Koozali.org: home of the SME Server
Contribs.org Forums => Koozali SME Server 10.x => Topic started by: Michail Pappas on September 10, 2024, 10:00:09 AM
-
(Note: I am aware that SME 10.1 is EOL, however there is no alternative for me for the time being).
We have a 10.1 SME box, minimal contribs (fail2ban basically). Today a user complained about not receiving email from a certain sender. Upon closer inspection, email was flagged as spam, but spam score did not exceed the spam threshold, so it was delivered to the queue.
However nothing appeared on the email client (thunderbird / POP-3) of the user. Checking the user's home directory it seems as though SPAM-marked messages ended under ~user/Maildir/.junkmail/new which, if I understand correctly, is a Horde thing.
I have to move all .junkmail residing messages to the respective ~user/Maildir/new (inbox) directory, however there are more than 100 mailboxes, so basically I'm afraid I might bork something here.
This behaviour seemed to have started some time before June 15th. What might have been installed at that time that modified SME behaviour this way? And more importantly, is it fixable? We have a large volume of mail messages and a lot of them are serious ones, having legal consequences.
Thanks in advance for any help provided!
EDIT: Open bug report https://bugs.koozali.org/show_bug.cgi?id=12733
-
this is the normal behaviour if you have spamassassin's SortSpam enabled
please post the result of
config show spamassassin | grep SortSpam
are the messages flagged as SPAM?
in any case, pop3 retrieves only INBOX messages; that's why you'd always use IMAP (or webmail)
-
this is the normal behaviour if you have spamassassin's SortSpam enabled
please post the result of
config show spamassassin | grep SortSpam
# config show spamassassin | grep SortSpam
SortSpam=enabled
are the messages flagged as SPAM?
Yes, they are.
in any case, pop3 retrieves only INBOX messages; that's why you'd always use IMAP (or webmail)
Fully understood. Now, is SortSpam a new feature? Because from all those years operating my SME box, the earliest these spam files appear (by searching in random users' .junkdirs) is June 15th this year. Wouldn't one expect to have a zillion spam files, dated from all these years back?
-
sortspam is there since.... well, I'd say SME5
-
sortspam is there since.... well, I'd say SME5
That's strange then....
In any case, in order to fix these for future emails I just have to disable this, do the post-upgrade dance and reboot, right?
-
That's strange then....
In any case, in order to fix these for future emails I just have to disable this, do the post-upgrade dance and reboot, right?
just change to disabled from server-manager and you're done
if you have to do via CLI
config setprop spamassassin SortSpam disabled
signal-event email-update
check the syntax and the case, I'm far from a server right now and can't check
-
Went like a charm!
Now, if only someone could fire up a small script to move all those messages .junkmail to the inbox, in order for POP-3 to pick them up :whistling: :roll:
EDIT: Was not difficult, did it!
-
Not sure why you are using POP3?
You should use IMAP which is more secure, and leave them on the server where they can be backed up.
It's also easy to move them about then as well.
-
... from all those years operating my SME box, the earliest these spam files appear (by searching in random users' .junkdirs) is June 15th this year. Wouldn't one expect to have a zillion spam files, dated from all these years back?
spamassassin -> "MessageRetentionTime" keeps user .junkmail folders from filling up your server:
# config show spamassassin
spamassassin=service
BayesAutoLearnThresholdNonspam=0.10
BayesAutoLearnThresholdSpam=12.00
DNSAvailable=yes
MessageRetentionTime=90
OkLanguages=all
OkLocales=all
RejectLevel=50
ReportSafe=0
Sensitivity=custom
SkipRBLChecks=0
SortSpam=enabled
SpamLearning=enabled
Subject=[SPAM]
SubjectTag=disabled
TagLevel=5
UseBayes=1
UseBayesAutoLearn=1
status=enabled
-
I don't have any mail on my SME server any more, so I can't test this, but...
Here's a quick command that
* scans all users except admin
* for messages in <user>/Maildir/$FOLDER
* with ctime in the last DAYS days
* outputs username, date, subject, from, and to
DAYS=7
FOLDER=.junkmail/new
cd /home/e-smith/files/users
# header, in case you want to open the output in excel...
printf "user\tdate\tsubject\tfrom\tto\n"
for user in *; do
if [[ "$user" != "admin" ]];
then
for msg in $(find $user/Maildir/$FOLDER/ -ctime -$DAYS -type f);
do
# 4 distinct 'grep's to create output field order consistently
# (another option would be 'egrep '^Date: |^Subject: |^From: |^To: ' |sort |tr "\n" "\t",
# but then the output is Date - From - Subject - To...)
printf "$user\t"
printf "$(grep '^Date: ' "$msg" |sed 's/\r//')\t"
printf "$(grep '^Subject: ' "$msg" |sed 's/\r//')\t"
printf "$(grep '^From: ' "$msg" |sed 's/\r//')\t"
printf "$(grep '^To: ' "$msg" |sed 's/\r//')\t"
printf "\n"
done
fi
done
-
alternatively, OP could use imapsync to move messages between folders..
the only requirement is that admin (or another user) must be dovecot master user
-
One more thought -
Most email clients have built-in spam sorting functionality.
If your spam is now being delivered to user Inboxes:
* Some of the SPAM may be automatically filed in Spam or Junk by the email client
* Potentially malicious email formerly hidden from users may now be accessible
-
Not sure why you are using POP3?
Leftovers from a time server storage was pretty high and I did not have a sufficiently powerful server.
spamassassin -> "MessageRetentionTime" keeps user .junkmail folders from filling up your server:
@mmcarn: that explains it mate, thanks! I really can't believe that after all these years of operation, it's only just now that I've stumbled upon this issue and its simple explanation...
One more thought -
Most email clients have built-in spam sorting functionality.
If your spam is now being delivered to user Inboxes:
* Some of the SPAM may be automatically filed in Spam or Junk by the email client
* Potentially malicious email formerly hidden from users may now be accessible
Thunderbird does have anti-spam measures and pretty good too (possibly also SA-based? ) however, I've disabled them for the fear of false positives. No matter how much training I've thrown in, for the majority of my users looking into a special folder daily is a feat.
-
No matter how much training I've thrown in, for the majority of my users looking into a special folder daily is a feat.
Make it a kpi or tell them if they find the bonus email in there they get an extra day added to their holidays.
Failing that tell them there will be an email in the folder that is their resignation unless they send a rebuttal.
-
Make it a kpi or tell them if they find the bonus email in there they get an extra day added to their holidays.
Failing that tell them there will be an email in the folder that is their resignation unless they send a rebuttal.
Been rolling laughing at this. Yeah, could be a decent approach in the private sector. But in the public sector I'm in, THESE employees could get away with almost anything; doing some major bummer by failing to notice that mail in the spam folder would not scratch them...
Some things I've come to accept with the passage of time, that's just how it is.
-
Leftovers from a time server storage was pretty high and I did not have a sufficiently powerful server.
> We have a large volume of mail messages and a lot of them are serious ones, having legal consequences.
In the context of this, and GDPR, and security, and backups, and loads of cheap storage, I really don't understand why you use POP3.
What happens if a client goes down and you lose their mail? If you say it leaves a copy on the server then that demolishes the storage issue and you may as well just use IMAP which is far more secure.
Where is all their sent mail which is part of the legal conversation?
Sorry - it just doesn't make any sense!
-
Been rolling laughing at this. Yeah, could be a decent approach in the private sector. But in the public sector I'm in, THESE employees could get away with almost anything; doing some major bummer by failing to notice that mail in the spam folder would not scratch them...
Some things I've come to accept with the passage of time, that's just how it is.
I managed a mail server for a small town municipality here in italy.
The first thing IT manager told me many, many years ago was: "I don't want any email client here.. I don't want to loose email because a f****ng dead hd or a virus on the client pc"
we moved all on SME and SOGo.. about 300 users.. none of them had used a webmail before.. but we did it.
The guy told he owes me at least 12 bottles of wine..
note to self: get in touch with him and go to the wine :-D
I work with/for medium enterprise with structured IT dept.. Sometimes users try to steer the IT decisions, but at the end, it's the IT dept that decides and users are forced to adhere to the rules. note easy, but the only way to go
-
> We have a large volume of mail messages and a lot of them are serious ones, having legal consequences.
In the context of this, and GDPR, and security, and backups, and loads of cheap storage, I really don't understand why you use POP3.
You're correct in that it does not make sense, but I have had no time to deal with it as well. There's a backlog of a zillion things to do, this would be another bullet in it.
Unfortunately, it has been announced that we'll be moving to M365 by the end of the year, hence no point in spending non-existent available time to something that will go to the wastebasket. I was against this decision, since it implicated moving mails and documents to Azure. Seems I'm the only one shouting.
@Stefano: that's a great success story there, glad it all worked out.
But not all stories get a happy ending, sometimes resources (personnel time+money) are simply not there for things to happen.
Wrote this while having a small break eating something. Don't know if I'll have the time to philosophize soon again.
-
You're correct in that it does not make sense, but I have had no time to deal with it as well. There's a backlog of a zillion things to do, this would be another bullet in it.
Unfortunately, it has been announced that we'll be moving to M365 by the end of the year, hence no point in spending non-existent available time to something that will go to the wastebasket. I was against this decision, since it implicated moving mails and documents to Azure. Seems I'm the only one shouting.
I use/manage 365 everyday.. once setup, it works.. yes, it lacks some debug tools I used to have in SME, but I can survive..
to be honest, debug tools are there, but sometimes it's not easy to find them and understand the result, but that's it
about moving.. migration is quite easy and almost fully automated.
forget about using any other client than outlook.. they simply won't work and if they will, you'll lose may features and integrations
@Stefano: that's a great success story there, glad it all worked out.
But not all stories get a happy ending, sometimes resources (personnel time+money) are simply not there for things to happen.
Wrote this while having a small break eating something. Don't know if I'll have the time to philosophize soon again.
I know what you mean, been in your shoes many times in the past
-
ps, received this the other day,
We're writing to let you know about a change that Microsoft and Google are making, that may affect you if you send emails using SMTP.
What's changing?
Microsoft are retiring Basic Authentication on Outlook.com, and Google are retiring Basic Authentication on Gmail.
You're only affected by this change to Basic Authentication if you send emails directly from your Sage software using SMTP via Outlook.com or Gmail.
When do the changes take effect?
Microsoft plan to make this change on 16 September 2024.
Google plan to make this change from 30 September 2024.
may not be changing on M365 but you never know with microsoft
-
ps, received this the other day,
We're writing to let you know about a change that Microsoft and Google are making, that may affect you if you send emails using SMTP.
What's changing?
Microsoft are retiring Basic Authentication on Outlook.com, and Google are retiring Basic Authentication on Gmail.
You're only affected by this change to Basic Authentication if you send emails directly from your Sage software using SMTP via Outlook.com or Gmail.
When do the changes take effect?
Microsoft plan to make this change on 16 September 2024.
Google plan to make this change from 30 September 2024.
may not be changing on M365 but you never know with microsoft
big guys like gmail and 365 are trying to limit spam and phishing problems..
google has some subnets (/16) in BL because of spam
and nowadays 2FA is a must..
so nothing new under the sun
-
big guys like gmail and 365 are trying to limit spam and phishing problems..
They ought to shut themselves down and do the rest of us a favour ;-)
So much junk comes from one or the other.
-
They ought to shut themselves down and do the rest of us a favour ;-)
So much junk comes from one or the other.
indeed, but the bigger the infra, the harder is to block violated accounts..
there are sooo many accounts with trivial password and no MFA.
You know I am not a 365/gmail fan, but your position is valid for every big ISP/Hoster.. for example, OVH has zillions of VM with unmaintained Wordpress instances..