Topic: SPAM-marked messages are not downloadable via POP-3

[Michail Pappas]

(Note: I am aware that SME 10.1 is EOL, however there is no alternative for me for the time being).

We have a 10.1 SME box, minimal contribs (fail2ban basically). Today a user complained about not receiving email from a certain sender. Upon closer inspection, email was flagged as spam, but spam score did not exceed the spam threshold, so it was delivered to the queue.

However nothing appeared on the email client (thunderbird / POP-3) of the user. Checking the user's home directory it seems as though SPAM-marked messages ended under ~user/Maildir/.junkmail/new which, if I understand correctly, is a Horde thing.

I have to move all .junkmail residing messages to the respective ~user/Maildir/new (inbox) directory, however there are more than 100 mailboxes, so basically I'm afraid I might bork something here.

This behaviour seemed to have started some time before June 15th. What might have been installed at that time that modified SME behaviour this way? And more importantly, is it fixable? We have a large volume of mail messages and a lot of them are serious ones, having legal consequences.

Thanks in advance for any help provided!

EDIT: Open bug report https://bugs.koozali.org/show_bug.cgi?id=12733

[Stefano]

this is the normal behaviour if you have spamassassin's SortSpam enabled
please post the result of

Code: [Select]

config show spamassassin | grep SortSpam

are the messages flagged as SPAM?

in any case, pop3 retrieves only INBOX messages; that's why you'd always use IMAP (or webmail)

[Michail Pappas]

this is the normal behaviour if you have spamassassin's SortSpam enabled
please post the result of

Code: [Select]

config show spamassassin | grep SortSpam


Code: [Select]

# config show spamassassin | grep SortSpam
    SortSpam=enabled


are the messages flagged as SPAM?

Yes, they are.

in any case, pop3 retrieves only INBOX messages; that's why you'd always use IMAP (or webmail)

Fully understood. Now, is SortSpam a new feature? Because from all those years operating my SME box, the earliest these spam files appear (by searching in random users' .junkdirs) is June 15th this year. Wouldn't one expect to have a zillion spam files, dated from all these years back?

[Stefano]

sortspam is there since.... well, I'd say SME5

[Michail Pappas]

sortspam is there since.... well, I'd say SME5

That's strange then....

In any case, in order to fix these for future emails I just have to disable this, do the post-upgrade dance and reboot, right?

[Stefano]

That's strange then....

In any case, in order to fix these for future emails I just have to disable this, do the post-upgrade dance and reboot, right?

just change to disabled from server-manager and you're done
if you have to do via CLI

Code: [Select]

config setprop spamassassin SortSpam disabled
signal-event email-update

check the syntax and the case, I'm far from a server right now and can't check

[Michail Pappas]

Went like a charm!

Now, if only someone could fire up a small script to move all those messages .junkmail to the inbox, in order for POP-3 to pick them up :whistling: 

EDIT: Was not difficult, did it!

[ReetP]

Not sure why you are using POP3?

You should use IMAP which is more secure, and leave them on the server where they can be backed up.

It's also easy to move them about then as well.

[mmccarn]

... from all those years operating my SME box, the earliest these spam files appear (by searching in random users' .junkdirs) is June 15th this year. Wouldn't one expect to have a zillion spam files, dated from all these years back?

spamassassin -> "MessageRetentionTime" keeps user .junkmail folders from filling up your server:

# config show spamassassin
spamassassin=service
    BayesAutoLearnThresholdNonspam=0.10
    BayesAutoLearnThresholdSpam=12.00
    DNSAvailable=yes
    MessageRetentionTime=90
    OkLanguages=all
    OkLocales=all
    RejectLevel=50
    ReportSafe=0
    Sensitivity=custom
    SkipRBLChecks=0
    SortSpam=enabled
    SpamLearning=enabled
    Subject=[SPAM]
    SubjectTag=disabled
    TagLevel=5
    UseBayes=1
    UseBayesAutoLearn=1
    status=enabled

[mmccarn]

I don't have any mail on my SME server any more, so I can't test this, but...

Here's a quick command that
* scans all users except admin
* for messages in <user>/Maildir/$FOLDER
* with ctime in the last DAYS days
* outputs username, date, subject, from, and to

Code: [Select]

DAYS=7
FOLDER=.junkmail/new
cd /home/e-smith/files/users
# header, in case you want to open the output in excel...
printf "user\tdate\tsubject\tfrom\tto\n"
for user in *; do
  if [[ "$user" != "admin" ]];
  then
    for msg in $(find $user/Maildir/$FOLDER/ -ctime -$DAYS -type f);
    do
      # 4 distinct 'grep's to create output field order consistently
      # (another option would be 'egrep '^Date: |^Subject: |^From: |^To: ' |sort |tr "\n" "\t",
      # but then the output is Date - From - Subject - To...)
      printf "$user\t"
      printf "$(grep '^Date: ' "$msg" |sed 's/\r//')\t"
      printf "$(grep '^Subject: ' "$msg" |sed 's/\r//')\t"
      printf "$(grep '^From: ' "$msg" |sed 's/\r//')\t"
      printf "$(grep '^To: ' "$msg" |sed 's/\r//')\t"
      printf "\n"
    done
  fi
done


[Stefano]

alternatively, OP could use imapsync to move messages between folders..
the only requirement is that admin (or another user) must be dovecot master user

[mmccarn]

One more thought -

Most email clients have built-in spam sorting functionality.

If your spam is now being delivered to user Inboxes:
* Some of the SPAM may be automatically filed in Spam or Junk by the email client
* Potentially malicious email formerly hidden from users may now be accessible

[Michail Pappas]

Not sure why you are using POP3?

Leftovers from a time server storage was pretty high and I did not have a sufficiently powerful server.

spamassassin -> "MessageRetentionTime" keeps user .junkmail folders from filling up your server:

@mmcarn: that explains it mate, thanks! I really can't believe that after all these years of operation, it's only just now that I've stumbled upon this issue and its simple explanation...

One more thought -

Most email clients have built-in spam sorting functionality.

If your spam is now being delivered to user Inboxes:
* Some of the SPAM may be automatically filed in Spam or Junk by the email client
* Potentially malicious email formerly hidden from users may now be accessible

Thunderbird does have anti-spam measures and pretty good too (possibly also SA-based? ) however, I've disabled them for the fear of false positives. No matter how much training I've thrown in, for the majority of my users looking into a special folder daily is a feat.

[sages]

No matter how much training I've thrown in, for the majority of my users looking into a special folder daily is a feat.

Make it a kpi or tell them if they find the bonus email in there they get an extra day added to their holidays.
Failing that tell them there will be an email in the folder that is their resignation unless they send a rebuttal.

[Michail Pappas]

Make it a kpi or tell them if they find the bonus email in there they get an extra day added to their holidays.
Failing that tell them there will be an email in the folder that is their resignation unless they send a rebuttal.

Been rolling laughing at this. Yeah, could be a decent approach in the private sector. But in the public sector I'm in, THESE employees could get away with almost anything; doing some major bummer by failing to notice that mail in the spam folder would not scratch them...

Some things I've come to accept with the passage of time, that's just how it is.