Koozali.org: home of the SME Server

SPAM-marked messages are not downloadable via POP-3

Offline Michail Pappas

  • *
  • 351
  • +1/-0
SPAM-marked messages are not downloadable via POP-3
« on: September 10, 2024, 10:00:09 AM »
(Note: I am aware that SME 10.1 is EOL, however there is no alternative for me for the time being).

We have a 10.1 SME box, minimal contribs (fail2ban basically). Today a user complained about not receiving email from a certain sender. Upon closer inspection, email was flagged as spam, but spam score did not exceed the spam threshold, so it was delivered to the queue.

However nothing appeared on the email client (thunderbird / POP-3) of the user. Checking the user's home directory it seems as though SPAM-marked messages ended under ~user/Maildir/.junkmail/new which, if I understand correctly, is a Horde thing.

I have to move all .junkmail residing messages to the respective ~user/Maildir/new (inbox) directory, however there are more than 100 mailboxes, so basically I'm afraid I might bork something here.

This behaviour seemed to have started some time before June 15th. What might have been installed at that time that modified SME behaviour this way? And more importantly, is it fixable? We have a large volume of mail messages and a lot of them are serious ones, having legal consequences.

Thanks in advance for any help provided!

EDIT: Open bug report https://bugs.koozali.org/show_bug.cgi?id=12733
« Last Edit: September 10, 2024, 10:22:04 AM by Michail Pappas »

Offline Stefano

  • *
  • 10,889
  • +3/-0
Re: SPAM-marked messages are sent to ~user/Maildir/.junkmail/new
« Reply #1 on: September 10, 2024, 10:21:31 AM »
this is the normal behaviour if you have spamassassin's SortSpam enabled
please post the result of
Code: [Select]
config show spamassassin | grep SortSpam

are the messages flagged as SPAM?

in any case, pop3 retrieves only INBOX messages; that's why you'd always use IMAP (or webmail)

Offline Michail Pappas

  • *
  • 351
  • +1/-0
Re: SPAM-marked messages are sent to ~user/Maildir/.junkmail/new
« Reply #2 on: September 10, 2024, 10:25:59 AM »
this is the normal behaviour if you have spamassassin's SortSpam enabled
please post the result of
Code: [Select]
config show spamassassin | grep SortSpam
Code: [Select]
# config show spamassassin | grep SortSpam
    SortSpam=enabled

Quote
are the messages flagged as SPAM?
Yes, they are.

Quote
in any case, pop3 retrieves only INBOX messages; that's why you'd always use IMAP (or webmail)

Fully understood. Now, is SortSpam a new feature? Because from all those years operating my SME box, the earliest these spam files appear (by searching in random users' .junkdirs) is June 15th this year. Wouldn't one expect to have a zillion spam files, dated from all these years back?
« Last Edit: September 10, 2024, 10:28:53 AM by Michail Pappas »

Offline Stefano

  • *
  • 10,889
  • +3/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #3 on: September 10, 2024, 10:27:25 AM »
sortspam is there since.... well, I'd say SME5

Offline Michail Pappas

  • *
  • 351
  • +1/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #4 on: September 10, 2024, 10:31:05 AM »
sortspam is there since.... well, I'd say SME5

That's strange then....

In any case, in order to fix these for future emails I just have to disable this, do the post-upgrade dance and reboot, right?

Offline Stefano

  • *
  • 10,889
  • +3/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #5 on: September 10, 2024, 10:34:56 AM »
That's strange then....

In any case, in order to fix these for future emails I just have to disable this, do the post-upgrade dance and reboot, right?

just change to disabled from server-manager and you're done
if you have to do via CLI
Code: [Select]
config setprop spamassassin SortSpam disabled
signal-event email-update

check the syntax and the case, I'm far from a server right now and can't check

Offline Michail Pappas

  • *
  • 351
  • +1/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #6 on: September 10, 2024, 10:51:28 AM »
Went like a charm!

Now, if only someone could fire up a small script to move all those messages .junkmail to the inbox, in order for POP-3 to pick them up :whistling:  :roll:

EDIT: Was not difficult, did it!
« Last Edit: September 10, 2024, 10:55:37 AM by Michail Pappas »

Offline ReetP

  • *
  • 3,911
  • +6/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #7 on: September 10, 2024, 12:29:27 PM »
Not sure why you are using POP3?

You should use IMAP which is more secure, and leave them on the server where they can be backed up.

It's also easy to move them about then as well.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline mmccarn

  • *
  • 2,649
  • +10/-0
Re: SPAM-marked messages are sent to ~user/Maildir/.junkmail/new
« Reply #8 on: September 10, 2024, 12:45:15 PM »
... from all those years operating my SME box, the earliest these spam files appear (by searching in random users' .junkdirs) is June 15th this year. Wouldn't one expect to have a zillion spam files, dated from all these years back?

spamassassin -> "MessageRetentionTime" keeps user .junkmail folders from filling up your server:

# config show spamassassin
spamassassin=service
    BayesAutoLearnThresholdNonspam=0.10
    BayesAutoLearnThresholdSpam=12.00
    DNSAvailable=yes
    MessageRetentionTime=90
    OkLanguages=all
    OkLocales=all
    RejectLevel=50
    ReportSafe=0
    Sensitivity=custom
    SkipRBLChecks=0
    SortSpam=enabled
    SpamLearning=enabled
    Subject=[SPAM]
    SubjectTag=disabled
    TagLevel=5
    UseBayes=1
    UseBayesAutoLearn=1
    status=enabled

Offline mmccarn

  • *
  • 2,649
  • +10/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #9 on: September 10, 2024, 02:39:18 PM »
I don't have any mail on my SME server any more, so I can't test this, but...

Here's a quick command that
* scans all users except admin
* for messages in <user>/Maildir/$FOLDER
* with ctime in the last DAYS days
* outputs username, date, subject, from, and to

Code: [Select]
DAYS=7
FOLDER=.junkmail/new
cd /home/e-smith/files/users
# header, in case you want to open the output in excel...
printf "user\tdate\tsubject\tfrom\tto\n"
for user in *; do
  if [[ "$user" != "admin" ]];
  then
    for msg in $(find $user/Maildir/$FOLDER/ -ctime -$DAYS -type f);
    do
      # 4 distinct 'grep's to create output field order consistently
      # (another option would be 'egrep '^Date: |^Subject: |^From: |^To: ' |sort |tr "\n" "\t",
      # but then the output is Date - From - Subject - To...)
      printf "$user\t"
      printf "$(grep '^Date: ' "$msg" |sed 's/\r//')\t"
      printf "$(grep '^Subject: ' "$msg" |sed 's/\r//')\t"
      printf "$(grep '^From: ' "$msg" |sed 's/\r//')\t"
      printf "$(grep '^To: ' "$msg" |sed 's/\r//')\t"
      printf "\n"
    done
  fi
done

Offline Stefano

  • *
  • 10,889
  • +3/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #10 on: September 10, 2024, 02:44:16 PM »
alternatively, OP could use imapsync to move messages between folders..
the only requirement is that admin (or another user) must be dovecot master user


Offline mmccarn

  • *
  • 2,649
  • +10/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #11 on: September 10, 2024, 02:47:13 PM »
One more thought -

Most email clients have built-in spam sorting functionality.

If your spam is now being delivered to user Inboxes:
* Some of the SPAM may be automatically filed in Spam or Junk by the email client
* Potentially malicious email formerly hidden from users may now be accessible

Offline Michail Pappas

  • *
  • 351
  • +1/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #12 on: September 11, 2024, 06:35:46 AM »

Not sure why you are using POP3?
Leftovers from a time server storage was pretty high and I did not have a sufficiently powerful server.

Quote
spamassassin -> "MessageRetentionTime" keeps user .junkmail folders from filling up your server:
@mmcarn: that explains it mate, thanks! I really can't believe that after all these years of operation, it's only just now that I've stumbled upon this issue and its simple explanation...

One more thought -

Most email clients have built-in spam sorting functionality.

If your spam is now being delivered to user Inboxes:
* Some of the SPAM may be automatically filed in Spam or Junk by the email client
* Potentially malicious email formerly hidden from users may now be accessible

Thunderbird does have anti-spam measures and pretty good too (possibly also SA-based? ) however, I've disabled them for the fear of false positives. No matter how much training I've thrown in, for the majority of my users looking into a special folder daily is a feat.

Offline sages

  • *
  • 189
  • +0/-0
    • http://www.sages.com.au
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #13 on: September 11, 2024, 09:25:18 AM »
No matter how much training I've thrown in, for the majority of my users looking into a special folder daily is a feat.
Make it a kpi or tell them if they find the bonus email in there they get an extra day added to their holidays.
Failing that tell them there will be an email in the folder that is their resignation unless they send a rebuttal.
...

Offline Michail Pappas

  • *
  • 351
  • +1/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #14 on: September 11, 2024, 09:54:47 AM »
Make it a kpi or tell them if they find the bonus email in there they get an extra day added to their holidays.
Failing that tell them there will be an email in the folder that is their resignation unless they send a rebuttal.
Been rolling laughing at this. Yeah, could be a decent approach in the private sector. But in the public sector I'm in, THESE employees could get away with almost anything; doing some major bummer by failing to notice that mail in the spam folder would not scratch them...

Some things I've come to accept with the passage of time, that's just how it is.

Offline ReetP

  • *
  • 3,911
  • +6/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #15 on: September 11, 2024, 04:47:34 PM »
Leftovers from a time server storage was pretty high and I did not have a sufficiently powerful server.

> We have a large volume of mail messages and a lot of them are serious ones, having legal consequences.

In the context of this, and GDPR, and security, and backups, and loads of cheap storage, I really don't understand why you use POP3.

What happens if a client goes down and you lose their mail? If you say it leaves a copy on the server then that demolishes the storage issue and you may as well just use IMAP which is far more secure.

Where is all their sent mail which is part of the legal conversation?

Sorry - it just doesn't make any sense!


...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Stefano

  • *
  • 10,889
  • +3/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #16 on: September 11, 2024, 05:37:48 PM »
Been rolling laughing at this. Yeah, could be a decent approach in the private sector. But in the public sector I'm in, THESE employees could get away with almost anything; doing some major bummer by failing to notice that mail in the spam folder would not scratch them...

Some things I've come to accept with the passage of time, that's just how it is.

I managed a mail server for a small town municipality here in italy.

The first thing IT manager told me many, many years ago was: "I don't want any email client here.. I don't want to loose email because a f****ng dead hd or a virus on the client pc"
we moved all on SME and SOGo.. about 300 users.. none of them had used a webmail before.. but we did it.
The guy told he owes me at least 12 bottles of wine..
note to self: get in touch with him and go to the wine :-D

I work with/for medium enterprise with structured IT dept.. Sometimes users try to steer the IT decisions, but at the end, it's the IT dept that decides and users are forced to adhere to the rules. note easy, but the only way to go

Offline Michail Pappas

  • *
  • 351
  • +1/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #17 on: September 12, 2024, 09:07:07 AM »
> We have a large volume of mail messages and a lot of them are serious ones, having legal consequences.

In the context of this, and GDPR, and security, and backups, and loads of cheap storage, I really don't understand why you use POP3.
You're correct in that it does not make sense, but I have had no time to deal with it as well. There's a backlog of a zillion things to do, this would be another bullet in it.

Unfortunately, it has been announced that we'll be moving to M365 by the end of the year, hence no point in spending non-existent available time to something that will go to the wastebasket. I was against this decision, since it implicated moving mails and documents to Azure. Seems I'm the only one shouting.

@Stefano: that's a great success story there, glad it all worked out.

But not all stories get a happy ending, sometimes resources (personnel time+money) are simply not there for things to happen.

Wrote this while having a small break eating something. Don't know if I'll have the time to philosophize soon again.

Offline Stefano

  • *
  • 10,889
  • +3/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #18 on: September 12, 2024, 09:18:17 AM »
You're correct in that it does not make sense, but I have had no time to deal with it as well. There's a backlog of a zillion things to do, this would be another bullet in it.

Unfortunately, it has been announced that we'll be moving to M365 by the end of the year, hence no point in spending non-existent available time to something that will go to the wastebasket. I was against this decision, since it implicated moving mails and documents to Azure. Seems I'm the only one shouting.

I use/manage 365 everyday.. once setup, it works.. yes, it lacks some debug tools I used to have in SME, but I can survive..
to be honest, debug tools are there, but sometimes it's not easy to find them and understand the result, but that's it
about moving.. migration is quite easy and almost fully automated.
forget about using any other client than outlook.. they simply won't work and if they will, you'll lose may features and integrations

Quote
@Stefano: that's a great success story there, glad it all worked out.

But not all stories get a happy ending, sometimes resources (personnel time+money) are simply not there for things to happen.

Wrote this while having a small break eating something. Don't know if I'll have the time to philosophize soon again.

I know what you mean, been in your shoes many times in the past

Offline Gary Douglas

  • *
  • 80
  • +1/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #19 on: September 12, 2024, 10:05:07 AM »
ps, received this the other day,

We're writing to let you know about a change that Microsoft and Google are making, that may affect you if you send emails using SMTP.
What's changing?
Microsoft are retiring Basic Authentication on Outlook.com, and Google are retiring Basic Authentication on Gmail.
You're only affected by this change to Basic Authentication if you send emails directly from your Sage software using SMTP via Outlook.com or Gmail.
When do the changes take effect?
Microsoft plan to make this change on 16 September 2024.
Google plan to make this change from 30 September 2024.

may not be changing on M365 but you never know with microsoft
« Last Edit: September 12, 2024, 10:10:34 AM by Gary Douglas »

Offline Stefano

  • *
  • 10,889
  • +3/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #20 on: September 12, 2024, 10:36:15 AM »
ps, received this the other day,

We're writing to let you know about a change that Microsoft and Google are making, that may affect you if you send emails using SMTP.
What's changing?
Microsoft are retiring Basic Authentication on Outlook.com, and Google are retiring Basic Authentication on Gmail.
You're only affected by this change to Basic Authentication if you send emails directly from your Sage software using SMTP via Outlook.com or Gmail.
When do the changes take effect?
Microsoft plan to make this change on 16 September 2024.
Google plan to make this change from 30 September 2024.

may not be changing on M365 but you never know with microsoft

big guys like gmail and 365 are trying to limit spam and phishing problems..
google has some subnets (/16) in BL because of spam

and nowadays 2FA is a must..
so nothing new under the sun

Offline ReetP

  • *
  • 3,911
  • +6/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #21 on: September 16, 2024, 03:34:31 PM »
big guys like gmail and 365 are trying to limit spam and phishing problems..

They ought to shut themselves down and do the rest of us a favour ;-)

So much junk comes from one or the other.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Stefano

  • *
  • 10,889
  • +3/-0
Re: SPAM-marked messages are not downloadable via POP-3
« Reply #22 on: September 16, 2024, 04:38:01 PM »
They ought to shut themselves down and do the rest of us a favour ;-)

So much junk comes from one or the other.


indeed, but the bigger the infra, the harder is to block violated accounts..
there are sooo many accounts with trivial password and no MFA.

You know I am not a 365/gmail fan, but your position is valid for every big ISP/Hoster.. for example, OVH has zillions of VM with unmaintained Wordpress instances..