Koozali.org: home of the SME Server

cvm_unix crashes during SMTP hack attempts

Offline Michail Pappas

  • *
  • 339
  • +1/-0
cvm_unix crashes during SMTP hack attempts
« on: December 07, 2021, 07:21:18 AM »
(Note: this is a different issue than the one described in #11792: focus here is on why auth_cvm_unix_local is crashing and not on why it doesn't always startup after a crash...)

Brute force cracking attempts seem to be taking down cvm-unix. Four such attempts made cvm-unix crash, the reason I'm posting them here is due to the reference in /var/log/qpsmtpd/current about an uninitialized value $ret. In the snippet below, first line is from /var/log/messages, second block is from /var/log/qpsmtpd/current and third one from running journalctl -u cvm-unix.service:

Code: [Select]
Dec  6 13:32:25 mail kernel: [280001.287486] cvm-unix[18933]: segfault at 0 ip 00007ff68e404037 sp 00007ffccc142fc8 error 4 in libc-2.17.so[7ff68e2c5000+1c4000]


Dec 06 13:32:25 mail.mydomain.gr systemd[1]: cvm-unix.service: main process exited, code=killed, status=11/SEGV
Dec 06 13:32:25 mail.mydomain.gr systemd[1]: Unit cvm-unix.service entered failed state.
Dec 06 13:32:25 mail.mydomain.gr systemd[1]: cvm-unix.service failed.
Dec 06 13:32:26 mail.mydomain.gr systemd[1]: cvm-unix.service holdoff time over, scheduling restart.
Dec 06 13:32:26 mail.mydomain.gr systemd[1]: Stopped Credential Validation Modules.
Dec 06 13:32:26 mail.mydomain.gr systemd[1]: Starting Credential Validation Modules...
Dec 06 13:32:26 mail.mydomain.gr systemd[1]: Started Credential Validation Modules.

2021-12-06 13:32:24.278943500 30919 Accepted connection 0/40 from bad.ip.2 / Unknown
2021-12-06 13:32:24.279312500 30919 Connection from Unknown [bad.ip.2]
2021-12-06 13:32:25.427041500 30919 (connect) earlytalker: pass, not spontaneous
2021-12-06 13:32:25.429481500 30919 (connect) relay: skip, no match
2021-12-06 13:32:25.453920500 30919 (connect) dnsbl: karma -1 (-1)
2021-12-06 13:32:25.454024500 30919 (connect) dnsbl: fail, NAUGHTY, zen.spamhaus.org
2021-12-06 13:32:25.454470500 30919 220 mail.mydomain.gr ESMTP
2021-12-06 13:32:25.524893500 30919 dispatching EHLO [bad.ip.2]
2021-12-06 13:32:25.529793500 30919 (ehlo) helo: karma -1 (-2)
2021-12-06 13:32:25.529939500 30919 (ehlo) helo: fail, NAUGHTY, no rDNS
2021-12-06 13:32:25.531254500 30919 250-mydomain.gr Hi Unknown [bad.ip.2]
2021-12-06 13:32:25.531364500 30919 250-PIPELINING
2021-12-06 13:32:25.531456500 30919 250-8BITMIME
2021-12-06 13:32:25.531550500 30919 250-SIZE 30000000
2021-12-06 13:32:25.531643500 30919 250-STARTTLS
2021-12-06 13:32:25.531733500 30919 250 AUTH PLAIN LOGIN
2021-12-06 13:32:25.616797500 30919 dispatching AUTH LOGIN
2021-12-06 13:32:25.617496500 30919 334 VXNlcm5hbWU6
2021-12-06 13:32:25.687290500 30919 334 UGFzc3dvcmQ6
2021-12-06 13:32:25.759328500 Use of uninitialized value $ret in unpack at /usr/share/qpsmtpd/plugins/auth/auth_cvm_unix_local line 124.
2021-12-06 13:32:25.759522500 30919 (auth-login) auth::auth_cvm_unix_local: skip: no response from cvm for operator
2021-12-06 13:32:25.759903500 30919 535 LOGIN authentication failed for operator
2021-12-06 13:32:25.830281500 30919 dispatching QUIT
2021-12-06 13:32:25.833171500 30919 221 mydomain.gr closing connection. Have a wonderful day.
2021-12-06 13:32:25.833173500 30919 click, disconnecting





Dec  6 15:15:25 mail kernel: [286181.300401] cvm-unix[30928]: segfault at 0 ip 00007f5807a81037 sp 00007ffde3a80b18 error 4 in libc-2.17.so[7f5807942000+1c4000]


Dec 06 15:15:25 mail.mydomain.gr systemd[1]: cvm-unix.service: main process exited, code=killed, status=11/SEGV
Dec 06 15:15:25 mail.mydomain.gr systemd[1]: Unit cvm-unix.service entered failed state.
Dec 06 15:15:25 mail.mydomain.gr systemd[1]: cvm-unix.service failed.
Dec 06 15:15:26 mail.mydomain.gr systemd[1]: cvm-unix.service holdoff time over, scheduling restart.
Dec 06 15:15:26 mail.mydomain.gr systemd[1]: Stopped Credential Validation Modules.
Dec 06 15:15:26 mail.mydomain.gr systemd[1]: Starting Credential Validation Modules...
Dec 06 15:15:26 mail.mydomain.gr systemd[1]: Started Credential Validation Modules.


2021-12-06 15:15:24.358558500 3079 Accepted connection 0/40 from bad.ip.3 / Unknown
2021-12-06 15:15:24.358966500 3079 Connection from Unknown [bad.ip.3]
2021-12-06 15:15:25.519747500 3079 (connect) earlytalker: pass, not spontaneous
2021-12-06 15:15:25.522334500 3079 (connect) relay: skip, no match
2021-12-06 15:15:25.549046500 3079 (connect) dnsbl: karma -1 (-1)
2021-12-06 15:15:25.549253500 3079 (connect) dnsbl: fail, NAUGHTY, zen.spamhaus.org
2021-12-06 15:15:25.549951500 3079 220 mail.mydomain.gr ESMTP
2021-12-06 15:15:25.623457500 3079 dispatching EHLO [bad.ip.3]
2021-12-06 15:15:25.628437500 3079 (ehlo) helo: karma -1 (-2)
2021-12-06 15:15:25.628579500 3079 (ehlo) helo: fail, NAUGHTY, no rDNS
2021-12-06 15:15:25.629838500 3079 250-mydomain.gr Hi Unknown [bad.ip.3]
2021-12-06 15:15:25.629963500 3079 250-PIPELINING
2021-12-06 15:15:25.630056500 3079 250-8BITMIME
2021-12-06 15:15:25.630187500 3079 250-SIZE 30000000
2021-12-06 15:15:25.630285500 3079 250-STARTTLS
2021-12-06 15:15:25.630386500 3079 250 AUTH PLAIN LOGIN
2021-12-06 15:15:25.698583500 3079 dispatching AUTH LOGIN
2021-12-06 15:15:25.699269500 3079 334 VXNlcm5hbWU6
2021-12-06 15:15:25.767238500 3079 334 UGFzc3dvcmQ6
2021-12-06 15:15:25.837383500 Use of uninitialized value $ret in unpack at /usr/share/qpsmtpd/plugins/auth/auth_cvm_unix_local line 124.
2021-12-06 15:15:25.837592500 3079 (auth-login) auth::auth_cvm_unix_local: skip: no response from cvm for adm
2021-12-06 15:15:25.837959500 3079 535 LOGIN authentication failed for adm
2021-12-06 15:15:25.907328500 3079 dispatching QUIT
2021-12-06 15:15:25.907331500 3079 221 mydomain.gr closing connection. Have a wonderful day.
2021-12-06 15:15:25.910170500 3079 click, disconnecting
2021-12-06 15:15:26.236406500 1041 cleaning up after 3079



Dec  6 15:28:42 mail kernel: [286978.082549] cvm-unix[3090]: segfault at 0 ip 00007f50e372f037 sp 00007ffc589f8268 error 4 in libc-2.17.so[7f50e35f0000+1c4000]

Dec 06 15:28:42 mail.mydomain.gr systemd[1]: cvm-unix.service: main process exited, code=killed, status=11/SEGV
Dec 06 15:28:42 mail.mydomain.gr systemd[1]: Unit cvm-unix.service entered failed state.
Dec 06 15:28:42 mail.mydomain.gr systemd[1]: cvm-unix.service failed.
Dec 06 15:28:42 mail.mydomain.gr systemd[1]: cvm-unix.service holdoff time over, scheduling restart.
Dec 06 15:28:42 mail.mydomain.gr systemd[1]: Stopped Credential Validation Modules.
Dec 06 15:28:42 mail.mydomain.gr systemd[1]: Starting Credential Validation Modules...
Dec 06 15:28:42 mail.mydomain.gr systemd[1]: Started Credential Validation Modules.


2021-12-06 15:28:41.241509500 3410 Accepted connection 0/40 from bad.ip.4 / Unknown
2021-12-06 15:28:41.241874500 3410 Connection from Unknown [bad.ip.4]
2021-12-06 15:28:42.401806500 3410 (connect) earlytalker: pass, not spontaneous
2021-12-06 15:28:42.404228500 3410 (connect) relay: skip, no match
2021-12-06 15:28:42.413695500 3410 (connect) dnsbl: karma -1 (-1)
2021-12-06 15:28:42.413817500 3410 (connect) dnsbl: fail, NAUGHTY, zen.spamhaus.org
2021-12-06 15:28:42.414326500 3410 220 mail.mydomain.gr ESMTP
2021-12-06 15:28:42.468059500 3410 dispatching EHLO User
2021-12-06 15:28:42.469861500 3410 (ehlo) helo: karma -1 (-2)
2021-12-06 15:28:42.470003500 3410 (ehlo) helo: fail, NAUGHTY, not FQDN
2021-12-06 15:28:42.471306500 3410 250-mydomain.gr Hi Unknown [bad.ip.1]
2021-12-06 15:28:42.471415500 3410 250-PIPELINING
2021-12-06 15:28:42.471506500 3410 250-8BITMIME
2021-12-06 15:28:42.471602500 3410 250-SIZE 30000000
2021-12-06 15:28:42.471694500 3410 250-STARTTLS
2021-12-06 15:28:42.471784500 3410 250 AUTH PLAIN LOGIN
2021-12-06 15:28:42.523279500 3410 dispatching AUTH LOGIN
2021-12-06 15:28:42.523908500 3410 334 VXNlcm5hbWU6
2021-12-06 15:28:42.574941500 3410 334 UGFzc3dvcmQ6
2021-12-06 15:28:42.627481500 Use of uninitialized value $ret in unpack at /usr/share/qpsmtpd/plugins/auth/auth_cvm_unix_local line 124.
2021-12-06 15:28:42.627704500 3410 (auth-login) auth::auth_cvm_unix_local: skip: no response from cvm for mysql
2021-12-06 15:28:42.628055500 3410 535 LOGIN authentication failed for mysql
2021-12-06 15:28:42.685314500 3410 dispatching QUIT
2021-12-06 15:28:42.685316500 3410 221 mydomain.gr closing connection. Have a wonderful day.
2021-12-06 15:28:42.685317500 3410 click, disconnecting


Dec  6 17:50:57 mail kernel: [295513.098775] cvm-unix[3421]: segfault at 0 ip 00007f2f29ab7037 sp 00007ffde2992b78 error 4 in libc-2.17.so[7f2f29978000+1c4000]

Dec 06 17:50:57 mail.mydomain.gr systemd[1]: cvm-unix.service: main process exited, code=killed, status=11/SEGV
Dec 06 17:50:57 mail.mydomain.gr systemd[1]: Unit cvm-unix.service entered failed state.
Dec 06 17:50:57 mail.mydomain.gr systemd[1]: cvm-unix.service failed.
Dec 06 17:50:58 mail.mydomain.gr systemd[1]: cvm-unix.service holdoff time over, scheduling restart.
Dec 06 17:50:58 mail.mydomain.gr systemd[1]: Stopped Credential Validation Modules.
Dec 06 17:50:58 mail.mydomain.gr systemd[1]: Starting Credential Validation Modules...
Dec 06 17:50:58 mail.mydomain.gr systemd[1]: Started Credential Validation Modules.


2021-12-06 17:50:56.207251500 5831 Accepted connection 0/40 from bad.ip.2 / Unknown
2021-12-06 17:50:56.207598500 5831 Connection from Unknown [bad.ip.2]
2021-12-06 17:50:57.379997500 5831 (connect) earlytalker: pass, not spontaneous
2021-12-06 17:50:57.382309500 5831 (connect) relay: skip, no match
2021-12-06 17:50:57.409439500 5831 (connect) dnsbl: karma -1 (-1)
2021-12-06 17:50:57.409617500 5831 (connect) dnsbl: fail, NAUGHTY, zen.spamhaus.org
2021-12-06 17:50:57.410393500 5831 220 mail.mydomain.gr ESMTP
2021-12-06 17:50:57.478911500 5831 dispatching EHLO [bad.ip.2]
2021-12-06 17:50:57.483670500 5831 (ehlo) helo: karma -1 (-2)
2021-12-06 17:50:57.483812500 5831 (ehlo) helo: fail, NAUGHTY, no rDNS
2021-12-06 17:50:57.485134500 5831 250-mydomain.gr Hi Unknown [bad.ip.2]
2021-12-06 17:50:57.485244500 5831 250-PIPELINING
2021-12-06 17:50:57.485336500 5831 250-8BITMIME
2021-12-06 17:50:57.485432500 5831 250-SIZE 30000000
2021-12-06 17:50:57.485537500 5831 250-STARTTLS
2021-12-06 17:50:57.485628500 5831 250 AUTH PLAIN LOGIN
2021-12-06 17:50:57.565743500 5831 dispatching AUTH LOGIN
2021-12-06 17:50:57.566568500 5831 334 VXNlcm5hbWU6
2021-12-06 17:50:57.662779500 5831 334 UGFzc3dvcmQ6
2021-12-06 17:50:57.735881500 Use of uninitialized value $ret in unpack at /usr/share/qpsmtpd/plugins/auth/auth_cvm_unix_local line 124.
2021-12-06 17:50:57.736107500 5831 (auth-login) auth::auth_cvm_unix_local: skip: no response from cvm for operator
2021-12-06 17:50:57.736480500 5831 535 LOGIN authentication failed for operator
2021-12-06 17:50:57.813278500 5831 dispatching QUIT
2021-12-06 17:50:57.815381500 5831 221 mydomain.gr closing connection. Have a wonderful day.
2021-12-06 17:50:57.815384500 5831 click, disconnecting
2021-12-06 17:50:58.072258500 1041 cleaning up after 5831

Hope these logs contain something useful for the devs to pin this issue. In all cases I've cvm_unix crashing is during a failed SMTP connect attempt from an unknown ip address.
« Last Edit: December 07, 2021, 07:25:46 AM by Michail Pappas »

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: cvm_unix crashes during SMTP hack attempts
« Reply #1 on: December 07, 2021, 01:19:09 PM »
Michail - these things should really be in the bug tracker. You are splattering information everywhere with no consistency making it very difficult to keep track and it takes a lot of time to try and tie it up and then respond correctly. Time which could be better spent on say building the new DMARC plugin that I am trying to do....

We also have this:
https://forums.contribs.org/index.php/topic,54708.0.html

And this:
https://forums.contribs.org/index.php/topic,54571.0.html

And bugs.

Please stick to one thread or bug per topic. They are all essentially related to one issue which is cvm-unix which is the plugin is used by both sqpsmtpd and qpsmtpd.

Quote
not on why it doesn't always startup after a crash...

JP can probably correct me if I am wrong on some of the following stuff.

As we stated in 11792 the service will automagically restart.

There is no evidence that it does not. Please refer to the bug for comments.

Quote
the reason I'm posting them here is due to the reference in /var/log/qpsmtpd/current about an uninitialized value $ret.

https://github.com/smtpd/qpsmtpd/blob/master/plugins/auth/auth_cvm_unix_local

Or /usr/share/qpsmtpd/plugins/auth/auth_cvm_unix_local

Clearly the socket is created or you would get connection failed here.

Code: [Select]
107    connect(SOCK, sockaddr_un($self->{_cvm_socket})) or do {
108        $self->log(LOGERROR, "skip: socket connection attempt for: $user");
109        return DENY, "authcvm, connection failed";
110    };

Your 'error' is here:

Code: [Select]
123    my $ret = <SOCK>;
124    my ($s) = unpack("C", $ret);

And correctly returned here:

Code: [Select]
126     if (!defined $s) {
127         $self->log(LOGERROR, "skip: no response from cvm for $user");
128         return DECLINED;
129     }

Have a read of the code in the plugin.

Yes, it could possibly be considered a bug, but that is really in the plugin which is upstream and they rarely accept patches. The plugin does not stop, and reports the correct error. Why it does not get a return from the socket I do not know - presumably it dies. But the $ret error is effectively log noise.

However, this is still related to cvm-unix used by both qpsmtp and sqpsmtpd so one bug will cover both areas.

We are aware of the cvm-unix issue. We are looking at updating the code but the latest code is not designed to run on Cent 7 and we currently have some build issues. Even then it may not fix the issue and as a result we are also looking at other alternatives to using it. Unfortunately we are all overwhelmed with stuff at the minute so our dev time is limited.

See: https://bugs.koozali.org/show_bug.cgi?id=11315

Note per your previous comments  on 11792

Quote
Stability regarding the core functionality should be our main concern here. Never had issues with 9.2, could this be a CentOS 7 thing?

It is our main concern. However. We did as much testing as we could before release, but despite multiple calls for support and help we were hampered by lack of users helping us.

We try our best to ensure that things are tested as well as possible, but good QA means lots of testing, and quite simply almost nobody bothered. And now people jump up and down and ask why things were not picked up which is 'irritating'.

This is open source. It is a collective responsibility. That means everyone is responsible. Not just the few who actually bothered to write the code, and test it.

For anyone experiencing this issues you can do the following to mitigate it:

1. Use full checking on s/qpmsptd - do not bypass any of the filters/plugins that are designed to keep the bad people out
2. Use failban to block multiple login attempts
3. Use Geoip IP or xt_tables to limit attacks

My qpsmtpd geoip:
 BadCountries=RU,VN,TF,CN,RO,MX,MY,ID,IR,JP,KR,AR,PH,HK,TH,IL,AE,TW,RS,CO,BO,BD,BG,SN,NG,UA,CZ,LT,SK,IQ,NP,IN,TR,EE,BR,PA


Note - when I first upgraded I had a few of these errors. Since installing fail2ban and geoip I have had none..... YMMV.

We will let users know as and when we have a fix - either an updated cvm rpm, or alternative authentication.


...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Michail Pappas

  • *
  • 339
  • +1/-0
Re: cvm_unix crashes during SMTP hack attempts
« Reply #2 on: December 07, 2021, 02:04:19 PM »
Was writing a response for 30' that got lost due to cookie expiration. Too tired and too pissed with this loss to continue right now. Perhaps over rocket chat at some point.

Offline Michail Pappas

  • *
  • 339
  • +1/-0
Re: cvm_unix crashes during SMTP hack attempts
« Reply #3 on: December 08, 2021, 09:42:43 AM »
There goes another 30' to respond:

Michail - these things should really be in the bug tracker. You are splattering information everywhere with no consistency making it very difficult to keep track and it takes a lot of time to try and tie it up and then respond correctly. Time which could be better spent on say building the new DMARC plugin that I am trying to do....
I believe I've been trying to isolate incidents of different nature to separate threads, regardless if the same component is responsible, simply because that's the proper way to do it from a non-dev side. Lacking the experience to actually say whether they are the same issue, I prefer to keep things isolated. A dev can merge threads, if he deems it reasonable.

Topic in hand, this post here is about the why's cvm is dying...

Quote
We also have this:
https://forums.contribs.org/index.php/topic,54708.0.html
... whereas this one is about cvm-unix not restarting after a crash (which in my mind might simply mean that other factors might be at play) ...

Quote
And this:
https://forums.contribs.org/index.php/topic,54571.0.html
... and this is about mitigating.

Quote
Please stick to one thread or bug per topic. They are all essentially related to one issue which is cvm-unix which is the plugin is used by both sqpsmtpd and qpsmtpd.
Please do treat me like an end user, that is actually following a "stick to one thread/bug per topic" rule in his perception! Let me know which thread you'd like me to provide my feedback on. Just don't tell me that I should be creating a new thread/bug for my feedback...

Quote
JP can probably correct me if I am wrong on some of the following stuff.

As we stated in 11792 the service will automagically restart.

There is no evidence that it does not. Please refer to the bug for comments.

We have a corpse, but no murder weapon: I'm waiting for the issue to re-surface and grab some logs for the devs.


Quote
https://github.com/smtpd/qpsmtpd/blob/master/plugins/auth/auth_cvm_unix_local

Or /usr/share/qpsmtpd/plugins/auth/auth_cvm_unix_local

Clearly the socket is created or you would get connection failed here.

[...]

Have a read of the code in the plugin.

Yes, it could possibly be considered a bug, but that is really in the plugin which is upstream and they rarely accept patches. The plugin does not stop, and reports the correct error. Why it does not get a return from the socket I do not know - presumably it dies. But the $ret error is effectively log noise.

However, this is still related to cvm-unix used by both qpsmtp and sqpsmtpd so one bug will cover both areas.
I can not read this code, it is incomprehensible to me. Therefore I cannot deduce which part of the code is at fault

Quote
Note per your previous comments  on 11792

It is our main concern. However. We did as much testing as we could before release, but despite multiple calls for support and help we were hampered by lack of users helping us.

We try our best to ensure that things are tested as well as possible, but good QA means lots of testing, and quite simply almost nobody bothered. And now people jump up and down and ask why things were not picked up which is 'irritating'.

This is open source. It is a collective responsibility. That means everyone is responsible. Not just the few who actually bothered to write the code, and test it.
Per my previous comments, I was discussing that core functionality stability should be prioritized. Over contrib fixes, currently at play for example. It was not hinting that the Q&A procedure was below par.

As for your comments regarding lack of testing before release, I believe that they are OT here, as well as the references to up-and-down-'irritating'-whiners... Let's keep this civil, I don't know you and you definitely don't know me.

Quote
For anyone experiencing this issues you can do the following to mitigate it:

1. Use full checking on s/qpmsptd - do not bypass any of the filters/plugins that are designed to keep the bad people out
2. Use failban to block multiple login attempts
3. Use Geoip IP or xt_tables to limit attacks

My qpsmtpd geoip:
 BadCountries=RU,VN,TF,CN,RO,MX,MY,ID,IR,JP,KR,AR,PH,HK,TH,IL,AE,TW,RS,CO,BO,BD,BG,SN,NG,UA,CZ,LT,SK,IQ,NP,IN,TR,EE,BR,PA


Note - when I first upgraded I had a few of these errors. Since installing fail2ban and geoip I have had none..... YMMV.
Personal feedback: I was still having the issue with 1 and 2 in place. Possibly tuning fail2ban can lower cvm crashes. Can't do 3 unfortunately, since we receive mail from all over the world.

Quote
We will let users know as and when we have a fix - either an updated cvm rpm, or alternative authentication.
Always appreciated your efforts. Major reason for sticking to SME.

Offline Jean-Philippe Pialasse

  • *
  • 2,747
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: cvm_unix crashes during SMTP hack attempts
« Reply #4 on: December 08, 2021, 01:39:12 PM »
you can still protect you sqpsmtpd with geoip allowing only your country and do not authorize auth on qpsmtpd. this will lower the incidence greatly. 

however we might need your feedback for the next event.
please see in rocket

Offline Michail Pappas

  • *
  • 339
  • +1/-0
Re: cvm_unix crashes during SMTP hack attempts
« Reply #5 on: December 08, 2021, 05:34:47 PM »
you can still protect you sqpsmtpd with geoip allowing only your country and do not authorize auth on qpsmtpd. this will lower the incidence greatly.

Not sure I understand this in terms of how it works and how I should approach it for configuration.