Koozali.org: home of the SME Server

Obsolete Releases => SME Server 9.x => Topic started by: Michail Pappas on November 08, 2017, 06:52:56 AM

Title: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 08, 2017, 06:52:56 AM
For the last days I've been receiving a large number of freshclam update errors like the following one, on three different 9.2 sites (hence, not a LAN issue):

Code: [Select]
2017-11-08 03:42:46.095752500 ClamAV update process started at Wed Nov  8 03:42:46 2017
2017-11-08 03:42:46.096035500 main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
2017-11-08 03:42:47.589090500 WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
2017-11-08 03:42:47.589453500 WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
2017-11-08 03:42:47.589818500 WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
2017-11-08 03:42:47.590193500 WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
2017-11-08 03:42:47.590575500 WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
2017-11-08 03:42:47.590954500 ERROR: getpatch: Can't download daily-24011.cdiff from database.clamav.net
2017-11-08 03:42:47.624341500 WARNING: Incremental update failed, trying to download daily.cvd
2017-11-08 03:42:55.601291500 WARNING: Mirror 195.222.33.229 is not synchronized.
2017-11-08 03:42:55.679739500 Giving up on database.clamav.net...
2017-11-08 03:42:55.679740500 Update failed. Your network may be down or none of the mirrors listed in /etc/freshclam.conf is working. Check http://www.clamav.net/doc/mirrors-faq.html for possible reasons.
I've tried a number of things, like using /usr/bin/refreshclam as described in the wiki https://wiki.contribs.org/Clamav:freshclam_update#Freshclam_update but to avail. For the record, db.local.clamav.net resolves to :
Code: [Select]
# dig db.local.clamav.net

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.4 <<>> db.local.clamav.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58318
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;db.local.clamav.net.           IN      A

;; ANSWER SECTION:
db.local.clamav.net.    785     IN      CNAME   db.southeu.clamav.net.
db.southeu.clamav.net.  7       IN      A       195.222.33.229
db.southeu.clamav.net.  7       IN      A       193.92.150.194

;; Query time: 9 msec
;; SERVER: 192.168.1.2#53(192.168.1.2)
;; WHEN: Wed Nov  8 07:28:49 2017
;; MSG SIZE  rcvd: 94
Manually doing a freshclam -v:
Code: [Select]
]# freshclam -v
Current working dir is /var/clamav
Max retries == 6
ClamAV update process started at Wed Nov  8 07:29:56 2017
Using IPv6 aware code
Querying current.cvd.clamav.net
TTL: 612
Software version from DNS: 0.99.2
main.cvd version from DNS: 58
main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr                                                                           )
daily.cvd version from DNS: 24024
Retrieving http://db.local.clamav.net/daily-24011.cdiff
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from db.local.clamav.net
Retrieving http://db.local.clamav.net/daily-24011.cdiff
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from db.local.clamav.net
Retrieving http://db.local.clamav.net/daily-24011.cdiff
Ignoring mirror 195.222.33.229 (due to previous errors)
Ignoring mirror 193.92.150.194 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from db.local.clamav.net
Retrieving http://db.local.clamav.net/daily-24011.cdiff
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from db.local.clamav.net
Retrieving http://db.local.clamav.net/daily-24011.cdiff
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from db.local.clamav.net
Retrieving http://db.local.clamav.net/daily-24011.cdiff
Ignoring mirror 195.222.33.229 (due to previous errors)
Ignoring mirror 193.92.150.194 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from db.local.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
Whitelisting short-term blacklisted mirrors
Retrieving http://db.local.clamav.net/daily.cvd
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
WARNING: Can't download daily.cvd from db.local.clamav.net
Trying again in 5 secs...
[...]
TTL: 577
Software version from DNS: 0.99.2
main.cvd version from DNS: 58
main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
daily.cvd version from DNS: 24024
Retrieving http://database.clamav.net/daily-24011.cdiff
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
Retrieving http://database.clamav.net/daily-24011.cdiff
Ignoring mirror 195.222.33.229 (due to previous errors)
Ignoring mirror 193.92.150.194 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
Retrieving http://database.clamav.net/daily-24011.cdiff
Ignoring mirror 195.222.33.229 (due to previous errors)
Ignoring mirror 193.92.150.194 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
Retrieving http://database.clamav.net/daily-24011.cdiff
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
Retrieving http://database.clamav.net/daily-24011.cdiff
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
WARNING: getpatch: Can't download daily-24011.cdiff from database.clamav.net
Retrieving http://database.clamav.net/daily-24011.cdiff
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
ERROR: getpatch: Can't download daily-24011.cdiff from database.clamav.net
WARNING: Incremental update failed, trying to download daily.cvd
Whitelisting short-term blacklisted mirrors
Retrieving http://database.clamav.net/daily.cvd
Ignoring mirror 193.92.150.194 (due to previous errors)
Ignoring mirror 195.222.33.229 (due to previous errors)
ERROR: Can't download daily.cvd from database.clamav.net
Giving up on database.clamav.net...
Update failed. Your network may be down or none of the mirrors listed in /etc/freshclam.conf is working. Check http://www.clamav.net/doc/mirrors-faq.html for possible reasons.

I've also tried using another DatabaseMirror. Same problem. Finally I deleted the failed mirrors and started again, using the following first:
Code: [Select]
sv d freshclam
rm -f /var/clamav/mirrors.dat
sv u freshclam

I've encountered the same issue in the past, but doing something of the above fixed things. This time, I can't get things right, no matter which site (of the 3) I'm trying on. Any ideas?

Title: Re: freshclam issues for the last 4-5 days
Post by: guest22 on November 08, 2017, 07:03:53 AM
https://lists.contribs.org/pipermail/devinfo/2017-November/014138.html
Title: Re: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 08, 2017, 11:38:08 AM
I see, so it's possibly an upstream issue...
Title: Re: freshclam issues for the last 4-5 days
Post by: brianr on November 08, 2017, 06:53:10 PM
Quote from: Michail Pappas on November 08, 2017, 11:38:08 AM
I see, so it's possibly an upstream issue...

Which seems to have fixed itself today....
Title: Re: freshclam issues for the last 4-5 days
Post by: devtay on November 09, 2017, 10:53:53 PM
It seems to be pretty common. I usually just wait it out unless there is something else not working on my server. Like 95% of the time it's an upstream issue. Annoying.
Title: Re: freshclam issues for the last 4-5 days
Post by: ReetP on November 10, 2017, 02:20:16 AM
According to the clamav lists it seems that the issues have been resolved, for now at least.
Title: Re: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 13, 2017, 07:46:04 AM
Hello again,

I'm afraid I'm still having the same kind of problems. Deleting mirrors.dat and using refreshmirrors does not help...
Title: Re: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 13, 2017, 08:50:22 AM
Is anyone else here experiencing the same issue here? If not, can you please tell me which database mirrors you are using?
Title: Re: freshclam issues for the last 4-5 days
Post by: TerryF on November 13, 2017, 09:23:32 AM
No issues, all good now, mine from AU

[root@home ~]#  freshclam --list-mirrors
Mirror #1
IP: 198.148.78.4
Successes: 85
Failures: 0
Last access: Mon Nov 13 15:28:12 2017
Ignore: No
-------------------------------------
Mirror #2
IP: 155.98.64.87
Successes: 55
Failures: 92
Last access: Mon Nov 13 14:28:09 2017
Ignore: No
-------------------------------------
Mirror #3
IP: 72.21.81.253
Successes: 0
Failures: 5
Last access: Sun Sep 10 11:24:34 2017
Ignore: No
-------------------------------------
Mirror #4
IP: 200.236.31.1
Successes: 47
Failures: 1
Last access: Thu Nov  2 01:45:08 2017
Ignore: No
-------------------------------------
Mirror #5
IP: 12.167.151.1
Successes: 56
Failures: 15
Last access: Mon Nov 13 08:27:48 2017
Ignore: No
-------------------------------------
Mirror #6
IP: 194.8.197.22
Successes: 40
Failures: 9
Last access: Wed Nov  8 00:08:48 2017
Ignore: Yes
-------------------------------------
Mirror #7
IP: 104.131.196.175
Successes: 2
Failures: 16
Last access: Fri Nov 10 21:21:19 2017
Ignore: Yes
-------------------------------------
Mirror #8
IP: 150.214.142.197
Successes: 99
Failures: 4
Last access: Mon Nov 13 04:27:41 2017
Ignore: No
-------------------------------------
Mirror #9
IP: 69.12.162.28
Successes: 82
Failures: 10
Last access: Mon Nov 13 00:26:30 2017
Ignore: No
-------------------------------------
Mirror #10
IP: 204.130.133.50
Successes: 63
Failures: 16
Last access: Mon Nov 13 16:28:17 2017
Ignore: No
-------------------------------------
Mirror #11
IP: 74.115.25.14
Successes: 57
Failures: 20
Last access: Mon Nov 13 03:27:36 2017
Ignore: Yes
-------------------------------------
Mirror #12
IP: 128.199.133.36
Successes: 3
Failures: 14
Last access: Tue Nov  7 15:07:13 2017
Ignore: No
-------------------------------------
Mirror #13
IP: 72.21.91.8
Successes: 0
Failures: 34
Last access: Sat Nov 11 04:21:58 2017
Ignore: Yes

Added: probably should reset it :-)
Title: Re: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 13, 2017, 11:00:46 AM
Ok, what I've found so far. First, there seems to be a huge problem to my db.local.clamav.net mirror, which essentially resolves to db.southeu.clamav.net. Which in turn resolves to a single system, 193.92.150.194 which is unresponsive:
Code: [Select]
dig db.local.clamav.net

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.4 <<>> db.local.clamav.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56548
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;db.local.clamav.net.           IN      A

;; ANSWER SECTION:
db.local.clamav.net.    2075    IN      CNAME   db.southeu.clamav.net.
db.southeu.clamav.net.  60      IN      A       193.92.150.194

;; Query time: 72 msec
;; SERVER: 192.168.1.2#53(192.168.1.2)
;; WHEN: Mon Nov 13 11:58:13 2017
;; MSG SIZE  rcvd: 78

Perhaps Stefano might have the same issues.

@TerryF: can you please provide me with your DatabaseMirror (config getprop clamav DatabaseMirror) setting? Is it db.au.clamav.net?
Title: Re: freshclam issues for the last 4-5 days
Post by: Stefano on November 13, 2017, 11:22:55 AM
I don't see any error from all servers of mine :-)
sorry
Title: Re: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 13, 2017, 11:40:02 AM
Hey Stefano, thanks for getting back to me! Is your DatabaseMirror set to db.southeu.clamav.net ?
Title: Re: freshclam issues for the last 4-5 days
Post by: Stefano on November 13, 2017, 11:46:49 AM
no.. I have some servers pointing to db.it.clamav.net and others to db.local.clamav.net

Code: [Select]
[root@mail ~]# dig db.local.clamav.net

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.62.rc1.el6_9.4 <<>> db.local.clamav.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36756
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;db.local.clamav.net.           IN      A

;; ANSWER SECTION:
db.local.clamav.net.    3438    IN      CNAME   db.it.clamav.net.
db.it.clamav.net.       60      IN      A       93.184.220.20
db.it.clamav.net.       60      IN      A       51.15.177.217
db.it.clamav.net.       60      IN      A       90.147.160.69
Title: Re: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 13, 2017, 01:02:47 PM
I see. Can I ask what does db.local.clamav.net resolve to?
Title: Re: freshclam issues for the last 4-5 days
Post by: TerryF on November 13, 2017, 01:21:30 PM
Quote from: Michail Pappas on November 13, 2017, 11:00:46 AM
@TerryF: can you please provide me with your DatabaseMirror (config getprop clamav DatabaseMirror) setting? Is it db.au.clamav.net?

[root@home ~]# config getprop clamav DatabaseMirror
db.local.clamav.net

freshclam.conf

#------------------------------------------------------------
Checks 24
DNSDatabaseInfo current.cvd.clamav.net

DatabaseDirectory /var/clamav
DatabaseMirror db.local.clamav.net
DatabaseMirror database.clamav.net
DatabaseOwner clamav
Foreground yes
LogVerbose yes
MaxAttempts 6
NotifyClamd /etc/clamd.conf
OnErrorExecute "/sbin/e-smith/freshclam-update-failed"
OnUpdateExecute "/sbin/e-smith/freshclam-update-ok"
Title: Re: freshclam issues for the last 4-5 days
Post by: Stefano on November 13, 2017, 02:17:13 PM
Quote from: Michail Pappas on November 13, 2017, 01:02:47 PM
I see. Can I ask what does db.local.clamav.net resolve to?

re read my previous post, it's all there ;-)
Title: Re: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 13, 2017, 03:18:45 PM
My error mate, sorry.
Title: Re: freshclam issues for the last 4-5 days
Post by: Knuddi on November 14, 2017, 09:01:10 AM
These issues happen from time to time and I have tried (during these periods) to alter Mirror to all kinds of places with little luck. I all previous cases the problem goes away on its own after 6-8 hours when mirrors are sync'ed correctly.

I have actually also made my own local mirror (as I have quite a few servers) to try to resolve this - based on this: https://www.clamav.net/documents/private-local-mirrors.

Title: Re: freshclam issues for the last 4-5 days
Post by: Michail Pappas on November 14, 2017, 09:24:47 AM
I am aware of these ups and downs, but this time this seems like a bigger problem is in the works. I've managed partially to obtain at least some updates to pass through, ie I receive less than 15 freshclam failures daily, by using the gr mirror. However, this is in stark contrast to the stability I have had: less than one failure in 3-6 months...

As for a local mirror: I have a single SME server running, helping a couple of admins on their own systems. So I'd like to avoid the overhead of maintaining another critical infrastructure component.

All in all, it's not a big problem since I am unable to use the unofficial ClamAV updates that would improve detection a lot, so the clients' AV does all the work here.