Koozali.org: home of the SME Server

Dansguardian how do I get regular blacklist updates?

Offline steve288

  • *
  • 336
  • +0/-0
Dansguardian how do I get regular blacklist updates?
« on: March 23, 2012, 06:59:20 PM »
I'm trying to sort how to update the blacklist of sites in Dansguardian.

It appears the original list is very old.
And there appears to be only category included with the Install   /etc/dansguardian/lists/blacklists/ads  The file domains says they are from 2005.

However
http://www.shallalist.de/ has free(ish) lists for Dansguardian (as well as for Squidguard. )

I would like to download the list, every day  (for argument sake) put it into the right folder
eg  /etc/dansguardian/lists/blacklists/etc

I think I also need to change the /etc/dansguardian/lists/bannedsitelist and  uncomment the #.Include</etc/dansguardian/lists/blacklists/adult/domains>
Line as indicated in The wiki http://wiki.contribs.org/Dansguardian/ConfigFiles#Show_weighted_phrases_found
Under the heading of bannedsitelist. It also points to a link at http://dansguardian.org/?page=extras
That says you can download blacklists but, now you have to click on another link on this page “Blacklists” which points you to a pay site.
There is a link here called “A set of blacklist scripts for automating downloading of black lists and restarting DansGuardian. More info within the archive." Written by Brian Walters” but when you download the file, its not clear to me how to use it.

There is a little more info on the http://wiki.contribs.org/Dansguardian/ConfigFiles#Show_weighted_phrases_found paged under "bannedurllist". But nothing about doing this automatically.
I know you can do this with squidGuard, but what about Dansguardian.

How can I enable possibly the free http://www.shallalist.de/ site to download and automatically install updates.
Should I also install squidGuard?  Or what?

How do I regularly update blacklists automatically so taht Dansguardian will use them.

Thanks.

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Dansguardian how do I get regular blacklist updates?
« Reply #1 on: March 26, 2012, 03:48:16 AM »
steve288

The download scripts are here
http://www.shallalist.de/helpers.html
Choose one that suits you and configure a cron job to regularly run the script.

I very vaguely recall there was info published somewhere re doing this, perhaps it was in an earlier version of the Dansguardian howto, so look at the History and go back a long way (years)
If I find it in my travels I'll post it here, but it won't be soon.
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline steve288

  • *
  • 336
  • +0/-0
Re: Dansguardian how do I get regular blacklist updates?
« Reply #2 on: March 27, 2012, 02:40:39 PM »
Thanks I  will take a look at it.
regards

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Dansguardian how do I get regular blacklist updates?
« Reply #3 on: March 27, 2012, 08:08:33 PM »
steve288

For a download script see old Dansguardian Howto at
http://distro.ibiblio.org/pub/linux/distributions/smeserver/contribs/rmitchell/smeserver/howto/deprecated/sme6/dansguardian%20instal%20&%20configure%20HOWTO%20for%20sme%20server.htm

The site mentioned (teledanmark.no), no longer exists.
A current blacklists.tar.gz is available from http://cri.univ-tlse1.fr/blacklists/download/blacklists.tar.gz
Google for other locations if preferred

Here is a modified (corrected) script

If you wish to make dansguardian use squidguard blocking rules & have them updated weekly then ad the following to the /etc/cron.weekly/dansguardian file
Please check the location of the blacklists is still current, if necessary search Google on "squidGuard blacklists" or "blacklists.tar.gz" to find a current location.

cd /etc/cron.weekly
pico -w dansguardian

Add the following lines

# blacklists update script for dansguardian
cd /etc/dansguardian
rm -f blacklists.tar.gz
wget -qnv http://cri.univ-tlse1.fr/blacklists/download/blacklists.tar.gz -O blacklists.tar.gz
tar -zxf blacklists.tar.gz
chown -R root.root blacklists
chmod -R 640 blacklists
find blacklists -name new\* -exec rm {} \;
rm -f blacklists/README
chmod ug+x blacklists
chmod ug+x blacklists/*

Then
Ctrl o
To save
Ctrl x
To exit

chmod +x dansguardian
service crond restart

The scripts runs OK (manually instigated), but I did not test or check full integration with Dansguardian.
« Last Edit: March 27, 2012, 10:58:51 PM by mary »
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline janet

  • ****
  • 4,812
  • +0/-0
Re: Dansguardian how do I get regular blacklist updates?
« Reply #4 on: March 28, 2012, 05:24:24 AM »
steve288

DG Wiki has been updated
http://wiki.contribs.org/Dansguardian/ConfigFiles#Blacklists

Would you try the suggested script and check integration with DG, if any corrections please advise here, or update the wiki yourself
Please search before asking, an answer may already exist.
The Search & other links to useful information are at top of Forum.

Offline steve288

  • *
  • 336
  • +0/-0
Re: Dansguardian how do I get regular blacklist updates?
« Reply #5 on: March 29, 2012, 05:49:49 PM »
Thanks been away for a bit.
I will check and return with feed back.
Regards

Offline steve288

  • *
  • 336
  • +0/-0
Re: Dansguardian how do I get regular blacklist updates?
« Reply #6 on: March 30, 2012, 09:21:09 PM »
Works great.
Thanks.

I added a troubleshooting section to blacklist. Hope helps others.

I am wondering about one thing. The script that puts all the blacklist files into a new folder
/etc/dansguardian/blacklist
The original version of DG that gets installed puts the BL files in /etc/dansguardian/lists/blacklist
Should the tar extract the files in the Original location or in the new location.

If you put it into the original location then many of paths in the bannedsitelists and bannedurllists are already there. Of course the new path can be added.
Can you suggest what you think the most appropriate approach is?

Regards
« Last Edit: March 30, 2012, 10:12:55 PM by steve288 »