Topic: squidGuard - updating databases

[Shaun R]

Followed the HOWTO for installing squidGuard.
Works OK.
I've noticed that the files containing the urls/expressions to be blocked were produced by a robot in October.
I've downloaded the most recent (May) black list.
I untarred it into the correct directories but find that the new ones aren't normal text files but database files.  They end in .db

How do I use them?  Do I somehow have to convert the .db files into new text files or should I alter squidGuard's conf file to use the .db files instead?

If anyone's used the new files please explain it to me.

[Pierluigi Miranda]

Shaun R wrote:
 
> Followed the HOWTO for installing squidGuard.
> Works OK.
> I've noticed that the files containing the urls/expressions
> to be blocked were produced by a robot in October.
> I've downloaded the most recent (May) black list.
> I untarred it into the correct directories but find that the
> new ones aren't normal text files but database files.  They
> end in .db
>
> How do I use them?  Do I somehow have to convert the .db
> files into new text files or should I alter squidGuard's conf
> file to use the .db files instead?
>
> If anyone's used the new files please explain it to me.

I did not download the most recent databases, but I can tell you that .db file are nothing but the binary compiled text files.

You have nothing to do to use them: if they are present in the right directory, squidGuard loads them straight instead of compliling the text files in memory.

You can verify this behaviour looking at the squidGuard.log file you'll find in the /usr/local/squidGuard/log directory.

P.S.: if you manage your trusted/untrusted sites databases often, and they grow big, you may want to compile them to load faster on squidGuard startup, issuing a "squidGuard -C domainlist|urllist" command.

--

Pierluigi Miranda

[Shad Lords]

I have written the following script and saved it as squidGuard in /etc/cron.weekly.  Once you have squidGuard installed and acls correct this will update the database with the latest definitions each week.

#! /bin/bash
cd /usr/local/squidGuard
rm blacklists.tar.gz
wget -nv http://ftp.ost.eltele.no/pub/www/proxy/squidGuard/contrib/blacklists.tar.gz
tar -zxf blacklists.tar.gz
mv blacklists/* db
kill -HUP cat /var/run/squid.pid


In case it doesn't come across correctly
"wget -nv http://...tar.gz" should all be on the same line.

[Shad Lords]

Sorry for the previous post.  The script was an older version that I was testing with.  This one is the correct one.

I have written the following script and saved it as squidGuard in /etc/cron.weekly.  Once you have squidGuard installed and acls correct this will update the database with the latest definitions each week.

#! /bin/bash

cd /usr/local/squidGuard
rm -r blacklists*
wget -nv http://ftp.ost.eltele.no/pub/www/proxy/squidGuard/contrib/blacklists.tar.gz
tar -zxf blacklists.tar.gz
cp -r blacklists/* db
kill -HUP cat /var/run/squid.pid

In case it doesn't come across correctly
"wget -nv http://...tar.gz" should all be on the same line.

[Shaun]

Superb.
Thanks Shad!
I'll try it.  Excellent!

[Shaun R]

Hi Shad,

Tried your second script and it didn't work unfortunately.
It downloaded the blacklists.tar.gz and untarred it.

The first time it reported an error that the directory db didn't exist so I created that.
My machine running 4.1.2 didn't like the kill -HUP 'cat /var/run/squid.pid' line and so I've replaced it with squid -k kill

But, whereas I did have a working filtering system now I can access any page I like!

[KevinR]

Hi,

I'm working on the same problem. I installed the script and executed it. I found a missing in the first line

#! /bin/bash

cd /usr/local/squidGuard
rm -r blacklists*
wget -nv http://ftp.ost.eltele.no/pub/www/proxy/squidGuard/contrib/blacklists.tar.gz
tar -zxf blacklists.tar.gz
cp -r blacklists/* db
kill -HUP cat /var/run/squid.pid

Probably a typo on my part. I ran it a second time and it ran without errors.

I also lost all filtering. I tried running the setacls script in /usr/local/squidGuard
and "service squid restart". I finally rebooted. Still without filters. This would be great if it gets working. Look forward to any insights.

Kevin

[Shad Lords]

#! /bin/bash

cd /usr/local/squidGuard
rm -r blacklists*
wget -nv http://ftp.ost.eltele.no/pub/www/proxy/squidGuard/contrib/blacklists.tar.gz
tar -zxf blacklists.tar.gz
cp -r blacklists/* db
find db -name \*.\* -exec rm {} \;
kill -HUP cat /var/run/squid.pid

==============================

I have found out that the .db files don't work and need to be removed.  If you look at the /usr/local/squidGuard/log/squidGuard.log file you will see that it is entering emergency mode.  I have added a line 'find ...' to the script that fixes this problem.  Also make sure the tick marks around the pid file are the ones next to the 1 not the ones next to the enter key.  If you look at the log file after running the script it should say something about 'squidGuard ready for requests'.  If you run setacls after running this script and you didn't have all the db's installed you will get an error.  You need to have an expressions, urls, and domains file in each directory.  You also need a .blocked file in all but the trusted dir.

Sorry for all the confusion.

-Shad

[KevinR]

Shad Lords wrote:

> log file after running the script it should say something
> about 'squidGuard ready for requests'.  If you run setacls
> after running this script and you didn't have all the db's
> installed you will get an error.  You need to have an
> expressions, urls, and domains file in each directory.  You
> also need a .blocked file in all but the trusted dir.
>
> Sorry for all the confusion.

No problem. That worked great. I still think the filters a bit weak. I've been looking at adding words to the exclusion list. Have you done that? I was also looking at other solutions but I don't believe they would work on e-smith.

Thanks,

Kevin

[Shaun R]

Hi again,
I've made the changes you suggested, Shad, but am still having problems.

I thought that perhaps there was some clashing with the old e-smith-squidGuard-db files so I have removed them and the actual Berkley, SquidGaurd and E-Smith SquidGuard rpms.  I then reinstalled the Berkley, SquidGuard and E-SmithSquidguard rpms but not the databases and rerun your script.

squidGuard.log shows this error:

[2481] init domainlist /usr/local/squidGuard/db/local/domains
[2481] /usr/local/squidGuard/db/local/domains: No such file or directory
[2481] going into emergency mode

When I look in the /usr/local/squidGuard/db directory there isn't a local directory.  Should there be one?

Very grateful for any advice you can offer.

[Shaun R]

Started over.
Installed all of the squidGuard packages including the databases then ran your script.
Blocking now works.
But .........
.....the lists in the db folder were created in October 2000!  typing ls -l shows this.

going to the blacklists folder it shows that the .db files were created 25/5/01 and the nondb lists 18/10/2000 (or similar).

The download from squidGuard when untarred only contains the .db files these days.  So by deleting them we aren't keeping our blocked lists up to date at all!
We are reverting to the October 2000 ones which we had anyway!

I hope we can solve this as keeping the lists up to date automatically would be brilliant.

[tranbert]

A french guy is providing up to date bases :

http://cache.univ-tlse1.fr/documentations/cache/squidguard_en.html

A little script to do, and the bases are recreated every night.

Attention : script done with a hammer, i'm not a guru, juste tried to make something working.. Feel free to make it better and publish it

first, create /root/update

ftpbatch : (to get the files by ftp)

#!/bin/sh

cd /root/update

ftp -v ftp.univ-tlse1.fr << EOT
bin
cd /pub/reseau/cache/squidguard_contrib
get publicite.tar.gz
get adult.tar.gz
bye


traitement : (to untar and copy text files, then update just porn database, add is not automatic for me, i don't need it)

#!/bin/sh

cd /root/update

tar xzf adult.tar.gz

cp /root/update/adult/domains /usr/local/squidGuard/db/porn

cp /root/update/adult/urls /usr/local/squidGuard/db/porn


rm -v /root/update/adult/domains
rm -v /root/update/adult/urls
rm -v  publicite.tar.gz
rm -v adult.tar.gz

squidGuard -C all  ######  <--- or just "porn" if U want
kill -HUP cat /var/run/squid.pid

echo "C'est fini"

[Shad Lords]

It appears that all but the porn database is being updated regularly.  I looked at all of the urls and domains and they were all within 3-4 days of today except for the porn ones.  The domains was about 2 months old and the urls was about 8 months old.

It appears that the .db files get created everytime there is a run.  I just did a test and I got the exact same .db file as in the blacklist.  If you are interested try the following.  Go to the db/porn directory.  Type "squidGuard -C domains" or "squidGuard -C urls" and compare the .db file that is created to the one in the blacklist directory.  They are identical.  It appears that the urls file is not being updated very regularly and that the date on the .db file doesn't mean much.  I would trust the date on the regular files.

Interesting note on the french database.  The domains file is about 600k larger than the one I am downloading however the urls is about 420k smaller and the expressions is empty.  An interesting aside, acording to the website the adult.tar.gz was last updated on March 31 2001 which is the same date as the blacklist.tar.gz.  Even though the dates in the adult.tar.gz are newer and the filesizes are diferent.  

I would love to get an updated porn database somewhere but it seem that the one in the blacklist file is the latest.

[Shaun R]

When I manually download the blacklists.tar.gz file and untar it, I find (by ls -l) that in the porn directory for example that the urls.db file is 1098752 bytes created on May 26 and the standard urls text file is only 673218 bytes created on Oct 18.

To me that suggests that the downloaded tar file contains a newer db file than the text file so we are making a mistake by deleting the db file or by transferring the text file urls with squidGaurd -c to the db file?

[Shad Lords]

No we are not making a mistake.  The db files that are in the tar are created after every run of the robot.  They contain the exact same thing that the normal file does.  The reason for the newer dates in the porn case is that the regular files haven't been updated in a while.  The db files were created at the end of the last run.  If you do a file compare on a db file that you create from the old file and the db file that comes in the tar you will see that they are identical.

-Shad