Topic: [Announce]: Dansguardian-stats - Statistics overview of web filter

[Knuddi]

I installed Dansguardian a while ago to protect my kids from unwanted content but even more from getting malware and virus though browsing. I realized that no proper statistics interface was existing (well I didn't find one) so I made my own variant. I have make a wiki for this on:

http://wiki.contribs.org/Dansguardian-stats

I have not had access to a lot of log files so if you see some unexpected stats then please let me know and please also provide the log file.

Enjoy,
Jesper

[Franco]

Hi Knuddi,
I'm currently using Sarg http://wiki.contribs.org/Sarg to read and keep the historical view of all accesses. The logformat for sarg needs to be in squid format.
I wanted to try your tool anyway, but unfortunatelly I can't access the package, the page is timing out.

Thanks,

[Knuddi]

The server was down this night due to power out, therefore the problem with access.

Sarg, as I understand it, "just" provides multiple summaries, I tried to also add some overview

[Franco]

I'm getting an error:

Code: [Select]

JPgraph error
font /usr/local/unjunkmgr/msttcorefonts/arialbd.ttf not readeable or does not exist.
Does it require your other contrib?

Thanks,

[Knuddi]

My mistake - I had not tested enough on a clean box. There should be no dependencies except from Dansguardian.

Upgrade to the new 1.0.2 version where I have fixed this - Wiki is also updated.

http://distro.ibiblio.org/pub/linux/distributions/smeserver/contribs/swerts-knudsen/SME7/Dansguardian-stats/smeserver-dansguardian-stats-1.0.2-1.noarch.rpm

[maykelsoft]

Hi,

I tried to install this contrib but graph is not showing. Did i miss something.

Version Installed: smeserver-dansguardian-stats-1.0.1-1.noarch.rpm

Anyhow, this is such a great contrib.

Thank you.

[Knuddi]

Do you mean it show an error or are all the numbers just zero (0)? The system requires you to you use logfileformat=4 in the Dansguardian configuration. This could be the reason.

You can see whether its working by manually running the statistic system:

Code: [Select]

/usr/local/dansguardian-stats/dansguardian-stats.pl -duration=hour
if you output looks like this and you have been browsing then you have a problem and the graphs will be zero

---Overall result ----------------
Start time    = Tue 28 Apr 2009, 08:15:56
End time      = Tue 28 Apr 2009, 09:15:59
Scanned URLs  = 0 (0 Bytes)
Infected      = 0 (0 Bytes)
URL mod       = 0
Content mod   = 0
Denied        = 0 (0 Bytes)

Also check the the dansguardian log to see whether its active and scanning:

Code: [Select]

tail -f /var/log/dansguardian/access.log

[Franco]

Knuddi,
Like maykelsoft stated, it's not working.
It just shows the two boxes in white.
My logformat is at 4:

cat /etc/dansguardian/dansguardian.conf | grep logfileformat
logfileformat = 4
logfileformat = 1

Thanks,

[Knuddi]

Why do you have 2 logfileformat statements in dansguardia.conf? I would imagine that the second entry takes precedence and forces logfileformat=1 rather that the desired "4"

[maykelsoft]

hi,

my logfileformat was set to 4.

but the graph is still 2 white blank page.

my dansguardian access.log is active...

cat /etc/dansguardian/dansguardian.conf | grep logfileformat
logfileformat = 4

Start time    = Wed 29 Apr 2009, 19:07:15
End time      = Wed 29 Apr 2009, 20:09:20
Scanned URLs  = 0 (0 Bytes)
Infected      = 0 (0 Bytes)
URL mod       = 0
Content mod   = 0
Denied        = 1368 (0 Bytes)

thanks

[Knuddi]

I can see that it thinks nothing is scanned (*SCANNED* tag not present in access.log) but some URLs are denied. if you could email me the log file then I can easier figure out what is wrong.

send to contribs@swerts-knudsen.dk

/Jesper

[maykelsoft]

hi jesper,

i already emailed the partial logs... as the original access logs are about 130mb.

thank you.

mike

[Franco]

Why do you have 2 logfileformat statements in dansguardia.conf? I would imagine that the second entry takes precedence and forces logfileformat=1 rather that the desired "4"

That's what I always had, I removed the last entry reloaded, but still nothing. Same white squares as maykelsoft states.

[Knuddi]

Its important that you restart dansguardian after you change the logfileformat to 4 - otherwise it has no effect. This fixed maykelsoft's problems.

[Knuddi]

Updated the contrib to 1.0.2 where I have changed pie charts to bars to give better overview and updated log parser to see some of the undocumented results from DG.

See wiki for download details:
http://wiki.contribs.org/Dansguardian-stats

Obviously I recommend all to upgrade...

[Franco]

Thanks Knuddi,
Now I wanted to use the unofficial ClamAV sigs, but not include the Sane Security, is it possible to just use the others?

Edit: Nevermind, found the instructions on the configuration file.
Thanks,

[bas]

Hello Jesper,

Same thing here, logfileformat set to 4 access.log shows *scanned*. But not on every line.

------------------
2009.5.7 23:19:25       -       192.168.10.250  http://forums.contribs.org/Themes/default/images/expand.gif             GET     162     0               1      200      image/gif                       -
2009.5.7 23:19:25       -       192.168.10.250  http://forums.contribs.org/Themes/default/images/bbc/resize-handle.gif          GET     827     0              1200     image/gif                       -
2009.5.7 23:19:25       -       192.168.10.250  http://forums.contribs.org/Themes/default/images/bbc/bbc_hoverbg.gif            GET     334     0              1200     image/gif                       -
2009.5.7 23:19:25       -       192.168.10.250  http://forums.contribs.org/index.php/topic,43922.0/prev_next,next.html          GET     0       0              1403     text/html                       -
2009.5.7 23:24:57       -       192.168.10.250  http://mirror2.mirrors.tds.net/pub/mozilla.org/firefox/releases/3.0.10/update/win32/nl/firefox-3.0.10.complete.mar      *SCANNED*       GET     300000  0               1       206     application/octet-stream                        -
------------------

The stats-webpage show two empty boxes "NO WEBPAGES SCANNED ..."
running the script dansguardian-stats.pl also shows nothing is scanned:
---Overall result ----------------
Start time    = Thu 7 May 2009, 22:14:46
End time      = Thu 7 May 2009, 23:14:46
Seen URLs     = 0 (0 Bytes)
Scanned URLs  = 0 (0 Bytes)
Infected      = 0 (0 Bytes)
URL mod       = 0
Content mod   = 0
Denied        = 0 (0 Bytes)

Is the script reading the right access.log? Or is there another .conf where i configure what to scan?

Bas

[Knuddi]

The access log that I scan is /var/log/dansguardian/access.log and the program will complain if not present.

You are absolutely sure that you have restarted dansguardian after you changed the logfileformat to 4? Yes yes, I have to ask 

What is the output of:
cat /etc/dansguardian/dansguardian.conf | grep logfileformat

If yes, then please email me the log file so I can try to determine why is doesn't parse correctly.

[bas]

Hello Jesper,

The result of the cat ..
[root@proxy ~]# cat /etc/dansguardian/dansguardian.conf | grep logfileformat
logfileformat = 4

And yes i did restart the dansguardian service in fact i restarted the whole server.
Now i'm gonna lookup your email address to mail the access.log

Bas

[Knuddi]

Just for the rest to know the resolution. The log file had both old and new (format 4) lines inside and the parser bailed out when it didn't see the right format in the start.

Therefore, until I release a new version I suggest you clear out the log (/var/log/dansguardian/access.log) before you restart dansguardian.

Code: [Select]

/etc/init.d/dansguardian stop
rm /var/log/dansguardian/access.log
/etc/init.d/dansguardian start

Wiki also updated to reflect this..

[Knuddi]

Just a small notice for those who followed this thread. I have released an updated version that fixes:

1) mixed log files with new (logfileformat 4) and old format causes nothing to be shown (graphs empty)
2) Some categories were represented as number rather than a textual description

Install instruction on wiki: http://wiki.contribs.org/Dansguardian-stats

[bloodshoteye]

@knuddi

Thanks for your contrib - looks great!
Can you contrib be adapted to work with dungog-dansguardian?

Cheers

[Knuddi]

I wouldn't know - I do not have access to this contrib. The only requirement would be the log file format which I do not know anything about? Can you enlighten me?

[Knuddi]

After a quick look at the wiki on Dungog it seems that they can co-operate just fine. In the "Global Settings" section you need to change the "format of Logfile" to something with "tab separated".

[bloodshoteye]

That did it:

Server Panel > Dansguardian > Modify global settings > Format of logfile > Tab Delimited > Save and Restart

Thanks, Knuddi