Koozali.org: home of the SME Server

Obsolete Releases => SME 7.x Contribs => Topic started by: Knuddi on April 23, 2009, 11:35:08 PM

Title: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on April 23, 2009, 11:35:08 PM
I installed Dansguardian a while ago to protect my kids from unwanted content but even more from getting malware and virus though browsing. I realized that no proper statistics interface was existing (well I didn't find one) so I made my own variant. I have make a wiki for this on:

http://wiki.contribs.org/Dansguardian-stats

I have not had access to a lot of log files so if you see some unexpected stats then please let me know and please also provide the log file.

Enjoy,
Jesper
Title: Re: [Announce]: Dansguardian-stats - Statsistics overview of web filter
Post by: Franco on April 24, 2009, 03:46:26 AM
Hi Knuddi,
I'm currently using Sarg http://wiki.contribs.org/Sarg to read and keep the historical view of all accesses. The logformat for sarg needs to be in squid format.
I wanted to try your tool anyway, but unfortunatelly I can't access the package, the page is timing out.

Thanks,
Title: Re: [Announce]: Dansguardian-stats - Statsistics overview of web filter
Post by: Knuddi on April 24, 2009, 06:55:00 AM
The server was down this night due to power out, therefore the problem with access.

Sarg, as I understand it, "just" provides multiple summaries, I tried to also add some overview :-)

Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Franco on April 25, 2009, 02:13:14 PM
I'm getting an error:
Code: [Select]
JPgraph error
font /usr/local/unjunkmgr/msttcorefonts/arialbd.ttf not readeable or does not exist.

Does it require your other contrib?

Thanks,
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on April 25, 2009, 06:41:01 PM
My mistake - I had not tested enough on a clean box. :-( There should be no dependencies except from Dansguardian.

Upgrade to the new 1.0.2 version where I have fixed this - Wiki is also updated.

http://distro.ibiblio.org/pub/linux/distributions/smeserver/contribs/swerts-knudsen/SME7/Dansguardian-stats/smeserver-dansguardian-stats-1.0.2-1.noarch.rpm
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: maykelsoft on April 28, 2009, 07:47:34 AM
Hi,

I tried to install this contrib but graph is not showing. Did i miss something.

Version Installed: smeserver-dansguardian-stats-1.0.1-1.noarch.rpm

Anyhow, this is such a great contrib.

Thank you.
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on April 28, 2009, 09:22:25 AM
Do you mean it show an error or are all the numbers just zero (0)? The system requires you to you use logfileformat=4 in the Dansguardian configuration. This could be the reason.

You can see whether its working by manually running the statistic system:

Code: [Select]
/usr/local/dansguardian-stats/dansguardian-stats.pl -duration=hour
if you output looks like this and you have been browsing then you have a problem and the graphs will be zero

---Overall result ----------------
Start time    = Tue 28 Apr 2009, 08:15:56
End time      = Tue 28 Apr 2009, 09:15:59
Scanned URLs  = 0 (0 Bytes)
Infected      = 0 (0 Bytes)
URL mod       = 0
Content mod   = 0
Denied        = 0 (0 Bytes)

Also check the the dansguardian log to see whether its active and scanning:
Code: [Select]
tail -f /var/log/dansguardian/access.log
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Franco on April 28, 2009, 02:08:55 PM
Knuddi,
Like maykelsoft stated, it's not working.
It just shows the two boxes in white.
My logformat is at 4:
Quote
cat /etc/dansguardian/dansguardian.conf | grep logfileformat
logfileformat = 4
logfileformat = 1
Thanks,
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on April 28, 2009, 02:27:37 PM
Why do you have 2 logfileformat statements in dansguardia.conf? I would imagine that the second entry takes precedence and forces logfileformat=1 rather that the desired "4"

Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: maykelsoft on April 29, 2009, 02:10:39 PM
hi,

my logfileformat was set to 4.

but the graph is still 2 white blank page.

my dansguardian access.log is active...

cat /etc/dansguardian/dansguardian.conf | grep logfileformat
logfileformat = 4

Start time    = Wed 29 Apr 2009, 19:07:15
End time      = Wed 29 Apr 2009, 20:09:20
Scanned URLs  = 0 (0 Bytes)
Infected      = 0 (0 Bytes)
URL mod       = 0
Content mod   = 0
Denied        = 1368 (0 Bytes)

thanks
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on April 29, 2009, 02:34:13 PM
I can see that it thinks nothing is scanned (*SCANNED* tag not present in access.log) but some URLs are denied. if you could email me the log file then I can easier figure out what is wrong.

send to contribs@swerts-knudsen.dk

/Jesper
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: maykelsoft on April 29, 2009, 02:56:37 PM
hi jesper,

i already emailed the partial logs... as the original access logs are about 130mb.

thank you.

mike
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Franco on April 30, 2009, 02:35:54 AM
Why do you have 2 logfileformat statements in dansguardia.conf? I would imagine that the second entry takes precedence and forces logfileformat=1 rather that the desired "4"
That's what I always had, I removed the last entry reloaded, but still nothing. Same white squares as maykelsoft states.
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on April 30, 2009, 07:47:52 AM
Its important that you restart dansguardian after you change the logfileformat to 4 - otherwise it has no effect. This fixed maykelsoft's problems.
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on April 30, 2009, 08:48:59 PM
Updated the contrib to 1.0.2 where I have changed pie charts to bars to give better overview and updated log parser to see some of the undocumented results from DG.

See wiki for download details:
http://wiki.contribs.org/Dansguardian-stats

Obviously I recommend all to upgrade...
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Franco on May 07, 2009, 02:40:40 PM
Thanks Knuddi,
Now I wanted to use the unofficial ClamAV sigs, but not include the Sane Security, is it possible to just use the others?

Edit: Nevermind, found the instructions on the configuration file.
Thanks,
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: bas on May 07, 2009, 11:28:43 PM
Hello Jesper,

Same thing here, logfileformat set to 4 access.log shows *scanned*. But not on every line.

------------------
2009.5.7 23:19:25       -       192.168.10.250  http://forums.contribs.org/Themes/default/images/expand.gif             GET     162     0               1      200      image/gif                       -
2009.5.7 23:19:25       -       192.168.10.250  http://forums.contribs.org/Themes/default/images/bbc/resize-handle.gif          GET     827     0              1200     image/gif                       -
2009.5.7 23:19:25       -       192.168.10.250  http://forums.contribs.org/Themes/default/images/bbc/bbc_hoverbg.gif            GET     334     0              1200     image/gif                       -
2009.5.7 23:19:25       -       192.168.10.250  http://forums.contribs.org/index.php/topic,43922.0/prev_next,next.html          GET     0       0              1403     text/html                       -
2009.5.7 23:24:57       -       192.168.10.250  http://mirror2.mirrors.tds.net/pub/mozilla.org/firefox/releases/3.0.10/update/win32/nl/firefox-3.0.10.complete.mar      *SCANNED*       GET     300000  0               1       206     application/octet-stream                        -
------------------

The stats-webpage show two empty boxes "NO WEBPAGES SCANNED ..."
running the script dansguardian-stats.pl also shows nothing is scanned:
---Overall result ----------------
Start time    = Thu 7 May 2009, 22:14:46
End time      = Thu 7 May 2009, 23:14:46
Seen URLs     = 0 (0 Bytes)
Scanned URLs  = 0 (0 Bytes)
Infected      = 0 (0 Bytes)
URL mod       = 0
Content mod   = 0
Denied        = 0 (0 Bytes)

Is the script reading the right access.log? Or is there another .conf where i configure what to scan?

Bas
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on May 08, 2009, 02:18:43 PM
The access log that I scan is /var/log/dansguardian/access.log and the program will complain if not present.

You are absolutely sure that you have restarted dansguardian after you changed the logfileformat to 4? Yes yes, I have to ask  :???:

What is the output of:
cat /etc/dansguardian/dansguardian.conf | grep logfileformat

If yes, then please email me the log file so I can try to determine why is doesn't parse correctly.
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: bas on May 08, 2009, 08:49:31 PM
Hello Jesper,

The result of the cat ..
[root@proxy ~]# cat /etc/dansguardian/dansguardian.conf | grep logfileformat
logfileformat = 4

And yes i did restart the dansguardian service in fact i restarted the whole server.
Now i'm gonna lookup your email address to mail the access.log

Bas

Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on May 09, 2009, 06:56:17 PM
Just for the rest to know the resolution. The log file had both old and new (format 4) lines inside and the parser bailed out when it didn't see the right format in the start.

Therefore, until I release a new version I suggest you clear out the log (/var/log/dansguardian/access.log) before you restart dansguardian.
Code: [Select]
/etc/init.d/dansguardian stop
rm /var/log/dansguardian/access.log
/etc/init.d/dansguardian start


Wiki also updated to reflect this..
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on May 20, 2009, 09:32:54 PM
Just a small notice for those who followed this thread. I have released an updated version that fixes:

1) mixed log files with new (logfileformat 4) and old format causes nothing to be shown (graphs empty)
2) Some categories were represented as number rather than a textual description

Install instruction on wiki: http://wiki.contribs.org/Dansguardian-stats

Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: bloodshoteye on May 21, 2009, 02:13:30 PM
@knuddi

Thanks for your contrib - looks great!
Can you contrib be adapted to work with dungog-dansguardian?

Cheers
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on May 21, 2009, 05:43:05 PM
I wouldn't know - I do not have access to this contrib. The only requirement would be the log file format which I do not know anything about? Can you enlighten me?
Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: Knuddi on May 21, 2009, 08:35:50 PM
After a quick look at the wiki on Dungog it seems that they can co-operate just fine. In the "Global Settings" section you need to change the "format of Logfile" to something with "tab separated".

Title: Re: [Announce]: Dansguardian-stats - Statistics overview of web filter
Post by: bloodshoteye on May 22, 2009, 08:36:12 AM
That did it:
Quote
Server Panel > Dansguardian > Modify global settings > Format of logfile > Tab Delimited > Save and Restart

Thanks, Knuddi