Koozali.org: home of the SME Server

Displaying dates in the squid/access logfile

Offline judgej

  • *
  • 375
  • +0/-0
Displaying dates in the squid/access logfile
« on: September 27, 2007, 01:09:26 PM »
The squid access logs contain dates in timestamp format, '1190890805.594' for example. I would like administrators to be able to view this log, through the normal 'View log files' admin panel, but see the date and time expanded to a human-readable form.

Is there any way to do this? I have found an example perl script to do this offline, but how about through the admin panels?

-- Jason
-- Jason

Offline chris burnat

  • *****
  • 1,135
  • +2/-0
    • http://www.burnat.com
Re: Displaying dates in the squid/access logfile
« Reply #1 on: September 27, 2007, 01:20:44 PM »
I have been battling with this issue which does not end there...  If you use Dansguardian, you need to adopt the Squid format for Dans logs, or SARG cannot produce reports.  Trying to track users in real time becomes a nightmare since you have to "translate" Squid timestamps in human readable form - in my case one at a time.  Maybe I am missing something.

Would you like to make a New Feature Request at Bugzilla?
And.. can you post your script here, I have been looking for something like this for a while.
Thanks
chris
- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org)  - check: http://wiki.contribs.org/Bugzilla_Help .  Thanks.

Offline judgej

  • *
  • 375
  • +0/-0
Re: Displaying dates in the squid/access logfile
« Reply #2 on: September 27, 2007, 01:35:23 PM »
The script is listed here:

http://forums.contribs.org/index.php?topic=20325.msg80117#msg80117

I'll try and get those details put into the wiki.
-- Jason

Offline chris burnat

  • *****
  • 1,135
  • +2/-0
    • http://www.burnat.com
Re: Displaying dates in the squid/access logfile
« Reply #3 on: September 27, 2007, 02:01:34 PM »
Thanks Judge.
Much appreciated, I missed it...
- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org)  - check: http://wiki.contribs.org/Bugzilla_Help .  Thanks.

Offline judgej

  • *
  • 375
  • +0/-0
Re: Displaying dates in the squid/access logfile
« Reply #4 on: September 27, 2007, 02:19:29 PM »
I've put some notes here:

http://wiki.contribs.org/Log_Files#What_they_are_and_what_they_mean

I expect this can be automated by customising the admin script that lists these files. Whether the logfile format detection can be automated or whether admin script needs to be more 'aware' of the logfile formats, I don't know.
« Last Edit: September 27, 2007, 05:11:44 PM by judgej »
-- Jason

Offline judgej

  • *
  • 375
  • +0/-0
Re: Displaying dates in the squid/access logfile (HACKED:)
« Reply #5 on: September 29, 2007, 02:08:47 AM »
I've made a change that seems to do the job. The file /etc/e-smith/web/functions/viewlogfiles contains the function tai64nlocal. I modified it to detect the squid file timestamp too, and format that as localtime. Here is the new function:

Code: [Select]
sub tai64nlocal
{
    my $line = shift;
    if ($line =~ /^(\@[0-9a-f]{24})(.*)/s)
    {
        return Time::TAI64::tai64nlocal($1) . $2;
    }
    if ($line =~ /^([0-9]{10}.[0-9]{3})(.*)/s)
    {
        return localtime($1) . $2;
    }
    return $line;
}

The old function, for completeness:

Code: [Select]
sub tai64nlocal
{
    my $line = shift;
    return $line unless ($line =~ /^(\@[0-9a-f]{24})(.*)/s);
    return Time::TAI64::tai64nlocal($1) . $2;
}

Also raised as a bug: http://bugs.contribs.org/show_bug.cgi?id=3434
« Last Edit: September 29, 2007, 02:18:11 AM by judgej »
-- Jason

Offline cactus

  • *
  • 4,880
  • +3/-0
    • http://www.snetram.nl
Re: Displaying dates in the squid/access logfile (HACKED:)
« Reply #6 on: September 29, 2007, 12:06:15 PM »
I've made a change that seems to do the job. The file /etc/e-smith/web/functions/viewlogfiles contains the function tai64nlocal.

Also raised as a bug: http://bugs.contribs.org/show_bug.cgi?id=3434
Your work looks very identical to mine and Charlie's from bug 3432. I don not want to seem rude but are you sure you got the idea yourself or did you just copy it from there?
Be careful whose advice you buy, but be patient with those who supply it. Advice is a form of nostalgia, dispensing it is a way of fishing the past from the disposal, wiping it off, painting over the ugly parts and recycling it for more than its worth ~ Baz Luhrmann - Everybody's Free (To Wear Sunscreen)

Offline judgej

  • *
  • 375
  • +0/-0
Re: Displaying dates in the squid/access logfile (HACKED:)
« Reply #7 on: September 29, 2007, 02:09:48 PM »
Your work looks very identical to mine and Charlie's from bug 3432. I don not want to seem rude but are you sure you got the idea yourself or did you just copy it from there?

Nope - I've never seen that bug. That's my programming style - I like to spell everything out in full with if-then statements. I do tend to use a single exit point too, but I'm not familiar enough with perl variables to do that, otherwise I would have. I know there is a bug in the RE - the dot in the middle probably needs escaping, but I'm not really sure how. For a permanent fix, I would also rename the function to something more appropriate, since it is more generic in what it decodes now. I found the script using 'find' from the /etc/e-smith directory, and greping for 'tai64' in every text file. There were three in total, the first one being right one. Need any more convincing...?

All I can say is great minds think alike :-) I would never try to pass off someone elses code or effort as mine. It wouldn't be right, and I would not last long here if I did. I guess we all happened to be looking at logfiles at the same time, with this problem in particular - I spotted that by the recent changes in the logfiles wiki page (which I thanked you for - check it out).

If it gets into the next set of patches, then everyone is a winner, regardless of where the code came from.
« Last Edit: September 29, 2007, 02:20:51 PM by judgej »
-- Jason