Koozali.org: home of the SME Server
Obsolete Releases => SME 8.x Contribs => Topic started by: purvis on April 24, 2013, 02:18:53 AM
-
http://wiki.contribs.org/GeoIP
after following directions in the GeoIP contrib wiki
The new GeoIp.dat file created by way of the crontab into the directory /var/lib/GeoIp does not update(copy) the GeoIP.dat file into /usr/share/GeoIP.
I understand the geoiplookup program defaults to looking for the file GeoIP.dat in the directory of /usr/share/GeoIP.
I tested the geoiplookup after deleting the file in the directory of /usr/share/GeoIP (rm -f /usr/share/GeoIp/GeoIP.dat) and the geoiplookup program does not return anything if the file is not there at that location. geoiplookup does have an override option to specify the database ( geoiplookup -f /var/lib/GeoIP/GeoIP.dat google.com).
to show your both the current GeoIP.dat files here is the commands
ls -l /var/lib/GeoIP
ls -l /usr/share/GeoIP
This bash file does download the new GeoIP.dat file and updates the /usr/share/GeoIP directory as well.
geoipdatupdate
#!/bin/bash
cd /
mkdir -p /var/lib/GeoIP
mkdir -p /usr/share/GeoIP
cd /var/lib/GeoIP
rm -f /var/lib/GeoIP/GeoIP.dat.gz
rm -f /var/lib/GeoIP/GeoIP.dat.gz.*
wget -q http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz > /dev/null
if [ -f "GeoIP.dat.gz" ];then
rm -f /var/lib/GeoIP/GeoIP.dat
gunzip GeoIP.dat.gz > /dev/null
fi
if [ -f "GeoIP.dat" ];then
rm -f /usr/share/GeoIP/GeoIP.dat
cp GeoIP.dat /usr/share/GeoIP
fi
#ls -l /var/lib/GeoIP
#ls -l /usr/share/GeoIP
exit 0
The crontab shows a once a month chance to update the GeoIP.dat file.
I think that this can be improved on by having more than one chance.
I will look into creating a hourly or daily cron that updates the GeoIP.dat file.
The GeoIP.dat file is changed monthly on the GeoIP.dat supplier side and that occurs more towards the beginning of the month, from what I understand.
-
I created a bash routine to update the GeoIP.dat files on our servers to be placed in the /etc/cron.hourly or /etc/cron.daily directory.
This should try to update the GeoIP.dat file starting immediately if the GeoIP.dat file does not exist, GeoIP.dat is 2 months old or after the first Tuesday of the month. From the Q&A on the Maxmind web site, the GeoIP.dat country database file is suppose to be updated on the first tuesday of each month.
If the GeoIP.dat file is considered up to date. No updating will be done.
for what is worth to others here is my code
#!/bin/bash
#geoipdatupdate
#date edited 2013-04-25
#can be placed in the directory /etc/cron.hourly or /etc/cron.daily directory
#this bash routine updates the GeoIP.dat country database file in /usr/share/Geoip from the internet
#website location of where to get geoip.dat.gz
weblocationofgeoipdatgz="geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz"
function getgeoipdatstatus {
updatethefiles=0
mkdir -p /var/lib/GeoIP
mkdir -p /usr/share/GeoIP
if [ ! -f /usr/share/GeoIP/GeoIP.dat ]
then
let updatethefiles=1
return
fi
# get the date of the file GeoIP.dat in /usr/share/GeoIP directory
geoipdatdate=$(stat -c %y /usr/share/GeoIP/GeoIP.dat)
geoipdatdate=${geoipdatdate%% *}
geoipdatdateyear=${geoipdatdate:0:4}
geoipdatdatemth=${geoipdatdate:5:2}
geoipdatdateday=${geoipdatdate:8:2}
# get the current date and place in months
currentyear=$(date +%Y)
currentmth=$(date +%m)
currentday=$(date +%d)
#if the date of the file matches the current month abort update
if [ $geoipdatdateyear == $currentyear ] && [ $geoipdatdatemth == $currentmth ]
then
return
fi
#if the date of the GeoIP.dat files is two months old, update it
if [ $updatethefiles -eq 0 ]
then
date1=$((geoipdatdateyear * 12 + geoipdatdatemth ))
date2=$((currentyear * 12 + currentmth ))
date3=$((date2-date1))
if [ $date3 -gt 1 ];then let updatethefiles=1;fi
fi
# if the date of the GeoIP.dat file is one month old and past the first tuesday of the month
# iwait till the 4rd day of the month before trying to update
if [ $geoipdatdatemth != $currentmth ] || [ $geoipdatdateyear != $currentyear ]
then
if [ $currentday -gt 7 ];then let updatethefiles=1;return;fi
dayofweek=$(date +%a)
if [ $dayofweek == "Wed" ] && [ $currentday -ge 1 ];then let updatethefiles=1;fi
if [ $dayofweek == "Thu" ] && [ $currentday -ge 2 ];then let updatethefiles=1;fi
if [ $dayofweek == "Fri" ] && [ $currentday -ge 3 ];then let updatethefiles=1;fi
if [ $dayofweek == "Sat" ] && [ $currentday -ge 4 ];then let updatethefiles=1;fi
if [ $dayofweek == "Sun" ] && [ $currentday -ge 5 ];then let updatethefiles=1;fi
if [ $dayofweek == "Mon" ] && [ $currentday -ge 6 ];then let updatethefiles=1;fi
if [ $dayofweek == "Tue" ] && [ $currentday -ge 1 ];then let updatethefiles=1;fi
fi
return
}
getgeoipdatstatus
if [ $updatethefiles -eq 0 ];then exit 0;fi
mkdir -p /var/lib/GeoIP
mkdir -p /usr/share/GeoIP
cd /var/lib/GeoIP
rm -f /var/lib/GeoIP/GeoIP.dat.gz
rm -f /var/lib/GeoIP/GeoIP.dat.gz.*
wget -q http://$weblocationofgeoipdatgz > /dev/null
if [ -f "GeoIP.dat.gz" ];then
rm -f /var/lib/GeoIP/GeoIP.dat
gunzip GeoIP.dat.gz > /dev/null
fi
if [ -f "GeoIP.dat" ];then
rm -f /usr/share/GeoIP/GeoIP.dat
cp -p GeoIP.dat /usr/share/GeoIP
fi
#ls -l /var/lib/GeoIP
#ls -l /usr/share/GeoIP
exit 0
-
After reading that Maxmind updates the GeoIP.dat country database file on the first Tuesday of each month.
I altered the script to update on any day after the first Tuesday of the month has passed.
I removed the part of the code where the updates where to be made starting xx(default was 4) number of days after the beginning of the month.
If the GeoIP country database is older than 1 month old, the update will try tried immediately.
An update is tried only once per script run, this is why my favorite place to hold this script will be in cron.hourly directory so that failures in downloading from the internet can be overcome in a near future if a failure occurred.
-
In the GeoIp contrib located here
http://wiki.contribs.org/GeoIP
There is written today are instructions to create a cron-job and here are the basic instructions
mkdir -p /etc/e-smith/templates-custom/etc/crontab
vim /etc/e-smith/templates-custom/etc/crontab/91_Update_GeoIP_db
#insert the below 2 lines in the file 91_Update_GeoIP_db and save the file
# Updating the GeoIP database monthly on the 5th at 0:00h.
0 0 5 * * root /usr/bin/wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz -O /var/lib/GeoIP/GeoIP.dat.gz; /bin/gunzip -f /var/lib/GeoIP/GeoIP.dat.gz
expand-template /etc/crontab
The code above does not copy the GeoIP.dat file into the /usr/share/GeoIP directory.
This code also it tries to update the GeoIP.dat file only once per month.
I am using the bash file as written above to update GeoIP.dat and place the file where it is needed. A good directory to place the bash is in the dialy cron directory "/etc/cron.daily".
I have cable internet and the bash file above only takes seconds to update the GeoIP.dat file. I have placed the bash routine in my hour cron directory "/etc/cron.hourly".
I deleted the cron-job created from the instructions at http://wiki.contribs.org/GeoIP
rm -f /etc/e-smith/templates-custom/etc/crontab/91_Update_GeoIP_db
expand-template /etc/crontab
The command to check the current date of the GeoIP.dat file in the /usr/share/GeoIp directory
geoiplookup -v /usr/share/GeoIP/GeoIP.dat
The returned text from the command will likely have a date of the file in the format of YYYYYMMDD.
As of today, the default location geolookup uses for the data file for GeoIP /usr/share/GeoIP/
-
I oddly found some bash code that did not work today.
It occurs when there is a zero before a number like in a day 08 for 8th day of the month.
I am new to bash programming and this is the kind of stuff that get's into your gut.
I think the new code to assign dates values number will work now.
I wanted to get this fixed and updated here on the contribs.org forum.
Here is the new bash code that updates the GeoIP.dat file.
Once again, I put the bash routine in the directory of /etc/cron.hourly so it will get run as soon as possible when there is a internet connection.
#!/bin/bash
#geoipdatupdate
#date edited 2013-05-15
#can be placed in the directory /etc/cron.hourly or /etc/cron.daily directory
#this bash routine updates the GeoIP.dat file in /usr/share/Geoip from the internet
#website location of where to get geoip.dat.gz
weblocationofgeoipdatgz="geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz"
function getgeoipdatstatus {
local stemp="0"
updatethefiles=0
mkdir -p /var/lib/GeoIP
mkdir -p /usr/share/GeoIP
if [ ! -f /usr/share/GeoIP/GeoIP.dat ]
then
let updatethefiles=1
return
fi
if [ ! -f /var/lib/GeoIP/GeoIP.dat ]
then
let updatethefiles=1
return
fi
# get the date of the file GeoIP.dat in /usr/share/GeoIP directory
geoipdatdate=$(stat -c %y /usr/share/GeoIP/GeoIP.dat)
geoipdatdate=${geoipdatdate%% *}
geoipdatdateyear=${geoipdatdate:0:4}
geoipdatdatemth=${geoipdatdate:5:2}
geoipdatdateday=${geoipdatdate:8:2}
geoipdatdateyear=`expr $geoipdatdateyear + 0`
geoipdatdatemth=`expr $geoipdatdatemth + 0`
geoipdatdateday=`expr $geoipdatdateday + 0`
# get the current date and place in months
currentyear=$(date +%Y)
currentmth=$(date +%m)
currentday=$(date +%d)
currentyear=`expr $currentyear + 0`
currentmth=`expr $currentmth + 0`
currentday=`expr $currentday + 0`
#if the date of the file matches the current month abort update
if [ $geoipdatdateyear == $currentyear ] && [ $geoipdatdatemth == $currentmth ]
then
return
fi
#if the date of the GeoIP.dat files is two months old, update it
if [ "$updatethefiles" -eq 0 ]
then
date1=$((geoipdatdateyear * 12 + geoipdatdatemth ))
date2=$((currentyear * 12 + currentmth ))
date3=$((date2-date1))
if [ $date3 -gt 1 ];then let updatethefiles=1;fi
fi
# if the date of the GeoIP.dat file is one month old and past the first tuesday of the month
# iwait till the 4rd day of the month before trying to update
if [ $geoipdatdatemth != $currentmth ] || [ $geoipdatdateyear != $currentyear ]
then
if [ $currentday -gt 7 ];then let updatethefiles=1;return;fi
dayofweek=$(date +%a)
if [ $dayofweek == "Wed" ] && [ $currentday -ge 1 ];then let updatethefiles=1;fi
if [ $dayofweek == "Thu" ] && [ $currentday -ge 2 ];then let updatethefiles=1;fi
if [ $dayofweek == "Fri" ] && [ $currentday -ge 3 ];then let updatethefiles=1;fi
if [ $dayofweek == "Sat" ] && [ $currentday -ge 4 ];then let updatethefiles=1;fi
if [ $dayofweek == "Sun" ] && [ $currentday -ge 5 ];then let updatethefiles=1;fi
if [ $dayofweek == "Mon" ] && [ $currentday -ge 6 ];then let updatethefiles=1;fi
if [ $dayofweek == "Tue" ] && [ $currentday -ge 1 ];then let updatethefiles=1;fi
fi
return
}
getgeoipdatstatus
if [ $updatethefiles -eq 0 ];then exit 0;fi
mkdir -p /var/lib/GeoIP
mkdir -p /usr/share/GeoIP
cd /var/lib/GeoIP
rm -f /var/lib/GeoIP/GeoIP.dat.gz
rm -f /var/lib/GeoIP/GeoIP.dat.gz.*
wget -q http://$weblocationofgeoipdatgz > /dev/null
if [ -f "GeoIP.dat.gz" ];then
rm -f /var/lib/GeoIP/GeoIP.dat
gunzip GeoIP.dat.gz > /dev/null
fi
if [ -f "GeoIP.dat" ];then
rm -f /usr/share/GeoIP/GeoIP.dat
cp -p GeoIP.dat /usr/share/GeoIP
fi
#ls -l /var/lib/GeoIP
#ls -l /usr/share/GeoIP
exit 0
-
alternatively
wiki.contribs.org/Talk:GeoIP
be aware that some versions of geoip will expect databases at another place, here is a modified version of cron part to avoid this :
# Updating the GeoIP database monthly on the 5th at 0:00h.
0 0 5 * * root /usr/bin/wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz -O /var/lib/GeoIP/GeoIP.dat.gz; /bin/gunzip -f /var/lib/GeoIP/GeoIP.dat.gz; cp /var/lib/GeoIP/*.dat /usr/share/GeoIP/
0 0 5 * * root /usr/bin/wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz -O /var/lib/GeoIP/GeoIPCity.dat.gz; /bin/gunzip -f /var/lib/GeoIP/GeoIPCity.dat.gz; cp /var/lib/GeoIP/*.dat /usr/share/GeoIP/
-
Because the newer GeoIp.dat files are posted some time on the first Tuesday of every month. It would make more since to have the cron do it's job on the 8th or the 9th of each month.
Personally, I think having a cron trying only once a month to update the GeoIP.dat file is too chancy.
-
Hi,
I've developed and maintain the smeserver-geoip rpm. Thanks for your interest in improving it. I can see that you've put much thought and energy into it recently.
Here are my thoughts after briefly reading through the info above:
-if something doesn't work as expected, file a bug. I happen to run into this thread but that won't always happen.
The new GeoIp.dat file created by way of the crontab into the directory /var/lib/GeoIp does not update(copy) the GeoIP.dat file into /usr/share/GeoIP.
-I'm not sure why I thought it's required to be the /var/lib/GeoIP/ directory. I must have had good reason to though. Maybe this changed somewhere along the way and the wiki never got updated.
The crontab shows a once a month chance to update the GeoIP.dat file.
I think that this can be improved on by having more than one chance.
What's the point? It updates once a month via cron. That's the end of it. Updating more than that will do absolutely nothing.
Thanks,