Obsolete Releases > SME 8.x Contribs
a script to visual analze emailheaders using country geoip
purvis:
This program was written to help me analyze how i might want to handle bad email by determining the country origin using the geoip lookup software that is easily installed following a wiki page from contribs.org.
This program help me see where email might be coming from. It will not be perfect for a variety of reason but it is a tool just the same.
Also this program can be use to test if email might be coming into the server from an unwanted country.
I am sure it can use it for other purposes too.
There is also a function that i added to see where emails marked with a certain spam level using astericks placed the email headers, presumably by spammassain, may be coming from.
In the future, i will not be allowing any email from countries outside of the usa to accounts on our servers as a general rule. But i will allow certain mail to the maillog account.
This program was more than i needed by i tried to make it for other users. My learning curve of bash and regular expressions has been slow down.
This script does not delete emails as written.
From what I have seen with some other addon programs for sme server, you have to exclude countries. That could be ok for somebody wanting that. But I need to filter in and not out emails within countries I want.
This script makes use of the GeoIp program found here http://wiki.contribs.org/GeoIP
You can find most of the countries listed there also.
The program has to be loaded on your sever for this script to run as is.
The script was written with as few functions as possible to keep it fast but also trying to keep it somewhat modular for changing.
There are some areas where i have commented out lines while testing. I have limited time to do this kind of stuff and it earns me zero dollars except for trouble the script may save me in down time in the future.
I will share another script in about a month that will be derived from this and it will be much leaner to process received emails full time as they arrive.
This script was set up to view emails headers that are received from outside of the United States. Any outside ip address for the countries wanted in the email header flags the email I the script as rejected.
I had a thought about placing countries into the email header of all received emails. That might prove useful somewhere down the road. But I am learning to crawl before I run.
--- Code: ---#!/bin/bash
### This program was written to help evaluate received emails to determine
### if emails need to be blocked by ip addresses in the email headers or
### to see if emails are being received for countries not wanted.
### There are other uses for this program one might want to use it for.
### This program makes use of a lot of functions to make it somewhat easier
### to alter to get other results
### This program does not alter emails as written and the users takes full use for its actions.
### This script will search emails that have emails where a ip address does not
### come from a safe list placed into a string.
### There is another option to search only on the spam asterick marks
### In the "X-Spam-Level:" email header.
### Those are compared to how may astericks are in the field.
### You set how many minuium astericks to have in the fiel.
### Any number or higer number of astericks will display the email
### spamastericks must be set to 0(zero) for script to work on ip addresses alon
###---------------------------------------------------------------------start of variables
### All countries need a space before and after the country and
### a comma immediately after the country abbreviation.
### Between each country needs to be a backslash and a bar together
safelistofcountries=" US, \| UM, \| GOV, \| MIL, "
### edit the wanipaddressofthisemailserver
### place you ip address in so that it will not be searched
### this is not necessary but may prove helpful.
### the program will not report on the ipaddresses in this string
### place a minus sign n front of each ip 4 address with no spaces in the string
### an example is "-78.152.2.1-645.1.284.291"
wanipaddressofthisemailserver=""
### Set nosafecountryduplicate to 1 to not record a second ip lookup of an address
### that was already found in a previous email that was geoip looked up if that ip
### address fell into one of the safe countries list.
### All addresses outside of the safe list will be shown and reocrded in the log files
### The setting of 0(zero) will record all ip addresses. This feature was designed to
### speed up the program when set to 1, but you could make good use of the log
### when you record all ip addresses.
### Any ip address will be listed only once for each email header scanned.
nosafecountryduplicate=1
### set readmaillogemails to 0 to exclude the emails to the maillog account
### set readmaillogemails to 1 to exclude the emails that are not in the maillog account
### set readmaillogemails to 2 to read all emails, both maillog and non maillog accounts
### if you are reading a directory full of email flles not under the MailDir directory set this value to 2
#readmaillogemails=0 # exclude emails under maillog account
#readmaillogemails=1 # read only the emails under the maillog account
#readmaillogemails=2 # read all emails
readmaillogemails=2 # read all emails
### set spamastericks to the number of how many spam astericks need to to be found in the spam header line
spamastericks=0
### assign the program to lookup geo ip address based on countries
geoiplookupcommand="/usr/bin/geoiplookup"
###---------------------------------------------------------------------end of variables
###---------------------------------------------------------------- start of functions
function whereisipfrom {
local stemp=""
iplocationaccept=0
local tempcount=0
if [ $nosafecountryduplicate -eq 1 ];then
tempcount=$(echo $sipacceptlist | grep -c "\-$1\-")
if [ $tempcount -gt 0 ]
then
iplocationaccept=1
return
fi
fi
stemp=$($geoiplookupcommand "$1" | sed -e 's/.*://')
# stemp=$(echo $stemp | tr '[:upper:]' '[:lower:]')
sgroupedcountries="$sgroupedcountries$1 $stemp\n"
echo -e "$1\t$stemp" >> /tmp/geoemailall
tempcount=$(echo $stemp | grep -i -c "$safelistofcountries")
if [ $tempcount -gt 0 ]
then
iplocationaccept=1
sipacceptlist="$sipacceptlist$1-"
echo -e "$1\t$stemp" >> /tmp/geoemailaccepted
return
fi
if [ $iplocationrejected -eq 0 ];then clear;fi
tempcount=$(echo $stemp | grep -i -c " not found")
if [ $tempcount -gt 0 ];then iplocationaccept=-1;fi
echo "$1 $stemp"
echo -e "$1\t$stemp" >> /tmp/geoemailrejected
if [ "$iplocationaccept" -eq 0 ] || [ "$iplocationaccept" -eq -1 ]
then
iplocationrejected=2
ipthisreceivedlinerejected=1
fi
return
}
function checkipaddresses {
local tempcount=0
local stemp=""
local kk=""
ipthisreceivedlinerejected=0
set $1
for item
do
tempcount=$(echo "-$item" | grep -c "\-127.0.0.1\|\-192.168.\|\-10.\|\-169.254.")
if [ "$tempcount" -gt 0 ];then continue;fi
tempcount=$(echo "-$item" | grep -c "\-172.")
if [ "$tempcount" -gt 0 ];then
tempcount=$(echo "-$item" | grep -c "\-172.16.\|\-172.17.\|\-172.18.\|\-172.19.\|\-172.20.\|\-172.21.\|\-172.22.\|\-172.23.\|\-172.24.\|\-172.25.\|\-172.26.\|\-172.27.\|\-172.28.\|\-172.29.\|\-172.30.\|\-172.31.")
if [ "$tempcount" -gt 0 ];then continue;fi
fi
tempcount=$(echo $ipaddresseschecked | grep -c "\-$item\-")
if [ "$tempcount" -gt 0 ];then continue;fi
ipaddresseschecked="$ipaddresseschecked$item-"
whereisipfrom "$item"
done
return
}
function locateipaddresses {
local stemp=""
local ipaddresses=""
iplocationrejected=0
ipaddresseschecked="-127.0.0.1-$wanipaddressofthisemailserver-"
for item in $semailheader
do
stemp=$(echo "$item" | grep -i "^Received:" )
if [ ! -z "$stemp" ]
then
ipaddresses=$(echo "$stemp" | grep -E -o '((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])')
#echo $ipaddresses
if [ ! -z "$ipaddresses" ]
then
checkipaddresses "$ipaddresses"
#the next line is testing, where you can view the recevied line in the email header where the rejected ip address is found
if [ $ipthisreceivedlinerejected -gt 0 ];then echo "$stemp";echo " ";fi
fi
fi
done
return
}
function isemailspamflagged {
local tempcount=0
local stemp=""
flaggedwithspam=0
for item in $semailheader
do
stemp=$(echo "$item" | grep -i "^X-Spam-Level:" )
if [ ! -z "$stemp" ];then
tempcount=$(echo "$stemp" | grep -c "$sspamstring" )
if [ $tempcount -gt 0 ];then flaggedwithspam=1;fi
fi
done
return
}
function echoemailheaderline {
for item in $semailheader
do
echo "$item" | grep -i "$1"
done
return
}
function getemailheaderfromemailfile {
semailheader=$(grep -B10000 -m1 ^$ $filename | tr -s '\t' '\040'|sed -e ':a;N;$!ba;s/\n / /g')
return
}
###----------------------------------------------------------------------end of functions
### -----------------------------------------------------------------------------------------------
### main program starts hear
###
if [ ! -f "$geoiplookupcommand" ]
then
echo "the program below does not exist and is needed"
echo "$geoiplookupcommand"
fi
rm -f /tmp/geoemailall
rm -f /tmp/geoemailaccepted
rm -f /tmp/geoemailrejected
echo "all email geoipaddress " > /tmp/geoemailall
echo "" >> /tmp/geoemailall
echo "all email geoipaddress accepted" > /tmp/geoemailaccepted
echo "" >> /tmp/geoemailaccepted
echo "all email geoipaddress rejected " > /tmp/geoemailrejected
echo "" > /tmp/geoemailrejected
counter=0
while [ $counter -lt $spamastericks ]
do sspamstring="$sspamstring\*"
let counter=counter+1
done
clear
echo "working ! readng emails ......................"
echo ""
echo ""
countfile=0
sipacceptlist="-"
IFS=$(echo -en "\n\b")
for filename in $(find /home/e-smith/files/users/*/Maildir/ -name "1*.*.$HOSTNAME*" -type f )
do
if [ "$readmaillogemails" -eq 0 ];then
if [[ $filename == */maillog/Maildir/* ]];then continue; fi
fi
if [ "$readmaillogemails" -eq 1 ];then
if [[ $filename != */maillog/Maildir/* ]];then continue; fi
fi
let countfile+=1
#echo $countfile
#echo $filename
getemailheaderfromemailfile "$filename"
#echo -e "$semailheader"
sgroupedcountries=""
iplocationrejected=0
locateipaddresses
flaggedwithspam=0
if [ $spamastericks -gt 0 ]
then
iplocationrejected=0
isemailspamflagged
fi
showemail=0
if [ $flaggedwithspam -gt 0 ];then showemail=1; fi
if [ "$iplocationrejected" -gt 0 ] && [ "$spamastericks" -eq 0 ];then showemail=1;fi
if [ $showemail -gt 0 ]
then
echoemailheaderline ":"
echo ""
echo "-------------------------------------------"
echo -e "$sgroupedcountries"
echoemailheaderline "^Date: "
echoemailheaderline "^TO: "
echoemailheaderline "^From: "
echoemailheaderline "^Return"
echoemailheaderline "^Subject: "
echoemailheaderline "^X-"
echoemailheaderline "^RECEIVE"
echo "filename $filename"
echo "--------------------------------------------------------------------------------------------------"
read -p "Press [Enter] key to continue ......."
echo "working ! readng emails ......................"
echo ""
echo ""
fi
done
echo ""
echo ""
echo "total emails read $countfile"
echo ""
echo "next! view 3 files with only geoiplookup returned iinformation"
echo "view geoip counties in the file /tmp/geoemailall"
echo "view geoip counties in the file /tmp/geoemailaccepted"
echo "view geoip counties in the file /tmp/geoemailrejected"
echo ""
echo " To quit viewing each list, press the q key to quit"
echo " You will have to delete the 3 files above manually"
echo " when are complete and done using this program."
echo " They are left undeleted so you can use them in other ways."
read -p "Press [Enter] key to continue ......."
cat /tmp/geoemailall | less
cat /tmp/geoemailaccepted | less
cat /tmp/geoemailrejected | less
exit 0
--- End code ---
Stefano:
be aware that a USA company could have its servers in mongolia..
blocking email with GeoIP could be dangerous
CharlieBrady:
--- Quote from: Stefano on April 12, 2013, 10:40:09 AM ---be aware that a USA company could have its servers in mongolia..
blocking email with GeoIP could be dangerous
--- End quote ---
Moreover, you won't solve the spam problem by allowing email only from the USA. 20-25% of global SPAM originates from USA IP addresses. Only China produces more.
http://www.securelist.com/en/analysis/204792282/Spam_in_January_2013
http://www.securelist.com/en/analysis/204792251/Spam_in_Q3_2012
purvis:
Stefano. You may have to whitelist some ip addresses.
I suggest using email server that is more likely to a scan emails better for those outside companies.
I don't think having personal mail on a business server is appropriate.
Email belongs to the server owner. Lets not start a discussion on that.
Charlie. I do acknowledge that spam can come from USA. If I can cut 80 to 75 percent out. That is ok with me.
purvis:
I am reviewing my email to get a view of what is coming in and how it is rated on spam.
I do not have a lot of email coming in to really do a top notch job of review.
But I think that I can make some remarks.
It appears now that mostly the emails containing vicious material is marked with an out of USA IP address in some way. Most of it will be the originators up address.
I have seen a lot of unwanted email either vicious or spam have a email header that
PHPmailer from sourceforge.net is used.
The bad email from my brothers yahoo.com account that somehow was compromised was spamming email to his contact list and some if not all his contact list in yahoo were in the email headers. The originator was from of the bad email was from outside the USA and the multiple (4)times this kind of email came was with originators outside USA in different countries each time.
Only the last email was marked as spam. This started over a year ago and all email was to vicious single links.
Navigation
[0] Message Index
[#] Next page
Go to full version