I had the same problem and it was the host.allow that was the problem.
After a search on google, I found this:
http://library.n0i.net/linux-unix/faq/ucd-snmp/#Why_am_I_getting__Connection_refused__--- SNIP ---
Why am I getting "Connection refused"?
-------------------------------------
This is actually nothing to do with the access control mechanism
(though that's an understandable mistake). This is the result of
the TCP wrapper mechanism using the files 'hosts.allow' and 'hosts.deny'
to control access to the service. Some distributions (such as RedHat
Linux) come with this enabled automatically - otherwise you need to
select it explicitly by configuring using '--with-libwrap'.
The simplest way to avoid this problem is to add the line
snmpd: ALL
in the file /etc/hosts.allow (or wherever this file is on your system).
--- /SNIP ---
/Mats