I'm having the exact same problem, with two SME 5.12 boxes, trying to get to a Microsoft Small Business Server behind one of them. Appropriate entries in LMHOSTS and HOSTS don't seem to help either. Interestingly enough, an NT, 2000 or XP box *will* see & authenticate, but a 9x box won't.
Another interesting note is that the 9x boxes also can't see a share on the SME box on the far side of the VPN.
I hope some IPSec whiz can help us! I'm going bug-eyed reading man pages, forums, listservs and Googling on this issue.