I use E-Smith as a firewall with thousands of connections being forwarded to an internal server, and thousands of emails being sent out every day. I also have 3 or 4 people VPN'ed in moving files across. It has been going for a year with no problems.
This is on a PIII 800 with 512M of RAM. I have never noticed a slowdown. As long as you have a decent machine you should be able to handle a pretty fair number of VPN connections. Hell, if you look at a lot of the CISCO boxes they are 200Mhz or less machines with 32M of RAM.
If the users that will be VPN'ed in are coming from one location behind a NAT firewall, you will have issues. The VPN server will only allow one connection from one IP. That is the standard, though I know the Microsoft PPTP server will allow multiple connections from one IP. If you need to do that, you would be best off creating one tunnel from the remote location (from the Firewall there for example) and having the users use that.