<~~not an expert but aren't "no load" ping times of 120 ms on an internal network excessive?
I'm on a wireless AP --- switch ---- gateway--modem (+some other stuff). No gigabit hardware anyware and wireless G. I have 1- 2 ms to get to gateway router, to my home network, I have 40 ms avg.
I ran the same speedtest and got Toronto - New York - 3235 kbps down and 3730 kbps up with almost no change internally and maybe doubling of external, see graph. Well I would show you the graph if I could get it into the post. I tried to cut and past bitmap and jpeg but neither would work.

Anyways I tried just pinging internal addresses with the speedtest running and I don't see the ping time to gateway at more than 2 ms for either up or down.
I guess I'm saying that something else is wrong inside the network, not at the gateway/firewall/modem level.