Nice picture
It looks like you have couple of routes for testing.
1> does local lan to SME private lan IP also slow down on pings?
2> does local lan PC to a branch office PC or Server slow down?
I like to use Colasoft's ping tool on a windows machine to troubleshoot these. You can ping multiple addresses at once and see at a glance on the graph if certain addresses are slower then others. The addresses I usually run are as follows,
My workstation to:
1> local file server or pdc
2> LAN internet gateway
3> ISP's closest internet gateway to me
4> google.com or other reliable WAN target
5> targets across various VPN tunnels
I then use pingplotter to watch which hop is causing trouble if I have one route that is not working correctly. (pingplotter does not help on my tunnels as all of them show as just one hop)
It sounds suspiciously like hardware to me (nic on server, switch port or could be a nic on a workstation.) That is assuming this was a stable situation that has just now degraded.