For what its worth - we have seen something like this once or twice in our customer base. It absolutely drove us nuts until we figured it out...
The first time was with a Draytek router (nice mid range unit). We didn't fit and had no access to it. The techs who looked after it were good guys, but they didn't know VoIP, and they had set up a lot of the advanced features on the Draytek, including the DOS UDP attack threshold... Figured it out yet? At one call on the circuit, everything was hunkydory, also at two and three, but at four, the circuit went down... for exactly 10 seconds and then resumed nornal operations.
Turns out that the standard Draytek reponse to an assumed DOS attack is to shut down all comms... for 10 seconds.
Solution, - increase the threshold at which the DOS attack mechanism kicks in.
Maybe your problem, maybe not
Best
S