Latency on initial VOIP connection, only occurring on Speaker-phone to speaker-phone calls

Hi all,

I've opened a ticket with support, but thought I'd get some input from the community first.

Here is the issues, as stated to support:

Greetings,
We're having an intermittent, but fairly frequent issue with latency on our VOIP phones. The issue only occurs speaker to speaker. Speaker to handset is fine. And the delay is not delay in voice, its on the initial call/answer. So if I dial or get dialed (speaker to speaker), and we answer on speaker, there’s an approximate 7 second lag until we hear anything at all. Then after that 7 seconds, voice is clear and fine. There's no latency when there’s audio.

We have a 100/100 circuit here, and our provider, Primus, has also ruled out the main circuit. We've been told by Polycom, that device to device is LAN only, and does not go over WAN at all. The phones are all connected to a Cisco SG500-28P Gigabit PoE Stackable Managed Switch. The switch is connected to interfaces on the UTM's (in an HA pair).

Even with QoS set on the UTM, we're still having the intermittent slow-to-connect issue. We suspect there is a configuration or other issue with the UTM.