Just looking through my old posts and realized I never posted my solution here. After a lot of internet searching I came up with the following and this problem was solved. It's been so long, I realize these notes are cryptic, but hopefully it pushes you in the right direction. Pretty sure the key was the step 3 below.
www.astaro.org/.../44614-utm9-active-passive-ha-vmware-3.html
www.astaro.org/.../57207-ha-switch-config.html
Thanks for the help. I solved the problem using mulitiple sources. I added them to one, which worked for me:
In the CLI of the UTM (on the console) you have the do the following [1][2]
1. Set MTU on HA interface to 1500 with the command "cc set ha advanced mtu 1500" in UTM CLI, to prevent use of jumbo frames. (Will not work)
2. Use a dedicated VLAN in ESXi for the two UTM HA interfaces.
3. Disable virtual HA MACs with the command: "cc set ha advanced virtual_mac 0" in UTM CLI to ensure that VMs residing on same vSwitch as the passive ASG can communicate with the active UTM on other vSwitch.
ethernet0.ignoreMACAddressConflict = "TRUE"
ethernet1.ignoreMACAddressConflict = "TRUE"
ethernet2.ignoreMACAddressConflict = "TRUE"