Additional IP's

Question

We have a block of 16 IPv4's which we have as additional IP's on the primary interface. We're starting to find this a little limited and I was thinking of moving these IP's of the primary interface. 
 Now the connection is currently like so: 
 ISP NTE > Switch (vlan 111> UTM WAN 1 with additional IP's (uses 2 physical ports on switch) 
 So I'm thinking here: 
 ISP NTE > Switch (Vlan 111) > UTM WAN 1, UTM WAN 2, UTM WAN 3 (which would use 4 physical ports and I could add additional IP's to any of the physicals) or purchase a UTM adapter (SG330 8 port adapter) 
 THis would allow me to select different physical WAN's for IPSec etc 
 
 Anyone tried this?

DouglasFoster · Answer

You should never have two NICS for the same device on the same subnet, unless you are doing NIC Teaming (Link Aggregation Groups). 
 Here is my understanding of why it is does not work: 
 
 Assume router has address a.b.c.1 and UTM has addresses a.b.c.2 (on interface A2) and a.b.c.3 (on interface A3) 
 The TCP/IP stack is layered -- at an upper level, the packet is created with a sender address of a.b.c.2 
 When the packet reaches the routing layer, the router is only looking for a path to the next hop (a.b.c.1). It picks one of the two NICs without trying to match the source address on the packet to the listen address associated with the NIC. Sometimes it picks wrong. 
 When other network devices see that a packet with source address a.b.c.3 came from interface A2, they all update their routing tables to send all future packets to the MAC address for A2. 
 When they try to use that new MAC address, the packet is rejected, so the sending device has to send a new ARP request, from which it learns that the packet so be sent to the MAC address for A3. This repeats and is called "ARP flapping" 
 Better network equipment will detect "ARP flapping", go into threat evasion mode, and block traffic from both NICs. Less intelligent devices will simply have lousy performance. 
 
 With Windows, it is even worse, I think because there are problems with LAN-level protocols as well. I have some unhappy experience here, and have read Microsoft's warnings not to do it. 
 With UTM, there may be some ways to limit the damage, by tying certain routes to certain interfaces, but I think it will be difficult to get implemented and even more difficult to sustain. 
 I believe the options are: 
 
 Use a 1Gbps NIC until it is saturated, then upgrade to a 10Gbps fiber connection if you need more capacity. 
 Use a 1Gbps NIC until it is saturated, then add a second NIC with NIC teaming. 
 Use a 1Gbps NIC until it is saturated, then add a second NIC on a different subnet, and use uplink balancing.