I have two Internet providers and use eth1 and eth6 for them. I have Uplink Balancing on with the eth6 interface preferred. I have a Multipath Rule in place for SMTP that is "Any -> SMTP-All -> Any -> Uplink Interfaces". The idea with that is to use whichever Uplink interface answered an incoming message and use the eth6 (preferred in balancing) for outbound.
At this point, I have both eth1 and eth6 on the same switch as the routers for both carriers. As you'd expect, both subnets are completely different and there is no interest in dealing with BGP.
I have a Multipath rule that pushes all web browsing out the eth6 interface and that has been flawless. This problem only involves SMTP and we have the email subscription.
I was doing an e-mail upgrade today, so the mail server came down. Prior to taking it down, however, I went to the vendor who bags our mail, scans for spam/malware, and then sends to us. They test availability of our mailservers with a quick EHLO. Both eth1 and eth6's addresses came back as good. I turned on mailbagging so that I wouldn't get any mail during the server upgrade.
After the server was done, I ran the availability test again and eth6 address failed (eth1 was fine)!
So to gather some data, I ssh'd to the firewall and kicked off a tcpdump on eth1 for just port 25 traffic. Then I went to the mailbagger and ran the availability test again. The output from the tcpdump is below - notice how the eth6 IP address is being responded to on eth1's physical interface!
firewall:/home/login # tcpdump -ni eth1 port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 376 bytes
11:53:47.593173 IP EMAIL_provider_IP.27736 > eth6_IP.25: S 4085297002:4085297002(0) win 5840
11:53:47.595668 IP eth6_IP.25 > EMAIL_provider_IP.27736: S 373234996:373234996(0) ack 4085297003 win 14480
11:53:50.593197 IP EMAIL_provider_IP.27736 > eth6_IP.25: S 4085297002:4085297002(0) win 5840
11:53:50.595620 IP eth6_IP.25 > EMAIL_provider_IP.27736: S 420109780:420109780(0) ack 4085297003 win 14480
11:53:56.590727 IP EMAIL_provider_IP.27736 > eth6_IP.25: S 4085297002:4085297002(0) win 5840
11:53:56.593146 IP eth6_IP.25 > EMAIL_provider_IP.27736: S 513821130:513821130(0) ack 4085297003 win 14480
11:54:08.593270 IP EMAIL_provider_IP.27736 > eth6_IP.25: S 4085297002:4085297002(0) win 5840
11:54:08.595645 IP eth6_IP.25 > EMAIL_provider_IP.27736: S 701360034:701360034(0) ack 4085297003 win 14480
11:54:17.587839 IP EMAIL_provider_IP.4081 > eth1_IP.25: S 2389992971:2389992971(0) win 5840
11:54:17.590267 IP eth1_IP.25 > EMAIL_provider_IP.4081: S 2592030501:2592030501(0) ack 2389992972 win 14480
11:54:17.648024 IP EMAIL_provider_IP.4081 > eth1_IP.25: . ack 1 win 5840
11:54:17.651802 IP eth1_IP.25 > EMAIL_provider_IP.4081: P 1:30(29) ack 1 win 14480
11:54:17.709671 IP EMAIL_provider_IP.4081 > eth1_IP.25: . ack 30 win 5840
11:54:17.709977 IP EMAIL_provider_IP.4081 > eth1_IP.25: F 1:1(0) ack 30 win 5840
11:54:17.712104 IP eth1_IP.25 > EMAIL_provider_IP.4081: P 30:68(38) ack 2 win 14480
11:54:17.712119 IP eth1_IP.25 > EMAIL_provider_IP.4081: F 68:68(0) ack 2 win 14480
11:54:17.769929 IP EMAIL_provider_IP.4081 > eth1_IP.25: R 2389992973:2389992973(0) win 0
So as you might expect, the eth6 address is now unable to accept incoming email. If I did something wrong, please let me know what I screwed up!
This thread was automatically locked due to age.
