Thread Info
  • State Suggested Answer
  • Locked Locked
  • Replies 8 replies
  • Answers 1 answer
  • Subscribers 27 subscribers
  • Views 2548 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Cannot SSH to bridged Eve-NG host through XG

Bryon Adams

I've set up Eve-NG on a host on the same VLAN as my PC, then set up OSPF to advertise any new networks I add to Eve-NG labs. That part seems to work fine, I can ping through the lab to VMs on different networks. Eve-NG lives on ESXi with configuration to allow forged packets configured as far as I can tell. The next part I'd like to get working is SSH.

I'm not sure what's happening to cause SSH to fail. SSH from my PC it times out, I do not see the inbound SSH attempts with debugging enabled on my IOSv router, and looking at logs from within XG throws a bunch of denied packets stating either "Invalid packet" or "Invalid TCP state." SSH from one IOSv to another works within the lab so that appears to be configured correctly. Interestingly I can successfully ping to hosts that I cannot SSH to.

PS C:\Users\Bryon> ssh 172.16.1.2
ssh_exchange_identification: read: Connection timed out
PS C:\Users\Bryon> ping 172.16.1.2

Pinging 172.16.1.2 with 32 bytes of data:
Reply from 172.16.1.2: bytes=32 time=2ms TTL=254
Reply from 172.16.1.2: bytes=32 time=1ms TTL=254

Topology:

tcpdump:

18:26:57.659145 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [S], seq 2463688266, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:26:57.659145 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [S], seq 2463688266, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:26:57.660112 Port1.4, OUT: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [S], seq 2463688266, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:26:57.660127 Port1, OUT: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [S], seq 2463688266, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
18:26:57.662913 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [.], ack 1819393893, win 65392, length 0
18:26:57.662913 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [.], ack 1, win 65392, length 0
18:26:57.665800 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:26:57.665800 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:26:57.966417 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:26:57.966417 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:26:58.567591 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:26:58.567591 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:26:59.663516 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [.], ack 1, win 65392, length 0
18:26:59.663516 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [.], ack 1, win 65392, length 0
18:26:59.768380 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:26:59.768380 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:27:02.168587 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:27:02.168587 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:27:03.665942 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [.], ack 1, win 65392, length 0
18:27:03.665942 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [.], ack 1, win 65392, length 0
18:27:06.969409 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:27:06.969409 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [P.], seq 0:33, ack 1, win 65392, length 33
18:27:11.665775 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [.], ack 1, win 65392, length 0
18:27:11.665775 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [.], ack 1, win 65392, length 0
18:27:16.570010 Port1, IN: ethertype IPv4, IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [R.], seq 33, ack 1, win 0, length 0
18:27:16.570010 Port1.4, IN: IP 192.168.4.104.59004 > 172.16.1.2.ssh: Flags [R.], seq 33, ack 1, win 0, length 0

I added this rule to try and help but it didn't seem to help. I see only outbound traffic hitting it. Ducknet is the zone, interface 1.4. Ducknet_v4 is the network 192.168.4.0/24. Eve-NG Networks is the network 172.16.0.0/16.

I'm more than happy to provide any information I've left out, tried to throw anything I could think of that will help but I'm new to firewalls in general so I'm sure I forgot something. Thanks!



This thread was automatically locked due to age.
Parents
  • 0

    Hello Bryon,

    Thank you for contacting the Sophos Community!

    It seems like the device with 172.16.1.2 is not replying to the SSH packets. However, I can see ACK packets being sent to 172.16.1.2 despite the fact that there are no reply packets, so I think you must have some assymetric routing going on, traffic going out one interface and coming in a different one. 

    Which is the reason you are seeing in the Logs the packets being dropped by the XG as Invalid packets.

    You could try doing the following from the console in the XG: (5>4)

    console > set advanced-firewall bypass-stateful-firewall-config add source_network 192.168.4.0 source_netmask 255.255.255.0 dest_network 172.16.1.0 dest_netmask 255.255.255.0 

    Try first this, I don't think you need a  reversed by-pass. (the Source Network and Dest Network would be the opposite).

    You can also set it for only a host instead of the whole network, you can use Tab to auto-complete. 

    Regards,

  • 0 in reply to emmosophos

    Setting that bypass configuration causes SSH to connect now to the remote IOSv host. I'm not sure if I 100% understand what a bypass is designed for so I'm not sure if that's something I would want to leave in place or if this is a normal troubleshooting step for Sophos XG. Could you explain for me? Seems like we're asking the firewall to not filter traffic for these source/destination networks based on any rules I have.

    Admittedly I'm a little confused as to why that works, I was following the asymmetric routing angle you suggested and saw that IOSv was in fact responding via some debugs but didn't actually see them on the ESXi host via tcpdump. Since it's all virtual networking I'm fine to say it's probably just a bug or I'm capturing traffic on ESXi wrong.

  • 0 in reply to Bryon Adams

    Hello Bryon,

    Thank you for the follow-up!

    So the XG checks for "stateful packets", meaning it checks the conntrack for every packet that traverses the XG, in your case what is happening is that the  traffic from 192.168.4.104 is going out let;s say interface Port1, and then the XG is expecting the reply packet to follow the same path, but in this case the tcpdump doesn't show any packet coming back to you from 172.16.1.2, yet, 192.168.4.104 is sending [.] ACK packets, but the XG is not seeing the [.] or even SYN packets coming to the 172.16.1.2

    So basically the return packet is following a different path to return, which is causing the XG to block the packets as it can't find the conntrack that matches those [.] and Syn packets.

    What by passing the stateful firewall does, is it tell the XG to ignore this, simply route the packets and don't inspect them, which is the reason why you can SSH now. 

    Sometimes it is not possible to avoid asymmetric routing due to how the traffic flows, but ideally, you would avoid this. 

    Regards,

  • 0 in reply to emmosophos

    Is this an oddity with my setup for XG then? I wouldn't expect asymmetric routing here since it's basically one flat network on VLAN 4, just that it hairpins back out the same interface it came in on. However it's sending and receiving on both Port1 and Port1.4, where I'd expect to only see Port1.4 being used since all traffic to/from XG should be on VLAN 4's interface, Port1.4.

    I see in on Port1, in on Port1.4, then out on Port1.4, out on Port1. The the replies are the inverse. It just doesn't make sense that pings work just fine but SSH doesn't without the bypass.

    If a bypass is a normal thing in this kind of scenario I'm 100% fine with it, I'm just fighting my own bias here I think.

Reply
  • 0 in reply to emmosophos

    Is this an oddity with my setup for XG then? I wouldn't expect asymmetric routing here since it's basically one flat network on VLAN 4, just that it hairpins back out the same interface it came in on. However it's sending and receiving on both Port1 and Port1.4, where I'd expect to only see Port1.4 being used since all traffic to/from XG should be on VLAN 4's interface, Port1.4.

    I see in on Port1, in on Port1.4, then out on Port1.4, out on Port1. The the replies are the inverse. It just doesn't make sense that pings work just fine but SSH doesn't without the bypass.

    If a bypass is a normal thing in this kind of scenario I'm 100% fine with it, I'm just fighting my own bias here I think.

Children
  • 0 in reply to Bryon Adams

    Hello Bryon,

    Just to add, for PINGs asymmetric routing doesn't affect the flow since it doesn't use TCP or UDP. The SSH breaks the community or doesn't work because the handshake can't be completed.

    Regards,

  • 0 in reply to emmosophos

    Oh, so when XG is looking at the ping traffic, it doesn't really have a way to say "this reply doesn't match" since there isn't enough information in a ping to say it's not a valid response. SSH on the otherhand, since it's TCP, gives XG more information to work off of and it can identify that the responses don't line up with the stream I sent out to begin the handshake. Am I on the right line of thinking there?

    Edit: I still don't really get why XG thinks it's asymmetric either, I think that's probably my biggest hangup.