Strange drops

James,

create a Firewall Rule from 10.10.90.2 TCP 8779 to 10.10.10.112 port 43470. Log ID: 0101021 means that traffic is dropped by Firewall.

Regards

You can see from the packet that this is not a SYN packet, this is a packet from an established connection that has been dropped for no obvious reason. The connection has not timed out - it is still present in conntrack. In this case, the connection came good again after a bit and the application unfroze. So for some reason XG is deciding that it occasionally doesn't like something about packets in the middle of a connection.

I have done the following:

set advanced-firewall bypass-stateful-firewall-config add source_network 10.10.90.0 source_netmask 255.255.255.0 dest_network 10.10.10.0 dest_netmask 255.255.255.0

set advanced-firewall bypass-stateful-firewall-config add source_network 10.10.10.0 source_netmask 255.255.255.0 dest_network 10.10.90.0 dest_netmask 255.255.255.0

which disables connection tracking and inspection between the two networks, and the problem has not occurred in the 8ish hours since I put that in place. Previously the problem would have occurred many times in that time.

I will raise a ticket with Sophos if I get another day of trouble free connectivity. If the customer is in agreement I might try removing those rules and see if the problem returns, just to prove the fix.

James,

I am having the exact same issue as you. Support could not find an answer yet. I had to move all routing to our core switches for it to work properly. I had the exact scenario where I had the firewall rules in place but sometimes the XG just didn't allow the connection. Unfortunately, this is the 3rd case I have open right now with no answers. Support is a joke. The enginner saw this happening live and was baffled. I am very curious as to what they say to you. Please keep us updated. My next step is to re-load the primary device via the ISO of firewall OS and restore a backup. Once restored, re-install the OS on the auxiliary unit and setup HA.

Mike 

Hi Mike,

Thanks for the confirmation that you are seeing similar problems. I wonder if it's possible to log a "class action" support case :)

I notice you mention you are using HA - have you tried reproducing this problem with HA turned off? That's what I might be trying next, although I should probably let the client have a few days of trouble free networking before I do that.

When faced with this sort of problem I normally turn to tcpdump, but to properly investigate I need to capture a whole days worth of traffic, or at least lots more than the 100000 packets that XG's tcpdump filedump lets me capture. I can capture on the PC, but the packets are getting dropped from the PBX to the PC, so the PC wouldn't see them and I wouldn't get the whole picture.

I wonder if tcpdump from Debian would run on XG... even if I have to copy all the deps into a chroot environment to make it work :)

thanks

James

Hi James,

I did try with a single unit for a day or so and that did not help in our case. I reloaded the OS on both units but I have not tested it again. I moved all of the routing back to our core switches as I fought it for a few weeks. I hope I get a chance in the next few days to put the gateway back on the XG. I can't believe the XG is failing at the most basic of firewall functions. My output is identical to yours. The connections show in the connection table and has a connection ID. In my case it is computers accessing a SQL database when the connections get dropped. Nothing strange about the packets at all.

2017-05-11 09:44:40 0101021 IP 10.11.0.31.53689 > 10.10.0.18.1433 : proto TCP: P 1318068363:1318068473(110) win 256 checksum : 5546
0x0000:<remainder of the packet redacted>
Date=2017-05-11 Time=09:44:40 log_id=0101021 log_type=Firewall log_component=Firewall_Rule log_subtype=Denied log_status=N/A log_priority=Alert duration=N/A in_dev=Uplink_LAG.19 out_dev=Uplink_LAG inzone_id=1 outzone_id=1 source_mac=18 6:da:1f:38:9b dest_mac=00:e0:20:11:0a:66 l3_protocol=IP source_ip=10.11.0.31 dest_ip=10.10.0.18 l4_protocol=TCP source_port=53689 dest_port=1433 fw_rule_id=0 policytype=1 live_userid=0 userid=0 user_gp=0 ips_id=0 sslvpn_id=0 web_filt _id=0 hotspot_id=0 hotspotuser_id=0 hb_src=0 hb_dst=0 dnat_done=0 proxy_flags=0 icap_id=0 app_filter_id=0 app_category_id=0 app_id=0 category_id=0 bandwidth_id=0 up_classid=0 dn_classid=0 source_nat_id=0 cluster_node=0 inmark=0x0 nfque =100 scanflags=0 gateway_offset=0 max_session_bytes=0 drop_fix=0 ctflags=40 connid=180176096 masterid=0 status=398 state=3 sent_pkts=N/A recv_pkts=N/A sent_bytes=N/A recv_bytes=N/A tran_src_ip=N/A tran_src_port=N/A tran_dst_ip=N/A tran st_port=N/A

(IP's changed)

I agree with you on trying to capture packets. My next plan is to port mirror the interface on the switch for one of the computers affected and the database server. Then run Wireshark until I capture it. That may be next week though before I can. 

Mike

James,

I am having another issue with authentication that may or may not be associated. Just trying to get all of the details I can. Is your customer using STAS as well?

Mike

Yes using STAS (and SATC), but not enforced - user information is logged if available but not required by any rules. I had considered SATC as an issue as it integrates into the system a bit deeply, but this is happening on PC's as well as Citrix.

I do notice from your packet that you seem to be using VLAN on LAG. Are you using LACP too?

I might try disabling a port so the LAG only uses one port. It's not the same as not using LAG at all but is easy enough to try - disabling the LAG would require an uncomfortable amount of reconfiguration on XG.

One thing I finally managed to capture is another case that i've seen a bit where users get "Page could not be displayed". A tcpdump on the PC shows SYN packets being sent but nothing is logged on the router - it's like the packets never reach it.

If that's happening mid connection too then that might explain the mid-stream packet drops we've both seen.

I have been running a tcpdump on one of the servers on this network, filtering by SYN packets, and looking for SYN retransmissions.

There are the expected retransmissions here and there which is expected, but there are definite periods where there are an excessive number of retransmissions, both to internal hosts that I know are up, and to external hosts. This matches my experience in the past where suddenly pages stop loading, even though already established connections seem to be okay (and my connection via Citrix appears unaffected).

What could be causing this? I don't have any IPS or DoS turned on on these networks, and there is nothing in any logs at this time.

The Sophos XG never logs any evidence of these packets, so I don't know if they are reaching the XG or not. My hope would be to run a tcpdump side by side on the server and the XG and compare results, but tcpdump is crippled on XG so this isn't possible.

I have enough now to open a case though, so I guess i'll do that next.

James

Hi,

Could you share the case details so we may check on our end ? 

I have the exact same problem and I've already been told to look for asymmetric routing problem but the XG is the only way between my two networks as well. I've had 3 other sets of eyes on this including 2 CCIE's and they all point back at the XG. 

"Strange Drops" is a good way to put it.

Matthew can you tell me a little about your setup? The other user who reported similar issues is using VLAN's on top of LAG interface. Are you doing this also? Are you using LACP? I have since tried disabling one of the LAG ports so it's now only on one port, but this isn't having any effect. If you are using LAG also then I might try removing the LAG setup to test if that is somehow affecting the problem.

I have a case open with Sophos for this. I got great response from support initially but having tried all the easy things (check asymmetric routing, turn of micro app discovery, etc) I haven't heard anything in 2 days.

It's a HA setup so one of the devices will be removed shortly and will have SG put back on it if I don't get anywhere with this. I'm really reluctant to do this as it means the problem will likely not get solved as I have no way to test it.

James

It goes from a pair of Cisco Routers running BGP outside because I don't want to do 500Mbps BGP links on the XG, it'd probably choke then to a pair of Cisco Core Switches on a VLAN then back to the XGs and then from the XGs to my Core Switching then to my access switching on the Native VLAN. There's no lags or port channel group or anything except between the core and access switches. I have 2 XG 330s in HA.

Again ISP(s) -> Router(s) -> Core on a VLAN -> XGs -> Core -> Access (This is a Channel from the Cores)

What I keep seeing is that random clients lose connection to the Internet (LAN->WAN) for 30-45 seconds and during that time I'm dropping what appears to be valid traffic. I've tried the micro-app off, no IPS, no web, no app for a single test machine I generate some internet load on and it craps out randomly. It's not the same machine dropping every time and it's random machines. I get no drops of packets pinging between router/firewall/switch/etc at that time. Just no traffic from Client -> Internet for a random client at random times.

We've already sat there and spanned some ports on my cores and sniffed out the traffic and it's not going asymmetrically. I am not incrementing errors on any interfaces, anywhere.

It goes from a pair of Cisco Routers running BGP outside because I don't want to do 500Mbps BGP links on the XG, it'd probably choke then to a pair of Cisco Core Switches on a VLAN then back to the XGs and then from the XGs to my Core Switching then to my access switching on the Native VLAN. There's no lags or port channel group or anything except between the core and access switches. I have 2 XG 330s in HA.

Again ISP(s) -> Router(s) -> Core on a VLAN -> XGs -> Core -> Access (This is a Channel from the Cores)

What I keep seeing is that random clients lose connection to the Internet (LAN->WAN) for 30-45 seconds and during that time I'm dropping what appears to be valid traffic. I've tried the micro-app off, no IPS, no web, no app for a single test machine I generate some internet load on and it craps out randomly. It's not the same machine dropping every time and it's random machines. I get no drops of packets pinging between router/firewall/switch/etc at that time. Just no traffic from Client -> Internet for a random client at random times.

We've already sat there and spanned some ports on my cores and sniffed out the traffic and it's not going asymmetrically. I am not incrementing errors on any interfaces, anywhere.

Hi guys, sorry for the delayed response. I haven't had much time in the last few days to look further into this. I have my case escalated to a tier 3 engineer who is going back and forth with the development team. The level 3 engineer is very intelligent about the XG and what we've come up with, is the XG seems to look for a user based firewall rule if a packet comes through with a, what I'll call a user header on it, even if its an inter-vlan rule and shouldn't. This seems to casue another connection to be tracked, thus making the firewall drop it because it not in the right order. It also seems to have bug for checking the user on a network even if that subnet is excluded in STAS. Right now it seems like multiple issues are causing this. I believe they have logged 2 bugs so far on my issue. To me, it seems like a very poor design of identifying users if they can't create a way to properly exclude networks. Further more, I wish it wasn't forced. I don't care what user is using inter-vlan routing. I created those rules by subnet but the tier 3 said if I chose to use STAS, I will have to created 2 rules for everything. One network based and 1 user based. Seems stupid to me.

James, I took down the LAG and it didn't change anything. If you can, disable STAS completely and see if your issue goes away. That will surely verify if it is authentication related like mine. I know that is alot to ask.

Matthew, I believe your issue may be helped by changing some of your settings from this link. https://community.sophos.com/kb/en-us/125468. What you are describing with the  is how the XG performs when it is in "learning mode" if you too are using STAS. The default unauth-traffic drop-period is 120 seconds.

Mike

Oh yeah, I got STAS on. I'll try that.

When you say "using STAS", do you mean you have a rule that requires a user to have been identified? or just that "system auth cta enable" has been run?

I have STAS and SATC enabled, but don't have any rules present that depend on a user having been identified. The user is just logged on a best effort basis.

I have disabled "system auth cta", I will see what happens next

From what support is telling me, just having STAS enabled causes some packets to have a "user tag" on them. If the packet has the user tag but no user rule exist, the XG can drop it for whatever reason. I really hope this is a bug and not the way it was designed. That is a terrible approach IMHO. I too did not have any user based rules between networks. I was hoping the XG would just identify the users best effort like you but that does not seem to be the logic of the device. Support had me create user based rules to mimic the network rules and most drops have gone away. We are now trying to figure out why it is looking for user tags on traffic that should be excluded. I hope this helps you guys. It has be a very frustrating few weeks for me once again using the great "XG". Please let me know how it goes. I will update this thread as well when I hear back from support.

Hi Michael,

I have turned off STAS (system auth cta disable) and so far the problem has just evaporated. This isn't be the first time i've thought i've fixed the problem only to find that users just weren't reporting back to me, but i've figured out a pretty reliable way to measure it now (wireshark running on a few devices watching for duplicate SYN packets) so it's looking good so far.

I had previously enabled firewall bypassing for LAN->LAN traffic, which greatly improved the local network performance (no more Citrix session host servers not reporting in etc), but obviously that workaround wouldn't work for WAN traffic requiring NAT.

Since turning off STAS earlier today no users have reported any issues, and since turning off the firewall bypass workaround a few hours ago, no SYN retransmissions have been reported, and I would expect to see 10-100 every 30 minutes or so as the Citrix session host server reports into the Citrix delivery controller and whatever other things Windows does in the background.

I really appreciate you taking the time to share this! In the absence of a solution this week I was planning to take the HA passive device and put SG on it and cut over to that, and I really wasn't looking forward to doing that - since working with XG I find the old SG approach to firewall rules very tedious :)

Could you PM me the case number you have open with Sophos? I want to pass it on to the engineers looking at my case so they aren't double handling what appears to be the same issue (i'm still getting requests to check my IPS rules as part of diagnosing the issue ).

Thanks again

James

I created a user specific rule yesterday morning on a whim because the engineers I am working with asked me to after we tried everything else. Funny enough that user had no drops all day. I hope it's really 'the' bug and not a fluke.

I do have STAS on but it was just for reporting purposes. I hadn't had a rule set to "Match Known Users" until yesterday.

James, glad to hear you may be seeing some light at the end of the tunnel too :). PM coming to you shortly. I hope this resolves your issue for now until Sophos can figure out how to fix it or Ill just stop using STAS all together. It is going to be very disappointing if Sophos cannot figure out a way to allow STAS for reporting purposes and not block or drop traffic. Let's see what happens. PM me back or post if you get any updates on your case.

Matthew, I am in the exact same situation. I only use STAS for reporting. I too hope it's a bug and not by design. Hopefully I will hear something back soon from the GES or dev team.

Mike 

Hi MichaelBolton,

Could you share with us the case# so we may look into it?

Aditya,

It is 7227282.

Mike

Anyone have any updates to this issues? Did disabling STAS seem to take care of it for you? From what I am told, the amount of authentication traffic going into the XG is causing a panic condition in the access_server. Development is working to get a fix in for MR5. I can no longer wait. XG is dropping too much traffic that it shouldn't even be looking at. What I was initally told was XG will try to route traffic to a firewall rule if the packet has a "user" tied to it. I have shown Sophos support that mine is still dropping traffic even though no user data is in the packet and the firewall rules are not user based. I really hope they get this fixed soon. AlanT I know you guys are working to fix issues with the development but I am ready to jump ship to another vendor. I have a firewall that can't be used when a feature Sophos advertises is enabled. I am also being told now that in MR4, firewall acceleration is disabled when you run in HA which is FastPath packet optimization. That is another feature that XG was bought for and can't be used and no one even know unless you get someone on the GES team.

Mike