Thread Info
  • State Suggested Answer
  • +5 person also asked this people also asked this
  • Locked Locked
  • Replies 51 replies
  • Answers 2 answers
  • Subscribers 40 subscribers
  • Views 101760 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Strange drops

jamesharper

We have a customer with a phone switchboard application that periodically freezes, either at an application level (can't click anything), or it just won't show incoming calls. In both cases it can sometimes unfreeze, and then all the calls that have come in in the meantime suddenly flash on the screen. We've ruled out AV as the cause and are now looking into the problem being at the network layer.

drop-packet-capture shows this at the time of freezing:

2017-05-23 08:58:14 0101021 IP 10.10.90.2.8779 > 10.10.10.112.43470 : proto TCP: P 3007061919:3007062115(196) win 330 checksum : 55314
0x0000:  4500 00ec 18b4 4000 3f06 a9d2 0a0a 5a02  E.....@.?.....Z.
0x0010:  <remainder of the packet redacted>
Date=2017-05-23 Time=08:58:14 log_id=0101021 log_type=Firewall log_component=Firewall_Rule log_subtype=Denied log_status=N/A log_priority=Alert duration=N/A in_dev=Lag.90 out_dev=Lag.10 inzone_id=1 outzone_id=8 source_mac=00:1a:e8:8b:15:b4 dest_mac=00:e0:20:11:08:fc l3_protocol=IP source_ip=10.10.90.2 dest_ip=10.10.10.112 l4_protocol=TCP source_port=8779 dest_port=43470 fw_rule_id=0 policytype=1 live_userid=0 userid=0 user_gp=0 ips_id=0 sslvpn_id=0 web_filter_id=0 hotspot_id=0 hotspotuser_id=0 hb_src=0 hb_dst=0 dnat_done=0 proxy_flags=0 icap_id=0 app_filter_id=0 app_category_id=0 app_id=0 category_id=0 bandwidth_id=0 up_classid=0 dn_classid=0 source_nat_id=0 cluster_node=0 inmark=0x0 nfqueue=101 scanflags=0 gateway_offset=0 max_session_bytes=0 drop_fix=0 ctflags=33554472 connid=2341170016 masterid=0 status=398 state=3 sent_pkts=N/A recv_pkts=N/A sent_bytes=N/A recv_bytes=N/A tran_src_ip=N/A tran_src_port=N/A tran_dst_ip=N/A tran_dst_port=N/A

then the same again exactly 2 minutes later (even the checksum is the same)

The connection came good another minute later.

Any idea where to look next?

thanks

James



This thread was automatically locked due to age.
Parents
  • 0

    James,

    create a Firewall Rule from 10.10.90.2 TCP 8779 to 10.10.10.112 port 43470. Log ID: 0101021 means that traffic is dropped by Firewall.

    Regards

     

  • 0 in reply to lferrara

    You can see from the packet that this is not a SYN packet, this is a packet from an established connection that has been dropped for no obvious reason. The connection has not timed out - it is still present in conntrack. In this case, the connection came good again after a bit and the application unfroze. So for some reason XG is deciding that it occasionally doesn't like something about packets in the middle of a connection.

    I have done the following:

    set advanced-firewall bypass-stateful-firewall-config add source_network 10.10.90.0 source_netmask 255.255.255.0 dest_network 10.10.10.0 dest_netmask 255.255.255.0

    set advanced-firewall bypass-stateful-firewall-config add source_network 10.10.10.0 source_netmask 255.255.255.0 dest_network 10.10.90.0 dest_netmask 255.255.255.0

    which disables connection tracking and inspection between the two networks, and the problem has not occurred in the 8ish hours since I put that in place. Previously the problem would have occurred many times in that time.

    I will raise a ticket with Sophos if I get another day of trouble free connectivity. If the customer is in agreement I might try removing those rules and see if the problem returns, just to prove the fix.

  • 0 in reply to jamesharper

    Hi,

    Could you share the case details so we may check on our end ? 

  • 0 in reply to jamesharper

    I have the exact same problem and I've already been told to look for asymmetric routing problem but the XG is the only way between my two networks as well. I've had 3 other sets of eyes on this including 2 CCIE's and they all point back at the XG. 

    "Strange Drops" is a good way to put it.

  • 0 in reply to Matthew Weiskopf

    Matthew can you tell me a little about your setup? The other user who reported similar issues is using VLAN's on top of LAG interface. Are you doing this also? Are you using LACP? I have since tried disabling one of the LAG ports so it's now only on one port, but this isn't having any effect. If you are using LAG also then I might try removing the LAG setup to test if that is somehow affecting the problem.

    I have a case open with Sophos for this. I got great response from support initially but having tried all the easy things (check asymmetric routing, turn of micro app discovery, etc) I haven't heard anything in 2 days.

    It's a HA setup so one of the devices will be removed shortly and will have SG put back on it if I don't get anywhere with this. I'm really reluctant to do this as it means the problem will likely not get solved as I have no way to test it.

    James

  • 0 in reply to jamesharper

    It goes from a pair of Cisco Routers running BGP outside because I don't want to do 500Mbps BGP links on the XG, it'd probably choke then to a pair of Cisco Core Switches on a VLAN then back to the XGs and then from the XGs to my Core Switching then to my access switching on the Native VLAN. There's no lags or port channel group or anything except between the core and access switches. I have 2 XG 330s in HA.

    Again ISP(s) -> Router(s) -> Core on a VLAN -> XGs -> Core -> Access (This is a Channel from the Cores)

    What I keep seeing is that random clients lose connection to the Internet (LAN->WAN) for 30-45 seconds and during that time I'm dropping what appears to be valid traffic. I've tried the micro-app off, no IPS, no web, no app for a single test machine I generate some internet load on and it craps out randomly. It's not the same machine dropping every time and it's random machines. I get no drops of packets pinging between router/firewall/switch/etc at that time. Just no traffic from Client -> Internet for a random client at random times.

    We've already sat there and spanned some ports on my cores and sniffed out the traffic and it's not going asymmetrically. I am not incrementing errors on any interfaces, anywhere.

  • 0 in reply to Matthew Weiskopf

    Hi guys, sorry for the delayed response. I haven't had much time in the last few days to look further into this. I have my case escalated to a tier 3 engineer who is going back and forth with the development team. The level 3 engineer is very intelligent about the XG and what we've come up with, is the XG seems to look for a user based firewall rule if a packet comes through with a, what I'll call a user header on it, even if its an inter-vlan rule and shouldn't. This seems to casue another connection to be tracked, thus making the firewall drop it because it not in the right order. It also seems to have bug for checking the user on a network even if that subnet is excluded in STAS. Right now it seems like multiple issues are causing this. I believe they have logged 2 bugs so far on my issue. To me, it seems like a very poor design of identifying users if they can't create a way to properly exclude networks. Further more, I wish it wasn't forced. I don't care what user is using inter-vlan routing. I created those rules by subnet but the tier 3 said if I chose to use STAS, I will have to created 2 rules for everything. One network based and 1 user based. Seems stupid to me.

    James, I took down the LAG and it didn't change anything. If you can, disable STAS completely and see if your issue goes away. That will surely verify if it is authentication related like mine. I know that is alot to ask.

    Matthew, I believe your issue may be helped by changing some of your settings from this link. https://community.sophos.com/kb/en-us/125468. What you are describing with the  is how the XG performs when it is in "learning mode" if you too are using STAS. The default unauth-traffic drop-period is 120 seconds.

    Mike

  • 0 in reply to MichaelBolton

    Oh yeah, I got STAS on. I'll try that.

  • 0 in reply to MichaelBolton

    When you say "using STAS", do you mean you have a rule that requires a user to have been identified? or just that "system auth cta enable" has been run?

    I have STAS and SATC enabled, but don't have any rules present that depend on a user having been identified. The user is just logged on a best effort basis.

    I have disabled "system auth cta", I will see what happens next

  • 0 in reply to jamesharper

    From what support is telling me, just having STAS enabled causes some packets to have a "user tag" on them. If the packet has the user tag but no user rule exist, the XG can drop it for whatever reason. I really hope this is a bug and not the way it was designed. That is a terrible approach IMHO. I too did not have any user based rules between networks. I was hoping the XG would just identify the users best effort like you but that does not seem to be the logic of the device. Support had me create user based rules to mimic the network rules and most drops have gone away. We are now trying to figure out why it is looking for user tags on traffic that should be excluded. I hope this helps you guys. It has be a very frustrating few weeks for me once again using the great "XG". Please let me know how it goes. I will update this thread as well when I hear back from support.

  • 0 in reply to MichaelBolton

    Hi Michael,

    I have turned off STAS (system auth cta disable) and so far the problem has just evaporated. This isn't be the first time i've thought i've fixed the problem only to find that users just weren't reporting back to me, but i've figured out a pretty reliable way to measure it now (wireshark running on a few devices watching for duplicate SYN packets) so it's looking good so far.

    I had previously enabled firewall bypassing for LAN->LAN traffic, which greatly improved the local network performance (no more Citrix session host servers not reporting in etc), but obviously that workaround wouldn't work for WAN traffic requiring NAT.

    Since turning off STAS earlier today no users have reported any issues, and since turning off the firewall bypass workaround a few hours ago, no SYN retransmissions have been reported, and I would expect to see 10-100 every 30 minutes or so as the Citrix session host server reports into the Citrix delivery controller and whatever other things Windows does in the background.

    I really appreciate you taking the time to share this! In the absence of a solution this week I was planning to take the HA passive device and put SG on it and cut over to that, and I really wasn't looking forward to doing that - since working with XG I find the old SG approach to firewall rules very tedious :)

    Could you PM me the case number you have open with Sophos? I want to pass it on to the engineers looking at my case so they aren't double handling what appears to be the same issue (i'm still getting requests to check my IPS rules as part of diagnosing the issue ).

    Thanks again

    James

  • 0 in reply to jamesharper

    I created a user specific rule yesterday morning on a whim because the engineers I am working with asked me to after we tried everything else. Funny enough that user had no drops all day. I hope it's really 'the' bug and not a fluke.

     

    I do have STAS on but it was just for reporting purposes. I hadn't had a rule set to "Match Known Users" until yesterday.

Reply
  • 0 in reply to jamesharper

    I created a user specific rule yesterday morning on a whim because the engineers I am working with asked me to after we tried everything else. Funny enough that user had no drops all day. I hope it's really 'the' bug and not a fluke.

     

    I do have STAS on but it was just for reporting purposes. I hadn't had a rule set to "Match Known Users" until yesterday.

Children
No Data