Thread Info
  • State Verified Answer
  • +1 person also asked this people also asked this
  • Locked Locked
  • Replies 9 replies
  • Answers 1 answer
  • Subscribers 29 subscribers
  • Views 2061 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Traffic Getting Lost In A Black Hole

Gary Parr

I'm trying to debug an issue with a Honeywell WiFi thermostat that can't seem to connect to the Honeywell servers. My firewall rule for the thermostat covers all my IOT devices and is wide open, no IPS or scanning, all protocols allowed, etc. The XG logs show nothing dropped or blocked. So I dug deeper and captured some packets.

I should stop here and explain that my network is currently a bit over complicated... I have an Ubiquiti ERL3 between the WAN and my DMZ, and the XG between the DMZ and the LAN.

Anyway, while running tcpdump on both the ERL3 and XG for all protocols, all ports, filtering on the Honeywell server IP as the host I noticed something odd. XG captures a ton of traffic destined for that Honeywell server but nothing shows on the ERL3 packet dump. So... the XG must be blocking something, right? Well.. the PCAP file shows normal SYN/ACK/PSH/ACK (and an unfortunate RST) and nothing getting dropped.

What?

So... try something else. Fire up a command prompt and ping the Honeywell server. Yup, that registers packets on both the XG and the ERL3.  So, point a web browser to the Honeywell server on 443 (same port thermostat is connecting to) and again, packets on both XG and ERL3. Try a telnet to 443... only XG gets those packets. The ERL3 doesn't see the telnet session happen. And it does happen... telnet connects.

I can't figure out how tcpdump on the ERL3 is not picking up that traffic unless the XG isn't actually sending it where it's supposed to... which isn't the case as far as I can tell.  Can anyone explain that to me?

Thanks,

Gary



This thread was automatically locked due to age.
  • +1

    Hi Gary,

    on the XG -> Web do you have disallow invalid https (443) settings ticked? A couple of my IoT devices or attempt to use 443 to a Chinese University helpdesk, so it gets blocked.

    Ian

  • 0 in reply to rfcat_vk

    Un-Freaking believable. That worked.

    I was stupid enough to assume that the settings under the "HTTPS Decryption and Scanning" section only affected, well, HTTPS Decryption and Scanning.  And, since the firewall rule matching the MAC of the thermostat is at the top of the list and has HTTPS Decryption and Scanning un-checked, I stupidly assumed that HTTPS traffic matching that rule was not being, you know, decrypted or scanned.

    To my understanding, there is no way to create an exceptions for the "Invalid Certificates" (there are some sites my wife hits that do cert pinning, so I learned this one the hard way) so I'm assuming there is also no way to create exceptions for the unrecognized SSL protocols either? Which, as I now know, affects the whole freaking FW even if you tell it not to decrypt and scan HTTPS.

    BAH!

    Follow-up... why did I see nothing obvious in the logs or the packet captures to indicate this was happening? Anyone know where the blocked SSL protocols get logged?

    -GP

  • 0 in reply to Gary Parr

    Hello 

    I've been having the same issue since upgrading to v18.  I've tried the suggestion above but it doesn't seem to help anymore.  

    What really baffles me is the logger show the traffic as Accepted with the firewall rule I've set. 

     

    Here are my current settings.

    Is there any way to debug this further?  I can see outbound traffic in the log to the Honeywell IP address range being send and Accepted in the log but there is not any response.  It doesn't seem that the traffic is truly being sent. 

  • 0 in reply to TexasRaptor

    Hi,

    which parts of logviewer have you reviewed?

    I suspect that you are being blocked by the SSL/TLS inspection, try enabling the web proxy with allow all. If as I suspect the controller does not use http/s there will not be any issues but you will see traffic in the logs.

    Ia

  • 0 in reply to rfcat_vk

    Ok I've updated the firewall rule for the thermostats to use the Web Proxy, and set the Web Policy to Allow All

    I'm not get any log under the SSL/TSL Inspection. I do see the outgoing connection from the Thermostat to the Honeywell Servers. 199.62.84.151-3

    This address doesn't appear under any other section category in the log viewer. beside Firewall 

    While the firewall Log show the outgoing connection is Accepted the byte count for the rule remains at 0 bytes recived.  So it seems like the traffic is getting dropped and never even sent to the server for a response.

     

     

     

  • 0 in reply to TexasRaptor

    Ok, I think there might be a bug in v18 with the New NAT rules and how their applied.  

    My understanding is a single NAT rule and cover multiple firewall rules.   I have a general LAN to WAN firewall rule with a linked source NAT.  This should SNAT all outbound traffic including my Thermostats.  My understanding was not every Firewall rule needs it own NAT rule.  But when I added a dedicated linked SNAT rule for the thermostats then traffic started coming in. 

    This seems like a bug in the v18.  Wasn't the one of the reasons for the new v18 way of decoupling firewall rules and NAT rules from each other so a single NAT could cover multiple firewall rules?  Is this a bug or did I miss something?

    It seems to me that the additional SNAT rule is redundant. Yet without it traffic was not being sent properly and its necessary for the it work. 

     

      

  • +1 in reply to TexasRaptor

    Hi,

    I tried all those sd-wan and linked Nat rules on migrating to the v18 and deleted all the sd-wan and linked Nat, they were too hard to manage.

    i ended upwith one general Masq rule for all my devices, voip phones, PCs, macs, iPhones, Iot devices and i see details in logs. If using the http proxy you will not see any details in tls log viewer.

    i use a mix of http proxy and tls inspection.

    i have seperate lans for iot, voip and general use Devices internet access,all managed with clientless groups and static ip assignments.

    ian

  • 0 in reply to TexasRaptor

    It appears to me that if a NAT is linked to a firewall rule than it will only perform the NAT for traffic matching the firewall rule # it is linked to.  So regardless of if the traffic would match the NAT rule it would not be applied if the traffic matched different firewall rule#.  So if a I add a generic SNAT at the top of the NAT table then it will get applied to all outbound firewall rules even ones with a linked NAT to a different NAT rule #.  So there seems to be an inconsistency. 

    This seems to be the behavior I'm not sure if it correct and the intended behavior for v18 or not.  

  • 0 in reply to TexasRaptor

    Hi,

    you have basically summarised why I gave up on linked Nat rules.

    ian