DHCP Relay

Hey, ah, so this bug is still there in v19. I expected this. We're having it since moving from v17 ti v18.

Sometimes DHCP Server on XG will fail, sometimes DHCP Relay to Windows Server will fail.

Re-Saving does not help on our XG, we need to delete and recreate a random DHCP Server or Relay to get it working again.

Have a case open currently but no technical feedback after weeks.

05158330 / 05128430

I ask you to open a case and refer to my case no#

do you have CheckMK?

We can see it in the DHCP Statistics on our Windows DHCP Server for Relay.

issue start= clients not getting DHCP address

issue fixed= we recreated one of a dozend DHCP Relays on XG

Likely it is actually a Flood Prevention of DHCP Relay and not a Bug.

The point is: DHCP Relay has a Builtin Feature called Flood Protection. 

If the DHCP Server was not reachable for the first DHCP requests, the relay will stop for some time to prevent a DHCP storm to the Servers. 

This prevents networks to go down from DHCP floods of requests which nobody answers. 

All the cases, i worked in, this was the case. The DHCP server was not reachable or did not answer for some reasons. 

And this screenshot actually makes me wonder: Why are there so many requests in this window? 

A Tcpdump of those requests in the affected area would help to see, who to blame. DHCP Relays work with the DHCP agent IPs. This means, if the server answers incorrectly, this could cause this. Only proveable in a tcpdump. 

I have provided full network dumps to the case while the issue was happening.

we have IPS Flood protection disabled on that XG

This happens throughout the day. I would say from my monitoring, there were no enormous high DHCP requests.

The Flood Protection is not the UDP Flood Protection. Instead a Flood Protection of the own service. 

So if you have a tcpdump, you should see the incoming requests, the forwarded Requests with DHCP agent IP and the packets going back. Where is the issue sitting in terms of DHCP Relay? Why is the client still requests? 
You can read about DHCP Relays here: https://www.netmanias.com/en/post/techdocs/6000/dhcp-network-protocol/understanding-dhcp-relay-agents#:~:text=Relay%20Agent%20IP%20address%3A%20The,DHCP%20Discover%20message%20was%20received.

Good to know; but I have given up opening cases with Sophos. It's like talking to my wall, so it's a waste of time. Only this forum offers some help - with special thanks to LuCar Toni - but Sophos support doesn't even deserve that name (and yes I have the Enhanced Pro Plus or whatever it's called)

The community cannot create Bug IDs. So if there is a issue, it has to go through deeper analyse with Support to get a Bug ID and sorted out. It has to be the go to for all scenarios. But issues like this is always hard to figure out, if you could resolve them by yourself by reapplying the config. Because no product has extended reporting and logging enabled all the time. So to figure out what happen, it is a nightmare for a support after the incidence in every product. 

Totally get your point. It's just our overall experience with the support - not only regarding this case. But no need to discuss here. I gave up as i said... But another "thank you" to you, cause your comments help us a lot...

Hi,

I have the same problem, but resaving or recreating the rules does not help. I had to roll back to 18.5.3 to get it working again.

Two of our 5 XGs are affected, the structure of our networks are exactly the same. The affected sides are twice as big as the not affected.

@LHerzog Are there any news about your tickets?

@LuCarToni Is it possible to configure the Flood Protection for the DHCP-Relay?

Hi LHerzog - reviewed your provided case ID's and it could possibly be tied to Development ID NC-86351. Once you've provided the output & pcap requested, please let us know and we can help escalate the case if needed.

our case is closed now until the next time it happens. Sophos Support is now requesting more information.

I post it here so someone else can collect that logs and dumps when the issue occurs.

This is regarding your service request number 05158330.
 
The below information will help us to proceed with troubleshooting the issue faster. Please provide us with the following information so we can provide you with the solution at the earliest.
 
 Please share the output of the below commands in both working and non-working scenario and share the timestamp with us whenever the issue happened.
 
1) ps -w | grep dhcre
 
Please note down a MAC address of any one system and please the collect below in both working and non working scenarios.
 
2)Wireshark Pcap from the affected system.
 
3) Please share the tcpdump, conntracks, and drop capture on XG | this needs to be collected on port 67 and port 68 , use 67 or 68 i syntax, please go through the below commands just for your reference. (Please open multiple putty sessions and collect the logs simultaneously).
 
 
In putty session 1:
 
#tcpdump -nei any host <dest IP> and port 67 or port 68
 
(Destination IP is which we are pinging)
 
 
In putty session 2:
 
Console> drop-packet-capture 'host <dest IP> and port 67 or port 68
 
Or
 
#drppkt host <dest IP> and port 67 or port 68
 
In putty session 3:
 
#mount -w -o remount /
#cish
Console> tcpdump verbose filedump count 10000 'host <dst IP> and port 67 or port 68 -s0
(At the time of issue and once we have 30-40 Packets, press Ctrl + C)
#exit
#cp /tmp/data/tcpdump.pcap /usr/share/userportal/tcpdump.pcap
Open Web Browser and enter:  x.x.x.x:4444/tcpdump.pcap [Replace with your Firewall IP and admin Port]
Go back to the Advanced Shell of the XG Firewall and then run the following command.
Note: It is important to run this command before closing the PuTTY session.
rm -rf /usr/share/userportal/tcpdump.pcap
mount -r -o remount /
 
In putty session 4:
 
#cd /log
#csc custom debug
#tail -f csc.log
 
 
In putty session 5:
 
#conntrack -L -d <dest IP>
 
In putty session 6:
 
#cd /log
#tail -f networkd.log
 
 
4)Please share the network topology along with IPSchema if possible
 
5)Please upload all logs on FTP server mentioning detailed time-stamps.
 
cd /
 
tar -cvzf tmp/AllXGLogss.tar.gz log/*
 
curl --insecure --ftp-ssl ftp://ftp.sophos.com:990 -u xxx:xxx -T '/tmp/AllXGLogss.tar.gz'