XG / XGS ongoing issues list

Let me rephrase this: 

1. This should be fixed by now, based on the PPPoE Protocol. There are certainly some scenarios, which could still lead to this problem (if the ISP is doing something odd). 

2. You can simply save the interface without change - This will reload the interface and essentially doing the same like "Turn off/on" on UTM. 

3. Mail relay is a question, what they do. SMTP Auth is not implemented on SFOS. This means: you cannot do a smtp auth against the firewall. You could do a host based relay. SFOS uses Exim like UTM does. BTW: SMTP Auth is a poor design decision IMO - You do not have a summarized Email summary / logs. I would always recommend to send all Emails first to the Email server. 

That is good news the PPPoE user interfaces now have a automatic renew on link failure, but what about the non PPPoE users who have a standard DHCP addressing function from their ISP/RSPs that is still broken. Using link manager to indicate a link has failed only sends notifications does not cause the address renew request to be sent.

Ian

Hi rfcat_vkDHCP client (XG) will not send new IP requests until its lease time is over. Making the modem ON/OFF or plugging/unplugging the physical link will not trigger any renewal events. This is how the DHCP client works in SFOS as per DHCP RFC. On interface update event we forcefully restart the DHCP client which will generate the request again.

Thank you for the explanation, but that does not work in real life. The time to live is an artificial XG parameter, the time to live used by the ISP/RSP is way shorter eg if you disconnect and reconnect a short time eg less than a minute you will usually be assigned a new IP address. So maybe it is time for Sophos to move away from the theoretical document to a real world process and reduce the stress on your customers?

Ian

Vishal, the issue is that the WAN link manager can detect the WAN service is down, but then does nothing about it to try and recover it.

We have seen this very issue with NBN services in Australia delivered via DHCP'd IP. The fault occurs when there is an outage where the XG does not see a physical interface down (as it would in some sense with PPP). The service comes back, but no traffic till a new DHCP request goes out, the WAN link manager knows the link is down, but nothing happens till the timeout.

On one hand the argument is you should have dual WAN, but that's not always going to be the case, and additionally your second WAN may be of a lower service as it's primarily a backup, it could be 4G/mobile data which also could be costly. So it really should do more to restore the service as fast as possible to return to both reduce costs, but also in many cases bring performance level back to normal thus reducing end user impact.

XG's in HA can also exacerbate the problem, as they will never see a physical interface go down in the normal sense, because there will often be something else in between to provide the WAN to both XG's. In many setups we have two XG's in HA, going into two switches (stacked or otherwise), so the WAN is delivered either as a VLAN to the XG, or a port on the switch, so they are in a sense abstracted away from the ISP's equipment and thus again never see a physical drop.

Now the other part of the problem is that we don't see this with other vendors. One client had both XG and Sonicwall, and when we saw the same outage knock all the XG's out (funnily enough were all on the same NBN POI so they had the same "exchange" so to speak and all saw the same fault where the fibre link didn't physically drop), the Sonicwalls recovered within minutes, the XG's needed a reboot or someone to go in and click save on the interface to issue a new DHCP. So if Sonicwall can manage to do this, why can't XG?

And to be honest using the statement around "oh well we follow the RFC" isn't really a good response, I've had that kind of mentality come from a few vendors and it's beyond frustrating when it's just not how the real world works. Or worse and sadly often the case, that while it is true they are within spec of an RFC, they could fix the problem by implementing the optional or extended parts of the spec. For example the way the Sonicwall seemed to get around it (at the time from what we saw anyway) was sending regular/early renews, which is not invalid based on what I can see in the RFC?

"A client MAY choose to renew or extend its lease prior to T1. The server may choose not to extend the lease (as a policy decision by the network administrator), but should return a DHCPACK message regardless."

https://www.rfc-editor.org/rfc/rfc2131

Though to be perfectly honest none of this stuff about the spec really matters, because the WAN link manager knows the service is down, so why not use that to trigger the WAN interface that is "down" to just do a release/renew? Just a tick box or something to enable it to trigger it? Do something to try and recover that link.

But there may be some light at the end of the tunnel, we have been chasing this with Sophos for some time now, and there is a feature request to try and resolve this under the following SFSW-I-1139. So I would suggest anyone seeing this DHCP issue, grab a list of the affected clients, XG serial numbers and chase your account manager, because as we were told "the best way for this FR to make noise is to align this up to as many customers as possible"

And finally, don't get me wrong about Sophos/XG firewalls, I like the XG product, and no firewall vendor is perfect, I've got gripes with all firewall vendors. But really do like the XG/XGS product, and use it daily at many different sites.

Vishal, the issue is that the WAN link manager can detect the WAN service is down, but then does nothing about it to try and recover it.

We have seen this very issue with NBN services in Australia delivered via DHCP'd IP. The fault occurs when there is an outage where the XG does not see a physical interface down (as it would in some sense with PPP). The service comes back, but no traffic till a new DHCP request goes out, the WAN link manager knows the link is down, but nothing happens till the timeout.

On one hand the argument is you should have dual WAN, but that's not always going to be the case, and additionally your second WAN may be of a lower service as it's primarily a backup, it could be 4G/mobile data which also could be costly. So it really should do more to restore the service as fast as possible to return to both reduce costs, but also in many cases bring performance level back to normal thus reducing end user impact.

XG's in HA can also exacerbate the problem, as they will never see a physical interface go down in the normal sense, because there will often be something else in between to provide the WAN to both XG's. In many setups we have two XG's in HA, going into two switches (stacked or otherwise), so the WAN is delivered either as a VLAN to the XG, or a port on the switch, so they are in a sense abstracted away from the ISP's equipment and thus again never see a physical drop.

Now the other part of the problem is that we don't see this with other vendors. One client had both XG and Sonicwall, and when we saw the same outage knock all the XG's out (funnily enough were all on the same NBN POI so they had the same "exchange" so to speak and all saw the same fault where the fibre link didn't physically drop), the Sonicwalls recovered within minutes, the XG's needed a reboot or someone to go in and click save on the interface to issue a new DHCP. So if Sonicwall can manage to do this, why can't XG?

And to be honest using the statement around "oh well we follow the RFC" isn't really a good response, I've had that kind of mentality come from a few vendors and it's beyond frustrating when it's just not how the real world works. Or worse and sadly often the case, that while it is true they are within spec of an RFC, they could fix the problem by implementing the optional or extended parts of the spec. For example the way the Sonicwall seemed to get around it (at the time from what we saw anyway) was sending regular/early renews, which is not invalid based on what I can see in the RFC?

"A client MAY choose to renew or extend its lease prior to T1. The server may choose not to extend the lease (as a policy decision by the network administrator), but should return a DHCPACK message regardless."

https://www.rfc-editor.org/rfc/rfc2131

Though to be perfectly honest none of this stuff about the spec really matters, because the WAN link manager knows the service is down, so why not use that to trigger the WAN interface that is "down" to just do a release/renew? Just a tick box or something to enable it to trigger it? Do something to try and recover that link.

But there may be some light at the end of the tunnel, we have been chasing this with Sophos for some time now, and there is a feature request to try and resolve this under the following SFSW-I-1139. So I would suggest anyone seeing this DHCP issue, grab a list of the affected clients, XG serial numbers and chase your account manager, because as we were told "the best way for this FR to make noise is to align this up to as many customers as possible"

And finally, don't get me wrong about Sophos/XG firewalls, I like the XG product, and no firewall vendor is perfect, I've got gripes with all firewall vendors. But really do like the XG/XGS product, and use it daily at many different sites.

Th question that then follows is, what is the value of the WAN link manager?

Ian

WAN Link manager is to load balancing. So the WAN Link manager is used for "Default Gateway behavior". If the firewall needs a Default GW, WAN link manager will give you a feedback, which Link to use. 

Essentially you are discussing a missing feature, which will pick up a renewal process of the IP based on the WAN Link Manager. And that is tracked to be discussed. 

It is based on the technology. PPPoE Interfaces will do this (based on the used technology). But it seems like, there are other ISPs, only giving you a IP based on ethernet protocol (which is unusual for my field - ISPs are not doing this often times here - I cannot speak for other regions).