Gateway 1 - Main ISP keeps getting disconnected, unable to find logs

Hello Sophos User1499,

Would it be possible for you to share the Case ID. 

How are you checking that when the internet goes down on #ISP1 you still have internet connectivity? Have you tried to connect the WAN port that goes to the ISP#1 to a computer when the issue is happening and run a Ping test?

I take you have changed the Failover rule to ping an IP address beyond the IP of the ISP router?

Regards,

@emmosophos,

I can confirm that the Internet is working on ISP#1.

I have taken out Sophos Firewall and the internet is working directly from the Mikrotik Router which is the ISP#1.

Also, from the 6 disconnects yesterday, I found out that there is a 20minute window that Sophos keeps showing that ISP#1 is down from GUI and CLI but when I checked the Public IP on all my devices, it is still getting ISP#1 IP Address. But most of the time, when ISP#1 is showing as down, it will failover to ISP#2, leaving my webservers on ISP#1 down.

I have told all this to Sophos Escalation Engineer but he keeps insisting that it is an ISP issue which clearly it is NOT an ISP issue.

I just want to know where to read more meaningful logs to understand why Sophos Firewall keeps disconnecting ISP#1.

Case#: 9927707

Hi,

what are the settings in the Mikrotik DHCP server?

Ian

Hi ian

The Mikrotik is not setup as a DHCP server. The issue only started last Thursday and the DUAL WAN has been setup for more than a year now.

I have 5 Static WAN IP Address setup on the Mikrotik Router, All WAN IP is working from the Mikrotik Router to all other devices except for the Sophos Firewall.

The WAN IP Address assigned to the Sophos Firewall is confirmed working on the Mikrotik Router when Sophos Firewall is showing as disconnected.

The DUAL WAN failover is previously setup to ping 8.8.8.8 and I have added to ping the ISP#1 DNS Address today but I still get the 20minutes disconnect after several hours.

I am hoping to find any meaningful logs so I can properly start to troubleshoot the issue as I have already physically isolated the issue to be on the Sophos side.

However, I have yet to find anything at all.

Thank you for your reply. I really appreciate it.

Hi,

I gave up on 8.8.8.8 as test site being not reliable. Also the XG does not automatically reconnect if it senses a link failure, this in an ongoing issue.

My experience.

ian

ian 

Thanks again for your reply.

I have already removed 8.8.8.8 and changed it to the ISP DNS Address but after 3 hours, the disconnect happens again.

I am stumped as to where could I possibly find any other logs in the Sophos Firewall.

In dgd.log, it only shows "Failed, Retrying(4) Ping : 123.100.x.x"

Thank you.

ian 

Thanks again for your reply.

I have already removed 8.8.8.8 and changed it to the ISP DNS Address but after 3 hours, the disconnect happens again.

I am stumped as to where could I possibly find any other logs in the Sophos Firewall.

In dgd.log, it only shows "Failed, Retrying(4) Ping : 123.100.x.x"

Thank you.

Hi,

Next trick to try is edit and save the offending interface without changing anything. See how long that stays up for.

Also what are the CPU.memory and load like?

Ian

rfcat_vk 

Disk Usage is 22% Used Only

CPU Usage IDLE is at 95.53%

Thanks for the mention of the trick, I will try this and see how long it stays up.

rfcat_vk 

I did the edit and save without any changes on both ISP1 and failover settings.

ISP1 just went down again just now.

Hi,

that CPU usage is that about 5% free?

What does TOP how as to what is hogging CPU?

Ian

rfcat_vk 

I meant, CPU IDLE 95%, so only 5% is currently used over this weekend. 

I generated a 48-hour report. So even with only 5% use on the CPU, I still experienced several disconnects.

Without proper logs in Sophos Firewall, I am troubleshooting based on isolation only.

I can confirm that when Sophos is showing ISP1 as disconnected, I can still ping the Sophos Public IP from outside network.

I have now used a different PORT interface in the Sophos Firewall for ISP1 and recreated all Firewall Policies pointing to the new port interface for ISP1.

However, I still experience the disconnect.

Hi,

the issue will be with the ping and XG not seeing the ping response and as a result it thinks the link is down when in reality there is nothing wrong with the link.

I haven't configured a fail over even though it is almost mandatory and you have to convince it not to activate on failure.

Ian

rfcat_vk - you are right. the issue is now with Ping and XG, as the link is actually NOT down. I really appreciate you staying with me on this one.

It is going to be something stupid like a watch timer failing to handle two different requests for the service correctly and the supervisor timer cutting in after a delay.

But that will be up to the Devs to identify and fix, by the way this is not the first time this issue has been raised.

ian

rfcat_vk 

It is going to be another long night for me. I deleted the original Port interface of ISP1 and it is now on a completely different port interface.

All Firewall rules related to ISP1 has been deleted and recreated. 

I am now on a waiting game, the last disconnect is at 2:00pm today so I will wait until 2:00am.

Hopefully, everything goes well and no more disconnect. Though if it happens again, I will completely remove the WAN Failover to see what will happen next.

Thanks

OKay, so despite all the changes, I still got the ISP1 disconnect after 8 hours.

I have now removed DUAL WAN Failover and will have to wait what happens next.