SophosUpdate not working through VPN

The failure to update is all about this line:

ERROR No network connectivity. Update cannot continue.

If you create a new DWORD named LogLevel value under HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Sophos\AutoUpdate\ and set it to 0, then start an update.

You should see an extra line, e.g.

DEBUG Environment::HasNetworkConnectivity Environment.cpp:757 Network connectivity: 66

What number do you have:

Also, in a PS command prompt, what does the following command return:

[Activator]::CreateInstance([Type]::GetTypeFromCLSID(`DCB00C01-570F-4A9B-8D69-199FDBA5723B'))

Regards,
Jak

Hi Jak,

The result with the extra logging is:

2020-04-10T06:46:35.342Z [10212: 7464] [v6.1.356.0] DEBUG Environment::HasNetworkConnectivity Environment.cpp:699 Network connectivity: 3

The result of the Powershell test is:

IsConnectedToInternet IsConnected
--------------------- -----------
                False        True

So this might be clear why it's not working, although this test is not exactly accurate … this is why I Always say when people come with the info I can (or can't) ping a certain address, that it should (shouldn't) work …
But I think I can continue with this … I keep you updated on the result, not sure whether this will be very fast, today I have a lot to do (although I would love to get this solved)

BR,

Alain

Thanks for the reply, OK, so 3 would suggest a connectivity state based on:

https://docs.microsoft.com/en-us/windows/win32/api/netlistmgr/ne-netlistmgr-nlm_connectivity

typedef enum NLM_CONNECTIVITY {
NLM_CONNECTIVITY_DISCONNECTED = 0x0000,
NLM_CONNECTIVITY_IPV4_NOTRAFFIC = 0x0001,
NLM_CONNECTIVITY_IPV6_NOTRAFFIC = 0x0002,
NLM_CONNECTIVITY_IPV4_SUBNET = 0x0010,       16dec
NLM_CONNECTIVITY_IPV4_LOCALNETWORK = 0x0020, 32dec
NLM_CONNECTIVITY_IPV4_INTERNET = 0x0040,     64dec
NLM_CONNECTIVITY_IPV6_SUBNET = 0x0100,
NLM_CONNECTIVITY_IPV6_LOCALNETWORK = 0x0200,
NLM_CONNECTIVITY_IPV6_INTERNET = 0x0400
} NLM_CONNECTIVITY;

Given I have 66, then I'm looking at:
NLM_CONNECTIVITY_IPV4_INTERNET and NLM_CONNECTIVITY_IPV6_NOTRAFFIC
Which would be correct as I just have IPV4.

For a return value of 3, this Windows API is returning the state to be:

NLM_CONNECTIVITY_IPV4_NOTRAFFIC and NLM_CONNECTIVITY_IPV6_NOTRAFFIC

This would suggest you have now IPv4 or IPv6 traffic and AutoUpate skips the update.

I would assume that on a computer that is updating, both IsConnectedToInternet and IsConnected are true.

Give me a moment and I might knock up an exe to help confirm/test this.

Regards,
Jak

Do you have Visual Studio?  E.g. 2019 Community will be fine and is free.

If so, you could create a new C# console project with the following code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using NETWORKLIST;

namespace NetTester
{
class Program
{
static void Main(string[] args)
{
var manager = new NetworkListManager();

//Default to all networks.
var connectedNetworks = manager.GetNetworks(NLM_ENUM_NETWORK.NLM_ENUM_NETWORK_ALL).Cast<INetwork>();

//List just connected with any argument passed
if (args.Length >= 1)
{
connectedNetworks = manager.GetNetworks(NLM_ENUM_NETWORK.NLM_ENUM_NETWORK_CONNECTED).Cast<INetwork>();
}

foreach (var network in connectedNetworks)
{
Console.WriteLine("Network ID: " + network.GetNetworkId());
Console.WriteLine("Network name: " + network.GetName());
Console.WriteLine("Description: " + network.GetDescription());
Console.WriteLine("Domain Type: " + network.GetDomainType());
Console.WriteLine("Connectivity: " + network.GetConnectivity());
Console.WriteLine("IsConnected: " + network.IsConnected);
Console.WriteLine("IsConnectedToInternet: " + network.IsConnectedToInternet);
Console.WriteLine("GetCategory: " + network.GetCategory());
Console.WriteLine("");
}
Console.ReadKey();
}
}
}

You need to reference in the solution the following:

I would suggest targeting the highest version of .NET available, e.g. 4.7.2

The output will be:

C:\Users\user\source\repos\NetTester\NetTester\bin\Debug>nettester.exe 1
Network ID: f407cd41-e852-4a7a-a1fe-94f2d9259c66
Network name: jak
Description: jak
Domain Type: NLM_DOMAIN_TYPE_NON_DOMAIN_NETWORK
Connectivity: 66
IsConnected: True
IsConnectedToInternet: True
GetCategory: NLM_NETWORK_CATEGORY_PRIVATE

Note: If you don't pass an argument it will list all networks not just connected ones.

We couldn't check with Visual Studio now, but with Powershell we had also the same info.  The Guest-Network is the home network, the one in "our domain" is the VPN-tunnel

Network Name: <our domain>
Description: <our domain>
Domain Type: NLM_DOMAIN_TYPE_DOMAIN_AUTHENTICATED
Connectivity: 3
IsConnected: False
IsConnectedToInternet: False
GetCategory: NLM_NETWORK_CATEGORY_DOMAIN_AUTHENTICATED

Network Name: xxxGuest
Description: xxxGuest
Domain Type: NLM_DOMAIN_TYPE_NON_DOMAIN_NETWORK
Connectivity: 66
IsConnected: True
IsConnectedToInternet: True
GetCategory: NLM_NETWORK_CATEGORY_PRIVATE

Hmm.. From some testing, it seems we have code that behaves like the following:

#include <Windows.h>
#include <iostream>
#include <atlbase.h>
#include <netlistmgr.h>

#pragma comment(lib, "ole32.lib")

bool Connected();

int main()
{
    if (SUCCEEDED(CoInitializeEx(NULL, COINIT_MULTITHREADED))){
        if (Connected()){
            std::cout << "We are connected." << std::endl;
            return 0;
        }
        else {
            std::cout << "We are not connected." << std::endl;
            return 1;
        }
    }
    else{
        std::cout << "CoInitializeEx failed, odd!" << std::endl;
    }
}

bool Connected(){
    NLM_CONNECTIVITY connectivity = NLM_CONNECTIVITY_DISCONNECTED;
    CComPtr<INetworkListManager> nlm;
    if (FAILED(nlm.CoCreateInstance(CLSID_NetworkListManager)) ||
        FAILED(nlm->GetConnectivity(&connectivity))){
        // pre-Vista, no NLM?
        return true;
    }

    std::cout << "Network connectivity: " << connectivity << std::endl;

    if (connectivity == NLM_CONNECTIVITY_DISCONNECTED){
        return false;
    }

    if ((0 != (connectivity & NLM_CONNECTIVITY_IPV4_NOTRAFFIC)) && (0 != (connectivity & NLM_CONNECTIVITY_IPV6_NOTRAFFIC))){
        return false;
    }

    return true;
}

I've also attached a file which prints out the details on a per network basis as well which might be insighful.

You could compile this in VS with a C++ console/empty project and should give the same results as AutoUpdate to simplify this test.

I suppose to more closely emulate what's going on you'd need to run the exe as the System user in a non-interactive session.  PsExec can do this with the -s switch.  I don't think the network list would be different for local system vs the logged on user though?

Regarding Powershell, something like the following this evaluates each network returned from GetNetworks which what I guess you did:

$NLM_ENUM_NETWORK_ALL = 3
$NetworkListManager = [Activator]::CreateInstance([Type]::GetTypeFromCLSID("DCB00C01-570F-4A9B-8D69-199FDBA5723B"))
$Networks = $NetworkListManager.GetNetworks($NLM_ENUM_NETWORK_ALL)
foreach($network in $Networks){
    write-host "Network name: $($network.GetName())"
    write-host "Description: $($network.GetDescription())"
    write-host "Domain Type: $($network.GetDomainType())"
    write-host "Connectivity: $($network.GetConnectivity())"
    write-host "IsConnected: $($network.IsConnected)"
    write-host "IsConnectedToInternet: $($network.IsConnectedToInternet)"
    write-host "GetCategory: $($network.GetCategory())"
    write-host "================================================"
}

Where:

#typedef enum NLM_ENUM_NETWORK {
# NLM_ENUM_NETWORK_CONNECTED,
# NLM_ENUM_NETWORK_DISCONNECTED,
# NLM_ENUM_NETWORK_ALL
#} ;

As might the cmdlet Get-NetConnectionProfile. 

What I find interesting is that your script reports 2 networks, one with connectivity 66 and one with 3.  We see in the AutoUpdate trace log, the connectivity result of 3. Where:

NLM_CONNECTIVITY_IPV4_NOTRAFFIC | NLM_CONNECTIVITY_IPV6_NOTRAFFIC = 3

NLM_CONNECTIVITY_IPV6_NOTRAFFIC | NLM_CONNECTIVITY_IPV4_INTERNET = 66

I would have thought that if the code calls GetConnectivity, the documentation says:

The GetConnectivity method returns the overall connectivity state of the machine.

Which seems at odds with one of the networks having the state of 66, unless that network is filtered out somehow?

I guess you could try and "break" the NetworkListManager COM registration, but to do that just for SophosUpdate.exe would require a shim created from the Application Compatibility Toolkit targeted as SophosUpdate.exe to prevent netprofm.dll from loading or the read to the CLSID reg key failing by using VirtualRegistry.   Also I see Endpoint Defense prevents shims being inserted for Sophos processes so awkward anyway to deploy.

Maybe there is something under: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\NetworkList to influence this call. 

Trying to influence the connectivity returned results of the above C++ code is probably the best I can offer.

Regards,
Jak

0181.test.txt

Thanks for the effort, Jak

It's clear now that the SophosUpdater does not exactly work as expected, but that there is nothing we can do about it...

I suppose the best thing is to provide a link to this forum thread in a Support ticket and ask for it to be escalated to Development.  It seems like a code change would be required to be more tolerant of such a state. 

That said, it seems like a common scenario so I would have thought more people would have encountered it.  Feels like some sort of config change should allow it to work and I still don't understand why what appears to be a network non-specific global state returned by GetConnectivity doesn't have your computer down as connected.

If I have any further thoughts I'll update the post.

Regards,

Jak

Hi Alain and Jak,

Thank you for all the information on this issue here.

I have already mentioned the community link into the case. I have received the logs on the case with the auto-update debug logs.

As I mentioned to you Alain, I'll proceed further to get the case escalated to GES.

Hi Alain Cloet 

The case for this issue has been closed and found that there was a default gateway missing on the SSL VPN adaptor through which traffic was passing and which was causing the issue.

You colleague has confirmed that he has run the command and was successfully able to connect it. The power shell script is available in the below thread.

https://community.sophos.com/products/xg-firewall/f/vpn/116909/problem-with-ssl-vpn-client-and-o365-in-windows-10

Hi Jasmin,

Although there is a workaround, I wouldn't consider this a solution.  The error is caused by Microsoft, but the Sophos Updater is using the wrong assumption that this data is correct, although it's proven to be wrong.

Maybe there isn't anything to add on the ticket, I wanted to clearify this view...

BR,

Alain

Hi Alain Cloet 

I understand your concern and the same was conveyed to your colleague on the ticket that this is a workaround and The global escalation engineer will pass on the issue of SSL VPN adaptor not providing default gateway to look from their side.

Hi Alain Cloet 

I understand your concern and the same was conveyed to your colleague on the ticket that this is a workaround and The global escalation engineer will pass on the issue of SSL VPN adaptor not providing default gateway to look from their side.