SophosUpdate not working through VPN

Hi,

As the title mentions, updates through a SSL VPN tunnel aren't working.  Seen from Sophos Central, some computers link to Sophos, others to the Update Server. but all mention issues in their logfiles...

The line that wonders me most, is "ERROR No network connectivity. Update cannot continue." …
It can be there is no connectivity to whatever it tries, but there is network available, as the VPN is open … also none of the logs on the Sophos firewall (packetfilter and http) mention anything about blocked content, so we have no clue what it is trying to do, or if the updater just can't find the route through the tunnel...
Any hints on what to look for?  Currently support didn't find more to say than "it must be the firewall" …

Is there an option to read the alc.log file when using Sophos Central?  Maybe this would make this more clear...

On my computer, today's attempts look like (and even then, Sophos Central currently states "Update Successful"... but the log is exactly the same as a on another computer which is seen as "Update failed")

2020-04-07T07:34:59.376Z [ 8860:15908] [v6.1.356.0] INFO  =========================
2020-04-07T07:34:59.376Z [ 8860:15908] [v6.1.356.0] INFO  SophosUpdate is starting.
2020-04-07T07:34:59.376Z [ 8860:15908] [v6.1.356.0] INFO  AutoUpdate version      : 6.1.356.0
2020-04-07T07:34:59.376Z [ 8860:15908] [v6.1.356.0] INFO  SophosUpdate version    : 6.1.356.0
2020-04-07T07:34:59.376Z [ 8860:15908] [v6.1.356.0] INFO  Build                   : 20190830114005-95a0922451e171e9dc54e46773bc3633f4b6b20b
2020-04-07T07:34:59.376Z [ 8860:15908] [v6.1.356.0] INFO  =========================
2020-04-07T07:34:59.376Z [ 8860:15908] [v6.1.356.0] INFO  Platform ID: WIN_10_X64 1909 18363.720
2020-04-07T07:34:59.377Z [ 8860:15908] [v6.1.356.0] INFO  Platform upgraded: 0
2020-04-07T07:34:59.377Z [ 8860:15908] [v6.1.356.0] INFO  Subscription: WindowsCloudNextGen RECOMMENDED 11
2020-04-07T07:34:59.377Z [ 8860:15908] [v6.1.356.0] INFO  Subscription: WindowsCloudClean RECOMMENDED 1
2020-04-07T07:34:59.377Z [ 8860:15908] [v6.1.356.0] INFO  Subscription: WindowsCloudAV RECOMMENDED 11
2020-04-07T07:34:59.377Z [ 8860:15908] [v6.1.356.0] INFO  Subscription: WindowsCloudHitmanProAlert RECOMMENDED 1
2020-04-07T07:34:59.377Z [ 8860:15908] [v6.1.356.0] INFO  Subscriptions changed: 0
2020-04-07T07:34:59.377Z [ 8860:15908] [v6.1.356.0] INFO  Features: APPCNTRL AV CLEAN CONNECT CORE DLP DVCCNTRL EFW HBT NTP SAV SDU WEBCNTRL XPD
2020-04-07T07:34:59.377Z [ 8860:15908] [v6.1.356.0] INFO  Features changed: 0
2020-04-07T07:34:59.380Z [ 8860:15908] [v6.1.356.0] INFO  Loading state file C:\ProgramData\Sophos\AutoUpdate\data\status\SophosUpdateStatus.xml
2020-04-07T07:34:59.538Z [ 8860:15908] [v6.1.356.0] ERROR No network connectivity. Update cannot continue.
2020-04-07T07:34:59.538Z [ 8860:15908] [v6.1.356.0] INFO  Telemetry::LoadTelemetrySupplement 215: Telemetry Interval set to 86400 seconds
2020-04-07T07:34:59.538Z [ 8860:15908] [v6.1.356.0] INFO  Telemetry::LoadDocument 202: C:\ProgramData\Sophos\AutoUpdate\\Config\TelemetryConfig.json loaded
2020-04-07T07:34:59.538Z [ 8860:15908] [v6.1.356.0] INFO  Telemetry::LoadTelemetrySupplement 256: Telemetry Interval updated to 86400 seconds
2020-04-07T07:34:59.538Z [ 8860:15908] [v6.1.356.0] INFO  Telemetry::CalculateLastTelemtryTime 145: Telemetry last ran at 2020-04-06 08:04:52, Offset 4201, Offset Time 2020-04-06 09:14:53
2020-04-07T07:34:59.538Z [ 8860:15908] [v6.1.356.0] INFO  Telemetry::HasTelemetrySchedulePeriodElapsed 164: Telemetry schedule has elapsed.
2020-04-07T07:34:59.538Z [ 8860:15908] [v6.1.356.0] INFO  Telemetry::SubmitTelemetry 278: Gathering Telemetry
2020-04-07T07:35:06.409Z [ 8860:15908] [v6.1.356.0] INFO  Overwriting state file C:\ProgramData\Sophos\AutoUpdate\data\status\SophosUpdateStatus.xml
2020-04-07T07:35:06.429Z [ 8860:15908] [v6.1.356.0] INFO  Verified state file can be loaded.
2020-04-07T07:35:06.430Z [ 8860:15908] [v6.1.356.0] INFO  SophosUpdate has completed with the result 2.
2020-04-07T07:35:06.430Z [ 8860:15908] [v6.1.356.0] INFO  SophosUpdate is exiting.

 

Thanks for any info,

Alain

  • Hi  

    Sophos Central Endpoint is unable to communicate to a central dashboard, would you please confirm if these domains and ports required are allowed? 

  • In reply to Shweta:

    Hi Shweta,

    As my collegue has spent many hours on the phone with support already, I do hope they checked this … As mentioned, we don't know how could this be a problem if we don't see problems in the logs on the firewall (it's Sophos as well), or someone should point out what might not be logged…

    I rather don't check the same several times, if no-one could explain what is tested… The general remark "No Network connectivity" is not true, so I'd like to know what the update tries at that moment... or why it can't be seen…

    Is there an option to force the update to start immediatelly (I have the idea that the trigger from Sophos Central doesn't do that, but if we can check this more frequently that once every hour, maybe Wireshark could come to the rescue)

    BR,

    Alain

  • In reply to Alain Cloet:

    Hi  

    We can force the update on the client and can run Wireshark on that client.

    Open the Sophos client on the endpoint and go to "about" and then client on update now which will trigger the update on the endpoint immediately.

  • In reply to Jasmin:

    Thanks, I tried this, it might help (but at the moment, I didn't see anything that seemed useful.

    What I did see, is that in the "Endpoint Self Help" for the Update Configuration it writes "No proxy used", although it shows one in C:\ProgramData\Sophos\AutoUpdate\Config\iconn.cfg.
    (I added the proxy in Sophos Central just yesterday, I hoped it would make a difference, but it seems it's not using this setting, or at least not at this point)

    A few weeks ago, I read that the update will use Sophos when it's not on the internal network.  How does it decide whether the computer is on the "internal network" (VPN isn't really internal, but not external either)

    BR,

    Alain

  • In reply to Alain Cloet:

    Hi  

    Thank you for the above note. As you said this endpoint is not using the proxy setting which you have applied on the Sophos Central as you might have installed the Sophos client on this machine before applying those changes in the Sophos Central. 

    When you add a proxy setting in the Sophos Central, it binds the proxy settings in the installer and then further installed client will use the proxy to download updates.

    Follow the below steps and see if they helps you redirect the endpoint to the proxy:

    1. Open a Command Prompt and run it as administrator.
    2. Type the following command:

      • For 32-bit: netsh winhttp import proxy source =ie then press the Enter button.
      • For 64-bit: cd C:\Windows\SysWOW64
        netsh winhttp import proxy source =ie then press the Enter button.
    3. Open Services and run it as administrator.
    4. Restart the following Sophos services:

      • Sophos AutoUpdate Service
      • Sophos MCS Agent
      • Sophos MCS Client
      • Sophos System Protection Service
  • In reply to Jasmin:

    Hi Jasmin,


    Unfortunatelly, this doesn't change... I modified Group Policy as well to make sure it was not caused by "bypassing local adresses"...


    Still no Proxy in the list (I had to restart computer, as the services can't be restarted, even when I open them with an admin account)

    But why doesn't it take it from the iconn-config?

    BR,

    Alain

  • In reply to Alain Cloet:

    Hi  

    It might be taking it from iconn-config file but when the next time policies will be synced with central, It will replace the edited file on the endpoint to make the complaint to the policies.

    As web proxy also doesn't change the result and result mentioned "No network connectivity", I suspect the traffic is not going out of the endpoint or VPN is blocking the network traffic from the machine.

    The Wireshark result can only help us here to drive us forward.

  • In reply to Jasmin:

    The iconn was set via Sophos Central, not manually...

    Originally (2 days ago) it looked like (users, passwords, server set to xxx):

    [PPI.WebConfig_Primary]
     AllowLocalConfig    =0
     AutoDialTimeout     =
     LocalPath           =
     DownloadGranularity =
    BandwidthLimit=1024
    UseHttps=0
    UserName=xxx
    UserPassword=xxx
    UseSophos=1
    UseDelta=1

    [PPI.ProxyConfig_Primary]
     AllowLocalConfig    =0

    ProxyType=0
    ProxyAddress=
    ProxyPortNumber=0
    ProxyUserName=
    ProxyUserPassword=

    Currently it is:

    [PPI.WebConfig_Primary]
     AllowLocalConfig    =0
     AutoDialTimeout     =
     LocalPath           =
     DownloadGranularity =
    BandwidthLimit=1024
    UseHttps=0
    UserName=xxx
    UserPassword=xxx
    UseSophos=1
    UseDelta=1

    [PPI.ProxyConfig_Primary]
     AllowLocalConfig    =0

    ProxyType=2
    ProxyAddress=xxx
    ProxyPortNumber=8080
    ProxyUserName=
    ProxyUserPassword=xxx

    I'll try to make a Wireshark trace this morning, but I will have to close as much as possible, to see what could be the relevant traffic...(and most likely, notice that there isn't traffic at all)

    To be continued…

    BR,

    Alain

  • In reply to Alain Cloet:

    :(

    Nothing to see via wireshark

    Nothing special to see via ProcMon

    Rules in Windows firewall are ok (and also nothing to see in those logfiles.. but they do see incomplete, but this might have to do with the fact that my collegue had to change settings for Sophos support)

  • In reply to Alain Cloet:

    Hi  

    Would you please PM me the case number you have already registered with Sophos Support? 

  • The failure to update is all about this line:

    ERROR No network connectivity. Update cannot continue.

    If you create a new DWORD named LogLevel value under HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Sophos\AutoUpdate\ and set it to 0, then start an update.

    You should see an extra line, e.g.

    DEBUG Environment::HasNetworkConnectivity Environment.cpp:757 Network connectivity: 66

    What number do you have:

    Also, in a PS command prompt, what does the following command return:

    [Activator]::CreateInstance([Type]::GetTypeFromCLSID(`DCB00C01-570F-4A9B-8D69-199FDBA5723B'))

    Regards,
    Jak

  • In reply to jak:

    Hi Jak,

    The result with the extra logging is:

    2020-04-10T06:46:35.342Z [10212: 7464] [v6.1.356.0] DEBUG Environment::HasNetworkConnectivity Environment.cpp:699 Network connectivity: 3

     

    The result of the Powershell test is:

    IsConnectedToInternet IsConnected
    --------------------- -----------
                    False        True

     

    So this might be clear why it's not working, although this test is not exactly accurate … this is why I Always say when people come with the info I can (or can't) ping a certain address, that it should (shouldn't) work …
    But I think I can continue with this … I keep you updated on the result, not sure whether this will be very fast, today I have a lot to do (although I would love to get this solved)

    BR,

    Alain

  • In reply to Alain Cloet:

    Thanks for the reply, OK, so 3 would suggest a connectivity state based on:

    https://docs.microsoft.com/en-us/windows/win32/api/netlistmgr/ne-netlistmgr-nlm_connectivity

    typedef enum NLM_CONNECTIVITY {
    NLM_CONNECTIVITY_DISCONNECTED = 0x0000,
    NLM_CONNECTIVITY_IPV4_NOTRAFFIC = 0x0001,
    NLM_CONNECTIVITY_IPV6_NOTRAFFIC = 0x0002,
    NLM_CONNECTIVITY_IPV4_SUBNET = 0x0010,       16dec
    NLM_CONNECTIVITY_IPV4_LOCALNETWORK = 0x0020, 32dec
    NLM_CONNECTIVITY_IPV4_INTERNET = 0x0040,     64dec
    NLM_CONNECTIVITY_IPV6_SUBNET = 0x0100,
    NLM_CONNECTIVITY_IPV6_LOCALNETWORK = 0x0200,
    NLM_CONNECTIVITY_IPV6_INTERNET = 0x0400
    } NLM_CONNECTIVITY;

    Given I have 66, then I'm looking at:
    NLM_CONNECTIVITY_IPV4_INTERNET and NLM_CONNECTIVITY_IPV6_NOTRAFFIC
    Which would be correct as I just have IPV4.

    For a return value of 3, this Windows API is returning the state to be:

    NLM_CONNECTIVITY_IPV4_NOTRAFFIC and NLM_CONNECTIVITY_IPV6_NOTRAFFIC

    This would suggest you have now IPv4 or IPv6 traffic and AutoUpate skips the update.

    I would assume that on a computer that is updating, both IsConnectedToInternet and IsConnected are true.

    Give me a moment and I might knock up an exe to help confirm/test this.

    Regards,
    Jak

  • In reply to jak:

    Do you have Visual Studio?  E.g. 2019 Community will be fine and is free.

    If so, you could create a new C# console project with the following code:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using NETWORKLIST;

    namespace NetTester
    {
    class Program
    {
    static void Main(string[] args)
    {
    var manager = new NetworkListManager();

    //Default to all networks.
    var connectedNetworks = manager.GetNetworks(NLM_ENUM_NETWORK.NLM_ENUM_NETWORK_ALL).Cast<INetwork>();

    //List just connected with any argument passed
    if (args.Length >= 1)
    {
    connectedNetworks = manager.GetNetworks(NLM_ENUM_NETWORK.NLM_ENUM_NETWORK_CONNECTED).Cast<INetwork>();
    }

    foreach (var network in connectedNetworks)
    {
    Console.WriteLine("Network ID: " + network.GetNetworkId());
    Console.WriteLine("Network name: " + network.GetName());
    Console.WriteLine("Description: " + network.GetDescription());
    Console.WriteLine("Domain Type: " + network.GetDomainType());
    Console.WriteLine("Connectivity: " + network.GetConnectivity());
    Console.WriteLine("IsConnected: " + network.IsConnected);
    Console.WriteLine("IsConnectedToInternet: " + network.IsConnectedToInternet);
    Console.WriteLine("GetCategory: " + network.GetCategory());
    Console.WriteLine("");
    }
    Console.ReadKey();
    }
    }
    }

    You need to reference in the solution the following:

    I would suggest targeting the highest version of .NET available, e.g. 4.7.2


    The output will be:

    C:\Users\user\source\repos\NetTester\NetTester\bin\Debug>nettester.exe 1
    Network ID: f407cd41-e852-4a7a-a1fe-94f2d9259c66
    Network name: jak
    Description: jak
    Domain Type: NLM_DOMAIN_TYPE_NON_DOMAIN_NETWORK
    Connectivity: 66
    IsConnected: True
    IsConnectedToInternet: True
    GetCategory: NLM_NETWORK_CATEGORY_PRIVATE

    Note: If you don't pass an argument it will list all networks not just connected ones.

     

  • In reply to jak:

    We couldn't check with Visual Studio now, but with Powershell we had also the same info.  The Guest-Network is the home network, the one in "our domain" is the VPN-tunnel

    Network Name: <our domain>
    Description: <our domain>
    Domain Type: NLM_DOMAIN_TYPE_DOMAIN_AUTHENTICATED
    Connectivity: 3
    IsConnected: False
    IsConnectedToInternet: False
    GetCategory: NLM_NETWORK_CATEGORY_DOMAIN_AUTHENTICATED

    Network Name: xxxGuest
    Description: xxxGuest
    Domain Type: NLM_DOMAIN_TYPE_NON_DOMAIN_NETWORK
    Connectivity: 66
    IsConnected: True
    IsConnectedToInternet: True
    GetCategory: NLM_NETWORK_CATEGORY_PRIVATE