Problems with updating services, particularly MCS, since approx end May

I already have a ticket open with Sophos - I'm just interested in seeing if other's have experienced this. 

Since approximately end of May, we're running into issues with the MCS Agent and Client and File Scanner services failing to update and requiring regular troubleshooting. Zap is required to get the best results, but the issues recur within a few days.

Sophos solutions thus far are deletion of cache, lots of restarts and Zap - but this isn't sustainable really. 

It's not all endpoints, but is a substantial number of servers and desktops and is consuming a lot of IT time to try resolve. 

Anyone else experiencing similar? 

Parents
  • When you mention Sophos MCS and Sophos File Scanner failing to update, do you have some install logs from \windows\temp\ for this?  Can you share them?

  • Actually now you mention it, that directory isn't one Sophos have asked to look at, and a glance in there seems to show, for example, the SFS failing because the service is already marked for deletion - it can't handle that. 

    This then repeats - it's like it's deleting it, but then not realising and trying to delete again, getting an error and then getting in a mess as can't rollback a deleted service. 

    From one log file:

    I Removing file (C:\Program Files\Sophos\Sophos File Scanner\SBK1622700636-1) => rollback backup (C:\Windows\TEMP\ef9c21a0b89728a395c4d5054c104886427c8d423ce823f336042a71063621c8.tmp)
    2021-06-10T10:13:23.001Z [54776:27868] A Executing step: SetRegistryValue(HKLM\SYSTEM\CurrentControlSet\services\Sophos Endpoint Defense\TamperProtection\Components\SFS, 64, Enable, 0)
    2021-06-10T10:13:23.004Z [54776:27868] A Executing step: Stop service step: Sophos File Scanner Service
    2021-06-10T10:13:23.005Z [54776:27868] I Service already stopped.
    2021-06-10T10:13:23.005Z [54776:27868] A Executing step: Delete service step: Sophos File Scanner Service
    2021-06-10T10:13:23.007Z [54776:27868] W DeleteService 'Sophos File Scanner Service' failed with error 1072: The specified service has been marked for deletion.

    2021-06-10T10:13:23.007Z [54776:27868] E Failed step: Delete service step: Sophos File Scanner Service, rolling back previous steps
    2021-06-10T10:13:23.008Z [54776:27868] A Rolling back step: Stop service step: Sophos File Scanner Service
    2021-06-10T10:13:23.008Z [54776:27868] W StartService failed with error 1058: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

    And another:

    2021-06-10T10:30:58.708Z [39096:29048] I Removing file (C:\Program Files\Sophos\Sophos File Scanner\SBK1622700636-1) => rollback backup (C:\Windows\TEMP\630e898d41b6963c6f69864e9420a0580df7d39f56a48d7abd31db32900e819e.tmp)
    2021-06-10T10:30:58.710Z [39096:29048] A Executing step: SetRegistryValue(HKLM\SYSTEM\CurrentControlSet\services\Sophos Endpoint Defense\TamperProtection\Components\SFS, 64, Enable, 0)
    2021-06-10T10:30:58.713Z [39096:29048] A Executing step: Stop service step: Sophos File Scanner Service
    2021-06-10T10:30:58.714Z [39096:29048] I Service already stopped.
    2021-06-10T10:30:58.714Z [39096:29048] A Executing step: Delete service step: Sophos File Scanner Service
    2021-06-10T10:30:58.715Z [39096:29048] W DeleteService 'Sophos File Scanner Service' failed with error 1072: The specified service has been marked for deletion.

    2021-06-10T10:30:58.716Z [39096:29048] E Failed step: Delete service step: Sophos File Scanner Service, rolling back previous steps
    2021-06-10T10:30:58.716Z [39096:29048] A Rolling back step: Stop service step: Sophos File Scanner Service
    2021-06-10T10:30:58.717Z [39096:29048] W StartService failed with error 1058: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

  • I suspect as a "marker" for the issue, under the service key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos File Scanner Service

    There will be a DeleteFlag DWORD set to 1.

    This is the way Windows, specially the Service Control Manager (services,exe) cleans up services at boot which can't be removed when the SCM is asked to remove them.  e.g. SC.exe delete <ServiceName>.

    At the point the SFS installer is removing the service, there is a good chance that another process still has a handle open to the service.  As a result, the service cannot be totally be removed and simply marked as deleted. When the installer tries to put it back it will fail with 1058 until the computer is restarted I suspect.

    I would suggest, get the machine into the state where the service is marked as deleted.  Then stop third party services one at a time that are most likely be interested in enumerating services.

    What should happen is, once the process exists, the handle will be removed and the SFS service will then be "fully" deleted from the SCM database. The service key will be removed so you can check in Regedit.exe for that, refreshing as you go under the key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos File Scanner Service

    If you can determine the process or processes (pretty unlikely to be multiple) that have left an open handle you'll will find the cause. It is likely to be a bug in the process keeping the handle open when it should be closed.

    Regards

Reply
  • I suspect as a "marker" for the issue, under the service key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos File Scanner Service

    There will be a DeleteFlag DWORD set to 1.

    This is the way Windows, specially the Service Control Manager (services,exe) cleans up services at boot which can't be removed when the SCM is asked to remove them.  e.g. SC.exe delete <ServiceName>.

    At the point the SFS installer is removing the service, there is a good chance that another process still has a handle open to the service.  As a result, the service cannot be totally be removed and simply marked as deleted. When the installer tries to put it back it will fail with 1058 until the computer is restarted I suspect.

    I would suggest, get the machine into the state where the service is marked as deleted.  Then stop third party services one at a time that are most likely be interested in enumerating services.

    What should happen is, once the process exists, the handle will be removed and the SFS service will then be "fully" deleted from the SCM database. The service key will be removed so you can check in Regedit.exe for that, refreshing as you go under the key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos File Scanner Service

    If you can determine the process or processes (pretty unlikely to be multiple) that have left an open handle you'll will find the cause. It is likely to be a bug in the process keeping the handle open when it should be closed.

    Regards

Children