This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Device Encryption Service randomly not starting/stopping on multiple endpoints since last week's outages?

Hello all.

Since last week's outage debacle, I've seen multiple random endpoints suddenly report that the device encryption service is not starting.

There seems to be no rhyme or reason to the timing (not when starting up, after restart, etc.).  Seems possibly related to policy push issues.

Sophos support asked me to remove policies from affected devices, remove endpoints, reinstall endpoints, reapply policies. I have not opted to do this as it is not a viable solution and really wouldn't not solve underlying issues with the central cloud services not pushing out policies in the first place.

Generally I've used PSEXEC to remotely start the service and the affected clients don't seem to be popping back up again after that, but still it's getting annoying.

Have any of you encountered this as of late? Any particular data points/extrapolation you've found (patterns like time of day, etc.)?

Lastly, is this all going to be a continuing issue with Sophos. I am in charge of maintaining Sophos on multiple endpoints, and trying to deploy policies, reinstall Cloud Web Gateway...I thought this product was designed to assist with reducing management loads for endpoints, not increase them?



This thread was automatically locked due to age.
Parents
  • I've noticed this issue on quite a lot of our machines here, although it's never really been too much of a concern as I just remotely restart the service when an alert comes up.  Obviously not much use for those with thousands of machines though!  I did, however, notice an improvement in the issue after looking into a separate problem with the Windows Audio service not starting.  The fix for this seems to have helped with the Sophos Device Encryption service too.

    Basically, I set a new registry entry at HKLM\SYSTEM\CurrentControlSet\Control

    New DWORD key ServicesPipeTimeout with a decimal value of 60000

    Source for this at http://support.microsoft.com/kb/922918

    Obviously it depends on the underlying issue so may not sort it for everyone, but it might be worth trying on a couple of test machines.

  • How well did this fix work for you?  I've tried on a few of my machines with varying success.

  • It's been pretty much the same here.  It looks like a lot of the original cases were due to the service timing out, so increasing the timeout has 'fixed' the problem, but there are still a few machines where the service is still failing to start.  I've not tried extending the timeout even further, but if it's taking longer than 1 minute to start I think there's an extra problem.

    Confusing, as the timeout was clearly an issue for some machines, but not all of them.

  • Great.  I'm not sure what's going on in my case.  Now in addition to Windows Audio service not starting on random machines, the wifi stops working and restarting the computer asks for a BitLocker key each time.  One machine I can understand but on multiple is a head-scratcher.

  • RBGE said:

    It's been pretty much the same here.  It looks like a lot of the original cases were due to the service timing out, so increasing the timeout has 'fixed' the problem, but there are still a few machines where the service is still failing to start.  I've not tried extending the timeout even further, but if it's taking longer than 1 minute to start I think there's an extra problem.

    Confusing, as the timeout was clearly an issue for some machines, but not all of them.

     

    I'm curious if you're still having issues with services not starting.  I've had the issue now on almost 10 machines where a lot of services are randomly not starting.  The ones I've noticed are Windows Audio, WLAN, Print Spooler, and multiple Sophos services.  I'm not sure if these issues are related to Sophos or if they're all symptoms of something else.  There aren't any recently installed Windows Updates so........ I'm pulling my hair out over here.  lol

  • James Aggrey said:

     

     
    RBGE

    It's been pretty much the same here.  It looks like a lot of the original cases were due to the service timing out, so increasing the timeout has 'fixed' the problem, but there are still a few machines where the service is still failing to start.  I've not tried extending the timeout even further, but if it's taking longer than 1 minute to start I think there's an extra problem.

    Confusing, as the timeout was clearly an issue for some machines, but not all of them.

     

     

     

    I'm curious if you're still having issues with services not starting.  I've had the issue now on almost 10 machines where a lot of services are randomly not starting.  The ones I've noticed are Windows Audio, WLAN, Print Spooler, and multiple Sophos services.  I'm not sure if these issues are related to Sophos or if they're all symptoms of something else.  There aren't any recently installed Windows Updates so........ I'm pulling my hair out over here.  lol

     

     

    An update: I uninstalled Sophos on that laptop and all the Windows services started working normally after a reboot.  Friggin Sophos...... now I'm gonna try restarting to see if all the problems come back.

  • Hi Everyone,

    The rollout of Central Device Encryption 1.4.103 for Windows is now complete.

    Regards,

    Gowtham Mani
    Community Support Engineer | Sophos Technical Support

    Knowledge Base  |  @SophosSupport | Sign up for SMS Alerts
    If a post solves your question use the 'This helped me' link.

  • James Aggrey said:

     

    James Aggrey

    I'm curious if you're still having issues with services not starting.  I've had the issue now on almost 10 machines where a lot of services are randomly not starting.  The ones I've noticed are Windows Audio, WLAN, Print Spooler, and multiple Sophos services.  I'm not sure if these issues are related to Sophos or if they're all symptoms of something else.  There aren't any recently installed Windows Updates so........ I'm pulling my hair out over here.  lol

    An update: I uninstalled Sophos on that laptop and all the Windows services started working normally after a reboot.  Friggin Sophos...... now I'm gonna try restarting to see if all the problems come back.

     

    Sorry for the delay in responding.  I've been away from work for a few days to commiserate becoming ancient!  I initially noticed this problem on our machines at roughly the same time we installed the Windows 10 1803 update (we'd already tested it with all internal applications, and wanted to install it on our timeframe, not Microsoft's), and as the Windows Audio service started exhibiting the same problem, it looked like the update was just adding a wee bit longer to startup time and services were timing out.  The fact that it was mostly older machines experiencing this problem and most of them were sorted with the ServicesPipeTimeout registry key just helped to reinforce this suspicion.  It's possible that removing Sophos would mean there was enough time left for the other Windows services to start up before timing out, although if this were the case I couldn't really blame Sophos as removing any service from the startup list would have had the same effect.

    In our case though, I've only noticed it with the Windows Audio and Sophos Device Encryption services.  I haven't seen issues with any of the other services you mention.

  • Does this need a reboot? Still not seeing any change to this service not running..

  • RBGE said:

     

     
    James Aggrey

     

    James Aggrey

    I'm curious if you're still having issues with services not starting.  I've had the issue now on almost 10 machines where a lot of services are randomly not starting.  The ones I've noticed are Windows Audio, WLAN, Print Spooler, and multiple Sophos services.  I'm not sure if these issues are related to Sophos or if they're all symptoms of something else.  There aren't any recently installed Windows Updates so........ I'm pulling my hair out over here.  lol

    An update: I uninstalled Sophos on that laptop and all the Windows services started working normally after a reboot.  Friggin Sophos...... now I'm gonna try restarting to see if all the problems come back.

     

     

     

    Sorry for the delay in responding.  I've been away from work for a few days to commiserate becoming ancient!  I initially noticed this problem on our machines at roughly the same time we installed the Windows 10 1803 update (we'd already tested it with all internal applications, and wanted to install it on our timeframe, not Microsoft's), and as the Windows Audio service started exhibiting the same problem, it looked like the update was just adding a wee bit longer to startup time and services were timing out.  The fact that it was mostly older machines experiencing this problem and most of them were sorted with the ServicesPipeTimeout registry key just helped to reinforce this suspicion.  It's possible that removing Sophos would mean there was enough time left for the other Windows services to start up before timing out, although if this were the case I couldn't really blame Sophos as removing any service from the startup list would have had the same effect.

    In our case though, I've only noticed it with the Windows Audio and Sophos Device Encryption services.  I haven't seen issues with any of the other services you mention.

     

    Your theory makes sense.  That would also explain why I haven't seen the issue on any of our newer computers.  I'll do some more experimenting.  Thanks man for your help!

Reply
  • RBGE said:

     

     
    James Aggrey

     

    James Aggrey

    I'm curious if you're still having issues with services not starting.  I've had the issue now on almost 10 machines where a lot of services are randomly not starting.  The ones I've noticed are Windows Audio, WLAN, Print Spooler, and multiple Sophos services.  I'm not sure if these issues are related to Sophos or if they're all symptoms of something else.  There aren't any recently installed Windows Updates so........ I'm pulling my hair out over here.  lol

    An update: I uninstalled Sophos on that laptop and all the Windows services started working normally after a reboot.  Friggin Sophos...... now I'm gonna try restarting to see if all the problems come back.

     

     

     

    Sorry for the delay in responding.  I've been away from work for a few days to commiserate becoming ancient!  I initially noticed this problem on our machines at roughly the same time we installed the Windows 10 1803 update (we'd already tested it with all internal applications, and wanted to install it on our timeframe, not Microsoft's), and as the Windows Audio service started exhibiting the same problem, it looked like the update was just adding a wee bit longer to startup time and services were timing out.  The fact that it was mostly older machines experiencing this problem and most of them were sorted with the ServicesPipeTimeout registry key just helped to reinforce this suspicion.  It's possible that removing Sophos would mean there was enough time left for the other Windows services to start up before timing out, although if this were the case I couldn't really blame Sophos as removing any service from the startup list would have had the same effect.

    In our case though, I've only noticed it with the Windows Audio and Sophos Device Encryption services.  I haven't seen issues with any of the other services you mention.

     

    Your theory makes sense.  That would also explain why I haven't seen the issue on any of our newer computers.  I'll do some more experimenting.  Thanks man for your help!

Children
  • One thing I'm starting to notice now is that I'll get an email alert saying "One or more Sophos services are missing or stopped", but by the time I check the machine, all services are running.  I'm guessing the fix means the services will start a bit later now, but the process that triggers the email alerts isn't taking this into account yet??

  • I feel like I've had that issue for a while, going back to "this device has suspended encryption" or whatever the error message was.  By the time I went to the machine it was fine.

  • I know some of the components have recovery such that if a service is missing, on the next update it will be re-added.

    Do you know which services were reported as missing?

    There is a trail here:
    C:\ProgramData\Sophos\Health\Event Store\Trail\ 

    The following PowerShell might help make sense of it:

    $strLoc = $env:ProgramData + "\Sophos\Health\Event Store\trail\"
    $strFileName = "*.json"
    $OutData = @()

    Get-ChildItem -Path $strLoc -File -Filter $strFileName | Foreach-Object {
    $j = [System.IO.File]::ReadLines($_.FullName) | ConvertFrom-Json
    $Arr = New-Object PSObject
    $Arr | Add-Member -Name "file" -MemberType NoteProperty -Value $_.Name
    $Arr | Add-Member -Name "id" -MemberType NoteProperty -Value $j.id
    $Arr | Add-Member -Name "familyId" -MemberType NoteProperty -Value $j.familyId
    $Arr | Add-Member -Name "timeStamp" -MemberType NoteProperty -Value $j.timeStamp
    $Arr | Add-Member -Name "app" -MemberType NoteProperty -Value $j.app
    $Arr | Add-Member -Name "sequence" -MemberType NoteProperty -Value $j.sequence
    $Arr | Add-Member -Name "severity" -MemberType NoteProperty -Value $j.severity
    $Arr | Add-Member -Name "resourceId" -MemberType NoteProperty -Value $j.resourceId
    $Arr | Add-Member -Name "componentName" -MemberType NoteProperty -Value $j.componentName
    $Arr | Add-Member -Name "showNotification" -MemberType NoteProperty -Value $j.showNotification
    $Arr | Add-Member -Name "updateSummary" -MemberType NoteProperty -Value $j.updateSummary
    $Arr | Add-Member -Name "serviceName" -MemberType NoteProperty -Value $j.serviceName
    $Arr | Add-Member -Name "counterName" -MemberType NoteProperty -Value $j.counterName
    $Arr | Add-Member -Name "userName" -MemberType NoteProperty -Value $j.userName
    $Arr | Add-Member -Name "userSid" -MemberType NoteProperty -Value $j.userSid
    $Arr | Add-Member -Name "path" -MemberType NoteProperty -Value $j.path
    $Arr | Add-Member -Name "reboot" -MemberType NoteProperty -Value $j.reboot
    $OutData += $Arr
    }
    #may not need all of these. Comment out one you don't need.
    $a = $OutData | Out-Gridview -Title "Health Event Store"
    $a = $OutData | ConvertTo-Html | Out-File -FilePath Report.html
    $a = $OutData | ConvertTo-CSV -NoTypeInformation | Out-File -FilePath Report.csv

    You could change the path to the directory to be a remote location, i.e. through a C$ share I suppose.

    The Sophos UI event list may also reveal what's being going on.

    Regards,

    Jak

  • Has anyone opened a ticket on this issue?   I have patently been waiting for a fix but I am ready to open a case on it. 

  • I have an open ticket.  The update did not fix my problem.  After the update, support gather more info and decided it was a corrupted file (log4net.dll) in my cases.  I had about 35 of them out of 400 systems.  Replacing this file with the original from the Sophos Autoupdate cache fixed the issue and allowed me to start the service.  Still waiting to see if it sticks or not.  I am now down to fewer than 5 devices with this issue after running in the 40s for months.