How to disable Sophos Endpoint Defense without booting into safe mode

Hello Christopher Thompson,

I'm neither a Central nor an SED expert but I fear I'm right when I say there's no other way than 124377. I wonder how this crippled installation came about though. 

they do have the AutoUpdate component
Is it working? If it is, it should try to install the missing components but then this very likely fails for the same reason as the initial attempt. Although SAV is not installed - is the SavService present? AFAIK (but this might have changed with SED) installs/uninstalls are only blocked when it is running.

In either case the failure rate suggests a common cause that you probably have to identify and correct before an install will succeed.

Christian

Hello Christopher,

Would it be possible to confirm the method of migration you have used here to understand the process that has happened here?

Additionally if Sophos Management Communications System is on the system do you know if the endpoint has created an entry for itself in the central dashboard? If so can you disable Tamper Protection from the console and does this feedback to the endpoint?

I would also highly recommend raising a case with Sophos Support if you have not already. If you have can you message me the case reference?

https://secure2.sophos.com/en-us/support/contact-support.aspx

As far as I can tell, the only two Sophos processes running are ALsvc.exe (AutoUpdate) and McsClient.exe. Sophos support directed me to replace C:\ProgramData\Sophos\AutoUpdate\Config\iconn.cfg on a bad install with a copy from a good install. This seems promising because the file is almost empty on bad installs and contains necessary parameters like the server URL, username and password. Unfortunately, if I try to update or replace the file I'm again thwarted by Endpoint Defense.

The workstations with bad installs are no different from workstations with the good installs, as far as I can tell. I have good and bad installs on all our images and in all of our offices. I haven't been able to figure out why some succeeded and others failed.

I performed the migrations using two different methods. First, I migrated about 30 clients using the Sophos Cloud Migration Tool and of those, 3 were failures. I migrated the remainder through SCCM using a deployment packager called PowerShell App Deployment Toolkit. The package took three actions, first uninstalling Sophos Client Firewall (if present), then running "SophosInstall.exe -q -tps remove" and forcing a reboot at the end.

Yes, all affected installs are registered in Sophos Central and communicating with the portal. Once a day, they try to update the agent and fail. Interestingly, they only display "medium" status even though they're completely missing AV and web filtering. I did try disabling tamper protection through Sophos Central on multiple clients but it does not deactivate Endpoint Defense.

I do have a case open with Sophos and will send you the number.

Thank you for your help.

Hello Christopher Thompson,

I hope they do something with the logs. Seems that under certain circumstances the Central installer paints itself into a corner. It's somewhat, err, funny that SED is apparently in full operation before the product is successfully installed.

Christian

We are having the exact same issue, I have migrated about 380 machines of about 1200 with the Migration Tool so far and have had about 40 fail with the exact same issue described.

We will in the future be putting on our Gold Windows 10 image, but until then im stuck with the safe mode option.

I have logged this with Sophos to see if there are any other options.

In the batch file or script that runs sophosinstall.exe as part of the migration.  Prior to that command, would it help to run:

"C:\Program Files\Sophos\Endpoint Defense\uninstall.exe" /quiet

If the migrations are coming from 10.6.4, then SED is installed but disabled unless you add a registry key locally.  but I assume that as part of the migration, as MCS becomes installed and the computer managed by Central a SED policy comes down mid way through the migration and enabled SED at an odd time.  Maybe this is why it is intermittent as it's a race condition.

I can't think of a way to disable the SED boot driver without using safe mode.  It might be possible to create a batch file/script that will reboot the computer next time automatically to safe mode, e.g.

bcdedit /set {default} safeboot minimal
shutdown /r /t 0

But to fully automate this, say out of hours, it would also have to setup a script to run at next startup (that could run in safe-mode, I think run locations are limited), maybe using the HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\ shell key to chain explore.exe and a batch file.  You can use a comma to declare multiple commands.
The safe mode batch file or script could then:
- edit the registry to disable the SED driver, reg.exe
- restore the shell key to just Explorer.exe 
- restore normal boot with:
bcdedit /deletevalue {default} safeboot
shutdown /r /t 0

The only issue you'd have is automating the login on reboot, into safemode you'd have to create a temporary admin with net commands and set a password.  This account be later deleted but the password would probably just set in clear text in the script.  Maybe use https://technet.microsoft.com/en-us/sysinternals/autologon.aspx to simplify the auto-logon if that would work.

I suppose if you have some encyption software, safe mode would be even harder to automate.

Either way you slice it that is quite a tricky script but one I suspect has already been done before in some way.

In terms of trying to do something before the SED driver starts, a quick look in the registry ("HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense") shows its group to be: "FSFilter Security Enhancer".   If I run LoadOrder (https://technet.microsoft.com/en-us/sysinternals/loadorder.aspx) there are a few groups that a driver could belong to to perhaps disable SED before it starts.  

Regards,

Jak

Jak,

Thank you for your feedback. I attempted to automate the repair process using a similar technique to the one you described (bcdedit and Run/RunOnce registry keys) but didn't have success and ultimately fixed the affected workstations by hand.

For Sophos and those affected by the issue, it appears to occur during the install sequence if policy is not received from the Sophos Update servers within 5 minutes.

2/8/2017,3:42:27 PM,Information,Started installing autoupdate ...,
2/8/2017,3:42:29 PM,Information,Sucessfully enabled Endpoint Defense for SAU.,
2/8/2017,3:42:29 PM,Information,Register with MCS...,
2/8/2017,3:42:30 PM,Information,The SAU configuration could not be retrieved, pre-check for configuration not done.,
2/8/2017,3:47:31 PM,ERROR,There was an unexpected problem with the installation of Sophos Endpoint Security and Control. Details: The updating policy failed to arrive in the expected time,
2/8/2017,3:47:31 PM,ERROR,Install failed!,
2/8/2017,3:47:31 PM,Information,Sending EBS feedback to Sophos...,
2/8/2017,3:47:31 PM,Information,Sending data back to Sophos...,
2/8/2017,3:47:31 PM,Success,Successfully connected to the URL d1.sophosupd.com/.../x.xml.,
2/8/2017,3:47:31 PM,Information,------------------ Found errors during installation: 137 ------------------,
2/8/2017,3:47:31 PM,Information,------------------ Installation program finishing with code 137 ------------------,

After examining our deployment logs, it appears that many but not all of the failures occurred in the morning between 7-11 am CST. I have also experienced performance problems with the Sophos Central portal (cloud.sophos.com) during this time so it's my belief that this issue is caused by a combination of factors: a faulty install process that enables "tamper protection" before the workstation receives its first policy update, too short of a policy update timeout, and a heavy load on Sophos Central in the morning when US users are logging in and receiving policy updates.

What about the following script...

It boots the computer into Safe Mode with Networking, the user doesn't need to login as the initial script, that needs to run as admin, creates a service that will run as the system user at start-up.  This second script run by the service will auto restart the computer back into regular mode having disabled the SED driver from starting and also disables MCS Agent to avoid that re-applying policy.

Batch file 1 - To be run as admin:

@echo off
instsrv.exe DisableSED "C:\Program Files (x86)\Windows Resource Kits\Tools\srvany.exe"
REG ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DisableSED\Parameters" /v Application /f /t REG_SZ /d "C:\RemoveSED\Startup.bat"
REG ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SafeBoot\Network\DisableSED" /t REG_SZ /d "Service" /f
bcdedit /set {default} safeboot network
shutdown /f /r /t 3

Batch file two: Startup.bat

@echo off
REG ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos Endpoint Defense" /v Start /d "4" /t REG_DWORD /f >> C:\RemoveSED\SEDRemove.txt 2>&1
REG ADD "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sophos MCS Agent" /v Start /d "4" /t REG_DWORD /f >> C:\RemoveSED\SEDRemove.txt 2>&1
"%windir%\sysnative\bcdedit.exe" /deletevalue {default} safeboot >> C:\RemoveSED\SEDRemove.txt 2>&1
REG DELETE "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SafeBoot\Network\DisableSED" /f >> C:\RemoveSED\SEDRemove.txt 2>&1
taskkill /F /IM srvany.exe >> C:\RemoveSED\SEDRemove.txt 2>&1
sc delete DisableSED >> C:\RemoveSED\SEDRemove.txt 2>&1
shutdown /r /f /t 5 >> C:\RemoveSED\SEDRemove.txt 2>&1

The first batch file could write out the second but for ease of troubleshooting.

I did try initially with a single batch file that created a few scheduled tasks to run on boot to run in safe mode but didn't have much success with Task Scheduler running in safe mode even with the safeboot key being updated to enabled the schedule service.  Too many dependencies maybe.

The files instsrv.exe and srvany.exe can be obtained from the 2003 Resource Kit - www.microsoft.com/.../details.aspx
Some details here: https://support.microsoft.com/en-gb/help/137890/how-to-create-a-user-defined-service

TBH, it might be as easy to create a service to carry out the above to remove the need for them but they should just work and they are from Microsoft.  The one thing to watch is that the service created is 32-bit.  Interestingly, on a 64-bit computer, as bcdedit.exe is only native, you have to call it with the path "%windir%\sysnative\bcdedit.exe".

You could use Winrar to configure this as a single SFX which will autorun the first script.

Regards,
Jak