Endpoint Agent Failing on Windows Server 2003

Hello Adam,

first of all, like XP Server 2003 has been retired as supported platform.

update fails
because of an installation error for SAVXP? Or? The install, uninstall, CustomAction in %windir%\Temp\ should provide some details. Could be that the update fails because service control can't start SAVService.exe (is it actually stopped or stopping?)
It might or might not be possible to solve the problem without a reboot.

But anyway, 2003 is no longer supported, updates will cease to work any time now.

Christian

Thanks for the quick reply Christian.

I know about the impending retirement of support for Server 2003 and we are in the process of migrating to a new server but I'm hoping to keep SAV running on the old one for just a little while longer.

It appears that the current installation is damaged (the following is an excerpt from the Major Install log):

2017-07-03 15:26:47 Checking the integrity of the extant SAV installation (noUI is 0)
2017-07-03 15:26:47 The file \WSCClient.exe does not exist(2)
2017-07-03 15:26:47 The file \SavService.exe does not exist(2)
2017-07-03 15:26:47 The file \SavAdminService.exe does not exist(2)
2017-07-03 15:26:47 The file \BackgroundScanClient.exe does not exist(2)
2017-07-03 15:26:47 The file \ComponentManager.dll does not exist(2)
2017-07-03 15:26:47 The file \ICAdapter.dll does not exist(2)
2017-07-03 15:26:47 The file \ICManagement.dll does not exist(2)
2017-07-03 15:26:47 The file \ICProcessors.dll does not exist(2)
2017-07-03 15:26:47 The file \ThreatDetection.dll does not exist(2)
2017-07-03 15:26:47 The file \VirusDetection.dll does not exist(2)
2017-07-03 15:26:47 The file \SavControl.dll does not exist(2)
2017-07-03 15:26:47 The file \SavMain.exe does not exist(2)
2017-07-03 15:26:47 The file \SavProgress.exe does not exist(2)
2017-07-03 15:26:47 The file \DesktopMessaging.dll does not exist(2)
2017-07-03 15:26:47 The file \SavShellExt.dll does not exist(2)
2017-07-03 15:26:47 There is an incomplete SAV installation, forcing a Major Update to recover
...
2017-07-03 15:27:17 Unable to create an instance of ComponentManager - SystemInformation will not be informed of the update (0x80080005)
...
2017-07-03 15:26:47 ERROR: GetVersion - Unable to load the new Factory file, path = C:\Documents and Settings\All Users\Application Data\Sophos\Sophos Anti-Virus\Config\Factory.xml

And this from the Uninstall log:

MSI (s) (14:8C) [15:45:41:164]: SOURCEMGMT: Trying source C:\Program Files\Sophos\AutoUpdate\cache\savxp\.
MSI (s) (14:8C) [15:45:41:164]: Note: 1: 2203 2: C:\Program Files\Sophos\AutoUpdate\cache\savxp\Sophos Anti-Virus.msi 3: -2147287037
MSI (s) (14:8C) [15:45:41:164]: SOURCEMGMT: Source is invalid due to missing/inaccessible package.

I'm puzzled as to why the uninstaller is using the Program Files path, the msi is actually in C:\Documents and Settings\All Users\Application Data\Sophos\AutoUpdate\Cache\savxp

The service is in the Stopped state and will not start.

If a reboot is necessary it is an option but if the major update failed to uninstall the software I'm guessing that a manual uninstall is also likely to fail?

Adam.

Hello Adam,

10.3.15 is quite old, it might still be available as fixed package (at least I see this version in SEC/SUM).

As for the paths it works like this:
The Windows Installer can be told to cache an MSI, this location is recorded together with location of the original package. When 10.3 was installed AutoUpdate's cache was still under ProgramFiles, so the original location is pointing there
The cache has moved since and the 10.6/10.7 package is in the new InstallFromPath. The updating logic says that if a previous version of SAVXP is found it must be uninstalled. So the Installer first looks for the cached package, for whatever reason it is gone for good. It's then trying to find the original source - at the old path which is no longer there. Trying to locate the original source can basically have one of three outcomes - the package isn't found (path wrong or .msi not there), the package is found but it is not the correct one (e.g. for SAVXP it's always called Sophos Anti-Virus.msi but obviously it must have different contents and/or logic for different versions) also causing the Uninstall to fail, the package is found and its version is acceptable. Thus copying the current .msi to the either the Installer or the "old" AU cache won't work.

You'd need the 10.3.15 .msi, I'd put it in the Installer cache as 39119dc1.msi. That's still no guaranty that the Uninstall will succeed but you should try this first.

Christian

Thanks for the explanation Christian.

Can I just check that the correct way to obtain the 10.3.15.msi is by adding a new Subscription in the EC (I do see that Extended version as an option when adding a new subscription)? If so, will I need to create a new share to download the files?

Many thanks,

Adam

Hello Adam,

if you don't have a "spare" subscription/CID you have to add a new one. You just have to name the subscription and select a package version. SUM takes care of creating and naming the CID (the \S000\ is changed to some other value).
You don't have to create another updating policy (or let the server update from this CID), you just want the .msi.

Christian

Hi Christian,

Thank you. I created the new subscription and SUM downloaded the package to a new S009 CID. I copied the Sophos Anti-Virus.msi file from \S009\SAVSCFXP\savxp to C:\Windows\Installer and renamed it 39119dc1.msi.

Running an update now results in more progress - the Uninstall log shows a lot of activity from the MSI (unregistering components, deleting registry keys, etc.) - but, unfortunately, the uninstall still seems to be failing with a 1603 error and the Major Install log reports the following:

ERROR: Uninstall of SAV, version = 10.3.15, succeeded but IsSAVInstalled is true (10.3.15).
ERROR: Upgrade failure
Info: Set Update Failed
Unable to create an instance of ComponentManager - SystemInformation cannot be informed of end of update

I'm thinking it may now be worth rebooting the server and trying again, unless you have any other options I could try?

Many thanks,

Adam.

Hello Adam,

succeeded but IsSAVInstalled is true
always found this rather ... strange. failing with a 1603 - the actual error is either somewhere in the Uninstall log or the CustomActions log. Don't think a reboot will help unless the log tells that a missing reboot is the cause of the failure. I have seen some endpoints "stuck" on 10.3.15 (or even earlier), didn't have much in common but all could be recovered.

Christian

Hi Christian,

It is a very strange error indeed. Thanks for your continued help with this.

I found a few errors in the Uninstall log detailing files which were locked by ALMon and swc_service so I terminated those processes and tried the update again. It's still failing with error 1603 but the only errors I can find now are the following:

Uninstall log:

MSI (s) (A0:60) Executing op: RegCreateKey()
MSI (s) (A0:60) Executing op: RegOpenKey(Root=976,Key=Products\5B3B929D6C65CC643B3A1A7A48BC8B4E\Usage,SecurityDescriptor=BinaryData,BinaryType=0,)
MSI (s) (A0:60) Executing op: RegAddValue(Name=SAVService,Value=#1256456200,)
MSI (s) (A0:60) Note: 1: 1402 2: UNKNOWN\Products\5B3B929D6C65CC643B3A1A7A48BC8B4E\Usage 3: 6
MSI (s) (A0:60) Error in rollback skipped.    Return: 3
Info 1402.Could not open key: UNKNOWN\Products\5B3B929D6C65CC643B3A1A7A48BC8B4E\Usage.  System error 6.  Verify that you have sufficient access to that key, or contact your support personnel.

CustomActions log:

UninstallBootDriverFromInf: Action started
UninstallBootDriverFromInf: Executing RunInfSection with DefaultUninstall and DefaultUninstall.Services
UninstallBootDriverFromInf: RunInfSection: Error calling SetupOpenInfFile() 0x80070002. ErrorLine is 0
UninstallBootDriverFromInf: Action failed

Startup log:

Entering wWinMain
The argument passed to the Service identified an invalid registry key. The default registry key 'SOFTWARE\SOPHOS\SAVService' will be used.
Leaving wWinMain

Major Install log:

ERROR: SetupPlugin: Unable to get buffer size for Application registry key Path value.
...
ERROR: StoreTempFiles - failed to copy machine file - not present, hr = 0x0

Adam

Hello Adam,

there must be some other location in the Uninstall log with an error that mentions or near a UninstallBootDriverFromInf. This part is "just" a rollback error.

Anyway, I knew I have seen the SetupOpenInfFile() before (at least two typos in my post). Found another post and it seems that there's not really a solution apart from the mentioned Fix-It. Guess the .inf files are still there.

Christian

Hi Christian

You are correct about the UninstallBootDriverFromInf error:

Executing op: CustomActionSchedule(Action=UninstallBootDriver,ActionType=1025,Source=BinaryData,Target=UninstallBootDriverFromInf,CustomActionData=C:\Program Files\Sophos\Sophos Anti-Virus\SOPHOSBOOTDRIVER.INF)
MSI (s) (A0:C8) Invoking remote custom action. DLL: C:\WINDOWS\Installer\MSI334.tmp, Entrypoint: UninstallBootDriverFromInf
MSI (s) (A0:60) User policy value 'DisableRollback' is 0
MSI (s) (A0:60) Machine policy value 'DisableRollback' is 0
Action ended: InstallFinalize. Return value 3.

That file is also missing from C:\Program Files\Sophos\Sophos Anti-Virus\

The Fix-It mentioned in your other post appears to be for Windows 7, 8 and 10 so not sure if it will work on Server 2003?

Adam

"That file is also missing from C:\Program Files\Sophos\Sophos Anti-Virus\"

NB: The folder itself exists (and contains other files) but the .inf file is missing.

Hello Adam,

the .inf file is missing
in this case please copy the .inf files (from the \wxp_i386\ subdirectories) to the Program directory and try again. I'm not sure if this will resolve the issue - normally a missing file results in a different error message but who knows.

As for the Fix-It, can't say if it will refuse to install or run. Technically it doesn't matter whether a desktop or server OS. It does no harm to try - you get a prompt to select the products for which to remove the Installer information before it modifies something.

Christian

Hello Adam,

the .inf file is missing
in this case please copy the .inf files (from the \wxp_i386\ subdirectories) to the Program directory and try again. I'm not sure if this will resolve the issue - normally a missing file results in a different error message but who knows.

As for the Fix-It, can't say if it will refuse to install or run. Technically it doesn't matter whether a desktop or server OS. It does no harm to try - you get a prompt to select the products for which to remove the Installer information before it modifies something.

Christian

Hi Christian

I've copied the .inf files as suggested and that seems to have resolved that particular error but the uninstall process is still failing with error 1603. The only error I can find in the log files is this one (in the Uninstall log):

MSI (s) (58:E8) Note: 1: 1402 2: UNKNOWN\Products\5B3B929D6C65CC643B3A1A7A48BC8B4E\Usage 3: 6
MSI (s) (58:E8) Error in rollback skipped.    Return: 3
Info 1402.Could not open key: UNKNOWN\Products\5B3B929D6C65CC643B3A1A7A48BC8B4E\Usage.  System error 6.  Verify that you have sufficient access to that key, or contact your support personnel.

I had the same thought as you with regards to the Fix-It but it appears that the Server 2003 OS does not recognise the .diagcab extension.

Adam.

Hello Adam,

the only error
as "last time" this is the rollback error, there should be somewhere "farther up" a Return value 3 following the actual error. I know, a tedious process.

Christian

Hi Christian

We finally have success!

I'd missed a couple of .inf files when I transferred the SOPHOSBOOTDRIVER.INF earlier. After transferring these missing files the AutoUpdate routine successfully removed version 10.3.15 and installed 10.7.2.49. I just need to reboot the server tonight as the updating log (alc.log) shows a restart is now needed before a 'normal' AutoUpdate can be run.

Thank you so much for your patience and help with this, your guidance has been invaluable.

Kind regards,

Adam

Hello Adam,

good to hear it works.
a restart is now needed
not immediately, but as new components have been installed and some replaced, full functionality is only given after a reboot. A simple example: Assume there's a DLL which is loaded by certain or all processes and this DLL is replaced with a new version. Already running processes will continue to use the old one. In order to have them use the updated version you have to restart these processes. To be sure all processes are using the new version a reboot is the best option. 

You should eventually reboot but it will continue to update the 10.7.2 (if it stops updating then because 2003 has been retired). If up- and downgraded (several times in succession) endpoints that had the reboot required without a reboot in between. Very rarely I've seen the AutoUpdate refuses to upgrade to a higher version - detection data updates always work (well, perhaps not for years but definitely for months).
Thus if reboot would be a pain set it aside for now.

Christian

Hi Christian

Thanks for the explanation. An out-of-hours reboot shouldn't be an issue now but it's good to know that there is an option to postpone it if needed. It seems a slight shame to have achieved victory when support is due to end so soon but the learning process has definitely been worthwhile.

Adam

Hi Christian

I may have celebrated slightly prematurely.

I rebooted the server last night and, although AutoUpdate is working correctly, on-access scanning is now showing a status of 'Unknown' (it was working before the reboot) and the following error has appeared in the Event Log:

Failed to connect to the on-access driver (0x80070002)

I found this article but the registry keys it mentions do not exist and running the Virus Removal Tool doesn't find anything.

Sorry to bother you again but do you have any suggestions (other than uninstalling and re-installing the SAV)?

Adam

Hello Adam,

the registry keys it mentions do not exist
HKLM\SYSTEM\CurrentControlSet\Services\SAVOnAccessControl? 0x80070002 is a not found, did you also get the mentioned Event ID 43?

Sophos Anti-Virus has seemingly been successfully installed? I fear there is no workaround, won't suggest anything I can't test (no more 2003 or XP at hand), especially with a server.

Christian

Hi Christian

HKLM\SYSTEM\CurrentControlSet\Services\SAVOnAccessControl?

Correct, the only SAV... keys in that location are SAVAdminService, SAVRKBootTasks and SAVService. There is no corresponding EventID 43 in the Event Log.

Sophos Anti-Virus has seemingly been successfully installed?

It would appear so. Everything apart from on-access scanning seems to be working (I can manually scan files, for example).

won't suggest anything I can't test

That sounds wise to me, thanks for taking the time to reply anyway. I think the best option may now be to remove the software and reinstall manually (at least the prior work should now allow an uninstall so wasn't wasted effort).

Adam.

Hello Adam,

does the key perhaps exist in one of the previous ControlSets? Is the SAVOnAccess driver loaded (driverquery /v | find /i "SAVOnAccess" and fltmc instances | find /i "SAVOnAccess")?

Christian

Hi Christian

It does appear in a few other areas:

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_SAVONACCESSCONTROL

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Eventlog\System\SAVOnAccessControl

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet003\Enum\Root\LEGACY_SAVONACCESSCONTROL

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet003\Services\Eventlog\System\SAVOnAccessControl

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\LEGACY_SAVONACCESSCONTROL

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\System\SAVOnAccessControl

Both driver searches return nothing.

Adam