Thread Info
  • State Verified Answer
  • Locked Locked
  • Replies 16 replies
  • Subscribers 2 subscribers
  • Views 19868 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Upgrade to Sec5.5 half successful. Patch Service Issues - stuck in staring state

Redfern

Hi Guys, 

So Im stuck in quite a strange problem. I will try to explain as best as i can below and also will lay out some background info:

3 days ago:

So we was happily running Sec5.4.1 on a Windows 2008r2 Standard and then we decided to upgrade the console to 5.5. 

However we did not have the SUM password or the SophosManagment Database access accout password. So we reset the passwords in AD and re-ran the sec5.4.1 Installer in to reset the passwords. If i remember correctly that went fine. 

Then we Ran the Sec5.5 installer to update it. Initially it failed because of a cached password policy we had. So I removed the policy and re-ran it. This time the installation a completed, but when i restarted the server, enterprise console failed to open. I think the database upgrade to from 540 to 550 failed. so i dropped the Sophos550 databse and recreated it manually using the UPgradeDB utility no the sophos console would open but it i had lost all my policies and computers. so i contacte Sophos support and they issues the folwoing commands: 

sqlcmd -S -S .\SOPHOS -d SOPHOS550 -q "Update Upgrade Set UpgradeStatus=1" 
sqlcmd -E -S .\SOPHOS -d SOPHOS550 -q "From540" 
sqlcmd -S -S .\SOPHOS -d SOPHOS550 -q "Update Upgrade Set UpgradeStatus=2" 

So after this everything seemed to work. I had al my policies and computers back,  The clients were updating fine and the console was updating fine and i could change policies and protect clienst form the console all fine. 

 

Today:

So we come in today with a plan to migrate the console to another machine. So i go into to services to stop them and i find the:

  1. The three Sophos Patch Services are stuck on starting – 
  2. I am getting this log in the Event Viewer: Event 0, PatchFeedProcessor, Processing stopped (handled error):'Failed to check upgrade complete status'
  3. Also I’m getting the following log in windows event logger like every 30 seconds: Event 18456, MSSQL$SOPHOS, Login failed for user 'DOMAIN\SophosManagement'. Reason: Failed to open the explicitly specified database. [CLIENT: <local machine>]

So I verified with the following :

SQLCMD -E -S SQLSERVER\SOPHOS

1> SELECT * FROM SYSDATABASES WHERE NAME LIKE "%PATCH%"
2> GO

This showed that SophosPatch52 is the database name which is correct. 

Then i went into SQL Servermanagement Studio and checked the properties of the SophosPatch52 database. it had no users/groups in the permissions tab. so i manualy added the Sophos DB Admins Group to it and gave it connect permissions (same as the other databases) - (THIS STEP CLEARED THE WINDOWS MSSQL$SOPHOS LOGIN FAILED ERROR)

But the services still did not start after taskilling them and re-trying. 

Then i tried this:

 

1> USE SOPHOSPATCH52
2> UPDATE Upgrade
3> SET UpgradeStatus=2 WHERE ID=1
4> GO

it seemed to do something but still the patch services did not start. 

 

i have also tried running: the updatepatchDB.bat Domain\SOPHOS NetbiosDomain SOPHOSpatch52 Sophos_updatepatchDB.log it seems to do something but nothing changes. 

 

So i spoke to Sophos Support and they said i should go ahead with the migration which might fix the issue. 

so i did backup up the old server and  then i restored them to the new server. however when i try tried to instal enterprise console on the new server it failed. i checked the services and the patch services were stuck in starting state. 

I dont know where to go from here..

 

Does any one have any advice suggestions please?

 

Cheers, 



This thread was automatically locked due to age.
Parents
  • 0

    Hello Redfern,

    when i tried to install enterprise console on the new server it failed [...] patch services were stuck
    more or less halfway through and not rolled back? You did install the database component, restore the database and then installed the management server (and console) and this failed?
    Errors in the Event log for the Patch services? Dunno if trace logging would give more insight.

    Christian

  • 0 in reply to QC

    Hi Christian. 

    Thank you for your response. 

     

    Yes it stopped about half way through then it started rolling back. When it was rolling back i checked the services and the 3 Sophos patch services were stuck in starting state. 

    Yes i Installed the database component and then restored it according to the guide. then i installed the management serve at which point it failed. 

    i checked the patch services logs just now in C:\ProgramData\Sophos\Patch\Logs and they are saying the below:

     

    2018-06-04 15:05:39 | PID 3484 | TID 4 | ID: 5000 | Severity: error | Error fetching upgrade status.-- System Exception Details --
    Message: The EXECUTE permission was denied on the object 'usp_UpgradeStatusGet', database 'SOPHOSPATCH52', schema 'dbo'.
    Type: SqlException
    Source: .Net SqlClient Data Provider
    Target: Void OnError(System.Data.SqlClient.SqlException, Boolean, System.Action`1[System.Action])
    Help Link:
    -- Evidence At Publish --

    do you think i need to give execute permissions through SQl for this to work? perhaps this issue is caused by me manually adding the sophos DB admins account to the SophosPatch52 in SQL?

     

    What do you reckon?

     

  • 0 in reply to Redfern

    Another update:

     

    i went in to SQL and gave the Sophos DB Admin group execute permissions on the SophosPatch52 Databse. 

    now the error in the Patch services log has changed

     

    FROM:

    2018-06-04 15:05:39 | PID 3484 | TID 4 | ID: 5000 | Severity: error | Error fetching upgrade status.-- System Exception Details --
    Message: The EXECUTE permission was denied on the object 'usp_UpgradeStatusGet', database 'SOPHOSPATCH52', schema 'dbo'.
    Type: SqlException
    Source: .Net SqlClient Data Provider
    Target: Void OnError(System.Data.SqlClient.SqlException, Boolean, System.Action`1[System.Action])
    Help Link: 
    -- Evidence At Publish

    TO:

    2018-06-04 16:36:08 | PID 3484 | TID 4 | ID: 5000 | Severity: error | Error fetching upgrade status.-- System Exception Details --
    Message: Input string was not in a correct format.
    Type: FormatException
    Source: mscorlib
    Target: Void StringToNumber(System.String, System.Globalization.NumberStyles, NumberBuffer ByRef, System.Globalization.NumberFormatInfo, Boolean)
    Help Link:
    -- Evidence At Publish --

     

     

  • 0 in reply to Redfern

    Also the "SQL Server Agent (sophos)" service starts and then immediately stops. 

  • 0 in reply to Redfern

    Hello Redfern,

    working from bottom to top
    the SQL Server Agent (SOPHOS) is disabled on my servers so apparently it's not involved.

    Type: FormatException
    Source: mscorlib
    doesn't really tell much. Not sure if it refers to the SQL error or an error during interpretation of the SQL error.

    updatepatchDB.bat
    just creates updated functions and procedures

    Sophos DB Admin should have the correct permissions

    It looks like the (more or less original) SOPHOSPATCH52 database is corrupted. Wonder if it is possible to simply start with a fresh SOPHOSPATCH52 or if SEC would complain that it can't find an endpoint's referenced Patch data. Unfortunately at the moment I don't have a machine available with which I could test.

    Christian

  • 0 in reply to QC

    Yes thats correct. The Sophospatch52 databse was having issues. so deleted itr and recreated it and set the upgrade to status 2. now the services are running fine so thats sorted. 

     

    the migration is still failing though. i get this error in when installing management server and console on the new machine (after backing up and restoring databases) :

     

    05/06/2018 11:16:43, WARNING : Could not copy SmAgentAPI.dll from C:\sec_550\ServerInstaller\pre-reqs\SmAgent\SmAgentAPI.dll to C:\Program Files (x86)\Business Objects\BusinessObjects Enterprise 11\win32_x86\plugins\auth\secLDAP\SmAgentAPI.dll, copyFile return 5 - Access is denied.

    05/06/2018 11:16:43, INFO : Verifying files in folder
    05/06/2018 11:16:43, INFO : Target folder verification completed successfully
    05/06/2018 11:16:43, INFO : About to install Database64.msi
    05/06/2018 11:16:48, INFO : Processing INSTALLMESSAGE_TERMINATE message from MSI
    05/06/2018 11:16:48, INFO : Ended installing Database64.msi
    05/06/2018 11:16:50, INFO : Installation of Database succeeded
    05/06/2018 11:16:50, INFO : Verifying files in folder
    05/06/2018 11:16:52, INFO : Target folder verification completed successfully
    05/06/2018 11:16:52, INFO : About to install Server64.msi
    05/06/2018 11:18:29, INFO : Custom action: CreateCDLTask has failed
    05/06/2018 11:18:29, INFO : Deactivate state: Installing
    05/06/2018 11:18:29, INFO : Activate state: Failing
    05/06/2018 11:18:54, INFO : Installation of Server64.msi failed with error code: 1603
    05/06/2018 11:18:54, INFO : Ended installing Server64.msi
    05/06/2018 11:18:56, INFO : Installation failed with error code: 1603
    05/06/2018 11:18:56, INFO : Deactivate state: Failing
    05/06/2018 11:18:56, INFO : Activate state: Failed
    05/06/2018 11:18:56, INFO : Entered Installation failed page.
    05/06/2018 11:19:01, INFO : Opening logs folder: C:\ProgramData\Sophos\Management Installer

     

    Any ideas?

     

Reply
  • 0 in reply to QC

    Yes thats correct. The Sophospatch52 databse was having issues. so deleted itr and recreated it and set the upgrade to status 2. now the services are running fine so thats sorted. 

     

    the migration is still failing though. i get this error in when installing management server and console on the new machine (after backing up and restoring databases) :

     

    05/06/2018 11:16:43, WARNING : Could not copy SmAgentAPI.dll from C:\sec_550\ServerInstaller\pre-reqs\SmAgent\SmAgentAPI.dll to C:\Program Files (x86)\Business Objects\BusinessObjects Enterprise 11\win32_x86\plugins\auth\secLDAP\SmAgentAPI.dll, copyFile return 5 - Access is denied.

    05/06/2018 11:16:43, INFO : Verifying files in folder
    05/06/2018 11:16:43, INFO : Target folder verification completed successfully
    05/06/2018 11:16:43, INFO : About to install Database64.msi
    05/06/2018 11:16:48, INFO : Processing INSTALLMESSAGE_TERMINATE message from MSI
    05/06/2018 11:16:48, INFO : Ended installing Database64.msi
    05/06/2018 11:16:50, INFO : Installation of Database succeeded
    05/06/2018 11:16:50, INFO : Verifying files in folder
    05/06/2018 11:16:52, INFO : Target folder verification completed successfully
    05/06/2018 11:16:52, INFO : About to install Server64.msi
    05/06/2018 11:18:29, INFO : Custom action: CreateCDLTask has failed
    05/06/2018 11:18:29, INFO : Deactivate state: Installing
    05/06/2018 11:18:29, INFO : Activate state: Failing
    05/06/2018 11:18:54, INFO : Installation of Server64.msi failed with error code: 1603
    05/06/2018 11:18:54, INFO : Ended installing Server64.msi
    05/06/2018 11:18:56, INFO : Installation failed with error code: 1603
    05/06/2018 11:18:56, INFO : Deactivate state: Failing
    05/06/2018 11:18:56, INFO : Activate state: Failed
    05/06/2018 11:18:56, INFO : Entered Installation failed page.
    05/06/2018 11:19:01, INFO : Opening logs folder: C:\ProgramData\Sophos\Management Installer

     

    Any ideas?

     

Children
  • 0 in reply to Redfern

    Hello Redfern,

    the details should be in the Sophos_Server64msi log. CreateCDLTask should create the Sophos Patch Feed task.

    Christian

  • 0 in reply to QC

    i had to delete the existing SmAgentAPI.dll file and then also had to disable the Network access: Do not allow storage of passwords and credentials for network authentication policy. 

    Now the install for the console  has completed. Im just completing the migration end steps now. 

    do you know roughly how long it takes for the clients to pick up the new updating policy? my compuyters have just been sat there with awaiting policy Transfer for about 30 mins.

     

    Cheers

  • 0 in reply to Redfern

    Hello Redfern,

    delete
    just a warning, shoudln't have been necessary (IIRC I simply ignored it)

    disable
    thought as much (sorry that I didn't already suggest it)

    awaiting policy Transfer
    first question: Are they communicating - and with the correct server? You said this is a migration and somehow the endpoints must come to know they have to bid farewell to the old and welcome the new one. How did you go about this?

    Christian


     

  • 0 in reply to QC

    Hi Christian, 

     

    I though they were but i think i was wrong. As currently the none of the PC's are showing connected!, 

    also it still says awaiting transfer policy. i tried to issue reprotect to all the PC's but it did not work. 

     

    What do you think im missing?

  • 0 in reply to Redfern

    Hello Redfern,

    is Protect supposed to work i.e. you've used it before? How did it not work - what's the error shown in the console? There's an article that describes how Protect works.

    Indeed Protect is the recommended method to redirect the endpoints. You did export and import the RMS certificates during migration, didn't you? I assume the new server has a different name and IP. Using an alias can work - it depends on your mrinit.conf - usually it contains not just an IP but the server's NetBIOS and FQDN. You'd have to turn off old, add old's name (preferably a DNS FQDN) as an alias for new - the endpoints should then find new. 

    Christian

  • 0 in reply to QC

    well protect worked on the old console just fine. Its how we always deployed Sophos. We never had to manually run setup.exe or use any scripts etc. the migration guide i was following instructed to do backup. I can see in the saved backup from old server that the certificationmanager REG Entry is backed up along with with 5 other reg files. also i restored these to the new server as part of the migration guide. 

    Yes the server has a different name and IP, but the old server cannot be turned off as it is DC and holds other critical services (one of the reasons why we are migrating the Sophos Console). 

    the error i am getting are genericish - it will say things like:

    Install failed. computer may need additional config. - awaiting response from computer

    if i check the Sophos communication report it says :

    There is a problem communicating with the server. - DNS issues

    State of outgoing communications to server - comunication failure 

     

    Whats strange is that when i try to push protect form the console, the client PC recived the task in tsk schduler but then it doesnt run. one of the clients said that there was already an instance of the task running. 

     

    Im sure i have my DNS fine, firewall rules are fine, shares and security is fine. its just so strange that it still wont work. :-(

     

  • 0 in reply to Redfern

    Hello Redfern,

    I'll have to think about it, it's almost 6pm (and I'm anyway not Sophos). Should be possible to make it work again if it worked before.
    Is your old SEC still alive and do the endpoints communicate?

    Christian

  • 0 in reply to QC

    Hi Christian,   I know you are not Sophos, Ive been following your advice for other issues for quite well over a year now (worked as tech support for an IT company, managing schools ICT who all used Sophos) and  I thank you for helping me and responding to my messages. 

    Yes the old SEC is still alive and communicating fine. 

    I think I'm going to have to revisit the Sonicwall and windows firewalls...

    Cheers. 

  • +1 in reply to Redfern

    Hello Redfern,

    it might be a firewall might preventing successful use of Protect.

    if i check the Sophos communication report
    the one on an endpoint you attempted to re-protect? First thing to check is whether the Parent addresses: are those for the new server. If they are then the install has principally succeeded. There shouldn't be DNS issues - but if there are they must have been present before. Can't say what they could be or whether they'd prevent communication in all cases.

    Christian

  • 0 in reply to QC

    Hi Christian, 

     

    It was the a firewall issue. Windows firewall had the old IP in the scope which needed changing. This allowed the clients communicate with the server properly. we did still have an issue where we couldn't re-protect over half the clients. however we solved that by using the script you referenced in another post! Thanks :-)

     

    so now that thats done, we will look at upgrading to 5.5.1 ( finger crossed it goes smoothly) and deploying a child Enter prise console aswell. 

    Cheers

    Thanks again. 

     

     

Unfiltered HTML