Update Manager stuck at Downloading Binaries

First check for me is if SUM can report status into the management server.

You could check the last message time of the SUM computer in the endpoint view - Computer Details column is fine.  This should be pretty recent and this will prove if RMS is working gernerally.  I.e. RMS on the SUM computer is able to send in a status or event message - either contribute to this timestamp.  

This status may not however include data about the SUM.  For this reason, it would be good to prove that status info about SUM can be returned.

One quick check - in the registry on the SUM server you can change the version of SUM, just temporarily. It is returned from this value: 

HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Sophos\UpdateManager
Product Version
e.g: 1.6.2.186

If you change it to say, 1.6.2.1861 and restart the "Sophos Agent" service, within 20 seconds, the UI of the Console should reflect this new string.

You should shortly after revert the value in the registry and restart the Sophos Agent to return the value.

This test will prove that the Sophos Agent service is able to load the SUM adapter DLL and is able to return a SUM status to the management server.

Also can I check - is the SUM server in this state on the management server or is it remote?

Regards,

Jak

Hi Jak,

Thanks for the info... 

I did what you mentioned, and when I check the last message time it lists Aug 15th, thats for all my endpoints along with the SUM... I noticed that the SUM has an IP of 169.254.X.X, while all my endpoints have their respective ips... don't know if that's an issue, never noticed it before.

as for the Registry temp change, it didn't reflect at all in the UI... So I switched it back.

I have sent more info to Sophos support yesterday and today, and haven't heard anything yet. I'm almost at the point of reinstalling the Enterprise Console to see if that would fix this issue.

Thanks

Denis

OK, well a little bit of an overview of the messaging system seems worth while.

When you install SEC, the management server installer determines if the server has a dynamic or static IP.  Of course it may have multiple IPs.  In either case, the file mrinit.conf is constructed.  If the computer has a static IP, then the ParentAddress will contain:

IP,fqdn,NetBIOS

If the computer has a dynamic IP, then the ParentAddress will contain:
FQDN,NetBIOS

These addresses are essentially how the clients find the management server.   It also has the ports needed.  I.e. 8192 and 8194.

The file mrinit.conf if added to the root of the distribution point.  When you deploy to a endpoint, setup.exe is run on the client and copies mrinit.conf and cac.pem to the client to the directory: C:\program files (x86)\sophos\remote management system\.  

AutoUpdate installs, pulls down the components which includes RMS.  RMS installs and the helper tool clientmrinit.exe reads the files from disk and adds the data to the registry.  The Sophos Message Router service uses the registry values from then on.  The only time the files will be used is if there is an update to RMS and clientmrinit.exe runs again.

So the first thing I would check is the ParentAddress value in the registry on the managed clients. 
64-bit: "HKLM\SOFTWARE\Wow6432Node\Sophos\Messaging System\Router\"
32-bit: "HKLM\SOFTWARE\Sophos\Messaging System\Router"

The clients will need to be able to resolve at least one of the addresses in that comma separated list to the management server.  It will take a few minutes to go between each in the list.

A good test from the client is therefore, can the client connect to TCP port 8192 and 8194 of the management server using at least one address in that list, ideally the first for speed. E.g.

telnet ip 8192
telnet ip 8194

Note: you may need to add the telnet client from appwiz.cpl.

The router log on the client will also log all of these taking place.  \programdata\sophos\remote management system\3\router\logs\

Maybe the firewall on the management server has been activated and prevented the client routers from connecting to these ports?  It would account for a sudden site wide communication issue.

Beyond that, the router on the client first reads the "IOR" string from port 8192 of the server.  This IOR string, should point the client back at the server and port 8194 (SSL) port.  You can decode the IOR using http://catior.org.  For each IP on the management server you will have a "profile".  I assume that the IOR does contain the IP of the management server.

This is perhaps the first checks, let me know how these check out.

I wouldn't re-install just yet as you may end up needing to re-install all the clients to get the certificates to re-align.

Regards,
Jak

Hi Jak,

I did as you mentioned, I ran the telnet tests from a CLient AND from the SUM server itself, and it came with this when i did telnet ip 8192

Type ID: "IDL:SophosMessaging/MessageRouter:1.0"
Profiles:
1. IIOP 1.2 169.254.98.103 8193 "....NUP...!........RootPOA.RouterPersistent.........MessageRouter"
TAG_ORB_TYPE 0x54414f00
TAG_CODE_SETS char native code set: ISO-8859-1
char conversion code set: UTF-8
wchar native code set: UTF-16
wchar conversion code set:

TAG_SSL_SEC_TRANS port = 8194 supports = 166 requires = 134

2. IIOP 1.2 169.254.105.84 8193 "....NUP...!........RootPOA.RouterPersistent.........MessageRouter"
TAG_ORB_TYPE 0x54414f00
TAG_CODE_SETS char native code set: ISO-8859-1
char conversion code set: UTF-8
wchar native code set: UTF-16
wchar conversion code set:

TAG_SSL_SEC_TRANS port = 8194 supports = 166 requires = 134

3. IIOP 1.2 169.254.223.156 8193 "....NUP...!........RootPOA.RouterPersistent.........MessageRouter"
TAG_ORB_TYPE 0x54414f00
TAG_CODE_SETS char native code set: ISO-8859-1
char conversion code set: UTF-8
wchar native code set: UTF-16
wchar conversion code set:

TAG_SSL_SEC_TRANS port = 8194 supports = 166 requires = 134

4. IIOP 1.2 SERVERIP 8193 "....NUP...!........RootPOA.RouterPersistent.........MessageRouter"
TAG_ORB_TYPE 0x54414f00
TAG_CODE_SETS char native code set: ISO-8859-1
char conversion code set: UTF-8
wchar native code set: UTF-16
wchar conversion code set:

TAG_SSL_SEC_TRANS port = 8194 supports = 166 requires = 134

5. IIOP 1.2 SERVERIP 8193 "....NUP...!........RootPOA.RouterPersistent.........MessageRouter"
TAG_ORB_TYPE 0x54414f00
TAG_CODE_SETS char native code set: ISO-8859-1
char conversion code set: UTF-8
wchar native code set: UTF-16
wchar conversion code set:

TAG_SSL_SEC_TRANS port = 8194 supports = 166 requires = 134

When i did TELNET ip 8194, it was a blank black screen, which i've read somewhere that's what is supposed to happen... I've done these tests last week, except for the decode part.

I looked at the logs for the Router on the client's side, and there aren't any errors either... it successfully validated parent router's IOR, and so on.

I will also mention this... when I go to the Configure Update Manager option and EDIT the Source (which is SOPHOS), when the Source Details comes up with Address of SOPHOS and username/password... when I click on OK... I get an error message after a min or so.... Says Timeout while attempting to conenct to the specified address...

See image below

And I also looked at the sophos-management-services.log, and noticed something happened... and I can't make sense of it... as if a version got rolled back from an update, and that's when things started to go south..

2017-08-15 12:15:45,489 [17] INFO {Sophos.Management.Services.Sddma.WarehouseAttributesSynchroniser.GetRequestMessage} ==> Generating getWarehouseAttributes request.
2017-08-15 12:15:45,504 [17] INFO {Sophos.Management.Services.Messaging.Handler.receiver_OnDoAction} ==> Received "do action" from Sophos.Management.Services.Sddma.MessageReceiver details Sophos.Management.Services.Sddma.MessageReceiver
2017-08-15 12:15:45,504 [17] INFO {Sophos.Management.Services.Com.Messaging.handler_SendDataPacket} ==> Marshaling a DoAction message for SDDM on Sophos.Management.Endpoint data size 453
2017-08-15 12:15:45,504 [17] INFO {Sophos.Management.Services.Com.Messaging.get_Sink} ==> Message Sink is ready.
2017-08-15 12:15:45,536 [17] INFO {Sophos.Management.Services.Sddma.ProductReleasesSynchroniser.GetRequestMessage} ==> Generating getProductReleases request.
2017-08-15 12:15:45,551 [17] INFO {Sophos.Management.Services.Messaging.Handler.receiver_OnDoAction} ==> Received "do action" from Sophos.Management.Services.Sddma.MessageReceiver details Sophos.Management.Services.Sddma.MessageReceiver
2017-08-15 12:15:45,551 [17] INFO {Sophos.Management.Services.Com.Messaging.handler_SendDataPacket} ==> Marshaling a DoAction message for SDDM on Sophos.Management.Endpoint data size 490
2017-08-15 12:15:45,551 [17] INFO {Sophos.Management.Services.Com.Messaging.get_Sink} ==> Message Sink is ready.
2017-08-15 12:16:14,912 [17] INFO {Sophos.Management.Services.Sddma.StatusMonitor.ExtractUpdateInfoAndRaiseEvent} ==> Self-update information found. Sending update notified event.
2017-08-15 12:16:15,146 [17] INFO {Sophos.Management.Services.Sddma.WarehouseAttributesSynchroniser.ProcessReply} ==> Received warehouse attributes reply.
2017-08-15 12:16:15,551 [17] INFO {Sophos.Management.Services.Sddma.ProductReleasesSynchroniser.ProcessReply} ==> Received product releases reply.
2017-08-15 12:16:16,222 [17] INFO {Sophos.Management.Services.Sddma.VersionRolloverService.PerformVersionRollover} ==> Version rollover service invoked
2017-08-15 12:16:16,347 [17] INFO {Sophos.Management.Services.Sddma.VersionRolloverService.FetchCurrentAndPreviousProductData} ==> Got current & previous data.
2017-08-15 12:16:16,362 [17] INFO {Sophos.Management.Services.Sddma.VersionRolloverService.PerformVersionRollover} ==> Version rollover service finished
2017-08-15 12:16:16,362 [17] INFO {Sophos.Management.Services.Sddma.VersionRolloverService.PerformPredefinedVersionRollover} ==> Predefined subscription Version rollover service invoked
2017-08-15 12:16:16,362 [17] INFO {Sophos.Management.Services.Sddma.VersionRolloverService.PerformPredefinedVersionRollover} ==> Updating subscription configuration for all servers.
2017-08-15 12:16:16,940 [17] INFO {Sophos.Management.Services.Sddma.VersionRolloverService.PerformPredefinedVersionRollover} ==> Sending subscription configuration to 1 server(s).
2017-08-15 12:16:17,096 [17] INFO {Sophos.Management.Services.Messaging.Handler.receiver_OnApplyPolicy} ==> Received "apply policy" from Sophos.Management.Services.Sddma.MessageReceiver details Sophos.Management.Services.Sddma.MessageReceiver
2017-08-15 12:16:17,127 [17] INFO {Sophos.Management.Services.Com.Messaging.handler_SendDataPacket} ==> Marshaling a ApplyPolicy message for SDDM on Sophos.Management.Endpoint data size 4
2017-08-15 12:16:17,127 [17] INFO {Sophos.Management.Services.Com.Messaging.get_Sink} ==> Message Sink is ready.
2017-08-15 12:16:17,143 [17] INFO {Sophos.Management.Services.Sddma.VersionRolloverService.PerformPredefinedVersionRollover} ==> Predefined subscription Version rollover service finished
2017-08-15 12:16:17,143 [17] INFO {Sophos.Management.Services.Sddma.SddmaService.EnsureDefaultSourceLocationExists} ==> The default source location SOPHOS exists.
2017-08-15 12:16:17,143 [17] INFO {Sophos.Management.Services.Sddma.SddmaService.EnsureSelfUpdateLocationExists} ==> The self-update target location C:\ProgramData\Sophos\Sophos Endpoint Management\5.2.1\Updates\Secure exists.
2017-08-15 12:16:17,548 [17] INFO {Sophos.Management.Services.Sddma.SddmaService.EnsurePredefinedSubscriptionsAddressIsPublished} ==> Set predefined subscriptions data download folder at the registry location HKLM\SOFTWARE\Sophos\EE\Products to C:\ProgramData\Sophos\Sophos Endpoint Management\5.2.1\Updates\Secure.
2017-08-15 12:16:17,626 [17] INFO {Sophos.Management.Services.Sddma.ServerDataMonitor.GetData} ==> Received data request for 1 servers.
2017-08-15 12:16:17,626 [17] INFO {Sophos.Management.Services.Sddma.AuthoritativeServerSelector.SelectCandidateAuthoritativeServer} ==> Selecting the server 'HCMTNDC02' with the endpoint address 'Router$HCMTNDC02' as the authoritative server.
2017-08-15 13:32:31,787 [174] INFO {Sophos.Management.HMPA.HMPAPolicy..ctor} ==> HMPA policy constructed with name {F7B644B0-FFEE-4544-A894-449A16E01624}.
2017-08-15 15:32:32,102 [25] INFO {Sophos.Management.HMPA.HMPAPolicy..ctor} ==> HMPA policy constructed with name {F7B644B0-FFEE-4544-A894-449A16E01624}.
2017-08-15 17:32:32,399 [37] INFO {Sophos.Management.HMPA.HMPAPolicy..ctor} ==> HMPA policy constructed with name {F7B644B0-FFEE-4544-A894-449A16E01624}.
2017-08-15 19:32:32,727 [213] INFO {Sophos.Management.HMPA.HMPAPolicy..ctor} ==> HMPA policy constructed with name {F7B644B0-FFEE-4544-A894-449A16E01624}.
2017-08-15 21:32:33,041 [51] INFO {Sophos.Management.HMPA.HMPAPolicy..ctor} ==> HMPA policy constructed with name {F7B644B0-FFEE-4544-A894-449A16E01624}.
2017-08-15 23:32:33,334 [97] INFO {Sophos.Management.HMPA.HMPAPolicy..ctor} ==> HMPA policy constructed with name {F7B644B0-FFEE-4544-A894-449A16E01624}.
2017-08-16 01:32:33,654 [153] INFO {Sophos.Management.HMPA.HMPAPolicy..ctor} ==> HMPA policy constructed with name {F7B644B0-FFEE-4544-A894-449A16E01624}.
2017-08-16 03:32:33,956 [21] INFO {Sophos.Management.HMPA.HMPAPolicy..ctor} ==> HMPA policy constructed with name {F7B644B0-FFEE-4544-A894-449A16E01624}.

Of the 5 IPs on the server, there are 3 169 addresses in the router and 2 other IPs. Of those 2 IPs, is the server managing computers via both those interfaces or could all the clients you're managing just use one?

If the IP is static you could explicitly define it by editing:

"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SophosMessageRouter" changing the value for 'ImagePath' from (for example):

"C:\Program Files\Sophos\Remote Management System\RouterNT.exe" -service -name Router -ORBListenEndpoints iiop://:8193/ssl_port=8194

to

"C:\Program Files\Sophos\Remote Management System\RouterNT.exe" -service -name Router -ORBListenEndpoints iiop://192.168.15.1:8193/ssl_port=8194

The IP address should be the one that the clients can connect to.

Then Edit "HKEY_LOCAL_MACHINE\SOFTWARE\sophos\Messaging System\Router", changing the value for ''ServiceArgs" from (for example):

"-ORBListenEndpoints iiop://:8193/ssl_port=8194"

to

"-ORBListenEndpoints iiop://192.168.15.1:8193/ssl_port=8194"
Again the IP address should be the one that the clients can connect to and the same as above.

If you restart the Sophos Message Router service then and check the IOR it will be shorter and only have reference to the one IP.  This way, after clients read this IOR string they will come straight back to the right IP and port 8194.

Once done.  I would be interested to see some logs of the messaging system.

Restart the Sophos Router and Agent service on a client, wait 1 minute (to guarantee a status message has been generated). Can we see the new Agent and Router log file that has been created?

Hi Jak,

Thanks for the help... after looking at your suggestions, I looked on my Sophos server, and I don't have this in the registry

 Edit "HKEY_LOCAL_MACHINE\SOFTWARE\sophos\Messaging System\Router"

I only have ManagementServer, Patch and ServerSecurity under Sophos.

Sorry, those paths should be under:
hklm\software\wow6432node\sophos\....
...as they are all 32-bit applications.

You're seeing the 64-bit components under the 'other' Sophos key.

Regards,

Jak

Hi Jak,

After doing what you suggested, now my Update Manager finally reported and it's saying its up to date.

I checked the Sophos-Management-Services.log, and it seems to be back as it was before, the same pattern i mean. :-)

That seemed to have solved the issue... but how did that issue start i'm wondering...

Thanks for all the Help!!!!

Glad it's sorted.

I assume those 169 addresses have always been there and hence the IOR string always had the 5 profiles?

Also, in the event log of the server, or the SUM, etc.. at the same as the time was frozen.  Did the RMS package get installed, there should be an event from MSI?  If so, what version was it?