Thread Info
  • State Suggested Answer
  • Locked Locked
  • Replies 8 replies
  • Answers 1 answer
  • Subscribers 13 subscribers
  • Views 4213 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Network interruptions: Installing Sophos updates component NTP64, MCS Client, SAU

LHerzog

Hi,

some colleagues reported network / soft-phone interruptions during the last days.

Today i picked one computer and found a lost heartbeat at 14:42 - the time where his phone call was interrupted.

I found out, hat SED64 and NTP64 had been updated on the client:

--------

SophosUpdate.log

2022-01-24T13:41:27.083Z [18548:18984] I Beginning decode
2022-01-24T13:41:35.839Z [18548:18984] I [SUL-Log] [I46431] Decoding file C:/ProgramData/Sophos/AutoUpdate/Cache/decoded/ntp64/2022012401.ips
2022-01-24T13:41:36.447Z [18548:18984] I [SUL-Log] [I46431] Decoding file C:/ProgramData/Sophos/AutoUpdate/Cache/decoded/sed64/Config/BehavioralRules/behave.dec
2022-01-24T13:41:38.372Z [18548:18984] I [SUL-Log] [I39856] Purging file C:\ProgramData\Sophos\AutoUpdate\Cache\decoded/ntp64/2022011801.ips

...

2022-01-24T13:42:04.390Z [18548:18984] I Installing component 1129226C-32AB-4B72-85E1-A9CC8DFBC859 (sed64) 3.0.1.873

...

2022-01-24T13:42:19.962Z [18548:18984] I Installing component NTP64 (ntp64) 1.15.783.0

-

SntpService.log:

2022-01-24T05:36:20.222Z [ 4724: 8244] A Starting a DetectionReporter thread: 8244
2022-01-24T13:42:25.545Z [ 4724: 4728] A The service has stopped.
2022-01-24T13:42:27.672Z [21940:22236] A Starting version 1.15.783.0 of the Sophos Network Threat Protection service.
2022-01-24T13:42:33.501Z [21940:22740] A Starting a DetectionReporter thread: 22740

---------

To me it looks like there is a new IPS policy pulled by Sophos Update Service: ntp64/2022012401.ips and does it needs to re-install NTP service to apply new IPS policies?

Is my thinking correct and is that the normal behaviour?

Do you have on your screen, that this (NTP restard) causes a Heartbeat failure for synchronized security with Sophos Firewall? Isn't it possible to update policies in a better, non-interruptive approach?

I mean, updating IPS patterns on XG and UTM  is also causing this kind of trouble especially on smaller appliances but it's not the smartest way to drop connections for updates.



This thread was automatically locked due to age.
Parents
  • 0

    Hello LHerzog,

    Thanks for reaching out to us.

    You are correct, the IPS updates will trigger a re-install of the Sophos Network Threat Protection component.

    As an immediate work-around, you could apply an "Updating Policy" so that the devices will only update at the given time. You could also use "Controlled Updates" to accomplish this. 

    I will inquire with our team to find out if there are plans to change this and let you know.

    Kushal Lakhan
    Team Lead, Global Community Support
    Connect with Sophos Support, get alerted, and be informed.
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  - Visit Sophos Techvids
  • 0 in reply to Qoosh

    Thanks for confirming the behaviour. Looking forward to your update about future plans.

  • 0 in reply to LHerzog

    yesterday NTP64, today Auto update XG (whatever this should be on a client)

    Sophos AutoUpdate XG. Product version: 6.12.86

    MCS Agent and MCS Client restarted. Causing Synced Heartbeat reset.

    If we put our clients into controlled: aka delayed updates is a bad option because it only allows one day per week to update.

    Why not allowing als many days as you like? I (probably manu admins) just don't want updates to be installed in regular work hours.

    I think MTR team also will not like this setting and it will be marked yellow or red in their next long excel report about our Central settings.

  • 0 in reply to LHerzog

    Even Heartbeat has changed today!

    .

    2022-01-25T07:44:49.391Z [ 4516: 5148] A Starting Heartbeat version 1.15.783.0
    2022-01-25T13:52:47.553Z [17084:15976]A Starting Heartbeat version 1.15.827.0
    .

    I wonder how it is possible that Sophos is sending out multiple updates a week, on different days, not even combined as one update, that are causing network interruptions if you use Sophos Hardware and Software the way you say we should do (synced security, heartbeat...). I'm really annoyed after a long day with many user complaints.

  • 0 in reply to LHerzog

    Network drops should not be occurring when an IPS update occurs.

    The feedback I have received is that when IPS rules are updated, there will be a re-build process for SophosSnort. This is similar to a SAV engine update. 

    The filter driver responsible for intercepting traffic, however, should not be affected. This filter driver is only reloaded when a major update occurs. This means that some traffic related to web browsers may be affected by the update, but something like a VOIP system should not be affected.

    Some steps to determine if the IPS update is what caused the issue is as follows.
    - Establish a phone call for troubleshooting purposes
    - Disable Tamper Protection 
    - Delete/rename the files ending in ".ips" from the following directory: C:\ProgramData\Sophos\Sophos Network Threat Protection\IPS
    - Stop the "Sophos AutoUpdate Service"
    - Delete "SophosUpdateStatus.xml" in: C:\ProgramData\Sophos\AutoUpdate\data\status\
    - Start the "Sophos AutoUpdate Service"
    - Trigger an update 

    This process will replicate the SophosSnort/IPS rebuild process. 

    The service stoppage/restart is something that will be looked into for future releases. I will reaching out to you via PM to request additional information for troubleshooting. 

    Kushal Lakhan
    Team Lead, Global Community Support
    Connect with Sophos Support, get alerted, and be informed.
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  - Visit Sophos Techvids
  • 0 in reply to Qoosh

    Hi,

    i followed your guideline above and it resulted in a reinstall of most Agent components, including NTP64

    	Line 18895: 2022-01-27T13:12:36.914Z [14824:15892] I Installing products.
	Line 18896: 2022-01-27T13:12:36.918Z [14824:15892] I Installing component SDU (SDU) 6.11.234
	Line 18906: 2022-01-27T13:12:39.094Z [14824:15892] I Installing component 243DECCD-8080-410D-A45F-77F2182715EE (UNINSTALLER64) 1.12.133.133
	Line 18919: 2022-01-27T13:12:39.471Z [14824:15892] I Installing component 1129226C-32AB-4B72-85E1-A9CC8DFBC859 (SED64) 3.0.1.878
	Line 18930: 2022-01-27T13:12:46.303Z [14824:15892] I Installing component 3799FB3E-808A-4F7D-AC6A-0C74F931C386 (MCS) 4.15.79.0
	Line 18942: 2022-01-27T13:12:51.687Z [14824:15892] I Installing component 0253775E-970D-4876-959C-21B422420E5A (SSE64) 1.8.8.1
	Line 18952: 2022-01-27T13:13:00.563Z [14824:15892] I Installing component 591706A7-9603-4255-A65F-EA49BB11E8AC (SFS64) 1.9.16.3
	Line 18962: 2022-01-27T13:13:06.939Z [14824:15892] I Installing component 3D8DC0A9-7F42-4CD5-AA7B-CF29296E7789 (SOPHOSCLEANM64) 3.9.14.1
	Line 18972: 2022-01-27T13:13:09.037Z [14824:15892] I Installing component 3CE954A1-0F41-4D9B-B2F0-58AA75334DFD (SHS) 2.8.130.0
	Line 18982: 2022-01-27T13:13:11.943Z [14824:15892] I Installing component 5CD1A7B6-812E-47A1-A986-3A6D5D5C19F5 (UI64) 2.4.230.0
	Line 18992: 2022-01-27T13:13:33.820Z [14824:15892] I Installing component 642A6FD9-A9D6-482D-BD8C-46661F241A0E (AMSI64) 1.8.59
	Line 19002: 2022-01-27T13:13:34.629Z [14824:15892] I Installing component 70FDD40E-986A-44E5-9620-2B894A06702A (SME64) 1.8.7.1
	Line 19015: 2022-01-27T13:13:38.236Z [14824:15892] I Installing component 7F682906-6E49-481B-89C5-2DCA36720F4F (ESH64) 3.1.88.0
	Line 19026: 2022-01-27T13:13:38.811Z [14824:15892] I Installing component BA3387BB-AE88-4403-A36D-F8C0E0B6AEB2 (LIVETERMINAL64) 1.4.80.0
	Line 19036: 2022-01-27T13:13:39.448Z [14824:15892] I Installing component CD297D6B-58A5-474F-8A0D-0A15803B8B50 (EFW64) 2.0.20.20
	Line 19046: 2022-01-27T13:13:41.626Z [14824:15892] I Installing component LiveQuery64 (LiveQuery64) 3.4.0.320
	Line 19057: 2022-01-27T13:13:52.741Z [14824:15892] I Installing component MTR64 (MTR64) 2.3.0.68
	Line 19067: 2022-01-27T13:13:56.409Z [14824:15892] I Installing component NTP64 (NTP64) 1.15.827.0
	Line 19077: 2022-01-27T13:14:06.810Z [14824:15892] I Installing component 244E68BF-E1BB-4A6B-AC18-A492DE0134C0 (HMPA64) 3.8.3.812
	Line 19087: 2022-01-27T13:14:42.171Z [14824:15892] I Installing component 1FE3E7DF-EFFA-408A-A1B0-89F15BA61F31 (SAUXG) 6.12.86

    NTP restarted internally but the windows service SntpService did not restart according to the windows eventlog

    2022-01-27T13:14:05.158Z [ 5576:18116] A Starting version 1.15.827.0 of the Sophos Network Threat Protection service.

    2022-01-27T13:14:08.651Z [ 5576: 3840] A Setting IPS health status to GREEN

    2022-01-27T13:14:02.847Z [ 4464: 4784] A Stopped Heartbeat
    2022-01-27T13:14:02.848Z [ 4464: 4784] A ----------------------------------------------------------------------------------------------------
    2022-01-27T13:14:05.369Z [ 5576: 3412] A ----------------------------------------------------------------------------------------------------
    2022-01-27T13:14:05.370Z [ 5576: 3412] A Starting Heartbeat version 1.15.827.0
    2022-01-27T13:14:05.372Z [ 5576: 3412] A ----------------------------------------------------------------------------------------------------
    2022-01-27T13:14:05.446Z [ 5576:10252] A Connection succeeded.

    Issues:

    lost heartbeat, re-authenticated to firewall.

    [2022-01-27 13:14:03.060Z] WARN HBSession.cpp[26955]:344 bufferDisconnectEvent - Incoming connection from 172.xxx.xxx.xx0 failed. SSL error:
[2022-01-27 13:14:05.455Z] INFO HBSession.cpp[26955]:504 logNewSession - New Session: [172.xxx.xxx.xx0]:20986 connected
[2022-01-27 13:14:05.515Z] INFO ModuleSacFirst.cpp[26955]:95 sendEacMessage - send EacSwitchRequest to endpoint (IP=172.xxx.xxx.xx0)
[2022-01-27 13:14:05.519Z] INFO EpStateListBroker.cpp[26955]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: 84c07cdd-xxxx-xxxx-bf06-xxxxxxxxxx8(172.xxx.xxx.xx0)
[2022-01-27 13:14:10.961Z] INFO ModuleStatus.cpp[26955]:137 processMessageStatus - Status request received from endpoint: 84c07cdd-xxxx-xxxx-bf06-xxxxxxxxxx8 (172.xxx.xxx.xx0) health: 1

    ongoing phone call got disconnected

    In an open firefox browser session, I cannot  open any new website until I close and restart firefox. Known issue here for us when Sophos Services restart in the background.

    error: Error code: SEC_ERROR_BAD_SIGNATURE

    So it could be reproduced.

Reply
  • 0 in reply to Qoosh

    Hi,

    i followed your guideline above and it resulted in a reinstall of most Agent components, including NTP64

    	Line 18895: 2022-01-27T13:12:36.914Z [14824:15892] I Installing products.
	Line 18896: 2022-01-27T13:12:36.918Z [14824:15892] I Installing component SDU (SDU) 6.11.234
	Line 18906: 2022-01-27T13:12:39.094Z [14824:15892] I Installing component 243DECCD-8080-410D-A45F-77F2182715EE (UNINSTALLER64) 1.12.133.133
	Line 18919: 2022-01-27T13:12:39.471Z [14824:15892] I Installing component 1129226C-32AB-4B72-85E1-A9CC8DFBC859 (SED64) 3.0.1.878
	Line 18930: 2022-01-27T13:12:46.303Z [14824:15892] I Installing component 3799FB3E-808A-4F7D-AC6A-0C74F931C386 (MCS) 4.15.79.0
	Line 18942: 2022-01-27T13:12:51.687Z [14824:15892] I Installing component 0253775E-970D-4876-959C-21B422420E5A (SSE64) 1.8.8.1
	Line 18952: 2022-01-27T13:13:00.563Z [14824:15892] I Installing component 591706A7-9603-4255-A65F-EA49BB11E8AC (SFS64) 1.9.16.3
	Line 18962: 2022-01-27T13:13:06.939Z [14824:15892] I Installing component 3D8DC0A9-7F42-4CD5-AA7B-CF29296E7789 (SOPHOSCLEANM64) 3.9.14.1
	Line 18972: 2022-01-27T13:13:09.037Z [14824:15892] I Installing component 3CE954A1-0F41-4D9B-B2F0-58AA75334DFD (SHS) 2.8.130.0
	Line 18982: 2022-01-27T13:13:11.943Z [14824:15892] I Installing component 5CD1A7B6-812E-47A1-A986-3A6D5D5C19F5 (UI64) 2.4.230.0
	Line 18992: 2022-01-27T13:13:33.820Z [14824:15892] I Installing component 642A6FD9-A9D6-482D-BD8C-46661F241A0E (AMSI64) 1.8.59
	Line 19002: 2022-01-27T13:13:34.629Z [14824:15892] I Installing component 70FDD40E-986A-44E5-9620-2B894A06702A (SME64) 1.8.7.1
	Line 19015: 2022-01-27T13:13:38.236Z [14824:15892] I Installing component 7F682906-6E49-481B-89C5-2DCA36720F4F (ESH64) 3.1.88.0
	Line 19026: 2022-01-27T13:13:38.811Z [14824:15892] I Installing component BA3387BB-AE88-4403-A36D-F8C0E0B6AEB2 (LIVETERMINAL64) 1.4.80.0
	Line 19036: 2022-01-27T13:13:39.448Z [14824:15892] I Installing component CD297D6B-58A5-474F-8A0D-0A15803B8B50 (EFW64) 2.0.20.20
	Line 19046: 2022-01-27T13:13:41.626Z [14824:15892] I Installing component LiveQuery64 (LiveQuery64) 3.4.0.320
	Line 19057: 2022-01-27T13:13:52.741Z [14824:15892] I Installing component MTR64 (MTR64) 2.3.0.68
	Line 19067: 2022-01-27T13:13:56.409Z [14824:15892] I Installing component NTP64 (NTP64) 1.15.827.0
	Line 19077: 2022-01-27T13:14:06.810Z [14824:15892] I Installing component 244E68BF-E1BB-4A6B-AC18-A492DE0134C0 (HMPA64) 3.8.3.812
	Line 19087: 2022-01-27T13:14:42.171Z [14824:15892] I Installing component 1FE3E7DF-EFFA-408A-A1B0-89F15BA61F31 (SAUXG) 6.12.86

    NTP restarted internally but the windows service SntpService did not restart according to the windows eventlog

    2022-01-27T13:14:05.158Z [ 5576:18116] A Starting version 1.15.827.0 of the Sophos Network Threat Protection service.

    2022-01-27T13:14:08.651Z [ 5576: 3840] A Setting IPS health status to GREEN

    2022-01-27T13:14:02.847Z [ 4464: 4784] A Stopped Heartbeat
    2022-01-27T13:14:02.848Z [ 4464: 4784] A ----------------------------------------------------------------------------------------------------
    2022-01-27T13:14:05.369Z [ 5576: 3412] A ----------------------------------------------------------------------------------------------------
    2022-01-27T13:14:05.370Z [ 5576: 3412] A Starting Heartbeat version 1.15.827.0
    2022-01-27T13:14:05.372Z [ 5576: 3412] A ----------------------------------------------------------------------------------------------------
    2022-01-27T13:14:05.446Z [ 5576:10252] A Connection succeeded.

    Issues:

    lost heartbeat, re-authenticated to firewall.

    [2022-01-27 13:14:03.060Z] WARN HBSession.cpp[26955]:344 bufferDisconnectEvent - Incoming connection from 172.xxx.xxx.xx0 failed. SSL error:
[2022-01-27 13:14:05.455Z] INFO HBSession.cpp[26955]:504 logNewSession - New Session: [172.xxx.xxx.xx0]:20986 connected
[2022-01-27 13:14:05.515Z] INFO ModuleSacFirst.cpp[26955]:95 sendEacMessage - send EacSwitchRequest to endpoint (IP=172.xxx.xxx.xx0)
[2022-01-27 13:14:05.519Z] INFO EpStateListBroker.cpp[26955]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: 84c07cdd-xxxx-xxxx-bf06-xxxxxxxxxx8(172.xxx.xxx.xx0)
[2022-01-27 13:14:10.961Z] INFO ModuleStatus.cpp[26955]:137 processMessageStatus - Status request received from endpoint: 84c07cdd-xxxx-xxxx-bf06-xxxxxxxxxx8 (172.xxx.xxx.xx0) health: 1

    ongoing phone call got disconnected

    In an open firefox browser session, I cannot  open any new website until I close and restart firefox. Known issue here for us when Sophos Services restart in the background.

    error: Error code: SEC_ERROR_BAD_SIGNATURE

    So it could be reproduced.

Children
  • 0 in reply to LHerzog

    I received word about users getting disconnected again today.

    A short check: Sophos Endpoint IPS received an update and sntp restarted.

    one example:

    2022-01-27T13:41:45.740Z [ 4880: 4884] A The service has stopped. 2022-01-27T13:41:48.078Z [ 4048:20952] A Starting version 1.15.827.0 of the Sophos Network Threat Protection service.


    The firewall log today is again flooded with user-logout followed by user-login some seconds later. That's the moment where they lost their connection for user authenticated XG rules.


    Who ever eventually feels responsible for Sophos Synchronized Security should review this! what a messed up software and deployment approach.

    You can't even control IPS updates with the scheduled updates feature in Sophos Central.... They are handled like normal pattern updates. Kicking the users ass out of our Sophos "Ecosystem"!

  • 0 in reply to LHerzog

    Powering Synchronized Security. Delivering real business impact.

    Sophos ACE enables Sophos products to share threat, health, and security information in real time and respond automatically to threats–we call it Synchronized Security. This approach elevates protection while slashing total cost of ownership (TCO).

    from www.sophos.com/.../adaptive-cybersecurity-ecosystem

    Business impact... You do.

Unfiltered HTML