SSP.exe creating lots of traffic

Hi,

I've recently noticed these too since upgrading to 10.6.3 on 12 April. FW logs were being flooded with these dropped connection attempts.

I logged a call and was told to disable the "Sophos System Protection Service" on a machine to see if it stopped, it did. I then received the reply at the end of this post. Just waiting to hear what the actual risks are of not allowing this traffic before I make the decision.

I also got the following information about ssp.exe:
Sophos System Protection, a new component installed in 10.6.3) uses HTTPS to perform lookups to Amazon-hosted Web servers in order to determine if processes that are connecting to external IPs/URLs/etc. without using a browser are talking to Command and Control servers. If they are, the process is quarantined and if need be, terminated. You can see the FAQ for Malicious Traffic Detection here: https://www.sophos.com/en-us/support/knowledgebase/121607.aspx
 

"Hello,

Since these machines do not have direct access to the internet, I would recommend disabling the Sophos System Protection service across your environment, as it doesn't seem like this feature will be usable unless you make some significant changes to your network.

If in the future you do want to enable this feature, you'll have to whitelist the domains referenced in here:

https://www.sophos.com/fr-fr/support/knowledgebase/117936.aspx

You'll also have to the Whitelist the Sophos Live Protect and Sophos System Protection Processes. 

Let me know if you have any further questions! 

Regards,"

On these computers without internet access, what about adding the addresses to the hosts file?

C:\ProgramData\Sophos\Sophos System Protection\Config\SXA.conf references:

https://4.sophosxl.net/lookup

C:\ProgramData\Sophos\Sophos System Protection\Config\FBA.conf references:

https://ssp.feedback.sophos.com/ssp/v1/

so "C:\Windows\System32\drivers\etc\hosts" could be updated with:

127.0.0.1  4.sophosxl.net

127.0.0.1  ssp.feedback.sophos.com

Regards,

Jak

Hi Craig,

Thanks for the information. I have a case open with Support at the moment, opened on 28th April and finally got a response on 2nd May and still waiting for another response despite this being a fairly urgent issue. The SSP.exe service is being blamed for a major slowdown on one of our firewalls because our Infrastructure Team are seeing massive amounts of traffic (think millions of hits over a few hours) from that service on the Firewall logs.

I have been told the following:

• The SSP should establish a connection to 4.sophosxl.net on startup and then repeat that connection every 30 minutes, unless it can't establish a connection and then it will retry each of the connection methods every 30 seconds?
• It will also connect to ssp.feedback.sophos.com at startup and will then connect again once per week, unless it can't establish a connection and then it will retry each of the connection methods every 30 seconds?

Is this correct?

• If the above is true, is there a way for us to change how often the service connects and the connection method it should use?
• Can you confirm that the connection will work through an authenticated proxy so long as HTTPS traffic is allowed?
• Can you explain why a machine with Sophos 10.3.15 has SSP logs on it if SSP is a feature of 10.6? I have actually found machines with SSP logs going back to February 2016, well before 10.6 was available.
• Have any Sophos customers experienced problems with connection speeds through their firewalls or millions of connection attempts by SSP.exe since the update to 10.6?

And the most important question:

• How would we disable this service from the Enterprise Console? What feature do we need to turn off in the "Antivirus and HIPS" policy?

SSP, just to be clear:

SSP performs SXL4 online queries on behalf of the Malicious Traffic Detection and File Download Reputation features. The queries are used to determine the reputation of a file or URL. The data sent includes file names, file hashes, URLs. The SXL4 queries are done over HTTPS connections to the Sophos SXL4 server.

On startup, SSP establishes an HTTPS connection to 4.sophosxl.net, which is the Sophos SXL4 server. It does that so that the proxy is available when a SXL4 query needs to be performed. To determine a proxy, it attempts to connect to the server using three methods in parallel: direct connection (no proxy), auto proxy detection (WPAD) and system default proxy (e.g. configured with netsh), and it chooses the first one that succeeds. **We have this KBA that might be useful in this case (Ingnore the fact that it's MCS mentioned) -https://www.sophos.com/support/knowledgebase/119263.aspx

On startup, SSP also establishes a connection to ssp.feedback.sophos.com, which is the feedback server used for telemetry. For that it uses the same three methods of determining a proxy (direct connection, auto proxy detection and system default proxy). .

If a connection to either of these servers cannot be established, SSP will retry periodically every 30 seconds. This can happen for example if no Internet connection is available, or if a proxy/firewall at the edge of the network blocks HTTPS traffic to these servers.

To answer you questions (as best as I can).

Can you explain why a machine with Sophos 10.3.15 has SSP logs on it if SSP is a feature of 10.6? I have actually found machines with SSP logs going back to February 2016, well before 10.6 was available.

Without seeing the full sophos logs or even just the alupdate logs, I can't tell you exactly. But as a guess  maybe the machine was on a preview subscription and it was changed back? You will have found mentions of SSP in alupdate.log from before February, alupdate was aware of the component for a while and would look in the CID for SSP even before release, you would see this in the logs. If you really want an answer, PM me and I can take a look at your logs then we'll know.

Have any Sophos customers experienced problems with connection speeds through their firewalls or millions of connection attempts by SSP.exe since the update to 10.6?

Live protection will generate traffic, for best possible protection it should be on.We haven't seen a lot of cases for this. Allowing access out will alleviate some of the symptoms, for sure.

How would we disable this service from the Enterprise Console? What feature do we need to turn off in the "Antivirus and HIPS" policy?

You can't disable SSP from SEC. You could disable the service via GPO, but I absolutely wouldn't recommend it. Disabling the SSP service would prevent Malicious Traffic Detection, and File Download Reputation features from working.

Hi,

Thanks for the detailed reply. I've bypassed authentication on the proxy for 4.sophosxl.net and  ssp.feedback.sophos.com, which I'm happy to do as it's likely only Sophos application traffic communicating with those domains.

However, the traffic I am seeing blocked at the firewall is all to AmazonWS and this traffic stops instantly as soon as the "Sophos System Protection Service" is stopped and disabled on the monitored system.

Are we expected to bypass proxy authentication for AWS as well?

No, our products will be connecting to the domains listed, not directly to Amazon EC host-names. 

What you might be seeing is your Firewall inspecting the connection, seeing the IP the client is connecting to, performing a reverse lookup and the reverse lookup giving an EC compute hostname?

Similar to:

Hi,

Sure. Will PM you now.

Paul

Hi Craig,

Thank you very much for all of that information. The query about disabling it has been born from the SSP service being blamed for causing performance issues through our Firewall so we wanted to be ready to disable it if necessary. We fully understand why it is needed and don't want to disable it but we were close to being asked to.

I would just like to mention that we are also seeing the SSP service contacting AmazonWS addresses in our FW logs so would also be interested in anything you can determine from Paul's FW logs.

Thanks again

Hi Craig,

To return to the 4.sophosxl.net address returning as an Amazon AWS address, I get the following from an nslookup on our web filter:

"DNS Lookup for 4.sophosxl.net (4.sophosxl.net) returns:    54.246.172.45    52.50.177.117    52.51.158.42    52.30.113.180    52.30.190.178    52.48.62.119    54.76.67.12    52.18.100.124"

When I use MXToolbox.com to reverse lookup all of those IP addresses, they all come back as an amazonaws.com address.

Similarly when I complete a DNS Lookup on MXtoolbox.com on "4.sophosxl.net" it returns completely different IP addresses:

Is this all correct?

Thanks

Martin

That's all correct, it's a massive load balanced system, so it will return multiple AWS addresses.

Best,

Craig

Hi Craig,

Sorry I thought you had said that these weren't actually Amazon addresses previously. So are you confirming that the sophosxl.net service is a load balanced system hosted on Amazon AWS servers?

Is there a particular range or list of IP addresses that Sophos use for this? We have found that some of them are blocked by our web filtering software as it thinks they are a "peer-to-peer" connection.

Thanks

Martin

I am slowly getting angry.. another two days without any helpful update on this bug.

I this the way Sophos cares for custumers? Especially in this case where the bug ist caused by a sophos antivirus product update

in combination with an other sophos product (UTM) . Our companys suffers from an absolutely wrecked proxy, an there is literally no

useful help... seriusly WTF? 

I am slowly getting angry.. another two days without any helpful update on this bug.

I this the way Sophos cares for custumers? Especially in this case where the bug ist caused by a sophos antivirus product update

in combination with an other sophos product (UTM) . Our companys suffers from an absolutely wrecked proxy, an there is literally no

useful help... seriusly WTF? 

Hi, I PM'd you last week - I haven't had a reply.

How is the state of investigation? Can you confirm that this is a Proxy bug?

Do you have found  a solution for your environment? We are in the similar situation (the proxy server with authentication) and going to  disable ssp.exe via GPO

The only real solution would be a hotfix by Sophps for this  bug. But Sophos support is quite horrible as it seems they dont really care. We have been told to disable SSP or to rollback to an older Sophos release ^^.  What helped a litte was to redirect the traffic to the WAF by using a WPAD Proxy config. Seems to bee less cpu sonsuming if the WAF recieves the packages, but still not a suitabble solution in any way.

What I don't understand is, why isn't Sophos at least using the systems proxy settings?
Sophos should implement the whole thing into the enterprise console and make it downloadable like the CIDs.
Or better the message router forwards it to the server and the server asks for the information via proxy and send the reply back to the client. 
Requesting direct internet Access for something like this just plain stupid and I don't think that is suitable for ANY business customers.
Blocking ssp.exe via GPO is another thing. Manually manipulating the services of a security suite?
THIS IS SOMETHING ONE SHOULD MANAGE THROUGH A GUI AKA (SEC).

Hi,

I too fail to understand why any company selling to business customers would expect them to punch holes in their perimeter security,or place major additional load on firewalls/UTMs by using dynamic objects just to make their products work. I really hope Sophos does not join the likes of Adobe in this respect. By all means move services to the cloud, but make sure you support secure access to them. Rant over!

I'd now like to update the community on the latest from my support case on this issue:

Hello Paul,

This case has been escalated to myself for further investigation.

This is a known issue currently with development and will be fixed in an update later in the year (after Sept).   Sophos suggest the following to reduce/stop the traffic hitting the firewall.
 
  1. Open up the firewall to allow connections to 4.sophosxl.net
  2. Add a local DNS record to deny access to 4.sophosxl.net and ssp.feedback.sophos.com
  3. Add a local host entry for 4.sophosxl.net and ssp.feedback.sophos.com to point to 127.0.0.1    (this would need to be on every endpoint)

As long as the SXL4 request receives a valid response (a deny or block will count as a valid response) it will stop sending the request every 30 seconds i would recommend as a short term fix that you look at setting a DNS record to allow these through

I then asked if the fix would make the SSP.exe traffic use the locally configured NTLM proxy and got the following reply:

 I am afraid I am not sure how this fix is going to be implemented as of yet however I believe they are going to force the policy to apply whether the sxl lookup goes through or not. Once its succeeds on this lookup it should  continue using NTLM its just the initial connection which appears to be the issue.

Just a shame that we cannot make use of the new security features of Sophos Endpoint and will rely on the old IDE files for now.

The Endpoint "Web Protection/Web Control" feature has always adhered to the client proxy settings supporting NTLM along the way, so they can do it. Hopefully after the fix, the new SSP features will too.

Whats the state of investigation? Are we going to get some kind of  bugfix in near future? Or did sophos decide to ignore this case ?