Random Port Exhaustion on Sophos Endpoint Security and Control clients version 10.6.3 and 10.6.4 on Windows 10 Enterprise workstations.

Hi All, 

Good morning! I have deployed on my environment several Windows 10 Enterprise Edition with the Sophos Endpoint Security and Control Security 10.6.3 and 10.6.4 versions installed and I'm facing randomly that suddenly the users are not able to browse the internet. Doing some troubleshooting and i found that I have from the Windows Logs several events 4227 and 4231, doing some monitoring(NETSTAT -anoq -p tcp) about the ports utilization i found that this file C:\Program Files (x86)\Common Files\Sophos\Web Intelligence\swi_fc.exe, is opening several ephemeral ports connections.

The only solution that we found so far is to have so far is to restart the machines, but the users are not quite happy.

Based on this, there's something else that I can do? 

Really appreciate your help and comments, 

  • We're also seeing similar symptoms to that which you have described. Did you find a resolution for this?

    Thanks, Tom

  • I believe this is fixed in swi_fc.exe in a later version.  What is the version of swi_fc.exe that is running?  The file is typically here:
    "C:\Program Files (x86)\Common Files\Sophos\Web Intelligence\swi_fc.exe".

    I assume you're not Central but SEC managed?

    If you first run (admin prompt):
    tasklist | find "swi_fc.exe"

    To find the pid of swi_fc.exe.  Example output:

    swi_fc.exe 4196 Services 0 16,164 K

    In this case 4196, then run:
    netstat -anoq | find "BOUND" | find "4196" /c
    This will count all the connections attributed to swi_fc.exe that are in the bound state.

    When all the browser processes (chrome.exe, firefox.exe, iexplore.exe, etc..) are closed this should be 0 as swi_fc.exe shouldn't be proxying anything just listening.  

    Is this what you see?  I assume when you have the issue, this count goes up into the thousands?

    If so, this should be fixed in a future swi_fc.exe.  The bound ports accumulate for connections that are made by swi_fc.exe that get ACCESS_DENIED, ERROR_SEM_TIMEOUT for example.

    For example, if I create a local Windows Firewall rule to block an outgoing TCP connection to a specific address and then visit this address, swi_fc.exe would get ACCESS_DENIED back from Get­Queued­Completion­Status API.  This would cause 1 BOUND socked. 

    If I have an upstream device which is blocking an address and swi_fc.exe makes a connection it might get back ERROR_SEM_TIMEOUT.  For each of these, I suspect it's keeping the connection in the BOUND state.

    Given this, do you have a device upstream blocking sites being accessed such that nothing is returned, i.e. not a block page?  More of a stealthy drop of the connection attempt?

    If you try the Sophos Central client (can create a trial via cloud.sophos.com and deploy to a test client), you could copy the swi_fc.exe from that version to "C:\Program Files (x86)\Common Files\Sophos\Web Intelligence\swi_fc.exe". To replace it you would need to stop all the browser processes and then stop the Sophos Web Filter service before restarting the Sophos Web Filter service which will start the new swi_fc.exe process.

    Hope it helps.

    Regards,
    Jak

  • In reply to jak:

    Amazing, thank you Jak! I will check out what you have suggested next time it occurs to one of our users. Do you know what version of "swi_fc.exe" this should be fixed in for an on-prem solution, or should I raise a ticket with the support desk?

    Although I cannot prove it, my suspicion is that this might be linked to a SCF issue we've started having on Win10 where localhost traffic is not being detected correctly as localhost and is then subsequently being blocked inbound by the SCF. 

    Thanks again for your help!

    Tom 

  • I am having the exact same issue as described here, i have found a work around of disabling the web intelligence and web filter services. When the issue occurs i can simply stop these services and kill the process to prevent needing to reboot. 

    I am having this issue on server 2016 RDS Session Hosts.

  • In reply to Michael Szafran:

    What version of SAV do you have installed?

    10.8.3?
    10.8.4?

    or something else?

    Regards,

    Jak

  • In reply to jak:

    Hi Jak,

     

    I am John. I have taken over this issue from Michael. We have SAV 10.8.2.

     

    Is there a new swi_fc.exe available that can resolve this port exhaustion issue?

     

    If not, disabling the Web Filter service is the current effective workaround. However, it seems to re-enable and start itsef possibly after a Sophos uodate etc. Not really sure why.

    Hence, is there a way not to deploy the Web Filter service altogether? It appears to be part of the Anti-Virus package and not a separate application we could exclude from deployment. I also could not find any MSI switches that could be applied to the ClientPackage.msi application to not istall the Web Filter this way as well.

    I beleive whenever the Sophos Managemnt Service deploys updates to the servers, it re-enables and re-starts the disable Web Filter service - this is not desirable.

     

    Regards,

    John

  • In reply to Michael Szafran:

    Hello John,

    not a long-term solution but if you disable Web Protection (in Central Realtime scanning (Internet)) and Web Control swi_fc.exe shouldn't interfere.

    Christian

  • In reply to QC:

    Christian is right, the issue which is now fixed as I understand it is to do with swi_fc.exe (the local web proxy) making outbound connections to addresses that aren't avialable so if you disable:

    Web Control in the linked policy

    AND

    The 2 Web Protection features:

    • Block Access To malicious Websites
    • Content Scanning

    https://community.sophos.com/kb/en-us/123446

    Then browsers do not send traffic to swi_fc.exe so the issue can't occur.

    Regards,
    Jak

  • In reply to jak:

    I've seen this issue happening in Windows Server 2016 as well. A Technical Support person suggested that the loopback address (127.0.0.1) is blocked in the Web Control Policy (under Website Exceptions)

    This seems to have worked - we did not observe any TCP port exhaustion issues in the last few days. So in my case I needn't block Web Protection features in the AV & HIPS policy. Might worth trying as an alternative.