This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Sophos Firewall WAF Policy Crashing System

Hello Sophos Community

Using the latest firmware as of today (SFOS 19.5.0 GA-Build197) on Sophos Firewall, installed as a virtual appliance in Proxmox 7.3-4.  It's a home license, on 4 virtual CPUs (host), and 6GB memory.  I'm using the official qcow2 images.

I am hosting a Wordpress blog behind it, using WAF, which I use mainly for sharing family pictures and videos.  Below is a screenshot of the policy.

When this policy is applied to the firewall rule, everything works fine until I attempt to upload a very large file to the blog.  Recently, I attempted to upload our family's Christmas morning video to my site, which was 1.3GB.  Not only did it fail, but the entire firewall crashed... and crashed hard.  The house went offline, DNS resolution went down, I couldn't connect to the firewall via the IP address, nothing.  It just died.  I had to go into Proxmox, kill the VM, then restart it.  Upon restart everything was fine.

If I remove the WAF policy, the file uploads.  If I enable the WAF policy and transfer the file, everything comes crashing down.  It's a very easily reproduced.  What could be causing this?



This thread was automatically locked due to age.
  • For what it's worth, I just tried with a 600 MB file, and the same result.  I can understand if a WAF rule would fail the request or result in an error... but I wouldn't expect the whole server to become unresponsive.

  • Hi,

    Deleted incorrect information.

    Ian

    XG115W - v20 GA - Home

    XG on VM 8 - v20 GA

    If a post solves your question please use the 'Verify Answer' button.

  • Basically the system should not crash based on a core dump of some kind of process. But in proxmox, do you have a console? Is there any kind of service still running? 

    __________________________________________________________________________________________________________________

  • I believe the console itself was completely frozen too, but I'll need to test that to answer conclusively.  I can probably do that tomorrow.  Reproducing this tonight would impact the other WAF (wife acceptance factor).

  • Thanks for your reply.  I often upload family videos greater than 10MB, and have not seen this limit before.  Do you have a link I could check out?

  • Hi John,

    To me it sounds like Wordpress is using some file transfer method that is not part of the core HTTP standard, e.g. something like WebDAV which WAF doesn't support. If that is the case, then that is known to cause increased memory usage in WAF, which can starve other processes on the system, leading to the crash you see.

    You can try to check if you can configure Wordpress to not use any non-standard protocol. If you can do this, that should fix the problem.

    Alternatively you can try to define an exception for the site path route the file is uploaded to and skip all CTF checks for that site path route. This is not guaranteed to work, but might reduce the memory footprint of WAF just enough to avoid a crash.

    Best regards,

    Attila Kovacs

  • If you can reproduce it, you could observe the memory footprint of the system to check, if the firewall is increasing. 

    __________________________________________________________________________________________________________________

  • I'm going to reproduce this in the next hour or so.  Is memory footprint the only thing you'd like me to keep an eye on?  Should I watch anything else?

  • I'll look into this as well, thank you for the suggestion.  Even if this ends up being the issue, I still wouldn't expect the system to flat out crash.  A more graceful failure would probably be better.  I'll report back.

  • Alright, here's what I saw on this test:

    • File is a 1.3gb MP4 video.  Using Wordpress's built-in media uploading feature.
    • After about 1 minute all connections went down and the firewall froze.
    • Below I was able to capture "top" numbers in the firewall before the SSH session failed/froze
    • I saw a spike in Proxmox for the VM's memory and CPU... but they don't strike me as problematic.
    • The CPU spike did coincide with the firewall becoming unavailable.
    • After another minute or two, the firewall recovered itself and everything came back to life (except for the SSH session).
    • The firewall's downtime coincided with the CPU spike.
    • The transfer is hanging indefinitely, and presumably will eventually fail.

    NOTE: If not explicitly approved by Sophos support, any modifications
    top - 11:33:07 up 22:21,  2 users,  load average: 4.52, 1.14, 0.43
    Tasks: 464 total,   3 running, 459 sleeping,   0 stopped,   2 zombie
    %Cpu0  :  13.1/48.8   62[|||||||||||||||||||||||||||||||||||||                       ]     %Cpu1  :  17.3/56.5   74[||||||||||||||||||||||||||||||||||||||||||||                ]
    %Cpu2  :  15.2/67.3   82[|||||||||||||||||||||||||||||||||||||||||||||||||           ]     %Cpu3  :  19.4/46.7   66[||||||||||||||||||||||||||||||||||||||||                    ]
    GiB Mem : 99.8/5.8      [                                                            ]
    GiB Swap: 74.9/4.0      [