Thread Info
  • State Verified Answer
  • Locked Locked
  • Replies 29 replies
  • Answers 1 answer
  • Subscribers 40 subscribers
  • Views 8237 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Sophos Firewall WAF Policy Crashing System

John Groller

Hello Sophos Community

Using the latest firmware as of today (SFOS 19.5.0 GA-Build197) on Sophos Firewall, installed as a virtual appliance in Proxmox 7.3-4.  It's a home license, on 4 virtual CPUs (host), and 6GB memory.  I'm using the official qcow2 images.

I am hosting a Wordpress blog behind it, using WAF, which I use mainly for sharing family pictures and videos.  Below is a screenshot of the policy.

When this policy is applied to the firewall rule, everything works fine until I attempt to upload a very large file to the blog.  Recently, I attempted to upload our family's Christmas morning video to my site, which was 1.3GB.  Not only did it fail, but the entire firewall crashed... and crashed hard.  The house went offline, DNS resolution went down, I couldn't connect to the firewall via the IP address, nothing.  It just died.  I had to go into Proxmox, kill the VM, then restart it.  Upon restart everything was fine.

If I remove the WAF policy, the file uploads.  If I enable the WAF policy and transfer the file, everything comes crashing down.  It's a very easily reproduced.  What could be causing this?



This thread was automatically locked due to age.
  • 0 in reply to John Groller

    Hello there,

    Thank you for contacting the Sophos Community.

    Did you try the suggestion Attila mentioned?

    What does the Sophos Firewall system graphs at around that time?

    Do you see anything under /var/cores?

    Regards,

     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support Videos | Product Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • 0 in reply to emmosophos

    I have not tried Attila's suggestion yet.  I work from home, so I can't be bringing my firewall down over and over :-).  I will report back when I know more, but on first glance, I'm not seeing an association between WebDAV and Wordpress for the media upload  capability.

    In the end it's less about the ability to upload the files, and more about the fact that the firewall crashes when I attempt to upload something which, perhaps, is unsupported.  A more graceful solution for that would definitely be appreciated... I could always upload stuff another way.  I'll report back as I learn more.

    As for /var/cores, I looked in there just now, during normal operation, and see nothing.

    SFVH_KV01_SFOS 19.5.0 GA-Build197# ls -l /var/cores
    SFVH_KV01_SFOS 19.5.0 GA-Build197#

    As for the Sophos Firewall system graphs, the test today was around 11:30.

  • 0 in reply to John Groller

    I have not been able to determine if Wordpress is using anything non-standard for these transfers... it doesn't seem like they are, but I don't know for sure.  I'm using the native media upload feature, no plugins or anything.

    Even so, from the tests I ran earlier today, nothing jumps out at me with regard to memory footprint or CPU cycles.  Everything appears to well within acceptable ranges during the crash.  I'll give another shot tomorrow while tailing the reverse proxy, and sys logs.

  • 0 in reply to John Groller

    Ok, I was able to capture this from the reverse proxy log the moment before the system crashed:

    [Tue Jan 31 05:23:03.054004 2023] [security2:error] [pid 22813:tid 139889534220032] [client 192.168.0.71:47856] [client 192.168.0.71] ModSecurity: Request body (Content-Length) is larger than the configured limit (1073741824).

    As this came from the Sophos logs, am I correct in understanding this configuration is in the firewall?  If so, is it adjustable?

  • 0 in reply to John Groller

    Yes, this is a logline from WAF. It basically means it received a request that is bigger than the 1GB limit allowed by ModSecurity. It is configurable, but 1GB is the maximum. It sounds like Wordpress is trying to transfer the whole file in a single HTTP request. However, this request should have simply been rejected by WAF with a 413 response and shouldn't cause a crash.

    Just guessing, but it might be that Wordpress tried a standard file upload in a single request first, but got rejected by WAF, which in turn triggered some non-standard chunked file upload from Wordpress causing the crash.

    It would be good to see a HAR capture from the client side to see the actual message flow between the browser and WAF, and also the full reverseproxy.log capturing all logs when the issue happens. If you are willing to do further testing on this and send the logs to me, I can check them.

  • 0 in reply to AttilaKovacs

    Thanks for your reply.  I ran a few more tests last night and am seeing the same behavior with both 600MB and 300MB files though.  While still large, they seem to be within this limit... and I agree, either way it shouldn't cause the firewall to seize up.

    I should be able to capture some of this today, and will report back.

  • 0 in reply to John Groller

    Ok, just ran another test... this time with a 600MB MP4.  Here's the setup now:

    • 600MB file
    • Exception applied for the /wp-admin path for all policy rules.
    • Within 5 seconds the server became unresponsive.  I let it go for about 5 minutes before finally killing it.
    • During this time my SSH connection failed and I could not even log into the SOPHOS console directly via the Proxmox UI.  Completely unresponsive every way I normally connect to it.
    • Before the test I deleted the /log/reverseproxy.log.
    • I restarted the daemon by making a rule change.
    • The reverseproxy log logged a couple entries before it looks like logging stopped... presumably due to the failure.
    • I noticed in Proxmox the CPU of the VM going to about 40% and just staying there.
    • I killed the machine around 11:39, and normal startup numbers are shown.

    I'll PM you the reverse proxy logs.

  • 0 in reply to John Groller

    Well, Sophos flagged my account as SPAM/malicious when I tried to PM you the logs... so I had to deal with that Rage.  I think I'm unlocked now and the PM went through.  Let me know what else I can provide.

  • 0 in reply to John Groller

    Sorry for the delay in response... SOPHOS somehow flagged me as SPAM/abusive and I had to go through the appeal process.  Rage.  You should have received the PM now.  Thanks for taking the time.

  • 0 in reply to John Groller

    can you do a atop logging? 

    So log atop into a log file (not /tmp) and then we check this via atop.

    https://unix.stackexchange.com/questions/276069/how-to-configure-atop-sampling-interval-log-path

    atop -w /var/atop.log should be fine 

    Then you can read it with atop -r /var/atop.log

    __________________________________________________________________________________________________________________

Unfiltered HTML