413 Request entity too large

I have been working with Sophos on this very issue since 7/25/19 as well! what a strange small world we live in.

So far I have been escalated through the tiers and I'm currently waiting for a developer to return a phone call. 

I know this much so far, The error is pertaining to the setting defined for ModSecurity : SecRequestBodyNoFilesLimit.  

The default setting is: 1048576bytes of course you already know that. 

It seems apparent that the UTM machines and the XG get and store this information in very different ways and even the top level engineers aren't sure how to apply the tweak.

The problem I see so far is this, older UTM KB documents tell the engineers to bump the tblwafsecurityprofile record item sec_request_body_no_files_limit ="<new value>" restart the WAF and call it good..

This doesn't work on the XG.

So for example lets say you update the db to the size you want, and then to trigger a regeneration of the /cfs/waf/reverseproxy.conf (Included to Apache via httpd.conf) file you add an exception(skip) to your policy from the GUI and click save, The page generates a new /cfs/waf/reverseproxy.conf file with the new exception and updates the database.

But if you take a look at the record now, What you will see is that the db will revert back to the default size setting.

The SecRequestBodyNoFilesLimit is defined in another file somewhere..and this is where the GUI is getting the default value for this setting that it uses to populate the /cfs/waf/reverseproxy.conf file that controls your WAF.

I can see that /usr/apache/conf/waf/base.conf contains an entry defining SecRequestBodyLimit 1GB (1073741824) which is ironically the hard stop max size for this configuration directive but for us we need to find where to define SecRequestBodyNofilesLimit..

I speculate that this is the file we're looking for.. I can only assume that if the configuration directives are not explicitly defined they will simply behave with their defaults.

Perhaps a simple addition of the entry would override the defaults to our size of choice? i.e. "SecRequestBodyNofilesLimit 10485760" for a 10Mb limit

The issue I'm dealing with is that we aren't permitted to browse the directory structure and make changes without the supervision of an engineer or you risk violating your service agreement, and since the XG doesn't have an ftp server enabled easy directory tree structure perusing isn't available.

I'm hoping to have a developer on the line today.. if so I will update this post and this issue should be in their KB.

If you hear anything let me know as well.

Good luck.

I have an XG at home that I could test this on, just let me know step by step what you are looking for and how to confirm it?

Learning is fun lol.....

Also, I only ask because I am following some of what you got here but I went into the /cfs/waf/reverseproxy.conf and noticed I have "SecRequestBodyNofilesLimit 10485760"  set.  So are you saying that this aspect is getting ignored in the WAF and the setting is being pulled from another config such as the base.conf?  

I'd be very curious to learn what you're seeing..

I'm suggesting that the value in mine continually reverts back to 1048576 whenever a change is applied to the policy in the GUI.

For instance if you add an exception to the WAF protection policy (for example, 981200) you are using and then click save.. (you can remove it later..)

Now go back and look at your reverseproxy.conf and the db record for that policy..

You can look at the policy records with:  psql -U nobody -d corporate -xc "select * from tblwafsecurityprofile"

Take a look and see if it's still at 10485760 in the sec_request_body_no_files_limit in the db and SecRequestBodyNofilesLimit in the reverseproxy.conf or if the values have gone back down to 1048576.

If you have multiple policies like we do, The reversion only takes place on the policy that is updated and then saved. meaning that the values in the other policy records wont be changed because you aren't editing their parent policy.

I checked and the policy did change to the 1048576 in both the reverseproxy.conf and db.  

Thanks for doing that test I appreciate it....

Good to know I'm not the only one seeing this behavior.

I'm hopeful that we will have an answer soon,

I had a top level guy on the phone banging around on this XG and even he said that he had to pass the logs over to the developers for analysis.

Id be curious to learn if you to enter SecRequestBodyNofilesLimit 10485760 just below the SecRequestBodyLimit 1073741824 entry in the /usr/apache/conf/waf/base.conf file and then do another change to your policy if it would do it and apply the new setting to the reverseproxy.conf and db

IDK if the WAF service would then need to be restarted to apply the changes.. I'd assume not, but if so you would need to do a  "service WAF:restart -ds nosync" 

Ultimately that's what I really would like to see tested.

Thanks for doing that test I appreciate it....

Good to know I'm not the only one seeing this behavior.

I'm hopeful that we will have an answer soon,

I had a top level guy on the phone banging around on this XG and even he said that he had to pass the logs over to the developers for analysis.

Id be curious to learn if you to enter SecRequestBodyNofilesLimit 10485760 just below the SecRequestBodyLimit 1073741824 entry in the /usr/apache/conf/waf/base.conf file and then do another change to your policy if it would do it and apply the new setting to the reverseproxy.conf and db

IDK if the WAF service would then need to be restarted to apply the changes.. I'd assume not, but if so you would need to do a  "service WAF:restart -ds nosync" 

Ultimately that's what I really would like to see tested.

I think I'll be firing up a VM and trying it myself as well.. 

I think that if this is correct it would be a useful addition to the KB.

Let me know if you had a chance to try it out.

Your reply could save me a ton of time wasted spinning up a VM and pushing my other projects aside.

Thanks again for checking it out.

I attempted to add the 

SecRequestBodyNofilesLimit 10485760 right under the SecRequestBodyLimit in the base.conf  but I cannot seem to save the conf file, even with :wq!

I am thinking this may require to stop a service, apache perhaps? But on this aspect I am not 100% sure.

Darn.. 

It appears that the file is owned by root..

The permissions are not there for the user "admin" to edit it.

XG330_WP02_SFOS 17.5.7 MR-7# ls -l
-rw-r--r-- 1 root 0 1844 Jul 4 08:40 base.conf 

In order to make any changes to it the admin user would need to be a member of the sudoers group and/or you'd need root access.. (which you probably do)

https://community.sophos.com/products/xg-firewall/f/sophos-xg-firewall-general-discussion/76101/advanced-shell

Wonderful.

So the only way to test this now is to have their dev team do it.

I guess I'll be getting back on the phone.. mind you, the hold music loop is atrocious at best.. but hey why not, I haven't got anything better to do with my day.. 

No root huh?.. grrrr.. 

Yes I do not know why they don't replace the hold music with IT related news.

I've spoken to some people in some pretty high places and I'm still waiting to hear from someone at the very very top.

So as of the time of this post the only thing we can currently do to get this applied and working is to manually update the db record and force an update to the waf_reconfig.

First is that you need to know all of the exclusions(skips) you'll need to enter into the policy under the GUI. 

Add them them in and then save the policy. 

You also need to know the policy ID, so you would run:  psql -U nobody -d corporate -c "select * from tblwafsecurityprofile;"

(my query returns)

Look for the number at the beginning of the line listing your policy, in my case the ID I'm interested in is 7.

Next you'd need to apply the change to the value in the sec_request_body_no_files_limit record. in my case I'm defining it to be 10MB and I'm telling the query to apply it to policy ID = 7

psql -U nobody -d corporate -c "update tblwafsecurityprofile set sec_request_body_no_files_limit ="10485760" where id ="7";"

Then finally you need to force the system to update the important files with:

opcode waf_reconfig -t json -b '{"Entity": "waf_advanced_config", "Event": "UPDATE"}' -ds nosync

This will rebuild the reverseporxy.conf and get the WAF up and running with the new larger limit.. 

After this is done, If you make ANY changes to this policy in the GUI you will need to do the above process again because the default 1048576 value will be reinserted into the db and reverseproxy.conf bringing you back to the broken and unusable 413 issue.

Hi

We also have a remote sesion with support, and also he use the commands you mention:

"We run the following commands in the XG to increase the file size allowed in the WAF from 1048576 [1MB] to 52428800 [50MB]. With the following command:

SFVUNL_VM01_SFOS 17.5.7 MR-7# psql -U nobody -d corporate -c "update tblwafsecur
ityprofile set sec_request_body_no_files_limit ="52428800" where id ="7";"
UPDATE 1
.
SFVUNL_VM01_SFOS 17.5.7 MR-7# opcode waf_reconfig -t json -b '{"Entity": "waf_ad
vanced_config", "Event": "UPDATE"}' -ds nosync
200 OK
{ "status": "200", "statusmessage": "success" }
.
SFVUNL_VM01_SFOS 17.5.7 MR-7# psql -U nobody -d corporate -c "select name,id,sec
_request_body_no_files_limit from tblwafsecurityprofile;"
           name            | id | sec_request_body_no_files_limit
---------------------------+----+---------------------------------
 Exchange AutoDiscover     |  1 |                         1048576
 Exchange General          |  2 |                         1048576
 Microsoft Lync            |  4 |                         1048576
 Microsoft RDG 2008        |  5 |                         1048576
 Microsoft RD Web 2008     |  6 |                         1048576
 Exchange Outlook Anywhere |  3 |                         1048576
 WAF Police                |  7 |                        52428800

but we got the error again with files of more than 49KB, smaller than these worked fine.

On the IIS we increase this value "" uploadReadAheadSize "which by default is -49KB to a larger one, after this all the files that failed previously it was possible to load them, although with 30MB files this still failed, but this change lets us see that the problem indicates that it is a configuration in the IIS that needs to be changed."

So we´re waiting for the reply to this, because if we try to upload a file without passing through the WAF, and we did´t have this problem.

I´ll let you know

I'm glad to read abut your progress. I'd be curious if you're working with Arjun.. He's been most helpful.

Yes, IIS uploadReadAhaeadSize will definitely cause this issue, Actually that's the first thing we changed before I dove into the rabbit hole chasing this configuration directive override.

If the WAF is blocking the transaction you can easily monitor it by watching the /log/reverseproxy.log during the post to the IIS server.

That way you can see if it's the WAF or IIS dropping your file transfer at 30mb.

tail -f reverseproxy.log | grep -i "<Testing PC IP Addy>"

There is the possibility that another scan is killing the transfer for some other reason like a false positive on a Restricted SQL Character Anomaly, or some other pattern match.

...then again you wouldn't get a 413..

Happy hunting. Just keep in mind that if you find that you need to add another rule ID to the skip rules list, you'll also need to reapply the value change to the db and rerun the opcode to apply the change again to your system.

...at least until they get a fix rolled out for this bug.