WAF issues after upgrading to 9.602-3

Hi,

 

after upgrading from 9.601-5 to 9.602-3 we experienced issues ( intermittently lost packets and rejected connections) at some Apps (Kerio Connect), all HTTPS only.

Rollback to 9.601-5 fixed the problems. Found no useful messages in the WAF/FW logs. We have 13 Apps in the WAS config defined, maybe a number with a bad Karma.

 

Thanks

 

Henri 

  • Hallo Henri,

    Occasionally, the Up2Date process "breaks" something in the configuration.  Restoring the configuration backup made just before the last Up2Dates were applied gives the system another shot at upgrading the configuration databases. This is often all that's needed to fix a situation like yours.  If it doesn't, then I recommend rebooting several times before deciding to re-image from ISO.

    Cheers - Bob

  • In reply to BAlfson:

    Hi Bob,

     

    thanks four your reply.

     

    I tried to restore config and upgrade again. Same issue. After tracing a bit, in case of such an error, the WAF does not receive a https packet, it's rejected in between.

    There is also no trace entry in the firewall log, a nmap receives a syn reject. A few seconds later on, the issue is gone, the connection works again fine, and so on ...

    Could you please give me a ping, when this issue is solved? Have here complex WAF exceptions, it took very long to create/test it, therefor a reinstall is no option.

    I will stay at this FW version.

     

    Thanks

     

    Henri

  • In reply to Henri04:

    Bob / Henri -

     

    How do I roll back to 9.601-5? I am having very similar issues after updating to 9.602-3. All traffic seems to hit the default web filter policy and gets blocked rather than hitting the profile it should. If you do a policy check it tells that the traffic is allowed but it actually blocks. If I set the default policy to allow all everything seems to work.  I don't see a way to downgrade the firmware from 9.602-3 down to 9.601-5? When I go to backup/restore it will not do anything.

     

    Thank you for any help you can offer.

  • In reply to Jesse Wulf:

    Hi Jesse,

     

    I use here (a) VMWARE appliance(s) (HA) and backup it with VEEAM, so it's very easy to step back, sorry, I assume this will not help in your case.

     

    Best regards

     

    Henri

  • In reply to Jesse Wulf:

    Hi Jesse and welcome to the UTM Community!  (Lurking since 2017, I see!)

    " When I go to backup/restore it will not do anything." - If you mean restoring doesn't change the version, that's correct.

    The only way to downgrade is to re-image from ISO unless you have a backup of a VM.  Is this a home-use situation?  If a business, you might want to copy off the logs first and then back on after re-imaging.  I don't know of an easy way to keep reporting and graphs.  Is this a software version or a Sophos appliance?

    Cheers - Bob

  • In reply to BAlfson:

    @Henri - You are correct we run the hardware version so restoring is not quite so simple.

    @Bob - Yes, I have read many of your posts.  In this case, I am not convinced it it the firmware anymore. After an hour and a half on hold and was told there was no known issues, I received another email from support stating their Web Categorization service was having issues. We are still having issues today, over 24 hours later. I have had to resort to turn off transparent filtering so our users stopped receiving messages of "refused to connect"

    I have emailed support a couple of times with no response. Know of a way to get through to tech support any quicker than email or and hour and a half on hold? I sent one email yesterday evening and one this morning and have heard nothing back.

    Thanks for your help!

    Jesse

  • In reply to Jesse Wulf:

    After another day of continued issues, I have found that if I change all the filtering profiles back to standard rather than transparent things will go back to normal. Obviously I don't want to leave it that way but after another hour and a half waiting for support they were convinced the issue was the AD SSO. I was instructed to put the profiles in transparent but leave the authentication on none. That is not the fix so I am currently waiting for support to call me back.

    These issues didn't start until this firmware but so far I have not been able to get anyone at Sophos to admit there is a problem. They tell me it is unusual to have call wait times an hour and a half so they must be aware of something... my end users are beyond frustrated.

    Anyone else having any luck?

  • In reply to Jesse Wulf:

    Jesse, the problem you are describing is with the Web Filtering, not Web Application Firewall --- and yes, your specific issue is a known issue; transparent web filtering is essentially broken for some customers (appears random; our production system is broken, yet a customer of mine who only uses transparent mode works fine, same firmware).  Workarounds include manually specifying proxy settings in your browsers, etc. or disabling https scanning in the web filtering profile(s).  Yeah not great workarounds but it's apparently really broken.  Tell your support folks to reference issue 10952

  • In reply to BrucekConvergent:

    Thank you for that information Bruce.

    There was a firmware updated that came out yesterday that specifically addressed 10952 which I installed this morning. Unfortunately, I didn't seem to resolve the issue. Is there some kind of cache or similar that needs to be cleared after installing firmware that could cause the issue to continue even though the new firmware has been applied? It did reboot the firewall.

    Here is the info I have on the update:

    Up2Date 9.603001 package description:

    Remark:
    System will be rebooted

    News:
    Maintenance Release

    Bugfix:
    Fix [NUTM-10932]: [AWS, Basesystem] License issue for AWS installations after the upgrading firmware to 9.602

    RPM packages contained:
    ep-confd-9.60-1396.g5df23ab47.i686.rpm
    ep-release-9.603-1.noarch.rpm

    I am also going to reply back to my case to let them know the issues isn't resolved but so far on this case, unless I sit on hold for an hour and a half to talk to a live person, they have not replied to my email case replies.

  • In reply to Jesse Wulf:

    9.603 does not include the fix we are all waiting for, only corrects an AWS licensing issue.

  • In reply to BrucekConvergent:

    Hi All,

    The issue NUTM-10952 is still being actively investigated by our team with critical priority. Please stay tuned as more information regarding a fix version will be announced as it becomes available.

    Regards,

  • In reply to FloSupport:

    Hi FloSupport,

    the issue NUTM-10952 doesn´t seem to match the issue described by the OP (WAF issues).

    My Sophos UTM support company is asking me constantly for occurrences of WAF failures and screenshots from our monitrings and even web check locations.

    Thanks,

    Claudio

  • In reply to Claudio Schnell da Silva:

    Hi  

    My apologies, I was replying back to the ID that  mentioned.

     I would advise that you raise a case so that our support team can follow up and further investigate your appliance. Please PM me with your case details as well.

  • In reply to FloSupport:

    Hi,

     

    I found in another topic that the Webfilter issues are solved by changing the SPX port.

    Have here also SPX and the SPX traffic ist redirected to the public IP of the UTM appliance with port 443, with is the WAF.

    I have a WAF rule to redirect the SPX traffic to a internal IP where SPX is listing. 

    The reason for that, have in case of a ISP failure a backup with only one IP and have to use port 443, otherwise I'am blocked by customers FWs.

    Could this configuration ends in bad ipfilter rules, after a firmware upgrade?

     

    Thanks

     

    Henri

     

    Henri