This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Antivirus blocking all ReverseProxy/WAF requests

Over the last several weeks, we have been discovering that our websites go down for a period of 30-60 seconds a couple of times a day - with a HTTP response code of 400 during the outage.

Looking into the reverseproxy.log file about these outages, we discover that EVERY request for 30-60 seconds is (erroneously) flagged by the avscan engine as containing a virus. Example snippet, deidentified:

2017:01:13-09:32:52 asg-2 reverseproxy: [Fri Jan 13 09:32:52.839143 2017] [avscan:error] [pid 20852:tid 3910048624] [client x.x.x.x:31754] [20852] virus daemon error found in request blah-blah/blah.php, referer: http://blah.blah/blah
2017:01:13-09:32:52 asg-2 reverseproxy: [Fri Jan 13 09:32:52.953336 2017] [avscan:notice] [pid 20852:tid 3910048624] [client x.x.x.x:31754] mod_avscan_input_filter: virus found, referer: http://blah.blah/blah
2017:01:13-09:32:52 asg-2 reverseproxy: [Fri Jan 13 09:32:52.954189 2017] [proxy_http:error] [pid 20852:tid 3910048624] (13)Permission denied: [client x.x.x.x:31754] AH01095: prefetch request body failed to y.y.y.y:80 (y.y.y.y) from 204.63.207.1 (), referer: http://blah.blah/blah

We first started noticing this behavior in about middle of December 2016. We have since upgraded the firmware to most recent 9.409-9 and still noticing the issue.

I am worried that the AVScan patterns are operating incorrectly - that's the only thing I can imagine at this point to be the issue.

Does anyone have any input or advice?

Cheers!



This thread was automatically locked due to age.
Parents
  • Hi Sam,

    Which is the AV engine selected? Make sure it is set to AVIRA.

    Any help?

    Sachin Gurung
    Team Lead | Sophos Technical Support
    Knowledge Base  |  @SophosSupport  |  Video tutorials
    Remember to like a post.  If a post (on a question thread) solves your question use the 'This helped me' link.

  • sachingurung,

    I have changed the AV engine to AVIRA and still the issue persists; we are still experiencing periods where the Reverse Proxy responds with HTTP code 400 to all requests.

    Please escalate this to your product engineers to see if the issue can be replicated. Also if you'd like log files of the behavior please see your support@sophos.com email, I have sent many log file snippets... (and, not surprisingly, I have not heard back for several weeks now!!)

    Cheers

  • Sam, this isn't the place to communicate with Sophos Support.  If you have a case #, send an email asking for immediate escalation.  If you don't have a case # and you don't have Premium Support, ask your reseller to get a case open (we do this for no charge for our customers).  If you have Premium Support, Open a ticket with Sophos Support at https://secure2.sophos.com/en-us/support/contact-support.aspx or https://myutm.sophos.com/supportCases.php.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • I have contacted Sophos Support multiple times regarding this Virus flagging issue.  Each time I request escalation, but after the initial response they simply did not reply to my follow-ups.  Yes, case #'s were opened, and yes, case #'s were ignored by the Sophos support team.

    I am completely upset with Sophos Support over this issue, because of the general Lack of Support.

  • In situations where one has a unique problem, there are three steps to take, in order from the least to the most disruptive:

    1. Make a new configuration backup and then restore one from before the Up2Date after which your problems began.
    2. Reboot the UTM.
    3. Take several backups off the UTM if you don't have them sent to you on a daily basis, reload the UTM from ISO and restore the most-recent backup.

    I'm guessing they recommended #3 already and that you don't want to do that as you lose logs, Reporting and graphs.  You can export the logs to your PC with WinSCP, but there's no way to save graphs and reporting.

    If the third solution doesn't fix the problem, you will have re-energized your claim to escalation.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • None of those options are realistic in a top-end production Web Application environment, mostly because A) there have been numerous unrelated production configuration changes that have been made since the events began, and restoring an old config would undo those production config changes - and B) Rebooting the UTM has already happened when the firmwares were updated after the events began.

    Since we are in a Highly Available production environment with active-passive HA nodes, I might be able to destroy the cluster and restore just one node using our USB Smart Installer.  This, of course, creates a single point of failure in the production environment which makes this solution not very attractive.

    I would hate to think that reinstalling from ISO is Sophos's official solution to me (they haven't suggested it as of Jan 9, the last I heard from them).  Instead, I would imagine they would give me confirmation that the Product Engineers have reviewed my log files that I sent them, and that they either A) know what the issue is and they are researching why it's happening or B) need to contact me directly for more information or clarification.  Obviously, none of those things has happened, and that's why I began this discussion on the Sophos Community.

    In fact at this point, I'm reckoning Sophos's Support system as bush league (read: second-rate).  I've even politely asked to collaborate with the supervisors/managers of the support team and I received crickets.  Completely unimpressed.

  • You have a PM, Sam.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Hi Sam,

    I will take care that the case gets a proper escalation and support. Can you just PM me the license number, I can't take any further step towards support unless you are having a paid subscription.

    Alongside, can you also post the reverseproxy logs  for us to investigate.

    Thanks

    Sachin Gurung
    Team Lead | Sophos Technical Support
    Knowledge Base  |  @SophosSupport  |  Video tutorials
    Remember to like a post.  If a post (on a question thread) solves your question use the 'This helped me' link.

Reply
  • Hi Sam,

    I will take care that the case gets a proper escalation and support. Can you just PM me the license number, I can't take any further step towards support unless you are having a paid subscription.

    Alongside, can you also post the reverseproxy logs  for us to investigate.

    Thanks

    Sachin Gurung
    Team Lead | Sophos Technical Support
    Knowledge Base  |  @SophosSupport  |  Video tutorials
    Remember to like a post.  If a post (on a question thread) solves your question use the 'This helped me' link.

Children
No Data