Thread Info
  • State Suggested Answer
  • Locked Locked
  • Replies 12 replies
  • Answers 1 answer
  • Subscribers 3 subscribers
  • Views 19703 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

WAF captures all backend errors

Harro Verton

Hello,

I'm in the process of converting a linux/apache box that was used as a reverse proxy to UTM 9 WAF, and I've bumped into an issue.

I' ve just spend over an hour trying to figure out why a migrated site gave a 403 error page on every request. Finally I found that the site itself (on the real server) generated a 503, with a message saying the database credentials were not correct.

Is there a way to configure the WAF to not hide these kind of messages?



This thread was automatically locked due to age.
  • 0

    Hello,


    in general, the WAF doesn't rewrite backend error messages. What's your configuration? And could you post the corresponding log lines when you get the 403 responses?


    Sabine

  • 0 in reply to Evianne

    I will try to reproduce it with a test site later today. This particular site is live now, so I can't touch that anymore.

    What info do you need about the config?  It's an HA cluster (active-passive), both nodes running 9.402-7.

  • 0 in reply to Harro Verton

    The configuration of the virtual webserver, firewall profile, exceptions to begin with.

  • 0 in reply to Evianne

    Ok. This is the virtualhost that had the problem:

    No exceptions defined. I tried "Advanced Protection" first, but the application (phpBB v3) has forms that don't pass, so I changed to "Basic Protection".

  • 0

    Here's what I think happened:

    The Common Threats filter of your firewall profile blocked the 503 response with a 403 error to prevent information leakage.

    The "Outbound" category of the Common Threats filter contains rules like (I'm quoting comments from the outbound rules file here):

    * SQL Errors leakage

    * IIS Errors leakage

    Probably one of those rules matched and triggered the 403 block page.

    You can check reverseproxy.log to see if that's really the case.

  • 0 in reply to ewadie

    Quite possible. I can't check anymore as the reverse proxy log only contains todays entries if I download it.


    When it happens again, I'll first remove the firewall profile, that would effectively disable those rules, right? Could you confirm it does, since I also now see entries in the log like:

    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.039155 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmc, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.039856 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmc' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.040394 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmt, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044190 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmt' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044478 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmz, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044665 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmz' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044860 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utma, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045094 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utma' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045286 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmb, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045462 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmb' from request due to missing/invalid signature, referer: https://www.example.org/abby/

    (IP and sitename changed to protect the innocent ;-)

    for a virtual webserver definition that has no firewall profile attached. This suggests that the WAF always intervenes at some point? Do I explicitly have to make a firewall profile that has all rules disabled?

  • 0 in reply to Harro Verton

    Harro Verton said:
    Quite possible. I can't check anymore as the reverse proxy log only contains todays entries if I download it.

    Older log files get rotated into /var/log/reverseproxy/

    Harro Verton said:
    When it happens again, I'll first remove the firewall profile, that would effectively disable those rules, right?

    Right. You can also do any of these:

    • Disable the Common Threats filter in the firewall profile
    • Disable the Outbound category in the firewall profile (assuming that it is in fact the Outbound category which is blocking your 503 response)
    • Skip the specific rule ID that is causing problems, also in the firewall profile

    Harro Verton said:
    Could you confirm it does, since I also now see entries in the log like:

    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.039155 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmc, referer: https://www.example.org/abby/

    for a virtual webserver definition that has no firewall profile attached.

    Are you sure? Those messages come from Cookie signing. You wouldn't see those without a firewall profile.

  • 0 in reply to Harro Verton

    Ok, silly me, found the archived logs.

    They don't really help though, the entries for the original problem are:

    2016:05:27-13:02:46 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="337" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="351288" url="/index.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:02:48 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="337" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="20973" url="/index.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:03:47 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="337" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="21215" url="/index.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:04:01 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="17699" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:04:05 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="13105" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:05:06 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="54674" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:05:08 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="22928" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:08:53 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="12488" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:08:54 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="15236" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:08:55 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="9957" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:11:32 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="50076" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:11:32 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="17070" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:11:50 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="335" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="36939" url="/faq.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:13:11 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="335" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="15075" url="/faq.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:13:12 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="335" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="12252" url="/faq.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:14:09 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="335" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="10073" url="/faq.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"

  • 0 in reply to ewadie

    ewadie said:
    Are you sure? Those messages come from Cookie signing. You wouldn't see those without a firewall profile.

    Yes, 100% sure. I just double checked. This is its definition:

    I've logged in via SSH and checked the generated reverseproxy.conf, and that doesn't show anything weird. It does not contain any "CookieDropUnsigned On".

  • 0 in reply to Harro Verton

    This is the live log from a few minutes ago, for that site:

    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586083 2016] [cookie:error] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] No signature found, cookie: __utma, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586438 2016] [cookie:warn] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] Dropping cookie '__utma' from request due to missing/invalid signature, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586636 2016] [cookie:error] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] No signature found, cookie: __utmb, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586805 2016] [cookie:warn] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] Dropping cookie '__utmb' from request due to missing/invalid signature, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586972 2016] [cookie:error] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] No signature found, cookie: __utmc, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.587132 2016] [cookie:warn] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] Dropping cookie '__utmc' from request due to missing/invalid signature, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.587306 2016] [cookie:error] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] No signature found, cookie: __utmt, referer: https://www.example.org/jade/

Unfiltered HTML