Thread Info
  • State Suggested Answer
  • Locked Locked
  • Replies 12 replies
  • Answers 1 answer
  • Subscribers 3 subscribers
  • Views 19707 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

WAF captures all backend errors

Harro Verton

Hello,

I'm in the process of converting a linux/apache box that was used as a reverse proxy to UTM 9 WAF, and I've bumped into an issue.

I' ve just spend over an hour trying to figure out why a migrated site gave a 403 error page on every request. Finally I found that the site itself (on the real server) generated a 503, with a message saying the database credentials were not correct.

Is there a way to configure the WAF to not hide these kind of messages?



This thread was automatically locked due to age.
Parents
  • 0

    Here's what I think happened:

    The Common Threats filter of your firewall profile blocked the 503 response with a 403 error to prevent information leakage.

    The "Outbound" category of the Common Threats filter contains rules like (I'm quoting comments from the outbound rules file here):

    * SQL Errors leakage

    * IIS Errors leakage

    Probably one of those rules matched and triggered the 403 block page.

    You can check reverseproxy.log to see if that's really the case.

  • 0 in reply to ewadie

    Quite possible. I can't check anymore as the reverse proxy log only contains todays entries if I download it.


    When it happens again, I'll first remove the firewall profile, that would effectively disable those rules, right? Could you confirm it does, since I also now see entries in the log like:

    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.039155 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmc, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.039856 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmc' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.040394 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmt, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044190 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmt' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044478 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmz, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044665 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmz' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044860 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utma, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045094 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utma' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045286 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmb, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045462 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmb' from request due to missing/invalid signature, referer: https://www.example.org/abby/

    (IP and sitename changed to protect the innocent ;-)

    for a virtual webserver definition that has no firewall profile attached. This suggests that the WAF always intervenes at some point? Do I explicitly have to make a firewall profile that has all rules disabled?

Reply
  • 0 in reply to ewadie

    Quite possible. I can't check anymore as the reverse proxy log only contains todays entries if I download it.


    When it happens again, I'll first remove the firewall profile, that would effectively disable those rules, right? Could you confirm it does, since I also now see entries in the log like:

    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.039155 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmc, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.039856 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmc' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.040394 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmt, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044190 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmt' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044478 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmz, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044665 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmz' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.044860 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utma, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045094 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utma' from request due to missing/invalid signature, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045286 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmb, referer: https://www.example.org/abby/
    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.045462 2016] [cookie:warn] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] Dropping cookie '__utmb' from request due to missing/invalid signature, referer: https://www.example.org/abby/

    (IP and sitename changed to protect the innocent ;-)

    for a virtual webserver definition that has no firewall profile attached. This suggests that the WAF always intervenes at some point? Do I explicitly have to make a firewall profile that has all rules disabled?

Children
  • 0 in reply to Harro Verton

    Harro Verton said:
    Quite possible. I can't check anymore as the reverse proxy log only contains todays entries if I download it.

    Older log files get rotated into /var/log/reverseproxy/

    Harro Verton said:
    When it happens again, I'll first remove the firewall profile, that would effectively disable those rules, right?

    Right. You can also do any of these:

    • Disable the Common Threats filter in the firewall profile
    • Disable the Outbound category in the firewall profile (assuming that it is in fact the Outbound category which is blocking your 503 response)
    • Skip the specific rule ID that is causing problems, also in the firewall profile

    Harro Verton said:
    Could you confirm it does, since I also now see entries in the log like:

    2016:05:31-00:00:08 firewall-1-1 reverseproxy: [Tue May 31 00:00:08.039155 2016] [cookie:error] [pid 8810:tid 4046146416] [client 130.115.xxx.xxx:50949] No signature found, cookie: __utmc, referer: https://www.example.org/abby/

    for a virtual webserver definition that has no firewall profile attached.

    Are you sure? Those messages come from Cookie signing. You wouldn't see those without a firewall profile.

  • 0 in reply to Harro Verton

    Ok, silly me, found the archived logs.

    They don't really help though, the entries for the original problem are:

    2016:05:27-13:02:46 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="337" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="351288" url="/index.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:02:48 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="337" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="20973" url="/index.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:03:47 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="337" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="21215" url="/index.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:04:01 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="17699" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:04:05 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="13105" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:05:06 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="54674" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:05:08 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="22928" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:08:53 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="12488" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:08:54 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="15236" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:08:55 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="9957" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:11:32 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="50076" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:11:32 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="328" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="17070" url="/" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:11:50 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="335" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="36939" url="/faq.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:13:11 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="335" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="15075" url="/faq.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:13:12 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="335" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="12252" url="/faq.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"
    2016:05:27-13:14:09 firewall-1-1 reverseproxy: id="0299" srcip="212.56.xxx.xxx" localip="5.135.xxx.xxx" size="335" user="-" host="212.56.xxx.xxx" method="GET" statuscode="403" reason="waf" extra="Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available" exceptions="-" time="10073" url="/faq.php" server="my-mc-phoenix.com" referer="-" cookie="-" set-cookie="-"

  • 0 in reply to ewadie

    ewadie said:
    Are you sure? Those messages come from Cookie signing. You wouldn't see those without a firewall profile.

    Yes, 100% sure. I just double checked. This is its definition:

    I've logged in via SSH and checked the generated reverseproxy.conf, and that doesn't show anything weird. It does not contain any "CookieDropUnsigned On".

  • 0 in reply to Harro Verton

    This is the live log from a few minutes ago, for that site:

    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586083 2016] [cookie:error] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] No signature found, cookie: __utma, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586438 2016] [cookie:warn] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] Dropping cookie '__utma' from request due to missing/invalid signature, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586636 2016] [cookie:error] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] No signature found, cookie: __utmb, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586805 2016] [cookie:warn] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] Dropping cookie '__utmb' from request due to missing/invalid signature, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.586972 2016] [cookie:error] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] No signature found, cookie: __utmc, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.587132 2016] [cookie:warn] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] Dropping cookie '__utmc' from request due to missing/invalid signature, referer: https://www.example.org/jade/
    2016:05:31-17:36:26 firewall-1-1 reverseproxy: [Tue May 31 17:36:26.587306 2016] [cookie:error] [pid 38509:tid 4104895344] [client 86.145.xxx.xxx:54939] No signature found, cookie: __utmt, referer: https://www.example.org/jade/

  • 0 in reply to Harro Verton

    Harro Verton said:
    They don't really help though, the entries for the original problem are:

    Outbound Anomaly Score Exceeded (score 4): Last Matched Message: The application is not available means that the request is in fact blocked by the Common Threats filter. But there should many more log lines from the Common Threats filter itself explaining which rules matched. Can't tell you why that's not the case on your machine.

  • 0 in reply to Harro Verton

    Harro Verton said:
    Yes, 100% sure. I just double checked. This is its definition:

    Sorry, then I don't know why those log lines appear.

Unfiltered HTML