This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Virtual Webservers problem after UTM restart in AWS

I have a UTM in AWS and I have two webservers configured in Web Application Firewall.  When UTM is rebooted(update, maintenance, testing, etc)  the Real Webservers are in error and their buttons are in orange color.  It does not come back automatically and I have to manually toggle the Enable and Disable button to get them to work again(Green color).   Is this a known problem?   I do not want to manually have to go in each time to enable WAF.  



This thread was automatically locked due to age.
  • Hi,

    it's not really clear from your post: is the WAF not working after reboot or is it just the buttons that are orange after reboot?

    What's in the reverseproxy.log after the reboot?

     

    Sabine

  • Aren't they usually related?   If WAF is not working, somewhere in Webserver Protection, there is orange color indication.  If there is Orange color indication, WAF is not functional. 

     

    Well, in my case, after reboot, I can't access the site that I need to goto which means WAF is not working.  So when I check UTM, there is orange color wanrnings to Site Path routes.  I have to manually reset them to have WAF working again. 

    There isn't much useful information in the log other than telling that the WAF is shutting down.  This log is generated right after it started:

     

    2017:05:25-09:46:33  reverseproxy: [Thu May 25 09:46:33.000671 2017] [security2:notice] [pid 6313:tid 4148139712] ModSecurity: LIBXML compiled version="2.7.6"
    2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.001002 2017] [proxy_protocol:notice] [pid 6331:tid 4148139712] ProxyProtocol: disabled on 127.0.0.1:4080
    2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.002470 2017] [mpm_worker:notice] [pid 6331:tid 4148139712] AH00292: Apache/2.4.10 (Unix) OpenSSL/1.0.2j-fips configured -- resuming normal operations
    2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.002497 2017] [core:notice] [pid 6331:tid 4148139712] AH00094: Command line: '/usr/apache/bin/httpd'
    2017:05:25-09:46:38  reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="447" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"
    2017:05:25-09:46:45  reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="363" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"
    2017:05:25-09:46:45  reverseproxy: [Thu May 25 09:46:45.679056 2017] [mpm_worker:notice] [pid 6331:tid 4148139712] AH00297: SIGUSR1 received. Doing graceful restart
    2017:05:25-09:46:45  reverseproxy: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroOptweb] does not exist

  • Leo, I've seen situations where a Real Server was orange and yet things worked just fine.  I've not seen an orange color associated to a Site Path Route - what version are you on?

    Go to the point in the log where you Disabled/Enabled the Virtual Server.  Show us the lines beginning with 20 lines before the Disable to 40 lines after the Enable.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • I have the latest version, 9.500-9.  

    I've tried to work with Sophos support but no help.  latest from him is that I have to disable "Stick Session" but I don't know what this got to do with the reverse proxy not coming back on reboot...  

     

    here is the log.  It was enabled around 2017:05:31-11:05.

     

    2017:05:31-10:57:03 abcpublic reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="274" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"

    2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000136 2017] [proxy_protocol:notice] [pid 5294:tid 4147439296] ProxyProtocol: disabled on 127.0.0.1:4080

    2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000208 2017] [security2:notice] [pid 5294:tid 4147439296] ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

    2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000211 2017] [security2:notice] [pid 5294:tid 4147439296] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"

    2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000214 2017] [security2:notice] [pid 5294:tid 4147439296] ModSecurity: PCRE compiled version="7.8 "; loaded version="7.8 2008-09-05"

    2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000216 2017] [security2:notice] [pid 5294:tid 4147439296] ModSecurity: LIBXML compiled version="2.7.6"

    2017:05:31-11:00:08 abcpublic reverseproxy: [Wed May 31 11:00:08.001000 2017] [proxy_protocol:notice] [pid 5459:tid 4147439296] ProxyProtocol: disabled on 127.0.0.1:4080

    2017:05:31-11:00:08 abcpublic reverseproxy: [Wed May 31 11:00:08.003399 2017] [mpm_worker:notice] [pid 5459:tid 4147439296] AH00292: Apache/2.4.10 (Unix) OpenSSL/1.0.2j-fips configured -- resuming normal operations

    2017:05:31-11:00:08 abcpublic reverseproxy: [Wed May 31 11:00:08.003427 2017] [core:notice] [pid 5459:tid 4147439296] AH00094: Command line: '/usr/apache/bin/httpd'

    2017:05:31-11:01:21 abcpublic reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="0" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="433" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"

    2017:05:31-11:02:02 abcpublic reverseproxy: [Wed May 31 11:02:02.049495 2017] [mpm_worker:notice] [pid 5459:tid 4147439296] AH00295: caught SIGTERM, shutting down

    2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001045 2017] [proxy_protocol:notice] [pid 6317:tid 4148152000] ProxyProtocol: disabled on 127.0.0.1:4080

    2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001212 2017] [security2:notice] [pid 6317:tid 4148152000] ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

    2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001222 2017] [security2:notice] [pid 6317:tid 4148152000] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"

    2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001229 2017] [security2:notice] [pid 6317:tid 4148152000] ModSecurity: PCRE compiled version="7.8 "; loaded version="7.8 2008-09-05"

    2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001233 2017] [security2:notice] [pid 6317:tid 4148152000] ModSecurity: LIBXML compiled version="2.7.6"

    2017:05:31-11:02:19 abcpublic reverseproxy: [Wed May 31 11:02:19.000212 2017] [proxy_protocol:notice] [pid 6325:tid 4148152000] ProxyProtocol: disabled on 127.0.0.1:4080

    2017:05:31-11:02:19 abcpublic reverseproxy: [Wed May 31 11:02:19.002154 2017] [mpm_worker:notice] [pid 6325:tid 4148152000] AH00292: Apache/2.4.10 (Unix) OpenSSL/1.0.2j-fips configured -- resuming normal operations

    2017:05:31-11:02:19 abcpublic reverseproxy: [Wed May 31 11:02:19.002180 2017] [core:notice] [pid 6325:tid 4148152000] AH00094: Command line: '/usr/apache/bin/httpd'

    2017:05:31-11:02:23 abcpublic reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="435" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"

    2017:05:31-11:05:38 abcpublic reverseproxy: id="0299" srcip="10.168.11.133" localip="10.168.11.189" size="1215" user="-" host="10.168.11.133" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="61772" url="/opt/" server="abcpublic.com" referer="-" cookie="AWSALB=dr15+izHNymddWqqtb5rK+iDDwpDrxNkT52VFClTqUubSprXHjJO88AEaZknuzkfVA7lfCCGLmcAaVtguDV7I8naQjiA3F78B82//0evm7pIAt7TBU7Pr6ri3yZP" set-cookie="AWSALB=dwCIXdvFhEGcM0bCVkCWSbKnmM0ikv4vA6jphUdP+Q7j5JZgjorUxoeV9CMkmUpoyMg3pQIZ2NgCCcj535k/yqD1F5Xx28FbstBp7h6gb0g6deTjFeBr59+P0eGx; Expires=Wed, 07 Jun 2017 15:05:38 GMT; Path=/"

    2017:05:31-11:05:39 abcpublic reverseproxy: id="0299" srcip="10.168.11.133" localip="10.168.11.189" size="14370" user="-" host="10.168.11.133" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="9029" url="/opt/css/opt.css" server="abcpublic.com" referer="https://abc.com/opt/" cookie="AWSALB=dwCIXdvFhEGcM0bCVkCWSbKnmM0ikv4vA6jphUdP+Q7j5JZgjorUxoeV9CMkmUpoyMg3pQIZ2NgCCcj535k/yqD1F5Xx28FbstBp7h6gb0g6deTjFeBr59+P0eGx" set-cookie="AWSALB=3Fr49MOJVEbbNlGv0/9TXFCzS2qmUT2TJu37ruZehGw249dV52cR412oA8WdQHTMZCrrcsBXqA0Fva8wNR/fOvNRPatCGHeX0xF88sFqCHEgv4/VW6aUrFcmJoXZ; Expires=Wed, 07 Jun 2017 15:05:39 GMT; Path=/"

    2017:05:31-11:05:39 abcpublic reverseproxy: id="0299" srcip="10.168.11.133" localip="10.168.11.189" size="13426" user="-" host="10.168.11.133" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="8753" url="/opt/css/uswds.css" server="abcpublic.com" referer="https://abc.com/opt/" cookie="AWSALB=dwCIXdvFhEGcM0bCVkCWSbKnmM0ikv4vA6jphUdP+Q7j5JZgjorUxoeV9CMkmUpoyMg3pQIZ2NgCCcj535k/yqD1F5Xx28FbstBp7h6gb0g6deTjFeBr59+P0eGx" set-cookie="AWSALB=6IwvFlfNPelT6Wh+zA12p4kJjDoM3VR/y8VAOmXnk/NPwOTAFKvu4+vuzSxjEkX9cGUAbLPdq9FvBFd/OcN/HeJDgLCTS6PzzY4QJkjAv8bh4XCWayl3kihyWCp8; Expires=Wed, 07 Jun 2017 15:05:40 GMT; Path=/"

    2017:05:31-11:05:40 abcpublic reverseproxy: id="0299" srcip="10.168.11.133" localip="10.168.11.189" size="10395" user="-" host="10.168.11.133" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="13989" url="/opt/js/vendor/angular/angular-resource.js" server="abcpublic.com" referer="https://abc.com/opt/" cookie="AWSALB=dwCIXdvFhEGcM0bCVkCWSbKnmM0ikv4vA6jphUdP+Q7j5JZgjorUxoeV9CMkmUpoyMg3pQIZ2NgCCcj535k/yqD1F5Xx28FbstBp7h6gb0g6deTjFeBr59+P0eGx" set-cookie="AWSALB=3qqyNWTM51u7p9d8ZkwVaepZ5SLOJb0xWmwGmxbM/OUsqSrTK08Lgz7ibWaJfXDLzCtJ7JZVl9IMgIjLww59idEnh+V2CSmK2/JAS6W36rWOrZmefHNGgPPLZa14; Expires=Wed, 07 Jun 2017 15:05:40 GMT; Path=/"

  • The buttons only indicate the status of the real webservers. After a reboot, they are always orange because the WAF has not checked yet if the real webservers are reachable.

    There are no requests before the manual restart of the WAF. So, my question is still: is the WAF reachable?

     

    Best
     Sabine

  • No. As I have initially pointed out that it does not come back automatically and reverse proxy is still not functional.   It always needs to be manually enabled again.  

  • Ok, then please add the log lines from request to the WAF when the WAF is not reachable.

  • Please refer to my replies on this thread.  They are attached.

  • Both 9.414 and 9.501 contain the following bugfix:

    NUTM-6930 [WAF] WAF not responding after reboot of the AWS UTM

    Can you confirm that this fixes your issue?