Virtual Webservers problem after UTM restart in AWS

Hi,

it's not really clear from your post: is the WAF not working after reboot or is it just the buttons that are orange after reboot?

What's in the reverseproxy.log after the reboot?

Sabine

Aren't they usually related?   If WAF is not working, somewhere in Webserver Protection, there is orange color indication.  If there is Orange color indication, WAF is not functional. 

Well, in my case, after reboot, I can't access the site that I need to goto which means WAF is not working.  So when I check UTM, there is orange color wanrnings to Site Path routes.  I have to manually reset them to have WAF working again. 

There isn't much useful information in the log other than telling that the WAF is shutting down.  This log is generated right after it started:

2017:05:25-09:46:33  reverseproxy: [Thu May 25 09:46:33.000671 2017] [security2:notice] [pid 6313:tid 4148139712] ModSecurity: LIBXML compiled version="2.7.6"
2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.001002 2017] [proxy_protocol:notice] [pid 6331:tid 4148139712] ProxyProtocol: disabled on 127.0.0.1:4080
2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.002470 2017] [mpm_worker:notice] [pid 6331:tid 4148139712] AH00292: Apache/2.4.10 (Unix) OpenSSL/1.0.2j-fips configured -- resuming normal operations
2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.002497 2017] [core:notice] [pid 6331:tid 4148139712] AH00094: Command line: '/usr/apache/bin/httpd'
2017:05:25-09:46:38  reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="447" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"
2017:05:25-09:46:45  reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="363" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"
2017:05:25-09:46:45  reverseproxy: [Thu May 25 09:46:45.679056 2017] [mpm_worker:notice] [pid 6331:tid 4148139712] AH00297: SIGUSR1 received. Doing graceful restart
2017:05:25-09:46:45  reverseproxy: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroOptweb] does not exist

Aren't they usually related?   If WAF is not working, somewhere in Webserver Protection, there is orange color indication.  If there is Orange color indication, WAF is not functional. 

Well, in my case, after reboot, I can't access the site that I need to goto which means WAF is not working.  So when I check UTM, there is orange color wanrnings to Site Path routes.  I have to manually reset them to have WAF working again. 

There isn't much useful information in the log other than telling that the WAF is shutting down.  This log is generated right after it started:

2017:05:25-09:46:33  reverseproxy: [Thu May 25 09:46:33.000671 2017] [security2:notice] [pid 6313:tid 4148139712] ModSecurity: LIBXML compiled version="2.7.6"
2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.001002 2017] [proxy_protocol:notice] [pid 6331:tid 4148139712] ProxyProtocol: disabled on 127.0.0.1:4080
2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.002470 2017] [mpm_worker:notice] [pid 6331:tid 4148139712] AH00292: Apache/2.4.10 (Unix) OpenSSL/1.0.2j-fips configured -- resuming normal operations
2017:05:25-09:46:34  reverseproxy: [Thu May 25 09:46:34.002497 2017] [core:notice] [pid 6331:tid 4148139712] AH00094: Command line: '/usr/apache/bin/httpd'
2017:05:25-09:46:38  reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="447" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"
2017:05:25-09:46:45  reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="363" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"
2017:05:25-09:46:45  reverseproxy: [Thu May 25 09:46:45.679056 2017] [mpm_worker:notice] [pid 6331:tid 4148139712] AH00297: SIGUSR1 received. Doing graceful restart
2017:05:25-09:46:45  reverseproxy: AH00112: Warning: DocumentRoot [/var/www/REF_RevFroOptweb] does not exist

Leo, I've seen situations where a Real Server was orange and yet things worked just fine.  I've not seen an orange color associated to a Site Path Route - what version are you on?

Go to the point in the log where you Disabled/Enabled the Virtual Server.  Show us the lines beginning with 20 lines before the Disable to 40 lines after the Enable.

Cheers - Bob

I have the latest version, 9.500-9.  

I've tried to work with Sophos support but no help.  latest from him is that I have to disable "Stick Session" but I don't know what this got to do with the reverse proxy not coming back on reboot...  

here is the log.  It was enabled around 2017:05:31-11:05.

2017:05:31-10:57:03 abcpublic reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="274" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"

2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000136 2017] [proxy_protocol:notice] [pid 5294:tid 4147439296] ProxyProtocol: disabled on 127.0.0.1:4080

2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000208 2017] [security2:notice] [pid 5294:tid 4147439296] ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000211 2017] [security2:notice] [pid 5294:tid 4147439296] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"

2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000214 2017] [security2:notice] [pid 5294:tid 4147439296] ModSecurity: PCRE compiled version="7.8 "; loaded version="7.8 2008-09-05"

2017:05:31-11:00:07 abcpublic reverseproxy: [Wed May 31 11:00:07.000216 2017] [security2:notice] [pid 5294:tid 4147439296] ModSecurity: LIBXML compiled version="2.7.6"

2017:05:31-11:00:08 abcpublic reverseproxy: [Wed May 31 11:00:08.001000 2017] [proxy_protocol:notice] [pid 5459:tid 4147439296] ProxyProtocol: disabled on 127.0.0.1:4080

2017:05:31-11:00:08 abcpublic reverseproxy: [Wed May 31 11:00:08.003399 2017] [mpm_worker:notice] [pid 5459:tid 4147439296] AH00292: Apache/2.4.10 (Unix) OpenSSL/1.0.2j-fips configured -- resuming normal operations

2017:05:31-11:00:08 abcpublic reverseproxy: [Wed May 31 11:00:08.003427 2017] [core:notice] [pid 5459:tid 4147439296] AH00094: Command line: '/usr/apache/bin/httpd'

2017:05:31-11:01:21 abcpublic reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="0" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="433" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"

2017:05:31-11:02:02 abcpublic reverseproxy: [Wed May 31 11:02:02.049495 2017] [mpm_worker:notice] [pid 5459:tid 4147439296] AH00295: caught SIGTERM, shutting down

2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001045 2017] [proxy_protocol:notice] [pid 6317:tid 4148152000] ProxyProtocol: disabled on 127.0.0.1:4080

2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001212 2017] [security2:notice] [pid 6317:tid 4148152000] ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001222 2017] [security2:notice] [pid 6317:tid 4148152000] ModSecurity: APR compiled version="1.5.1"; loaded version="1.5.1"

2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001229 2017] [security2:notice] [pid 6317:tid 4148152000] ModSecurity: PCRE compiled version="7.8 "; loaded version="7.8 2008-09-05"

2017:05:31-11:02:18 abcpublic reverseproxy: [Wed May 31 11:02:18.001233 2017] [security2:notice] [pid 6317:tid 4148152000] ModSecurity: LIBXML compiled version="2.7.6"

2017:05:31-11:02:19 abcpublic reverseproxy: [Wed May 31 11:02:19.000212 2017] [proxy_protocol:notice] [pid 6325:tid 4148152000] ProxyProtocol: disabled on 127.0.0.1:4080

2017:05:31-11:02:19 abcpublic reverseproxy: [Wed May 31 11:02:19.002154 2017] [mpm_worker:notice] [pid 6325:tid 4148152000] AH00292: Apache/2.4.10 (Unix) OpenSSL/1.0.2j-fips configured -- resuming normal operations

2017:05:31-11:02:19 abcpublic reverseproxy: [Wed May 31 11:02:19.002180 2017] [core:notice] [pid 6325:tid 4148152000] AH00094: Command line: '/usr/apache/bin/httpd'

2017:05:31-11:02:23 abcpublic reverseproxy: id="0299" srcip="127.0.0.1" localip="127.0.0.1" size="142" user="-" host="127.0.0.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="435" url="/lb-status" server="localhost" referer="-" cookie="-" set-cookie="-"

2017:05:31-11:05:38 abcpublic reverseproxy: id="0299" srcip="10.168.11.133" localip="10.168.11.189" size="1215" user="-" host="10.168.11.133" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="61772" url="/opt/" server="abcpublic.com" referer="-" cookie="AWSALB=dr15+izHNymddWqqtb5rK+iDDwpDrxNkT52VFClTqUubSprXHjJO88AEaZknuzkfVA7lfCCGLmcAaVtguDV7I8naQjiA3F78B82//0evm7pIAt7TBU7Pr6ri3yZP" set-cookie="AWSALB=dwCIXdvFhEGcM0bCVkCWSbKnmM0ikv4vA6jphUdP+Q7j5JZgjorUxoeV9CMkmUpoyMg3pQIZ2NgCCcj535k/yqD1F5Xx28FbstBp7h6gb0g6deTjFeBr59+P0eGx; Expires=Wed, 07 Jun 2017 15:05:38 GMT; Path=/"

2017:05:31-11:05:39 abcpublic reverseproxy: id="0299" srcip="10.168.11.133" localip="10.168.11.189" size="14370" user="-" host="10.168.11.133" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="9029" url="/opt/css/opt.css" server="abcpublic.com" referer="https://abc.com/opt/" cookie="AWSALB=dwCIXdvFhEGcM0bCVkCWSbKnmM0ikv4vA6jphUdP+Q7j5JZgjorUxoeV9CMkmUpoyMg3pQIZ2NgCCcj535k/yqD1F5Xx28FbstBp7h6gb0g6deTjFeBr59+P0eGx" set-cookie="AWSALB=3Fr49MOJVEbbNlGv0/9TXFCzS2qmUT2TJu37ruZehGw249dV52cR412oA8WdQHTMZCrrcsBXqA0Fva8wNR/fOvNRPatCGHeX0xF88sFqCHEgv4/VW6aUrFcmJoXZ; Expires=Wed, 07 Jun 2017 15:05:39 GMT; Path=/"

2017:05:31-11:05:39 abcpublic reverseproxy: id="0299" srcip="10.168.11.133" localip="10.168.11.189" size="13426" user="-" host="10.168.11.133" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="8753" url="/opt/css/uswds.css" server="abcpublic.com" referer="https://abc.com/opt/" cookie="AWSALB=dwCIXdvFhEGcM0bCVkCWSbKnmM0ikv4vA6jphUdP+Q7j5JZgjorUxoeV9CMkmUpoyMg3pQIZ2NgCCcj535k/yqD1F5Xx28FbstBp7h6gb0g6deTjFeBr59+P0eGx" set-cookie="AWSALB=6IwvFlfNPelT6Wh+zA12p4kJjDoM3VR/y8VAOmXnk/NPwOTAFKvu4+vuzSxjEkX9cGUAbLPdq9FvBFd/OcN/HeJDgLCTS6PzzY4QJkjAv8bh4XCWayl3kihyWCp8; Expires=Wed, 07 Jun 2017 15:05:40 GMT; Path=/"

2017:05:31-11:05:40 abcpublic reverseproxy: id="0299" srcip="10.168.11.133" localip="10.168.11.189" size="10395" user="-" host="10.168.11.133" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="13989" url="/opt/js/vendor/angular/angular-resource.js" server="abcpublic.com" referer="https://abc.com/opt/" cookie="AWSALB=dwCIXdvFhEGcM0bCVkCWSbKnmM0ikv4vA6jphUdP+Q7j5JZgjorUxoeV9CMkmUpoyMg3pQIZ2NgCCcj535k/yqD1F5Xx28FbstBp7h6gb0g6deTjFeBr59+P0eGx" set-cookie="AWSALB=3qqyNWTM51u7p9d8ZkwVaepZ5SLOJb0xWmwGmxbM/OUsqSrTK08Lgz7ibWaJfXDLzCtJ7JZVl9IMgIjLww59idEnh+V2CSmK2/JAS6W36rWOrZmefHNGgPPLZa14; Expires=Wed, 07 Jun 2017 15:05:40 GMT; Path=/"

The buttons only indicate the status of the real webservers. After a reboot, they are always orange because the WAF has not checked yet if the real webservers are reachable.

There are no requests before the manual restart of the WAF. So, my question is still: is the WAF reachable?

Best
 Sabine

No. As I have initially pointed out that it does not come back automatically and reverse proxy is still not functional.   It always needs to be manually enabled again.  

Ok, then please add the log lines from request to the WAF when the WAF is not reachable.

Please refer to my replies on this thread.  They are attached.