Weird issue that has just started today. After awhille the proxy just seems to stop working than start again. According to the logs it is not restarting and top is showing activity. When the problem is starting to happen cpu usage spikes to 100% across all cores (httpproxy)
Below are the live logs as it starts to happen.
2015:04:23-14:03:39 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="(nil)" function="confd_network_filter" file="confd-client.c" line="1851" message="failed to get network: Operation not permitted (req=[{ "id": 3990, "method": "get_object", "params": [ "" ] }])"
2015:04:23-14:03:40 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="(nil)" function="confd_network_filter" file="confd-client.c" line="1851" message="failed to get network: Operation not permitted (req=[{ "id": 4482, "method": "get_object", "params": [ "" ] }])"
2015:04:23-14:03:45 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="(nil)" function="aptp_reload" file="aptpscanner.c" line="133" message="reloading ATP pattern"
2015:04:23-14:03:45 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="(nil)" function="aptp_reload" file="aptpscanner.c" line="151" message="reloading ATP pattern finished"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="(nil)" function="plain_write_vector" file="epoll.c" line="1098" message="Write error on the epoll handler 395 (Broken pipe)"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0001" severity="info" sys="SecureWeb" sub="http" request="0xd543c000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd543a800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="(nil)" function="confd_config_reload_func" file="confd-client.c" line="634" message="reloading config done, new version 1017"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd543d000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0001" severity="info" sys="SecureWeb" sub="http" request="0xa500000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd542c000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0001" severity="info" sys="SecureWeb" sub="http" request="0xa500800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd542b800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa4fe000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd58b2800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd540c800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xe49d5800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd5af3800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xe33d7800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd5af3000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xe33d7000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xe33d6800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd5af2000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xe33d6000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd5af0800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xe33d5000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa75f800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa75f000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd4b76800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa75e000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0001" severity="info" sys="SecureWeb" sub="http" request="0xa75d800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xe580b800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa43c800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa75c800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa43c000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd56f7800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa43b800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd56f7000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xa43a000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd5d15800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd58b0800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xdd397800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd52fa000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xdd396800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xdd396000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xdd395800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xd4eab800" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
2015:04:23-14:07:09 HOSTNAME_HERE httpproxy[22795]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0xdd395000" function="p0f_add_request" file="p0f-client.c" line="345" message="connect: Resource temporarily unavailable"
Things tried:
- Stopping Proxy, Clearing Cache, Restarting
- Rebooting VM
- During the outage if i bypass the proxy by going straight to the gateway (our edge router on dmz) using the same dns servers, it all works.
System CPU usage is usually in the 10-40% range; memory is only 40% of 8gb.
Proxy is in standard mode w/ http/s filtering only + AV scanning.
This thread was automatically locked due to age.