Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Thread Info
  • State Not Answered
  • Locked Locked
  • Replies 6 replies
  • Subscribers 38 subscribers
  • Views 22327 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

awarrenhttp high cpu usage?

JonathanH

I've seen twice now in the time I've been running XG (under two days) where the AwarrenHttp process eats up the entirety of my cpu's.

I have 4 CPU's and 4.5GB memory assigned to the virtual XG.

Currently I do not have any policies configured for malware inspection of FTP, HTTP or HTTPS. I have shut off WAF for inbound 443 to my ADFS server.

I do however run web content inspection on outbound traffic from around 5 systems. When it gets to high usage, my only option appears to be to reboot the VM. After a reboot it comes back up with minimal usage.

I'm still trying to find more information on this one, but wanted to ask if anyone had pointers for troubleshooting the "AwarrenHttp" process when it gets to this level of CPU usage? 

Example TOP output:

top - 20:54:53 up 1 day, 1:50, 1 user, load average: 7.44, 7.35, 7.32
Tasks: 338 total, 2 running, 336 sleeping, 0 stopped, 0 zombie
Cpu(s): 15.4%us, 12.2%sy, 0.0%ni, 37.3%id, 34.6%wa, 0.0%hi, 0.5%si, 0.0%st
Mem: 3081836k total, 1823252k used, 1258584k free, 7920k buffers
Swap: 1048572k total, 974496k used, 74076k free, 712620k cached

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1863 20 0 131m 50m 924 S 99.9 1.7 46:11.46 awarrenhttp



This thread was automatically locked due to age.
Parents
  • 0

    You need to take the awarrenhttp in to debug mode by firing the below command
    console> system diagnostics subsystems WebProxy debug on
    Debug for WebProxy subsystem is enabled.
    from the shell you can look at the awarrenhttp.log in debug mode to see if there is any suspicious logs
    xxxxxx_WP01_SFOS 15.01.0# tail -f /log/awarrenhttp.log

    copy to the debug log to a log file and attach it here You can also monitor the process using this command

    console>system diagnostics utilities process-monitor

  • 0 in reply to Kranthi
    The problem has not yet occurred, however it may have been a result of some of the policies I had set to "log". I've since turned several of them off, but I had a couple that were logging allowed traffic and not just failures.

    I did want to post a couple of these here, in the rotated awarrenhttp.log.0 file, I see may of these entries:
    1447383324.909463946 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447383324.909468092 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447383324.909472193 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor

    The non rotated file awarrenhttp.log shows:
    1447473668.772053197 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=5 stats: 0: 0 0%, 1: 2 40%, 2: 2 40%, 3: 1 20%,
    1447473729.348090116 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=8 stats: 0: 1 13%, 1: 2 25%, 2: 4 50%, 3: 1 13%,
    1447473787.034425946 [ 2515/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473788.788263364 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=6 stats: 0: 0 0%, 1: 2 33%, 2: 3 50%, 3: 1 17%,
    1447473815.564220894 [ 2514/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473815.624756469 [ 2513/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473848.984349056 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=10 stats: 0: 2 20%, 1: 4 40%, 2: 2 20%, 3: 2 20%,
    1447473897.642068160 [ 2514/ (nil)] http_parser_context.c:80 http_parser_context_error Unable to parse a http message of 1460 bytes [00 00 00 00 00 00 00 00 00 00] (HPE_INVALID_METHOD: invalid HTTP method)
    1447473897.666008311 [ 2515/ (nil)] http_parser_context.c:80 http_parser_context_error Unable to parse a http message of 1460 bytes [00 00 00 00 00 00 00 00 00 00] (HPE_INVALID_METHOD: invalid HTTP method)
    1447473909.516425292 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=5 stats: 0: 0 0%, 1: 2 40%, 2: 2 40%, 3: 1 20%,


    For now I will keep an eye out for the issue to repeat, and will capture the debug information when it does.

    Thanks again fro the reply.
Reply
  • 0 in reply to Kranthi
    The problem has not yet occurred, however it may have been a result of some of the policies I had set to "log". I've since turned several of them off, but I had a couple that were logging allowed traffic and not just failures.

    I did want to post a couple of these here, in the rotated awarrenhttp.log.0 file, I see may of these entries:
    1447383324.909463946 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447383324.909468092 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447383324.909472193 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor

    The non rotated file awarrenhttp.log shows:
    1447473668.772053197 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=5 stats: 0: 0 0%, 1: 2 40%, 2: 2 40%, 3: 1 20%,
    1447473729.348090116 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=8 stats: 0: 1 13%, 1: 2 25%, 2: 4 50%, 3: 1 13%,
    1447473787.034425946 [ 2515/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473788.788263364 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=6 stats: 0: 0 0%, 1: 2 33%, 2: 3 50%, 3: 1 17%,
    1447473815.564220894 [ 2514/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473815.624756469 [ 2513/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473848.984349056 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=10 stats: 0: 2 20%, 1: 4 40%, 2: 2 20%, 3: 2 20%,
    1447473897.642068160 [ 2514/ (nil)] http_parser_context.c:80 http_parser_context_error Unable to parse a http message of 1460 bytes [00 00 00 00 00 00 00 00 00 00] (HPE_INVALID_METHOD: invalid HTTP method)
    1447473897.666008311 [ 2515/ (nil)] http_parser_context.c:80 http_parser_context_error Unable to parse a http message of 1460 bytes [00 00 00 00 00 00 00 00 00 00] (HPE_INVALID_METHOD: invalid HTTP method)
    1447473909.516425292 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=5 stats: 0: 0 0%, 1: 2 40%, 2: 2 40%, 3: 1 20%,


    For now I will keep an eye out for the issue to repeat, and will capture the debug information when it does.

    Thanks again fro the reply.
Children
No Data
Unfiltered HTML