Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Thread Info
  • State Not Answered
  • Locked Locked
  • Replies 6 replies
  • Subscribers 38 subscribers
  • Views 22324 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

awarrenhttp high cpu usage?

JonathanH

I've seen twice now in the time I've been running XG (under two days) where the AwarrenHttp process eats up the entirety of my cpu's.

I have 4 CPU's and 4.5GB memory assigned to the virtual XG.

Currently I do not have any policies configured for malware inspection of FTP, HTTP or HTTPS. I have shut off WAF for inbound 443 to my ADFS server.

I do however run web content inspection on outbound traffic from around 5 systems. When it gets to high usage, my only option appears to be to reboot the VM. After a reboot it comes back up with minimal usage.

I'm still trying to find more information on this one, but wanted to ask if anyone had pointers for troubleshooting the "AwarrenHttp" process when it gets to this level of CPU usage? 

Example TOP output:

top - 20:54:53 up 1 day, 1:50, 1 user, load average: 7.44, 7.35, 7.32
Tasks: 338 total, 2 running, 336 sleeping, 0 stopped, 0 zombie
Cpu(s): 15.4%us, 12.2%sy, 0.0%ni, 37.3%id, 34.6%wa, 0.0%hi, 0.5%si, 0.0%st
Mem: 3081836k total, 1823252k used, 1258584k free, 7920k buffers
Swap: 1048572k total, 974496k used, 74076k free, 712620k cached

PID PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1863 20 0 131m 50m 924 S 99.9 1.7 46:11.46 awarrenhttp



This thread was automatically locked due to age.
  • 0

    You need to take the awarrenhttp in to debug mode by firing the below command
    console> system diagnostics subsystems WebProxy debug on
    Debug for WebProxy subsystem is enabled.
    from the shell you can look at the awarrenhttp.log in debug mode to see if there is any suspicious logs
    xxxxxx_WP01_SFOS 15.01.0# tail -f /log/awarrenhttp.log

    copy to the debug log to a log file and attach it here You can also monitor the process using this command

    console>system diagnostics utilities process-monitor

  • 0 in reply to Kranthi
    Thanks, I'll work on trying to capture this information tonight if the problem surfaces again.
  • 0 in reply to Kranthi
    The problem has not yet occurred, however it may have been a result of some of the policies I had set to "log". I've since turned several of them off, but I had a couple that were logging allowed traffic and not just failures.

    I did want to post a couple of these here, in the rotated awarrenhttp.log.0 file, I see may of these entries:
    1447383324.909463946 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447383324.909468092 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447383324.909472193 [ 2616/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor

    The non rotated file awarrenhttp.log shows:
    1447473668.772053197 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=5 stats: 0: 0 0%, 1: 2 40%, 2: 2 40%, 3: 1 20%,
    1447473729.348090116 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=8 stats: 0: 1 13%, 1: 2 25%, 2: 4 50%, 3: 1 13%,
    1447473787.034425946 [ 2515/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473788.788263364 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=6 stats: 0: 0 0%, 1: 2 33%, 2: 3 50%, 3: 1 17%,
    1447473815.564220894 [ 2514/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473815.624756469 [ 2513/ (nil)] dns.c:229 dns_active_epoll DNS reloaded
    1447473848.984349056 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=10 stats: 0: 2 20%, 1: 4 40%, 2: 2 20%, 3: 2 20%,
    1447473897.642068160 [ 2514/ (nil)] http_parser_context.c:80 http_parser_context_error Unable to parse a http message of 1460 bytes [00 00 00 00 00 00 00 00 00 00] (HPE_INVALID_METHOD: invalid HTTP method)
    1447473897.666008311 [ 2515/ (nil)] http_parser_context.c:80 http_parser_context_error Unable to parse a http message of 1460 bytes [00 00 00 00 00 00 00 00 00 00] (HPE_INVALID_METHOD: invalid HTTP method)
    1447473909.516425292 [ 2513/ (nil)] epoll.c:522 output_event_threads_stats Open Sessions=5 stats: 0: 0 0%, 1: 2 40%, 2: 2 40%, 3: 1 20%,


    For now I will keep an eye out for the issue to repeat, and will capture the debug information when it does.

    Thanks again fro the reply.
  • 0 in reply to Kranthi
    Ok here you go, it appears to happened again. The log is filling up with the following at a very high rate:

    1447535638.125233020 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125237466 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125241183 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125244990 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125248656 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125252478 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125256149 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125259935 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125263595 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
    1447535638.125267327 [ 2516/ (nil)] dns.c:613 dns_poll dns: Bad file descriptor
  • 0

    Under CLI option 4 please run this command
    I need to know the Build you are on
    > system diagnostics show version-info and post the output.

    Under the advanced shell please fire this and see if there is an output for this command  only when you see a high CPU on awarren http
    "grep Open /log/awarrenhttp.log | tail -n 10"

    Thanks,
    Kranthi

  • 0 in reply to Kranthi

    Since the last report I have built a new VM XG because I was running a trial license. I have rebuilt, activated as home, and restored my backup from the first XG. No repeat so far.

    I did however fire back up the old appliance so  you could have the version information:

    I will let you know if the problem occurs on the new appliance.

    For future reference, here is my new appliance version info as it stands today:

    console> system diagnostics show version-info

    Serial Number: xxxxxxx
    Device-Id: xxxxxx
    Appliance Model: SFVH
    Firmware Version: SFOS 15.01.0
    Firmware Build: 376
    Firmware Loader version: 0x00000005
    HW version: VM01
    Config DB version: 15.045
    Signature DB version: 15.045
    Report DB version: 15.045
    Webcat Signature version: 0.0.0.76
    Web Proxy version: HTTP-Proxy.b66befa10
    SMTP Proxy version: 1.0.6.4
    POP/IMAP Proxy version: 1.0.0.3.4
    Logging Daemon version: 0.0.0.17
    AP Firmware: 3.0.001
    ATP: 1.0.0039
    Avira AV: 1.0.0040
    Authentication Clients: 1.0.0013
    IPS and Application signatures: 3.12.48
    RED Firmware: 1.0.004
    Sophos AV: 1.0.0040
    SSLVPN Clients: 1.0.004
    WAF: 1.0.0006
    Hot Fix version: 1

    console>

Unfiltered HTML