Thread Info
  • State Suggested Answer
  • Locked Locked
  • Replies 14 replies
  • Answers 1 answer
  • Subscribers 7 subscribers
  • Views 13202 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

HTTP Proxy failures - general oddness

BC68

For the past several weeks my UTM has been acting flaky - Issues with the WiFi (AP55) staying connected, web not responding, email quarantine release website giving errors... just general not behaving. I've tried clearing out a few things, turned off EndPoint protect (which as a problem, but resetting my UTM ID seemed to have fixed it), and rebooted. But I'm still getting issues.

I've been sorting through logs trying to see if I can figure out what's going on. I was having some WiFi issues before, but I found some lose connection on my patch panel so i attributed it to that - however after correcting that issue still having some connectivity issues - I'm not suspecting any issues with the AP55 at this point though.

Right now I get an email ~ every hour:

INFO-141] Http proxy not running - restarted

Http proxy not running - restarted
--
System Uptime      : 9 days 15 hours 46 minutes
System Load        : 5.69
System Version     : Sophos UTM 9.508-10

Digging through the log's I'm seeing:

Kernel Log:

2018:04:02-09:20:04 utm kernel: [837290.810516] ConfdReload[60002]: segfault at 33f0e996 ip 00000000f73a9103 sp 00000000e4e7ae70 error 4 in libtcmalloc.so.4.1.0[f7382000+48000]
2018:04:02-09:21:08 utm kernel: [837355.307681] ConfdReload[60674]: segfault at 30459 ip 00000000f759ae08 sp 00000000e4e25bd0 error 6 in libcrypto.so.1.0.0[f74a8000+1b4000]
2018:04:02-09:25:22 utm kernel: [837608.670408] ConfdReload[61080]: segfault at 33f0e996 ip 00000000f739d103 sp 00000000e4e6ee70 error 4 in libtcmalloc.so.4.1.0[f7376000+48000]
2018:04:02-09:29:13 utm kernel: [837839.989362] ConfdReload[61823]: segfault at 0 ip 00000000f7379103 sp 00000000e4e4afe0 error 4 in libtcmalloc.so.4.1.0[f7352000+48000]
2018:04:02-09:36:15 utm kernel: [ 217.193988] ConfdReload[8439]: segfault at 8070455 ip 00000000f686e5ba sp 00000000e4ea68ac error 7 in libc-2.11.3.so[f67f2000+16c000]
2018:04:02-09:36:52 utm kernel: [ 254.219874] ConfdReload[8906]: segfault at 30459 ip 00000000f75e1e08 sp 00000000e4e6cbd0 error 6 in libcrypto.so.1.0.0[f74ef000+1b4000]
2018:04:02-09:38:08 utm kernel: [ 330.497729] ConfdReload[9258]: segfault at 9060455 ip 00000000f73d9b96 sp 00000000e4e99c40 error 4 in libtcmalloc.so.4.1.0[f73a1000+48000]
2018:04:02-09:42:27 utm kernel: [ 588.853698] Confd[9414]: segfault at 6 ip 00000000f7392697 sp 00000000f64aab80 error 4 in libtcmalloc.so.4.1.0[f7359000+48000]
2018:04:02-09:43:25 utm kernel: [ 647.012638] ConfdReload[10419]: segfault at 9007375 ip 00000000f735eb96 sp 00000000e4e1e950 error 4 in libtcmalloc.so.4.1.0[f7326000+48000]

 

Self monitoring log:

2018:04:02-10:03:46 utm selfmonng[4917]: W child returned status: exit='0' signal='0'
2018:04:02-10:04:06 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 1 - 3
2018:04:02-10:04:11 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 2 - 3
2018:04:02-10:04:16 utm selfmonng[4917]: W check Failed increment httpproxy_running counter 3 - 3
2018:04:02-10:04:16 utm selfmonng[4917]: W NOTIFYEVENT Name=httpproxy_running Level=INFO Id=141 suppressed
2018:04:02-10:04:16 utm selfmonng[4917]: W triggerAction: 'cmd'
2018:04:02-10:04:16 utm selfmonng[4917]: W actionCmd(+):  '/var/mdw/scripts/httpproxy restart'
2018:04:02-10:04:18 utm selfmonng[4917]: W child returned status: exit='0' signal='0'
2018:04:02-10:04:48 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 1 - 3
2018:04:02-10:04:53 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 2 - 3
2018:04:02-10:04:58 utm selfmonng[4917]: W check Failed increment httpproxy_running counter 3 - 3
2018:04:02-10:04:58 utm selfmonng[4917]: W NOTIFYEVENT Name=httpproxy_running Level=INFO Id=141 suppressed
2018:04:02-10:04:58 utm selfmonng[4917]: W triggerAction: 'cmd'
2018:04:02-10:04:58 utm selfmonng[4917]: W actionCmd(+):  '/var/mdw/scripts/httpproxy restart'
2018:04:02-10:05:00 utm selfmonng[4917]: W child returned status: exit='0' signal='0'
2018:04:02-10:07:10 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 1 - 3
2018:04:02-10:07:15 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 2 - 3
2018:04:02-10:07:20 utm selfmonng[4917]: W check Failed increment httpproxy_running counter 3 - 3
2018:04:02-10:07:20 utm selfmonng[4917]: W NOTIFYEVENT Name=httpproxy_running Level=INFO Id=141 suppressed
2018:04:02-10:07:20 utm selfmonng[4917]: W triggerAction: 'cmd'
2018:04:02-10:07:20 utm selfmonng[4917]: W actionCmd(+):  '/var/mdw/scripts/httpproxy restart'
2018:04:02-10:07:57 utm selfmonng[4917]: W child returned status: exit='0' signal='0'
2018:04:02-10:10:37 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 1 - 3
2018:04:02-10:10:42 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 2 - 3
2018:04:02-10:10:47 utm selfmonng[4917]: W check Failed increment httpproxy_running counter 3 - 3
2018:04:02-10:10:47 utm selfmonng[4917]: W NOTIFYEVENT Name=httpproxy_running Level=INFO Id=141 suppressed
2018:04:02-10:10:47 utm selfmonng[4917]: W triggerAction: 'cmd'
2018:04:02-10:10:47 utm selfmonng[4917]: W actionCmd(+):  '/var/mdw/scripts/httpproxy restart'
2018:04:02-10:10:53 utm selfmonng[4917]: W child returned status: exit='0' signal='0'
2018:04:02-10:13:23 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 1 - 3
2018:04:02-10:13:28 utm selfmonng[4917]: I check Failed increment httpproxy_running counter 2 - 3
2018:04:02-10:13:33 utm selfmonng[4917]: W check Failed increment httpproxy_running counter 3 - 3
2018:04:02-10:13:33 utm selfmonng[4917]: W NOTIFYEVENT Name=httpproxy_running Level=INFO Id=141 suppressed
2018:04:02-10:13:33 utm selfmonng[4917]: W triggerAction: 'cmd'
2018:04:02-10:13:33 utm selfmonng[4917]: W actionCmd(+):  '/var/mdw/scripts/httpproxy restart'
2018:04:02-10:13:35 utm selfmonng[4917]: W child returned status: exit='0' signal='0'

When I get quarantine report and click on it via e-mail I get this:
  

I do have a wildcard SSL cert applied, however it works no problem when accessing either the admin page nor the user portal (both produce green bars):
 


The UTM is running on an ESXi 6.0 (on an old dell R710), the VM has 8GB of RAM, 200GB HDD and 4 CPU's - all hardware indicator's seen to be fine:
 

One thing to note, since I have the home license I'm getting close to the 50 max - have about 45 devices (yeah, I know... between gaming consoles, computers, roku's, voip, work laptops, kid's school ipads, cell phones... we got lot's of crap), so I'm getting low on licenses but the hardware spec's shouldn't have any issues handling it.

Any suggestions, help, chicken sacrifices, etc.. would be greatly appreciated.


This thread was automatically locked due to age.
Parents
  • 0

    This proxy crash is not a problem seen here frequently, but there does seem to be an issue that several people have reported in the last month.  I think at least two of the other reports were on VMs.

    First (it's a long shot), try restoring a backup from 9.506 made just before the 507 and 508 Up2Dates were applied.  If that doesn't work, try installing a new VM from ISO using 64-bit SUSE Enterprise Linux.  Any luck with that?

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • 0 in reply to BAlfson

    Are you saying to restore the just config backup from 9.506 on the current 9.508 OS? Or restore using 9.506 OS?

    Also, the current VM is configured as SUSE Enterprise x64

     

     

    Or are you just saying to make a new VM install the OS & restore the config on that?

     

    Thanks!

  • 0 in reply to BC68

    Although it's rare, sometimes the Up2Date process will damage the configuration.  When something unusual happens, my first suggestion is to try a restore in place.  If that doesn't resolve the issue, restore the latest backup to a newly-generated system or VM.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
Reply
  • 0 in reply to BC68

    Although it's rare, sometimes the Up2Date process will damage the configuration.  When something unusual happens, my first suggestion is to try a restore in place.  If that doesn't resolve the issue, restore the latest backup to a newly-generated system or VM.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
Children
  • 0 in reply to BAlfson

    So I've resolved part of the issue in which the Quarantine report wasn't working due to SSL errors. I had to edit httpd-spam.conf and insert the SSL cert info as somehow it got removed from the config file. 

    I am still getting the HTTP proxy not running issue as I haven't figured it out yet - I'd really like to avoid the Microsoft approach of troubleshooting "Have you rebooted? OK, then try reinstalling". 

  • 0 in reply to BC68

    There is a definite bug in dealing with the Email Quarantine page. I was making some WAF changes and had to change some information on my Webadmin page (for port 443 stuff) and the quarantine release page broke again. 

     

    I had to go back into the httpd-spam.conf file and change

    SSLEngine Off

    to

    SSLEngine On
    SSLCertificateFile /etc/httpd/WebAdminCert.pem
    SSLCertificateKeyFile /etc/httpd/WebAdminKey.pem
    SSLCACertificateFile /etc/httpd/WebAdminCertCA.pem

    and it's working again. 

  • 0 in reply to BC68

    Does the latest update from July 2018 (u2d-sys-9.509003-510004) solve this issue with http proxy constantly restarting?

  • 0 in reply to Catfriend

    Unknown, I wasn't aware of this release since apparently it's a soft release as of the 19th (so like 4 days ago) , and I haven't received any news about it until your post either :). 

     

    Do we know if this will be pushed out via Up2Dates or if it requires a manual update?

  • 0 in reply to Catfriend

    Since I had a backup as of this weekend, and I have a backup connection for work I decided to give the 9.510 update a try. Came up OK, I didn't have to manually edit the  httpd-spam.conf to keep my SSL cert in there, and so far no restarts - however sometimes after a reboot it appeared "clean" for a few hours, so I'll watch and see. But so far so good.

  • 0 in reply to Catfriend

    Applying the 9.510 update seems to have resolved my issues. No longer getting http restarts, and not seeing segfault crashes any more. Overall the UTM is acting better. Wish it hadn't taken so long for the fix, but it seems to be working now.

  • 0 in reply to BC68

    As I look back in my inbox at HUNDREDS of notifications of "Http proxy not running - restarted", I noticed the last notification I got was about six weeks ago with Sophos UTM 9.509-3 and as I check my current version, I see that I'm still running 9.509-3 but haven't had the proxy failures.  I don't know what caused them and I have no idea what cleared them up, but I'm glad to notice I went from several notifications per day to none since June 14th.  

    I'd actually forgotten about this issue (which didn't seem to be negatively affecting anything, it was just a nagging concern that it was happening) until I read this message.  I haven't seen 9.510 available, but I'm happy to see the problem seems to be gone now.

  • 0 in reply to Sean Workman

    Interesting that it magically cleaned up for you... check your logs to see if it's still happening, just not getting e-mail notifications on it. It was causing issues for me, this patch seems to have cleared up lot's of minor issues - my wifi was pretty flaky (but I couldn't nail down the issue), and it seems to be better now as well. So at least for me this patch solved a lot of issues and makes things way better around here :)


    You can manually download and apply the patch - that's what I did. ftp://ftp.astaro.com/pub/UTM/v9/up2date/

Unfiltered HTML