Sophos AV update failed - broken all web access when malware enabled - This keeps happening

I have had this problem a few times before, and the only way I've been able to fix it is to completely re-image the device and restore from backup.  Until this issue is fixed, there must be an easier way.

 

This has been shown in a previous thread - https://community.sophos.com/products/xg-firewall/f/sophos-xg-firewall-general-discussion/86815/proxy-broken-with-latest-malware-av-pattern-update---lost-all-web-access/321434#321434

 

I have malware enabled on my FW rule to handle endpoints, using the sophos engine.  Sometimes the sophos AV pattern update fails.  When this happens, it breaks access to the web, with the proxy returning a 500 response to all requests.

If I remove malware checking from the FW rule and/or change the malware engine - it fixes web access.   However, I cannot get the sophos AV to update, and thus remains broken.

 

I am currently on SFOS 16.05.0 GA

The most recent patten update showing the following....

Sophos AV
1.0.10583
-
20:56:17, Mar 02 2017
Failed

This is the tail from the up2date_av.log

2017-03-05 04:56:31 AM: applying incremental update update
2017-03-05 04:56:31 AM: updating /sdisk/savi/engine signatures
2017-03-05 04:56:31 AM: updating /sdisk/savi/vdl signatures
2017-03-05 04:56:33 AM: New savi full update Failed
2017-03-05 06:56:31 AM: Got the lock for updating savi (savi_10576-10596.tar.gz
)
2017-03-05 06:56:31 AM: applying incremental update update
2017-03-05 06:56:31 AM: updating /sdisk/savi/engine signatures
2017-03-05 06:56:31 AM: updating /sdisk/savi/vdl signatures
2017-03-05 06:56:33 AM: New savi full update Failed
2017-03-05 08:56:31 AM: Got the lock for updating savi (savi_10577-10597.tar.gz
)
2017-03-05 08:56:31 AM: applying incremental update update
2017-03-05 08:56:31 AM: updating /sdisk/savi/engine signatures
2017-03-05 08:56:31 AM: updating /sdisk/savi/vdl signatures
2017-03-05 08:56:34 AM: New savi full update Failed
2017-03-05 10:56:31 AM: Got the lock for updating savi (savi_10577-10597.tar.gz
)
2017-03-05 10:56:31 AM: applying incremental update update
2017-03-05 10:56:31 AM: updating /sdisk/savi/engine signatures
2017-03-05 10:56:31 AM: updating /sdisk/savi/vdl signatures
2017-03-05 10:56:33 AM: New savi full update Failed
2017-03-05 12:56:30 PM: Got the lock for updating savi (savi_10577-10597.tar.gz
)
2017-03-05 12:56:30 PM: applying incremental update update
2017-03-05 12:56:30 PM: updating /sdisk/savi/engine signatures
2017-03-05 12:56:31 PM: updating /sdisk/savi/vdl signatures
2017-03-05 12:56:33 PM: New savi full update Failed

 

Is there a way to clear the current AV pattern so it can update, without the need to reinstall the image????  This happens at least once a month and is causing real issues.

 

Any help greatly appreciated.

 

  • Michael,

    this should not occur one per month. Something is broken on your appliance. In my case, it happened 2 times this year. Did you try to rename the folder as suggested from this thread?

    https://community.sophos.com/products/xg-firewall/f/sophos-xg-firewall-general-discussion/73626/avira-up2date-error-is-there-any-solution#pi2132219853=4

    If it does not work, open a ticket with Support.

    Regards

  • In reply to lferrara:

    I've only started using this platform since late last year - so given we have only past February - it has indeed happened every month.  :D

    Thanks for the link - let me go through it and try and I'll reply back.

     

    Mike.

  • In reply to lferrara:

    Hi lferrara...

    I've read the thread as well as my original thread.

    Looking more into the logs - it does seem to be the exact same problem I had last time....

    Sun Mar 05 16:56:31 2017 Download for file savi_1.00_1.0.10599_fdiff20.tar.gz.g
    pg passed integrity and gpg checks
    Sun Mar 05 16:56:31 2017 Either FILE or MSID received in U2DVERSION is blank, s
    avi_10579-10599.tar.gz,
    Sun Mar 05 16:56:31 2017 Current savi patterns are at /content/savi_1.00/1.0.10
    583
    Sun Mar 05 16:56:31 2017 New updated patterns are now at /content/savi_1.00/1.
    0.10599
    Sun Mar 05 16:56:33 2017 Callback u2d_pt_installed failed for savi, version = 1
    .0.10599.
    Sun Mar 05 16:56:33 2017 Setting status 'fail' in DB and reverting link for sav
    i to old version = 1.0.10583.
    Sun Mar 05 16:56:33 2017 savi patterns are again at /content/savi_1.00/1.0.1058
    3

    You mentioned last time to rename the content folder (At least I think). I hate to be a pain, but  can you be explicit in what folder to rename - and to what?

    Only because last time I renamed the folder, it killed my whole platform and I had to re-image - so clearly I misunderstood something. 

    Here is the listing of my /contents/savi_1.00/

     

    SFVH_SO01_SFOS 16.05.0 GA# cat U2DVERSION

    FILE=savi_10582-10583.tar.gz

    CV=1.00

    VERSION=1.0.10583

    TYPE=immdiff

    MSID=

    SFVH_SO01_SFOS 16.05.0 GA# cd ..

    SFVH_SO01_SFOS 16.05.0 GA# cd savi_1.00/

    SFVH_SO01_SFOS 16.05.0 GA# ls -l

    drwxr-xr-x    2 root     0             1024 Mar  2 20:55 1.0.10583

    drwxr-xr-x    2 root     0             1024 Mar  3 02:56 1.0.10584

    drwxr-xr-x    2 root     0             1024 Mar  3 06:56 1.0.10585

    drwxr-xr-x    2 root     0             1024 Mar  3 08:56 1.0.10586

    drwxr-xr-x    2 root     0             1024 Mar  3 12:55 1.0.10587

    drwxr-xr-x    2 root     0             1024 Mar  3 18:55 1.0.10588

    drwxr-xr-x    2 root     0             1024 Mar  3 22:55 1.0.10589

    drwxr-xr-x    2 root     0             1024 Mar  4 04:56 1.0.10590

    drwxr-xr-x    2 root     0             1024 Mar  4 08:56 1.0.10591

    drwxr-xr-x    2 root     0             1024 Mar  4 16:55 1.0.10592

    drwxr-xr-x    2 root     0             1024 Mar  4 20:56 1.0.10593

    drwxr-xr-x    2 root     0             1024 Mar  5 00:56 1.0.10594

    drwxr-xr-x    2 root     0             1024 Mar  5 04:56 1.0.10595

    drwxr-xr-x    2 root     0             1024 Mar  5 06:56 1.0.10596

    drwxr-xr-x    2 root     0             1024 Mar  5 14:56 1.0.10597

    drwxr-xr-x    2 root     0             1024 Mar  5 16:56 1.0.10599

    SFVH_SO01_SFOS 16.05.0 GA#

     

    Thanks again for all your help!

     

    - Mike

  • In reply to Michael Allen:

    "mv /content/u2d/pattern /content/u2d/pattern.org
    This will rename the pattern file to pattern.org.
    Now update the pattern files with the GUI using System > Administration > Updates.
    Give the firewall some time to succeed the update process."

    From the thread.

    It worked for me last time without break XG.

  • In reply to lferrara:

    Thanks heaps for the quick reply - I'll try tonight and report results.

  • In reply to Michael Allen:

    I forgot to mention.   I tried this is it work perfectly.  Thanks again for your help!

  • In reply to lferrara:

    I'm chiming in as i just had this SAME problem after upgrading to MR3.

     

    but the solution does not work, my pattern file has 0 bytes  (and it's a file, not a directory)

    CR15iNG_AM02_SFOS 16.05.3 MR-3# ls -l
    drwxr-xr-x    2 root     0             1024 Apr  6 17:47 downloads
    -rw-r--r--    1 root     0                0 Apr  6 17:31 dr
    -rw-r--r--    1 root     0                0 Apr  6 17:46 firmware
    -rw-r--r--    1 root     0                0 Apr  6 17:47 pattern
    -rw-r--r--    1 root     0                0 Apr  6 17:36 pattern.org

    and the update keeps failing.

    the u2d log says:

    Thu Apr 06 17:47:32 2017 Download for file savi_1.00_1.0.10764_fdiff20.tar.gz.gpg passed integrity and gpg checks
    Thu Apr 06 17:47:32 2017 Either FILE or MSID received in U2DVERSION is blank, savi_10738-10764.tar.gz,
    Thu Apr 06 17:47:32 2017 Current savi patterns are at /content/savi_1.00/1.0.10762
    Thu Apr 06 17:47:32 2017 New updated  patterns are now at /content/savi_1.00/1.0.10764
    Thu Apr 06 17:47:35 2017 Callback u2d_pt_installed failed for savi, version = 1.0.10764.
    Thu Apr 06 17:47:35 2017 Setting status 'fail' in DB and reverting link for savi to old version = 1.0.10762.
    Thu Apr 06 17:47:35 2017 savi patterns are again at /content/savi_1.00/1.0.10762

     

    and the up2date_av:

    2017-04-06 05:47:33 PM: Got the lock for updating savi (savi_10738-10764.tar.gz)
    2017-04-06 05:47:33 PM: applying incremental update update
    2017-04-06 05:47:33 PM: updating /sdisk/savi/engine signatures
    2017-04-06 05:47:33 PM: updating /sdisk/savi/vdl signatures
    2017-04-06 05:47:35 PM: New savi full update Failed

    i'm not clear on that directory rename thing what directory?

  • In reply to Mast_01:

    I feel you're pain buddy.

     

    This keeps happening to me.  For the 4th time just today.

     

    As a first step for a quick work around, change the malware engine from sophos to Avira - in the GUI under System Service  --> Malware Protection.  That should temporarily fix the problem.

     

    It does look like the same problem that happens to me.  However that solution does indeed work for me.  If rename that pattern file, it should start to work.

    Just run the command mv /content/u2d/pattern /content/u2d/pattern.old

    Once you click update the patterns in the GUI again, that pattern file should be recreated. 

    However, you do have to wait for the actual pattern file to update.  Usually takes a few hours... however it should hopefully eventually work.

     

    Once it has updated and installed successfully, don't forget to change back to using the sophos malware engine.

     

    This is clearly a bug I hope sophos will fix.   I am running MR2 release, but I just noticed MR3 is now available.  So hopefully this bug has been addressed in the new release.

     

     

     

  • In reply to Michael Allen:

    "System Service  --> Malware Protection.  " ¿¿where is that?, i can't find it on my SF16. the closest is web->protection but i can't find even chose if i want dual scanning let alone the engine

    the pattern rename is not fixing it for me, also, this issue happened because of the MR3 update so don't get your hopes up

  • In reply to Mast_01:

    Are you running XG Firewall or UTM?

    I've never used the UTM device, so can't help there.  But on the XG, you should be able to see in the screen shot below to change the malware engine. 

     

  • In reply to Michael Allen:

    i'm using XG but that tab does not appear on any of the XG devices i've installed

  • In reply to Mast_01:

    Mast_01,

    go back to previous firmware and see if the issue persists. If not, I advice you to download the configuration and do a clean reinstallation using XG v15 MR-3 ISO directly.

    Let us know.

    Thanks

  • In reply to lferrara:

    isn't going back to an old firmware reset the config?, last times i did that it reset config and/or IP, and even the chance of that happening is not acceptable

    reimaging also is not possible

     

  • In reply to Mast_01:

    I agree that losing your config isn't ideal... but you can backup you config. Goto Backup and Firmware then Backup and Restore.  Back you config to a file off box.  If you lose everything after rolling back, your can restore from backup.  However, sometimes that is still difficult if you need to try and connect again to do the initial setup.

     

    I can also mention that even removing the pattern file didn't work for me this time.  The update failed again.  I am trying the same process again and see if it fixes it again.

    This is becoming very annoying.  I could never put these into production environment as they are now.  Sticking with the Ironports for now  :P

     

     

  • In reply to Michael Allen:

    Michael,

    it's beyond "not ideal" i simply CAN'T lose the config in this box as it's located on a very remote location with no support staff there and travel is simply not feasible for this, so i can't reset, can't restore, can't do anything that will bring the box down in any way or shape, this needs to work as it is now.

     

    also this is not a config problem, NONE of my XG boxes have the AV engine option so it must be a x86 thing and these small CR15's must be an ARM or something of the sort that don't support the other.

     

    edit: seems the problem fixed itself with a new signature as now i have success in the sigs