This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Configuration gone after upgrage SFOS 19.0.0 to 19.0.1

Hello everybody,

yesterday I updated two Firewalls (XGS 126 and XG 125) from Version 19.0.0 to 19.0.1. After the upgrade both Firewalls has SFOS 19.0.1 firmware installed but lost their configuration. The problem was both firewalls are on remote site and lost all external connections (Sophos Central und VPN), so I can't reach the firewalls. Today we connect on site via serial console to the Firewalls, and I saw that Firmware 19.0.1 was installed, but (at minimum) the network configuration was gone. I made some tests:

- Booting the 19.0.0 Firmware the Firewall runs as expected. 

- Booting the 19.0.1 Firmware via Bootloader or via WebAdmin Firewall has no configuration.

On the other hand, I made the upgrade on serval firewalls without any problem (2x XG 125, 1x XGS 5500 HA, a Virtual and a Software Firewall). 

How can I remove the 19.0.1 Firmware from the non-working firewalls to get a 2nd try to upload the firmware again and install the 19.0.1?

Thanks,

Ben



This thread was automatically locked due to age.
  • Hello @Ben@Network,

    Thank you for reaching out to the community, You can download the v19.0 GA from the following links:
    https://www.sophos.com/en-us/support/downloads/firewall-installers
    https://download.sophos.com/firmware/HW/index.html

    And then re-image the appliance: https://support.sophos.com/support/s/article/KB-000036812?language=en_US

    Then you can download firmware latest firmware 19.0.1  MR-1 from Sophos Licensing Portal: https://docs.sophos.com/nsg/sophos-firewall/18.5/Help/en-us/webhelp/onlinehelp/AdministratorHelp/BackupAndFirmware/Firmware/FirmwareDownloadFirmware/index.html

    Between after the firmware upgrade did your appliance went into the failsafe mode ? If that is the case then you may check the reason by following the article below:
    Know the cause of hardware appliance going in failsafe mode: https://support.sophos.com/support/s/article/KB-000036375?language=en_US

    Thanks & Regards,
    _______________________________________________________________

    Vivek Jagad | Team Lead, Global Support & Services 

    Log a Support Case | Sophos Service Guide
    Best Practices – Support Case


    Sophos Community | Product Documentation | Sophos Techvids | SMS
    If a post solves your question please use the 'Verify Answer' button.

  • Hello Vivek,

    Thanks for the answer. Since the firewall is not directly in my access, I did not want to install a new image on the firewall. With the version 19.0.0 the firewall runs without problems. However, now when I click on "Boot Firmware Image" on the 19.0.1 in WebAdmin, the firewall boots again with the broken configuration. I am now looking for a way to upload the 19.0.1 version again and have a 2nd attempt to boot a working 19.0.1.

    Is there a way that the firewall automatically reboots after a certain time and boots with the previous firmware version, unless the automatic reboot is cancelled by the administrator?

    Ben

    If a post solves your question please use the 'Verify Answer' button.

  • Hey Ben@Network ,

    SFOS provides you to slots of the firmware, where in you can switch between the two slots anytime without loosing the configurations. 
    So, before moving to the SFOS v19.01 MR-1 if you have the backup of v19.0.0 GA that is excellent. 

    And now that the configurations are broken on v19.01 MR-1 then that backup file will be handy. You can load the previous firmware using *SFLOADERhttps://docs.sophos.com/nsg/sophos-firewall/19.0/Help/en-us/webhelp/onlinehelp/AdministratorHelp/BackupAndFirmware/Firmware/FirmwareLoadFirmwareSFLoader/index.html

    Once you are able to start the v19.0.0 GA without appliance being broke or into the fail-safe mode then you may restore the backup with the file you saved. 

    *WARNING: The option to load firmware using SFLoader isn't available for XGS devices. To update corrupt firmware for XGS devices, see Reimage Sophos Firewall. OR the link shared previously !! 

    Thanks & Regards,
    _______________________________________________________________

    Vivek Jagad | Team Lead, Global Support & Services 

    Log a Support Case | Sophos Service Guide
    Best Practices – Support Case


    Sophos Community | Product Documentation | Sophos Techvids | SMS
    If a post solves your question please use the 'Verify Answer' button.

  • Check if both of those KIL Items are not affected: 

    __________________________________________________________________________________________________________________

  • If I understood your answer correctly, the only way to boot a running19.0.1 on the firewalls is to install a new image, update to 19.0.1 and import the 19.0.0 configuration backup? There is no more "remote" friendly way?

    If a post solves your question please use the 'Verify Answer' button.

  • Nope if it is broken or in a fail-safe mode then there is no other remote friendly way of restoring it !!
    You should take a look into the suggestion given by Ben@Network

    Thanks & Regards,
    _______________________________________________________________

    Vivek Jagad | Team Lead, Global Support & Services 

    Log a Support Case | Sophos Service Guide
    Best Practices – Support Case


    Sophos Community | Product Documentation | Sophos Techvids | SMS
    If a post solves your question please use the 'Verify Answer' button.

  • Hi LuCar Toni,

    I did not used a Cyberroam backup for the Firewalls. Both firewalls are SFOS firewalls starting with SFOS 18.x. The regport setting are the default values:


    The uprade from 18.5.3 to 19.0.0 works without any trouble.

    Ben

    If a post solves your question please use the 'Verify Answer' button.

  • Interesting. Could you review the /log/migration.log? 

    __________________________________________________________________________________________________________________

  • There are some database errors:  

    2022-08-02 14:49:39.768 GMT starting old version corporate db
    Starting conf database
    332 2022-08-02 14:49:40.553 GMTLOG: could not connect socket for statistics collector: Network is unreachable
    332 2022-08-02 14:49:40.553 GMTLOG: disabling statistics collector for lack of working socket
    334 2022-08-02 14:49:40.554 GMTLOG: database system was shut down at 2022-08-02 14:48:45 GMT
    332 2022-08-02 14:49:40.559 GMTLOG: database system is ready to accept connections
    2022-08-02 14:49:42.286 GMT
    2022-08-02 14:49:42.331 GMT : Database started after 0 seconds
    DROP SCHEMA
    UPDATE 3
    Stopping database
    332 2022-08-02 14:49:45.817 GMTLOG: received fast shutdown request
    332 2022-08-02 14:49:45.817 GMTLOG: aborting any active transactions
    335 2022-08-02 14:49:45.818 GMTLOG: shutting down
    335 2022-08-02 14:49:45.902 GMTLOG: database system is shut down
    2022-08-02 14:49:46.844 GMT : Database stopped after 1 seconds
    /sdisk/oldpgconfdump.sql is created
    Starting conf database
    377 2022-08-02 14:49:48.484 GMTLOG: could not connect socket for statistics collector: Network is unreachable
    377 2022-08-02 14:49:48.484 GMTLOG: disabling statistics collector for lack of working socket
    379 2022-08-02 14:49:48.485 GMTLOG: database system was shut down at 2022-07-19 19:35:51 GMT
    377 2022-08-02 14:49:48.493 GMTLOG: database system is ready to accept connections
    2022-08-02 14:49:50.463 GMT
    2022-08-02 14:49:50.468 GMT : Database started after 0 seconds
    DROP SCHEMA config CASCADE
    DROP SCHEMA
    DROP SCHEMA public CASCADE
    DROP SCHEMA
    DROP PROCEDURAL LANGUAGE plpgsql
    391 2022-08-02 14:49:52.959 GMTERROR: cannot drop language plpgsql because extension plpgsql requires it
    391 2022-08-02 14:49:52.959 GMTHINT: You can drop extension plpgsql instead.
    391 2022-08-02 14:49:52.959 GMTSTATEMENT: DROP PROCEDURAL LANGUAGE plpgsql
    ERROR: cannot drop language plpgsql because extension plpgsql requires it
    HINT: You can drop extension plpgsql instead.
    CREATE SCHEMA public
    CREATE SCHEMA
    psql:/sdisk/oldpgconfdump.sql:19183: WARNING: column "senderemail" has type "unknown"
    DETAIL: Proceeding with relation creation anyway.
    psql:/sdisk/oldpgconfdump.sql:19183: WARNING: column "receipientemail" has type "unknown"
    DETAIL: Proceeding with relation creation anyway.
    setval
    --------
    1263
    (1 row)

    setval
    --------
    1
    (1 row)

    setval
    --------
    1
    (1 row)

    setval
    --------
    1
    (1 row)

    setval
    --------
    1
    (1 row)

    setval
    --------
    1
    (1 row)

    setval
    --------
    592
    (1 row)

    setval
    --------
    1
    (1 row)

    380 2022-08-02 14:49:57.336 GMTLOG: checkpoints are occurring too frequently (9 seconds apart)
    380 2022-08-02 14:49:57.336 GMTHINT: Consider increasing the configuration parameter "checkpoint_segments".
    Stopping database
    377 2022-08-02 14:50:04.349 GMTLOG: received fast shutdown request
    377 2022-08-02 14:50:04.349 GMTLOG: aborting any active transactions
    380 2022-08-02 14:50:05.170 GMTLOG: shutting down
    380 2022-08-02 14:50:05.447 GMTLOG: database system is shut down
    2022-08-02 14:50:06.409 GMT : Database stopped after 2 seconds
    old conf to new conf migrated with return value :: 0
    2022-08-02 14:50:06.620 GMT starting migration log
    Starting conf database
    446 2022-08-02 14:50:06.781 GMTLOG: could not connect socket for statistics collector: Network is unreachable
    446 2022-08-02 14:50:06.781 GMTLOG: disabling statistics collector for lack of working socket
    448 2022-08-02 14:50:06.782 GMTLOG: database system was shut down at 2022-08-02 14:50:05 GMT
    446 2022-08-02 14:50:06.786 GMTLOG: database system is ready to accept connections
    2022-08-02 14:50:08.764 GMT
    2022-08-02 14:50:08.769 GMT : Database started after 0 seconds
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    INSERT 0 0
    UPDATE 1
    INSERT 0 1
    UPDATE 1
    nvram_get failed with -16
    Old version is 19.003 and currentversion is 19.004
    Database is upgrading to dbv19.004
    Check migration for version dbv19.004
    Applying migration for version dbv19.004
    1457 2022-08-02 14:50:11.569 GMTERROR: duplicate key value violates unique constraint "tblclientservices_pkey"
    1457 2022-08-02 14:50:11.569 GMTDETAIL: Key (servicekey)=(vpn_flush_conn_failover) already exists.
    1457 2022-08-02 14:50:11.569 GMTSTATEMENT: INSERT INTO tblclientservices (servicekey, servicevalue) VALUES ('vpn_flush_conn_failover', 'non_tcp');
    psql:/_conf/DB/dbv19.004/corporate.sql:44: ERROR: duplicate key value violates unique constraint "tblclientservices_pkey"
    DETAIL: Key (servicekey)=(vpn_flush_conn_failover) already exists.
    /bin/psql -1 -p 5432 -U pgroot -q -d corporate -f /_conf//DB/dbv19.004/corporate.sql Failed
    /bin/sh /_conf//DB/dbv19.004/migration.sh Failed
    UPDATE 1
    Stopping database
    446 2022-08-02 14:50:13.260 GMTLOG: received fast shutdown request
    446 2022-08-02 14:50:13.260 GMTLOG: aborting any active transactions
    449 2022-08-02 14:50:13.260 GMTLOG: shutting down
    449 2022-08-02 14:50:13.406 GMTLOG: database system is shut down
    2022-08-02 14:50:14.288 GMT : Database stopped after 1 seconds
    applymigration.sh exited with 1
    2022-08-02 14:50:35.111 GMT: Before mountconf unmount

    If a post solves your question please use the 'Verify Answer' button.

  • The same error on the XGS 126:    

    1510 2022-08-02 15:12:50.688 GMTERROR: duplicate key value violates unique constraint "tblclientservices_pkey"
    1510 2022-08-02 15:12:50.688 GMTDETAIL: Key (servicekey)=(vpn_flush_conn_failover) already exists.
    1510 2022-08-02 15:12:50.688 GMTSTATEMENT: INSERT INTO tblclientservices (servicekey, servicevalue) VALUES ('vpn_flush_conn_failover', 'non_tcp');
    psql:/_conf/DB/dbv19.004/corporate.sql:44: ERROR: duplicate key value violates unique constraint "tblclientservices_pkey"
    DETAIL: Key (servicekey)=(vpn_flush_conn_failover) already exists.
    /bin/psql -1 -p 5432 -U pgroot -q -d corporate -f /_conf//DB/dbv19.004/corporate.sql Failed
    /bin/sh /_conf//DB/dbv19.004/migration.sh Failed

    Last week I set this values on all firewalls:

    set vpn conn-remove-tunnel-up disable
    set vpn conn-remove-on-failover non-tcp

    If a post solves your question please use the 'Verify Answer' button.