This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Weird UTM freezes randomly approximately once a day ...

I have experienced a strange lockup on my "new" UTM box, but I checked log files and they don't reveal anything, just a bunch of weird characters ...

2023:03:16-01:32:01 escape75 /usr/sbin/cron[25494]: (root) CMD (  nice -n19 /usr/local/bin/gen_inline_reporting_data.plx)
2023:03:16-01:35:01 escape75 /usr/sbin/cron[25649]: (root) CMD (   /usr/local/bin/reporter/system-reporter.pl)
�����������������������������������������������������������������������������������������������������������
2023:03:16-09:03:10 escape75 syslog-ng[4942]: syslog-ng starting up; version='3.4.7' 2023:03:16-09:03:12 escape75 ddclient[5361]: WARNING: cannot connect to checkip.dyndns.org:80 socket: IO::Socket::INET: Bad hostname 'checkip.dyndns.org' 2023:03:16-09:03:24 escape75 system: System was restarted



So,- I've been running the software version of UTM (9.714) on my old unit (an XG115 r2) for a couple of years without any issues,
and recently I have migrated my saved config over to a new unit (XG115 r3) and a few hours after setting up the new unit (at night)

it froze up, and interfaces were not pingable (LAN) so I powered it down and rebooted. It's working again ...

Just wondering if there's something more I can look at to see what the issue was .. I have a hunch maybe it was DHCP related,
as my devices on the LAN were renewing the IP addresses and they were not in the table on the new unit, but it's a wild guess,
so if this doesn't happen again then maybe it's nothing to worry about.

I don't know if there would be an issue moving the config file (and license) from the old unit, but I wouldn't think so.

The new unit was installed the same way as the old unit, using the ssi-9.714-4.1.iso file and removing the /etc/asg with a software license,
and the old unit hasn't experienced any weird issues in years, and the ethernet ports and devices are setup in an identical way, nothing changed.

Just looking for thoughts and ideas ...

Stats from top:

top - 11:32:20 up 2:31, 1 user, load average: 0.09, 0.29, 0.25
Tasks: 163 total, 1 running, 160 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.6%us, 0.5%sy, 0.0%ni, 98.5%id, 0.1%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 3898468k total, 3558768k used, 339700k free, 111124k buffers
Swap: 4194300k total, 112k used, 4194188k free, 1352808k cached

Zombies:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 18256 0.0 0.0 0 0 ? Z 11:30 0:00 [aua.bin] <defunct>
root 18595 0.6 0.0 0 0 ? Z 11:32 0:00 [confd.plx] <defunct>



This thread was automatically locked due to age.
Parents
  • Hello there!

    Good day and thanks for reaching out to Sophos Community and hope you are well. 

    Wanted to check if this freeze happens often at night? or just a single occurence? 

    If ths happens often I may recommend you to open a support ticket to have this further checked, if just a single occurence we may want to observe and see if this would persist in the future. 

    Have a nice day ahead and thank you for choosing Sophos

    Cheers,

    Raphael Alganes
    Community Support Engineer | Sophos Technical Support
    Sophos Support Videos Product Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.

  • Unfortunately the reload using asg-9.714-4.1.iso instead of ssi-9.714-4.1.iso did not solve the freezing, I used the 64 bit kernel.

    I have performed a disk check and memory check and it passes without any error.
    When the unit is frozen, you can't ping the LAN interfaces, and USB keyboard doesn't respond in order to login.
    The freezing seems to happen randomly, it looks like it happens every day ...


    I am including all fail, error, invalid messages from boot log, I do not know if these are useful at all:

    2023:03:19-14:09:12 escape75 kernel: [    0.000000] ACPI BIOS Warning (bug): FADT (revision 6) is longer than ACPI 5.0 version, truncating length 276 to 268 (20130725/tbfadt-311)
    2023:03:19-14:09:12 escape75 kernel: [    0.000000] ACPI Error: Gpe0Block - 32-bit FADT register is too long (32 bytes, 256 bits) to convert to GAS struct - 255 bits max, truncating (20130725/tbfadt-202)

    2023:03:19-14:09:12 escape75 kernel: [    1.785222] TCP: TFO aes cipher alloc error: -2

    2023:03:19-14:09:12 escape75 kernel: [    5.318546] i801_smbus 0000:00:1f.1: can't derive routing for PCI INT A
    2023:03:19-14:09:12 escape75 kernel: [    5.318554] i801_smbus 0000:00:1f.1: PCI INT A: no GSI
    2023:03:19-14:09:12 escape75 kernel: [    5.318591] i801_smbus 0000:00:1f.1: Failed to allocate irq 255: -22
    2023:03:19-14:09:12 escape75 kernel: [    5.318598] i801_smbus: probe of 0000:00:1f.1 failed with error -22

    2023:03:19-14:09:12 escape75 kernel: [    0.004000] tsc: Fast TSC calibration failed
    2023:03:19-14:09:12 escape75 kernel: [    0.012000] tsc: PIT calibration matches HPET. 1 loops
    2023:03:19-14:09:12 escape75 kernel: [    0.012000] tsc: Detected 1592.856 MHz processor

    2023:03:19-14:09:12 escape75 kernel: [    0.199653] acpi PNP0A08:00: ACPI _OSC support notification failed, disabling PCIe ASPM
    2023:03:19-14:09:12 escape75 kernel: [    0.199661] acpi PNP0A08:00: Unable to request _OSC control (_OSC support mask: 0x08)

    2023:03:19-20:09:37 escape75 [daemon:notice] rrdcached[3786]:  queue_thread_main: rrd_update_r (/var/log/reporting/rrd/ips.rrd) failed with status -1. (/var/log/reporting/rrd/ips.rrd: illegal attempt to update using time 1679255404 when last update time is 1679256005 (minimum one second step))
    

    2023:03:19-14:09:12 escape75 kernel: [ 0.199505] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) 2023:03:19-14:09:12 escape75 kernel: [ 0.199579] \_SB_.PCI0:_OSC invalid UUID 2023:03:19-14:09:12 escape75 kernel: [ 0.199582] _OSC request data:1 8 0 2023:03:19-14:09:12 escape75 kernel: [ 0.199643] \_SB_.PCI0:_OSC invalid UUID

  • Are you able to install some kind of windows or linux OS to the device?  Maybe run something like prime95 or equiv to load it up.  Could be cooling issues (or lack there of) causes instability.

  • I don't know, it's a regular XG115 R3 from Sophos ...

    I would expect that there would be some kind of troubleshooting path, as this appears caused by my config possibly ...

    I wonder if I could load SFOS HW Firmware and get a trial license and would be able to tell within a few days if it freezes ...

  • Why would you need a trial license?  Just install it and let it run without importing your config.  I would start from scratch and see how that works, if it's still freezing.  If it's not after a few days, it might be something in the config.  If it freezes, it's probably hardware related. 

    OPNSense 64-bit | Intel Xeon 4-core v3 1225 3.20Ghz
    16GB Memory | 500GB SSD HDD | ATT Fiber 1GB
    (Former Sophos UTM Veteran, Former XG Rookie)

Reply
  • Why would you need a trial license?  Just install it and let it run without importing your config.  I would start from scratch and see how that works, if it's still freezing.  If it's not after a few days, it might be something in the config.  If it freezes, it's probably hardware related. 

    OPNSense 64-bit | Intel Xeon 4-core v3 1225 3.20Ghz
    16GB Memory | 500GB SSD HDD | ATT Fiber 1GB
    (Former Sophos UTM Veteran, Former XG Rookie)

Children
  • I didn't know I could run it without a license ...

  • Well UTM you can just use your existing license file or it comes with a 30-day trial, and XG assigns you one when you download it, and you can see it under Administration > Licensing. 

    OPNSense 64-bit | Intel Xeon 4-core v3 1225 3.20Ghz
    16GB Memory | 500GB SSD HDD | ATT Fiber 1GB
    (Former Sophos UTM Veteran, Former XG Rookie)

  • Perfect, that's what I'm doing right now, UTM hardware ISO using the 30 day trial, and a basic setup, this will test my XG115 R3.

    I have also re-loaded my XG115 R2 just to see if the other box exhibits the same issues, this should give me some more clues.

  • I tried XG115 R2 re-install and use the same config file I've had issues with on my R3, and it's been fine for over 24 hours.

    Then I tried a basic setup with R3, eth1 - WAN, and eth0, eth2, eth3 - LAN (br0) and it went down in about 15 minutes!

    Temperatures seem ok, and memory tested fine, and storage is also OK ... I'm at a loss!

    loginuser@escape75:/home/login > sensors
    acpitz-virtual-0
    Adapter: Virtual device
    temp1: +36.0&***;C (crit = +125.0&***;C)

    coretemp-isa-0000
    Adapter: ISA adapter
    Physical id 0: +36.0&***;C (high = +110.0&***;C, crit = +110.0&***;C)
    Core 0: +35.0&***;C (high = +110.0&***;C, crit = +110.0&***;C)
    Core 1: +35.0&***;C (high = +110.0&***;C, crit = +110.0&***;C)
    Core 2: +35.0&***;C (high = +110.0&***;C, crit = +110.0&***;C)
    Core 3: +35.0&***;C (high = +110.0&***;C, crit = +110.0&***;C)

  • I don't believe this is a configuration issue, and it could be either a software and/or hardware issue.

    Did you by chance run a smart test on the disk via SSH?

    smartctl -a /dev/sda

    OPNSense 64-bit | Intel Xeon 4-core v3 1225 3.20Ghz
    16GB Memory | 500GB SSD HDD | ATT Fiber 1GB
    (Former Sophos UTM Veteran, Former XG Rookie)

  • Yes, smart test both short and long passes, as well as memory test, as well as temperatures seem ok.
    I've been monitoring temperatures as the box has been up for over an hour, so I'm just checking what I can.

  • Can you paste the SMART test information here? 

    OPNSense 64-bit | Intel Xeon 4-core v3 1225 3.20Ghz
    16GB Memory | 500GB SSD HDD | ATT Fiber 1GB
    (Former Sophos UTM Veteran, Former XG Rookie)

  • Of course, here it is:

    escape75:/home/login # smartctl -d ata --all /dev/sda
    smartctl 6.3 2015-02-08 r4039 [x86_64-linux-3.12.74-0.424574463.ge309b77.rb7-smp64] (SUSE RPM)
    Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
    
    === START OF INFORMATION SECTION ===
    Device Model:     ADATA_IM2S3134N-064GM
    Serial Number:    2I3920032044
    LU WWN Device Id: 5 707c18 1006e2fd0
    Firmware Version: 6.8E
    User Capacity:    64,023,257,088 bytes [64.0 GB]
    Sector Size:      512 bytes logical/physical
    Rotation Rate:    Solid State Device
    Form Factor:      2.5 inches
    Device is:        Not in smartctl database [for details use: -P showall]
    ATA Version is:   ACS-2 (minor revision not indicated)
    SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
    Local Time is:    Tue Mar 21 06:31:32 2023 CET
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    General SMART Values:
    Offline data collection status:  (0x00) Offline data collection activity
                                            was never started.
                                            Auto Offline Data Collection: Disabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                (   32) seconds.
    Offline data collection
    capabilities:                    (0x5b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            No Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   1) minutes.
    Extended self-test routine
    recommended polling time:        (   1) minutes.
    SCT capabilities:              (0x0039) SCT Status supported.
                                            SCT Error Recovery Control supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x000a   100   100   000    Old_age   Always       -       0
      2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
      3 Spin_Up_Time            0x0007   100   100   050    Pre-fail  Always       -       0
      5 Reallocated_Sector_Ct   0x0013   100   100   050    Pre-fail  Always       -       0
      7 Unknown_SSD_Attribute   0x000b   100   100   050    Pre-fail  Always       -       0
      8 Unknown_SSD_Attribute   0x0005   100   100   050    Pre-fail  Offline      -       0
      9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       20706
     10 Unknown_SSD_Attribute   0x0013   100   100   050    Pre-fail  Always       -       0
     12 Power_Cycle_Count       0x0012   100   100   000    Old_age   Always       -       147
    167 Unknown_Attribute       0x0022   100   100   000    Old_age   Always       -       0
    168 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       -       0
    169 Unknown_Attribute       0x0013   100   100   010    Pre-fail  Always       -       196611
    170 Unknown_Attribute       0x0013   100   100   010    Pre-fail  Always       -       0
    173 Unknown_Attribute       0x0012   128   128   000    Old_age   Always       -       4365747135
    175 Program_Fail_Count_Chip 0x0013   100   100   010    Pre-fail  Always       -       0
    180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   100   100   020    Pre-fail  Always       -       553
    192 Power-Off_Retract_Count 0x0012   100   100   000    Old_age   Always       -       116
    194 Temperature_Celsius     0x0022   052   052   030    Old_age   Always       -       48 (Min/Max 44/49)
    197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
    231 Temperature_Celsius     0x0033   069   069   005    Pre-fail  Always       -       31
    233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       127659751936
    234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       192211142656
    240 Unknown_SSD_Attribute   0x0013   100   100   050    Pre-fail  Always       -       0
    241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       23934369819
    242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       5005309732
    
    SMART Error Log Version: 1
    No Errors Logged
    
    SMART Self-test log structure revision number 1
    Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
    # 1  Extended offline    Completed without error       00%     20667         -
    # 2  Short offline       Completed without error       00%     20633         -
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.

  • Oh God, thank you for formatting that, lol. 

    Yeah that's a very healthy disk.  I was wanting the values specifically, as most people just read the pass/fail line and don't worry about anything else.  All the values look really good.

    OPNSense 64-bit | Intel Xeon 4-core v3 1225 3.20Ghz
    16GB Memory | 500GB SSD HDD | ATT Fiber 1GB
    (Former Sophos UTM Veteran, Former XG Rookie)