Weird UTM freezes randomly approximately once a day ...

I think you've put in a good effort to try to get it going. What was the source of this device? If you can return it, I would. No point in wasting more time trouble shooting.

I've bought used pc equip on ebay before. If it works great, if I can't get it going within an evening or 3, it goes back.

Yes, it was ebay in fact.

I have purchased a previous XG115 r2 and it's still running great, but this one has issues from the start ...

I was just trying to figure out if I'm running into some weird bug possible with one of my LAN devices causing a crash,
I know it would be strange but I've seen strange things, and the seller claims it was running just fine before it was replaced.

You know, when I see stuff like this it makes me wonder if it was another bad unit or:

 XG115 Rev 3 freezing sometimes on SFOS 18.5.2 MR-2-Build380 

Just another update, I was able to find some weird entries for the last 3 hours before the crash in syslog ...

192.168.5.1 Mar 22 12:46:18 daemon warning 2023:03:22-19:27:59 escape75 URID[7870] T=7870 ------ 2 - sxl2_internal_get_time: The clock was set back from 1679510202 to 1679509679\n

192.168.5.1 Mar 22 13:22:48 daemon warning 2023:03:22-19:27:49 escape75 URID[7870] T=7870 ------ 2 - sxl2_internal_get_time: The clock was set back from 1679510204 to 1679509669\n

That doesn't look very good ... that's a 30 seconds time jump back!

This in turn causes DHCP to do this:

192.168.5.1 Mar 22 13:04:32 daemon debug 2023:03:22-19:27:52 escape75 dhcpd reuse_lease: lease age -337 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.5.5

One more update, the unit also freezes on SFOS hardware version, no access to ports or serial possible.

I have contacted the seller and got a refund. I ended up paying some duties, he paid a little more for shipping,
but in the end it's a wash and I'm happy that we were able to resolve this situation as best as we could.

I will maybe play with it some more, it seems like it's starting to lose track of time (RTC clock issue)
but I'm wondering if this could be a CMOS battery issue, even though it's not losing bios settings ...

It certainly strange as it was pulled from a working environment, and now that I have excluded the SSD,
as well as memory issues, there's not much that remains ...

Thanks for everyone's help!

Doubt it's the battery.  Battery comes into play when device is off. When on, power supply provides power for everything. I wonder however if the ps isn't somehow flakey causing the issues.  That would make sense for all the instability.

Yes I would also think that’s the case …

I don’t think it’s the power supply because I am using the same one I’m also using for my XG115 r2.

It’s the original 12v 3a power supply, and I tried both plugs in the r3, I believe they are for failover.

Unless somehow the r3 requires more power than what’s labeled on the unit …

Well, I opened it up and cleaned the board with alcohol, then coated with wd-40, now it's nice and shiny.
Also replaced the CMOS battery which was reading 3.0v, so theoretically still good but not 3.3v.

We'll see what happens ... if it still freezes up then I guess it's a hardware fault of some sort.

I assume the chassis is the heatsink?

What's the wd40 for?

Yes, the underside of the chassis has a big heatsink mounted to it, and the CPU connects to it with a gummy thermal pad that can be re-installed after being taken off.

I imagine the heatsink is big as it's meant to possibly be used for other models that have other chips that need cooling, as it's basically almost the size of the motherboard.

The wd40 is meant as a corrosion inhibitor, but in my case I was using it to clean the board after the alcohol, as apparently it's safe to do so, as it's non-conductive.

The unit was never opened before, or if it was, it was opened by Sophos as the security sticker was left intact, however I've noticed several places on the underside of the board that had weird residue, possibly thermal paste of some sort, so that's why I gave it a good cleaning.

I guess I'll find out, if it stays up over the next few days I will be happy,- it's worth trying 

Just a quick update, it's been almost 24 hours and still up, too early to tell if it's good.

I opened up my old XG115 r2 and tested the battery, which is older, and to my surprise it was 3.2v ...

It's also interesting that the old unit has a 2450 vs a 2032, with nearly a double capacity!

Unfortunately it went down again, lasted about 24 hours ...

Oh well, it's a hardware fault after all I guess so nothing I can do

One more update ...

I tried with another power supply (higher amperage) but the issue remains.

However, I have been testing Windows 10 on the unit, as well as performed Intel CPU tests and it passed!
I am also now running pfSense on the unit, and hopefully I will get some more answers as to what could be happening.

One more update ...

I tried with another power supply (higher amperage) but the issue remains.

However, I have been testing Windows 10 on the unit, as well as performed Intel CPU tests and it passed!
I am also now running pfSense on the unit, and hopefully I will get some more answers as to what could be happening.

Feels good to have gone through this exercise? :)

Run some prime95 torture tests on it for a few hours.. I'd be surprised if it doesn't lock up sooner than later.

Has *sense been stable on it? I think their igb drivers are more picky than linux's.

I think it feels good, I'm learning more details about what the issue might be

So, the interesting part is that my pfSense is setup in a similar configuration,
port 1 is WAN, and ports 2,3,4 are just bridged LAN, but so far pfSense has
not gone down and uptime is at 1 day and 23 hours!

This is much longer than UTM or SFOS has been able to do, and I've also tried taxing
the CPU by issuing multiple 'yes > /dev/null &' for each core, and there were no issues.

So, since I am sure there's other people here on UTM using an XG115 r3 without issues,
I must be running into a hardware fault that's only exposed by UTM and not pfSense ...

I don't know how that is possible

Did you do a prime95 test?

I did not, yet.

I wish there was a way to run prime95 on pfSense, maybe there's a way, I just don't know how,
I don't want to jump every 2 days to a different OS as that wouldn't be a reliable test either ...

I guess I could try: https://www.ultimatebootcd.com/

You could setup a windows to go usb flash drive to run it off off (or external ssd).

Assuming no issues after a few hours of prime95. Perhaps the nics are faulty?

As this is a multiple interface device, you could just run one ethernet cable between 2 ports then set up iperf3 to generate traffic.

The server and client will need to be bound to the separate interfaces using -B.  IIRC the format is -B {ip of interface).

Lets say you assign 192.168.1.1 to interface 1, 192.168.1.2 to interface 2

-B 192.168.1.1 will bind to interface 1, -B 192.168.1.2 will bind to interface 2.

Run that for a period of time to see if it causes some sort of failure.

Some good points, I will keep working in it ...

My uptime is now 2 Days 17 Hours.

I would think that if the issue related to a faulty NIC, it would also show up in pfSense, hmm!

I wish I had access to another XG115 r3, could it be different BIOS settings, or something ...

Uptime over 3 days and no issues with pfSense, but I did a memtest using ubcd539.iso and it froze!

The only issue is that I did the multicore test which I've read can cause false positives, but this thing acually froze ...

It was stuck on the above screen, only + was blinking but test time was not incrementing, I left it for a minutes just to see.

So I'm thinking maybe it's bad RAM after all ?!?

(Of course I wish I had another XG115 r3 to test the RAM using multiple CPU's to see if it also behaves this way)

Sort of makes sense.  If the bad memory is never access then there's no issue. I bet *sense has a smaller memory foot print than utm/xg.  Same in windows. Prime95 has a test which uses up most of the memory. 

Why wouldn't you run the mem test with all cores going? It should work.

Choose the red option to fill the ram during test.  Choose the blue option to stress the cpu/cache only.

I use both when validating a new build.

Bought some teamgroup ram a while back from amz.  Two kits, 2x16gb each, 3600mhz.  Running both together at 3200mhz no problem.  Running both at 3600mhz I'd get weird stability issues. Eventually tracked it down to one of the kits needing more than the xmp set 1.35V ddr voltage. 1.40V.

Didn't care much for this and exchanged the bad (questionable) kit. IMO memory should run at rated xmp settings without having to bump up ram voltage (mem controller might need a boost, but that's a different matter).  With the new kit, all 4 slots populated, 3600mhz 1.35v ran without any further issues.

Gets even weirder, memtest86 actually passes, but memtest86+ freezes always at the same place, 2:38 minutes, hmm!

I actually didn't realize I already run Mersenne Prime Test v28.x using blend which says lots of memory tested, and it passed.