v19 VMware disk usage, reporting full

This could be:  NC-94291

Fixed in MR1. 

I think and believe that this problem (NC-94291) is fixed in build 350. But it is possible to correct this problem on a "live system"? When we updated another firewall, the /var filesystem REMAIN small. I have found some info about "increasing /var size" in https://support.sophos.com/support/s/article/KB-000036775?language=en_US. It is worth trying this after update? Or do we have reinstall XG firewall again and restore config from backup?

IT IS reason.!!

We have "consistently" only 5%-10% free. Sophos report this nearly every day as errors. AND our monitoring system report this permanently. AND we keep losing reports older than several days. AND email system sometimes refuse sending bigger emails. AND we have aloocated 80GB disc for 4 GB /var. AND....

SFV4C6_VM01_SFOS 19.0.0 GA-Build317# df -h
Filesystem                Size      Used Available Use% Mounted on
none                    640.6M     11.6M    582.3M   2% /
none                      2.9G     24.0K      2.9G   0% /dev
none                      2.9G    968.3M      2.0G  33% /tmp
none                      2.9G     14.7M      2.9G   0% /dev/shm
/dev/boot               127.7M     26.6M     98.4M  21% /boot
/dev/mapper/mountconf
                        560.3M     95.4M    460.9M  17% /conf
/dev/content             11.8G    493.8M     11.3G   4% /content
/dev/var                  3.7G      3.3G    328.9M  91% /var

Ok lets rephrase this. 

It is another problem, you are facing. 

So the disk is not increasing due an issue. Instead the size of your partition is to small. This has nothing to do with any of the IDs i called. The IDs are tackling issues, which increases the amount of data stored and resolve this.

But if you partition is not utilizing the proper disk size, this is an different issue. You can follow up on this one with support. 

"Instead the size of your partition is to small."

primary disk 16 GB
secondondary 80 GB (as recommended by Sophos)

I do not feel taken seriously here!

We have more than 170 firewalls online, so you can assume some experience.

Please verify your own Post on https://community.sophos.com/sophos-xg-firewall/sfos-v19-early-access-program/f/discussions/133616/v19-hyperv-disk-usage-reporting-full

Gerd

Ok lets rephrase this.

NO and NO. Disk increasing IS NOT AN ISSUE. ISUE IS:

Size of /var on 80GB disk is only 3.8 GB after intalation.

After new/fresh install /var is simply TOOO small. And is not ANY documented method how to expand it(or repair it, or anything you have to rephrase it).

THUS we have "bad install = with small /var" and we don´t have ANY method to recover this BAD condition.

@ Jiri

Thank you for your support!

This is exactly what I would like to point out!

Again: My point is another. The firewall releases will tackle issues, which let the existing space fill up due bugs.

You are facing an problem, which is a problem in the size of the partition of /var/ (Do not confuse this with the disk size you are using). 

This is already addressed in the newest Installation medium. So if you use the V19.0 MR1 installation files for hyper-v/vmware, it will be fixed.

It only affect installations based on v19.0 GA or EAP2. V19.0 MR1 or <V19.0 GA (V18.5 etc.) are not affected.

It is currently not possible to fix this with on board tools / firmware updates. So you need to check for a reimage, as far as i know. 

Hi Lucar,

This is again a very bad cheap solution! But very expensive for us!

And once again: WHERE CAN I FIND THE 19.1 INSTALLATION IMAGE?

Of course I know where to find installatiosnsimages, but there is only VI-19.0.0_GA.KVM-317.zip available for download. As already written several times!!!

Gerd

... and please escalate this so your developers can find a workable solution quickly.

You could use V18.5 MR4 and upgrade to V19.0 MR1 as well. 

V19.0 MR1 is not released in this form, as it is in the soft release. 

https://download.sophos.com/network/SophosFirewall/installers/VI-18.5.4_MR-4.VMW-418.zip

https://download.sophos.com/network/SophosFirewall/installers/VI-18.5.4_MR-4.HYV-418.zip

Feel free to do this via your Support Channel. 

Feel free to do this via your Support Channel. 

Just to document the problem further:

Initial problem:
Installation image HW-19.0.0_GA-317 is buggy. Appliances that are set up with this image have the report partition filled to 100% after a few weeks.

1st statement Sophos:
Problem is known. There are two fixes: NC-94291 & NC-100679
Try an update to 19.0.1 MR-1
DOES NOT WORK!

2. statement Sophos:
Do a fresh install with 19.0.1 MR-1 and then restore the backup.
DOES NOT WORK, because an installation image 19.0.1 MR-1 does not exist.

3. statement Sophos:
Do a fresh install with 18.5.4 and then restore the backup.
DOES NOT WORK,
After reinstallation "check for new firmware" -> no updates available.
Watch out, because you can not restore the backup from 19.x to 18.5.4!
You have to download and install the update manually, but Update from 18.5.4 to 19.0.0 GA-317 is not supported!!!!
Verify update matrix https://docs.sophos.com/releasenotes/index.html?productGroupID=nsg&productID=xg&versionID=19.0

The Sophos support did not offer a solution until today!

So at the moment we only have to keep an eye on the already installed machines and if necessary keep them alive with "Flush Device Reports".

What's going on at Sophos again?

BR Gerd

Let me rephrase this: 

There are different points. And you are starting to mix up certain points: 

First of all, there are Bugs, which let the /var/ partition fill up. Those root causes are fixed in V19.0 MR1. So the /var/ is not increasing "unexpectedly". Due the fact, that /var/ is used for other things like reporting, it is natural to increase over time, but will be flushed after a while by the settings of the system (Report settings).
This is fixed by  NC-94291 & NC-100679 - So no unexpected increase of /var/. 

The current ISO of V19.0 MR1 is not available due the fact, V19.0 MR1 got a respin. Therefore we do not want to push Build350 to the public anymore. The team is currently working on getting ISOs pushed out. The problem is: You have regulations, which has to be hit (Software Export compliance etc.). 

You can install V18.5 MR4 and upgrade to V19.0 MR1 with the SIG File, you find on your portal. V19.0 MR1 is not available for machines via up2date. And i posted no advise to "restore" the backup. I talked about do a V18.5 MR4 installation, upgrade it and then restore. Maybe this was not clear from my end. 

After a week off from the Internet, I'm trying to solve this problem again.

I ran offered update SFOS 19.0.1 MR1-Build365 on our other SFV4C6 VM. Use of /var is still 60% there (just like before update, this firewall is not so loaded).

But I'm not able to upload VI-19.0.1_MR-1.VMW-365.sig to the main firewall (neither VI-19.0.1_MR-1.VMW-350.sig used on the other firewall). I tried it several times, I can select the file, agree "Are you sure you want to upload the firmware?" question, the wheel spins for a while and then it returns to the "Firmware upgrade/downgrade" window. Maybe the file can§t be uploaded due to low space on /var?

"Check for new firmware" shows "No updates available". Is any other way how to install this update?

LuCar Toni said:

This could be:  NC-94291

Fixed in MR1. 

What exactly is being addressed in NC-94291? I can see

NC-94291 Firmware Management Small var partition created for VM image using an auxiliary disk.

The problem is "Small var partition created for VM image using an auxiliary disk.". How is this solved?

Hi Thomas

The update does not seem to do anything (see below, checked on a brand-new appliance)

I'm still waiting for a solution from Sophos support, but it's very tough here as usual :-(
Although the error should actually be known.......

Before Update

console> system diagnostics show disk

Partition        Utilization(%)

===============================

configuration         9%

content               3%

report               34%

SFVUNL_HV01_SFOS 19.0.0 GA-Build317# df -h

Filesystem                Size      Used Available Use% Mounted on

none                    640.9M     11.6M    582.7M   2% /

none                      1.9G     24.0K      1.9G   0% /dev

none                      1.9G     12.9M      1.9G   1% /tmp

none                      1.9G     14.6M      1.9G   1% /dev/shm

/dev/boot               127.7M     26.9M     98.1M  21% /boot

/dev/mapper/mountconf

                        560.3M     51.9M    504.4M   9% /conf

/dev/content             11.8G    393.0M     11.4G   3% /content

/dev/var                  3.7G      1.2G      2.4G  34% /var

After Update

console> system diagnostics show disk

Partition        Utilization(%)

===============================

configuration        13%

content               3%

report               35%

console>

SFVUNL_HV01_SFOS 19.0.1 MR-1-Build365# df -h

Filesystem                Size      Used Available Use% Mounted on

none                    613.2M      1.5M    567.0M   0% /

none                      1.9G     32.0K      1.9G   0% /dev

none                      1.9G     12.6M      1.9G   1% /tmp

none                      1.9G     14.6M      1.9G   1% /dev/shm

/dev/boot               127.7M     34.3M     90.7M  27% /boot

/dev/mapper/mountconf

                        560.3M     71.6M    484.7M  13% /conf

/dev/content             11.8G    386.1M     11.4G   3% /content

/dev/var                  3.7G      1.3G      2.4G  35% /var

LuCar Toni said:

We are not increasing the the /var/ because there is no reason to do. But 41% is actually fine from my perspective. /var/ is rarely used. 

It´s not fine! As I wrote several times!

We have appliances that are 100% filled after a few weeks.

The only thing that helped here was to flush device reports and set the report storage duration to the shortest time interval!

But obviously, once again, no one at Sophos wants to know anything about it.

WE NEED A SUPPORT SOLUTION WITHOUT REINSTALLATION!

I do not wont to be rude, but my answer is pretty clear: 

I don't want to be rude either. But my question is pretty clear too:

What exactly is being addressed in NC-94291? I can see

NC-94291 Firmware Management Small var partition created for VM image using an auxiliary disk.

The problem is "Small var partition created for VM image using an auxiliary disk.". How is this solved?

Not a word about "Bugs, which let the /var/ partition fill up". Maybe I don't understand well Sophos language.

I consider logs and reports to be "expected increase of /var/" and I'm surprised that Sophos doesn't see it as their problem that installing VI-19.0.0_GA.VMW-317.zip creates /var of a size that fills up with logs and reports in a week on a busy firewall and prevents the MTA from using /var/spool.

I have reported the issue to local support and am awaiting a response.

Essentially MR1 and this ID: NC-94291 will fix the installation size of the /var/ partition. We cannot change this after an installation. So if you use the new (not released V19.0 MR1 Installer), it will use the proper Size of the Aux and use it for /var/. 

It will not solve the current situation of your installation. This was not proper communicated through the channel as well. It will be entered to the KIL (known issue list) as well soon to inform everybody. 

The current only solution, would be using V18.5 Installers and upgrade to V19.0 MR1 with a SIG file. 

Here you find all installers: https://download.sophos.com/network/SophosFirewall/installers/index.html 

Then use the V18.5 MR4 installer. Due the installations. Jump to the end and do not restore your backup. Then upgrade via SIG to V19.0 MR1. After upgrade restore you backup and you should be fine. With reboots, you should be good to go in 15 minutes. 

LuCar Toni said:

Then use the V18.5 MR4 installer. Due the installations. Jump to the end and do not restore your backup. Then upgrade via SIG to V19.0 MR1. After upgrade restore you backup and you should be fine. With reboots, you should be good to go in 15 minutes. 

15 minutes for deploying and configuring VM (if I must set default GW in shell after every reboot), upgrading, deregistering old VM from Central, backup conf and stopping old VM, restoring conf and trying to register to Central? You must be Superman. It took me two hours for standby SFV4C6 and I'm still not able to registr it to Central neither via OTP nor via account :-( I'll open new discussion for it.

I'll prepare VM for the main SFV4C6 but I will have to wait with transfer for maintenance time.