v19 VMware disk usage, reporting full

This could be:  NC-94291

Fixed in MR1. 

I think and believe that this problem (NC-94291) is fixed in build 350. But it is possible to correct this problem on a "live system"? When we updated another firewall, the /var filesystem REMAIN small. I have found some info about "increasing /var size" in https://support.sophos.com/support/s/article/KB-000036775?language=en_US. It is worth trying this after update? Or do we have reinstall XG firewall again and restore config from backup?

Hi Thomas

The update does not seem to do anything (see below, checked on a brand-new appliance)

I'm still waiting for a solution from Sophos support, but it's very tough here as usual :-(
Although the error should actually be known.......

Before Update

console> system diagnostics show disk

Partition        Utilization(%)

===============================

configuration         9%

content               3%

report               34%

SFVUNL_HV01_SFOS 19.0.0 GA-Build317# df -h

Filesystem                Size      Used Available Use% Mounted on

none                    640.9M     11.6M    582.7M   2% /

none                      1.9G     24.0K      1.9G   0% /dev

none                      1.9G     12.9M      1.9G   1% /tmp

none                      1.9G     14.6M      1.9G   1% /dev/shm

/dev/boot               127.7M     26.9M     98.1M  21% /boot

/dev/mapper/mountconf

                        560.3M     51.9M    504.4M   9% /conf

/dev/content             11.8G    393.0M     11.4G   3% /content

/dev/var                  3.7G      1.2G      2.4G  34% /var

After Update

console> system diagnostics show disk

Partition        Utilization(%)

===============================

configuration        13%

content               3%

report               35%

console>

SFVUNL_HV01_SFOS 19.0.1 MR-1-Build365# df -h

Filesystem                Size      Used Available Use% Mounted on

none                    613.2M      1.5M    567.0M   0% /

none                      1.9G     32.0K      1.9G   0% /dev

none                      1.9G     12.6M      1.9G   1% /tmp

none                      1.9G     14.6M      1.9G   1% /dev/shm

/dev/boot               127.7M     34.3M     90.7M  27% /boot

/dev/mapper/mountconf

                        560.3M     71.6M    484.7M  13% /conf

/dev/content             11.8G    386.1M     11.4G   3% /content

/dev/var                  3.7G      1.3G      2.4G  35% /var

LuCar Toni said:

We are not increasing the the /var/ because there is no reason to do. But 41% is actually fine from my perspective. /var/ is rarely used. 

It´s not fine! As I wrote several times!

We have appliances that are 100% filled after a few weeks.

The only thing that helped here was to flush device reports and set the report storage duration to the shortest time interval!

But obviously, once again, no one at Sophos wants to know anything about it.

WE NEED A SUPPORT SOLUTION WITHOUT REINSTALLATION!

I do not wont to be rude, but my answer is pretty clear: 

I don't want to be rude either. But my question is pretty clear too:

What exactly is being addressed in NC-94291? I can see

NC-94291 Firmware Management Small var partition created for VM image using an auxiliary disk.

The problem is "Small var partition created for VM image using an auxiliary disk.". How is this solved?

Not a word about "Bugs, which let the /var/ partition fill up". Maybe I don't understand well Sophos language.

I consider logs and reports to be "expected increase of /var/" and I'm surprised that Sophos doesn't see it as their problem that installing VI-19.0.0_GA.VMW-317.zip creates /var of a size that fills up with logs and reports in a week on a busy firewall and prevents the MTA from using /var/spool.

I have reported the issue to local support and am awaiting a response.

Essentially MR1 and this ID: NC-94291 will fix the installation size of the /var/ partition. We cannot change this after an installation. So if you use the new (not released V19.0 MR1 Installer), it will use the proper Size of the Aux and use it for /var/. 

It will not solve the current situation of your installation. This was not proper communicated through the channel as well. It will be entered to the KIL (known issue list) as well soon to inform everybody. 

The current only solution, would be using V18.5 Installers and upgrade to V19.0 MR1 with a SIG file. 

Here you find all installers: https://download.sophos.com/network/SophosFirewall/installers/index.html 

Then use the V18.5 MR4 installer. Due the installations. Jump to the end and do not restore your backup. Then upgrade via SIG to V19.0 MR1. After upgrade restore you backup and you should be fine. With reboots, you should be good to go in 15 minutes. 

Hello Gerd,

according to the SFOS version, you have a virtual version of SFOS. How big a virtual disks do you have in Hyper-V/ESXi hypervisor? In my case, I have 16GB and 80GB virtual drives and disk partition sizes are as follows:

SFV4C6_VM01_SFOS 19.0.1 MR-1-Build350# df -h
Filesystem Size Used Available Use% Mounted on
none 612.8M 11.4M 556.6M 2% /
none 2.9G 48.0K 2.9G 0% /dev
none 2.9G 26.7M 2.9G 1% /tmp
none 2.9G 14.7M 2.9G 0% /dev/shm
/dev/boot 127.7M 34.0M 91.0M 27% /boot
/dev/mapper/mountconf
560.3M 74.7M 481.6M 13% /conf
/dev/content 11.8G 638.0M 11.2G 5% /content
/dev/var 70.7G 15.9G 54.8G 22% /var

I think your /var disk size is only about 4GB because it was the original version of that disk for v17/v17.5, if I remember correctly.
The installer from v18+ creates two virtual disks of 16 GB and 80 GB. I know that in the release notes for v18.0 it was recommended to install SFOS as a new system due to the greater demands on disk space for this and other versions.

Regards

alda

This issue only occur by using the V19.0 EAP2 or GA Installer. Other installers are not affected. 

Hi Luca,

I don't want to be offensive, but do you read your own posts?
A large manufacturer only offers solutions at the expense of the customer????

You can read all about it here!
Just once more.

- Sophos distributes a faulty installer.
- Solution is the new MR1 version, which is not available! (checked just one Minute ago!)
(That means faulty new installs every day!).
- Solution 2 "just install the old 18.5.x ! (which is officially no longer available!)

Hi Luca, who bears the costs if this measure has to happen at a customer 500 km away????
I know it like this: Whoever makes the mistake has to pay.

I am currently leaning towards solution 2, but with a firmware from another manufacturer.....


Unfortunately, this is not the first time I have experienced something like this at SOPHOS!

When will you finally wake up?

Greetings Gerd

... and Luca,

stop minimising the problem, even though you're getting paid for it.

Sometimes it's better to admit your mistakes!

Greetings Gerd

It´s a BUG from Sophos but they ignore it until today!

BR Gerd

It´s a BUG from Sophos but they ignore it until today!

BR Gerd

I will not respond to this thread anymore, as i cannot help you any further. 

All information are available. The bug is fixed. You need to reinstall with the new V19.0 MR1 installer or an old installer. 

Hi Lucar,

I can understand your despair. But there is still no acceptable solution from your employer SOPHOS!

Your last solution approach in unacceptable, as now really described several times!

Sorry dear community members, but actually this time I wanted to give Sophos a chance to really admit mistakes and generate a sound solution.
BR Gerd

LuCar Toni said:

All information are available. The bug is fixed. You need to reinstall with the new V19.0 MR1 installer or an old installer. 

this one is not available!

As I write now for several post!
What is this solution approach??

We are working on publishing the MR1 installers as we speak, I'll update this when they're available. 

Hello guys,

as long as we don't get a solution from Sophos, the only steps you can take are as follows:

1. reduce the report holding time to the lowest possible value
2. flush report data on the shell.

I unfortunately have to do this every 2nd day now because our customer appliance is more than 500 km away.

@Sophos great bug, keeps us at work

SOLVE IT!

Gerd, 

I do apologize for the trouble & extra work this issue has caused you & your customer. It was our fault this bug was introduced in v19.0 GA, and we are doing a full RCA to figure out how it happened and what we can improve internally to prevent this type of issue from happening again. 

Generally we only publish the installer of a version to MySophos when we fully GA, but I am working with the team now to publish the MR1 installer ASAP to provide you (& others affected) a viable solution. 

Bobby,

thanks for your answer, which sounds a bit more reliable to me.

Please remember the many installations that were made with 19.0.x.

A new installation costs you reputation and us money.
There must be another solution.

BR Gerd

Hi all, 

We have now posted the v19.0 MR1 installers at https://support.sophos.com/support/s/article/KB-000043162?language=en_US. 

Like Luca said, this issue affects VMs which were deployed with the v19.0 GA installer. If you re-deploy the VM using this MR1 installer, it would resolve the issue. 

I understand re-deploying a VM is sizeable effort, so in the meantime we're trying to find another solution to dynamically grow the partition size on a running VM without re-deploying as well. 

Hi Bobby,

Thank you for your answer, which initially gives us hope.

Please stay tuned to the topic.
A solution without reinstallation must be worked out.

Thanks Gerd

Hi Gerd Rehders1, TomasLavicky, Jiri Hadamek,

We have developed a workaround that will resize the /var partition on a running VM (while keeping your data), but requires a reboot of the VM. 

Can you please PM me your email address, so I can send you the workaround & instructions how to run it?