XG VM on Hyper V 2019

Hyper-V Guest Backup has been modified in Server 2016 and 2019 and requires updated Hyper-V Integration Services components within the guest. I suspect that the XG codebase isn't up-to-date in this regard.

The workaround is to disable Backup (volume shadow copy) within the guest VM configuration. This will revert to the guest VM being backed up using Saved state. The guest VM will be offline for the time it takes to put the guest VM into a saved state along with coordinating all other VMs to take a VSS snapshot so that the Hyper-V host can take a point-in-time consistent backup. If this isn't acceptable then remove the guest VM from being backed up and factor in a manual backup of the guest VM as part of your maintenance schedule. Backing up the guest VM configuration file(s) and VHD/VHDX(s) along with their paths is sufficient enough to be able to import them into a working Hyper-V system in the event of a Hyper-V host failure.

FWIW this isn't just limited to the XG virtual machine image - all Linux and FreeBSD VMs with outdated (or just plain broken) Integration Services components exhibit this same backup misbehaviour.

As i mentioned in previous post even with NO integration services selected in the configuration of the guest VM for XG, that means backup and shadow are NOT selected, backup will fail, server will malfunction and hang if rebooted if XG VM is running on Hyper v server and windows back up runs. if XG VM is off it works fine. And it still tries live backup, no saved state. So that workaround is not effective.

Consider this scenario;

remove XG VM from hyper V.

run backup configuration while XG VM is absent.*1

Import XG VM into Hyper V no integration components selected.

Do not rerun Windows backup configuration.

backup runs successfully! XG is not 'saved' during shadow copy not does it appear in backup results. running live whole time.

on next backup backup fails at create shadow copy 

XG VM can not be shut down or rebooted using the command line menu after logged in as admin or by adding integration component and rt click for menu.

All windows VM's cannot be shut down/rebooted either. 

The Hyper visor management service (vmms.exe)cannot be stopped using net stop commands, stuck at 'stopping'

If you restart server with out ending the vmms.exe service server will hang during shutdown trying to stop that service.

Hold power button only way to recover. Or kill vmms.exe and restart service. Interesting that windows VM can be shut down using rt click menu but XG still trying to shut down.

any attempt to correct XG VM hangs vmms.exe

If you stop service before shutting down server will restart and XG VM will be running happily when you check .....until the next backup runs.

Clearly there are 2 workarounds. remove XG VM and use something else. Or have vendor fix code that is not compatible, Microsoft and/or Sophos. Something that is triggered by running backup or one of its components.

all other methods do not allow for reliable server. problem goes away when XG VM is not on server running! Must be off or removed backup will run weeks without issue.

This issue is holding up a server deployment.

*1 imagine trying to remember to do this each time you needed to modify the backup. 

Yuck.

I vaguely remember a similar problem when FreeBSD's Hyper-V Integration Components were broken - even with the Guest VM services turned off, the FreeBSD components were still registering services with the Hyper-V host. From memory the only way around this was to disable the Hyper-V kernel modules within the Guest VM.

What happens if you modify the Windows Server Backup configuration to exclude the XG VM and then run a backup?

2019 server has been running well a week now with XG off. when turned on even if its excluded from backup it tries to shadow copy the VM and will not do a saved stat then server malfunctions as i stated before . when XG VM is off back up is normal . i built a 2016 server and imported the XG VM and it all works for 4 days now. next will be another 2019 server on different hardware in case the HP hardware is the cause. Can't use the HP OEM media as it checks the hardware and refused to install on Dell. so downloading a trial 2019 from Microsoft. stay tuned. I contacted Sophos support last week, they asked me for an image of the error message. WHAT? i asked them to read this page as there is no one image that will illustrate what is going on. must be first level support. waiting for reply.  ticket #9338465

Just to confirm - you've gone into Windows Server Backup, modified the backup schedule and removed the XG VM from the backup set? And it's still trying to create a snapshot of the XG VM as part of the backup?

yes even if i remove XG VM, configure backup then import xg VM, making sure no services are checked it still happens. 

Update; since patching the 2019 server with 2019-11 updates XG is now working normally 4 days. kb4523205 mentions some virtualization update, must have fixed it. backup successful, server will restart without hanging the management service vmms.exe. also Sophos support was less than helpful with this issue. looking elsewhere for my FW needs.

Update 2;

Something strange yesterday. The VM restarted many times, i got these emails;

8:55am gateway came up

11:10am gateway came up

12:22pm gateway came up

12:22pm gateway went down

1:00pm gateway went down

1:00pm gateway came up

1:45pm gateway came up

1:45pm gateway went down

2:38pm  gateway came up

notice the odd number of went down to up emails? Been quiet since then. Keep in mind this is a lab environment there are no users or workstations. Also the XG is running with an expired trial license, not sure what effect that has on its functionality. Sill not ready to try this in production.

Update 2;

Something strange yesterday. The VM restarted many times, i got these emails;

8:55am gateway came up

11:10am gateway came up

12:22pm gateway came up

12:22pm gateway went down

1:00pm gateway went down

1:00pm gateway came up

1:45pm gateway came up

1:45pm gateway went down

2:38pm  gateway came up

notice the odd number of went down to up emails? Been quiet since then. Keep in mind this is a lab environment there are no users or workstations. Also the XG is running with an expired trial license, not sure what effect that has on its functionality. Sill not ready to try this in production.

so some interesting new from the production server. Above was the lab server which i have had shut down.

So backup started to fail intermittently on the server on both backup drives. these are USB bays with SATA 3GB disks in them, a few years old. the error would say something cryptic, no real hint says; delete and recreate backup job.  once on each drive. sometime it failed at last volume 'F' other times never started. 3rd time says error writing to drive. happening twice on same disk but one error did not identify disk name , used a random volume string. So i decided to replace the USB bays with new ones.

so guess what ? backup fails within minutes on first volume on both NEW bup drives and lots of warnings in the system log about uapstor driver resetting and disk retried warnings. was better with old bup bays. Should i put them back?

So did chkdsk on bup disk, was ok. so what gives here? is it USB problem,disk problem, USB bus maybe? So i looked again at error logs and noticed some failed at volume 'F'.
' so i looked at the drive; and that is where XG firewall VM is located. Of course i have has this VM off since November as it causes backup to malfunction. still waiting for the word from Sophos that they support it on server 2019.
Wait a minute!!!

I dropped the VM and deleted the files (to recycle bin)and Vola backup now works and finished successfully in 80 min. still a few warnings in system log but far less. 

So should i wait for second bup bay, rerun bup wizard or maybe empty recycle bin of XG files? Something about the XG VM disks is causing this.  and why was it ok for 3 months? last MSpatch 2019-12 was 3 weeks before first failure. I will wait for bup drive bay change tomorrow. stay tuned.