First off let me say I know this has been a frequent question posted here regarding slow upload speeds on the UTM. I have been a UTM user for well over a decade probably closer to 15 years. I may not be a Bob expert but I know my way around the software.
I recently got an ATT symmetric 1GB fiber line and plugged it up to my UTM VM. At first, performance was normal and expected. I was getting close enough to max throughput with the understanding that I would be getting less because it was running in a VM on older hardware. All fine. Then one day, and I honestly don't know when, upload performance just took a nose dive. I would get 3-5Mbps on uploads. Sometimes spiking to 7%.
I did all the normal things, turned off IDS, turned off web filtering, confirmed 1500 MTU on all NICs, you name it. I have a small Ubiquiti FW as a backup and it was able to get full 1Gb from all my VM's and my physical boxes, so I know my internal network is fine and can handle it. I even built not 1 but 2 brand new UTMs. One I did a restore of a config and another I did unconfigured, no settings. The performance of all these UTMs is the same 5Mbps upload. Download speeds are fine and close to theoretical maximums. I am at a complete loss. I do NOT want to migrate my services to Ubiquiti. I want to keep using the UTM. What else can I do? there are no IDS or filtering logs to check since all of that is disabled. Is this an ATT thing? Is there some special setting I need to make on the ATT FW or UTM interface setting I need to make to get this to work? It doesn't have to be perfect, but symmetrical upload is all I want.
I have a similar set up to the OP in terms of 1gb att fiber and virtualized utm.
Unless the OP forgot to mention any bypass, my config differs in that i'm using a full gateway bypass - the gateway box (bgw210) sits useless in a box on a shelf. This uses extracted certs and wpa_supplicant to handle 802.1x auth.
Also, the wan port is directly passed through to utm (i211).
Have you tested uploading directly from the utm shell? Not sure how as there's no ftp/tftp client. Maybe using curl or iperf3 (manually installed).
Is netstat -i or ifconfig reporting any ethernet errors?
The ubiquiti works properly with the same cables?
Given the OP is using a vnic for wan, it's worth exploring advanced configuration options for the vnic and vswitch. Maybe something got changed. Also check the network stack configuration. It's been about 18 months since I last used esxi. Esxi doesn't support the rtl8125 nic in my upgraded host (x570, 5800x cpu, 64gb ram).
You can also do a dumb switch bypass to test to ensure the gateway is not somehow affecting this - https://www.dslreports.com/forum/r32094182- .
This method still works if your area has not been upgraded to xgs-pon. This eliminates any issues the gateway may have with your nic.
I switched to proxmox in early 2019. Prior to that was running esxi with utm. Full bw upload/download (940/940 mbps). That was on a 6600K box where utm had 4 vcores assigned. Virtualizing utm is prob not best for large scale/enterprise applications but works very well for home lab.
I have the ATT box in passthrough mode as well. Public IP's are assigned to all FW's. The physical Unifi gets the full 1Gb so I don't think there is anything on the ATT that needs adjusting. And it's JUST uploads, too, so I don't get that.
I've moved the internal VMNIC between vswitches, too. The physical nics are enterprise grade. No RealTek. Broadcom and Intel enterprise 10GB SFP+ NICs. Maybe a firmware update messed them up? But no, if I can get full Gb from a VM by swapping to a physical FW that means the same connections can handle full gig both ways. Just typing out loud to see if my thinking makes any sense.
No passthrough. Out of the picture entirely... removed, gone, does not exist.
That's why I suggested the dumb switch bypass just to rule out some weird issues between the gateway and the server nic.
So not sure I can do that. Before the ATT box is the actual fiber itself. The ATT box is a media converter. The fiber comes from the curb and goes directly into the ATT device which has multiple RJ45 ports on the back. One goes to a physical FW and one goes to a port on my core switch.
Sounds like you got the new set up - bgw320. There's ways of bypassing that too but more complicated. Dslreports has a lengthy thread on the topic - https://www.dslreports.com/forum/r33442912-AT-T-Fiber-Bye-bye-802-1x-you-will-not-be-missed .
Only a single device behind the att gateway/media converter can get the public ip. Are you subscribed to multiple static ip's?