DevOps administration ASMX authentication issue over UTM proxy

Cześć Mateusz,

What do you see in the WAF log related to  /TeamFoundation/Administration/v3.0/LocationService.asmx?

Cheers - Bob

Good call! Something's wrong, but I don't yet know what. Here's what's logged:

2022:03:27-19:50:58 firewall httpd[19792]: [proxy_http:error] [pid 19792:tid 3966999408] [client 10.150.1.42:53922] AH01086: read less bytes of request body than expected (got 0, expected 240)
2022:03:27-19:50:58 firewall httpd[19792]: [proxy_http:error] [pid 19792:tid 3966999408] [client 10.150.1.42:53922] AH10154: pass request body failed to 10.150.1.42:443 (srv-tfs-p03.local.net) from 10.150.1.42 () with status 500
2022:03:27-19:50:58 firewall httpd: id="0299" srcip="10.150.1.42" localip="<our public IP>" size="530" user="-" host="10.150.1.42" method="POST" statuscode="500" reason="-" extra="-" exceptions="-" time="75115" url="/TeamFoundation/Administration/v3.0/LocationService.asmx" server="<public server URL>" port="443" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="<UID>"

The first two lines indicate some kind of error... 

Apparently this is an ongoing issue, as a similar problem is described here:

https://community.sophos.com/utm-firewall/f/hardware-installation-up2date-licensing/132837/waf-issues-after-updating-to-9-709-3/490536

Now the big question is should I open a new Sophos support ticket or hope that one gets resolved... Sophos Support is... a bit slow and my experience was that they're mainly great at raising blood pressure...

     statuscode="500"

What happens if you make an Exception for that path in 'Firewall Profiles'?

Cheers - Bob

I don't have a firewall profile for this service, so I don't think that applies.

You must have one in 'Webserver Protection' in order for there to be a line in that log.

Cheers - Bob

I don't know what to tell you... but there's no profile.

Unless even with the profile disabled there are still some rules being applied?

EDIT: One "profile" thing I have is a custom exception as described by the following thread:

https://community.sophos.com/utm-firewall/f/web-server-security/94853/encoded-uri-being-broken-by-waf

Essentially DevOps has some URLs which Sophos UTM failed to parse correctly, and I've used the instructions in the linked thread to add a custom exception. Still, no profile is applied, and with that exception turned off the issue remains exactly the same. Additionally, what would the exception that you're suggesting even be?

Hmmm, haven't seen that before Mateusz!

statuscode="500" means a conflict between the Proxy and the web server, so my guess is that you need a Firewall Profile with an Exception for that path.  Any luck with that?

Cheers - Bob

Hmmm, haven't seen that before Mateusz!

statuscode="500" means a conflict between the Proxy and the web server, so my guess is that you need a Firewall Profile with an Exception for that path.  Any luck with that?

Cheers - Bob

I've tried adding an exception for EVERYTHING for that path (didn't help)... But I really don't see how it can have any effect if there's no firewall profile turned on for that web server either.

I think the 500 error is actually coming from the service itself because that's what I'm getting when I try to use said service when it's exposed via UTM. In reality the underlying exception is of the 400-variety (looking at the Event Viewer stuff for DevOps), as the server thinks it's getting an anonymous request which it can't process and thus throws a 500...

I agree that the 500 is coming from the DevOps service.  Is there a configuration option for it?

What if you make the Exception for 

     /TeamFoundation/Administration/v3.0

Cheers - Bob

Same thing with that path. And the service doesn't have any parameters - it doesn't even have its own web.config file, sharing the same configuration that the rest of DevOps uses (of course it's anyone's guess if MS doesn't do hack'y stuff in the backend).

Is there a community for the DevOps application?  Maybe someone else has run into this with another Web Server Protection supplier.

Cheers - Bob

I fear the on-prem DevOps gets much less support than the on-line DevOps, both in terms of community and official support. And, given this doesn't seem like an isolated incident (see link to other forum post where other MS services are affected) I'm more inclined to believe this is a UTM issue.