Trouble with Per-Connection Direct Proxy Authentication

Can you show us your firewall rules? SFOS will pick up two different connections (From Client to SFOS 3128) and from SFOS to WAN Server (HTTP/HTTPS). Therefore both traffic needs to be allowed. You should allow in General LAN to WAN HTTP/S as well. Does it work then? 

What is the best way to show you my rules?  It's a little cumbersome to send screenshots.

Transparent proxy continues to work fine, which I assume demonstrates that HTTP/S traffic is flowing.  Based on the idea that SFOS would see "two different connections" and that maybe both SFOS 3128 and HTTP/HTTPS could not occupy the same firewall rule, I created a separate firewall rule for just the Client (LAN) to SFOS 3128 traffic (-i.e. LAN to WAN / services TCP3128).  This rule was placed above the standard HTTP/HTTPS rule.  The standard HTTP/HTTPS rule has my Custom Web Policy filter applied and is performing correctly for transparent HTTP/S traffic.  Again, the new separate "Proxy" firewall rule correctly identifies direct proxy traffic.  Web browsing traffic sent to the proxy port shows up in the firewall log assigned to the new "Proxy" firewall rule.  The firewall rule happily shows it is accepted.   But, when it comes to the web policy filter logs, the same traffic shows it set to the "Deny All" web policy, even though there is no web filter policy assigned to the "Proxy" rule??  The web policy logs do list the user who was the source of the traffic, so it appears proxy authentication is working.

Moverover, there is some proxy-bound traffic that is successfully traversing the firewall / web filter and is working!  This traffic is destined to sites that are specifically listed under Web >> Exceptions.  Again, pointing to the fact that this problem is with the Web Policy Filter, not the Firewall rules.  I know it seems crazy, but it's like the proxy service is "hard-coded" to use the "Deny All" web policy.

Just to be sure: You did remove the Terminal Server from all other Auth methods (SATC etc.) - Correct? Also there is no Clientless Auth for this server? 

Correct.  There is no clientless auth for this (or any other) server.  I have uninstalled SATC from the terminal server (using the standard un-installer).  I have left the terminal server listed in the list of Login and Logoff exclusions in the STAS control panel on my domain controller, since I still do not want IP-based STAS for this server.  I've tried adding and removing the server from the list under "system auth thin-client" in the device console, but the presence or absence of the server IP in this list does not seem to make a difference.  From the authentication logs, and the fact that packets are correctly identified as the various terminal server users, it does appear that proxy authentication is "working."

Years ago, when we first implemented the XG, we tried everything and worked with your engineers to try and get "proxy authentication" to work as we expected (-i.e. the way it purportedly works now).  Our old firewall used proxy authentication as the sole means for controlling outbound access and setting outbound site-access rules.  So, it was a hard mental shift to switch to the STAS "IP-based" firewall world.  We made many configuration changes trying to get the proxy to work before abandoning the idea and drinking the STAS/SATC kool-aid.  I wonder if any of those low-level proxy config changes are still lurking in the bowels of our XG?  Is there anyway to see definitively how packets route through the system and the "choices" the firewall makes as a single packet moves through the web of rules and modules?

So - As far as i can remember using this feature in early EAP days, you need to setup Kerberos/NTLM to get this running.

All the NTLM stuff needs to be there. See: https://support.sophos.com/support/s/article/KB-000035722?language=en_US

And it needs to be removed from SATC in the firewall. So you need to remove it from auth thin-client. 

In the end, this feature works like this: The separate connections (TS to firewall and firewall to webserver) will be handled differently. We are using the first connection to authenticate and the second connection to apply the correct web filter policy. 

I have removed the terminal server from the list in auth thin-client, but it does not make a difference.

Again my assessment is that NTLM stuff is there and working, and proxy authentication is working, since I'm getting the terminal server user listed in the list of authenticated clients as a "multi-host client" (note: not a "thin client" type like SATC users).  This authenticated user does not show up until the user browses for a website on the terminal server, as you would expect.

The firewall log shows traffic to the proxy port is happily accepted.  The web filter log shows the user who is generating the traffic. . .

"Proxy Access" Firewall Rule Web filtering set to "None". . .

So, I agree with you that the problem is with the "second" connection.  But I never "see" a second connection (firewall to external webserver) because the web policy of the first connection is blocking it (yet, the web policy is set to None).

If you put HTTP and Proxy into this firewall rule? Why is there no Proxy rule in this firewall rule? You need one firewall rule, with 80/443/3128 and using Proxy there. 

Yes.  This is exactly what I tried to do originally.  I used my existing Web Access FW rule, which continues to work perfectly for transparent proxy traffic (HTTP/HTTPS) and (only for transparent HTTP/HTTPS traffic) correctly applies the custom Web Policy "Fulton Homes Web Access Policy."   The only configuration change I originally made was to add "Proxy" to the list of destination services.  "Proxy" is defined as TCP/UDP 3128.  I only created the separate "Proxy Access" rule above to test the idea that HTTP/HTTPS and Proxy had to be in two different rules because of the "two different connections" thing.  So, I agree that 80/443/3128 should be in the same rule.  To this end, I have eliminated the separate Proxy Access rule and re-converged 80/443/3128 into a single rule.  But, the result is the same behavior.  Here's the whole rule:

Yet, while transparent HTTP/HTTPS traffic gets the "Fulton Homes Web Access Policy" applied, proxy bound traffic matching the same FW rule gets the "Deny All" web policy applied.  Here are the firewall/web filter logs for direct proxy traffic from my terminal server (192.168.10.95).

Here are firewall / web filter logs for transparent HTTP/HTTPS traffic (authenticated using STAS).

I am curious to know what the "Do not apply this migrated rule to system-destined traffic" is all about and if it could be the source of the problem.  This rule has been in place for years and migrated forward from previous versions.

I have exactly the same issue. Multi-Host live users are showing up just fine, but log viewer returns web_policy="Deny All" even though the correct rules (fwid and webpolicy id, i have verified this via psql) are matching. This seems to be a bug.

In normal use (before per-connection authentication) setting the web policy to "2" is part of how the firewall component signals to the web proxy that it should do the regular NTLM/Kerberos or Captive Portal authentication for the connection. It happens when the firewall believes it needs to know user identity to match a rule, but the user identity is not known. After the first authentication, the firewall remembers the user/IP association and treats subsequent connections as already authenticated.

With per-connection authentication, the firewall component doesn't know who the user is at connection time, because the authentication won't happen until the proxy sees the connection and challenges the browser for authentication.

It looks like you have "Match known users" and/or "Use web authentication for unknown users" set on both these firewall rules - the first screenshot doesn't show it directly, but the "Identity" section in the Summary suggests this is the case.

It's not necessary to enable either of these when using per-connection authentication. The firewall component will never be able to do anything based on user ID traffic from for per-connection auth hosts. Could you please try unchecking them and see if that helps.

If that does help, please let me know. We should probably find a way to make this more obvious or less consequential (i.e. document it better, or change the firewall behaviour for multi-user host IPs).

On this basis, it is most likely that you will have to have two separate firewall rules - one for the multi-user hosts (where you should specify those hosts in the 'Source' part of the rule) and another for regular endpoints using transparent mode auth.