User field in WebProtection populated only occasionally

1) Your users have to hit a Filter Profile that specifies Authentication.

2) If a website has an exclusion for Authentcation, the Authentication process is not invoked so the user is not logged.

3) With HTTPS Inspection ("decrypt and scan") off, UTM cannot do AD SSO authentication because the SSO information is in the encrypted portion of the packet.   As a workaround, it assume the user is the last username seen on an http request from the same source IP address.  If there is no prior http request, the user is unauthenticated and the unauthenticated user policy is applied.   Workarounds:

• Add an http internet page to your startup page policy.   Google, Youtube, and many other sites use only https, but will redirect an http request to https.  Any of these should be sufficient to trigger authentication.
• Or enable decrypt-and-scan.

Also be aware that with decrpt-and-scan disabled, UTM only sees the host name (FQDN), not the full path and querystring, because this information is also encrypted.   Consequently, regular expressions that reference the path will not be matched.

If you use regular expressions, check your syntax carefully, it is easy to make mistakes.

1) Your users have to hit a Filter Profile that specifies Authentication.

2) If a website has an exclusion for Authentcation, the Authentication process is not invoked so the user is not logged.

3) With HTTPS Inspection ("decrypt and scan") off, UTM cannot do AD SSO authentication because the SSO information is in the encrypted portion of the packet.   As a workaround, it assume the user is the last username seen on an http request from the same source IP address.  If there is no prior http request, the user is unauthenticated and the unauthenticated user policy is applied.   Workarounds:

• Add an http internet page to your startup page policy.   Google, Youtube, and many other sites use only https, but will redirect an http request to https.  Any of these should be sufficient to trigger authentication.
• Or enable decrypt-and-scan.

Also be aware that with decrpt-and-scan disabled, UTM only sees the host name (FQDN), not the full path and querystring, because this information is also encrypted.   Consequently, regular expressions that reference the path will not be matched.

If you use regular expressions, check your syntax carefully, it is easy to make mistakes.

Thanks for the Reply, I have now figured out with your help!

now I have the quota for all http sites, and we use nowhere SSL Interception, but still if I open up any https page it shows the SelfSigned cert from the UTM.

Am I missing something again, or can it be that it automatically Intercepts SSL? 

Thanks for all the Help so far!

UTM has to impersonate the remote server when:

• HTTPS inspection is enabled
• A Block or Warn page needs to be displayed

This means that you need the CA Root certificate distributed to your clients, even though you do not use HTTPS Inspection.   For an HTTPS page, the block has to be returned using HTTPS, so for the page to pass certificate verification, the CA Root certificate needs to be available.

Side Note from your earlier question:   

In the logs, you will see many entries with status code = 407 and no user information.   This is normal chatter between UTM and the browser, where UTM is asking the browser for credentials.   As long as the browser replies with acceptable credentials, a subsequent entry will indicate that the request was allowed.   

Fat client applications typically cannot provide AD SSO information, so they will fail authentication and the logs will show multiple 407 entries but no success entry.   Some fat client applications can be configured with explicit proxy credentials, but more often you will need to either create an exception for them or allow a default set of browsing permissions for the unauthenticated users.

Proxy Types:

I recommend using both Standard and Transparent Proxies.  I use Standard Proxy with AD SSO for web-browser traffic, and Transparent Proxy with no authentication to handle the non-browser traffic and operating system traffic.   See my "Lessons Learned" post in the Web Filtering section for additional details.

One more tip:   

You MUST block UDP 443 in your firewall, or Chrome will bypass your web filter for any server that allows their QUIC protocol.

Dear Douglas,

Thank you for the clarification, I am avare of how SSL Interception works. and it is all easier with our own CA for sake of seamless description. 

I have now the policy set up and the filter profile is set to Transparent and AD SSO.

created a Tag fr the 3 domains including all subdomains:

• googlevideo.com
• youtube.com
• ytimg.l.google.com

and this is set up in the profile as the only object under "Decrypt and scan the following"

youtube works just fine, no cert error... Most probably due to QUICK (I now remember having troubleshooted about 4 hours with a customer 3 years ago and saw UDP 443 traffic, totally forgot that, and thanks for reminding me!)

but, all HTTPS is decrypted and the Sophos Proxies the HTTPS traffic.

If I set the same policy to "do not proxy HTTPS traffic in Transparent mode" it still does it, all the time. and the weird thing is, every HTTPS site gets a cert error, except youtube... that works just fine... 

for example jewtube.com is simple HTTP site where YouTube videos are embedded is hitting the quota page. that is what bothers me and was thinking something may be off, as the policy clearly states, do not proxy HTTPS but apparently it does. 

or am I missing something?

P.S.: thanks for all your help so far! Can I somehow get you a beer? :)

If you know that the site should be safe, and you are getting a certificate error, you could bypass the error to see what is actually being displayed, and then check the certificate attributes.

Browser issues:

Firefox uses its own per-user certificate store, which can float with the user's Firefox login.   Getting the CA certificate inserted into every user's certificate store is difficult.   There has been talk that they were adding an option for it to use the Windows certificate store, but I do not know if it has ever been implemented, and it would still require a per-user selection to change stores.  This is one of several evidences that Mozilla products have no place in a business, because they are not designed for centralized administration.

Microsoft Edge will block invalid certificates and not allow an override.   It also has no way to view the details about a valid certificate.   Of course, this may have changed as I do not use Edge either.   My substantive experience is with IE v11 and Chrome.

Website Tags

I do not know that this explains your symptoms, but I have recently learned that the option to include subdomains only applies to one level.    So "example.com" with "include subdomains" will match "server1.example.com", but it will not match "server2.division1.example.com"

Streaming Site Issues

There is a checkbox option to bypass content scanning for streaming sites.   It is recommended (and probably necessary) for acceptable video performance, but it may be part of your problem.

I am actually surprised that streaming video is working well with https inspection enabled.

Other

I don't know that any of this explains your symptoms.  

To check whether https inspection actually occurred, check the log file to see if the URL includes a path and query string.   If you only have an FQDN for the path, it is probably not inspecting.    When https is not decrypted, you get a single log entry, with method="CONNECT", after the session closes.  The size=value token represents the total bytes transferred for the session.   When https is decrypted, you get one entry for each URL referenced, and the size=value parameter is the data returned by that one web request.   I have not evaluated video closely, and I have given up on https inspection for now, so the size=value may be different with video streaming, based on how you have set the bypass option.