Web Filter Policy for User / Group Not Working

Your Filter Profile specifies Authentication = None, so it never asks for the user ID.    The log entry shows this:  user="".   So the default policy applies.

You need to specify an authentication method (AD SSO recommended if it is possible.)

For everyone else, it depends whether they will be authenticated or not.   If yes, then the default policy will apply.   If they will be unknown and you wish to allow access, you will need to create a policy with the option "Use this policy for users that skip authentication".

Check the Wiki for my longer discussions of web filter security and processing logic.

Douglas,

Thank you for your reply. I have checked a couple of things on the UTM and followed a number of steps which I will outline.

1.) I checked "Definitions & Users → Authentication Servers → Servers" and verified the link between the UTM and our AD Server. Test was successful.

2.) Ran a manual AD sync with the UTM and a new user I had created on AD appeared on the UTM so the connection between server and UTM is operational.

3.) As you suggested, I changed Authentication Method to "Active Directory SSO" on the Web Filter Profile (both the Default Profile and my Test Profile) as per the screenshots here:

4.) I rebooted the computer I am testing on numerous times in case the UTM needed to see it logging in. Again, the account I logged in with was called "training" and the IP on the machine today is 10.253.1.78

5.) I verified that the correct filter profile is being applied to the specific user. In this example the user is "training" and the website is "bbc.co.uk". On the Policy Test, the user is showing as being blockedfrom this site which I would expect.

6.) On the actual computer, the user can still access the site. The log entry is still not showing a username or domain - log extract here: 2018:10:27-14:08:14 cc-utm02-2 httpproxy[5664]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="GET" srcip="10.253.1.78" dstip="151.101.192.81" user="" group="" ad_domain="" statuscode="301" cached="0" profile="REF_HttProContaInterNetwo (Test Web Filter Policy)" filteraction="REF_DefaultHTTPCFFAction (Default content filter action)" size="0" request="0x1e997800" url="http://bbc.co.uk/" referer="" error="" authtime="6" dnstime="210" cattime="131" avscantime="0" fullreqtime="35379" device="1" auth="2" ua="Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" exceptions="" category="134" reputation="trusted" categoryname="General News" application="bbc" app-id="1213"

7.) There is one other thing I have noticed which I cannot explain. On the users computer, when I first open Internet Explorer, I see an entry in the live log with DOES show the user and domain (blanked) correctly as I would expect. The URL on this entry is the actual UTM so it seems to be picking up the correct information when the url is the utm but not on a website: 2018:10:27-14:17:23 cc-utm02-2 httpproxy[5664]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="GET" srcip="10.253.1.78" dstip="" user="training" group="" ad_domain="XXXXXXXXXX" statuscode="307" cached="0" profile="REF_HttProContaInterNetwo (Test Web Filter Policy)" filteraction=" ()" size="0" request="0x18a6be00" url="cc-utm02/auth referer="" error="" authtime="73" dnstime="0" cattime="0" avscantime="0" fullreqtime="171125" device="1" auth="2" ua="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; Tablet PC 2.0)" exceptions=""

The fact it picks up the user and domain on at least one log entry is making me think (or hope) the setup is very close to allowing me to properly use the Web Protection they way I wish.

If you or anybody else could shed more light on this I'd really appreciate it.

Thanks again, Niall Corcoran.

What type of test device are you using?   device="1" indicates that the device is not a PC.   I would expect device="0"

It's a standard Dell laptop running Windows 7 32bit. I'm logged into our network domain as the user "Training".

For AD SSO to work, the browser has to pass NTLM information to the proxy.   This browser is not doing so:

 ua="Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko"

so the user is unknown and the default action is taken

This browser is supplying the NTLM information, so it works:

 ua="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; Tablet PC 2.0)" 

Ok. Thanks for the reply. I'll have a check to see if I can enable it and run the tests again. Should I be surprised to see Mozilla on the log as the browser I'm using under the "Training" username is IE11.

Niall

Everything has Mozilla in the name, so that is not a surprise.  IE 11 does pass NTLM.   Compatibility mode changes the UA string, but it does not disable NTLM in any configuration that I have seen, so I would still expect to find another application.

My experience:   

• All major browsers will pass NTLM information (on Windows operating system)
• Nearly all non-browser applications which use HTTP or HTTPS will not pass NTLM.   
• Some of your non-browser applications can be manually configured with proxy credentials, but you will have a trouble finding them all, getting them configured, and solving problems when they have credential problems.   Even if you get this right, you will be stuck with some non-browser applications that cannot be configured for the proxy.
• You also need to consider whether web access will be needed if someone logs into a PC-local (non-AD) account.

UTM constraints:

• The proxy mode, source IP address, and device type determine the Filter Profile, which determines the authentication method.   If a request fails authentication, it will not drop down to the next Filter Profile.   Instead, the current filter profile will do whatever is configured for unauthenticated users (which can be block, default filter action, or a specific policy-filter_action pair.
• In some cases, non-AD requests will trigger a Basic Authentication prompt.  I have not tried to define when that does or does not occur.   In general, I do not expect users to respond correctly to Basic Authentication prompts, so I do not want them to appear.

I have discussed these issue in more detail in a post at the top of the Web Filtering sub-forum, titled "Web Filtering Lessons Learned".   

My solution has been to require AD SSO for Standard Mode, and to use Authentication=None for transparent mode.  Both modes are enabled for all internal IP addresses.   If you want to grant authentication bypass only as needed, the permission is granted by creating an Exception with the option for Bypass Authentication, then match to specific destinations or sources.

Consider that you can also enforce a configuration by giving the client device a static IP, then creating a Filter Profile for that IP address.

Transparent Mode Filter Profiles also act as Standard Mode profiles (undocumented feature).   So if you create any Standard Mode Filter Profiles, ensure that they have higher precedence than any Transparent Mode Filter Profiles.

About device codes, I stand corrected.

From this KB article about log files:

community.sophos.com/.../126660   

device codes
0: Unknown, 1: Windows, 2: Mac OSX, 3: Linux, 4: iOS, 5: Android, 6: Kindle, 7: Blackberry

I only see 0, null, or empty string.  I will ask support why my device detection is broken.

I wonder if the device type is based on MAC address, and therefore depends on a LAN connection between client and UTM

Hi Niall and welcome to the UTM Community!

Joining the UTM to AD on the 'Single Sign-On' tab seems to be the only thing you missed in Configuring HTTP/S proxy access with AD SSO.  Although the article is aimed at Standard mode, 98% of it applies to Transparent mode, too.

Cheers - Bob

Hi Niall and welcome to the UTM Community!

Joining the UTM to AD on the 'Single Sign-On' tab seems to be the only thing you missed in Configuring HTTP/S proxy access with AD SSO.  Although the article is aimed at Standard mode, 98% of it applies to Transparent mode, too.

Cheers - Bob