UTM best practise guide for strict webfiltering

My two cents:

UTM Web Filtering Best Practice Recommendation

General

Upload your organization logo using:

           Management… Customization… Global… Company logo

This helps your users detect block and warn messages as legitimate, reducing help desk calls while promoting compliance.

Categories

1) Review and optionally refine the supplied categories.

UTM has two types of categories  -- website categories and user permission categories.   Website categories are grouped together to create user permission categories.   All of the reporting is based on website category.  If a user is blocked or allowed incorrectly, you will have to manually follow the path from the website category to the user category to the user.   I created two of my own user permission categories

• High Risk or Illegal, (BLOCK for all users) which now contains these website categories:

Anonymizers, Anonymizing utilities, Browser exploits, Gambling, Gambling related, Hacking/computer crime, Illegal software, Illegal UK, Malicious downloads, Malicious sites, Media sharing, P2P/file sharing, Parked domain, Phishing, Pornography, Potential unwanted programs, Profanity, Residential IP addresses. School cheating information, Spam URLs, Spyware/adware

• Web Filtering Problems,  (WARN for all users) which now contains these website categories:

Categorization failed, Uncategorized

2) Ensure all website categories are assigned to a single user permission category.

Website categories are assigned to user categories, but there is no requirement for uniqueness.   A website category can be assigned to more than one user permission category, or it could be excluded from all user categories.   In either of these situations, the webfilter results may be unpredictable or undesired, so you need to ensure that every web category is assigned to exactly one user category.  A spreadsheet can be helpful for this purpose.

Log Files

I found that I could not manage the web filtering process without creating a Microsoft Access application to parse the webfilter log files.   The format is difficult to parse:   a space-delimited row header, followed by a space-delimited set of attributes in the format: keyword=”value”.   Several of the data elements are returned as both a code and a text translation of the code:

• Id and Name

• Category and CategoryName

• AppID and Application

Certificate problems

I am operating with HTTPS scanning enabled, so UTM looks closely at every certificate chain. It will block any connection with an imperfect certificate chain.   (I don’t know if the behavior is less strict if UTL-only scanning is enabled.)  The UTM administrator can create an exception to ignore certificate checks, but it seems unwise to do so globally,

I have encountered a surprising number of sites with SSL certificate problems.   The typical problems are:

• Broken certificate chain:   The system manager loaded the server certificate but did not load the intermediate certificate.

• Wildcard nesting:   The system manager assumes that a certificate for *.company.com can be applied to any and every web path.   The actual rule is that wildcards can only be applied to one level, so *.company.com is valid for abc.company.com, but it is invalid for abc.def.company.com.

• Extra self-signed certificate:   The site began with a self-signed certificate, then switched over to a commercial certificate, but never removed the self-signed certificate from the configuration.   UTM sees the self-signed certificate and rejects the connection.  I am hoping that a future release of UTM will ignore extra certificates as long as one certificate path is valid.

• Expired certificates:  Their system manager messed up.

The challenge for the UTM administrator is to identify when these problems are occurring, and decide how to respond to the other organization’s mistake.  The standard UTM reports have been of no value to me for this process, which is why I download and parse the log files in my own application.  

Invalid certificates problems are evidenced in the log file entries with these attributes:

• ID=”0002”, (Synonym:  Name=”web request blocked”)

• ERROR=”’ or not present

• URL starting https://

This selection rule will identify all sites that were blocked because of certificate problems, but it will not indicate the nature of the problem.   For that I use an SSL installation tester.  Most certificate issuers have a test utility, and SSL Labs has one as well

Our certificate vendor's test is quicker and suitable for most of my purposes, while the SSL Labs site is more comprehensive but slower.   Once you know the problem, you have three options:

• Ignore the problem.   This prevents your users from reaching that site.   This is the appropriate response for sites that are least important to your organization.

• Contact the server owner to make them aware of the problem.   There is no quick and simple method for knowing whom to contact.   You can either bypass UTM and browse the website to look for a “Contact Us” option (preferably for Webmaster), or use WHOIS to find an email address and hope that it will be routed to the right person.    I attempt contact for sites that are likely to be important to our staff but are not business critical.

• Create an exception to disable certificate checking.   The easiest way to do this is to browse to the problem site to display the navigation blocked page, then click the link for creating an exception.  You must know the webadmin username and password, so most users cannot do this.   For those who know the password, UTM will create the exception record if necessary, or will add to the existing record if it has already been established.

Certificate chain problems can be mitigated if you have access to the missing intermediate certificate from your own certificate vendor or other sources.  Intermediate certificates can be uploaded using Web Protection…  Filtering Options… HTTPS CAS… Verifications CAs… Upload local CA.   The same mechanism is used whether you are loading a CA root or a CA Intermediate certificate.

The certificate problem review process must be repeated continuously, preferably reviewing yesterday’s results today.

Uncategorized Websites

A similar process is needed for Uncategorized websites.    As indicated earlier, I have configured UTM to WARN on uncategorized websites, because these are the ones most likely to be malware traps, but they will also include a wide array of legitimate websites that are important to your organization but are too small to have been noticed by the categorization process.

Warned and uncategorized websites can be identified in the log file with this selection:

• ID=”0071”, (Synonym:  Name=” web request warned, forbidden category detected”)  and

• Category=” 9998,9998” (Synonym: CategoryName=” Uncategorized,Uncategorized”)

The second selection rule is probably unique to my configuration and may not be necessary if you want to review all WARNed sites.

While it appears that UTM can and does apply categories based on both fully qualified domain name and path, I reduce my results to unique FQDNs.   Then I review the list to decide whether I can safely place it in a category so that users will not be warned in the future.  One a list of updates is identified, I add those sites to the Webites override list.   For sites that seem to be generating a lot of different warn conditions, I may create the exception for the site/organization (company.com) instead of the FQDN.   For sites that I don’t recognize, I do nothing so that the warning remains in place.

The uncategorized site review process should also be performed daily or frequently.

Some follow-ups:

As frustrating as it is to identify and workaround websites with incorrectly-configured SSL certificates, this is a benefit not a problem.  I recently received a newsletter that talked about TLS interception tools, and argued that most of them were risky because they ignored SSL certificate errors themselves, while hiding the certificate problem from the user.   Part of the purpose of a certificate is to prove that the site you found is the site that you asked DNS to find, to protect against DNS attacks.  This protection only works if you enforce the rules strictly, the way that UTM does by default.  You can make the problem go away by creating an exception to ignore certificate errors for all sites, but this would be compromising your defenses significantly.

For uncategorized websites, I initially assigned categories as website exceptions, based on my best judgement.   This was inappropriate, as I don't have a good sandbox environment to test websites and I should not categorize a website that has not been evaluated.  The easier and safer solution is to crate an account on https://www.trustedsource.org, and submit the uncategorized websites to them for evaluation.   UTM documentation indicates that this is the service that UTM uses for categories and reputations.   The site asks for the McAfee product that you are using, and Sophos support told me that the correct choice is McAfee SmartFilter XL.   I tend to have about 10 sites per day to be evaluated, and they usually process those requests in a few hours.  This process makes the category and reputation available to all Sophos and McAfee users, while avoiding the performance penalty that would be expected if I configured a long list of website exceptions in UTM.

DouglasFoster said:

For uncategorized websites, I initially assigned categories as website exceptions, based on my best judgement. 

Where did you found the list of Uncategorized websites in UTM reports ?

DouglasFoster said:

For uncategorized websites, I initially assigned categories as website exceptions, based on my best judgement. 

Where did you found the list of Uncategorized websites in UTM reports ?

Not easily done.   In my first post, I indicated that I had to build my own report tool.  Details below will allow you to create your own if you have an Access developer and a license.   Something similar could be done in another tool.

From Logging and Reporting, go to View Log Files... Achived Log Files...  Web Filtering

Download yesteday's log file (it is in .gz format).  
(Working with today's log file is fairly difficult because the current-day file may be large, and must be saved using copy-and-paste from the log viewer window.)

Download 7-zip (free) or another utility that can decompress .gz files.

Change the file extension of the unzipped file to .TXT so that it can be read from Microsoft Access.

Use custom VBA code and nested queries to put the file into a relational format.  Each record in the log file is a list of phrases in the form keyword="value", and you need to make it behave like a .CSV file.

The Access code follows for those who want to try:

In a Module, paste this function:

Public Function ParseLog(msg As String, itm As String) As String
Dim tmppos As Long, tmpstr As String
tmpstr = " " & itm & "="""
tmppos = InStr(1, msg, tmpstr)
If tmppos = 0 Then
ParseLog = ""
Exit Function
End If
tmpstr = Mid(msg, tmppos + Len(tmpstr))
tmppos = InStr(1, tmpstr & """", """")
ParseLog = Left(tmpstr, tmppos - 1)
End Function

Save a sample log file to "weblog.txt".   Create a linked table to the file using fixed-length format.   Access does not allow text strings over 255 characters, so I broke the message body into 250-character chunks.   Then I used this query to assemble it back together:

Query name:  Full_Message

SELECT W.EventTime, W.ProxyName,
Left([W].[Msg0],10) AS Function,
" " & Left(W.Msg0 & " ",250) &
Left(W.Msg1 & " ",250) &
Left(W.Msg2 & " ",250) &
Left(W.Msg3 & " ",250) &
Left(W.Msg4 & " ",250) &
Left(W.Msg5 & " ",250) &
Left(W.Msg6 & " ",250) &
Left(W.Msg7 & " ",250) &
Left(W.Msg8 & " ",250) &
Left(W.Msg9 & " ",250) AS msg
FROM Weblog AS W
cxWHERE (((Left([W].[Msg0],10))="httpproxy[") AND ((InStr(1,[W].[Msg0],'id="0003'))=0));

(Message ID=0003 appears to document the connection process for an htttps connection, and I found it of no value so it is excluded in the above query syntax)

Next, you need to parse the keyword=value pairs.   Assuming that you created the function shown earlier, this query gets the job done:

QUERY NAME:  ParsedFile

SELECT DateValue(Mid([F].[EventTime],1,4) & "-" & Mid([F].[EventTime],6,2) & "-" & Mid([F].[EventTime],9,2))+TimeValue(Mid([F].[EventTime],12,8)) AS EventTime, F.ProxyName,
F.Function AS WebFunction,
ParseLog(F.msg,"id") AS ItmId,
ParseLog(F.msg,"severity") AS ItmSeverity,
ParseLog(F.msg,"sys") AS ItmSys,
ParseLog(F.msg,"sub") AS ItmSub,
ParseLog(F.msg,"name") AS Itmname,
ParseLog(F.msg,"action") AS ItmAction,
ParseLog(F.msg,"method") AS ItmMethod,
ParseLog(F.msg,"srcip") AS ItmSrcip,
ParseLog(F.msg,"dstip") AS ItmDstip,
ParseLog(F.msg,"user") AS ItmUser,
ParseLog(F.msg,"ad_domain") AS ItmAd_domain,
ParseLog(F.msg,"statuscode") AS ItmStatuscode,
ParseLog(F.msg,"cached") AS ItmCached,
ParseLog(F.msg,"profile") AS ItmProfile,
ParseLog(F.msg,"filteraction") AS ItmFilteraction,
CDbl(ParseLog([F].[msg],"size")) AS ItmSize,
ParseLog(F.msg,"request") AS ItmRequest,
ParseLog(F.msg,"url") AS ItmUrl,
ParseLog(F.msg,"referer") AS ItmReferer,
ParseLog(F.msg,"error") AS ItmError,
ParseLog(F.msg,"authtime") AS ItmAuthtime,
ParseLog(F.msg,"dnstime") AS ItmDnstime,
ParseLog(F.msg,"cattime") AS ItmCattime,
ParseLog(F.msg,"avscantime") AS ItmAvscantime,
ParseLog(F.msg,"fullreqtime") AS ItmFullreqtime,
ParseLog(F.msg,"device") AS ItmDevice,
ParseLog(F.msg,"auth") AS ItmAuth,
ParseLog(F.msg,"ua") AS ItmUa,
ParseLog(F.msg,"exceptions") AS ItmExceptions,
ParseLog(F.msg,"category") AS ItmCategory,
ParseLog(F.msg,"reputation") AS ItmReputation,
ParseLog(F.msg,"categoryname") AS ItmCategoryname,
ParseLog([F].[msg],"application") AS ItmApplication,
ParseLog([F].[msg],"app-id") AS ItmAppid,
ParseLog(F.msg,"content-type") AS ItmContenttype,
ParseLog(F.msg,"overridecategory") AS ItmOverrideCategory
FROM FullMessage AS F
WHERE (((ParseLog([F].[msg],"id"))<>"0003"));

I note that the where clause to exclude id=0003 is redundant, since it was already excluded in the previous query.

Now you can treat the ParsedFile query as a relational table to build specific queries.

I also found it useful to have a tool to parse a full URL into its component parts:

Public Function ParseURL(parmUrl As Variant, parmitem As String) As String
Dim urlraw As String, item As String, tmppos As Long, proto As String, fqdn As String, path As String

item = "" & UCase("" & parmitem)
If item = "" Then
item = "D"
End If

urlraw = "" & parmUrl
tmppos = InStr(1, urlraw, "://")
If tmppos > 0 Then
proto = Left(urlraw, tmppos - 1)
urlraw = Mid(urlraw, tmppos + 3)
Else
proto = ""
End If

tmppos = InStr(1, urlraw, "/")
If tmppos > 0 Then
fqdn = Left(urlraw, tmppos - 1)
path = Mid(urlraw, tmppos + 1)
Else
fqdn = urlraw
path = ""
End If

If item = "P" Then
ParseURL = proto
ElseIf item = "D" Then
ParseURL = fqdn
ElseIf item = "F" Then
ParseURL = path
ElseIf item = "PF" Then
ParseURL = proto & "://" & fqdn
End If

End Function

To find uncategorized websites, search for T.ItmId="0071"

This query uses the ParsedLog table and the ParseURL function to return a list of uncategorized websites, with a count of how many times they were referenced in http or https mode.   The path after the host name is discarded.

SELECT ParseURL([T].[ItmUrl],"D") AS FQDN,
Sum(IIf(ParseURL([T].[ItmUrl],"P")="https",0,1)) AS Http,
Sum(IIf(ParseURL([T].[ItmUrl],"P")="https",1,0)) AS Https
FROM ParsedFile AS T
WHERE (((T.ItmId)="0071"))
GROUP BY ParseURL([T].[ItmUrl],"D")
ORDER BY ParseURL([T].[ItmUrl],"D");

The same concept can be used for parsing the other files as well.

Based on the previous framework, below is the query to identify all sites that were blocked because of SSL Certificate problems:

I export the query results to Excel, then fill in the blank contact column with an email address (obtained from WHOIS).
I often use an SSL Checker to determine why the site failed, and put that reason into the blank "Reason" column.
Once the Excel file is populated, I use Word mail-merge to send the emails.   I average about 10-15 problem sites per day.;

SELECT Max(T.EventTime) AS LastSeen, 

"" AS Reason,
"" AS Contact,
ParseURL([T].[ItmUrl],"D") AS FQDN,
T.ItmDstip,
T.ItmReferer,
Count(T.EventTime) AS Events

FROM ParsedFile AS T
WHERE (((T.ItmError)="") AND ((T.ItmId)="0002"))
GROUP BY ParseURL([T].[ItmUrl],"D"), T.ItmDstip, T.ItmReferer
ORDER BY ParseURL([T].[ItmUrl],"D"), T.ItmReferer;

I made this Access database as per your instructions and it works wonderfully for pulling logs apart. Thanks for taking the time to share it with us. I noticed that the forum is replacing [ W ] with a icon of a withered rose. So to get around that click "Reply" to the post with the icon in it and it gives the original text.