UTM best practise guide for strict webfiltering

Hi, Christian, and welcome to the UTM Community!

I like what you are doing although I don't use the whitelist approach.

Using the blacklist approach, I usually create a Category "_ Malicious" containing Malicious downloads & Malicious sites. Also, for each client a list that usually includes Anonymizers, Anonymizing utilities, Browser exploits, Extreme, Hacking/computer crime, Remote access, Spam URLs, Spyware/adware & Web ads.

I have explored the question of FLASH blocking with Sophos support.

If the flash object is invoked with an <object> tag, UTM will block (strip out) the Flash content. However, several of the sites I tested invoke Flash using Javascript, and it seems difficult or impossible to block this material at UTM because the possible complexity of JavaScript is unlimited. Since the most malicious sites are the ones most likely to use Javascript to obscure their use of Flash, the UTM defense seems unlikely to accomplish much.

The workaround is to require click-to-play, so that Flash cannot run unless the user asks it to run. This article is one that I found which explains the details and appears to be accurate:
www.howtogeek.com/.../

Hello together,

thanks for your ideas.

@BAlfons:
- we dropped whitelist again, after enforcing stricter computer rules.
- for Suspicous category: we did not notice it at first. At beginning we blocked several categories and got false positives, removing everything, unfortunately also suspicious. We also included now the crimial stuff.

@DouglasFoster
They write it in some documents (but not in the admin book), that the filter is not safe. I find it very misleading as I've got uncoutinous after your location IT upgraded the PCs to Win7. Firefox, which does the blocking automatically, was not enforced.
I have to allow Java sites, e.g. govermental sites, but Malware just gets thru.

Also when setting up the url filter, we noticed that it was very slow at beginning. After some research, we noticed that we should active Endpoint Protection, to use the fast filter list - even if no computer is managed by the UTM.

I miss a best practise book

- the recommanded blocking categories: like suspicous to block ad-networks, which have no safe infrastructure and deliver ransomware, which current virus tools cannot detect (even Sophos itself!)
- features that should be activiated, but it should be also told that they are not 100% safe. E.g. firewall is 100% blocking the ports, when setting to block, but Flash is not 100% blocked, when setting to block. That should be also told in the administrator handbook...

My two cents:

UTM Web Filtering Best Practice Recommendation

General

Upload your organization logo using:

           Management… Customization… Global… Company logo

This helps your users detect block and warn messages as legitimate, reducing help desk calls while promoting compliance.

Categories

1) Review and optionally refine the supplied categories.

UTM has two types of categories  -- website categories and user permission categories.   Website categories are grouped together to create user permission categories.   All of the reporting is based on website category.  If a user is blocked or allowed incorrectly, you will have to manually follow the path from the website category to the user category to the user.   I created two of my own user permission categories

• High Risk or Illegal, (BLOCK for all users) which now contains these website categories:

Anonymizers, Anonymizing utilities, Browser exploits, Gambling, Gambling related, Hacking/computer crime, Illegal software, Illegal UK, Malicious downloads, Malicious sites, Media sharing, P2P/file sharing, Parked domain, Phishing, Pornography, Potential unwanted programs, Profanity, Residential IP addresses. School cheating information, Spam URLs, Spyware/adware

• Web Filtering Problems,  (WARN for all users) which now contains these website categories:

Categorization failed, Uncategorized

2) Ensure all website categories are assigned to a single user permission category.

Website categories are assigned to user categories, but there is no requirement for uniqueness.   A website category can be assigned to more than one user permission category, or it could be excluded from all user categories.   In either of these situations, the webfilter results may be unpredictable or undesired, so you need to ensure that every web category is assigned to exactly one user category.  A spreadsheet can be helpful for this purpose.

Log Files

I found that I could not manage the web filtering process without creating a Microsoft Access application to parse the webfilter log files.   The format is difficult to parse:   a space-delimited row header, followed by a space-delimited set of attributes in the format: keyword=”value”.   Several of the data elements are returned as both a code and a text translation of the code:

• Id and Name

• Category and CategoryName

• AppID and Application

Certificate problems

I am operating with HTTPS scanning enabled, so UTM looks closely at every certificate chain. It will block any connection with an imperfect certificate chain.   (I don’t know if the behavior is less strict if UTL-only scanning is enabled.)  The UTM administrator can create an exception to ignore certificate checks, but it seems unwise to do so globally,

I have encountered a surprising number of sites with SSL certificate problems.   The typical problems are:

• Broken certificate chain:   The system manager loaded the server certificate but did not load the intermediate certificate.

• Wildcard nesting:   The system manager assumes that a certificate for *.company.com can be applied to any and every web path.   The actual rule is that wildcards can only be applied to one level, so *.company.com is valid for abc.company.com, but it is invalid for abc.def.company.com.

• Extra self-signed certificate:   The site began with a self-signed certificate, then switched over to a commercial certificate, but never removed the self-signed certificate from the configuration.   UTM sees the self-signed certificate and rejects the connection.  I am hoping that a future release of UTM will ignore extra certificates as long as one certificate path is valid.

• Expired certificates:  Their system manager messed up.

The challenge for the UTM administrator is to identify when these problems are occurring, and decide how to respond to the other organization’s mistake.  The standard UTM reports have been of no value to me for this process, which is why I download and parse the log files in my own application.  

Invalid certificates problems are evidenced in the log file entries with these attributes:

• ID=”0002”, (Synonym:  Name=”web request blocked”)

• ERROR=”’ or not present

• URL starting https://

This selection rule will identify all sites that were blocked because of certificate problems, but it will not indicate the nature of the problem.   For that I use an SSL installation tester.  Most certificate issuers have a test utility, and SSL Labs has one as well

Our certificate vendor's test is quicker and suitable for most of my purposes, while the SSL Labs site is more comprehensive but slower.   Once you know the problem, you have three options:

• Ignore the problem.   This prevents your users from reaching that site.   This is the appropriate response for sites that are least important to your organization.

• Contact the server owner to make them aware of the problem.   There is no quick and simple method for knowing whom to contact.   You can either bypass UTM and browse the website to look for a “Contact Us” option (preferably for Webmaster), or use WHOIS to find an email address and hope that it will be routed to the right person.    I attempt contact for sites that are likely to be important to our staff but are not business critical.

• Create an exception to disable certificate checking.   The easiest way to do this is to browse to the problem site to display the navigation blocked page, then click the link for creating an exception.  You must know the webadmin username and password, so most users cannot do this.   For those who know the password, UTM will create the exception record if necessary, or will add to the existing record if it has already been established.

Certificate chain problems can be mitigated if you have access to the missing intermediate certificate from your own certificate vendor or other sources.  Intermediate certificates can be uploaded using Web Protection…  Filtering Options… HTTPS CAS… Verifications CAs… Upload local CA.   The same mechanism is used whether you are loading a CA root or a CA Intermediate certificate.

The certificate problem review process must be repeated continuously, preferably reviewing yesterday’s results today.

Uncategorized Websites

A similar process is needed for Uncategorized websites.    As indicated earlier, I have configured UTM to WARN on uncategorized websites, because these are the ones most likely to be malware traps, but they will also include a wide array of legitimate websites that are important to your organization but are too small to have been noticed by the categorization process.

Warned and uncategorized websites can be identified in the log file with this selection:

• ID=”0071”, (Synonym:  Name=” web request warned, forbidden category detected”)  and

• Category=” 9998,9998” (Synonym: CategoryName=” Uncategorized,Uncategorized”)

The second selection rule is probably unique to my configuration and may not be necessary if you want to review all WARNed sites.

While it appears that UTM can and does apply categories based on both fully qualified domain name and path, I reduce my results to unique FQDNs.   Then I review the list to decide whether I can safely place it in a category so that users will not be warned in the future.  One a list of updates is identified, I add those sites to the Webites override list.   For sites that seem to be generating a lot of different warn conditions, I may create the exception for the site/organization (company.com) instead of the FQDN.   For sites that I don’t recognize, I do nothing so that the warning remains in place.

The uncategorized site review process should also be performed daily or frequently.

Great job, Douglas!

"A website category can be assigned to more than one user permission category, or it could be excluded from all user categories. In either of these situations, the webfilter results may be unpredictable or undesired ..." - I haven't heard of this before. What is the source of this knowledge?

Cheers - Bob

Thanks Douglas for your post. As this thread may come up within a search, I made an edit in my first post and mentioned your extraordinary post.

0-day for my ransomware is found here: helpx.adobe.com/.../apsb16-04.html
Distributed through ad networks on trusted sites.
Not blocked by flash filter of Sophos UTM.
(Ads have not been blocked to that time, Firefox was not used to that time, Ad-Blocker on client was not used at that time)

Sophos told on phone:
Ah yeah you know. We see that the admins are getting more careless. Our software is just an additional layer, but 100% security - it's not possible. It's just one layer and does not release the admin from his duties.

Dear Sophos (ok, they do not read here, but one never knows): Maybe you should also not write that you block flash in your UTM.
This information needs to be at the checkbox for flash blocking, so one knows, this feature cannot be trusted. I mean, I should expect that something that is written is working. This javascript tunneling seems not to be new.
Then saying...it's just a layer which you cannot trust, seems a bit, well,...strange.

Edit 26th Feb
Sophos sent a best practise guide per mail (Sophos Sales Team DACH) againt Crypto-Trojans covering also the Web Protection.

Thumbs up! Thanks.

Some follow-ups:

As frustrating as it is to identify and workaround websites with incorrectly-configured SSL certificates, this is a benefit not a problem.  I recently received a newsletter that talked about TLS interception tools, and argued that most of them were risky because they ignored SSL certificate errors themselves, while hiding the certificate problem from the user.   Part of the purpose of a certificate is to prove that the site you found is the site that you asked DNS to find, to protect against DNS attacks.  This protection only works if you enforce the rules strictly, the way that UTM does by default.  You can make the problem go away by creating an exception to ignore certificate errors for all sites, but this would be compromising your defenses significantly.

For uncategorized websites, I initially assigned categories as website exceptions, based on my best judgement.   This was inappropriate, as I don't have a good sandbox environment to test websites and I should not categorize a website that has not been evaluated.  The easier and safer solution is to crate an account on https://www.trustedsource.org, and submit the uncategorized websites to them for evaluation.   UTM documentation indicates that this is the service that UTM uses for categories and reputations.   The site asks for the McAfee product that you are using, and Sophos support told me that the correct choice is McAfee SmartFilter XL.   I tend to have about 10 sites per day to be evaluated, and they usually process those requests in a few hours.  This process makes the category and reputation available to all Sophos and McAfee users, while avoiding the performance penalty that would be expected if I configured a long list of website exceptions in UTM.

For sites that need Flash or Java enabled, one of the challenges is how to (a) know which version to run, (b) ensure that every PC is running the correct version, and (c) for Java, ensure a consistent configuration.   I am looking for "Windows Update for Java" and 'Windows Update for Flash", but have not found suitable and affordable products yet.  We have mitigated the problem by moving to virtual desktops, since all of those devices run whatever versions we install on our master image  But since one is only as strong as the weakest link, we have to worry about even one PC that has a wrong version of Java or Flash, or a poor security configuration for Java.

UTM also provides an option to disable Javascript, (which is part of the web browser and different from the Java plug-in).   A bit of testing will quickly show that too many sites depend on Javascript, so there is no realistic option other than to leave it enabled.

DouglasFoster said:

For uncategorized websites, I initially assigned categories as website exceptions, based on my best judgement. 

Where did you found the list of Uncategorized websites in UTM reports ?

Not easily done.   In my first post, I indicated that I had to build my own report tool.  Details below will allow you to create your own if you have an Access developer and a license.   Something similar could be done in another tool.

From Logging and Reporting, go to View Log Files... Achived Log Files...  Web Filtering

Download yesteday's log file (it is in .gz format).  
(Working with today's log file is fairly difficult because the current-day file may be large, and must be saved using copy-and-paste from the log viewer window.)

Download 7-zip (free) or another utility that can decompress .gz files.

Change the file extension of the unzipped file to .TXT so that it can be read from Microsoft Access.

Use custom VBA code and nested queries to put the file into a relational format.  Each record in the log file is a list of phrases in the form keyword="value", and you need to make it behave like a .CSV file.

The Access code follows for those who want to try:

In a Module, paste this function:

Public Function ParseLog(msg As String, itm As String) As String
Dim tmppos As Long, tmpstr As String
tmpstr = " " & itm & "="""
tmppos = InStr(1, msg, tmpstr)
If tmppos = 0 Then
ParseLog = ""
Exit Function
End If
tmpstr = Mid(msg, tmppos + Len(tmpstr))
tmppos = InStr(1, tmpstr & """", """")
ParseLog = Left(tmpstr, tmppos - 1)
End Function

Save a sample log file to "weblog.txt".   Create a linked table to the file using fixed-length format.   Access does not allow text strings over 255 characters, so I broke the message body into 250-character chunks.   Then I used this query to assemble it back together:

Query name:  Full_Message

SELECT W.EventTime, W.ProxyName,
Left([W].[Msg0],10) AS Function,
" " & Left(W.Msg0 & " ",250) &
Left(W.Msg1 & " ",250) &
Left(W.Msg2 & " ",250) &
Left(W.Msg3 & " ",250) &
Left(W.Msg4 & " ",250) &
Left(W.Msg5 & " ",250) &
Left(W.Msg6 & " ",250) &
Left(W.Msg7 & " ",250) &
Left(W.Msg8 & " ",250) &
Left(W.Msg9 & " ",250) AS msg
FROM Weblog AS W
cxWHERE (((Left([W].[Msg0],10))="httpproxy[") AND ((InStr(1,[W].[Msg0],'id="0003'))=0));

(Message ID=0003 appears to document the connection process for an htttps connection, and I found it of no value so it is excluded in the above query syntax)

Next, you need to parse the keyword=value pairs.   Assuming that you created the function shown earlier, this query gets the job done:

QUERY NAME:  ParsedFile

SELECT DateValue(Mid([F].[EventTime],1,4) & "-" & Mid([F].[EventTime],6,2) & "-" & Mid([F].[EventTime],9,2))+TimeValue(Mid([F].[EventTime],12,8)) AS EventTime, F.ProxyName,
F.Function AS WebFunction,
ParseLog(F.msg,"id") AS ItmId,
ParseLog(F.msg,"severity") AS ItmSeverity,
ParseLog(F.msg,"sys") AS ItmSys,
ParseLog(F.msg,"sub") AS ItmSub,
ParseLog(F.msg,"name") AS Itmname,
ParseLog(F.msg,"action") AS ItmAction,
ParseLog(F.msg,"method") AS ItmMethod,
ParseLog(F.msg,"srcip") AS ItmSrcip,
ParseLog(F.msg,"dstip") AS ItmDstip,
ParseLog(F.msg,"user") AS ItmUser,
ParseLog(F.msg,"ad_domain") AS ItmAd_domain,
ParseLog(F.msg,"statuscode") AS ItmStatuscode,
ParseLog(F.msg,"cached") AS ItmCached,
ParseLog(F.msg,"profile") AS ItmProfile,
ParseLog(F.msg,"filteraction") AS ItmFilteraction,
CDbl(ParseLog([F].[msg],"size")) AS ItmSize,
ParseLog(F.msg,"request") AS ItmRequest,
ParseLog(F.msg,"url") AS ItmUrl,
ParseLog(F.msg,"referer") AS ItmReferer,
ParseLog(F.msg,"error") AS ItmError,
ParseLog(F.msg,"authtime") AS ItmAuthtime,
ParseLog(F.msg,"dnstime") AS ItmDnstime,
ParseLog(F.msg,"cattime") AS ItmCattime,
ParseLog(F.msg,"avscantime") AS ItmAvscantime,
ParseLog(F.msg,"fullreqtime") AS ItmFullreqtime,
ParseLog(F.msg,"device") AS ItmDevice,
ParseLog(F.msg,"auth") AS ItmAuth,
ParseLog(F.msg,"ua") AS ItmUa,
ParseLog(F.msg,"exceptions") AS ItmExceptions,
ParseLog(F.msg,"category") AS ItmCategory,
ParseLog(F.msg,"reputation") AS ItmReputation,
ParseLog(F.msg,"categoryname") AS ItmCategoryname,
ParseLog([F].[msg],"application") AS ItmApplication,
ParseLog([F].[msg],"app-id") AS ItmAppid,
ParseLog(F.msg,"content-type") AS ItmContenttype,
ParseLog(F.msg,"overridecategory") AS ItmOverrideCategory
FROM FullMessage AS F
WHERE (((ParseLog([F].[msg],"id"))<>"0003"));

I note that the where clause to exclude id=0003 is redundant, since it was already excluded in the previous query.

Now you can treat the ParsedFile query as a relational table to build specific queries.

I also found it useful to have a tool to parse a full URL into its component parts:

Public Function ParseURL(parmUrl As Variant, parmitem As String) As String
Dim urlraw As String, item As String, tmppos As Long, proto As String, fqdn As String, path As String

item = "" & UCase("" & parmitem)
If item = "" Then
item = "D"
End If

urlraw = "" & parmUrl
tmppos = InStr(1, urlraw, "://")
If tmppos > 0 Then
proto = Left(urlraw, tmppos - 1)
urlraw = Mid(urlraw, tmppos + 3)
Else
proto = ""
End If

tmppos = InStr(1, urlraw, "/")
If tmppos > 0 Then
fqdn = Left(urlraw, tmppos - 1)
path = Mid(urlraw, tmppos + 1)
Else
fqdn = urlraw
path = ""
End If

If item = "P" Then
ParseURL = proto
ElseIf item = "D" Then
ParseURL = fqdn
ElseIf item = "F" Then
ParseURL = path
ElseIf item = "PF" Then
ParseURL = proto & "://" & fqdn
End If

End Function

To find uncategorized websites, search for T.ItmId="0071"

This query uses the ParsedLog table and the ParseURL function to return a list of uncategorized websites, with a count of how many times they were referenced in http or https mode.   The path after the host name is discarded.

SELECT ParseURL([T].[ItmUrl],"D") AS FQDN,
Sum(IIf(ParseURL([T].[ItmUrl],"P")="https",0,1)) AS Http,
Sum(IIf(ParseURL([T].[ItmUrl],"P")="https",1,0)) AS Https
FROM ParsedFile AS T
WHERE (((T.ItmId)="0071"))
GROUP BY ParseURL([T].[ItmUrl],"D")
ORDER BY ParseURL([T].[ItmUrl],"D");

The same concept can be used for parsing the other files as well.