Regarding website filtering

User or group determines Policy, Policy determines Filter Action, Filter Action determines how Uncategorized sites are handled.

So you need to create at least one non-default Policy.  You also need to a user authentication method configured on the Filter Profile.

Hi, Kiran, and welcome to the UTM Community!

There is a trick with "Uncategorized" as there is a Category named "Uncategorized" and there are sites that are "Uncategorized" because no Category has been determined for the site.

Cheers - Bob

I have to quibble Bob.  I just rechecked and I have only one type of Uncategorized in the administrative interface for web proxy

Perhaps you were remembering that there are options for both "Uncategorized" and "Categorization Failure".  The latter would mean that the Categorization database is not reachable.

Assuming that you're not looking at Sophos at home because I haven't seen that but there should be a catergory selector for "Uncategorized websites" at the bottom of the Categories tab in your filter action.  This is where you set whether to allow or block uncategorized sites.

However I'd be wary of doing that big bang so to speak.  You'll block a lot of legitimate sites.  Just run a few reports on activity logged against the Uncategorized category and you'll see just how many sites are still in that default category.

Again, guys, there is a sub-category "Uncategorized" that you can select when you define a new Category on the 'Categories' tab in 'Web Filtering Options'.  This is an assigned category, not the lack of any category assignment.

At the bottom of the Categories section of a Filter Action, you can select to Allow 'Uncategorized websites'.  These are sites that have had no category assigned.  They haven't even been assigned 'Uncategorized'.  Perhaps this is what you mean as "categorization failure," Doug?

At least, that's how I've understood and used these concepts - I'd be happy to have my knife sharpened though!

Cheers - Bob

cheers Bob

been that long since we built our category list that I'd forgotten about that   :)

Well Bob, that is probably a bug and it probably has existed since...  at least 9.0 and maybe even before that.  :)

Although I cannot be sure without testing, I believe that if you create a Category Group containing "uncategorized" it will do nothing.

I don't think it's a bug, Michael, unless Sophos doesn't have an "Uncategorized" category in it's SXL database like CFF does.  I guess that's what you're telling me - my understanding no longer applies to the UTM as CFF is no longer used?

Hmmmm, in my lab, I still get 0 when I do:

cc get http use_sxl_urid

Cheers - Bob

AFAIK both CFF and SXL have the same thing.  A single "response from the server" that is a hexadecimal number (0xFF) that maps to the the word "Uncategorized".  AFAIK, the underlying database does not have an "explicitly defined" Uncategorized that is seperate from the "I have no information" Uncategorized.  Or if it does, I don't think that gets passed back to the client in either CFF or SXL.  Regardless the httpproxy only has a single Uncategorized concept.  The fact that there are two ways to configure it in the UI is a problem and I don't know what the system does if there is a conflict.  I suspect the one in the Category Group is ignored.

Of course, I could be wrong.

cc get http use_sxl_urid
cc get http sc_local_db

If use_sxl_urid is 1 then it will use SXL
If use_sxl_urid is 0 AND sc_local_db is none then it will use CFFS
If use_sxl_urid is 0 AND sc_local_db is mem|disk then it will use CFF with a local database


Fun Fact:
We are currently updating the TrustedSource SDK that the SXL servers use to the latest version.
In an upcoming UTM release we will be updating the TrustedSource SDK that the local CFF database uses.
We are not going to update the SDK that the CFFS servers use.

Therefore SXL users will get very slightly better categorization automatically as soon as we finish the SXL server.  Local db will get very slightly better categorization when they upgrade to the new version.  Anyone still using CFFS (which is anyone on 9.0 or earlier, or anyone who has manually changed their settings) will have the same categorization.
By "slightly better" I mean less than 0.01% improvement.  :)

Just revisited this to understand your point.   I was burned on the sub-category / super-category distinction.   My "Uncategorized" sub-category is in the "web filtering problems" super-category, and only super-categories appear in the Filter Action menus.

In the logs, I have all of the folllowing CategoryCode/Category pairs from the web logs:

9998 - Uncategorized

9998,9998 - Uncategorized,Uncategorized

9999 - Categorization failed

The "9998,9998" entry is the most common.   For three days of logs:

Category#   Log Entries
9998                 94
9998,9998   52,239
9999                 43

(out of 1,668,457 total log entries)

Many of these entries appear to be categorized sites that become uncategorized when the path gets long, and McAfee shows most of them with valid categories.   It would help to know which Uncategorized code is which.

Support had led me to believe that the new search engine was going to be a fix for Uncategorized sites.   Michael Dunn's comment is discouraging, as it suggests that there may be no fix.