Setting up policy. From block all, to allow specific sites but filter them

Will this work:

For the most restricted users

In the policy, 

• In the web policy, Enable "Block all content, except as specified below", and set all categories to BLOCK
• In the web policy, Enable "Block all websites with a reputation below: Suspicious", so that infected sites will be blocked even if you would otherwise trust them.
• In website exceptions, give allowed domains a TAG, and check the box to apply to subdomains.
• In the web policy, apply that TAG name(s) that are applicable.
• In the web policy "Websites" list, add exceptions for the any domains that you want to allow or block differently.

I am pretty confident that Reputation blocks take precedence over everything except website reputation overrides.   I would never use a website exception to raise a site's reputation ---  make them go through the TrustedSites.org reevaluation if they think they have been cleaned up from an infection.

After reputation, this is the order of processing, based on a support question that I raised awhile back:

• Policy website Allowed Sites are all evaluated first.   If there is a match, the site is allowed.
• Policy website Blocked Sites are all evaluated second.  If there is a match, the site is blocked.
• Policy website Tags are evaluated third.   If there is a match, the site is allowed.
• Policy Category rules are evaluated next, and can result in block, allow, or quota.  As you said, you need these to be all blocked.

Other gimmicks:

• Consider configuring DNAT rules to redirect to a non-existent address.   These can be configured based on source ip, user, or group.  Don't use firewall rules.

Cautions:

• Most commercial websites include content from lots of other sites.   There is a significant risk that your "Allowed" websites will not work because of your desired restrictions.

Will this work:

For the most restricted users

In the policy, 

• In the web policy, Enable "Block all content, except as specified below", and set all categories to BLOCK
• In the web policy, Enable "Block all websites with a reputation below: Suspicious", so that infected sites will be blocked even if you would otherwise trust them.
• In website exceptions, give allowed domains a TAG, and check the box to apply to subdomains.
• In the web policy, apply that TAG name(s) that are applicable.
• In the web policy "Websites" list, add exceptions for the any domains that you want to allow or block differently.

I am pretty confident that Reputation blocks take precedence over everything except website reputation overrides.   I would never use a website exception to raise a site's reputation ---  make them go through the TrustedSites.org reevaluation if they think they have been cleaned up from an infection.

After reputation, this is the order of processing, based on a support question that I raised awhile back:

• Policy website Allowed Sites are all evaluated first.   If there is a match, the site is allowed.
• Policy website Blocked Sites are all evaluated second.  If there is a match, the site is blocked.
• Policy website Tags are evaluated third.   If there is a match, the site is allowed.
• Policy Category rules are evaluated next, and can result in block, allow, or quota.  As you said, you need these to be all blocked.

Other gimmicks:

• Consider configuring DNAT rules to redirect to a non-existent address.   These can be configured based on source ip, user, or group.  Don't use firewall rules.

Cautions:

• Most commercial websites include content from lots of other sites.   There is a significant risk that your "Allowed" websites will not work because of your desired restrictions.

Dear Douglas thanks for your reply.

But since I am a newbie to the Web Filtering interface I got a bit confused with your instructions.

Please check my comments with red font. 

In the policy, 

• In the web policy, Enable "Block all content, except as specified below", and set all categories to BLOCK
• I created a new Policy and named it "Block all content, except as specified below", in order to Block all
• In the web policy, Enable "Block all websites with a reputation below: Suspicious", so that infected sites will be blocked even if you would otherwise trust them.
• I have enabled filtering per reputation as advised.
• In website exceptions, give allowed domains a TAG, and check the box to apply to subdomains.
• ? Should I go to Web Protection--> Filtering Options --> Exceptions and create a new Exception List? And set "For all requests" to "Going to websites tagged as" and set the "Allowed Websites" tag.
  Or should i got Web Protection--> Filtering Options --> Websites and create "New Sites" as suggested below?
• In the web policy, apply that TAG name(s) that are applicable.
• In the web policy "Websites" list, add exceptions for any domains that you want to allow or block differently.
  ? Sorry I got confused

I am pretty confident that Reputation blocks take precedence over everything except website reputation overrides.   I would never use a website exception to raise a site's reputation ---  make them go through the TrustedSites.org reevaluation if they think they have been cleaned up from an infection.

After reputation, this is the order of processing, based on a support question that I raised a while back:

How are the policies processed? I have tried to set the priority of each policy but did not work as expected. If I have a website to allowed it skips the block web page policy created. I have not found a Document or instruction in order to create policies and that work with priority.

• Policy website Allowed Sites are all evaluated first.   If there is a match, the site is allowed.
• Policy website Blocked Sites are all evaluated second.  If there is a match, the site is blocked.
• Policy website Tags are evaluated third.   If there is a match, the site is allowed.
• Policy Category rules are evaluated next, and can result in block, allow, or quota.  As you said, you need these to be all blocked.

Other gimmicks:

• Consider configuring DNAT rules to redirect to a non-existent address.   These can be configured based on source ip, user, or group.  Don't use firewall rules.

Cautions:

• Most commercial websites include content from lots of other sites.   There is a significant risk that your "Allowed" websites will not work because of your desired restrictions.
  The issue here is the blocked websites are not working and the allowed does. I expect some websites to not fully work since the content in most cases is hosted to other domains or CDNs but I will add the required allowed domains or FQDN per page.

Thanks in advance.

Almost all of my answer was about features within a single FILTER ACTION.

A note on tags.   You make up your own keywords to use as tags, and assign them to websites using website exceptions.    The tag name itself has no significance, other than to invoke a result when the same tag name appears in a Filter Action.

It sounds like your other problem is that policy selection is not doing as you intend.

1) You need to determine if users are being assigned to the intended groups.    If you are using Active Directory or LDAP, you can enter a username and password to find out what groups are being applied to that user.   Some of the other authentication methods may or may not have this feature, as I have not used them.   NOTE: firmware 408-4 has a bug which will cause this test to fail, even with the correct password.  The following version 9.409-9 has it fixed and the earlier firmware do not have the bug.

2) You need to see what policy and filteraction are being applied to the user.   To verify this, you need to use Live Log:

You need two PCs, one to run WebAdmin, and one to use for testing.   If you try to do both on one PC, the WebAdmin updates will add confusion to your analysis of the log files.

On one PC, start UTM WebAdmin, navigate to Logging and Reporting... View Log Files... Today's Log Files (tab) ... Web Filtering... Live Log (button)

When the Live Log window opens, put the IP address of the test PC into the Fillter box, and press Enter to have it take effect.   Ignore the entries that are displayed, as you are only interested in the new entries that appear after the Filter has been defined.   (Using the Reload will not make the unwanted entries go away.)

On the test machine, navigate to one or more of the websites that are handled incorrectly, and do your test browsing.   Allow at least 15 seconds after the end of your test for all of the log records to be displayed.   Then copy and paste the new rows into Notepad or another text editor for review.

Each log entry is a fixed-length header, followed by a long string of entries in the format of ' keyword1="Value1" keyword2="value2"  This is one log entry, reformatted onto multiple lines for readability, with a lot of the keyword pairs removed:

2017:03:18-10:10:39
name="http access"
action="pass"
srcip="192.168.26.143"
dstip="66.256.1.99"
user="--omitted---"
group=""
ad_domain="--omitted---"
profile="REF_HttProContaInterNetwo3 (Active Directory & LDAP)"
filteraction="REF_DefaultHTTPCFFAction (Default content filter action)"
url="https://---omitted-----/"
referer=""
error=""
ua="Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
exceptions="content,ssl,fileextension,patience"

From the logs, you will be interested in USER, PROFILE, FILTERACTION, and EXCEPTIONS.

To figure out what happened, you will also need to infer the FILTER PROFILE.

Processing Logic:

• Source IP, Authentication Method, and Proxy method are used to determines the FILTER PROFILE.
• The FILTER PROFILE determines whether HTTPS inspection will be used.
• The FILTER PROFILE also determines the list of Policies that will be considered, and their order of evaluation.   The first successful profile match is selected.
• The PROFILE determines the FILTER ACTION.
• The FILTER ACTION determines most of the Block/Allow logic, and some of the exceptions.   Other exceptions are defined in the Web Site Exceptions list.

So when things don't work as expected, you need to ask:

• Was the intended FILTER ACTION selected?  If so, then the FILTER ACTION needs corrections to its configuration.
• If the wrong FILTER ACTION was selected, was the correct PROFILE selected?  If not, is it an issue with profile order, group membership, or filter profile?
• Is the user authenticated correctly, or is the user="value" empty, implying a default policy was used.

Hope this helps

Dear all,

I really appreciate your assistance and your time.

Upon my evaluation, i did not manage to find a way to deploy our required rules.
Despite all your help the rules sometimes were working and most of the times did not work as expected.

I had contacted Sophos Support Team for two major cases. The first was a related with an incorrect configuration I had regarding the AD integration. It took 7+ days for the Sophos Team to investigate it.
For the second issue (regarding the question submitted also here), there has been over a week and they never contacted me back. They never responded to my ticket.

To conclude using a product that has some bugs is ok, but with almost no support it is a no go for my company.

I will move on in order to find other tools which will suit for our requirements.

I appreciate your help and effort. I hope sometime soon that Sophos will use Cyberoam's support procedure which was awesome last time I required their help.

Best Regards.

Nikos, my business has done large roll-outs of new technologies many times over the years.  In each case, we hired one of the top consultants we could find to guide us in our initial project.  I wanted us to develop a culture that would create solid, robust solutions from the beginning and would make it easy for top talent to help us if needed.

In the case of the UTM, I've seen more than one WebAdmin configuration done for the first time by a very talented CCIE.  Such configurations are a mess unless the CCIE worked with an experienced installer of the Sophos UTM.  I typically have to charge twice as much to fix such installations than if I'd done the initial configuration myself.

If you're not allowed to spend money on an experienced UTM consultant, you should stay with tools you already know.  If that's only TMG, then you may want to prep your management to spend money for a consultant with the next tool you try.

I don't mean to sound harsh, but my advice is heartfelt and I didn't want to leave any doubt as to my meaning.

Cheers - Bob