HTTPS decrypt and scan - not identifying embedded URLs

Web > Exceptions >  Apple Update.

By default, apple.com and a few other sites are not HTTPS decrypted or have any policy enforced.  If the "embedded advertising" urls come from apple domains they will not be blocked.  If you know the URL you can also look in Log Viewer (which should tell you the policy, exceptions, and decision) as well as Policy Tester.

Thank you Michael,

I understand about the Apple sites, they are not what I am asking about, it is the advertisements embedded in other sites eg dpreview. 

Ian

Added :- if you click on a link the XG then sees the link and blocks it as an advertisement.

If the page displaying the ad causes a http/s request to a site that is categorized as advertising then that request should be blocked.

If the page displaying the ad causes a http/s request to an site that is in an exception then that request will be allowed and the ad displayed.

If the page displaying the ad has the ad embedded in its own html then the ad is displayed.

My point is rather than saying "why am I seeing ads" you have to say "why is the request to site x being allowed".  You need to look at what is actually going on in the html and http requests.

>Why aren't the embedded advertisement URLs identified and classified then blocked?

What are the URLs?
What does Log Viewer say that it did about those URLs?
What does Policy Tester say that should be done about them?

Hi Michael,

the screenshot below is what I am asking about. Now if you click on an advertisement it is identified as an advertisement and blocked. Sometime in the past some of these embedded site were blocked, but that seems to have gone.

Ian

I just tested.  The only thing in my test policy is block Advertisements.




I have no top or side advertisements.

If I look at the Log Viewer, filtering for that category:



Double check your web policy and your exceptions.  For example if you have "Allow image files" before "Block Advertisements" then some of those might be allowed.  Of if you have an exception for googlesyndication.com then all the ads from that will be allowed.

I just tested.  The only thing in my test policy is block Advertisements.




I have no top or side advertisements.

If I look at the Log Viewer, filtering for that category:



Double check your web policy and your exceptions.  For example if you have "Allow image files" before "Block Advertisements" then some of those might be allowed.  Of if you have an exception for googlesyndication.com then all the ads from that will be allowed.

Hi Michael,

thank you of rthe detailed response. I have re-organised the web policy order which seems to have some detrimantal affect on accessing this site, makes it very slow.

I don't have the googlesyndication.com URL anywhere that I can find.

The issue not blocking adverts appears to be a safari issue, because FF blocks the ads. though none of the blocked sites appear in logviewer - web page. I do use IPv6 and see some of the preview sites appearing with IPv6 addresses.

An interesting side affect is that access to some other sites have sped up.

Ian

WebAdmin > Firewall Rules > IPv6

Remember that IPv6 has its own set of firewall rules.  You may need to make sure that you have the same policy applied in both IPv4 and IPv6.

For example, in IPv6 are you hitting an Allow All policy.


There is nothing that the Web Proxy does that makes sites slow, but there are things that a site/page itself may do if some of the resources that it tries to load are blocked.

Again, things like Log Viewer and the Browser F12 - Network tab are helpful in seeing what is actually loading and what is blocked.

Hi Michael,

thank you for the follow-up.

I have the same policies applied to both IP4 and IPv6 firewall rules.

The issue being that the XG does not recognise IPv6 FQDNs in the IPv6 firewall rules, so you need to use IPv6 addresses, which need to be continually checked for new ranges for a specific site. Now this was not an issue when I used UTM.

Ian