Blocking Ads best practice (bulk updates to URL Groups?)

We use mostly Apple equipment (MacBook, iPhone, iPad) and on our MacBooks we have Little Snitch which is an application-aware outgoing firewall that kills attempts to reach out to advertising sites, trackers, etc. This doesn't help the phones and tablets though.

We could do something through DNS perhaps, but it seems like adding the worst offenders to a URL Group to block would work as well. The only problem is, how would you save, modify, or update a URL Group in bulk (i.e. not one-at-a-time through the GUI)?

Or is this not best-practice and we should use a different mechanism (DNS Pi Hole or something similar)?

(I'm thinking I could feed my discoveries to Sophos somehow to get them to reclassify some sites to Advertising, but that feels slow, they might not agree with my classification, and some folks might want to actually visit the sites in question because they use their services.)

P.S. In the Managed TLS Exclusion List there is "ecure.echosign.com" which I imagine is a copy/paste error and should say "secure", but maybe not.



Edited TAGs
[edited by: emmosophos at 7:34 PM (GMT -7) on 17 May 2022]
  • You can create a custom category by importing a txt file with all domains or using an external URL DB.

    The problem of using the external URL is, It only works over plain-text HTTP or FTP, and It's limited to 2,000 domains, if the file have more it won't work.

    The same applies if you import a txt file with all domains, It will also be limited to 2,000 domains.

    If you don't want to go over all the hassle of having to automate everything from fetching the lists, cutting them to fit 2,000 domains per file and having to use the Firewall API to mass import them, then use something as Pi-Hole or AdGuard.

    PS; If still want to do this over the Firewall, be careful when importing the domains, depending on how fast the processor of your box currently is you will have around >5 minutes of slowness since apparently everything gets inserted over a DB and the process hangs the CPU for a long time.

    At last It will look something like this:


    If a post solves your question use the 'Verify Answer' link.

    XG 115w Rev.3 v19 GA @ Home.

  • So Web > Categories > Add and it'll let me select a file. I assume to update I'd Delete and then Add again?

    Thanks for the warning on the add process performance concerns. I do have thousands of domains/hosts in Little Snitch, but I think a hundred or so of the major ones would do for site-wide banning. So hopefully if done off-peak it'll not have much effect.

    Right now, I can still afford to sample web filtering logs -- looking specifically at phones, which don't have Little Snitch, and are also lower-volume -- and pick out the more common offenders. And also perhaps rank Little Snitch entries and add the top offenders. So I think manual curation is good enough for me, and then it's a GUI upload from my laptop, which is easy.

  • Hi,

    are you using the web proxy with HTPS scanning enabled, if so you should be able to block advertising sites with policies. I have implemented a block advert policy on my XG and it works, though there are a lot of adverts inbuilt into web sites which only get blocked when you click on the link within the web page.

    Ian

    XG115W - v19 GA - Home

    1225v5 6gb ram, SSID, 4 NICs 20w - v19 EAP - on holiday.

    If a post solves your question please use the 'Verify Answer' button.

  • Not using the web proxy, am using HTTPS scanning. I'm hoping that the blocking takes place whenever a web page (in particular) reaches out to something on the list. But maybe it only applies under other circumstances, so I'll have to poke around.

    And I realize now that one confusion I had was the URL Groups can't be bulk-updated, but Categories can. Even though you can by-hand add addresses to either, only one is appropriate for this kind of use.

  • Hi Wayne,

    how can use use HTTPS scanning withourtusing the proxy, SSL/TLS and DPI only do part of the job?

    Ian

    XG115W - v19 GA - Home

    1225v5 6gb ram, SSID, 4 NICs 20w - v19 EAP - on holiday.

    If a post solves your question please use the 'Verify Answer' button.

  • It's my impression that the web proxy is a legacy application and that if you check the Web boxes -- except the "Use web proxy instead of DPI engine" checkbox -- you get everything the web proxy provided and then some. Am I just misunderstanding how it works?

  • The dpi engine does not do any of the google functions, does not scan web policies completely (block web sites), does not scan UDP.

    If you check any of the web boxes but not use the web proxy, you still get the web proxy, not the DPI.

    Scans all TCP ports but not URLs.

    Ian

    XG115W - v19 GA - Home

    1225v5 6gb ram, SSID, 4 NICs 20w - v19 EAP - on holiday.

    If a post solves your question please use the 'Verify Answer' button.

  • So just to be clear the Web Boxes are under Web Filtering:

    • Web Policy

    • Apply web category-based traffic shaping

    • Block QUIC protocol

    • Scan HTTP and decrypted HTTPS

    • Use zero-day protection

    • Scan FTP for malware

    • Use web proxy instead of DPI engine

    • Decrypt HTTPS during web proxy filtering

    And I have a Web Policy, Scan HTTP and decrypted HTTPS, and Use Zero-Day Protection. So does that mean I'm actually using the Web Proxy instead of DPI?

    You need to use the web proxy to get Youtube and other restrictions, but none of the other things you mention are in the documentation. Is the documentation just wrong?

    I'm definitely seeing Web filtering disallowing things, though I have to admit a lot fewer than I had anticipated. (On the other hand, my laptop's Little Snitch will stop almost everything before it leaves the laptop, so the firewall will never see it.)

  • I suspect not wrong just not considered needed in the documentation. Not scanning UDP is documented somewhere maybe in SSL/TLS which uses the DPI engine.

    You need to install the XG CA to use https scanning.

    Ian

    XG115W - v19 GA - Home

    1225v5 6gb ram, SSID, 4 NICs 20w - v19 EAP - on holiday.

    If a post solves your question please use the 'Verify Answer' button.

  • Yes, I've installed the CA on all devices on my LAN. Got a couple of VLANs for IoT, Guest, work-owned/admin'd machines on which I don't do TLS decryption. That plus the many exceptions (apple.com, etc) means 2/3+ to TLS traffic is not decrypted. And hence not scanned. But I try.

    And maybe UDP isn't considered. I'll have to poke around on that. (In one place, they say that services are port-only and ignore the protocol, so my hope was that it does everything it can. On the other hand, blocking QUIC (port 443 UDP I think) does something about that issue.

    It should be a little clearer in the docs. Just as there should be a central doc on Traffic Shaping that covers the four types and how each is set up differently.