Blocking Ads best practice (bulk updates to URL Groups?)

We use mostly Apple equipment (MacBook, iPhone, iPad) and on our MacBooks we have Little Snitch which is an application-aware outgoing firewall that kills attempts to reach out to advertising sites, trackers, etc. This doesn't help the phones and tablets though.

We could do something through DNS perhaps, but it seems like adding the worst offenders to a URL Group to block would work as well. The only problem is, how would you save, modify, or update a URL Group in bulk (i.e. not one-at-a-time through the GUI)?

Or is this not best-practice and we should use a different mechanism (DNS Pi Hole or something similar)?

(I'm thinking I could feed my discoveries to Sophos somehow to get them to reclassify some sites to Advertising, but that feels slow, they might not agree with my classification, and some folks might want to actually visit the sites in question because they use their services.)

P.S. In the Managed TLS Exclusion List there is "" which I imagine is a copy/paste error and should say "secure", but maybe not.

Edited TAGs
[edited by: emmosophos at 7:34 PM (GMT -7) on 17 May 2022]
  • You can create a custom category by importing a txt file with all domains or using an external URL DB.

    The problem of using the external URL is, It only works over plain-text HTTP or FTP, and It's limited to 2,000 domains, if the file have more it won't work.

    The same applies if you import a txt file with all domains, It will also be limited to 2,000 domains.

    If you don't want to go over all the hassle of having to automate everything from fetching the lists, cutting them to fit 2,000 domains per file and having to use the Firewall API to mass import them, then use something as Pi-Hole or AdGuard.

    PS; If still want to do this over the Firewall, be careful when importing the domains, depending on how fast the processor of your box currently is you will have around >5 minutes of slowness since apparently everything gets inserted over a DB and the process hangs the CPU for a long time.

    At last It will look something like this:

    If a post solves your question use the 'Verify Answer' link.

    XG 115w Rev.3 v19 GA @ Home.

  • So Web > Categories > Add and it'll let me select a file. I assume to update I'd Delete and then Add again?

    Thanks for the warning on the add process performance concerns. I do have thousands of domains/hosts in Little Snitch, but I think a hundred or so of the major ones would do for site-wide banning. So hopefully if done off-peak it'll not have much effect.

    Right now, I can still afford to sample web filtering logs -- looking specifically at phones, which don't have Little Snitch, and are also lower-volume -- and pick out the more common offenders. And also perhaps rank Little Snitch entries and add the top offenders. So I think manual curation is good enough for me, and then it's a GUI upload from my laptop, which is easy.

  • Not using the web proxy, am using HTTPS scanning. I'm hoping that the blocking takes place whenever a web page (in particular) reaches out to something on the list. But maybe it only applies under other circumstances, so I'll have to poke around.

    And I realize now that one confusion I had was the URL Groups can't be bulk-updated, but Categories can. Even though you can by-hand add addresses to either, only one is appropriate for this kind of use.

  • Hi Wayne,

    how can use use HTTPS scanning withourtusing the proxy, SSL/TLS and DPI only do part of the job?


    XG115W - v19 GA - Home

    1225v5 6gb ram, SSID, 4 NICs 20w - v19 EAP - on holiday.

    If a post solves your question please use the 'Verify Answer' button.

  • It's my impression that the web proxy is a legacy application and that if you check the Web boxes -- except the "Use web proxy instead of DPI engine" checkbox -- you get everything the web proxy provided and then some. Am I just misunderstanding how it works?

  • The dpi engine does not do any of the google functions, does not scan web policies completely (block web sites), does not scan UDP.

    If you check any of the web boxes but not use the web proxy, you still get the web proxy, not the DPI.

    Scans all TCP ports but not URLs.


    XG115W - v19 GA - Home

    1225v5 6gb ram, SSID, 4 NICs 20w - v19 EAP - on holiday.

    If a post solves your question please use the 'Verify Answer' button.

  • So just to be clear the Web Boxes are under Web Filtering:

    • Web Policy

    • Apply web category-based traffic shaping

    • Block QUIC protocol

    • Scan HTTP and decrypted HTTPS

    • Use zero-day protection

    • Scan FTP for malware

    • Use web proxy instead of DPI engine

    • Decrypt HTTPS during web proxy filtering

    And I have a Web Policy, Scan HTTP and decrypted HTTPS, and Use Zero-Day Protection. So does that mean I'm actually using the Web Proxy instead of DPI?

    You need to use the web proxy to get Youtube and other restrictions, but none of the other things you mention are in the documentation. Is the documentation just wrong?

    I'm definitely seeing Web filtering disallowing things, though I have to admit a lot fewer than I had anticipated. (On the other hand, my laptop's Little Snitch will stop almost everything before it leaves the laptop, so the firewall will never see it.)

  • I suspect not wrong just not considered needed in the documentation. Not scanning UDP is documented somewhere maybe in SSL/TLS which uses the DPI engine.

    You need to install the XG CA to use https scanning.


    XG115W - v19 GA - Home

    1225v5 6gb ram, SSID, 4 NICs 20w - v19 EAP - on holiday.

    If a post solves your question please use the 'Verify Answer' button.

  • Yes, I've installed the CA on all devices on my LAN. Got a couple of VLANs for IoT, Guest, work-owned/admin'd machines on which I don't do TLS decryption. That plus the many exceptions (, etc) means 2/3+ to TLS traffic is not decrypted. And hence not scanned. But I try.

    And maybe UDP isn't considered. I'll have to poke around on that. (In one place, they say that services are port-only and ignore the protocol, so my hope was that it does everything it can. On the other hand, blocking QUIC (port 443 UDP I think) does something about that issue.

    It should be a little clearer in the docs. Just as there should be a central doc on Traffic Shaping that covers the four types and how each is set up differently.

  • For your purposes the web filtering done by the web proxy and done by the DPI engine is the same. While the web proxy is older technology it is still absolutely supported. There are a few things the web proxy can do that DPI cannot, they are all documented in WebAdmin.

    HTTP over UDP is the QUIC protocol - I think we changed the name of the option in WebAdmin at some point. Neither the proxy or dpi can handle QUIC.

    rfcat is incorrect/misleading when he says that DPI "does not scan web policies completely (block web sites)".

    With your configuration if you have traffic going through port 80/443 it will use DPI mode. However if you configure the computer/browser to use a proxy at 3128 then it uses the web proxy.

  • As the XG does have categorization it will block a large number of sites automatically. However it will not know about all sites, categorization is imperfect.  Part of the problem is context - when your browser visits it sends all sorts of cookies and headers and it gets back ads.  When our categorizer visits the same url without any of those details sometimes it doesn't get ads.  Or there are things that we know that the domain gets used for multiple things.

    AFAIK Little Snitch works on the laptops and stops things before they are sent to the XG. So you won't be able to tell what things would have been caught by the XG if it made it there.
    So if you just took the list from Little Snitch and put it in the XG, you might end up duplicating a huge number of entries that are already categorized and blocked.

    One by one you can use the policy tester to report on the category of every url so you can see if it would have been blocked had it made it to the XG.

    Editing URL Groups and Custom Categories can also be done using the XML API, if you wanted a more programmatic / bulk way of doing things. I don't know the limit in the number of entries in URL Groups. AFAIK for Custom Categories is it 2000 domains, even using XML API.

  • Hi Michael,

    my statement is based on information provided in the XG GUI screens.


    XG115W - v19 GA - Home

    1225v5 6gb ram, SSID, 4 NICs 20w - v19 EAP - on holiday.

    If a post solves your question please use the 'Verify Answer' button.

Reply Children
No Data