This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

User reports don't show hardly any of the traffic users are generating

If we parse the raw logs from the Sophos UTM, we can build our own reports that seem to show all the URL's that our users are visiting. However, if we try to use any of the native reporting tools they results only show a fraction of the sites visited. Why the disparity? What is the purpose of the built in reports?

All I want to do is select a User/Computer/IP address and show all the sites they have visited in the past Day/Week/Month.

I have tried the Logging/Reporting > Web Protection reports and Logging/Reporting > View Log Files > Search Log Files > Web Filtering. The latter tends to show more, but not all.


Am I missing something required to do this simple reporting on users? Does reporting require a third party product?

Sophos proudly touts the number of built in reports, but I find that the quantity is irrelevant if the data is questionable or not easily extracted.



This thread was automatically locked due to age.
Parents
  • Tim, it's hard to work without examples of what you're seeing...

    In 'Reporting Settings', you can select the detail retained in Reporting.

    If you have a firewall rule like 'Internal (Network) -> Web Surfing -> Internet : Allow', accesses can occur that don't transit the Proxy.  This can happen if Web Filtering is in Standard mode, but the individual client isn't configured correctly.  It can happen with HTTPS if the Proxy is in Transparent but is not inspecting HTTPS traffic.  It also can happen in Transparent if someone uses anything other than port 80/443 and there is a firewall rule allowing the traffic.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • There was a firewall rule like 'Internal (Network) -> Web Surfing -> Internet : Allow', so I disabled it. So I guess this indicates that the proxy component does not rely on the firewall for access.

    Web Filtering was in Standard mode with 'Block access on authentication failure' disabled.

    Created a new web filter profile scoped to my computer to test. No change when blocking enabled. Checked policies and found that a custom Filter Action > Additional Options > Activity Logging has "Log accessed pages" disabled. After I enable that more traffic started showing up in the logs.

    Basically this was dropping all the authenticated traffic from the Web Filter log, which is apparently used for reporting. Going to let it run tomorrow with all users and see how it reports.

  • Everything looked good today.

    We are also evaluating Fastvue Sophos Reporter and noticed an omission. We can't get details on outbound file transfers. Fastvue support informed me that the Web Filter on Sophos does not report on uploaded data. The could be useful for tracking DLP and malware on the network. Anyone know if this is true?

  • DLP is done for Email, but not Web Filtering.  Web Filtering relies on Categorization and Reputation to block access to sites that accept stolen information.  Anti-Virus and Advanced Threat Protection should address questions of malware.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Wow......

    I'm so glad I come across this post. This is a very easy mistake to make when setting up the UTM.

    Most users (including myself) will basically bring up the UTM with a lan and then a wan. To get access to the internet, you would then put rules eg dns & web browsing in the firewall rules.

    Web filtering comes later once you know you are on the internet. And because you see things going through the logs etc, it is easy to assume that it's working and the initial dns/web browsing rules in the firewall would be easily forgotten about.

    So ultimately, if you use the web proxy, dns proxy & smtp proxy, you don't need any of those rules in place on the firewall......

    Maybe they should add some sort of warning or popup to say that you have both enabled and should disable one or the other eg a conflicts page on the UTM which lists questionable setups eg you have port 25 open on the firewall but the smtp proxy is enabled. You might want to review this?

    It never once dawned on me that you get the lan going, the wan going and simply turn on web filtering and put your lan in the allowed networks to get to the internet. And I bet lots of others have done the same......

  • Careful with deleting the rules if you have hosts that are skipping the proxy, especially if running in transparent mode.  If they skip the proxy, and there are no rules allowing them to the internet, they will not have internet access.  I would definitely use a host group for each (the skip list and the firewall rule - haven't tried so assuming you can do this) to make sure you only have to make changes in one place.

  • I disabled the web browsing rule (not deleted) for all of our subnets and they are all still going through. Not had any complaints yet.

    I've also enabled caching which does seem to have improved browsing performance with no complaints either.

    Early days, but will keep you posted.

  • Louis, you might be interested in DNS Best Practice.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Hi Bob,

    yes we do that. I'm setting another UTM SG330 today so will try without adding any FW rules once we're connected and see how we get on.

    It's just a break from the norm because most will be used to setting up DNS/Web browsing rules at the beginning.....

  • What a great idea!

    I came across this when we were adding another UTM into the network. All web traffic was proxied and certain download sizes were in force.

    The new UTM would not update as it was going through the proxy which was denying the download size.

    Now the answer would be to put the new UTM in the skip host section on the proxy but you would also have to put a FW rule in to allow the new UTM to http.

    So the great idea from darrellr is rather than adding hosts (the UTM in this case) to both the skip list and fw rules, simply creat a "Proxy skip Host Group" and add this to the skip hosts and fw rule. That way in the future, when you need a host to skip the proxy, you simply add it into the "skip proxy group" and voila......

Reply
  • What a great idea!

    I came across this when we were adding another UTM into the network. All web traffic was proxied and certain download sizes were in force.

    The new UTM would not update as it was going through the proxy which was denying the download size.

    Now the answer would be to put the new UTM in the skip host section on the proxy but you would also have to put a FW rule in to allow the new UTM to http.

    So the great idea from darrellr is rather than adding hosts (the UTM in this case) to both the skip list and fw rules, simply creat a "Proxy skip Host Group" and add this to the skip hosts and fw rule. That way in the future, when you need a host to skip the proxy, you simply add it into the "skip proxy group" and voila......

Children
No Data