We'd love to hear about it! Click here to go to the product suggestion community
I have various rules for users with different time restrictions, and then a final rule to always allow various sites and services.
I have added an application policy called "Always Allowed" which includes the application "NTP". It isn't working though.
The application log shows the destination port of 123, but no Application Category or Application, and an Action of Denied. The Policy ID is my catchall rule, and the Message ID is 17051. I'm guessing that port 123 isn't getting correctly detected as NTP and so is being blocked.
The particulars of my rule are:
Source Zone / Network / Time: LAN / Any / All the time
Dest Zone / Network / Services: WAN / Any / Any
Match known users: unticked
Malware scanning: Only HTTP ticked
Intrusion Policy: None
Traffic Shaping Policy: None
Web Policy: None (I have tried Allow All too)
Application Policy: Always Allowed (my rule that includes NTP)
Any idea why this isn't working?
In reply to lferrara:
UTM does non-HTTP/HTTPS as well. I just checked, there are application definitions for things like DHCP. Definitions like Clash of Clans probably (I don't know for sure) apply to the higher level ports they connect to the game servers with.
In scenario 6, the traffic is blocked by the firewall. The application filter does not come into play. An application filter Allow cannot override a firewall Block.
Let me rephrase 2 more ways:
1) In order for HTTPS traffic to be allowed it must be allowed by all of these systems (Firewall, Category, Certificate, Antivirus, Application) and if any one of those systems says Blocked then the traffic is blocked. In order for NTP traffic to be allowed it must be allowed by all of these systems (Firewall, Application) and if any one of those systems says Blocked then the traffic is blocked.
2) Traffic passes through a series of filters, in order. Each filter has the ability to block the traffic, or allow it through its own filter and on to the next filter. A packet on port 123 (which is what NTP uses) arrives with a particular source and destination. The firewall tries to find a firewall rule that says matches that source, destination, and port. If no firewall rules are matched the packet is dropped. If a firewall rule does match then that rule is selected and it looks at the configured Application Filter for that firewall rule. The IPS then looks at the packet on port 123 and it can apply its own filter about whether that packet is NTP or not and whether to drop it or to pass through its filter. Possibly after IPS there may be other filters.
One point in (2) that explains this specific scenario. Lets say you have 5 rules, each of them have different Application Filters. None of the rules apply to port 123. Given that no firewall rule matches, how would the system even know what application filter you wanted? You must have a matching firewall rule before you can even choose which policy that IPS applies.
In reply to jamesharper:
There is hope. I have no idea why this started working. I just know that it did.
Could it be this?
In reply to Michael Dunn:
Reading the scenario 6 again gave me the idea why traffic is blocked (because the Application filter exists but it is not attached to any firewall rule). Sorry but it was not clear for me the scenario 6.
I know how the packets are inspected and how the flow works.
So in terms of Application filter, this is something newer and should be improved. I mean, if the application filter is used to allow/block application (even non-http/s), you should add more and more application inside the filter. For example, by creating application for Office365, Playstation games, etc...instead of using Firewall rules.
At the moment if a certain application is allowed, on XG admins can get mad because allowed Applications are not even logged. So imagine 20/30 rules with each one with its own Application filters, finding which App and which firewall rules (because combination of them can exist) can be very hard to debug.
So you should log even allowed applications, as UTM9 does or find a proper way to give us more log without using tcpdump.
I'm not the one who is determining which applications should be included or not. But if you think there are applications that should be added, feel free to raise that request. But let me challenge you on your examples, lets say Office365.Can you give me a real world example where you would ever create a Application Filter with a rule that would Allow Office365? Remember, an application Allow rule will do nothing to allow Office365 unless you also have an application rule that blocks Office365.Can you give me a real world example where you would ever create a Application Filter with a rule that would Block Office365? Is there a business need to ever do this?If you want to know about application detection (not enforcement), as far as I know any firewall rule that includes any Application Filter (such as Allow All) will mark and log all traffic that successfully gets through that firewall rule. If you find that there are applications that the XG supports in the list that are not being logged, it could be that they are not logged because they are allowed via a rule that does not contain an Application Filter, or it could be that they are not logged because the traffic is not matching any firewall rule and is blocked.
If you want to know about applications that the firewall is blocking, you need to use tcpdump. IPS / Application Filter cannot tell you because it was dropped before it got there.Remember - the Application Filter is used to Block applications that are making it through your firewall. It is not used to Allow applications. Therefore the application detection is only used on traffic that is allowed via a firewall rule, where that firewall rule has an application filter assigned.
Thank you for taking the time to jump in and explain a few things. This has helped clarify some of the expected behavior. That said, there is some very inconsistent (buggy) behavior in the way the Application Filter works within XG.
The three firewall rules in the above screenshot correspond to your scenarios 1,2, and 4 using LDAP as the test protocol. I skipped 3 for no good reason other than I missed it. Anyway, scenario 1 and 2 are as expected but scenario 4 failed miserably.
The application filter log above shows the denied result of scenario 2 on the bottom, which was expected. The FW rule allowed but filter denied LDAP protocol specifically. The top line of the log however is the result of scenario 4. The cloned FW rule was given an application filter which was base deny and explicit LDAP allow. In this instance, the LDAP traffic was not even identified by XG as being LDAP traffic and was blocked in contradiction to the application filter.
This misbehavior is consistent with what I've seen trying to test NTP traffic. When the application filter is set to deny NTP it identifies NTP on port 123 correctly and blocks the traffic. When set to allow NTP traffic, XG does not recognize NTP and blocks the traffic anyway.
I'm not trying to pick apart XG or be difficult... just think we've identified a bug in the application filter and trying to help understand it.
Thanks Michael....however allowed Apps are not moniterd. XG Application filters log only blocked Apps.
Imagin you have a firewall rule where the Application filter is deny all but allow few application. How can users determine if that App is allowed and which App is allowing it?
At least this happens at the moment with current version. Aditya Patel and sachingurung replied many times that XG logs only denied App and not allowed one.
Anyway, I appreciate yours replies here...even if it is not your topic!
In reply to Gary Parr:
It's not working for me, even with that pattern update.
I have an application filter with a default of "Allow", but with a rule that denies all applications, and NTP still gets through the firewall.
When I change that to an application filter with a default of "Deny", but with a rule that allows NTP, NTP is still blocked, and the log entry for port 123 does not identify the application as NTP.
James, can you double check your last test? When I run the different scenarios identified by Michael (and include your scenario as well) I get the correct result on all tests except where NTP is specifically allowed on a default-deny filter. This result is similar to my testing with the LDAP protocol.
In the above FW rules, each rule has an application filter as specified:
My testing results are as follows:
And, just like LDAP traffic was not identified as LDAP traffic in scenario 4 in my previous tests, NTP traffic was not identified as NTP traffic in the logs when I ran scenario 4 here as well. The issue I was experiencing previously, where NTP was not handled at all by the application filter, is something I have not been able to reproduce after receiving the signature update. Did that update fix part of the issue? Or was there a flaw in my previous test? I have no idea. But... I have repeatedly been able to replicate a bug where a default block filter with explicit allow (scenario 4) does not behave as expected.
I support this request and you can raise the request on Sophos Ideas for logging Allowed Applications in the LogViewer. Check #2 in my
Just for a bit more information here, Check #2 in my Troubleshooting Guide, IPS(Application Filter) in XG is accounted at the bottom of the list and any traffic that is allowed will be processed by all the other check before falling into Application Filter check. So considering that the blocks are logged in Log Viewer, if something is allowed and not logged then, it becomes certain that Application Filter is not blocking it and one must eventually take a step back and configure a Firewall Rule to block the services.
In reply to sachingurung:
but I will not open a feature request for standard feature. It is like opening a feature request for logging traffic on Firewall. Reports report which applications are used but live logs do not.
Still incredible how certain things work on this "Cyberoam OS like."
Great testing Gary. Just when I think, I know everything that is there to learn Michael Dunn throws a curve ball when explaining XG I am guessing you have run into a bug because the way the behavior is intended does make sense.
However, I generally don't use application rules for simple things like NTP where firewall rule gives me a lot more flexibility. Here I agree with Luk that application rules are more suitable when distinguishing for example between http downloads and http streaming for netflix specially when using QoS. You can't throttle http whereas its easy to throttle netflix using application filters.
Having said all that, this is another case where things are complicated and non intuitive because of the way firewall rules incorporate everything including appfilter/QoS/IPS. Long time ago people did some testing on UTM on behaviors of different firewall/DNAT/Proxy rules. Once you understand that behavior rules become easy to write. I suspect a similar test or KB from sophos is in order to explain some of the nuances that Michael explains in many posts throughout the forum. Because the way it is right now, there is really no guide to follow to see how http proxy affects DNS, application filters dos and don'ts and how the whole application control is dependent on intrusion protection.
In reply to Billybob:
here on some aspects it seems we are tilting at windmills. It does not make sense and this create confusion on Application filters. So Sophos should decide if use the application filters for all APPS or use them only for HTTP/S Applications. If they proceed with the first option, the application list is small and so they should allow us to add more applications using a custom application filter like custom IPS filter or another way to capture the packets and create the application. The other way makes more sense because the rest of the Application can be allowed/blocked by Firewall rules.
I'm in complete agreement that something simple like NTP could (and should) be handled by a simple FW rule. And yes, your point about app rules being well suited for differentiating different types of HTTP traffic is well taken. A counterpoint would be... Sohpos put those network protocols (and client/server and P2P) in the application list and by golly I expect them to work!
That said, it is a bit confusing. Is XG using DPI on application rules to determine the traffic type? If I allow SMTP in the app rules, will XG block SSH sent across port 25? Who knows. Perhaps we need to break "applications" into two sections... one for filtering of HTTP(S) and one for "other" with an explanation of just how deep the inspection of that "other" will be.
But... I do see the value here. Coming from a Shorewall/IP Tables setup, I like the idea of being able to create application filter bundles that are reusable and transportable. Imagine a "base" for servers that is a "deny all" with explicitly allowed services for network management. Then (assuming the v17 release has the add-able filters) you could add on a web server filter or an LDAP filter or a DB filter and, assuming your web servers and LDAP servers and DB servers are in host groups... you can have a single FW rule for each host group that holds the composite filter. Or something like that.
Granted, this could all just be the simple case of me completely misunderstanding next-gen firewalls.
I have worked with different firewall vendors over the years and like everything else, they come up with new terminology and fancy words. DPI (deep packet inspection) is the new word. Back about 10 years ago we called them UTM appliances. The only difference is that UTM9 does layer7 (kind of like DPI) using iptables whereas DPI firewalls lately use snort or similar. Snort was being used as intrusion detection system where it did (DPI) to see what was actually being carried in those packets and since it could distinguish between different applications, some firewall vendors started using it for application classification also. XG being one of those firewalls... hence the next gen DPI label
I suspect your telnet test to port 25 would be classified as SMTP traffic if you actually send the HELO message because at that point there is no difference between an SMTP server communication and your telnet session.
I am old fashioned and I trust the firewall rules that I create. If I need to open 10 ports to make an application work, I like to know what I did instead of the firewall using application control doing it for me. Almost reminds me of upnp... you have no idea what is going on. That is why I like the approach of firewall rule and then augment it with application filter if the application uses the same firewall port otherwise such traffic should be denied. But different people have different needs and Sophos knows a lot more about industry trends and needs of their clientele so we will get what they give us. This however doesn't mean that the product shouldn't work as advertised and you are absolutely correct that if it is there, you should expect it to work.
Well stated! For non-pet-project stuff, things I want to just plain work, I share a very similar philosophy.
But this? XG is my current "shiny new toy" and to that end I'm going to push it until it breaks and learn what I can along the way.