Bandwidth meter for SD-WAN routes is unreliable.

Hey Prism,

SDWAN-route and Firewall stats reported are not comparable apple-to-apple. Routing is per-packet operation and only captures those packets stats which passes thru it. Firewall rule works in stateful manner. All request and reply (client to server and server to client) fall under same firewall rule. 

Having said that, it is quite possible for these two meters to report different reading. For example, if asymmetric routing is configured. or SDWAN routing is configured only request direction, SDWAN-route will account only for the upload stats as downloads might be following a different SDWAN route (or other routes).

Let us know if this is not the case in your observation.

HTH

Moheed

Same issue with VLANs.

ian

The problem could also be Service vs Application Detection.

OneDrive for example is a App, which you cannot decrypt. Therefore it can be hard to route this app properly. SFOS is trying to find and match the apps and hosts properly but this can take "time" in the process. Therefore the 12 GB data could be a unmarked app in the process, mean the firewall did not pick up onedrive at this point to match it to the proper route. Application based routing has the same problem like decryption based on app, which Sophos tried to do in V18.0 and canceled. You have to match the rule by packet one, which means you have to figure out, which app this is by the near SYN packet. You cannot simply "reroute the app" later if you are sure, it is onedrive.  

That's why I created a secondary route with FQDN's for OneDrive, even then the primary route with App detection is still reliable. (Works really well.)

What I noticed later is what Moheed Ahmad  said, I've had asymmetric routing enabled and the meter only accounts for upload data. (Which is the request direction.)

I has wrong on how the meter works, I thought it would work just as a Firewall Policy, which accounts for both download/upload data. (Maybe I'm still wrong.)

Hi Prism,

If your download traffic too is using sd-wan route, you can cross-check by observing the increase in traffic stats against that rule. 

HTH

Moheed

That's something I still didn't understood with the meter for SD-WAN, hopefully you can teach me why and where I'm wrong.

On a Firewall policy, the meter accounts for both download/upload (stateful manner), but with SD-WAN the accounting is different. (As stated by you.)

Currently I only have LAN => WAN SD-WAN routes, using the image of my post as a example; If I download some data from OneDrive, which passes through either Route #5 or #3, the meter will only account the uploaded data from the client, the downloaded data which went to the same route isn't accounted to the meter.

That's what I don't understand, looking at the Logs I can see both download/upload are going through the same SD-WAN route, so why the download data isn't available?

Even then, there's a mismatch with what the meter shows, and how much data went through a certain route. I've uploaded 2 GB of data to OneDrive and while checking the Logs, I saw all data went through Route #3, but even then the bandwidth that the meter showed didn't match the amount of traffic that went through. (Not even close.)

On the example above there's no need to use FQDN's or App objects, even with a "Any" destination you can replicate It.

"Current precedence for routing: SD-WAN route, Static route, VPN route." & "Route only through specified gateways" is enabled. 

Also, thank you for answering my questions! (And sorry if my questions are... "bad".)

I haven't used SDWAN except to monitor my single gateway's quality, so take it with a grain of salt... But is your "download" (i.e. incoming, right) data actually touched by SDWAN at all? Or does SDWAN only enter the picture when a routing decision has to be made and that's really only done for "upload" (outgoing) packets which need to figure out which gateway to use?

Maybe incoming data uses whatever port/gateway it uses and SDWAN never sees it? (i.e. it's SD-WAN, not SD-LAN.) But I think you've said you do see some incoming traffic in SDWAN?

This is hard to know, since the Firewall logs doesn't show "enough" information.

But if you monitor the uplinks you can see the SD-WAN routing is working as expected for both download and upload traffic.

Wayne Folta said:

Maybe incoming data uses whatever port/gateway it uses and SDWAN never sees it?

 I don't think that's possible, how would the server or a web server know the IPv4 address of the other interface if the connection has already been established?

This could be different with VPN or IPsec traffic, but I'm using SD-WAN solely for Internet access.

Anyways, from my perspective this is only a interface issue, It doesn't affect the routing itself which is working well. (Even with application routing.)

I don't think an "established" connection is based on firewall ports, rather it's my impression that it's based on endpoints and things known by the endpoints (IP ports, sequence numbers, etc) isn't it? If that's right -- and I could be totally wrong -- then return traffic could use whatever port it happens to come in. (Based on decisions of upstream routers over which your local router has no control... unless you're using BGP or something that I don't understand.)

So my question boils down to how SD-WAN actually works. Is it "active" or "passive"? What I mean by "active" is: is there a separate SD-WAN subsystem that's involved on an ongoing basis with decisions about where traffic goes -- in particular, which gateway it will egress. Or is it "passive": something that's more configurational/setup that handles non-traditional routing specification, and once things are set up there is no process you could point to and say, "Yep, your using SD-WAN and it's up, since I can see this process running (or this function being called in the routing process, or whatever)."

If it's the former, I'm trying to say that there's no SD-LAN, so return traffic would not necessarily be routed through/via the SD-WAN subsystem, only outgoing traffic. So if stats were kept by this subsystem it might not see return traffic at all, since it is simply not involved in routing it. (Obviously, some subsystem in the router is involved, just saying that if there's a separate SD-WAN subsystem it may not be involved.)

If it's the latter, then I imagine SD-WAN connections need to be tagged. And there's a whole set of databases that tag traffic in many ways: established, security calls by IPS, inspection status, traffic shaping, etc. I forget the commands, but had to dip my toes in it when I was investigating if I'd set up Traffic Shaping correctly and if it was actually working.) In which case it might be possible that the initial outgoing SD-WAN is tagged, but return is not. Maybe an oversight (bug) by Sophos that's simple to fix. Or maybe it's more complicated than that and they're still working on it. So the overall routing system knows about the traffic, of course, but it's not tagged as SD-WAN so reporting misses it.

I'm really just speculating on the "who" that would be the subsystem that actually has access to and tracks the amounts of traffic. Depending on "what" that "who" is involved in, it may or may not be cognizant of all traffic in both directions.

Prism said:

On the example above there's no need to use FQDN's or App objects, even with a "Any" destination you can replicate It.

Prism said:

If I download some data from OneDrive, which passes through either Route #5 or #3, the meter will only account the uploaded data from the client, the downloaded data which went to the same route isn't accounted to the meter.

That is correct. Only uploaded data from client will show up, as there is no reply direction SDWAN route configured and SFOS accounts SDWAN-route data direction wise.

Prism said:

I've uploaded 2 GB of data to OneDrive and while checking the Logs, I saw all data went through Route #3, but even then the bandwidth that the meter showed didn't match the amount of traffic that went through. (Not even close.)

First connection from an app is routed using default wan link load balance. The application-based SD-WAN route applies to subsequent connections after Sophos Firewall learns the session details.

It seems in your case, appcache hasn't learned (or it had aged from appcache) about ip-addresses of onedrive's storage lakes. That's why missing accounting. Had it been subsequent uploads they should get accounted.

There is a nice documentation over here.

https://docs.sophos.com/nsg/sophos-firewall/18.5/Help/en-us/webhelp/onlinehelp/AdministratorHelp/Routing/SDWANPolicyRouting/RoutingSDWANPolicyRoutingUserApplication/

Hope this clarifies your query.

HTH

Moheed