BUG - SSL/TLS Inspection seems to break Honeywell smart devices

Hi,

you should be able to add an exception on the web page. If it breaks why have it go through a SSL/TLS enabled rule and cause yourself unnecessary device management grief?

Some of my IoT devices are showing up a secure connection error, then when tried a second time connect. This has only started in the last 30 minutes or so, the EAP 3 was installed about 4 hours ago.

Ian

Thats my confusion though. I only have the two default SSL/TLS inspection rules and they're both configured to "Do Not Encrypt". Why would I have to create an exception for something that shouldn't be decrypting anyway?

The error log I posted earlier is in the Web Filter log and for HTTP traffic, yet that event only occurs when The advanced settings of SS/TLS inspection settings is set to enabled even though the SSL/TLS rules are all configured as "Do Not Encrypt". When I completely disable the SSL/TLS inspection settings Advanced setting to "Disabled" everything works normally.

reason="HTTP parsing error encountered." 

I don't recall seeing that error before.

It is possible to do a packet dump?

Another possibility, can you go into console (this is the special command line, not ssh) and do

set http_proxy relay_invalid_http_traffic on

Let me know if that makes a difference.  If it does not, please turn it off again.

Note: I am off until Jan 2.

Hi Michael,

have a Merry Christmas and a Happy New Year and thank you for your  ongoing forum support.

Ian

Thanks for the response. Setting "set http_proxy relay_invalid_http_traffic on" seems to have resolved this specific issue for me.

Regarding the packet capture, I can provide you with anything you'd like. Do you have instructions or steps on how to gather and provide any needed captures or logs?

I am most interested if there is a difference between proxy mode and DPI mode.

Can you please try the following:

Switch from DPI mode to proxy mode and confirm it is still working.

Now turn off the invalid traffic setting:

set http_proxy relay_invalid_http_traffic off

Does it work in proxy mode?

If it still works, go to Web > General Settings > Block unrecognized SSL Protocols and select it (eg turn on blocking).

Does it work in proxy mode?

Michael Dunn I have the same issue with Honeywell smart devices and alarm systems. Following the steps you provide, I can confirm the traffic passes through the proxy with no issues as long as you have a web policy enabled. If you do not, the DPI engine is still processing the traffic even when the use proxy box is checked. Block unrecognized SSL Protocols enabled, showed no change.

Excluding the traffic from the DPI engine does not help either. Only enabling "set http_proxy relay_invalid_http_traffic" will allow the traffic through the DPI.

Thanks.  This is something we will want to investigate.

For the record this is how it works:

If you have either web policy != None or malware scan enabled then the traffic must be scanned.  The "use web proxy" checkbox defines whether the scanning is done by the proxy or DPI.  Changing the web policy does not affect which process is responsible for checking the traffic (and which process is responsible for decrypting).

That being said, if neither web policy or malware scan is enabled then the "use web proxy" checkbox become unchecked and disabled.  The traffic will go through DPI engine and may or may not be decrypted according to the SSL/TLS inspection rules.

There is no case where the "use web proxy" checkbox is checked but the DPI engine processes the traffic.

Once the data is decrypted you now have HTTP messages that must follow the HTTP specification.  Neither the DPI engine or the the proxy can do enforcement if the traffic does not conform to the HTTP spec.  By default if they get traffic that does not follow the HTTP specification they drop/block the connection.  Turning relay_invalid_http_traffic flag on will allow the non-spec traffic to pass through without being scanned.  The proxy has a slight difference in that some types of HTTPS invalid traffic follows the block unrecognized SSL Protocols instead of the relay_invalid_http_traffic flag, depending on exactly how it is invalid/unrecognized.

Most likely the smart devices, because the one company controls both the client and server, have implemented something slightly out of spec.  That is fine, that is why we have the flag to allow it.  But AFAIK both the proxy and DPI should be enforcing the same HTTP spec so either both should allow or both should block.  If one blocks and the other allows given the same flag settings, it is something I'd like to know more about.

We can start with a packet dump but because this is encrypted SSL traffic I don't know if it will be useful.

Thanks.  This is something we will want to investigate.

For the record this is how it works:

If you have either web policy != None or malware scan enabled then the traffic must be scanned.  The "use web proxy" checkbox defines whether the scanning is done by the proxy or DPI.  Changing the web policy does not affect which process is responsible for checking the traffic (and which process is responsible for decrypting).

That being said, if neither web policy or malware scan is enabled then the "use web proxy" checkbox become unchecked and disabled.  The traffic will go through DPI engine and may or may not be decrypted according to the SSL/TLS inspection rules.

There is no case where the "use web proxy" checkbox is checked but the DPI engine processes the traffic.

Once the data is decrypted you now have HTTP messages that must follow the HTTP specification.  Neither the DPI engine or the the proxy can do enforcement if the traffic does not conform to the HTTP spec.  By default if they get traffic that does not follow the HTTP specification they drop/block the connection.  Turning relay_invalid_http_traffic flag on will allow the non-spec traffic to pass through without being scanned.  The proxy has a slight difference in that some types of HTTPS invalid traffic follows the block unrecognized SSL Protocols instead of the relay_invalid_http_traffic flag, depending on exactly how it is invalid/unrecognized.

Most likely the smart devices, because the one company controls both the client and server, have implemented something slightly out of spec.  That is fine, that is why we have the flag to allow it.  But AFAIK both the proxy and DPI should be enforcing the same HTTP spec so either both should allow or both should block.  If one blocks and the other allows given the same flag settings, it is something I'd like to know more about.

We can start with a packet dump but because this is encrypted SSL traffic I don't know if it will be useful.

I can tell you on my unit, if I don't have a web policy selected ie none, have maleware scanning enabled, and have "Use web proxy instead of DPI engine" checked, the XG is using the DPI engine. Only when I select a web policy will it actually use the proxy.

Also checking "Decrypt HTTPS during web proxy filtering", the XG will use the proxy with the web policy set to "none". As previously stated though, if the web policy is set to none, malware scanning is enabled, use proxy is checked, and Decrypt HTTPS during web proxy filtering is unchecked, all SSL traffic still goes through the DPI engine. Definitely an issue.

MichaelBolton said:

I can tell you on my unit, if I don't have a web policy selected ie none, have maleware scanning enabled, and have "Use web proxy instead of DPI engine" checked, the XG is using the DPI engine. Only when I select a web policy will it actually use the proxy.

I assure you this is not the case.  What are you using to determine whether it is using the proxy (eg awarrenhttp) or DPI (eg snort).

HTTPS traffic stills show up in the log under "SSL/TLS inspection". The second I apply a web policy, the HTTPS traffic stops showing up in "SSL/TLS inspection" in the log.

Also, my work around for Honeywell traffic before your console setting, was to use the proxy but I had to set a policy then too. I used allow all and the traffic worked again. I will be happy to give you a remote session but I can assure you, the DPI engine is messing with the traffic somehow when no web policy is selected or "Decrypt HTTPS during web proxy filtering" is not selected.

I need to look at your configuration.  Can you please go to Diagnostics, Support Access and enable.  Post or PM me the Access ID.

Michael Dunn said:

 

I assure you this is not the case.  What are you using to determine whether it is using the proxy (eg awarrenhttp) or DPI (eg snort).

It looks like I am wrong.  Or rather, I am describing what should be happening and not what really is happening.  Looks like we have a bug in the proxy/dpi selection when web filter is none.

In Michael Bolton's case, with web filter = none port 443 is processed by snort and port 80 is processed by awarrenhttp.

Bypassing that issue by setting a web filter, can you please clarify the results here:

Results of further testing.

Web Filter=Allow All       Use Proxy            relay_invalid_http_traffic off      Block unrecognized protocols on               Error

Web Filter=Allow All       Use Proxy            relay_invalid_http_traffic on       Block unrecognized protocols on               Error

Web Filter=Allow All       Use Proxy            relay_invalid_http_traffic off      Block unrecognized protocols off                Allowed

Web Filter=Allow All       Use Proxy            relay_invalid_http_traffic on       Block unrecognized protocols off                Allowed

Thanks.  So even though they are controlled by different variables (snort is "more correct" than the proxy) the results are the same.

There is something against spec in the traffic from the IoT devices which both proxy and DPI do not like.  Since this is not a regression issue, it is currently low priority for investigation - especially as the investigation will most likely just confirm that they are non-spec in a way we don't want to allow.  There is a workaround - setting the global flag.

The issue of traffic going to DPI when it should be going to proxy has been reproduced and sent to the firewall team, who is responsible for that decision.

Thanks for the info and the help. Let me know if you guys want packet captures or any other info to help when it comes time to look into it further.