BUG - SSL/TLS Inspection seems to break Honeywell smart devices

I've noticed this behavior with both EAP 2 and now EAP 3. When SSL/TLS inspection is enabled Honeywell smart thermostats cannot connect to the internet. The only way they can connect is buy going the SSL/TLS inspection rules, SSL/TLS Inspection settings, advanced settings, and completely disabling the inspection (disable only when troubleshooting).

None of the default SSL/TLS inspection rules are configured to decrypt data.

Logs noticed in the webfilter when the issue is occurring is below.

2019-12-18 17:51:19Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="10" user="" user_group="" web_policy_id="13" web_policy="" category="" category_type="Acceptable" url="" content_type="" override_token="" response_code="" src_ip="x.x.x.x-internal IP" dst_ip="199.62.84.152" protocol="TCP" src_port="57393" dst_port="443" bytes_sent="0" bytes_received="0" domain="" exception="" activity_name="" reason="HTTP parsing error encountered." user_agent="" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="0" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

  • Hi,

    you should be able to add an exception on the web page. If it breaks why have it go through a SSL/TLS enabled rule and cause yourself unnecessary device management grief?

    Some of my IoT devices are showing up a secure connection error, then when tried a second time connect. This has only started in the last 30 minutes or so, the EAP 3 was installed about 4 hours ago.

    Ian

  • In reply to rfcat_vk:

    Thats my confusion though. I only have the two default SSL/TLS inspection rules and they're both configured to "Do Not Encrypt". Why would I have to create an exception for something that shouldn't be decrypting anyway?

    The error log I posted earlier is in the Web Filter log and for HTTP traffic, yet that event only occurs when The advanced settings of SS/TLS inspection settings is set to enabled even though the SSL/TLS rules are all configured as "Do Not Encrypt". When I completely disable the SSL/TLS inspection settings Advanced setting to "Disabled" everything works normally.

  • In reply to myparadigm:

    reason="HTTP parsing error encountered." 

    I don't recall seeing that error before.

    It is possible to do a packet dump?

     

    Another possibility, can you go into console (this is the special command line, not ssh) and do

    set http_proxy relay_invalid_http_traffic on

    Let me know if that makes a difference.  If it does not, please turn it off again.

     

    Note: I am off until Jan 2.

  • In reply to Michael Dunn:

    Hi Michael,

    have a Merry Christmas and a Happy New Year and thank you for your  ongoing forum support.

    Ian

  • In reply to Michael Dunn:

    Thanks for the response. Setting "set http_proxy relay_invalid_http_traffic on" seems to have resolved this specific issue for me.

     

    Regarding the packet capture, I can provide you with anything you'd like. Do you have instructions or steps on how to gather and provide any needed captures or logs?

  • In reply to myparadigm:

    I am most interested if there is a difference between proxy mode and DPI mode.

     

     

    Can you please try the following:

    Switch from DPI mode to proxy mode and confirm it is still working.

     

    Now turn off the invalid traffic setting:

    set http_proxy relay_invalid_http_traffic off

    Does it work in proxy mode?

     

    If it still works, go to Web > General Settings > Block unrecognized SSL Protocols and select it (eg turn on blocking).

    Does it work in proxy mode?

  • In reply to Michael Dunn:

     I have the same issue with Honeywell smart devices and alarm systems. Following the steps you provide, I can confirm the traffic passes through the proxy with no issues as long as you have a web policy enabled. If you do not, the DPI engine is still processing the traffic even when the use proxy box is checked. Block unrecognized SSL Protocols enabled, showed no change.

    Excluding the traffic from the DPI engine does not help either. Only enabling "set http_proxy relay_invalid_http_traffic" will allow the traffic through the DPI.

  • In reply to MichaelBolton:

    Thanks.  This is something we will want to investigate.

     

    For the record this is how it works:

    If you have either web policy != None or malware scan enabled then the traffic must be scanned.  The "use web proxy" checkbox defines whether the scanning is done by the proxy or DPI.  Changing the web policy does not affect which process is responsible for checking the traffic (and which process is responsible for decrypting).

    That being said, if neither web policy or malware scan is enabled then the "use web proxy" checkbox become unchecked and disabled.  The traffic will go through DPI engine and may or may not be decrypted according to the SSL/TLS inspection rules.

    There is no case where the "use web proxy" checkbox is checked but the DPI engine processes the traffic.

     

    Once the data is decrypted you now have HTTP messages that must follow the HTTP specification.  Neither the DPI engine or the the proxy can do enforcement if the traffic does not conform to the HTTP spec.  By default if they get traffic that does not follow the HTTP specification they drop/block the connection.  Turning relay_invalid_http_traffic flag on will allow the non-spec traffic to pass through without being scanned.  The proxy has a slight difference in that some types of HTTPS invalid traffic follows the block unrecognized SSL Protocols instead of the relay_invalid_http_traffic flag, depending on exactly how it is invalid/unrecognized.

    Most likely the smart devices, because the one company controls both the client and server, have implemented something slightly out of spec.  That is fine, that is why we have the flag to allow it.  But AFAIK both the proxy and DPI should be enforcing the same HTTP spec so either both should allow or both should block.  If one blocks and the other allows given the same flag settings, it is something I'd like to know more about.

    We can start with a packet dump but because this is encrypted SSL traffic I don't know if it will be useful.

  • In reply to Michael Dunn:

    I can tell you on my unit, if I don't have a web policy selected ie none, have maleware scanning enabled, and have "Use web proxy instead of DPI engine" checked, the XG is using the DPI engine. Only when I select a web policy will it actually use the proxy.

  • In reply to MichaelBolton:

    Also checking "Decrypt HTTPS during web proxy filtering", the XG will use the proxy with the web policy set to "none". As previously stated though, if the web policy is set to none, malware scanning is enabled, use proxy is checked, and Decrypt HTTPS during web proxy filtering is unchecked, all SSL traffic still goes through the DPI engine. Definitely an issue.

  • In reply to MichaelBolton:

    MichaelBolton

    I can tell you on my unit, if I don't have a web policy selected ie none, have maleware scanning enabled, and have "Use web proxy instead of DPI engine" checked, the XG is using the DPI engine. Only when I select a web policy will it actually use the proxy.

    I assure you this is not the case.  What are you using to determine whether it is using the proxy (eg awarrenhttp) or DPI (eg snort).

     

  • In reply to Michael Dunn:

    HTTPS traffic stills show up in the log under "SSL/TLS inspection". The second I apply a web policy, the HTTPS traffic stops showing up in "SSL/TLS inspection" in the log.

     

    Also, my work around for Honeywell traffic before your console setting, was to use the proxy but I had to set a policy then too. I used allow all and the traffic worked again. I will be happy to give you a remote session but I can assure you, the DPI engine is messing with the traffic somehow when no web policy is selected or "Decrypt HTTPS during web proxy filtering" is not selected.

  • In reply to MichaelBolton:

    I need to look at your configuration.  Can you please go to Diagnostics, Support Access and enable.  Post or PM me the Access ID.

  • In reply to Michael Dunn:

    Michael Dunn

     

    I assure you this is not the case.  What are you using to determine whether it is using the proxy (eg awarrenhttp) or DPI (eg snort).

    It looks like I am wrong.  Or rather, I am describing what should be happening and not what really is happening.  Looks like we have a bug in the proxy/dpi selection when web filter is none.

    In Michael Bolton's case, with web filter = none port 443 is processed by snort and port 80 is processed by awarrenhttp.

     

    Bypassing that issue by setting a web filter, can you please clarify the results here:

     

    Web Filter=Allow All Use DPI relay_invalid_http_traffic off Block unrecognized protocols on/off Error
    Web Filter=Allow All Use DPI relay_invalid_http_traffic on Block unrecognized protocols on/off Allowed
             
    Web Filter=Allow All Use Proxy relay_invalid_http_traffic off Block unrecognized protocols on ?
    Web Filter=Allow All Use Proxy relay_invalid_http_traffic on Block unrecognized protocols on ?
    Web Filter=Allow All Use Proxy relay_invalid_http_traffic off Block unrecognized protocols off ?
    Web Filter=Allow All Use Proxy relay_invalid_http_traffic on Block unrecognized protocols off ?
  • In reply to Michael Dunn:

    Results of further testing.

     

    Web Filter=Allow All       Use Proxy            relay_invalid_http_traffic off      Block unrecognized protocols on               Error

    Web Filter=Allow All       Use Proxy            relay_invalid_http_traffic on       Block unrecognized protocols on               Error

    Web Filter=Allow All       Use Proxy            relay_invalid_http_traffic off      Block unrecognized protocols off                Allowed

    Web Filter=Allow All       Use Proxy            relay_invalid_http_traffic on       Block unrecognized protocols off                Allowed