BUG - SSL/TLS Inspection seems to break Honeywell smart devices

I've noticed this behavior with both EAP 2 and now EAP 3. When SSL/TLS inspection is enabled Honeywell smart thermostats cannot connect to the internet. The only way they can connect is buy going the SSL/TLS inspection rules, SSL/TLS Inspection settings, advanced settings, and completely disabling the inspection (disable only when troubleshooting).

None of the default SSL/TLS inspection rules are configured to decrypt data.

Logs noticed in the webfilter when the issue is occurring is below.

2019-12-18 17:51:19Web filtermessageid="16002" log_type="Content Filtering" log_component="HTTP" log_subtype="Denied" status="" fw_rule_id="10" user="" user_group="" web_policy_id="13" web_policy="" category="" category_type="Acceptable" url="" content_type="" override_token="" response_code="" src_ip="x.x.x.x-internal IP" dst_ip="199.62.84.152" protocol="TCP" src_port="57393" dst_port="443" bytes_sent="0" bytes_received="0" domain="" exception="" activity_name="" reason="HTTP parsing error encountered." user_agent="" status_code="403" transaction_id="" referer="" download_file_name="" download_file_type="" upload_file_name="" upload_file_type="" con_id="0" app_name="" app_is_cloud="0" override_name="" override_authorizer="" used_quota="0"

  • In reply to MichaelBolton:

    Thanks.  So even though they are controlled by different variables (snort is "more correct" than the proxy) the results are the same.

    There is something against spec in the traffic from the IoT devices which both proxy and DPI do not like.  Since this is not a regression issue, it is currently low priority for investigation - especially as the investigation will most likely just confirm that they are non-spec in a way we don't want to allow.  There is a workaround - setting the global flag.

     

    The issue of traffic going to DPI when it should be going to proxy has been reproduced and sent to the firewall team, who is responsible for that decision.

  • In reply to Michael Dunn:

    Thanks for the info and the help. Let me know if you guys want packet captures or any other info to help when it comes time to look into it further.

  • 1) There is a known issue with "Use proxy instead of DPI mode" not being respected when Web Filter=None. This is being worked on for GA.

    2) There is a known issue with handling and logging of unscannable traffic. This is also being looked at, post-GA

    The traffic is on port 443 and matches a Firewall Rule for DPI mode and matches an TLS Rule Do Not decrypt rule.
    The sslx system does not recognize it as an TLS handshake, or there is something invalid about it.
    The sslx system, not being able to handle it, passes it to the HTTP parser hoping it can.
    The HTTP parser does not recognize it as plaintext HTTP and logs an error "HTTP parsing error encountered"

    Therefore it might appear to an administrator that the "Do not decrypt" was ignored, that it was decrypted and had trouble with the data after. In fact, what occurs is that neither the SSL system nor the HTTP system could parse the packets.

    Therefore some/most "HTTP parsing error encountered" in the log is actually a failure in handling the SSL handshake.

    In v18.0 EAP3 and most likely in v18 GA if you are in this scenario:
    proxy mode - In Web > General Settings > turn Block Unrecognized SSL Protocols off.
    DPI mode - In the command line (not ssh) use 'set http relay_invalid_http traffic on'

    This may change post-GA.

  • In reply to Michael Dunn:

    1) Has not changed for GA. Please use "Allow All".
    2) We have done several fixes for related defects, but we do not think that this particular problem is resolved. However we would like someone who was previously experiencing this problem to report whether it is working in GA.

    Specifically for the Honeywell smart thermostat, as far as I know:
    In 17.5 (proxy) it fails with "Block unrecognized protocols" Checked
    In 17.5 (proxy) it works with "Block unrecognized protocols" unchecked
    In 18.0 GA (proxy) it fails with "Block unrecognized protocols" Checked
    In 18.0 GA (proxy) it works with "Block unrecognized protocols" unchecked
    In 18.0 GA (DPI) it fails with "relay_invalid_http_traffic" off
    In 18.0 GA (DPI) it works with "relay_invalid_http_traffic" on

    Please confirm if this is true.

    Block unrecognized protocols can be changed in Web > General Settings.
    relay_invalid_http_traffic can be changed in command line (no ssh/advanced shell) with 'set http_proxy relay_invalid_http_traffic on/off'.

  • In reply to Michael Dunn:

    Hi Michael,

    so what you are advising is there is no way to stop a firewall rule from being scanned eg LAN to LAN rules?

    Ian

  • In reply to rfcat_vk:

    rfcat_vk

    so what you are advising is there is no way to stop a firewall rule from being scanned eg LAN to LAN rules?

    Ian, I love you man and I like all the bug reports you are giving us.  I'm a QA tester - I love testing, I love bugs.  But please do not hijack threads.

    This thread is about Honeywell smart devices in a LAN contacting to a server on the Internet.

     

  • In reply to Michael Dunn:

    Hi Michael,

    I could not find my original thread on the subject and this one has lots of explanations about the same issue, so I added a question because I have a device which has similar issues with DPI?

    Ian

  • In reply to Michael Dunn:

    Hi Michael,

    that answer was about the display when you click on the field on the firewall rule not about turning off DPI.

    Ian

  • In reply to rfcat_vk:

    rfcat_vk

    So how do I turn off inspection of internal rules only?

    RichBaldry

    Hi Ian,

    Our intent is for it not to be necessary to turn this off, so that the DPI Engine has no impact on traffic if there are no scanning or inspection policies applicable to it, but so that we can still include information about it in our overall accounting of traffic.

    There are still a couple of outstanding situations that we've come across in the EAP that we are aiming to fix before the GA release of v18, including a few relating to our handling of traffic that is not recognized as TLS or HTTP on port 443. One of the major impacts of this right now is OpenVPN SSL VPN connections, which send a few packets of custom handshake protocol before beginning a TLS handshake. We've had reports of a few IoT devices that use OpenVPN to tunnel traffic to the cloud - is it possible that's what's going on with your camera?

    So to be clear on your question "so what you are advising is there is no way to stop a firewall rule from being scanned eg LAN to LAN rules" the answer from the program manager is "the intent is for it not to be necessary to turn this off".

  • In reply to Michael Dunn:

    Therein lies the issue, you cannot use the http proxy on internal traffic (which does not break the application). but DPI does, so how do you overcome this because there is no log to show what needs to be exempted.

    Ian

  • In reply to rfcat_vk:

    I will test out the original issue and get back to you Mike.

     

    As far as what Ian said, I know it is off topic but it is also part of this issue overall. If I could have disabled any scanning whatsoever on a firewall rule, I could have bypassed this issue all together.

    If what Ian is saying is true and the DPI is scanning even LAN to LAN traffic, how in the world can XG meet the performance numbers that are posted for the firewall itself? Snort can't put up those numbers with any of the hardware appliances. 20 GBPS throughout on an XG230? Yeah right. If DPI wasn't involved maybe. Sorry I continued off topic but it is related.

  • In reply to MichaelBolton:

    Off topic, but currently in v18 GA if you create a Rule without any Security profile on it, It will bypass Snort. The issue of DPI scanning LAN to LAN Traffic in my believe has only present on v18 EAP.

    I've just tested it out with HTTP(s)-Download/SFTP/SMB and all of them bypassed Snort/AV/DPI using a Rule without any Security profile being applied.

    So it's pretty much working as expected right now in v18 GA.

     

    Thanks!

  • In reply to Prism:

    I have tested it on v18GA. All of my firewall rules, including LAN to LAN have all security, scanning, IPS, etc. disabled and the web policy set to non. Under "SSL/T LS Inspection Rules" you can see the traffic counters increasing, proving that the DPI is still being used.

  • In reply to MichaelBolton:

    MichaelBolton
    All of my firewall rules, including LAN to LAN have all security, scanning, IPS, etc. disabled and the web policy set to non. Under "SSL/T LS Inspection Rules" you can see the traffic counters increasing, proving that the DPI is still being used.

     

    The new DPI engine is always running, by default it have a hidden rule ANY=>ANY, Do not decrypt and Maximum compatibility.

    I currently have a SSL/TLS Inspection Rule LAN => ANY, Do not decrypt, Maximum compatibility (For the Logs). And above that rules, the correct ones for LAN=>WAN traffic.

    If there's no rules implicating Decryption over LAN=>LAN Traffic,  then SSL/TLS Inspection won't kick in and do It's thing. But of course, you can still apply Decryption Profiles over it to block certain Chipers or un-trusted certificates, so on.

     

    There is a new video from Sophos showing fast-path, if you want the traffic not being scanned at all, you will need to disable all security profiles over that traffic, that includes AV and IPS + AppID.

    Disabling only the Web Policy isn't right thing if you want the best throughput XG can give to you

    Also apparently disabling the Web Policies =/= disabling SSL/TLS Inspection Rules. You can have a rule/user without any web policies and still have a decryption rule over SSL/TLS Inspection for a category/Websites. Web Policies are in my understatement, being used only to apply actual policies over HTTP(s), such as block/allow/quota/warn for the categories/etc you have.

     

    Thanks!