Reverse proxy (WAF) UniFI - odd (and irreversible) behaviour after enabling websocket pass-through

Hi Folks and a very happy new year to you all! :)

Firstly, please note that I am merely a home user (so not paying my way, here) and that that this is a very low priority question! It's just puzzling the heck out of me and I wondered if anyone had any ideas on why it doesn't work (in other words, I'm mostly just asking this to help enhance my understanding)?

I've been using UTM's WAF to reverse proxy an Icecast stream (so both an http page and an audio stream*) and I use another virtual server instance to reverse proxy an Apache page (also running on the same Raspberry Pi) and using encrypt and redirect (using a LE cert) and of course, it all works flawlessly.

Mostly just as an experiment (though it could be useful to have it enable-able) I also tried setting up a reverse proxy instance of my UniFi server (which is running on a different Raspberry Pi) and with UniFi already having an https interface, I configured the virtual and real Webservers as is shown below:

  

From memory, I initially had the re-write html and re-write cookies ticked and using  https://ddns:9666 showed the UniFi log-in page, so initially it all looked to be working, but the page also had a couple of websocket error message pop-ups, so having spotted the new feature in the site path routing settings, I elected to enable websocket pass-through to see if that would remove the pop-up error messages.

This is where it almost gets quite interesting as after enabling that websocket pass-through feature, the browser now showed just a plain white page, then after disabling the websocket pass-through feature, the 'fault' remained (plain white page). Of course, I have tried several browsers (with cleared caches) and things like toggling the re-write html and re-write cookies options, but no matter what I have tried, all I can now see is a blank page.

Assuming it was something that had been corrupted on the Pi, I removed then re-installed UniFi, and access by using https://ddns:port showed the initial UniFi set-up page (so I thought I'd fixed things). I then set up the UniFI basics (password, language and region) and when it got to the stage where the log-in screen should appear, back appears my blank screen.

The Pi is on its own VLAN, but I have a firewall route to it, so I can access it LAN side (https://192.168.1.7:port) and after accepting the self-signed cert, the login page shows perfectly in the browser, so my next step was to view the source code from both the https//192.168.1.7:port (log in screen showing as expected) and https://ddns:port (plain white screen) and though I've yet to go through it all (using diff on large html files - which are full of containers - is not an easy task), I have already noticed that whilst the first section on each is identical (other than the <!DOCTYPE html> being on a separate line) and is shown below, I noticed that there are a few subtle differences in the lines following that first section.

First section:

<!DOCTYPE html><html lang="en" ng-controller="ManageController as manageCtrl" ng-strict-di><head><meta charset="utf-8"><title ng-bind="manageCtrl.system.getName() || manageCtrl.system.getHostname() || 'UniFi Network'">UniFi Network</title><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no" unifi-prevent-focus-zoom><meta name="apple-itunes-app" content="app-id=1057750338"><base href="/manage/"><link rel="apple-touch-icon-precomposed" href="images/favicons/favicon-152.png?v=2"><meta name="msapplication-TileColor" content="#0193d7"><meta name="msapplication-TileImage" content="images/favicons/favicon-144.png?v=2"><link rel="apple-touch-icon-precomposed" sizes="152x152" href="images/favicons/favicon-152.png?v=2"><link rel="apple-touch-icon-precomposed" sizes="144x144" href="images/favicons/favicon-144.png?v=2"><link rel="apple-touch-icon-precomposed" sizes="120x120" href="images/favicons/favicon-120.png?v=2"><link rel="apple-touch-icon-precomposed" sizes="72x72" href="images/favicons/favicon-72.png?v=2"><link rel="apple-touch-icon-precomposed" href="images/favicons/favicon-57.png?v=2"><link rel="icon" href="images/favicons/favicon-32.png?v=2" sizes="32x32"><script src="config/config.v5.12.35.0.js"></script><script type="text/javascript">window.unifiConfig.version = '5.12.35.0';

When using 192.168 access, the above is followed by:

</script><script src="js/initial.v5.12.35.0.js"></script><script src="js/components.v5.12.35.0.js" defer></script><script src="js/base.v5.12.35.0.js" defer></script><script src="js/main.manage.v5.12.35.0.js" defer></script><style>

When using the reverse proxy to access it, it is instead followed by the below:

</script><script src="js/initial.v5.12.35.0.js"></script><script src="js/components.v5.12.35.0.js" defer="defer"></script><script src="js/base.v5.12.35.0.js" defer="defer"></script><script src="js/main.manage.v5.12.35.0.js" defer="defer"></script><style>

I use notepad to build my web pages and I have zero knowledge of JS, so I'm just assuming that the above variants of the same, but if not, could that be what's causing the page not to show correctly (and if so, I wonder what's causing that to happen)?

Or - and I suspect this to be a more likely explanation - am I not understanding something that's really basic (and obvious to everyone else) about what can and cannot be done when attempting to reverse proxy on an https page?

Kind regards,

Briain

*The audio stream is to facilitate other radio amateurs' monitoring an audio from a 10 GHz beacon receiver, located at my house.

  • Hello  

    Thanks for the detailed post. It certainly looks to be a really good question. I see you have applied the Basic Protection firewall profile. Have you tried configuring it without any firewall profile? Since it seems to me as something with Javascript, the firewall profile can play some role in it. You can also check the reverse-proxy logs for the time when it displays the blank white page. It might give you any error logs helpful to understand the issue.

    On a side note, please go through this recommended read article created by Douglas Sophos UTM: Securing Web Application Firewall (WAF) It explains different parts of WAF in detail. 

  • In reply to Jaydeep:

    Hi Jaydeep

    Thank you for the advice and yes, I did try it with no firewall profile (sorry, I forgot to mention that). Thank you for alerting me to the article from Douglas; I'll read that this afternoon.

    Of course, it was foolish of me not to include the WAF log files in my original post; oops! The fewest log entries appear when the websocket passthrough feature is enabled, with the results being as is shown below. The odd thing was that it did originally work (albeit with the two websocket error popups) and that I cannot get back to that state implies to me that it's changed something in the R Pi. It is a barebones installation (network install, desktop install, Gedit text editor, Java and IniFi) so I guess it's also possible that I'm missing a dependency (though my understanding is that UniFi shouldn't require anything more than what I have already installed).

    2020:01:07-13:34:43 hadrian httpd: id="0299" srcip="192.168.6.1" localip="192.168.6.2" size="0" user="-" host="192.168.6.1" method="GET" statuscode="302" reason="-" extra="-" exceptions="-" time="257261" url="/" server="REDACTED.ddns.net:9666" port="9666" query="" referer="-" cookie="csrf_token=3ZaPjpdAXQKhihh7yisupSkZFcnGWHCM; unifises=4EKkcV7ESOcek2yeGIGMKPkRxw4HrVBU" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="XhSI80Itc-vugsNC2QqDSgAAAAE"
    2020:01:07-13:34:43 hadrian httpd: id="0299" srcip="192.168.6.1" localip="192.168.6.2" size="0" user="-" host="192.168.6.1" method="GET" statuscode="302" reason="-" extra="-" exceptions="-" time="22138" url="/manage" server="REDACTED.ddns.net:9666" port="9666" query="" referer="-" cookie="unifises=4EKkcV7ESOcek2yeGIGMKPkRxw4HrVBU; csrf_token=3ZaPjpdAXQKhihh7yisupSkZFcnGWHCM" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="XhSI80Itc-vugsNC2QqDSwAAAAE"
    2020:01:07-13:34:43 hadrian httpd: id="0299" srcip="192.168.6.1" localip="192.168.6.2" size="0" user="-" host="192.168.6.1" method="GET" statuscode="304" reason="-" extra="-" exceptions="-" time="25176" url="/manage/account/login" server="REDACTED.ddns.net:9666" port="9666" query="?redirect=%2Fmanage" referer="-" cookie="unifises=4EKkcV7ESOcek2yeGIGMKPkRxw4HrVBU; csrf_token=3ZaPjpdAXQKhihh7yisupSkZFcnGWHCM" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="XhSI80Itc-vugsNC2QqDTAAAAAE"
    2020:01:07-13:34:43 hadrian httpd: id="0299" srcip="192.168.6.1" localip="192.168.6.2" size="5510" user="-" host="192.168.6.1" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="73969" url="/manage/fonts/5.12.35.0/ubnt-icon/style.css" server="REDACTED.ddns.net:9666" port="9666" query="" referer="REDACTED.ddns.net:9666/.../login cookie="csrf_token=3ZaPjpdAXQKhihh7yisupSkZFcnGWHCM; unifises=4EKkcV7ESOcek2yeGIGMKPkRxw4HrVBU" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="XhSI80Itc-vugsNC2QqDTQAAAAE"

    Curiously, I repeated the test several times and it varies; sometimes there are just the first three of the above four entries, then at other times, it ends with that statuscode="200".

    I've just installed another instance of UniFi on a Windows PC, so I'll try reverse proxy on that to see if it shows the login page correctly (and also to see if accessin it generates any log entries).

    Briain

  • In reply to Jaydeep:

    Hi

    Well that was almost quite interesting! I used the reverse proxy on the new instance of UniFI (running on a Windows 10 machine) and it behaved similarly to the Pi originally did; I got to the login page and it showed two websocket errors. I then enabled websocket passthrough and that cleared them, but after logging in, it only showed the UniFi settings for a few seconds before dropping back to the login screen. This was exactly what happened with the Pi, so I followed the same steps as I'd originally followed (I cleared the browser cache) then tried accessing the UniFi login page again (that's when the Pi instance started showing the blank screen) but this time, after clearing the browser cache the Windows instance works perfectly!

    The log entries start identically to what's shown in my previous post, but they extend considerably onwards from that point (there are a lot of entries) but it all does seem to be working as it should.

    Just for information (in case anyone else is mad enough to be reading all this) the virtual webserver is currently set with Rewrite HTML, Rewrite cookies, Pass host header all enabled and with basic protection also enabled (and with websocket passthrough enabled in the site path routing section) and it certainly all appears to be working as it should.

    It's most interesting that when following the above steps, it all behaved identically to how it had behaved when doing the Pi, up to the point where I enabled websocket passthrough (and cleared the browser cache). I can only assume that something must have corrupted the Pi version of UniFi, so as soon as time permits, I'll start again from square one (a completely new barebones Pi build) and see if it happens again. It'll be a few days before I have time to do all that, but I will post back to let folks know how it goes (hopefully with success, and perhaps even a smiley symbol).

    Kind regards,

    Briain

  • In reply to Briain:

    Interesting turn of events!

    I'm more curious now to know what went wrong with Pi. Pi is a far more economical option (energy and resource wise). I will happily wait to know what happens next.

  • In reply to Jaydeep:

    Hi

    It's great that the reverse proxy worked perfectly with the Windows instance of UniFi as that clearly shows that it's not a UTM 9.700-5 issue (and indeed, I've also now verified which settings enable it to work properly, so that's very useful) but of course, it is still mildly intriguing why the Pi variant failed, so I will take a deeper dive into it (I now have one new idea on what might be upsetting the UniFI installation on this particular Pi build) and though it isn't a UTM 9 issue, the answer could be useful for someone, so I will update this thread (in a few days time).

    Bri