Thread Info
  • Locked Locked
  • Replies 13 replies
  • Subscribers 4 subscribers
  • Views 39324 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

WAF - a couple of learning points (learnt the hard way)

Louis-M

Last year, I replaced some clustered Cisco ASA 5510's with some clustered SG310's.

There was a fair amount of work to do as you simply could not export the rules and the terminology was a bit different. Nonetheless, the SG310's went in smoothly and we've been in the adhoc process of adding more and more functionality to users that the UTM's offer.

Part of this is WAF which we are using to retire the last ASA 5510 out as this is solely being used as a clientless SSL vpn (using java)

So I've been coming back to this a bit at a time as we discovered issues that took a little thinking about. The main root of our problems was DNS although you probably wouldn't have thought it was this looking at the errors.

The two big mistakes I made was:

1. Apply request routing for DNS for our external domain. As it was a new concept to us, we applied this from the UTM to all of our internal domain namespaces which was ok at the time for us to do due to the way our DNS was previously setup. We now have it setup so internal clients > internal dns servers > UTM. The UTM is the only device to perform recursive requests to the web.
The big mistake at the time was to apply DNS request routing  to our external domain from the UTM.

Our internal network is like any other eg 6 DNS servers with the internal domain and the external domain as well so that any external clients trying to get to serverA.mydomain.com would be able to use the internal ip address when they were inside the network and use the external public ip address when they were outside the network.

By getting the UTM to request route to the external DNS namespace on our internal DNS servers, the UTM was resolving serverA.mydomain.com to the internal ip address when it should have been resolving it to the external public ip address.

This had a massive detrimental effect on WAF. It was getting confused with the FQDN's. This wasn't noticed until we attempted to implement WAF

So, think very carefully when applying request routing on DNS and never "request route" the external domain name which you want WAF to use.

2. The 2nd issue was again a DNS issue and the mistake here was to add additional external hostnames to a host entry/definition on the UTM. So the UTM would see our internal  web server as an internal private ip but resolve the hostnames of the internal as well as the external hostname. Again, this caused confusion for the poor old WAF which by now, didn't know whether it was coming or going.

Bottom line is when using WAF (or the UTM) is to make sure it doesn't get confused with internal hostnames and external hostnames. So if you have an external FQDN pointing to an internal server, make sure that the UTM can only resolve the external FQDN to the external public ip and the internal server FQDN to the internal private ip. If it can resolve the external to the internal and vice versa, it will get confused and offer up mixed results that sometime work and sometimes don't.

Hopefully this will help somebody in the future and I think the moral of the story is:

The UTM offers a whole host of functionality but please be 100% sure as to what they mean before applying them otherwise you could end up with unpredictable behavior.



This thread was automatically locked due to age.
Parents

  • Pont 2 is very interesting.  I have an internal website that isn't loading CSS files or javascript when published for public access through WAF.  The Web Dev advised that those links were loading as the internal resources using the ip address, ie:@import url("WEBSERVERIP/.../comment.css instead of @import url("DOMAIN/.../comment.css certificate is for the domain, so it doesn’t load the css.  So far this only seems to happen on this Word Press site but I've only tested 3 sites so far.

    I'm wondering if this could be our issue?  If I do a DNS lookup on the UTM for the FQDN of the site it does resolve to the internal webserver IP.  UTM is primarily used as an AD integrated forward proxy so integrates with internal DNS.  This is an internal site that is accessible from internal clients so will resolve that FQDN from internal DNS.

    Do I need to create a host definition for this FQDN resolving to the public IP of the website?  Does mean that we'll have to revisit our browser proxy exceptions if we'll have to do this for every single published site because we have a lot of different website domains.  We are a Local Authority and it seems that services like to brand every single project that they run that needs a website e.g. go smarter, go green, go by bike....  I'm sure you get the idea  :(

  • in reply to mahaddon

    Our DNS & UTM is set up like so:

    INTERNAL NETWORK
    4x DNS servers. All clients within the network point to these. On these dns servers, we have 2 zones, mydomain.local (our internal dns) and mydomain.com (our external dns)

    It is setup like this because we have mobile users and if we give them a shortcut to mydomain.local (internal resolution), when they are external, they can't resolve. So we give them links to mydomain.com which on our internal dns server points to a host with the internal ip eg 10.0.0.1
    When they leave the network and use their ISP's server, they will resolve to the external ip eg 1.2.3.4

    The internal dns servers forward to the UTM for external requests and only these are allowed access to the UTM dns proxy. That way nobody can resolve from within unless they go to our dns servers which in turn can only recursively resolve to the UTM.

    THE UTM setup

    Now the UTM might want to resolve to some internal clients. So we put in dns request routing for mydomain.local ONLY and not mydomain.com. Internal hosts should only ever have the internal name so the UTM doesn't resolve it to the external name.

    If you run a dns lookup on the UTM is should return the following:

    hostA.mydomain.local = 10.0.0.1 (because it is using request dns routing for mydomain.local)
    hostA.mydomain.com = 1.2.3.4 (because it can't resolve the hostname and will then recursively resolve to your external dns eg 1.2.3.4

    WAF

    When you set this up, the virtual webserver eg 1.2.3.4 points to your internal server 10.0.0.1
    It should not be allowed to mix this up with ie virtual webserver 1.2.3.4 points to 10.0.0.1 & 1.2.3.4

    If you think about it, you wouldn't DNAT your external ip 1.2.3.4 to an internal ip of 1.2.3.4 so I suppose it's the same principle. My WAF was getting mixed up with this and now it works fine. It's hard to spot because the additional hostnames are hidden but tell tale sign is to do a dns lookup from the UTM to see what it returns.

  • in reply to Louis-M

    Unfortunately we can't segregate like that.  We have an internal AD local domain that's not publically visible but all of our non AD domains need to be accessed both publically and internally so we need to have zones in both internal and public DNS.  We'd never get away with setting up 2 domains for each website not least because we'd need to buy a SAN cert for every site but also because the officers in the business want their specific pretty domain name for their website.  By default our UTMs will resolve the internal IPs through internal DNS because it's configured as a backend firewall model and not an edge firewall in the same way as our TMGs are configured as a backend firewall.  It's also used primarily as a forward proxy/web filter solution.

    Yes we have virtual webservers set with a UTM listener IP and the real webservers have their own internal IP.  This is what worries me though.  This internal webserver IP should not be publically visible especially when using WAF and I don't know why this is the case.  I'm hoping that it's simply because the internal Word Press site is configured to use IPs in it's links but I don't know for certain.  I'm also not looking forward to tasking the web devs with trawling through all of their websites to look for links with webserver IPs in them  :(

  • in reply to mahaddon

    Hi, when I said we had 2 domains, it was for simplicity's sake. You can have just 2 domains but 100 zones within dns that aren't in the slightest related to your AD eg internally, I can send anybody who types google.com to 10.1.0.1 if I want.

    The main thing here is to make sure that the UTM doesn't resolve to the internal domain name of the server you are publishing. Therefore, you can't have a host with the external hostname pointing to the internal ip. The internal host should only have the internal ip and internal hostname.

    If you need mobile users to resolve to an internal ip when inside the network, you simply create a zone in dns and become authoritative for it. That way, mobile users will always resolve to the internal ip of the host when within the network. As they move outside, your external dns server becomes authoritative for the domain and they resolve to the external address.

  • in reply to mahaddon

    So, to summarize:

    You UTM is not on the edge of the network, so I assume you are NATing external HTTP/S to the UTM' internal IP.

    You have internal clients that access your webservers, and for that to work you have external DNS zones created on your AD DNS pointing to the webservers' inernal IPs.

    You are now trying to publish your websites through WAF and noticing that some requests contains the webservers' internal IPs.

    I think that your sites are not using relative links. It's not WAF's fault if your websites are passing links with internal IPs, as WAF will not rewrite the request unless you enable "Rewrite HTML" option at the virtual webserver profile (which I noticed you did not enable from another post you wrote). Have you tried enabling it?

    Also, I'm pretty sure that if your UTM were on the edge of your network you would not need to bother with creating your external DNS zones internally, as WAF would be able to server these sites even if the request came from inside the network.

    Regards,

    Giovani

     

  • in reply to giomoda

    Would the UTM allow that? ie external access back on itself? I know some FW's do not allow an internal ip to come back through dnat's or proxies.

    And yes, you are quite right, it's not needed. We purely do it to keep all requests within our cloud which means that the only things hitting the UTM's from the inside are external bound traffic.

  • in reply to Louis-M

    WAF would. That's how we access our internal ticketing system actually, both internally and externally. WAF is used for protection and loadbalancing between my two webservers.

    And I'm not saying it's wrong to have an internal DNS and taking this load off the UTM. It all depends on the size of your UTM and how much requests you have to deal from the internal network. But I think it's a lot simpler to use WAF than to have the same DNS zones on multiple DNS servers, so I always use WAF when possible.

    Regards - Giovani

  • in reply to giomoda

    thanks Giovani

    we have UTMs with 2 network interfaces.  One on the internal network used predominantly for internal forward proxy clients and one on the DMZ network.  DMZ network interface has additional IPs that are used for UTM Virtual webservers and these IPs NAT out through the edge firewall.

    All webservers are on the internal network so internal DNS resolves websites to those webserver IPs and internal clients route directly to the webservers bypassing the UTMs by web browser proxy exceptions.

    We are testing UTM WAF to publish the internal websites to public clients.  UTM Real webservers resolve to internal IPs of webservers.  Public DNS resolves to edge firewall public facing IP addresses which NAT to the UTM DMZ interface additional IPs.

    I've inserted a very crude diagram

    That's an interesting suggestion about using rewrite URL.  I had considered that but the UTM help suggested that we shouldn't use this for javascript and/or CSS files?  I will test using that though.

    Also my web dev thinks the sites are relative but I can confirm that.  He used this site to test from his phone using dev tools in Chrome on his pc to debug the browser session on his phone

    https://developers.google.com/web/tools/chrome-devtools/remote-debugging/

  • in reply to mahaddon

    Hey mahaddon.

    I don't see any recommendations against "Rewrite HTML" at the Sophos UTM help page, just some warnings about poorly coded webpages. 

    Rewrite HTML (optional): Select this option to have the UTM rewrite links of the returned webpages in order for the links to stay valid. Example: One of your real webserver instances has the hostname yourcompany.local but the virtual webserver's hostname on the UTM is yourcompany.com. Thus, absolute links like <a href="http://yourcompany.local/"> will be broken if the link is not rewritten to <a href="http://yourcompany.com/"> before delivery to the client. However, you do not need to enable this option if either yourcompany.com is configured on your webserver or if internal links on your webpages are always realized as relative links. It is recommended to use the option with Microsoft's Outlook Web Access and/or Sharepoint Portal Server.

    Note – It is likely that some links cannot be rewritten correctly and are therefore rendered invalid. Ask your website author(s) to format links consistently.

    Apart from URL rewriting, the HTML rewriting feature also fixes malformed HTML, for example:

    • <title> tags are moved in DOM tree from node html > title to correct html > head > title
    • Quotes around HTML attribute values are fixed (e.g., name="value becomes name="value")

    Note – HTML rewriting affects all files with a HTTP content type of text/* or *xml*, where * is a wildcard. Make sure that other file types, e.g. binary files, have the correct HTTP content type, otherwise they may get corrupted by the HTML rewriting feature.

     

    Just by reading this I'm confident enabling "Rewrite HTML" will solve most of your issues. I don't think that this specific issue has anything to do with your DNS setup. To me it just looks like absolute links are being used. Don't take your webdev's word on it. Enable it and see how it goes.

    That being said, as a general best practice you should follow Louis suggestion and allow your UTM to resolve your websites hostnames to their external IPs, otherwise some strange behavior can occur.

    I never used UTM as a second firewall, I usually have it sitting on the edge of the network and I usually follow this for DNS:

    Internal Clients -> Internal (AD) DNS Server  -> UTM DNS resolver

    And for the UTM DNS setup I usually use my ISPs' DNS Server or Google' DNS servers as forwarders. Check these best practices.

    Also, I never create any DNS requests routing on the UTM resolving external DNS names to internal addresses. Whatever needs to be resolved internally should be resolved by the internal DNS.

    Regards - Giovani

Reply
  • in reply to mahaddon

    Hey mahaddon.

    I don't see any recommendations against "Rewrite HTML" at the Sophos UTM help page, just some warnings about poorly coded webpages. 

    Rewrite HTML (optional): Select this option to have the UTM rewrite links of the returned webpages in order for the links to stay valid. Example: One of your real webserver instances has the hostname yourcompany.local but the virtual webserver's hostname on the UTM is yourcompany.com. Thus, absolute links like <a href="http://yourcompany.local/"> will be broken if the link is not rewritten to <a href="http://yourcompany.com/"> before delivery to the client. However, you do not need to enable this option if either yourcompany.com is configured on your webserver or if internal links on your webpages are always realized as relative links. It is recommended to use the option with Microsoft's Outlook Web Access and/or Sharepoint Portal Server.

    Note – It is likely that some links cannot be rewritten correctly and are therefore rendered invalid. Ask your website author(s) to format links consistently.

    Apart from URL rewriting, the HTML rewriting feature also fixes malformed HTML, for example:

    • <title> tags are moved in DOM tree from node html > title to correct html > head > title
    • Quotes around HTML attribute values are fixed (e.g., name="value becomes name="value")

    Note – HTML rewriting affects all files with a HTTP content type of text/* or *xml*, where * is a wildcard. Make sure that other file types, e.g. binary files, have the correct HTTP content type, otherwise they may get corrupted by the HTML rewriting feature.

     

    Just by reading this I'm confident enabling "Rewrite HTML" will solve most of your issues. I don't think that this specific issue has anything to do with your DNS setup. To me it just looks like absolute links are being used. Don't take your webdev's word on it. Enable it and see how it goes.

    That being said, as a general best practice you should follow Louis suggestion and allow your UTM to resolve your websites hostnames to their external IPs, otherwise some strange behavior can occur.

    I never used UTM as a second firewall, I usually have it sitting on the edge of the network and I usually follow this for DNS:

    Internal Clients -> Internal (AD) DNS Server  -> UTM DNS resolver

    And for the UTM DNS setup I usually use my ISPs' DNS Server or Google' DNS servers as forwarders. Check these best practices.

    Also, I never create any DNS requests routing on the UTM resolving external DNS names to internal addresses. Whatever needs to be resolved internally should be resolved by the internal DNS.

    Regards - Giovani

Children
  • in reply to giomoda

    thanks Giovani

    yes my mistake.  I had miss remembered and I actually read the warning about javascript in the Static URL Hardening help section.  I'll retest using Rewrite HTML I think.

    And yes I did think about creating definitions on UTM resolving the websites that I'm reverse proxying to their public IPs but I'll have to check with our AD GP guys to make sure the Proxy Exceptions policy is up to date.  I think we're running out of characters in that policy  :)

    thanks for your help  :)

     

  • in reply to giomoda

    OK a bit of progress.

    I spoke to my web devs and enabled Rewrite HTML and also Rewrite cookies and it looks as if this resolved the issue.  Web dev confirmed no IPs in the browser content and I saw the following in the UTM WAF logs (sanitised)

    reverseproxy: [Tue Apr 11 10:09:40.513196 2017] [proxy_http:error] [pid :] ()End of file found: [client 149.###:###:###:46450] AH01102: error reading status line from remote server WEBSITE_IP:443, referer: https://WEBSITE_URL/sites/all/themes/css/h.css?on3ztz

    reverseproxy: srcip="149.###:###:###" localip="UTM_VIRTUAL_SERVER_IP" size="0" user="-" host="149.###:###:###" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="3538" url="/sites/all/themes/images/PoT.gif" server="WEBSITE_URL referer="https://WEBSITE_URL/sites/all/themes/css/h.css?on3ztz" cookie="has_js=1; _ga=############" set-cookie="-"

    Looks as if the webserver IP is sent with the css file URL which fails and then UTM rewrites it with the website domain which works but I admit that's just my uninformed interpretation of the WAF log entries

    Note I didn't create any definitions on UTM resolving the website domain to the public IP.

    thanks for your help Giovani

Unfiltered HTML