Thread Info
  • State Not Answered
  • Locked Locked
  • Replies 11 replies
  • Subscribers 1 subscriber
  • Views 73257 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

AH00898 Error reading from remote server returned by

IngoPohlschneider
Hello everyone,
I am having trouble with some HTTPS services deployed via WAF.

Previously those services were deployed using DNAT.

When accessing the URL I get and Proxy error

This is what the log shows:
2013:09:23-13:41:18  reverseproxy: [Mon Sep 23 13:41:18.313296 2013] [proxy_http:error] [pid 5211:tid 4010797936] (20014)Internal error: [client :58915] AH01102: error reading status line from remote server :443
2013:09:23-13:41:18  reverseproxy: srcip="" localip="" size="0" user="-" host="" method="GET" statuscode="200" reason="-" extra="-" time="14736" url="/mifs/c/apix/v1/client//appstore" server="" referer="-" cookie="JSESSIONID=" set-cookie="-"
2013:09:23-13:41:18  reverseproxy: [Mon Sep 23 13:41:18.363377 2013] [proxy_http:error] [pid 5211:tid 3985619824] (20014)Internal error: [client :58917] AH01102: error reading status line from remote server :443
2013:09:23-13:41:18  reverseproxy: [Mon Sep 23 13:41:18.363398 2013] [proxy:error] [pid 5211:tid 3985619824] [client :58917] AH00898: Error reading from remote server returned by /mifs/c/apix/v1/client//appstore
2013:09:23-13:41:18  reverseproxy: srcip="" localip="" size="461" user="-" host="" method="GET" statuscode="502" reason="-" extra="-" time="13823" url="/mifs/c/apix/v1/client//appstore" server="" referer="-" cookie="JSESSIONID=" set-cookie="-"
2013:09:23-13:41:18  reverseproxy: srcip="" localip="" size="39" user="-" host="" method="GET" statuscode="404" reason="-" extra="-" time="14408" url="/apple-touch-icon-152x152-precomposed.png" server="" referer="-" cookie="-" set-cookie="JSESSIONID=; Path=/mifs; Secure; HttpOnly, HASH_JSESSIONID=; Path=/mifs; Secure"
2013:09:23-13:41:18  reverseproxy: srcip="" localip="" size="39" user="-" host="" method="GET" statuscode="404" reason="-" extra="-" time="13898" url="/apple-touch-icon-152x152.png" server="" referer="-" cookie="-" set-cookie="JSESSIONID=; Path=/mifs; Secure; HttpOnly, HASH_JSESSIONID=; Path=/mifs; Secure"
2013:09:23-13:41:18  reverseproxy: srcip="" localip="" size="39" user="-" host="" method="GET" statuscode="404" reason="-" extra="-" time="14605" url="/apple-touch-icon-precomposed.png" server="" referer="-" cookie="-" set-cookie="JSESSIONID=; Path=/mifs; Secure; HttpOnly, HASH_JSESSIONID=; Path=/mifs; Secure"
2013:09:23-13:41:18  reverseproxy: srcip="" localip="" size="45" user="-" host="" method="GET" statuscode="404" reason="-" extra="-" time="14330" url="/apple-touch-icon.png" server="" referer="-" cookie="-" set-cookie="JSESSIONID=; Path=/mifs; Secure; HttpOnly, HASH_JSESSIONID=; Path=/mifs; Secure" 


I masked some information in 

This happens only with this subpath. There is another portal running on this webserver (same real and virtual server) and thats running smoothly

Any ideas?

WAF is kinda hart to debug since the logs give not much of a clue what is the issue [:(]

best regards
chaser


This thread was automatically locked due to age.
  • 0
    I'm confused - is this an attempt to access an internal server from an internal workstation?  That shouldn't have worked with a DNAT.

    Cheers - Bob
     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • 0
    Hello Balfson,
    I am aware of this KB-article. Nevertheless it is currently set via NAT to the following:
    Any -> HTTPS -> Ext IP -> DNAT to Int. IP

    Works from external and internal. Internal users are using transparent proxy.

    But thats not my issue. My issue is that its not working once I use the WAF to do it:

    Virtual server listening on Ext. IP with some hostnames
    Realserver on Int IP 

    Both are set to HTTPS (with correct cert).
  • 0
    I think that reverseproxy only works with traffic in the INPUT chain, so I'm confused that the other one appears to work.  Can you see lines in the reveseproxy log where an internal user successfully accesses the internal Real Server via the Virtual Server on the External interface?

    I also don't think that httpproxy can talk to the reverseproxy.  Can you see lines in the Web Filtering log showing an internal user accessing an internal server (via either its private or public IP)?

    I think the only way to have the WAF handle internal->internal traffic is to use split DNS, aiming the FQDN at an address on the Internal interface, and then adding a Virtual Server on the Internal inteface.

    Cheers - Bob
     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • 0
    Hello
    I myself am able to access a webadmin available via a certain address on that machine (https://sub.domain.tld/admin) from my internal admin PC (10.***x address) going via the public address

    The error from the log appears when accessing a certain service (on that same https://sub.domain.tld/otherpath)

    cheers - chas0r
  • 0
    Chas0r, if you do nslookup sub.domain.tld from your admin PC, do you get the public IP and not the 10. IP?

    Is SSL scanning selected in Web Filtering?

    Sorry to pepper you with questions, but what you're describing doesn't fit with my understanding of how this all works, so I can't see how to fix your issue.

    Cheers - Bob
     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • 0
    Hello
    yes I get the public (external) IP of the DNS-record. So no splitt-DNS.

    I understand your confusion.

    Nevertheless the issue is not the crazy DNAT [;)] I want to get rid of that anyways.
    Its the issue that I get this errors for certain paths of the request done via WAF (and this error appears if you are an internal user as well as external)

    Best regards and sorry for the short answers...busy these days [:(]

    EDIT on the NAT-topic:
    If one reads closely one might get the opinion that this only is a problem if the accessed IP is the primary IP of the UTM:

    Cause
    DNS resolutions will reference the external interface address (given by the external DNS). 
    When the http proxy is enabled this results in a loop as the external address of the ASG is trying to access itself. 


    Also not sure if the KB still applies (though I think so since we have several (REQUIRED) FNATs for internal services. 

    Known to apply to the following Sophos product(s) and version(s)


    Sophos UTM Software Appliance

    Operating systems
    ASG 7, ASG 8


    Those FullNATs where required once the systems accessing the destination-system where in the same subnet as the webserver.
  • 0
    Full NATs are applied at the same time as DNATs, so it's even more confusing that the reverseproxy even sees the traffic from "Internal (Network)".

    Can you see lines in the reveseproxy log where an internal user successfully accesses the internal Real Server via the Virtual Server on the External interface?
     
    Can you see lines in the Web Filtering log showing an internal user accessing an internal server (via either its private or public IP)?

    Have you tried enabling sticky sessions for this path?

    Cheers - Bob
     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • 0
    Hey Balfson,
    I just accessed an internal page via external address from my internal host:

    Proxy only shows my client side-loading some external jquery-Lib:
    2013:09:25-13:46:13  httpproxy[15985]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="GET" srcip="" dstip="108.161.188.209" user="" statuscode="200" cached="0" profile="REF_HttProTestTransp (Transparent Proxy Test)" filteraction="REF_ACC_GBL_9c410826b3e94d2bbd93a6b9b267dfd4dfd4 (Grimme Group)" size="273199" request="0x1ca40c00" url="code.jquery.com/.../Hardware"


    The reverseproxy log shows the following for my IP (attached due to length)

    I exchanged some metadata but the external IP is the real external address (additional address on the external interface of the UTM) and the internal address is a private IP (10.x.x.x)

    I can access this page without any issues

    Cheers chas0rde

    EDIT: About sticky sessions: This is not required since currently only one real server is protected per virtual server. Therefore this would have no impact (at least if I get sticky sessions correct [;)] )
    reverseproxy.zip
  • 0
    I hope you will let us know what Sophos Support has to say about this...

    It looks like you might want to make an Exception for Cookie signing for traffic from internal users.

    Cheers - Bob
     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • 0
    Hello
    I am sporadically getting the error on other sites as well. I don't know if maybe timeouts are the problem causing this issue?

    On the other site logs show the following: 
    2013:10:02-08:16:31 DEDAM-A000-1 reverseproxy: srcip="193.165.77.11" localip="" size="7228" user="-" host="193.165.77.11" method="GET" statuscode="200" reason="-" extra="-" time="595027" url="/gwla/gwla_archiv.aspx" server="" referer="start.aspx" cookie="ASP.NET_SessionId=ttjhd2qnjuob3f3dnnblgpfa" set-cookie="-"

    2013:10:02-08:42:35 DEDAM-A000-1 reverseproxy: srcip="87.48.154.79" localip="" size="1404" user="-" host="87.48.154.79" method="GET" statuscode="200" reason="-" extra="-" time="479705" url="/gwla/gwla_start.aspx" server="" referer="https:///start.aspx" cookie="ASP.NET_SessionId=t1oix3yv1nrom0u2e502wxuq" set-cookie="-"

    2013:10:02-08:42:37 DEDAM-A000-1 reverseproxy: srcip="87.48.154.79" localip="" size="2602" user="-" host="87.48.154.79" method="GET" statuscode="200" reason="-" extra="-" time="6808" url="/gwla/gwla_seite1.aspx" server="" referer="https:///gwla/gwla_start.aspx" cookie="ASP.NET_SessionId=t1oix3yv1nrom0u2e502wxuq" set-cookie="-"

    2013:10:02-09:16:54 DEDAM-A000-1 reverseproxy: srcip="87.48.154.79" localip="" size="1404" user="-" host="87.48.154.79" method="GET" statuscode="200" reason="-" extra="-" time="498766" url="/gwla/gwla_start.aspx" server="" referer="https:///start.aspx" cookie="ASP.NET_SessionId=t1oix3yv1nrom0u2e502wxuq" set-cookie="-"

    2013:10:02-09:16:57 DEDAM-A000-1 reverseproxy: srcip="87.48.154.79" localip="" size="2602" user="-" host="87.48.154.79" method="GET" statuscode="200" reason="-" extra="-" time="6861" url="/gwla/gwla_seite1.aspx" server="" referer="https:///gwla/gwla_start.aspx" cookie="ASP.NET_SessionId=t1oix3yv1nrom0u2e502wxuq" set-cookie="-"

    2013:10:02-09:31:38 DEDAM-A000-1 reverseproxy: srcip="194.76.180.113" localip="" size="421" user="-" host="194.76.180.113" method="GET" statuscode="502" reason="-" extra="-" time="3307" url="/gwla/gwla_seite1.aspx" server="" referer="https:///gwla/gwla_start.aspx" cookie="ASP.NET_SessionId=0kehhebku0do2t55wklftk45" set-cookie="-"

    2013:10:02-10:05:13 DEDAM-A000-1 reverseproxy: srcip="87.48.154.79" localip="" size="1404" user="-" host="87.48.154.79" method="GET" statuscode="200" reason="-" extra="-" time="428801" url="/gwla/gwla_start.aspx" server="" referer="https:///start.aspx" cookie="ASP.NET_SessionId=t1oix3yv1nrom0u2e502wxuq" set-cookie="-"

    2013:10:02-10:05:15 DEDAM-A000-1 reverseproxy: srcip="87.48.154.79" localip="" size="2602" user="-" host="87.48.154.79" method="GET" statuscode="200" reason="-" extra="-" time="7316" url="/gwla/gwla_seite1.aspx" server="" referer="https:///gwla/gwla_start.aspx" cookie="ASP.NET_SessionId=t1oix3yv1nrom0u2e502wxuq" set-cookie="-"

    2013:10:02-10:23:08 DEDAM-A000-1 reverseproxy: srcip="" localip="" size="441" user="-" host="" method="GET" statuscode="502" reason="-" extra="-" time="3001" url="/gruppen/gwla_gruppenleiter.aspx" server="" referer="start.aspx" cookie="ASP.NET_SessionId=3mrbfs45cb3jb255dog4waaj" set-cookie="-"

    2013:10:02-10:50:06 DEDAM-A000-1 reverseproxy: srcip="87.48.154.79" localip="" size="1404" user="-" host="87.48.154.79" method="GET" statuscode="200" reason="-" extra="-" time="412563" url="/gwla/gwla_start.aspx" server="" referer="https:///start.aspx" cookie="ASP.NET_SessionId=t1oix3yv1nrom0u2e502wxuq" set-cookie="-"

    2013:10:02-10:50:07 DEDAM-A000-1 reverseproxy: srcip="87.48.154.79" localip="" size="2602" user="-" host="87.48.154.79" method="GET" statuscode="200" reason="-" extra="-" time="6818" url="/gwla/gwla_seite1.aspx" server="" referer="https:///gwla/gwla_start.aspx" cookie="ASP.NET_SessionId=t1oix3yv1nrom0u2e502wxuq" set-cookie="-"

    2013:10:02-10:59:18 DEDAM-A000-1 reverseproxy: srcip="" localip="" size="419" user="-" host="" method="GET" statuscode="502" reason="-" extra="-" time="3487" url="/gwla/gwla_start.aspx" server="" referer="start.aspx" cookie="ASP.NET_SessionId=3mrbfs45cb3jb255dog4waaj" set-cookie="-"

    2013:10:02-11:05:14 DEDAM-A000-1 reverseproxy: srcip="" localip="" size="419" user="-" host="" method="GET" statuscode="502" reason="-" extra="-" time="2664" url="/gwla/gwla_start.aspx" server="" referer="start.aspx" cookie="ASP.NET_SessionId=3mrbfs45cb3jb255dog4waaj" set-cookie="-"

    2013:10:02-11:09:44 DEDAM-A000-1 reverseproxy: srcip="" localip="" size="419" user="-" host="" method="GET" statuscode="502" reason="-" extra="-" time="2788" url="/gwla/gwla_start.aspx" server="" referer="start.aspx" cookie="ASP.NET_SessionId=3mrbfs45cb3jb255dog4waaj" set-cookie="-"
Unfiltered HTML