Let's encrypt: Error challenge is invalid

I'm trying to get a LE certificate for 2 domainnames that both point to the same IP-address (which is the same UTM on it's external interface).

However cert doesn't get generated, see log for errors (I have partly obfuscated the domainnames and my IP-address):

2018:09:26-14:54:02 firewall letsencrypt[9407]: I Renew certificate: handling CSR REF_CaCsrLetsEncryUtm for domain set [firewall.****s.eu,utm.****s.eu]
2018:09:26-14:54:02 firewall letsencrypt[9407]: I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain firewall.****s.eu --domain utm.****s.eu
2018:09:26-14:54:08 firewall letsencrypt[9407]: I Renew certificate: command completed with exit code 256
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: {
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "type": "http-01",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "status": "invalid",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "error": {
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "type": "urn:acme:error:unauthorized",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "detail": "Invalid response from firewall.****s.eu/.../TNi4gpGYdTnXumYHVFQIjuEw0f1wweFhNbEfIceKr38: q%!(EXTRA string=\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n\u003chead\u003e\n\u003cmeta charset=\"utf-8\"\u003e\n\u003cstyle\u003ebody{font-family:Arial,Helvetica,sans-serif;font-size:12px;text-alig)",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "status": 403
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: },
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "uri": "acme-v01.api.letsencrypt.org/.../7689019129",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "token": "TNi4gpGYdTnXumYHVFQIjuEw0f1wweFhNbEfIceKr38",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "validationRecord": [
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: {
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "url": "firewall.****s.eu/.../TNi4gpGYdTnXumYHVFQIjuEw0f1wweFhNbEfIceKr38",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "hostname": "firewall.****s.eu",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "port": "80",
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "addressesResolved": [
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "84.*.*.89"
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: ],
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: "addressUsed": "84.*.*.89"
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: }
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: ]
2018:09:26-14:54:08 firewall letsencrypt[9407]: E Renew certificate: COMMAND_FAILED: })
2018:09:26-14:54:08 firewall letsencrypt[9407]: I Renew certificate: sending notification WARN-603
2018:09:26-14:54:08 firewall letsencrypt[9407]: [WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service
2018:09:26-14:54:08 firewall letsencrypt[9407]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1)

Parents
  • It looks like something's turning the plaintext ACME challenge into a HTML page. See the "EXTRA string" in the log with DOCTYPE and other HTML stuff.

    Just to make sure: There's nothing between WAF and Let's Encrypt, is that correct? Like a load balancer or some other "active" network component that might mess with the responses from WAF?

    Here's how you can help us to figure this out:

    • Post the line in /var/log/reverseproxy.log that belongs to the ACME challenge request (search for "/.well-known/acme-challenge")
    • Disable the certificate renewal process on the command line for the next steps
      touch /tmp/disable_letsencrypt_renew_certificates
    • In WebAdmin click on the "Renew" button next to your failing certificate
    • Attach /var/chroot-reverseproxy/usr/apache/conf/reverseproxy.conf to your post
    • Of course anonymize any sensitive information in there
    • If this file is too big or you feel uncomfortable posting it here, we will find another way to get the information in it we need
    • Create a small text file in the ACME challenge location
      echo "test" > /var/chroot-reverseproxy/var/letsencrypt/acme-challenge/test
    • It would be ideal if you could request that file with a command line tool like curl including the HTTP headers
      curl -i http://<your domain>/.well-known/acme-challenge/test
    • Otherwise use a browser and check if it's returned as a plaintext file or a HTML page
    • If you used curl, post it's response (including the HTTP headers) here
    • When done you can remove the ACME challenge test file
      rm /var/chroot-reverseproxy/var/letsencrypt/acme-challenge/test
    • Re-enable the certificate renewal process by removing the file that we created at the beginning. It will execute the certificate renewal a little bit later and you will probably receive another failure notification.
      rm /tmp/disable_letsencrypt_renew_certificates

    That's a lot of steps, so thanks for your help!

Reply
  • It looks like something's turning the plaintext ACME challenge into a HTML page. See the "EXTRA string" in the log with DOCTYPE and other HTML stuff.

    Just to make sure: There's nothing between WAF and Let's Encrypt, is that correct? Like a load balancer or some other "active" network component that might mess with the responses from WAF?

    Here's how you can help us to figure this out:

    • Post the line in /var/log/reverseproxy.log that belongs to the ACME challenge request (search for "/.well-known/acme-challenge")
    • Disable the certificate renewal process on the command line for the next steps
      touch /tmp/disable_letsencrypt_renew_certificates
    • In WebAdmin click on the "Renew" button next to your failing certificate
    • Attach /var/chroot-reverseproxy/usr/apache/conf/reverseproxy.conf to your post
    • Of course anonymize any sensitive information in there
    • If this file is too big or you feel uncomfortable posting it here, we will find another way to get the information in it we need
    • Create a small text file in the ACME challenge location
      echo "test" > /var/chroot-reverseproxy/var/letsencrypt/acme-challenge/test
    • It would be ideal if you could request that file with a command line tool like curl including the HTTP headers
      curl -i http://<your domain>/.well-known/acme-challenge/test
    • Otherwise use a browser and check if it's returned as a plaintext file or a HTML page
    • If you used curl, post it's response (including the HTTP headers) here
    • When done you can remove the ACME challenge test file
      rm /var/chroot-reverseproxy/var/letsencrypt/acme-challenge/test
    • Re-enable the certificate renewal process by removing the file that we created at the beginning. It will execute the certificate renewal a little bit later and you will probably receive another failure notification.
      rm /tmp/disable_letsencrypt_renew_certificates

    That's a lot of steps, so thanks for your help!

Children