[Issue has resolved itself] LetsEncrypt: An 'unexpected' certificate renewal failure (anybody else, or is it just me)?

Edit: The issue resolved itself overnight (see post after this).

Hi Folks

I hope everybody here is keeping well and not under too much stress in these uncertain times.

I've been using UTM for about 4 years, so my settings are by now quite 'matured' and thus these days, I don't often need change DNAT or firewall rules. I've also been using the LetsEncrypt feature since not long after it was added to UTM and up until now, it's been working flawlessly. This morning, I noted a certificate renewal failure e-mail in my inbox, so I undertook the following steps to see if I could manually renew it.

1. Checked nothing silly in my DNAT rules (all okay)

2. Toggled off country blocking (I'm in UK and USA wasn't blocked, so that shouldn't have mattered, but I disabled all country blocking just to be sure)

3. Toggled off the virtual web servers in the WAF (one was set with http->https redirection, so I was pondering whether that might be upsetting things)

I cannot think what else might be causing the below 'challenge is invalid' and 'timeout' problem (I am double-NAT, but the outer router has both ports 80 and 443 opened to the UTM; prior to toggling off the virtual web servers, I also tested that http://mydomain redirected to https:/mydomain and showed the web page from the real server, which is Apache running on a Raspberry Pi) so I was just wondering if anybody else is having any LE renewal problems (or new certificate generation problems; I did try generating a fresh certificate, but that yielded identical failure results in the log files). Or does anybody else have a cool idea about what I might try (I still believe it's likely my doing, but I have run out of ideas on what to next check).

Kind regards, 

Briain

2020:04:11-11:48:03 hadrian letsencrypt[14112]: I Renew certificate: handling CSR REF_CaCsrMyDomainNameLeCert for domain set [MyDomainName.uk,www.MyDomainName.uk]
2020:04:11-11:48:03 hadrian letsencrypt[14112]: I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain MyDomainName.uk --domain www.MyDomainName.uk
2020:04:11-11:48:38 hadrian letsencrypt[14112]: I Renew certificate: command completed with exit code 256
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: {
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:   "type": "http-01",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:   "status": "invalid",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:   "error": {
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:     "type": "urn:ietf:params:acme:error:connection",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:     "detail": "Fetching MyDomainName.uk/.../N6euNRl12BJYeF954aWAZg3BL4rgb4e9pgPwlnj11eA: Timeout during connect (likely firewall problem)",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:     "status": 400
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:   },
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:   "url": "acme-v02.api.letsencrypt.org/.../TkIHWg",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:   "token": "N6euNRl12BJYeF954aWAZg3BL4rgb4e9pgPwlnj11eA",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:   "validationRecord": [
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:     {
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:       "url": "MyDomainName.uk/.../N6euNRl12BJYeF954aWAZg3BL4rgb4e9pgPwlnj11eA",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:       "hostname": "MyDomainName.uk",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:       "port": "80",
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:       "addressesResolved": [
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:         "111.222.333.444"
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:       ],
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:       "addressUsed": "111.222.333.444"
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:     }
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED:   ]
2020:04:11-11:48:38 hadrian letsencrypt[14112]: E Renew certificate: COMMAND_FAILED: })
2020:04:11-11:48:38 hadrian letsencrypt[14112]: I Renew certificate: sending notification WARN-603
2020:04:11-11:48:38 hadrian letsencrypt[14112]: [WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service
2020:04:11-11:48:38 hadrian letsencrypt[14112]: I Renew certificate: execution completed (CSRs renewed: 0, failed: 1)

NB I was getting tempted to just purchase a really cheap certificate, but it would be better to understand what's causing the above failure. :-)

  • Hi Folks

    All is now good; I now have a refreshed certificate (and the answer was to do nothing). 

    After posting the above, I re-enabled the virtual web servers (so including the one that does http->https redirect) and this morning, when I checked the status (expecting to see the red failure banner) I was pleasantly surprised to see that the automated certificate renewal process had worked at 01:38 this morning, so it looks like yesterday's debacle might have been due a problem beyond my network.

    Clearly I should have reflected upon the excellent advice from that fountain of wisdom, The Hitchhiker's Guide to the Galaxy:  

    At that moment the ship suddenly stopped rocking and swaying, the engine pitch settled down to a gentle hum.

    ``Hey, Ford,'' said Zaphod, ``that sounds good. Have you worked out the controls of this boat?''

    ``No,'' said Ford, ``I just stopped fiddling with them."

    Kind regards to all,

    Briain :-)