PHP script for uploading Lets Encrypt certs is broken since 19.0 MR1

EdmundSackbauer

I have not gotten 19 MR1 as of yet, but I will make it a point to manually update to it so I can try and replicate / resolve this issue.  

In the meantime, can you try manually creating a temp cert via the gui, and then applying that temp cert to your rules and see if you get a similar error?  Thanks!

Yes, this is what I did in the meantime, I manually created a temp cert and applied it to the rules, deleted the old LE cert, uploaded the new one with the same name, applied the new cert to the rules. That worked manually without problem.

Basically a script gets executed in different steps. Do you know, which step fails? 

So to speak, on which step are we having a problem, is the new cert there but does not get replaced etc. 

Script works in creating the temp cert, and the first rule is properly changed to the temp certificate.

Even the second rule is changed to the temp certificate, however the return string from the API cannot be parsed properly.

You could check the /log/apiparser.log if there is any error as well. 

INFO      Aug 02 10:09:13Z [11248]: Start Login Handler,Component : Login
ERROR     Aug 02 10:09:13Z [11248]: Key:ISCrEntity is not found in RequestMap File for Login.
INFO      Aug 02 10:09:13Z [11248]: Mapping file for Login component is /_conf/csc/IOMappingFiles//1800.1/Login/Login.xml
ERROR     Aug 02 10:09:13Z [11248]: Flag setting for this opcode is 18.
INFO      Aug 02 10:09:14Z [11248]: Opcode response: status:200
INFO      Aug 02 10:09:14Z [11248]: Authentication Successful
INFO      Aug 02 10:09:14Z [11248]: Start Set Handler,Component : FirewallRule
ERROR     Aug 02 10:09:14Z [11248]: Key:ISCrEntity is not found in RequestMap File for FirewallRule.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempsourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempexceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempsourceid" to handle logicaloperator.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="sourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempexceptionid" to handle logicaloperator.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="exceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: Flag setting for this opcode is 16.
INFO      Aug 02 10:10:20Z [11248]: Opcode response: status:200
INFO      Aug 02 10:10:20Z [11248]: End  SET Handler, Status : Success,  Component : FirewallRule, Transaction : , Operation : update.
MESSAGE   Aug 02 10:10:20Z [11248]: ENTITY 'FirewallRule' IMPORT Success
INFO      Aug 02 10:10:20Z [11248]: Command:/scripts/apiparser_generate_tar.sh /sdisk/api-1659434953040779.txt /sdisk/API-1659434953040779 /sdisk/APIXMLOutput/1659434952687.xml /sdisk/API-1659434953040779.tar /sdisk/API-1659434953040779.log 0 status:3
INFO      Aug 02 10:10:20Z [11248]: No need to create Tar file. Response file is /sdisk/APIXMLOutput/1659434952687.xmlSFVH_SO01_SFOS 19.0.1

INFO      Aug 02 10:09:13Z [11248]: Start Login Handler,Component : Login
ERROR     Aug 02 10:09:13Z [11248]: Key:ISCrEntity is not found in RequestMap File for Login.
INFO      Aug 02 10:09:13Z [11248]: Mapping file for Login component is /_conf/csc/IOMappingFiles//1800.1/Login/Login.xml
ERROR     Aug 02 10:09:13Z [11248]: Flag setting for this opcode is 18.
INFO      Aug 02 10:09:14Z [11248]: Opcode response: status:200
INFO      Aug 02 10:09:14Z [11248]: Authentication Successful
INFO      Aug 02 10:09:14Z [11248]: Start Set Handler,Component : FirewallRule
ERROR     Aug 02 10:09:14Z [11248]: Key:ISCrEntity is not found in RequestMap File for FirewallRule.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempsourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempexceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempsourceid" to handle logicaloperator.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="sourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempexceptionid" to handle logicaloperator.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="exceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: Flag setting for this opcode is 16.
INFO      Aug 02 10:10:20Z [11248]: Opcode response: status:200
INFO      Aug 02 10:10:20Z [11248]: End  SET Handler, Status : Success,  Component : FirewallRule, Transaction : , Operation : update.
MESSAGE   Aug 02 10:10:20Z [11248]: ENTITY 'FirewallRule' IMPORT Success
INFO      Aug 02 10:10:20Z [11248]: Command:/scripts/apiparser_generate_tar.sh /sdisk/api-1659434953040779.txt /sdisk/API-1659434953040779 /sdisk/APIXMLOutput/1659434952687.xml /sdisk/API-1659434953040779.tar /sdisk/API-1659434953040779.log 0 status:3
INFO      Aug 02 10:10:20Z [11248]: No need to create Tar file. Response file is /sdisk/APIXMLOutput/1659434952687.xmlSFVH_SO01_SFOS 19.0.1

EdmundSackbauer

I just got my box upgraded to 19.1-MR1, and ran my script against it and it ran without error.

Can you please share how this section of the script is setup for you please:

$update_certs = array("a.b.c","d.e.f"); //names of the certificates on XG that you are wanting to update
$xg_certificate_files = array("fullchain.pem","fullchain.pem"); //names of your cert files to be used in xg
$xg_certificate_keys = array("privkey.key","privkey.key"); // name of your cert key files to be used in xg ( note it must end in .key )
$certificate_file_uploads = array("/etc/letsencrypt/live/a.b.c/fullchain.pem","/etc/letsencrypt/live/d.e.f/fullchain.pem"); //local locations of cert file to upload
$certificate_key_uploads = array("/etc/letsencrypt/live/a.b.c/privkey.pem","/etc/letsencrypt/live/d.e.f/privkey.pem"); //local locations of key file to upload ( filename does not have to be .key but the filename above does have to be )

Also, starting around line 155, please set the script to the following and re-run it, and see if you get results to spit out for each of your firewall rules:

*** Be sure to make a backup copy of your original script before making these temporary changes ***

        $xml = new SimpleXMLElement($xmlstring);

        //print $xml->asXML();
        foreach($xml->FirewallRule as $policy)
        {
                if(!isset($policy->HTTPBasedPolicy[0]->Certificate[0]))
                        continue;
                if($policy->HTTPBasedPolicy[0]->Certificate[0] == $update_cert)
                {
                        print $policy->asXML();
                }
        }
        continue;

        unset($rule_names);
        print "\e[35mFINDING POLICIES USING CERT: $update_cert\e[39m\n";   

Also, around line 228, add this new line as well, and see what results you get from it for the rule that isn't working:

 print "\e[35mAPPLYING TEMP CERT FOR POLICY RULE: ".$policy->Name[0]."\e[39m\n";
                                $policy->HTTPBasedPolicy[0]->Certificate[0] = $temp_cert;


                                $update_policy = '<Set operation="update">';
                                $update_policy .= $policy->asXML();
                                $update_policy .= '</Set>';
                                //print $update_policy."\n";
                                $xmlstring = xml_curl($xg_ip,$username,$password,$update_policy,$cert_uploads);
//***********************ADD THIS NEW LINE BELOW
                                print "\n\nXML RETURN STRING: ".$xmlstring."\n\n";
                                $xg_result = new SimpleXMLElement($xmlstring);    

Here is my config section:

$update_certs = array("owncloud");
$xg_certificate_files = array("fullchain.pem");
$xg_certificate_keys = array("privkey.key");
$certificate_file_uploads = array("/etc/letsencrypt/live/owncloud.xxxx.net/fullchain.pem");
$certificate_key_uploads  = array("/etc/letsencrypt/live/owncloud.xxxx.net/privkey.pem");

And this is the output after running the changed lines above:

root@app01:~# php test.php

 CREATING TEMP CERT...
<?xml version="1.0" encoding="UTF-8"?>
<Response APIVersion="1800.1" IPS_CAT_VER="1">
  <Login>
    <status>Authentication Successful</status>
  </Login>
  <Certificate transactionid="">
    <Status code="200">Configuration applied successfully.</Status>
  </Certificate>
</Response>



<FirewallRule transactionid="">
    <Name>owncloud</Name>
    <Description/>
    <IPFamily>IPv4</IPFamily>
    <Status>Enable</Status>
    <Position>After</Position>
    <PolicyType>HTTPBased</PolicyType>
    <After>
      <Name>status.xxxx.net</Name>
    </After>
    <HTTPBasedPolicy>
      <HostedAddress>#Port3</HostedAddress>
      <HTTPS>Enable</HTTPS>
      <ListenPort>443</ListenPort>
      <Domains>
        <Domain>owncloud.xxxx.net</Domain>
      </Domains>
      <AccessPaths>
  <AccessPath>
    <allowed_networks>Any IPv4</allowed_networks>
    <auth_profile/>
    <backend>owncloud</backend>
    <be_path/>
    <hot_standby>0</hot_standby>
    <path>/</path>
    <stickysession_status>0</stickysession_status>
    <websocket_passthrough>0</websocket_passthrough>
  </AccessPath>
</AccessPaths>
      <Exceptions>
</Exceptions>
      <ProtocolSecurity>Owncloud</ProtocolSecurity>
      <CompressionSupport>Disable</CompressionSupport>
      <RewriteHTML>0</RewriteHTML>
      <PassHostHeader>Disable</PassHostHeader>
      <RewriteCookies>Disable</RewriteCookies>
      <IntrusionPrevention>WAN TO LAN</IntrusionPrevention>
      <TrafficShapingPolicy>None</TrafficShapingPolicy>
      <Certificate>owncloud</Certificate>
      <RedirectHTTP>Enable</RedirectHTTP>
    </HTTPBasedPolicy>
  </FirewallRule><FirewallRule transactionid="">
    <Name>Clone_owncloud</Name>
    <Description/>
    <IPFamily>IPv4</IPFamily>
    <Status>Enable</Status>
    <Position>After</Position>
    <PolicyType>HTTPBased</PolicyType>
    <After>
      <Name>owncloud</Name>
    </After>
    <HTTPBasedPolicy>
      <HostedAddress>#Port2</HostedAddress>
      <HTTPS>Enable</HTTPS>
      <ListenPort>443</ListenPort>
      <Domains>
        <Domain>owncloud.xxxx.net</Domain>
      </Domains>
      <AccessPaths>
  <AccessPath>
    <allowed_networks>Any IPv4</allowed_networks>
    <auth_profile/>
    <backend>owncloud</backend>
    <be_path/>
    <hot_standby>0</hot_standby>
    <path>/</path>
    <stickysession_status>0</stickysession_status>
    <websocket_passthrough>0</websocket_passthrough>
  </AccessPath>
</AccessPaths>
      <Exceptions>
</Exceptions>
      <ProtocolSecurity>Owncloud</ProtocolSecurity>
      <CompressionSupport>Disable</CompressionSupport>
      <RewriteHTML>0</RewriteHTML>
      <PassHostHeader>Disable</PassHostHeader>
      <RewriteCookies>Disable</RewriteCookies>
      <IntrusionPrevention>WAN TO LAN</IntrusionPrevention>
      <TrafficShapingPolicy>None</TrafficShapingPolicy>
      <Certificate>owncloud</Certificate>
      <RedirectHTTP>Enable</RedirectHTTP>
    </HTTPBasedPolicy>
  </FirewallRule>





DELETING TEMP CERT...
<?xml version="1.0" encoding="UTF-8"?>
<Response APIVersion="1800.1" IPS_CAT_VER="1">
  <Login>
    <status>Authentication Successful</status>
  </Login>
  <Certificate transactionid="">
    <Status code="200">Configuration applied successfully.</Status>
  </Certificate>
</Response>


the XML RETURN STRING is empty.

But only on the second rule. This is a SAN certificate with 2 hostnames.
Nevertheless, on BOTH rules the temporary cert is set, it seems just the second result of the API is wrong (empty).

This is interesting.... I just mimicked your setup, and compared our policy output from the test i asked for, and nothing stands out differently, but mine still worked.  It's odd that the first policy works, and the second policy "works / applies the change to the temp" but comes back with no response.  

Lets try this next and see if you notice anything different from the policy that works "returns a result for applying the temp cert" and the one that does not......

Have this be the only changes for this test!

Around line 82, set it to the following:

                        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                        curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
                       $result = curl_exec($ch); // execute
// *** remove the "//" from each of the following 3 lines below ***
            var_dump($result);
            $info = curl_getinfo($ch);
            print_r($info);
// *** end changes ***
                        curl_close($ch);

I'm wondering if maybe the curl is timing out? or getting some sort of error...if so this will show us what is returned, and the detailed "curl ifno" from running the curl against the api.

This is the output for the working first policy:

Array
(
    [url] => https://10.0.0.254:4444/webconsole/APIController
    [content_type] => text/xml;  charset=UTF-8
    [http_code] => 200
    [header_size] => 392
    [request_size] => 209
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 51.260151
    [namelookup_time] => 1.8E-5
    [connect_time] => 0.000478
    [pretransfer_time] => 0.01046
    [size_upload] => 1695
    [size_download] => 286
    [speed_download] => 5
    [speed_upload] => 33
    [download_content_length] => -1
    [upload_content_length] => 1695
    [starttransfer_time] => 0.011947
    [redirect_time] => 0
    [redirect_url] =>
    [primary_ip] => 10.0.0.254
    [certinfo] => Array
        (
        )

    [primary_port] => 4444
    [local_ip] => 10.0.0.9
    [local_port] => 51450
    [http_version] => 2
    [protocol] => 2
    [ssl_verifyresult] => 0
    [scheme] => HTTPS
    [appconnect_time_us] => 10423
    [connect_time_us] => 478
    [namelookup_time_us] => 18
    [pretransfer_time_us] => 10460
    [redirect_time_us] => 0
    [starttransfer_time_us] => 11947
    [total_time_us] => 51260151
)

And this the output for the second policy just before the exception happens:

bool(false)
Array
(
    [url] => https://10.0.0.254:4444/webconsole/APIController
    [content_type] =>
    [http_code] => 100
    [header_size] => 25
    [request_size] => 209
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 60.001212
    [namelookup_time] => 1.7E-5
    [connect_time] => 0.000431
    [pretransfer_time] => 0.010152
    [size_upload] => 1686
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 28
    [download_content_length] => -1
    [upload_content_length] => 1686
    [starttransfer_time] => 0.011703
    [redirect_time] => 0
    [redirect_url] =>
    [primary_ip] => 10.0.0.254
    [certinfo] => Array
        (
        )

    [primary_port] => 4444
    [local_ip] => 10.0.0.9
    [local_port] => 51456
    [http_version] => 2
    [protocol] => 2
    [ssl_verifyresult] => 0
    [scheme] => HTTPS
    [appconnect_time_us] => 10124
    [connect_time_us] => 431
    [namelookup_time_us] => 17
    [pretransfer_time_us] => 10152
    [redirect_time_us] => 0
    [starttransfer_time_us] => 11703
    [total_time_us] => 60001212
)

Strangely the total_time_us of the failed policy update is more or less exact 60 seconds. Could that be the timeout if curl? If so how do I change it?

Seems CURLOPT_ACCEPTTIMEOUT_MS is the default 60000000 us = 60 seconds

Yep, looks like we found the issue! The curl api call has a hard-set timeout of 60 seconds, so we need to adjust that.

Odd that your responses take so long.  My response time for setting the temp cert is "[total_time] => 12.153309" 

Anyways, you will just need to increase this setting around line 66

curl_setopt($ch, CURLOPT_TIMEOUT, 60);

I would just go ahead and double yours to the following:

curl_setopt($ch, CURLOPT_TIMEOUT, 120);

Let me know if that fixes the issue.

Now its working! I changed the line where you set the timeout from 60 to 120 and it runs through!!

Well I have a lot of rules (>40) and those have a lot of path specific routings etc. It is also taking so much time via the GUI.

Great!  Glad we got it figured out!  Thanks for being willing to do all the testing, and report your findings here to help solve it!

LuCar Toni Any chance you can get the OP, with the entire script, updated to reflect the fix we applied here, to help prevent others who may come across the OP and use the code, from having the issue EdmundSackbauer had?  Also, you still have the link to my post for this script going to someone else's github page on this post "(+) [LetsEncrypt] How To in XG - Recommended Reads - Sophos Firewall - Sophos Community"  Thanks!