PHP script for uploading Lets Encrypt certs is broken since 19.0 MR1

EdmundSackbauer

I have not gotten 19 MR1 as of yet, but I will make it a point to manually update to it so I can try and replicate / resolve this issue.  

In the meantime, can you try manually creating a temp cert via the gui, and then applying that temp cert to your rules and see if you get a similar error?  Thanks!

Yes, this is what I did in the meantime, I manually created a temp cert and applied it to the rules, deleted the old LE cert, uploaded the new one with the same name, applied the new cert to the rules. That worked manually without problem.

Basically a script gets executed in different steps. Do you know, which step fails? 

So to speak, on which step are we having a problem, is the new cert there but does not get replaced etc. 

Script works in creating the temp cert, and the first rule is properly changed to the temp certificate.

Even the second rule is changed to the temp certificate, however the return string from the API cannot be parsed properly.

You could check the /log/apiparser.log if there is any error as well. 

INFO      Aug 02 10:09:13Z [11248]: Start Login Handler,Component : Login
ERROR     Aug 02 10:09:13Z [11248]: Key:ISCrEntity is not found in RequestMap File for Login.
INFO      Aug 02 10:09:13Z [11248]: Mapping file for Login component is /_conf/csc/IOMappingFiles//1800.1/Login/Login.xml
ERROR     Aug 02 10:09:13Z [11248]: Flag setting for this opcode is 18.
INFO      Aug 02 10:09:14Z [11248]: Opcode response: status:200
INFO      Aug 02 10:09:14Z [11248]: Authentication Successful
INFO      Aug 02 10:09:14Z [11248]: Start Set Handler,Component : FirewallRule
ERROR     Aug 02 10:09:14Z [11248]: Key:ISCrEntity is not found in RequestMap File for FirewallRule.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempsourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempexceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempsourceid" to handle logicaloperator.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="sourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempexceptionid" to handle logicaloperator.
ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="exceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
ERROR     Aug 02 10:09:14Z [11248]: Flag setting for this opcode is 16.
INFO      Aug 02 10:10:20Z [11248]: Opcode response: status:200
INFO      Aug 02 10:10:20Z [11248]: End  SET Handler, Status : Success,  Component : FirewallRule, Transaction : , Operation : update.
MESSAGE   Aug 02 10:10:20Z [11248]: ENTITY 'FirewallRule' IMPORT Success
INFO      Aug 02 10:10:20Z [11248]: Command:/scripts/apiparser_generate_tar.sh /sdisk/api-1659434953040779.txt /sdisk/API-1659434953040779 /sdisk/APIXMLOutput/1659434952687.xml /sdisk/API-1659434953040779.tar /sdisk/API-1659434953040779.log 0 status:3
INFO      Aug 02 10:10:20Z [11248]: No need to create Tar file. Response file is /sdisk/APIXMLOutput/1659434952687.xmlSFVH_SO01_SFOS 19.0.1

EdmundSackbauer

I just got my box upgraded to 19.1-MR1, and ran my script against it and it ran without error.

Can you please share how this section of the script is setup for you please:

$update_certs = array("a.b.c","d.e.f"); //names of the certificates on XG that you are wanting to update
$xg_certificate_files = array("fullchain.pem","fullchain.pem"); //names of your cert files to be used in xg
$xg_certificate_keys = array("privkey.key","privkey.key"); // name of your cert key files to be used in xg ( note it must end in .key )
$certificate_file_uploads = array("/etc/letsencrypt/live/a.b.c/fullchain.pem","/etc/letsencrypt/live/d.e.f/fullchain.pem"); //local locations of cert file to upload
$certificate_key_uploads = array("/etc/letsencrypt/live/a.b.c/privkey.pem","/etc/letsencrypt/live/d.e.f/privkey.pem"); //local locations of key file to upload ( filename does not have to be .key but the filename above does have to be )

Also, starting around line 155, please set the script to the following and re-run it, and see if you get results to spit out for each of your firewall rules:

*** Be sure to make a backup copy of your original script before making these temporary changes ***

        $xml = new SimpleXMLElement($xmlstring);

        //print $xml->asXML();
        foreach($xml->FirewallRule as $policy)
        {
                if(!isset($policy->HTTPBasedPolicy[0]->Certificate[0]))
                        continue;
                if($policy->HTTPBasedPolicy[0]->Certificate[0] == $update_cert)
                {
                        print $policy->asXML();
                }
        }
        continue;

        unset($rule_names);
        print "\e[35mFINDING POLICIES USING CERT: $update_cert\e[39m\n";   

Also, around line 228, add this new line as well, and see what results you get from it for the rule that isn't working:

 print "\e[35mAPPLYING TEMP CERT FOR POLICY RULE: ".$policy->Name[0]."\e[39m\n";
                                $policy->HTTPBasedPolicy[0]->Certificate[0] = $temp_cert;


                                $update_policy = '<Set operation="update">';
                                $update_policy .= $policy->asXML();
                                $update_policy .= '</Set>';
                                //print $update_policy."\n";
                                $xmlstring = xml_curl($xg_ip,$username,$password,$update_policy,$cert_uploads);
//***********************ADD THIS NEW LINE BELOW
                                print "\n\nXML RETURN STRING: ".$xmlstring."\n\n";
                                $xg_result = new SimpleXMLElement($xmlstring);    

Here is my config section:

$update_certs = array("owncloud");
$xg_certificate_files = array("fullchain.pem");
$xg_certificate_keys = array("privkey.key");
$certificate_file_uploads = array("/etc/letsencrypt/live/owncloud.xxxx.net/fullchain.pem");
$certificate_key_uploads  = array("/etc/letsencrypt/live/owncloud.xxxx.net/privkey.pem");

And this is the output after running the changed lines above:

root@app01:~# php test.php

 CREATING TEMP CERT...
<?xml version="1.0" encoding="UTF-8"?>
<Response APIVersion="1800.1" IPS_CAT_VER="1">
  <Login>
    <status>Authentication Successful</status>
  </Login>
  <Certificate transactionid="">
    <Status code="200">Configuration applied successfully.</Status>
  </Certificate>
</Response>



<FirewallRule transactionid="">
    <Name>owncloud</Name>
    <Description/>
    <IPFamily>IPv4</IPFamily>
    <Status>Enable</Status>
    <Position>After</Position>
    <PolicyType>HTTPBased</PolicyType>
    <After>
      <Name>status.xxxx.net</Name>
    </After>
    <HTTPBasedPolicy>
      <HostedAddress>#Port3</HostedAddress>
      <HTTPS>Enable</HTTPS>
      <ListenPort>443</ListenPort>
      <Domains>
        <Domain>owncloud.xxxx.net</Domain>
      </Domains>
      <AccessPaths>
  <AccessPath>
    <allowed_networks>Any IPv4</allowed_networks>
    <auth_profile/>
    <backend>owncloud</backend>
    <be_path/>
    <hot_standby>0</hot_standby>
    <path>/</path>
    <stickysession_status>0</stickysession_status>
    <websocket_passthrough>0</websocket_passthrough>
  </AccessPath>
</AccessPaths>
      <Exceptions>
</Exceptions>
      <ProtocolSecurity>Owncloud</ProtocolSecurity>
      <CompressionSupport>Disable</CompressionSupport>
      <RewriteHTML>0</RewriteHTML>
      <PassHostHeader>Disable</PassHostHeader>
      <RewriteCookies>Disable</RewriteCookies>
      <IntrusionPrevention>WAN TO LAN</IntrusionPrevention>
      <TrafficShapingPolicy>None</TrafficShapingPolicy>
      <Certificate>owncloud</Certificate>
      <RedirectHTTP>Enable</RedirectHTTP>
    </HTTPBasedPolicy>
  </FirewallRule><FirewallRule transactionid="">
    <Name>Clone_owncloud</Name>
    <Description/>
    <IPFamily>IPv4</IPFamily>
    <Status>Enable</Status>
    <Position>After</Position>
    <PolicyType>HTTPBased</PolicyType>
    <After>
      <Name>owncloud</Name>
    </After>
    <HTTPBasedPolicy>
      <HostedAddress>#Port2</HostedAddress>
      <HTTPS>Enable</HTTPS>
      <ListenPort>443</ListenPort>
      <Domains>
        <Domain>owncloud.xxxx.net</Domain>
      </Domains>
      <AccessPaths>
  <AccessPath>
    <allowed_networks>Any IPv4</allowed_networks>
    <auth_profile/>
    <backend>owncloud</backend>
    <be_path/>
    <hot_standby>0</hot_standby>
    <path>/</path>
    <stickysession_status>0</stickysession_status>
    <websocket_passthrough>0</websocket_passthrough>
  </AccessPath>
</AccessPaths>
      <Exceptions>
</Exceptions>
      <ProtocolSecurity>Owncloud</ProtocolSecurity>
      <CompressionSupport>Disable</CompressionSupport>
      <RewriteHTML>0</RewriteHTML>
      <PassHostHeader>Disable</PassHostHeader>
      <RewriteCookies>Disable</RewriteCookies>
      <IntrusionPrevention>WAN TO LAN</IntrusionPrevention>
      <TrafficShapingPolicy>None</TrafficShapingPolicy>
      <Certificate>owncloud</Certificate>
      <RedirectHTTP>Enable</RedirectHTTP>
    </HTTPBasedPolicy>
  </FirewallRule>





DELETING TEMP CERT...
<?xml version="1.0" encoding="UTF-8"?>
<Response APIVersion="1800.1" IPS_CAT_VER="1">
  <Login>
    <status>Authentication Successful</status>
  </Login>
  <Certificate transactionid="">
    <Status code="200">Configuration applied successfully.</Status>
  </Certificate>
</Response>


the XML RETURN STRING is empty.

But only on the second rule. This is a SAN certificate with 2 hostnames.
Nevertheless, on BOTH rules the temporary cert is set, it seems just the second result of the API is wrong (empty).