Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

PHP script for uploading Lets Encrypt certs is broken since 19.0 MR1

Hi, I am using this script from user 

https://community.sophos.com/sophos-xg-firewall/f/discussions/129768/letsencrypt-api-update-script---dynamically-handles-multiple-certs-multiple-rules-including-re-grouping-of-policies-rules

However since MR1 it is broken when it tries to set the 2nd rule with the temporary certificate:
FINDING POLICIES USING CERT: owncloud
FOUND MATCHING POLICY RULE TO BE UPDATED: Clone_owncloud
FOUND MATCHING POLICY RULE TO BE UPDATED: owncloud
FINDING ASSIGNED GROUPS FOR RULE: Clone_owncloud
FOUND MATCHING FIREWALL RULE GROUP: Traffic to Internal Zones
FINDING ASSIGNED GROUPS FOR RULE: owncloud
FOUND MATCHING FIREWALL RULE GROUP: Traffic to Internal Zones



APPLYING TEMP CERT FOR POLICY RULE: Clone_owncloud
<?xml version="1.0" encoding="UTF-8"?>
<Response APIVersion="1800.1" IPS_CAT_VER="1">
  <Login>
    <status>Authentication Successful</status>
  </Login>
  <FirewallRule transactionid="">
    <Status code="200">Configuration applied successfully.</Status>
  </FirewallRule>
</Response>


APPLYING TEMP CERT FOR POLICY RULE: owncloud
PHP Fatal error:  Uncaught Exception: String could not be parsed as XML in /root/update_xg_cert_owncloud.php:226
Stack trace:
#0 /root/update_xg_cert_owncloud.php(226): SimpleXMLElement->__construct()
#1 {main}
  thrown in /root/update_xg_cert_owncloud.php on line 226

Any idea why that happens? I am completely noob in PHP and API.



This thread was automatically locked due to age.
Parents
  • I have not gotten 19 MR1 as of yet, but I will make it a point to manually update to it so I can try and replicate / resolve this issue.  

    In the meantime, can you try manually creating a temp cert via the gui, and then applying that temp cert to your rules and see if you get a similar error?  Thanks!

  • Yes, this is what I did in the meantime, I manually created a temp cert and applied it to the rules, deleted the old LE cert, uploaded the new one with the same name, applied the new cert to the rules. That worked manually without problem.

  • Basically a script gets executed in different steps. Do you know, which step fails? 

    So to speak, on which step are we having a problem, is the new cert there but does not get replaced etc. 

    __________________________________________________________________________________________________________________

  • Script works in creating the temp cert, and the first rule is properly changed to the temp certificate.

    Even the second rule is changed to the temp certificate, however the return string from the API cannot be parsed properly.

  • You could check the /log/apiparser.log if there is any error as well. 

    __________________________________________________________________________________________________________________

  • INFO      Aug 02 10:09:13Z [11248]: Start Login Handler,Component : Login
    ERROR     Aug 02 10:09:13Z [11248]: Key:ISCrEntity is not found in RequestMap File for Login.
    INFO      Aug 02 10:09:13Z [11248]: Mapping file for Login component is /_conf/csc/IOMappingFiles//1800.1/Login/Login.xml
    ERROR     Aug 02 10:09:13Z [11248]: Flag setting for this opcode is 18.
    INFO      Aug 02 10:09:14Z [11248]: Opcode response: status:200
    INFO      Aug 02 10:09:14Z [11248]: Authentication Successful
    INFO      Aug 02 10:09:14Z [11248]: Start Set Handler,Component : FirewallRule
    ERROR     Aug 02 10:09:14Z [11248]: Key:ISCrEntity is not found in RequestMap File for FirewallRule.
    ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempsourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
    ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempexceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
    ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempsourceid" to handle logicaloperator.
    ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="sourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
    ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempexceptionid" to handle logicaloperator.
    ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="exceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
    ERROR     Aug 02 10:09:14Z [11248]: Flag setting for this opcode is 16.
    INFO      Aug 02 10:10:20Z [11248]: Opcode response: status:200
    INFO      Aug 02 10:10:20Z [11248]: End  SET Handler, Status : Success,  Component : FirewallRule, Transaction : , Operation : update.
    MESSAGE   Aug 02 10:10:20Z [11248]: ENTITY 'FirewallRule' IMPORT Success
    INFO      Aug 02 10:10:20Z [11248]: Command:/scripts/apiparser_generate_tar.sh /sdisk/api-1659434953040779.txt /sdisk/API-1659434953040779 /sdisk/APIXMLOutput/1659434952687.xml /sdisk/API-1659434953040779.tar /sdisk/API-1659434953040779.log 0 status:3
    INFO      Aug 02 10:10:20Z [11248]: No need to create Tar file. Response file is /sdisk/APIXMLOutput/1659434952687.xmlSFVH_SO01_SFOS 19.0.1

Reply
  • INFO      Aug 02 10:09:13Z [11248]: Start Login Handler,Component : Login
    ERROR     Aug 02 10:09:13Z [11248]: Key:ISCrEntity is not found in RequestMap File for Login.
    INFO      Aug 02 10:09:13Z [11248]: Mapping file for Login component is /_conf/csc/IOMappingFiles//1800.1/Login/Login.xml
    ERROR     Aug 02 10:09:13Z [11248]: Flag setting for this opcode is 18.
    INFO      Aug 02 10:09:14Z [11248]: Opcode response: status:200
    INFO      Aug 02 10:09:14Z [11248]: Authentication Successful
    INFO      Aug 02 10:09:14Z [11248]: Start Set Handler,Component : FirewallRule
    ERROR     Aug 02 10:09:14Z [11248]: Key:ISCrEntity is not found in RequestMap File for FirewallRule.
    ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempsourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
    ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="tempexceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
    ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempsourceid" to handle logicaloperator.
    ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="sourceid", xmlelement="/FirewallRule/HTTPBasedPolicy/SourceNetworks/Network" cannot be found in request file.
    ERROR     Aug 02 10:09:14Z [11248]: json object not found with key="tempexceptionid" to handle logicaloperator.
    ERROR     Aug 02 10:09:14Z [11248]: Parser Error: xmlvalue for jsonkey="exceptionid", xmlelement="/FirewallRule/HTTPBasedPolicy/ExceptionNetworks/Network" cannot be found in request file.
    ERROR     Aug 02 10:09:14Z [11248]: Flag setting for this opcode is 16.
    INFO      Aug 02 10:10:20Z [11248]: Opcode response: status:200
    INFO      Aug 02 10:10:20Z [11248]: End  SET Handler, Status : Success,  Component : FirewallRule, Transaction : , Operation : update.
    MESSAGE   Aug 02 10:10:20Z [11248]: ENTITY 'FirewallRule' IMPORT Success
    INFO      Aug 02 10:10:20Z [11248]: Command:/scripts/apiparser_generate_tar.sh /sdisk/api-1659434953040779.txt /sdisk/API-1659434953040779 /sdisk/APIXMLOutput/1659434952687.xml /sdisk/API-1659434953040779.tar /sdisk/API-1659434953040779.log 0 status:3
    INFO      Aug 02 10:10:20Z [11248]: No need to create Tar file. Response file is /sdisk/APIXMLOutput/1659434952687.xmlSFVH_SO01_SFOS 19.0.1

Children
  • I just got my box upgraded to 19.1-MR1, and ran my script against it and it ran without error.

    Can you please share how this section of the script is setup for you please:

    $update_certs = array("a.b.c","d.e.f"); //names of the certificates on XG that you are wanting to update
    $xg_certificate_files = array("fullchain.pem","fullchain.pem"); //names of your cert files to be used in xg
    $xg_certificate_keys = array("privkey.key","privkey.key"); // name of your cert key files to be used in xg ( note it must end in .key )
    $certificate_file_uploads = array("/etc/letsencrypt/live/a.b.c/fullchain.pem","/etc/letsencrypt/live/d.e.f/fullchain.pem"); //local locations of cert file to upload
    $certificate_key_uploads = array("/etc/letsencrypt/live/a.b.c/privkey.pem","/etc/letsencrypt/live/d.e.f/privkey.pem"); //local locations of key file to upload ( filename does not have to be .key but the filename above does have to be )

    Also, starting around line 155, please set the script to the following and re-run it, and see if you get results to spit out for each of your firewall rules:

    *** Be sure to make a backup copy of your original script before making these temporary changes ***

            $xml = new SimpleXMLElement($xmlstring);
    
            //print $xml->asXML();
            foreach($xml->FirewallRule as $policy)
            {
                    if(!isset($policy->HTTPBasedPolicy[0]->Certificate[0]))
                            continue;
                    if($policy->HTTPBasedPolicy[0]->Certificate[0] == $update_cert)
                    {
                            print $policy->asXML();
                    }
            }
            continue;
    
            unset($rule_names);
            print "\e[35mFINDING POLICIES USING CERT: $update_cert\e[39m\n";   

  • Also, around line 228, add this new line as well, and see what results you get from it for the rule that isn't working:

     print "\e[35mAPPLYING TEMP CERT FOR POLICY RULE: ".$policy->Name[0]."\e[39m\n";
                                    $policy->HTTPBasedPolicy[0]->Certificate[0] = $temp_cert;
    
    
                                    $update_policy = '<Set operation="update">';
                                    $update_policy .= $policy->asXML();
                                    $update_policy .= '</Set>';
                                    //print $update_policy."\n";
                                    $xmlstring = xml_curl($xg_ip,$username,$password,$update_policy,$cert_uploads);
    //***********************ADD THIS NEW LINE BELOW
                                    print "\n\nXML RETURN STRING: ".$xmlstring."\n\n";
                                    $xg_result = new SimpleXMLElement($xmlstring);    

  • Here is my config section:

    $update_certs = array("owncloud");
    $xg_certificate_files = array("fullchain.pem");
    $xg_certificate_keys = array("privkey.key");
    $certificate_file_uploads = array("/etc/letsencrypt/live/owncloud.xxxx.net/fullchain.pem");
    $certificate_key_uploads  = array("/etc/letsencrypt/live/owncloud.xxxx.net/privkey.pem");

    And this is the output after running the changed lines above:

    root@app01:~# php test.php

     CREATING TEMP CERT...
    <?xml version="1.0" encoding="UTF-8"?>
    <Response APIVersion="1800.1" IPS_CAT_VER="1">
      <Login>
        <status>Authentication Successful</status>
      </Login>
      <Certificate transactionid="">
        <Status code="200">Configuration applied successfully.</Status>
      </Certificate>
    </Response>



    <FirewallRule transactionid="">
        <Name>owncloud</Name>
        <Description/>
        <IPFamily>IPv4</IPFamily>
        <Status>Enable</Status>
        <Position>After</Position>
        <PolicyType>HTTPBased</PolicyType>
        <After>
          <Name>status.xxxx.net</Name>
        </After>
        <HTTPBasedPolicy>
          <HostedAddress>#Port3</HostedAddress>
          <HTTPS>Enable</HTTPS>
          <ListenPort>443</ListenPort>
          <Domains>
            <Domain>owncloud.xxxx.net</Domain>
          </Domains>
          <AccessPaths>
      <AccessPath>
        <allowed_networks>Any IPv4</allowed_networks>
        <auth_profile/>
        <backend>owncloud</backend>
        <be_path/>
        <hot_standby>0</hot_standby>
        <path>/</path>
        <stickysession_status>0</stickysession_status>
        <websocket_passthrough>0</websocket_passthrough>
      </AccessPath>
    </AccessPaths>
          <Exceptions>
    </Exceptions>
          <ProtocolSecurity>Owncloud</ProtocolSecurity>
          <CompressionSupport>Disable</CompressionSupport>
          <RewriteHTML>0</RewriteHTML>
          <PassHostHeader>Disable</PassHostHeader>
          <RewriteCookies>Disable</RewriteCookies>
          <IntrusionPrevention>WAN TO LAN</IntrusionPrevention>
          <TrafficShapingPolicy>None</TrafficShapingPolicy>
          <Certificate>owncloud</Certificate>
          <RedirectHTTP>Enable</RedirectHTTP>
        </HTTPBasedPolicy>
      </FirewallRule><FirewallRule transactionid="">
        <Name>Clone_owncloud</Name>
        <Description/>
        <IPFamily>IPv4</IPFamily>
        <Status>Enable</Status>
        <Position>After</Position>
        <PolicyType>HTTPBased</PolicyType>
        <After>
          <Name>owncloud</Name>
        </After>
        <HTTPBasedPolicy>
          <HostedAddress>#Port2</HostedAddress>
          <HTTPS>Enable</HTTPS>
          <ListenPort>443</ListenPort>
          <Domains>
            <Domain>owncloud.xxxx.net</Domain>
          </Domains>
          <AccessPaths>
      <AccessPath>
        <allowed_networks>Any IPv4</allowed_networks>
        <auth_profile/>
        <backend>owncloud</backend>
        <be_path/>
        <hot_standby>0</hot_standby>
        <path>/</path>
        <stickysession_status>0</stickysession_status>
        <websocket_passthrough>0</websocket_passthrough>
      </AccessPath>
    </AccessPaths>
          <Exceptions>
    </Exceptions>
          <ProtocolSecurity>Owncloud</ProtocolSecurity>
          <CompressionSupport>Disable</CompressionSupport>
          <RewriteHTML>0</RewriteHTML>
          <PassHostHeader>Disable</PassHostHeader>
          <RewriteCookies>Disable</RewriteCookies>
          <IntrusionPrevention>WAN TO LAN</IntrusionPrevention>
          <TrafficShapingPolicy>None</TrafficShapingPolicy>
          <Certificate>owncloud</Certificate>
          <RedirectHTTP>Enable</RedirectHTTP>
        </HTTPBasedPolicy>
      </FirewallRule>





    DELETING TEMP CERT...
    <?xml version="1.0" encoding="UTF-8"?>
    <Response APIVersion="1800.1" IPS_CAT_VER="1">
      <Login>
        <status>Authentication Successful</status>
      </Login>
      <Certificate transactionid="">
        <Status code="200">Configuration applied successfully.</Status>
      </Certificate>
    </Response>

  • the XML RETURN STRING is empty.

    But only on the second rule. This is a SAN certificate with 2 hostnames.
    Nevertheless, on BOTH rules the temporary cert is set, it seems just the second result of the API is wrong (empty).

  • This is interesting.... I just mimicked your setup, and compared our policy output from the test i asked for, and nothing stands out differently, but mine still worked.  It's odd that the first policy works, and the second policy "works / applies the change to the temp" but comes back with no response.  

    Lets try this next and see if you notice anything different from the policy that works "returns a result for applying the temp cert" and the one that does not......

    Have this be the only changes for this test!

    Around line 82, set it to the following:

                            curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
                            curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
                           $result = curl_exec($ch); // execute
    // *** remove the "//" from each of the following 3 lines below ***
                var_dump($result);
                $info = curl_getinfo($ch);
                print_r($info);
    // *** end changes ***
                            curl_close($ch);

    I'm wondering if maybe the curl is timing out? or getting some sort of error...if so this will show us what is returned, and the detailed "curl ifno" from running the curl against the api.

  • This is the output for the working first policy:

    Array
    (
        [url] => https://10.0.0.254:4444/webconsole/APIController
        [content_type] => text/xml;  charset=UTF-8
        [http_code] => 200
        [header_size] => 392
        [request_size] => 209
        [filetime] => -1
        [ssl_verify_result] => 0
        [redirect_count] => 0
        [total_time] => 51.260151
        [namelookup_time] => 1.8E-5
        [connect_time] => 0.000478
        [pretransfer_time] => 0.01046
        [size_upload] => 1695
        [size_download] => 286
        [speed_download] => 5
        [speed_upload] => 33
        [download_content_length] => -1
        [upload_content_length] => 1695
        [starttransfer_time] => 0.011947
        [redirect_time] => 0
        [redirect_url] =>
        [primary_ip] => 10.0.0.254
        [certinfo] => Array
            (
            )
    
        [primary_port] => 4444
        [local_ip] => 10.0.0.9
        [local_port] => 51450
        [http_version] => 2
        [protocol] => 2
        [ssl_verifyresult] => 0
        [scheme] => HTTPS
        [appconnect_time_us] => 10423
        [connect_time_us] => 478
        [namelookup_time_us] => 18
        [pretransfer_time_us] => 10460
        [redirect_time_us] => 0
        [starttransfer_time_us] => 11947
        [total_time_us] => 51260151
    )
    

    And this the output for the second policy just before the exception happens:

    bool(false)
    Array
    (
        [url] => https://10.0.0.254:4444/webconsole/APIController
        [content_type] =>
        [http_code] => 100
        [header_size] => 25
        [request_size] => 209
        [filetime] => -1
        [ssl_verify_result] => 0
        [redirect_count] => 0
        [total_time] => 60.001212
        [namelookup_time] => 1.7E-5
        [connect_time] => 0.000431
        [pretransfer_time] => 0.010152
        [size_upload] => 1686
        [size_download] => 0
        [speed_download] => 0
        [speed_upload] => 28
        [download_content_length] => -1
        [upload_content_length] => 1686
        [starttransfer_time] => 0.011703
        [redirect_time] => 0
        [redirect_url] =>
        [primary_ip] => 10.0.0.254
        [certinfo] => Array
            (
            )
    
        [primary_port] => 4444
        [local_ip] => 10.0.0.9
        [local_port] => 51456
        [http_version] => 2
        [protocol] => 2
        [ssl_verifyresult] => 0
        [scheme] => HTTPS
        [appconnect_time_us] => 10124
        [connect_time_us] => 431
        [namelookup_time_us] => 17
        [pretransfer_time_us] => 10152
        [redirect_time_us] => 0
        [starttransfer_time_us] => 11703
        [total_time_us] => 60001212
    )
    

    Strangely the total_time_us of the failed policy update is more or less exact 60 seconds. Could that be the timeout if curl? If so how do I change it?

    Seems CURLOPT_ACCEPTTIMEOUT_MS is the default 60000000 us = 60 seconds

  • Yep, looks like we found the issue! The curl api call has a hard-set timeout of 60 seconds, so we need to adjust that.

    Odd that your responses take so long.  My response time for setting the temp cert is "[total_time] => 12.153309" 

    Anyways, you will just need to increase this setting around line 66

    curl_setopt($ch, CURLOPT_TIMEOUT, 60);

    I would just go ahead and double yours to the following:

    curl_setopt($ch, CURLOPT_TIMEOUT, 120);

    Let me know if that fixes the issue.

  • Now its working! I changed the line where you set the timeout from 60 to 120 and it runs through!!

    Well I have a lot of rules (>40) and those have a lot of path specific routings etc. It is also taking so much time via the GUI.

  • Great!  Glad we got it figured out!  Thanks for being willing to do all the testing, and report your findings here to help solve it!

    Any chance you can get the OP, with the entire script, updated to reflect the fix we applied here, to help prevent others who may come across the OP and use the code, from having the issue EdmundSackbauer had?  Also, you still have the link to my post for this script going to someone else's github page on this post "(+) [LetsEncrypt] How To in XG - Recommended Reads - Sophos Firewall - Sophos Community"  Thanks!