Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Sophos Firewall: Creating XML Objects with notepad++ for mass import

Disclaimer: This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment.


Overview

This Recommended Read discusses how to mass import objects using XML objects in the Sophos Firewall.

As some of you might want to import multiple objects at once to the Sophos Firewall, there are simple tools in the field to convert a list to an XML File.

Exporting the Configuration

For example, I will use Notepad++ & the find & replace function within Notepad++.

Based on several guides on the internet on how to do so, we’ll start with the XML import/export option.

The Sophos Firewall allows us to export and import different configuration options in the GUI. You’ll find this in Backup & firmware > Import export.

This is the starting point of our task. As we want to import "FQDN objects," for example, we’ll export the current FQDN objects. 

This can take some time to generate.

We’re doing this to get the syntax that we need to use for the import later. 

 Editing the XML file

The result should be a file called API-XXXXXXXXXXXXXX.tar. "tar" is another format of ZIP. https://en.wikipedia.org/wiki/Tar_(computing)

Using a ZIP tool (for example, 7zip), we can decompress this file and get an entities.xml file.

Open this .xml file with your Notepad++, and you’ll get the correct syntax, which we need to import this file with new data.

I'll now delete every entry except one to get the syntax correct.

Let's take a look at our "raw" data, which we want to import. 

I have prepared a list of 4 domains, but this could actually be as big as you like. 

Pressing STRG+F in Notepad++ will open the search / replace mode.

This is another useful guide on how to do that: https://www.launch2success.com/guide/advanced-find-and-replace-in-notepad/

I will now do the quick mode for you.

As your syntax dictates, an FQDN host needs to look like this: 

<FQDNHost transactionid="">
<Name>*.staticforce.com</Name>
<FQDN>*.staticforce.com</FQDN>
</FQDNHost>

In Notepad++, we can actually select each entry and replace it with something else. The cool part about Notepad++ is it can replace the entry and put the original content into the replaced content.

(German Screenshot):

It's important to select the regular expression. 

You’ll look for: ^(.+)$      <--- This will actually find each entry in each row. 

We’re going to replace it with the following: 

<FQDNHost transactionid="">
<Name>(\1)</Name>
<FQDN>(\1)</FQDN>
</FQDNHost>

It’s the same syntax as above. But each entry has a (\1).

Putting this into place, let's replace each entry in our test:

      

This code will now be placed into the Entities.xml we had earlier.

Quickly save this file and place it back to the original API.tar File. (I am using 7zip for this).

Replace the current Entities.xml. You might notice the change date will be the current time frame.

Importing the XML file

Back to the Sophos Firewall web admin. Now use this API.tar, which we updated, and "import" this file.

The import can take some time, depending on the size of your API File. 

After the import, let's quickly verify that everything is correct:


 

Perfect.

If you encounter an error, you can verify the import in the log file on the CLI.

Go to Advanced Shell > /log/apiparser.log. The apiparser.log will represent all imported objects. 

You can do this for all sorts of configurations.

Redo the same steps: Get the object's syntax, replace your raw data, and import this new data into the Sophos Firewall. 

Feel free to share your replacement Syntax.




Added TAG
[edited by: Erick Jan at 1:16 AM (GMT -8) on 14 Nov 2024]
Parents
  • Importing the huge Office 365 list with some modifications. The import fails after some time. But I can see in the web exceptions, that some elements have been imported. I hoped, the logfile would point me to the error but this is all just generic failures.

    Any hint how to get more information about which section or line of the import is failing?

    INFO      Oct 22 16:46:39 [17564]: Opcode response: status:200
    INFO      Oct 22 16:46:39 [17564]: Import for this component is done sucessfully!!!INFO      Oct 22 16:46:39 [17564]: End  SET Handler, Status : Success,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:39 [17564]: ENTITY 'WebFilterException' IMPORT Success
    INFO      Oct 22 16:46:39 [17564]: Start Set Handler,Component : WebFilterException
    ERROR     Oct 22 16:46:39 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException.
    WARNING   Oct 22 16:46:39 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'.
    ERROR     Oct 22 16:46:39 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:39 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file.
    ERROR     Oct 22 16:46:39 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:40 [17564]: Opcode response: status:500
    WARNING   Oct 22 16:46:40 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'.
    ERROR     Oct 22 16:46:40 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:40 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file.
    ERROR     Oct 22 16:46:40 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:41 [17564]: Opcode response: status:200
    INFO      Oct 22 16:46:41 [17564]: Import for this component is done sucessfully!!!INFO      Oct 22 16:46:41 [17564]: End  SET Handler, Status : Success,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:41 [17564]: ENTITY 'WebFilterException' IMPORT Success
    INFO      Oct 22 16:46:41 [17564]: Start Set Handler,Component : WebFilterException
    ERROR     Oct 22 16:46:41 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException.
    WARNING   Oct 22 16:46:41 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'.
    ERROR     Oct 22 16:46:41 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:41 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:42 [17564]: Opcode response: status:500
    WARNING   Oct 22 16:46:42 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'.
    ERROR     Oct 22 16:46:42 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:42 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:43 [17564]: Opcode response: status:200
    INFO      Oct 22 16:46:43 [17564]: Import for this component is done sucessfully!!!INFO      Oct 22 16:46:43 [17564]: End  SET Handler, Status : Success,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:43 [17564]: ENTITY 'WebFilterException' IMPORT Success
    INFO      Oct 22 16:46:43 [17564]: Start Set Handler,Component : WebFilterException
    ERROR     Oct 22 16:46:43 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException.
    WARNING   Oct 22 16:46:43 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'.
    ERROR     Oct 22 16:46:43 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:43 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file.
    ERROR     Oct 22 16:46:43 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:44 [17564]: Opcode response: status:500
    WARNING   Oct 22 16:46:44 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'.
    ERROR     Oct 22 16:46:44 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:44 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file.
    ERROR     Oct 22 16:46:44 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:45 [17564]: Opcode response: status:200
    INFO      Oct 22 16:46:45 [17564]: Import for this component is done sucessfully!!!INFO      Oct 22 16:46:45 [17564]: End  SET Handler, Status : Success,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:45 [17564]: ENTITY 'WebFilterException' IMPORT Success
    INFO      Oct 22 16:46:45 [17564]: Start Set Handler,Component : WebFilterException
    ERROR     Oct 22 16:46:45 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException.
    WARNING   Oct 22 16:46:45 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'.
    ERROR     Oct 22 16:46:45 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:45 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:46 [17564]: Opcode response: status:500
    WARNING   Oct 22 16:46:46 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'.
    ERROR     Oct 22 16:46:46 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:46 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:46 [17564]: Opcode response: status:500
    ERROR     Oct 22 16:46:46 [17564]: Opcode return status is neither 528 nor 200 for ImportSo Exiting.....
    INFO      Oct 22 16:46:46 [17564]: End  SET Handler, Status : Fail,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:46 [17564]: ENTITY 'WebFilterException' IMPORT Failed
    

    EEdit: i found the error in my xml - was a comma behind a destination IP. Anyway, it would be great to have such errors in the import log. It's hard to review a 1700 liner.

Reply
  • Importing the huge Office 365 list with some modifications. The import fails after some time. But I can see in the web exceptions, that some elements have been imported. I hoped, the logfile would point me to the error but this is all just generic failures.

    Any hint how to get more information about which section or line of the import is failing?

    INFO      Oct 22 16:46:39 [17564]: Opcode response: status:200
    INFO      Oct 22 16:46:39 [17564]: Import for this component is done sucessfully!!!INFO      Oct 22 16:46:39 [17564]: End  SET Handler, Status : Success,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:39 [17564]: ENTITY 'WebFilterException' IMPORT Success
    INFO      Oct 22 16:46:39 [17564]: Start Set Handler,Component : WebFilterException
    ERROR     Oct 22 16:46:39 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException.
    WARNING   Oct 22 16:46:39 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'.
    ERROR     Oct 22 16:46:39 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:39 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file.
    ERROR     Oct 22 16:46:39 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:40 [17564]: Opcode response: status:500
    WARNING   Oct 22 16:46:40 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'.
    ERROR     Oct 22 16:46:40 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:40 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file.
    ERROR     Oct 22 16:46:40 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:41 [17564]: Opcode response: status:200
    INFO      Oct 22 16:46:41 [17564]: Import for this component is done sucessfully!!!INFO      Oct 22 16:46:41 [17564]: End  SET Handler, Status : Success,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:41 [17564]: ENTITY 'WebFilterException' IMPORT Success
    INFO      Oct 22 16:46:41 [17564]: Start Set Handler,Component : WebFilterException
    ERROR     Oct 22 16:46:41 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException.
    WARNING   Oct 22 16:46:41 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'.
    ERROR     Oct 22 16:46:41 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:41 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:42 [17564]: Opcode response: status:500
    WARNING   Oct 22 16:46:42 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'.
    ERROR     Oct 22 16:46:42 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:42 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:43 [17564]: Opcode response: status:200
    INFO      Oct 22 16:46:43 [17564]: Import for this component is done sucessfully!!!INFO      Oct 22 16:46:43 [17564]: End  SET Handler, Status : Success,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:43 [17564]: ENTITY 'WebFilterException' IMPORT Success
    INFO      Oct 22 16:46:43 [17564]: Start Set Handler,Component : WebFilterException
    ERROR     Oct 22 16:46:43 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException.
    WARNING   Oct 22 16:46:43 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'.
    ERROR     Oct 22 16:46:43 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:43 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file.
    ERROR     Oct 22 16:46:43 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:44 [17564]: Opcode response: status:500
    WARNING   Oct 22 16:46:44 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'.
    ERROR     Oct 22 16:46:44 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:44 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file.
    ERROR     Oct 22 16:46:44 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:45 [17564]: Opcode response: status:200
    INFO      Oct 22 16:46:45 [17564]: Import for this component is done sucessfully!!!INFO      Oct 22 16:46:45 [17564]: End  SET Handler, Status : Success,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:45 [17564]: ENTITY 'WebFilterException' IMPORT Success
    INFO      Oct 22 16:46:45 [17564]: Start Set Handler,Component : WebFilterException
    ERROR     Oct 22 16:46:45 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException.
    WARNING   Oct 22 16:46:45 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'.
    ERROR     Oct 22 16:46:45 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:45 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:46 [17564]: Opcode response: status:500
    WARNING   Oct 22 16:46:46 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'.
    ERROR     Oct 22 16:46:46 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file.
    ERROR     Oct 22 16:46:46 [17564]: Flag setting for this opcode is 16.
    INFO      Oct 22 16:46:46 [17564]: Opcode response: status:500
    ERROR     Oct 22 16:46:46 [17564]: Opcode return status is neither 528 nor 200 for ImportSo Exiting.....
    INFO      Oct 22 16:46:46 [17564]: End  SET Handler, Status : Fail,  Component : WebFilterException, Transaction : , Operation : NONE.
    MESSAGE   Oct 22 16:46:46 [17564]: ENTITY 'WebFilterException' IMPORT Failed
    

    EEdit: i found the error in my xml - was a comma behind a destination IP. Anyway, it would be great to have such errors in the import log. It's hard to review a 1700 liner.

Children
  • In such cases, sometimes you need to cross reference this: You see the last object, which is failing: Your Webexception object. So you can go to the applog.log to the same timeframe and it should show you, why the import is failing. 

    __________________________________________________________________________________________________________________