Note: Please contact Sophos Professional Services if you require assistance with your specific environment.
Hi all,
This Recommended Read goes over how to mass import objects using XML objects in the Sophos Firewall
As some of you might want to import multiple objects at once to the XG Firewall, there are simple tools in the field to convert a list to an XML File.
For example, I will use Notepad++ & the find & replace function within Notepad++.
Based on several guides on the internet on how to do so, we will start with the XML import/export option.
The XG Firewall gives us the option to export and import different configuration options in the GUI. You will find this in Backup & firmware > Import export.
This is the starting point of our task. As we want to import "FQDN objects" for example, we will export the current FQDN objects.
This can take some time to generate.
We are doing this to get the syntax that we need to use for the import later.
The result should be a file called: API-XXXXXXXXXXXXXX.tar. "tar" is another format of ZIP. https://en.wikipedia.org/wiki/Tar_(computing)
Using a ZIP tool (for example 7zip), we can decompress this file and get an entities.xml file.
Open this .xml file with your Notepad++ and you will get the correct syntax, which we need to import this file with new data.
I'll now delete every entry except one to get the syntax correct.
Let's take a look at our "raw" data, which we want to import.
I have prepared a list of 4 domains but this could actually be as big as you like.
Pressing STRG+F in notepad++ will open the search / replace mode.
This is another useful guide on how to do that: https://www.launch2success.com/guide/advanced-find-and-replace-in-notepad/
I will now do the quick mode for you.
As your syntax dictates, an FQDN host needs to look like this:
<FQDNHost transactionid=""> <Name>*.staticforce.com</Name> <FQDN>*.staticforce.com</FQDN> </FQDNHost>
<FQDNHost transactionid="">
<Name>*.staticforce.com</Name>
<FQDN>*.staticforce.com</FQDN>
</FQDNHost>
In Notepad++, we can actually select each entry and replace it with something else. The cool part about Notepad++ is, it can replace the entry and put the original content into the replaced content.
(German Screenshot):
It's important to select the regular expression.
You will look for: ^(.+)$ <--- This will actually find each entry in each row.
And we are going to replace it with:
<FQDNHost transactionid=""> <Name>(\1)</Name> <FQDN>(\1)</FQDN> </FQDNHost>
<Name>(\1)</Name>
<FQDN>(\1)</FQDN>
It is the same syntax as above. But each entry has a (\1).
Putting this into place, let's replace each entry in our test:
This code will now be placed into the Entities.xml, which we had earlier.
Quickly save this file and place it back to the original API.tar File. (I am using 7zip for this).
Replace the current Entities.xml. You should notice, the change date should be the current time frame.
Back to the XG web admin. Now use this API.tar, which we updated and "import" this file.
The import can take some time, depending on the size of your API File.
After the import, let's quickly verify that everything is correct:
Perfect.
If you have an error, you can verify the import in the log file on the CLI.
Go to Advanced Shell > /log/apiparser.log. The apiparser.log will represent all imported objects.
You can do this for all sorts of configurations.
Simply redo the same steps as above: Get the syntax of the object, replace your raw data, and import this new data into the XG Firewall.
Feel free to share your replacement Syntax, if you have any.
Thanks for writing this HowTo! But this is still a pain in the *** to do this. I found my self using this XML converter together with CSV files a few times now.
Using this XML Template for IP Hosts.
<IPHost transactionid=""> <Name>##1##</Name> <IPFamily>IPv4</IPFamily> <HostType>Network</HostType> <IPAddress>##2##</IPAddress> <Subnet>##3##</Subnet> </IPHost>
Thanks for your Feedback. There are plenty tools out there.
I just wanted to give such a tool to the world, as this could actually make your IT daily business much easier (its not bounded to XML or what ever).
__________________________________________________________________________________________________________________
Importing the huge Office 365 list with some modifications. The import fails after some time. But I can see in the web exceptions, that some elements have been imported. I hoped, the logfile would point me to the error but this is all just generic failures.
Any hint how to get more information about which section or line of the import is failing?
INFO Oct 22 16:46:39 [17564]: Opcode response: status:200 INFO Oct 22 16:46:39 [17564]: Import for this component is done sucessfully!!!INFO Oct 22 16:46:39 [17564]: End SET Handler, Status : Success, Component : WebFilterException, Transaction : , Operation : NONE. MESSAGE Oct 22 16:46:39 [17564]: ENTITY 'WebFilterException' IMPORT Success INFO Oct 22 16:46:39 [17564]: Start Set Handler,Component : WebFilterException ERROR Oct 22 16:46:39 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException. WARNING Oct 22 16:46:39 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'. ERROR Oct 22 16:46:39 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file. ERROR Oct 22 16:46:39 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file. ERROR Oct 22 16:46:39 [17564]: Flag setting for this opcode is 16. INFO Oct 22 16:46:40 [17564]: Opcode response: status:500 WARNING Oct 22 16:46:40 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'. ERROR Oct 22 16:46:40 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file. ERROR Oct 22 16:46:40 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file. ERROR Oct 22 16:46:40 [17564]: Flag setting for this opcode is 16. INFO Oct 22 16:46:41 [17564]: Opcode response: status:200 INFO Oct 22 16:46:41 [17564]: Import for this component is done sucessfully!!!INFO Oct 22 16:46:41 [17564]: End SET Handler, Status : Success, Component : WebFilterException, Transaction : , Operation : NONE. MESSAGE Oct 22 16:46:41 [17564]: ENTITY 'WebFilterException' IMPORT Success INFO Oct 22 16:46:41 [17564]: Start Set Handler,Component : WebFilterException ERROR Oct 22 16:46:41 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException. WARNING Oct 22 16:46:41 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'. ERROR Oct 22 16:46:41 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file. ERROR Oct 22 16:46:41 [17564]: Flag setting for this opcode is 16. INFO Oct 22 16:46:42 [17564]: Opcode response: status:500 WARNING Oct 22 16:46:42 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'. ERROR Oct 22 16:46:42 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file. ERROR Oct 22 16:46:42 [17564]: Flag setting for this opcode is 16. INFO Oct 22 16:46:43 [17564]: Opcode response: status:200 INFO Oct 22 16:46:43 [17564]: Import for this component is done sucessfully!!!INFO Oct 22 16:46:43 [17564]: End SET Handler, Status : Success, Component : WebFilterException, Transaction : , Operation : NONE. MESSAGE Oct 22 16:46:43 [17564]: ENTITY 'WebFilterException' IMPORT Success INFO Oct 22 16:46:43 [17564]: Start Set Handler,Component : WebFilterException ERROR Oct 22 16:46:43 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException. WARNING Oct 22 16:46:43 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'. ERROR Oct 22 16:46:43 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file. ERROR Oct 22 16:46:43 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file. ERROR Oct 22 16:46:43 [17564]: Flag setting for this opcode is 16. INFO Oct 22 16:46:44 [17564]: Opcode response: status:500 WARNING Oct 22 16:46:44 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'. ERROR Oct 22 16:46:44 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file. ERROR Oct 22 16:46:44 [17564]: Parser Error: xmlvalue for jsonkey="dstiplist", xmlelement="/WebFilterException/DomainList/DstIp" cannot be found in request file. ERROR Oct 22 16:46:44 [17564]: Flag setting for this opcode is 16. INFO Oct 22 16:46:45 [17564]: Opcode response: status:200 INFO Oct 22 16:46:45 [17564]: Import for this component is done sucessfully!!!INFO Oct 22 16:46:45 [17564]: End SET Handler, Status : Success, Component : WebFilterException, Transaction : , Operation : NONE. MESSAGE Oct 22 16:46:45 [17564]: ENTITY 'WebFilterException' IMPORT Success INFO Oct 22 16:46:45 [17564]: Start Set Handler,Component : WebFilterException ERROR Oct 22 16:46:45 [17564]: Key:ISCrEntity is not found in RequestMap File for WebFilterException. WARNING Oct 22 16:46:45 [17564]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'. ERROR Oct 22 16:46:45 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file. ERROR Oct 22 16:46:45 [17564]: Flag setting for this opcode is 16. INFO Oct 22 16:46:46 [17564]: Opcode response: status:500 WARNING Oct 22 16:46:46 [17564]: Opcode failed with 'Add' operation. So call opcode with 'Update'. ERROR Oct 22 16:46:46 [17564]: Parser Error: xmlvalue for jsonkey="webcategoryid", xmlelement="/WebFilterException/DomainList/WebCategory" cannot be found in request file. ERROR Oct 22 16:46:46 [17564]: Flag setting for this opcode is 16. INFO Oct 22 16:46:46 [17564]: Opcode response: status:500 ERROR Oct 22 16:46:46 [17564]: Opcode return status is neither 528 nor 200 for ImportSo Exiting..... INFO Oct 22 16:46:46 [17564]: End SET Handler, Status : Fail, Component : WebFilterException, Transaction : , Operation : NONE. MESSAGE Oct 22 16:46:46 [17564]: ENTITY 'WebFilterException' IMPORT Failed
EEdit: i found the error in my xml - was a comma behind a destination IP. Anyway, it would be great to have such errors in the import log. It's hard to review a 1700 liner.
In such cases, sometimes you need to cross reference this: You see the last object, which is failing: Your Webexception object. So you can go to the applog.log to the same timeframe and it should show you, why the import is failing.
Can I do this for SecurityPolicy? This tag is for firewall section... The problem here is that all rules have sequential IDs, not to mention that XML associated to those rules have 2 tags called Position and After. Tag After shows the name of preceding rule... I think this set creates the sequence of all rules and if I remove one object, certainly the rule that is configured to be after the deleted rule will not find its location...
here:
<SecurityPolicy transactionid=""> <Name>DNS e PING</Name> <Description/> <IPFamily>IPv4</IPFamily> <Status>Enable</Status> <Position>After</Position> <PolicyType>Network</PolicyType> <After> <Name>Cameras_Pena</Name> </After> [ REMOVED UNNECESSARY TAGS ] </SecurityPolicy>
This method is more likely to create objects, which are "stateless" in that terms, that you can create 100 objects without any "dependencies". Host Objects, Exceptions etc. That is likely to be a use case for a mass import.
To actually create firewall rules, you need to have some more logic to it (Because a firewall has a lot of dependencies, like you describe).
The use case to create firewall rules would be more likely to interact directly with the API and push those changes to the API with a programming language in the first place. Something, where you can work with variables, like python or powershell, which makes this job a lot easier to begin with.
Hi,
i always get the error:
INFO Feb 07 10:36:26Z [8887]: Sanity check not required. And XML file is valid. xml: /sdisk/api-2023-02-07-10-36-25/Entities.xml. INFO Feb 07 10:36:26Z [8887]: Start Set Handler,Component : IPHost ERROR Feb 07 10:36:26Z [8887]: Key:ISCrEntity is not found in RequestMap File for IPHost. WARNING Feb 07 10:36:26Z [8887]: Can't get the <Add/Update> element from map file, So Mode value is 'Add'. ERROR Feb 07 10:36:26Z [8887]: Parser Error: xmlvalue for jsonkey="hostgroupid", xmlelement="/IPHost/HostGroupList/HostGroup" cannot be found in request file. ERROR Feb 07 10:36:26Z [8887]: Flag setting for this opcode is 16. INFO Feb 07 10:36:26Z [8887]: Opcode response: status:500 WARNING Feb 07 10:36:26Z [8887]: Opcode failed with 'Add' operation. So call opcode with 'Update'. ERROR Feb 07 10:36:26Z [8887]: Parser Error: xmlvalue for jsonkey="hostgroupid", xmlelement="/IPHost/HostGroupList/HostGroup" cannot be found in request file. ERROR Feb 07 10:36:26Z [8887]: Flag setting for this opcode is 16. INFO Feb 07 10:36:27Z [8887]: Opcode response: status:500 ERROR Feb 07 10:36:27Z [8887]: Opcode return status is neither 528 nor 200 for ImportSo Exiting..... INFO Feb 07 10:36:27Z [8887]: End SET Handler, Status : Fail, Component : IPHost, Transaction : , Operation : NONE. MESSAGE Feb 07 10:36:27Z [8887]: ENTITY 'IPHost' IMPORT Failed
but i cant find the problem, i edit the original exportet entities.xml but it seems that there is something missing within of the export (Line 5 shows the error)
the XML file contains, currently, only this:
<?xml version="1.0" encoding="UTF-8"?> <Configuration APIVersion="1905.1" IPS_CAT_VER="1"> <IPHost transactionid=""> <Name>Azure-Europe-01</Name> <IPFamily>IPv4</IPFamily> <HostType>Network</HostType> <IPAddress>4.207.0.0</IPAddress> <Subnet>255.555.0.0</Subnet> </IPHost> </Configuration>
But we want import the whole IP Address-Range of Azure from central Europe (~250 Entries)
Our current Version of the XG: SFOS 19.5.0 GA-Build197
Maybe someone know where the problem is
Edit / Add:
Thanks to LuCar Toni, Typo mistake in the subnet field "xxx.555...", my bad.
Your firewall claims, there is a hostgroup in the XML.
Can you send me via DMs the xml file?