This discussion has been locked.

You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Office365 deployment best practice

We are in the process of migrating to Office365. As part of the process, the networked is evaluated and the first recommendation by Microsoft is to remove any proxies from the path between the user and Office365. The problem with this is that MS has a ton of IP Address ranges and URLs.

The primary guidelines are:

Use a proxy PAC files to send all the MS URLs direct.
Create policies on the firewall to allow all IP ranges and URLs

https://support.office.com/en-us/article/managing-office-365-endpoints-99cab9d4-ef59-4207-9f2b-3728eb46bf9a?ui=en-US&rs=en-US&ad=US

Really good overview of their philosophy from Ignite:

https://www.youtube.com/watch?v=19a8s90HboQ&feature=youtu.be

Here is the entire IP/URL List in XML format: https://support.content.office.net/en-us/static/O365IPAddresses.xml

The problem I see is managing the list of IP Addresses and URLs. The list is long and changes somewhat frequently, so it's not just a matter of doing it once, you have to maintain it. As far as I know, there is no Network object in the UTM that let's you drop a list of subnets. That wouldn't be bad. But it appears that each subnet has to be created as a network definition and them maybe added to a group. But some places in Sophos do not accept groups, so then each subnet would have to be dragged one at a time in the interface. Again tedious to implement and more tedious to maintain.

I could use the API, but that would have to be run against each UTM. This will take a bit of work to implement, but may be the best solution long term.

Has anyone discovered an easy solution to keeping this type of thing up to date?

This thread was automatically locked due to age.

Parents

0 TimBoggs over 4 years ago

Next question, after downloading the MS IP addresses, the total count is 659 subnets. Can the Sophos handle that many network objects? Sometimes it struggles enumerating network definitions and I only have 450 now.
Cancel
Vote Up 0 Vote Down

Cancel
0 TimBoggs over 4 years ago in reply to TimBoggs

Done. Updated all three UTM devices using PowerShell and the Sophos API. I am relatively new to PowerShell, so this was quite an exercise, I hope this might help others map the Sophos API examples to PowerShell commands.

# 2018-03-17 TJB

<#
This script downloads the xml IP list from Microsoft, adds/removes Sophos Network Objects and creates/modifies a Group of Networks.
Requirements:
    Sophos API Enabled
    Local Sophos account with admin privelages configures with an API Key

Input Parameter: $UTM
    Script expects a hashtable to be passed in the following format:
        @{UTM = 'SophosDeviceName';
          URL = 'SophosDeviceName.company.com:4444/api;
          KEY = 'apiKeyFromSophosDevice';
          DBG = $true/$false - activates pause in script}
#>
Param([parameter(Mandatory=$true)]$utm)

$comment = 'Microsoft IPv4 | ' + (Get-Date).ToString("yyyy-MM-dd") + ' PS1'
$apiURL = $utm.URL
$networkURI = $apiURL + '/objects/network/network/'
$groupURI = $apiURL + '/objects/network/group/'
$resultCSV = '\\server\folder\resultList.csv'
$msGroup = @()    #Sophos Network Group that will contain all of the network object created
$utmNetList = @() #All Networks retrieved from the UTM
$msNetList = @() #UTM Networks filtered for MS-
$ipList = @()     #IP Addresses downloaded from Microsoft XML
$ipNetList = @() #IP Addresses parsed into Network format
$resultList = @() #Dispositon of networks for notification
$emailHeader = @{smtpserver = 'mail';
                 subject = 'Microosft IPv4 Address Update for ' + $utm.UTM;
                 to = 'user1@company.com';
                 from = 'user2@company.com'}



#Must have corresponding account configured with token on UTM
$token = $utm.KEY
$tokenBase64 = [Convert]::ToBase64String([System.Text.Encoding]::Default.GetBytes("token:" + $token))

#Common headers required by Sophos API
$headers = @{}
$headers.add("Authorization",'Basic ' + $tokenBase64)
$headers.add("Content-Type", "application/json")
$headers.add("Accept", "application/json")

#Sets the TLS level to match Sophos
$AllProtocols = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols

#Retrieve the existing list of MS subnets from Sophos
$utmNetList = Invoke-RestMethod -Uri $networkURI -Method Get -Headers $headers
$msNetList = $utmNetList | where {$_.name -like 'MS-*/*' -and $_.comment -like 'Microsoft*PS1'}

# Jordan's code to retrieve and parse the Microsoft XML IP List
[xml]$xml = (New-Object System.Net.WebClient).DownloadString("support.content.office.net/.../O365IPAddresses.xml");

$xml.products.product | % {
    $_.addresslist | ? {$_.type -like 'IPv4' } | Select -Expand address -EA SilentlyContinue | Sort -Unique | % {
        $ipList += $_
        }
}
$ipList = $ipList | sort -Unique

#Format IP List into Network hashtable
foreach ($ip in $ipList)
{
    $net = $ip.split("/")
    $ipAddress = $net[0]
    $netmask = $net[1]
    $name = 'MS-' + $ip
    $subnet = @{address = $ipAddress;
                address6 = "";
                comment = $comment;
                interface = "";
                name = $name;
                netmask = $netmask;
                netmask6 = "0";
                resolved = $true;
                resolved6 = $false}
    $subnet = $subnet | ConvertTo-Json | ConvertFrom-Json
    $ipNetList += $subnet
}

# Add new subnets to Sophos UTM
foreach ($ipNet in $ipNetList)
{
    $action = ""
    $ref = ""
    $result = @()
    if ($msNetList -match $ipNet.name)
    {
        Write-Host $ipNet.name already exists
        $action = 'Exists'
        $ref = ($msNetList | where {$_.name -eq $ipNet.name})._ref
    }
    else
    {
        Write-Host Need to create network object $ipNet.name on Sophos UTM
        $result = Invoke-RestMethod -Uri $networkURI -Method Post -Headers $headers -Body (ConvertTo-Json $ipNet)
        if ($result.name -eq $ipNet.name)
        {
            Write-Host Subnet created successfully
            $action = 'Added'
            $ref = $result._ref
        }
        else
        {
            $action = 'AddFailed'
        }
    }
    $resultList += @{action = $action; network = $ipNet.name; _ref = $ref}
    #if ($utm.DBG){pause}
}

# Old Subnets to remove from Sophos UTM
foreach ($msNet in $msNetList)
{
    $action = ""
    if ($ipNetList -match $msNet.name)
    {
        Write-Host $msNet.name is still valid
    }
    else
    {
        Write-Host Need to remove network object $msNet.name from Sophos UTM
        $action = "Remove"
        $resultList += @{action = $action; network = $msNet.name; _ref = $msNet._ref}
    }
    #if ($utm.DBG){pause}
}

#Update "Microsoft IPv4 Subnets" group if any subnets added or removed
if (($resultList | where {$_.action -eq "Added"}).count -gt 0 -or ($resultList | where {$_.action -eq "Remove"}).count -gt 0)
{
    $result = @()
    $msGrpMembers = @()
    $grpBody = @{}
    $networkGroups = Invoke-RestMethod -Uri $groupURI -Method Get -Headers $headers
    $msGroup = $networkGroups | where {$_.name -eq 'Microsoft IPv4 Subnets'}
    if ($msGroup.name -ne 'Microsoft IPv4 Subnets')
    {
        $grpMethod = 'Post'
        $msGroupUri = $groupURI
    }
    else
    {
        $grpMethod = 'Patch'
        $msGroupUri = $groupURI + $msGroup._ref
    }
    foreach ($result in $resultList)
    {
        if ($result.action -ne 'Remove')
        {
            $msGrpMembers += $result._ref
        }
    }
    $grpBody = @{comment = $comment;
                 name = "Microsoft IPv4 Subnets";
                 members = $msGrpMembers}

    Invoke-RestMethod -Uri $msGroupUri -Method $grpMethod -Headers $headers -Body (ConvertTo-Json $grpBody)
    $resultList.ForEach({[PSCustomObject]$_}) | Export-Csv $resultCSV -Force -NoTypeInformation
    $emailHeader.Add('Body', 'Result file attached')
    $emailHeader.Add('Attachments', $resultCSV)
    #if($utm.DBG){pause}
}
else
{
    $emailHeader.Add('Body','No IP address changes made this run')
}

#Remove unused subnets from Sophos UTM
if (($resultList | where {$_.action -eq "Remove"}).count -gt 0)
{
    $result = @()
    $msGrpMembers = @()
    $grpBody = @{}
    $headers.add("X-Restd-Err-Ack", "all")
    foreach ($result in $resultList)
    {
        if ($result.action -eq 'Remove')
        {
            Write-Host Deleting Network object $result.network
            $delNetURI = $networkURI + $result._ref
            Invoke-RestMethod -Uri $delNetURI -Method Delete -Headers $headers -Body (ConvertTo-Json $ipNet)
        }
    }
}

#Sends e-mail with list of ranges and action taken
Send-MailMessage @emailHeader
Cancel
Vote Up 0 Vote Down

Cancel

Reply

0 TimBoggs over 4 years ago in reply to TimBoggs

Done. Updated all three UTM devices using PowerShell and the Sophos API. I am relatively new to PowerShell, so this was quite an exercise, I hope this might help others map the Sophos API examples to PowerShell commands.

# 2018-03-17 TJB

<#
This script downloads the xml IP list from Microsoft, adds/removes Sophos Network Objects and creates/modifies a Group of Networks.
Requirements:
    Sophos API Enabled
    Local Sophos account with admin privelages configures with an API Key

Input Parameter: $UTM
    Script expects a hashtable to be passed in the following format:
        @{UTM = 'SophosDeviceName';
          URL = 'SophosDeviceName.company.com:4444/api;
          KEY = 'apiKeyFromSophosDevice';
          DBG = $true/$false - activates pause in script}
#>
Param([parameter(Mandatory=$true)]$utm)

$comment = 'Microsoft IPv4 | ' + (Get-Date).ToString("yyyy-MM-dd") + ' PS1'
$apiURL = $utm.URL
$networkURI = $apiURL + '/objects/network/network/'
$groupURI = $apiURL + '/objects/network/group/'
$resultCSV = '\\server\folder\resultList.csv'
$msGroup = @()    #Sophos Network Group that will contain all of the network object created
$utmNetList = @() #All Networks retrieved from the UTM
$msNetList = @() #UTM Networks filtered for MS-
$ipList = @()     #IP Addresses downloaded from Microsoft XML
$ipNetList = @() #IP Addresses parsed into Network format
$resultList = @() #Dispositon of networks for notification
$emailHeader = @{smtpserver = 'mail';
                 subject = 'Microosft IPv4 Address Update for ' + $utm.UTM;
                 to = 'user1@company.com';
                 from = 'user2@company.com'}



#Must have corresponding account configured with token on UTM
$token = $utm.KEY
$tokenBase64 = [Convert]::ToBase64String([System.Text.Encoding]::Default.GetBytes("token:" + $token))

#Common headers required by Sophos API
$headers = @{}
$headers.add("Authorization",'Basic ' + $tokenBase64)
$headers.add("Content-Type", "application/json")
$headers.add("Accept", "application/json")

#Sets the TLS level to match Sophos
$AllProtocols = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols

#Retrieve the existing list of MS subnets from Sophos
$utmNetList = Invoke-RestMethod -Uri $networkURI -Method Get -Headers $headers
$msNetList = $utmNetList | where {$_.name -like 'MS-*/*' -and $_.comment -like 'Microsoft*PS1'}

# Jordan's code to retrieve and parse the Microsoft XML IP List
[xml]$xml = (New-Object System.Net.WebClient).DownloadString("support.content.office.net/.../O365IPAddresses.xml");

$xml.products.product | % {
    $_.addresslist | ? {$_.type -like 'IPv4' } | Select -Expand address -EA SilentlyContinue | Sort -Unique | % {
        $ipList += $_
        }
}
$ipList = $ipList | sort -Unique

#Format IP List into Network hashtable
foreach ($ip in $ipList)
{
    $net = $ip.split("/")
    $ipAddress = $net[0]
    $netmask = $net[1]
    $name = 'MS-' + $ip
    $subnet = @{address = $ipAddress;
                address6 = "";
                comment = $comment;
                interface = "";
                name = $name;
                netmask = $netmask;
                netmask6 = "0";
                resolved = $true;
                resolved6 = $false}
    $subnet = $subnet | ConvertTo-Json | ConvertFrom-Json
    $ipNetList += $subnet
}

# Add new subnets to Sophos UTM
foreach ($ipNet in $ipNetList)
{
    $action = ""
    $ref = ""
    $result = @()
    if ($msNetList -match $ipNet.name)
    {
        Write-Host $ipNet.name already exists
        $action = 'Exists'
        $ref = ($msNetList | where {$_.name -eq $ipNet.name})._ref
    }
    else
    {
        Write-Host Need to create network object $ipNet.name on Sophos UTM
        $result = Invoke-RestMethod -Uri $networkURI -Method Post -Headers $headers -Body (ConvertTo-Json $ipNet)
        if ($result.name -eq $ipNet.name)
        {
            Write-Host Subnet created successfully
            $action = 'Added'
            $ref = $result._ref
        }
        else
        {
            $action = 'AddFailed'
        }
    }
    $resultList += @{action = $action; network = $ipNet.name; _ref = $ref}
    #if ($utm.DBG){pause}
}

# Old Subnets to remove from Sophos UTM
foreach ($msNet in $msNetList)
{
    $action = ""
    if ($ipNetList -match $msNet.name)
    {
        Write-Host $msNet.name is still valid
    }
    else
    {
        Write-Host Need to remove network object $msNet.name from Sophos UTM
        $action = "Remove"
        $resultList += @{action = $action; network = $msNet.name; _ref = $msNet._ref}
    }
    #if ($utm.DBG){pause}
}

#Update "Microsoft IPv4 Subnets" group if any subnets added or removed
if (($resultList | where {$_.action -eq "Added"}).count -gt 0 -or ($resultList | where {$_.action -eq "Remove"}).count -gt 0)
{
    $result = @()
    $msGrpMembers = @()
    $grpBody = @{}
    $networkGroups = Invoke-RestMethod -Uri $groupURI -Method Get -Headers $headers
    $msGroup = $networkGroups | where {$_.name -eq 'Microsoft IPv4 Subnets'}
    if ($msGroup.name -ne 'Microsoft IPv4 Subnets')
    {
        $grpMethod = 'Post'
        $msGroupUri = $groupURI
    }
    else
    {
        $grpMethod = 'Patch'
        $msGroupUri = $groupURI + $msGroup._ref
    }
    foreach ($result in $resultList)
    {
        if ($result.action -ne 'Remove')
        {
            $msGrpMembers += $result._ref
        }
    }
    $grpBody = @{comment = $comment;
                 name = "Microsoft IPv4 Subnets";
                 members = $msGrpMembers}

    Invoke-RestMethod -Uri $msGroupUri -Method $grpMethod -Headers $headers -Body (ConvertTo-Json $grpBody)
    $resultList.ForEach({[PSCustomObject]$_}) | Export-Csv $resultCSV -Force -NoTypeInformation
    $emailHeader.Add('Body', 'Result file attached')
    $emailHeader.Add('Attachments', $resultCSV)
    #if($utm.DBG){pause}
}
else
{
    $emailHeader.Add('Body','No IP address changes made this run')
}

#Remove unused subnets from Sophos UTM
if (($resultList | where {$_.action -eq "Remove"}).count -gt 0)
{
    $result = @()
    $msGrpMembers = @()
    $grpBody = @{}
    $headers.add("X-Restd-Err-Ack", "all")
    foreach ($result in $resultList)
    {
        if ($result.action -eq 'Remove')
        {
            Write-Host Deleting Network object $result.network
            $delNetURI = $networkURI + $result._ref
            Invoke-RestMethod -Uri $delNetURI -Method Delete -Headers $headers -Body (ConvertTo-Json $ipNet)
        }
    }
}

#Sends e-mail with list of ranges and action taken
Send-MailMessage @emailHeader
Cancel
Vote Up 0 Vote Down

Cancel

Children

0 TimBoggs over 4 years ago in reply to TimBoggs

I am calling the script above with this script. This script maintains the list of Sophos devices and they API Keys. Loops through the list calling other specified script. I plan to create a few other scripts to update other components and this way the API keys are maintained in a single script I can secure a bit more than usual.

<#
.SYNOPSIS
2018-03-17 TJB - Calls scripts for set of defined UTM Devices

.DESCRIPTION
Contains the list of UTM devices and their API keys. Keep safe.

.PARAMETER Script
Full path to the powershell script to be run against the list of UTM devices. Script must accept hashtable variable.

.PARAMETER debugPause
Whether or not to pause between servers. Can be passed along to the called script.
#>

Param([parameter(Mandatory=$true)]$Script,
      [Bool]$debugPause = $false)

$utmList = @(
    @{UTM = 'Sophos1';
      URL = 'sophos1.domain.com:4444/api';
      KEY = 'Sophos1RandomAPIKey';
      DBG = $debugPause},
    @{UTM = 'Sophos2';
      URL = 'sophos2.domain.com:4444/api';
      KEY = 'Sophos2RandomAPIKey';
      DBG = $debugPause},
    @{UTM = 'Sophos3';
      URL = 'sophos3.domain.com:4444/api';
      KEY = 'Sophos3RandomAPIKey';
      DBG = $debugPause})

foreach ($utm in $utmList)
{
    #Write-Host $utm.URL
    & $Script $utm
    if ($debugPause){pause}
}
Cancel
Vote Up 0 Vote Down

Cancel
0 BAlfson over 4 years ago in reply to TimBoggs

Thanks, Tim - this thread is a great contribution! I'll prioritize this at the top of this forum with "forever" selected.

Cheers - Bob

Sophos UTM Community Moderator
Sophos Certified Architect - UTM
Sophos Certified Engineer - XG
Gold Solution Partner since 2005

MediaSoft, Inc. USA
Cancel
Vote Up 0 Vote Down

Cancel
0 harrisonpensa over 3 years ago in reply to TimBoggs

I am about to figure this all out for my ASG320 UTM9 setup and our move to O365. I know very little about PS and using Sophos RestfulAPI but I am eager to try this. I am using transparent proxy and plan to add the resulting group(s) to the transparent destination skip list. I think this will do most of what is required. FYI, 80 and 443 are open outbound for everyone.

I believe the only other things that need to be done is to open the other required ports for SfB and try to limit those ports just to the group created by your scripts.

Does this make sense and is that what you ultimately did?
Cancel
Vote Up 0 Vote Down

Cancel
0 harrisonpensa over 3 years ago in reply to TimBoggs

Hey Tim, as I parse your script, it looks like this only does the IPv4 addresses and no URL's, IPv6 etc.

Has this worked well for you to date? Did you have to do any manual tweaking after the fact? Great work by the way. Certainly it must have been a challenge!
Cancel
Vote Up 0 Vote Down

Cancel
0 TimBoggs over 3 years ago in reply to harrisonpensa

I have run the script manually several times and it is updating fine. However, since I wrote the script MS has released an API interface for getting this data and I need to go back and refactor everything for the API. I started down that path and have already created a script to update the list of URL's in the Web Protection > Filtering Options > Exception list.

So, for Office365, need to create:

1. Network Definitions / Groups for IP based policies.

2. URL based exceptions for web filtering.

3. Proxy PAC file to tell browsers which way to send the traffic. Currently using UTM Manager for this, but if I can automate creation of the pac file for each site I might just do pure scripting. UTM manager has some frustrating limitations and lack of feature parity with the UTM interface.

The network definitions are the worst, creating 800 network objects really slows things down in the interface. Sophos really needs a network object that can be a collection of networks without requiring the creation of individual networks. And, the existing Network Group can't be used everywhere in the interface.
Cancel
Vote Up 0 Vote Down

Cancel
0 harrisonpensa over 3 years ago in reply to TimBoggs

It looks like Oct 2018 is when the XML will die and the web version will be the only way. I suppose it is best to just work towards the web version. I am pretty green at this so it may take me some considerable time and effort. If I come up with something I will share as you did. At the very least, your PS scripting shows how it needs to go to the UTM. I may play with grabbing the data from the web version, maybe use some other scripting and see if I can modify your script to deal with CSV or similar.

By the time I get it done, maybe Sophos will have a simple "enable O365" button that does it all for you.

Wish me luck. I have about 4 weeks to pull it off and two of those are vacation weeks.
Cancel
Vote Up 0 Vote Down

Cancel

0 TimBoggs over 3 years ago in reply to harrisonpensa

This might help? It's a bit of a merge of the MS sample and my script to update URL's for the exception list.

ms-ipRestApiShared.txt

# 2018-04-3 TJB

<#
This script downloads the xml IP list from Microsoft, adds/removes Sophos Network Objects and creates/modifies a Group of Networks.
Requirements:
    Sophos API Enabled
    Local Sophos account with admin privelages configures with an API Key

Input Parameter: $UTM
    Script expects a hashtable to be passed in the following format:
        @{UTM = 'SophosDeviceName';
          URL = 'https://SophosDeviceName.company.com:4444/api;
          KEY = 'apiKeyFromSophosDevice';
          DBG = $true/$false - activates pause in script}
#>
Param([parameter(Mandatory=$true)]$utm)

$ws = "https://endpoints.office.com" # webservice root URL
$datapath = "C:\scripts\sophos\endpoints_clientid_latestversion.txt" # path where client ID and latest version number will be stored
$exceptionUrlList = @() # Sophos Exception list
# Personalize the Tenant name.
$tenantName = 'tenantID' # API Tenant ID
$comment = 'Microsoft URL | ' + (Get-Date).ToString("yyyy-MM-dd") + ' PS1'
$apiURL = $utm.URL
$exceptionUri = $apiURL + '/objects/http/exception/'
$networkURI = $apiURL + '/objects/network/network/'
$groupURI = $apiURL + '/objects/network/group/'
$resultCSV = 'C:\scripts\sophos\resultList.csv'
$exceptionUrls = @()  # processed list of URLs for Sophos Formatting
$msGroup = @()    #Sophos Network Group that will contain all of the network object created
$utmNetList = @() #All Networks retrieved from the UTM
$msNetList = @()  #UTM Networks filtered for MS-
$ipList = @()     #IP Addresses downloaded from Microsoft XML
$ipNetList = @()  #IP Addresses parsed into Network format
$resultList = @() #Dispositon of networks for notification
# Personalize the mail server and recipient/sender information
$emailHeader = @{smtpserver = 'mailserver';
                 subject = 'Microosft IPv4 Address Update for ' + $utm.UTM;
                 to = 'user@company.com';
                 from = 'script@company.com'}

                 

#Must have corresponding account configured with token on UTM
$token = $utm.KEY
$tokenBase64 = [Convert]::ToBase64String([System.Text.Encoding]::Default.GetBytes("token:" + $token))

#Common headers required by Sophos API
$headers = @{}
$headers.add("Authorization",'Basic ' + $tokenBase64)
$headers.add("Content-Type", "application/json")
$headers.add("Accept", "application/json")

#Sets the TLS level to match Sophos
$AllProtocols = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols

#Retrieve the existing list of MS subnets from Sophos
$utmNetList = Invoke-RestMethod -Uri $networkURI -Method Get -Headers $headers
$msNetList = $utmNetList | where {$_.name -like 'MS-*/*' -and $_.comment -like 'Microsoft*PS1'}

#Retrieve the existing list of HTTP Exceptions from Sophos
$utmExceptionList = Invoke-RestMethod -Uri $exceptionUri -Method Get -Headers $headers
$msExceptionList = $utmExceptionList | where {$_.name -match 'Microsoft Office365 URLs' -and $_.comment -like 'Microsoft*PS1'}

# fetch client ID and version if data file exists; otherwise create new file
if (Test-Path $datapath) {
    $content = Get-Content $datapath
    $clientRequestId = $content[0]
    $lastVersion = $content[1]
}
else {
    $clientRequestId = [GUID]::NewGuid().Guid
    $lastVersion = "0000000000"
    @($clientRequestId, $lastVersion) | Out-File $datapath
}

# call version method to check the latest version, and pull new data if version number is different
$version = Invoke-RestMethod -Uri ($ws + "/version/O365Worldwide?clientRequestId=" + $clientRequestId)

if ($version.latest -ge $lastVersion) {
    Write-Host "New version of Office 365 worldwide commercial service instance endpoints detected"
    
    # write the new version number to the data file
    @($clientRequestId, $version.latest) | Out-File $datapath
    
    # invoke endpoints method to get the new data
    $endpointSets = Invoke-RestMethod -Uri ($ws + "/endpoints/O365Worldwide?clientRequestId=" + $clientRequestId + "&TenantName=" + $tenantName)

    # filter results for Allow and Optimize endpoints, and transform these into custom objects with port and category
    $flatUrls = $endpointSets | ForEach-Object {
        $endpointSet = $_
        $allowUrls = $(if ($endpointSet.allowUrls.Count -gt 0) { $endpointSet.allowUrls } else { @() })
        $optimizeUrls = $(if ($endpointSet.optimizeUrls.Count -gt 0) { $endpointSet.optimizeUrls } else { @() })
        
        $allowUrlCustomObjects = $allowUrls | ForEach-Object {
            [PSCustomObject]@{
                category = "Allow";
                url      = $_;
                # Allow URLs should permit traffic across both Allow and Optimize ports
                tcpPorts = (($endpointSet.allowTcpPorts, $endpointSet.optimizeTcpPorts) | Where-Object { $_ -ne $null }) -join ",";
                udpPorts = (($endpointSet.allowUdpPorts, $endpointSet.optimizeUdpPorts) | Where-Object { $_ -ne $null }) -join ",";
            }
        }
        $optimizeUrlCustomObjects = $optimizeUrls | ForEach-Object {
            [PSCustomObject]@{
                category = "Optimize";
                url      = $_;
                tcpPorts = $endpointSet.optimizeTcpPorts;
                udpPorts = $endpointSet.optimizeUdpPorts;
            }
        }
        $allowUrlCustomObjects, $optimizeUrlCustomObjects
    }

    $flatIps = $endpointSets | ForEach-Object {
        $endpointSet = $_
        $ips = $(if ($endpointSet.ips.Count -gt 0) { $endpointSet.ips } else { @() })
        # IPv4 strings have dots while IPv6 strings have colons
        $ip4s = $ips | Where-Object { $_ -like '*.*' }
        
        $allowIpCustomObjects = @()
        if ($endpointSet.allowTcpPorts -or $endpointSet.allowUdpPorts) {
            $allowIpCustomObjects = $ip4s | ForEach-Object {
                [PSCustomObject]@{
                    category = "Allow";
                    ip = $_;
                    tcpPorts = $endpointSet.allowTcpPorts;
                    udpPorts = $endpointSet.allowUdpPorts;
                }
            }
        }
        $optimizeIpCustomObjects = @()
        if ($endpointSet.optimizeTcpPorts -or $endpointSet.optimizeUdpPorts) {
            $optimizeIpCustomObjects = $ip4s | ForEach-Object {
                [PSCustomObject]@{
                    category = "Optimize";
                    ip       = $_;
                    tcpPorts = $endpointSet.optimizeTcpPorts;
                    udpPorts = $endpointSet.optimizeUdpPorts;
                }                
            }
        }
        $allowIpCustomObjects, $optimizeIpCustomObjects
    }
    
    Write-Output "IPV4 Firewall IP Address Ranges"
    ($flatIps.ip | Sort-Object -Unique) -join "," | Out-String

    Write-Output "URLs for Proxy Server"
    ($flatUrls.url | Sort-Object -Unique) -join "," | Out-String


    # Modifies the URLs for Sophos and populates the $exceptionUrls array
    foreach ($url in $flatUrls.url){
        $newUrl = ''
        if ($url -like '`*-*') {
            $newUrl = ($url).Replace('.','\.').Replace('*-','^https?://companyname-') # Replace with specific company name - might could use the TenantID variable
        } elseif ($url -like '`*.*'){
            $newUrl = ($url).Replace('.','\.').Replace('*\.','^https?://[^.]*\.')
        } else {
            $newUrl = '^https?://' + ($url -replace '\.','\.')        
        }
        $exceptionUrls = $exceptionUrls += $newUrl
    }

    if ($msExceptionList.name.count -eq 0){
        $exceptionMethod = 'Post'
        $msExceptionUri = $exceptionUri
    } elseif ($msExceptionList.name.count -eq 1){
        $exceptionMethod = 'Patch'
        $msExceptionUri = $exceptionUri + $msExceptionList._ref
    }


    <#Options for the skiplist
        av                     -- anti-virus
        cache                  -- Caching
        certcheck              -- Certificate Trust Check
        certdate               -- Certificate Date Check
        check_max_download     -- Block by download size
        content_removal        -- Content removal
        contenttype_blacklist  -- MIME type blocking
        extensions             -- Extension blocking
        log_access             -- Logging Accessed pages
        log_blocked            -- Logging Blocked pages
        patience               -- Do not display Download/Scan progress page
        ssl_scanning           -- SSL Scanning
        url_filter             -- URL Filter
        user_auth              -- Authentication
    #>
    $skipList = @('av',
                  'content_removal',
                  'user_auth',
                  'extensions',
                  'url_filter',
                  'contenttype_blacklist',
                  'cache',
                  'log_access',
                  'log_blocked',
                  'ssl_scanning',
                  'check_max_download',
                  'patience'
                 )

    #$msUrlList = ConvertTo-Json $exceptionUrls

    $exceptionBody = @{'aaa' = @();
                       'comment' = $comment;
                       'domains' = $exceptionUrls;
                       'endpoints_groups' = @();
                       'name' = 'Microsoft Office365 URLs';
                       'networks' = @();
                       'operator' = 'AND';
                       'skiplist' = $skipList;
                       'sp_categories' = @();
                       'status' = $true;
                       'tags' = @();
                       'user_agents' = @()
                      }


    Invoke-RestMethod -Uri $msExceptionUri -Method $exceptionMethod -Headers $headers -Body (ConvertTo-Json $exceptionBody)    # TODO Call Send-MailMessage with new endpoints data
}
else {
    Write-Host "Office 365 worldwide commercial service instance endpoints are up-to-date"
}

0 HansSchenkelaars over 3 years ago in reply to TimBoggs

Hi Tim,

I'm struggling with your script,

Where to put the following things

        UTM = '10.8.4.1';

        URL = '10.8.4.1:4444/api;

       KEY = 'LeqBiAUukVMlZgfdJsoHKwuLHRybVYZs';

When I start yout script it asks : Supply values for the following parameters: utm

So I put in

10.8.4.1:4444/api

10.8.4.1:4444

10.8.4.1

https://10.8.4.1:4444

Nothing works. I get a message like.

PS C:\WINDOWS\system32> #Retrieve the existing list of MS subnets from Sophos

PS C:\WINDOWS\system32> $utmNetList = Invoke-RestMethod -Uri $networkURI -Method Get -Headers $headers

Invoke-RestMethod : Invalid URI: The hostname could not be parsed.

At line:1 char:15

+ ... tmNetList = Invoke-RestMethod -Uri $networkURI -Method Get -Headers $ ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], UriFormatException

    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

PS C:\WINDOWS\system32> $msNetList = $utmNetList | where {$_.name -like 'MS-*/*' -and $_.comment -like 'Microsoft*PS1'}

Hope you can give me some guidance.

Best Regards,

Hans Schenkelaars
Cancel
Vote Up 0 Vote Down

Cancel
0 TimBoggs over 3 years ago in reply to HansSchenkelaars

You have to define the values as a hashtable then pass it to the script
$utm = @{UTM = 'SophosDeviceName';
   URL = 'SophosDeviceName.company.com:4444/api;
KEY = 'apiKeyFromSophosDevice'}

I designed this to have a calling script with an array of hashtables for all of my UTM's so I am only storing my api keys in one file. There are other scripts I run to update various UTM configurations.

Param([parameter(Mandatory=$true)]$script)

$utmList = @(
    @{UTM = 'Sophos1';
      URL = 'sophos1.corp.com:4444/api'
      KEY = 'apikey1'},
    @{UTM = 'Sophos2';
      URL = 'sophos2.corp.com:4444/api'
      KEY = 'apikey2'},
    @{UTM = 'Sophos3';
      URL = 'sophos3.corp.com:4444/api'
      KEY = 'apikey3'})

foreach ($utm in $utmList)
{
    #Write-Host $utm.URL
    & $script $utm
}

You would then call it with:

masterscript.ps1 -script runscript.ps1
Cancel
Vote Up 0 Vote Down

Cancel
0 HansSchenkelaars over 3 years ago in reply to TimBoggs

Hi Tim,

Thx for your quick response..

I created the 2 files masterscript.ps1 and runscript.ps1. Had to make some changes to PowerShell to allow running scripts but that works.\

When running the script I got an error regarding plain http traffic, so I changed the url in the masterscript to https://10.8.4.1:4444

.\masterscript.ps1 -script .\runscript.ps1
Invoke-RestMethod : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At D:\runscript.ps1:59 char:15
+ ... tmNetList = Invoke-RestMethod -Uri $networkURI -Method Get -Headers $ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At D:\runscript.ps1:63 char:21
+ ... ptionList = Invoke-RestMethod -Uri $exceptionUri -Method Get -Headers ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Out-File : Could not find a part of the path 'C:\scripts\sophos\endpoints_clientid_latestversion.txt'.
At D:\runscript.ps1:75 char:41
+ @($clientRequestId, $lastVersion) | Out-File $datapath
+ ~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OpenError: (:) [Out-File], DirectoryNotFoundException
+ FullyQualifiedErrorId : FileOpenFailure,Microsoft.PowerShell.Commands.OutFileCommand

Invoke-RestMethod : Invalid Instance string specified or ClientRequestId not specified in the Url.
At D:\runscript.ps1:79 char:12
+ $version = Invoke-RestMethod -Uri ($ws + "/version/O365Worldwide?clie ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
Office 365 worldwide commercial service instance endpoints are up-to-date

After Changing This I get an error regarding SSL/TLS.

.\masterscript.ps1 -script .\runscript.ps1
Invoke-RestMethod : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At D:\runscript.ps1:59 char:15
+ ... tmNetList = Invoke-RestMethod -Uri $networkURI -Method Get -Headers $ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At D:\runscript.ps1:63 char:21
+ ... ptionList = Invoke-RestMethod -Uri $exceptionUri -Method Get -Headers ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Out-File : Could not find a part of the path 'C:\scripts\sophos\endpoints_clientid_latestversion.txt'.
At D:\runscript.ps1:75 char:41
+ @($clientRequestId, $lastVersion) | Out-File $datapath
+ ~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OpenError: (:) [Out-File], DirectoryNotFoundException
+ FullyQualifiedErrorId : FileOpenFailure,Microsoft.PowerShell.Commands.OutFileCommand

Invoke-RestMethod : Invalid Instance string specified or ClientRequestId not specified in the Url.
At D:\runscript.ps1:79 char:12
+ $version = Invoke-RestMethod -Uri ($ws + "/version/O365Worldwide?clie ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
Office 365 worldwide commercial service instance endpoints are up-to-date

Maybe it's a small setting because I used both of your examples and I'm not sure what to change. Do you have an idea ?

We really need this to work hope you can help.. :-)

My Current Masterscript.ps1

Param([parameter(Mandatory=$true)]$script)
$utmList = @(
@{UTM = '10.8.4.1';
URL = 'https://10.8.4.1:4444/api'
KEY = 'LeqBiAUukVMlZguiJsoHKwuLHRybVYZs'})
foreach ($utm in $utmList)
{
#Write-Host $utm.URL
& $script $utm
}

My Current Runscript.ps1

# 2018-04-3 TJB

<#
This script downloads the xml IP list from Microsoft, adds/removes Sophos Network Objects and creates/modifies a Group of Networks.
Requirements:
Sophos API Enabled
Local Sophos account with admin privelages configures with an API Key

Input Parameter: $UTM
Script expects a hashtable to be passed in the following format:
@{UTM = '10.8.4.1';
URL = '10.8.4.1:4444/api;
KEY = 'LeqBiAUukVMlZguiJsoHKwuLHRybVYZs';
DBG = $true/$false - activates pause in script}
#>
Param([parameter(Mandatory=$true)]$utm)

$ws = "https://endpoints.office.com" # webservice root URL
$datapath = "C:\scripts\sophos\endpoints_clientid_latestversion.txt" # path where client ID and latest version number will be stored
$exceptionUrlList = @() # Sophos Exception list
# Personalize the Tenant name.
$tenantName = 'tenantID' # API Tenant ID
$comment = 'Microsoft URL | ' + (Get-Date).ToString("yyyy-MM-dd") + ' PS1'
$apiURL = $utm.URL
$exceptionUri = $apiURL + '/objects/http/exception/'
$networkURI = $apiURL + '/objects/network/network/'
$groupURI = $apiURL + '/objects/network/group/'
$resultCSV = 'C:\scripts\sophos\resultList.csv'
$exceptionUrls = @() # processed list of URLs for Sophos Formatting
$msGroup = @() #Sophos Network Group that will contain all of the network object created
$utmNetList = @() #All Networks retrieved from the UTM
$msNetList = @() #UTM Networks filtered for MS-
$ipList = @() #IP Addresses downloaded from Microsoft XML
$ipNetList = @() #IP Addresses parsed into Network format
$resultList = @() #Dispositon of networks for notification
# Personalize the mail server and recipient/sender information
$emailHeader = @{smtpserver = 'mailserver';
subject = 'Microosft IPv4 Address Update for ' + $utm.UTM;
to = 'h.schenkelaars@hcc-online.nl';
from = 'utm@hcc-online.nl'}

#Must have corresponding account configured with token on UTM
$token = $utm.KEY
$tokenBase64 = [Convert]::ToBase64String([System.Text.Encoding]::Default.GetBytes("token:" + $token))

#Common headers required by Sophos API
$headers = @{}
$headers.add("Authorization",'Basic ' + $tokenBase64)
$headers.add("Content-Type", "application/json")
$headers.add("Accept", "application/json")

#Sets the TLS level to match Sophos
$AllProtocols = [System.Net.SecurityProtocolType]'Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols

#Retrieve the existing list of MS subnets from Sophos
$utmNetList = Invoke-RestMethod -Uri $networkURI -Method Get -Headers $headers
$msNetList = $utmNetList | where {$_.name -like 'MS-*/*' -and $_.comment -like 'Microsoft*PS1'}

#Retrieve the existing list of HTTP Exceptions from Sophos
$utmExceptionList = Invoke-RestMethod -Uri $exceptionUri -Method Get -Headers $headers
$msExceptionList = $utmExceptionList | where {$_.name -match 'Microsoft Office365 URLs' -and $_.comment -like 'Microsoft*PS1'}

# fetch client ID and version if data file exists; otherwise create new file
if (Test-Path $datapath) {
$content = Get-Content $datapath
$clientRequestId = $content[0]
$lastVersion = $content[1]
}
else {
$clientRequestId = [GUID]::NewGuid().Guid
$lastVersion = "0000000000"
@($clientRequestId, $lastVersion) | Out-File $datapath
}

# call version method to check the latest version, and pull new data if version number is different
$version = Invoke-RestMethod -Uri ($ws + "/version/O365Worldwide?clientRequestId=" + $clientRequestId)

if ($version.latest -ge $lastVersion) {
Write-Host "New version of Office 365 worldwide commercial service instance endpoints detected"

# write the new version number to the data file
@($clientRequestId, $version.latest) | Out-File $datapath

# invoke endpoints method to get the new data
$endpointSets = Invoke-RestMethod -Uri ($ws + "/endpoints/O365Worldwide?clientRequestId=" + $clientRequestId + "&TenantName=" + $tenantName)

# filter results for Allow and Optimize endpoints, and transform these into custom objects with port and category
$flatUrls = $endpointSets | ForEach-Object {
$endpointSet = $_
$allowUrls = $(if ($endpointSet.allowUrls.Count -gt 0) { $endpointSet.allowUrls } else { @() })
$optimizeUrls = $(if ($endpointSet.optimizeUrls.Count -gt 0) { $endpointSet.optimizeUrls } else { @() })

$allowUrlCustomObjects = $allowUrls | ForEach-Object {
[PSCustomObject]@{
category = "Allow";
url = $_;
# Allow URLs should permit traffic across both Allow and Optimize ports
tcpPorts = (($endpointSet.allowTcpPorts, $endpointSet.optimizeTcpPorts) | Where-Object { $_ -ne $null }) -join ",";
udpPorts = (($endpointSet.allowUdpPorts, $endpointSet.optimizeUdpPorts) | Where-Object { $_ -ne $null }) -join ",";
}
}
$optimizeUrlCustomObjects = $optimizeUrls | ForEach-Object {
[PSCustomObject]@{
category = "Optimize";
url = $_;
tcpPorts = $endpointSet.optimizeTcpPorts;
udpPorts = $endpointSet.optimizeUdpPorts;
}
}
$allowUrlCustomObjects, $optimizeUrlCustomObjects
}

$flatIps = $endpointSets | ForEach-Object {
$endpointSet = $_
$ips = $(if ($endpointSet.ips.Count -gt 0) { $endpointSet.ips } else { @() })
# IPv4 strings have dots while IPv6 strings have colons
$ip4s = $ips | Where-Object { $_ -like '*.*' }

$allowIpCustomObjects = @()
if ($endpointSet.allowTcpPorts -or $endpointSet.allowUdpPorts) {
$allowIpCustomObjects = $ip4s | ForEach-Object {
[PSCustomObject]@{
category = "Allow";
ip = $_;
tcpPorts = $endpointSet.allowTcpPorts;
udpPorts = $endpointSet.allowUdpPorts;
}
}
}
$optimizeIpCustomObjects = @()
if ($endpointSet.optimizeTcpPorts -or $endpointSet.optimizeUdpPorts) {
$optimizeIpCustomObjects = $ip4s | ForEach-Object {
[PSCustomObject]@{
category = "Optimize";
ip = $_;
tcpPorts = $endpointSet.optimizeTcpPorts;
udpPorts = $endpointSet.optimizeUdpPorts;
}
}
}
$allowIpCustomObjects, $optimizeIpCustomObjects
}

Write-Output "IPV4 Firewall IP Address Ranges"
($flatIps.ip | Sort-Object -Unique) -join "," | Out-String

Write-Output "URLs for Proxy Server"
($flatUrls.url | Sort-Object -Unique) -join "," | Out-String

# Modifies the URLs for Sophos and populates the $exceptionUrls array
foreach ($url in $flatUrls.url){
$newUrl = ''
if ($url -like '`*-*') {
$newUrl = ($url).Replace('.','\.').Replace('*-','^https?://companyname-') # Replace with specific company name - might could use the TenantID variable
} elseif ($url -like '`*.*'){
$newUrl = ($url).Replace('.','\.').Replace('*\.','^https?://[^.]*\.')
} else {
$newUrl = '^https?://' + ($url -replace '\.','\.')
}
$exceptionUrls = $exceptionUrls += $newUrl
}

if ($msExceptionList.name.count -eq 0){
$exceptionMethod = 'Post'
$msExceptionUri = $exceptionUri
} elseif ($msExceptionList.name.count -eq 1){
$exceptionMethod = 'Patch'
$msExceptionUri = $exceptionUri + $msExceptionList._ref
}

<#Options for the skiplist
av -- anti-virus
cache -- Caching
certcheck -- Certificate Trust Check
certdate -- Certificate Date Check
check_max_download -- Block by download size
content_removal -- Content removal
contenttype_blacklist -- MIME type blocking
extensions -- Extension blocking
log_access -- Logging Accessed pages
log_blocked -- Logging Blocked pages
patience -- Do not display Download/Scan progress page
ssl_scanning -- SSL Scanning
url_filter -- URL Filter
user_auth -- Authentication
#>
$skipList = @('av',
'content_removal',
'user_auth',
'extensions',
'url_filter',
'contenttype_blacklist',
'cache',
'log_access',
'log_blocked',
'ssl_scanning',
'check_max_download',
'patience'
)

#$msUrlList = ConvertTo-Json $exceptionUrls

$exceptionBody = @{'aaa' = @();
'comment' = $comment;
'domains' = $exceptionUrls;
'endpoints_groups' = @();
'name' = 'Microsoft Office365 URLs';
'networks' = @();
'operator' = 'AND';
'skiplist' = $skipList;
'sp_categories' = @();
'status' = $true;
'tags' = @();
'user_agents' = @()
}

Invoke-RestMethod -Uri $msExceptionUri -Method $exceptionMethod -Headers $headers -Body (ConvertTo-Json $exceptionBody) # TODO Call Send-MailMessage with new endpoints data
}
else {
Write-Host "Office 365 worldwide commercial service instance endpoints are up-to-date"
}
Cancel
Vote Up 0 Vote Down

Cancel