Approved
Sign in to vote on ideas
+2
Sign in to vote on ideas

Queries from the March SophSkills presentation

Video: https://vimeo.com/519661823

Queries used:

Queries used during SophSkills Demo

DATA LAKE - List all EP and FW tables in the data lake

This query will need to run against the data lake.  As we add more sensors to the data lake we will be extending this query with additional sections to list the tables for the additional sensor. For example Email is coming to the data lake in just a few weeks so we will be adding that.   Note the query does not yet detect if the sensor is present so for account without FW Data in the lake you should comment out that section or the query will report an error that it could not find the xgfw_data table.

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-- LIST ALL EP and XG TABLES
WITH Full_Data_Set AS (
(SELECT DISTINCT 'XG_Firewall' Source, log_component Table_Name, COUNT(dist_key) Entries
FROM xgfw_data
GROUP BY log_component
ORDER By log_component ASC)
UNION ALL
(SELECT DISTINCT 'Endpoint' Source, query_name Table_Name, COUNT(endpoint_id) Entries
FROM xdr_data
GROUP BY query_name
ORDER BY query_name ASC)
)
SELECT UPPER(Source) Sensor_Type, LOWER(Table_Name) Table_Name, Entries FROM Full_Data_Set ORDER BY Sensor_Type, Table_Name, Entries
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 

ENDPOINT - Data Lake Hydration Queries

This query runs as an Endpoint Query is supported on Windows MAC and Linux, you only need to run it on a single device for each OS type as the data lake hydration queries are common across all instances of a supported OS

Fullscreen
1
2
3
-- Data Lake Hydration query
SELECT name, query, interval
FROM osquery_schedule
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

DATA LAKE - Windows programs Inventory Search (Datalake)

This query takes variables that will need to be defined when you save the query.

Descriptive Name

Variable type

SQL variable name

Value

Application Name

String

$$Application Name$$

%

Application Version

String

$$Application Version$$

%

Host Name

String

$$Host Name$$

%

Publisher Name

String

$$Publisher Name$$

%

 

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
WITH Counted_Apps AS (
WITH App_List AS (
SELECT DISTINCT meta_hostname, name, version, publisher, install_date
FROM xdr_data
WHERE query_name = 'windows_programs' AND name > '' AND
LOWER(name) LIKE LOWER('%$$Application Name$$%') AND LOWER(version) LIKE LOWER('%$$Application Version$$%') AND
LOWER(publisher) LIKE LOWER ('%$$Publisher Name$$%') AND LOWER(meta_hostname) LIKE LOWER ('%$$Host Name$$%')
)
-- WHEN Name, version and Publisher are wildcards group by publisher and put the name, version and devices info into a list
SELECT
publisher, COUNT(meta_hostname) Instances,
array_join(array_agg(DISTINCT name), ','||CHR(10)) App_name_List,
array_join(array_agg(DISTINCT version), ','||CHR(10)) version_LIST,
array_join(array_agg(DISTINCT meta_hostname), ','||CHR(10)) DeviceName_LIST, MIN(install_date) Earliest_Install, MAX(install_date) Last_Install
FROM App_List WHERE
'$$Application Name$$' = '%' AND '$$Application Version$$' = '%' AND '$$Publisher Name$$' = '%'
GROUP BY publisher
UNION ALL
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

DATA LAKE – MITRE ATT&CK - EXFILTRATION

This query scans all data in the data lake looking for IOC’s that match a list of 67 detection rules. It does take variables

Descriptive Name

Variable type

SQL variable name

Value

Any Field String Match

String

$$Any Field String Match$$

%

Device Name

String

$$Device Name$$

%

MITRE ATT&CK ID

String

$$MITRE ATT&CK ID$$

%

Tactic

String

$$Tactic$$

%

Technique

String

$$Technique$$

%

 

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
-- Hunt for EXFILTRATION IOC
-- VARIABLE $$Any Field String Match$$ STRING
-- VARIABLE $$Device Name$$ STRING
-- VARIABLE $$MITRE ATT&CK ID$$ STRING
-- VARIABLE $$Tactic$$ STRING
-- VARIABLE $$Technique$$ STRING
-- Calculate detection counts for similar IOC's across multiple devices or multiple instances of the same IOC on a device for the same process-name and cmdline
WITH Detection_Counts AS (
WITH Detections AS (
-- Detect MITRE ATT&CK Impact Tactic using Process and cmdline info only
WITH Mitre_map (ID,Tactic,Technique,SubTechnique,Mitre_Link,Method,Condition,SubCondition,Refrence) AS (
-- EXFILTRATION DETECTION SET
SELECT 'T1020.001','Exfiltration','Automated Exfiltration','Traffic Duplication','https://attack.mitre.org/techniques/T1020/001','Process_Cmd','powershell.exe','%Set-VMNetworkAdapter%PortMirroring%',''
UNION ALL SELECT 'T1020.001','Exfiltration','Automated Exfiltration','Traffic Duplication','https://attack.mitre.org/techniques/T1020/001','Process_Cmd','powershell.exe','%Add-NetEventPacketCaptureProvider%',''
UNION ALL SELECT 'T1030','Exfiltration','Data Transfer Size Limits','','https://attack.mitre.org/techniques/T1030','Process_Cmd','powershell.exe','%GetRequestStream%Write%length%',''
UNION ALL SELECT 'T1030','Exfiltration','Data Transfer Size Limits','','https://attack.mitre.org/techniques/T1030','Process_Cmd','powershell.exe','%upload%chunksize%',''
UNION ALL SELECT 'T1048.003','Exfiltration','Exfiltration Over Alternative Protocol','Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol','https://attack.mitre.org/techniques/T1048/003','Process_Cmd','telnet.exe','%localhost 80%','https://www.codingame.com/playgrounds/13710/conversation-over-http'
UNION ALL SELECT 'T1048.003','Exfiltration','Exfiltration Over Alternative Protocol','Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol','https://attack.mitre.org/techniques/T1048/003','Process_Cmd','telnet.exe','% 80%','https://www.codingame.com/playgrounds/13710/conversation-over-http'
UNION ALL SELECT 'T1048.003','Exfiltration','Exfiltration Over Alternative Protocol','Exfiltration Over Unencrypted/Obfuscated Non-C2 Protocol','https://attack.mitre.org/techniques/T1048/003','Process_Cmd','telnet.exe','% 25%','https://www.pentestpartners.com/security-blog/data-exfiltration-techniques/'
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

ENDPOINT – Process Tree for a Sophos PID

This query takes the SophosPID and provides the process tree. It is a bit different than the Sophos Canned query for the same, in that it adds some different fields in the result. 

Descriptive Name

Variable type

SQL variable name

Value

SPID

String

$$SPID$$

%

 

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
SELECT *
FROM (
WITH RECURSIVE Trace(SophosPID, LEVEL, parent, TIME, cmdline, sha256, pathname, endtime) AS (
VALUES (
'$$SPID$$',
'▬▬▬▬▬▬▬▬▬▬▬',
(SELECT ParentSophosPID FROM sophos_process_journal WHERE sophos_process_journal.SophosPID = '$$SPID$$' AND TIME = replace('$$SPID$$', rtrim('$$SPID$$', replace('$$SPID$$', ':', '')), '') / 10000000 - 11644473600),
(SELECT TIME FROM sophos_process_journal WHERE sophos_process_journal.SophosPID = '$$SPID$$' AND TIME = replace('$$SPID$$', rtrim('$$SPID$$', replace('$$SPID$$', ':', '')), '') / 10000000 - 11644473600),
'▬▬▬▬▬▬▬▬▬▬▬', /* not required as this row will be excluded */
'▬▬▬▬▬▬▬▬▬▬▬', /* not required as this row will be excluded */
'▬ ANCESTORS ABOVE ▬', /* Create a seperator to indicate where the target SPID is in the tree */
'▬▬▬▬▬▬▬▬▬▬▬'
)
UNION ALL
SELECT spj.SophosPID,
Trace.LEVEL - 1,
spj.parentSophosPID,
spj.TIME,
spj.cmdline,
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

DATA LAKE – MITRE ATT&CK CALDERA

This query will check for 141 different indicators of compromise as demonstrated by the Caldera test tool. You can download that tool from GIT https://github.com/mitre/caldera  

Descriptive Name

Variable type

SQL variable name

Value

Any Field String Match

String

$$Any Field String Match$$

%

Device Name

String

$$Device Name$$

%

MITRE ATT&CK ID

String

$$MITRE ATT&CK ID$$

%

Tactic

String

$$Tactic$$

%

Technique

String

$$Technique$$

%

 

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
-- Hunt for CALDERA IOCS
-- VARIABLE $$Any Field String Match$$ STRING
-- VARIABLE $$Device Name$$ STRING
-- VARIABLE $$MITRE ATT&CK ID$$ STRING
-- VARIABLE $$Tactic$$ STRING
-- VARIABLE $$Technique$$ STRING
-- Calculate detection counts for similar IOC's across multiple devices or multiple instances of the same IOC on a device for the same process-name and cmdline
WITH Detection_Counts AS (
WITH Detections AS (
-- Detect MITRE ATT&CK Impact Tactic using Process and cmdline info only
WITH Mitre_map (ID,Tactic,Technique,SubTechnique,Mitre_Link,Method,Condition,SubCondition,Refrence) AS (
-- CALDERA DETECTION SET
SELECT 'T1002','exfiltration','Compress staged directory','Data Compressed','https://attack.mitre.org/techniques/T1002','Process_Cmd','%powershell%','%Compress-Archive%','Compress a directory on the file system'
UNION ALL SELECT 'T1003','Process Memory-access','Leverage Procdump','Memory Dump','https://attack.mitre.org/techniques/T1003','Process_Cmd','%powershell%','%procdump%','Dump a processes memory'
UNION ALL SELECT 'T1003','credential-access','GetComputers (Alice','Credential Dumping','https://attack.mitre.org/techniques/T1003','Process_Cmd','%powershell%','%Get-NetComputer%','Get a list of all computers in a domain'
UNION ALL SELECT 'T1003','credential-access','Leverage Procdump for lsass memory','Credential Dumping','https://attack.mitre.org/techniques/T1003','Process_Cmd','%','%ma lsass.exe%','Dump lsass for later use with mimikatz'
UNION ALL SELECT 'T1003','credential-access','MiniDumpWriteDump (Spooky','Credential Dumping','https://attack.mitre.org/techniques/T1003','Process_Cmd','%','%\creds.dmp','Custom GO credential dumper using minidumpwritedump'
UNION ALL SELECT 'T1003','credential-access','Powerkatz (Staged','Credential Dumping','https://attack.mitre.org/techniques/T1003','Process_Cmd','%','%DumpCreds%','Use Invoke-Mimikatz'
UNION ALL SELECT 'T1003','credential-access','Credentials in Registry','Credential Dumping','https://attack.mitre.org/techniques/T1003','Process_Cmd','%','%reg query HKLM%/f password% /t REG_SZ% ','Search for possible credentials stored in the HKLM Hive'
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

DATA LAKE – Generic Search on Process Details

This looks across all process records in the data lake for a variety of things. It will auto group information based on Parent, Child, cmdline and user. (it does take into account the SHA256 of processes so even if the same process name is used it will not group if the SHA256 was different). We provide a number of variables to perform filtering.

Descriptive Name

Variable type

SQL variable name

Value

Command Line

String

$$Command Line$$

%

Device

String

$$Device$$

%

Ignore when > N Duplicates

String

$$Ignore when > N Duplicates$$

10000

Parent Process Name

String

$$Parent Process Name$$

%

Process Name

String

$$Process Name$$

%

User Name

String

$$User Name$$

%

 

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
-- Generic Search
-- VARIABLE: $$Device$$ STRING
-- VARIABLE: $$Command Line$$ STRING
-- VARIABLE: $$Process Name$$ STRING
-- VARIABLE: $$Parent Process Name$$ STRING
-- VARIABLE: $$User Name$$ STRING
-- VARIABLE: $$Ignore when > N Duplicates$$ STRING
WITH Count_limt AS (
With DeDuplicate AS (
SELECT
xdr_data.meta_hostname epName,
xdr_data.meta_os_type Device_Type,
replace(replace(CAST(from_unixtime(xdr_data.time) AS VARCHAR),'.000','Z'),' ','T') Date_Time, -- add the T to help excel understand this is a date and time
CASE CAST(xdr_data.username > '' AS INT)
WHEN 1 THEN xdr_data.username --|| ' : ' || xdr_data.meta_username
ELSE xdr_data.meta_username
END User_Name,
xdr_data.parent_name Parent_Process_Name,
xdr_data.name Process_Name,
xdr_data.cmdline Cmd_Line,
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

ENDPOINT – Decode Encoded powershells

This query will look for either ALL powershells or just encoded powershells by setting the first variable then for any encoded powershells it will perform a base64 decode of the encoded data.

Descriptive Name

Variable type

SQL variable name

Value

ALL powershell Commands (1) or just encoded powershell(0)

String

$$ALL powershell Commands (1) or just encoded powershell(0)$$

0

From N days ago

String

$$From N days ago$$

20

To N days ago

String

$$To N days ago$$

0

 

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
-- Powershell decoder
-- VARIABLE $$ALL powershell Commands (1) or just encoded powershell(0)$$ STRING
-- VARIABLE $$From N days ago$$ STRING
-- VARIABLE $$To N days ago$$ STRING
-- Go find processes that have command lines with encodedcommand data
WITH Encoded_Data AS (
SELECT
datetime(time,'unixepoch') DATE_TIME,
-- replace(pathname, rtrim(pathname, replace(pathname, '\', '')), '') ProcessName,
cmdline,
sophospid,
replace(substr(cmdline, instr(cmdline, ' -e')+2, length(cmdline)),rtrim(substr(cmdline, instr(cmdline, ' -e')+2, length(cmdline)),replace(substr(cmdline, instr(cmdline, ' -e')+2, length(cmdline)),' ','')),'') Encoded_Data,
sid,
parentSophosPID,
u.username
FROM sophos_process_journal spj LEFT JOIN users u ON spj.sid = u.uuid
WHERE pathname LIKE '%powershell%' AND eventtype = 0 AND time > strftime('%s','now','-$$From N days ago$$ days') and time < strftime('%s','now','-$$To N days ago$$ days')
AND ( cmdline LIKE '% -e %' OR cmdline LIKE '% -en %' OR cmdline LIKE '% -enc %' OR cmdline LIKE '% -enco %' OR cmdline LIKE '% -encod %' OR cmdline LIKE '% -encode %' OR cmdline LIKE '% -encoded %'
OR cmdline LIKE '% -encodedc %' OR cmdline LIKE '% -encodedco %' OR cmdline LIKE '% -encodedcom %' OR cmdline LIKE '% -encodedcomm %' OR cmdline LIKE '% -encodedcomma %'
OR cmdline LIKE '% -encocodedcomman %' OR cmdline LIKE '% -encodedcommand %' OR cmdline LIKE '% -ec %')
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • 1 comment
  • 0 members are here
  • Thanks Karl for these awesome queries!

    For anyone implementing, make sure you assign them to Endpoint Queries or Data Lake Queries based on the indicator in the name of each query.