All log files description/documentation?

Salut G,

Wise words from Doug!

You might also be interested in Packetfilter logfiles on the Sophos UTM.

Cheers - Bob
PS Thanks for the link to the new article on the Web Filtering log.

EDIT: Sophos took down the A Guide to Logfiles and Output KB article and has not replaced it.

Hello Both,

Thanks for your feedback. Just to give you more context, I'm trying to ingest Sophos UTM logs in Splunk and make most of these logs events compliant with the Splunk Common Information Model (CIM). Especially with the Authentication, Web, Network traffic and Network Session models.

For this I need to alias the UTM logs field to Splunk CIM models one, I need to do calculation/conversion (from microsecond to seconds for instance), etc.

Having a full description of all (most) UTM log files and fields will greatly help me in this work. So far, I had to "guess" fields name, values and units meaning to do this work.

@Bob: your first link to "A guide to Logfiles and Output" seems not working.

Best Regards,

G

This is a worthy goal.

Start with the simplest file format: the firewall log

Take all of my code from this post and use it to load into a SQL database

https://community.sophos.com/products/unified-threat-management/f/management-networking-logging-and-reporting/100770/how-to-using-a-sql-database-to-interpret-utm-log-files

This will allow you to see the logs in structured form, which will essentially answer your original question for this one file.

Then figure out how to do equivalent parsing in SPLUNK-speak, and prove that it produces an equivalent result to the SQL table.

When you are ready to make your head hurt, ask me for the code to the more complex files.   Then repeat the process.

Thanks for the link to your SQL loading post. Impressive work and could be still helpful.

In fact, I'm a bit further in my work :)

Thanks to the clean format of most UTM logs (field names=field value), Splunk does its magic by splitting and extracting them "automatically". For httpd logs, I've used regex to achieve the same.

After this, I can see the logs events in clear "table" views .

For instance, this is how the FW logs events appears in Splunk (I've removed IPs and MACs):

I've also done already some aliasing/conversions to make them Splunk CIM compliant. So far, the mappings below is where i am at this moment (in Splunk "language" sorry but should be straightforward to read and understand I feel).

I was therefore wondering if these mappings were semantically correct...

[sophos:utm:aua]
EVAL-action = case(name == "Authentication successful", "success", name == "Authentication failed", "failure")
FIELDALIAS-sophos-signature_id = id AS signature_id
EVAL-app = mvindex(split(sourcetype, ":"), -1)
FIELDALIAS-sophos-src = srcip AS src
FIELDALIAS-sophos-signature = name AS signature
FIELDALIAS-sophos-dest = host AS dest

[sophos:utm:httpproxy]
EVAL-action = case(action == "pass", "allowed", action == "block", "blocked")
EVAL-app = mvindex(split(sourcetype, ":"), -1)
FIELDALIAS-sophos-src = srcip AS src
FIELDALIAS-sophos-src_ip = srcip AS src_ip
FIELDALIAS-sophos-dest = url AS dest
FIELDALIAS-sophos-dest_ip = dstip AS dest_ip
EVAL-protocol = "ip"
FIELDALIAS-sophos-bytes = size AS bytes
EVAL-direction = "outbound"
EVAL-duration = round(fullreqtime/1000000)
FIELDALIAS-sophos-dvc = host AS dvc
FIELDALIAS-sophos-rule = filteraction as rule
FIELDALIAS-sophos-rule_id = id as rule_id

[sophos:utm:fw]
EVAL-action = case(action == "drop", "blocked")
EVAL-app = mvindex(split(sourcetype, ":"), -1)
EVAL-direction = "inbound"
FIELDALIAS-sophos-src = srcip AS src
FIELDALIAS-sophos-src_ip = srcip AS src_ip
FIELDALIAS-sophos-src_mac = srcmac AS src_mac
FIELDALIAS-sophos-src_port = srcport AS src_port
FIELDALIAS-sophos-dest = dstip AS dest
FIELDALIAS-sophos-dest_ip = dstip AS dest_ip
FIELDALIAS-sophos-dest_mac = dstmac AS dest_mac
FIELDALIAS-sophos-dest_port = dstport AS dest_port
FIELDALIAS-sophos-dvc = host AS dvc
FIELDALIAS-sophos-rule = name as rule
FIELDALIAS-sophos-rule_id = fwrule as rule_id
FIELDALIAS-sophos-tcp_flag = tcpflags as tcp_flag
FIELDALIAS-sophos-bytes = length AS bytes
LOOKUP-auto_transport_protocols = transport_protocols PROTOCOL_ID AS proto OUTPUTNEW PROTOCOL_NAME AS transport PROTOCOL_DESCRIPTION AS transport_desc

[sophos:utm:httpd]
EVAL-app = "Sophos UTM"
EXTRACT-access_log = .+httpd:\s(?<src>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\s-\s-\s.+\]\s"(?<http_method>\S*)\s((?<uri_path_qry>\S*)(?<uri_query>\?\S*)|(?<uri_path>\S*))\s(?<protocol>\S*)"\s(?<status>\d{3})\s(?<bytes>\d*)
FIELDALIAS-sophos-uri_path = uri_path_qry AS uri_path

Resources:   application master list at /etc/afc/applications

I built this list of all known Item IDs and the log file where they appear.   Sys and Sub are constants associated with each log file

Note that some of the obscure itmids are not numeric.

ItmId,Itmname,ItmSys,ItmSub,LogFile
0001,http access,SecureWeb,http,http
0002,web request blocked,SecureWeb,http,http
0003,http access.,SecureWeb,http,http
0056,"web request blocked, virus detected",SecureWeb,http,http
0057,"web request blocked, PUA detected",SecureWeb,http,http
0059,"web request blocked, forbidden tag detected",SecureWeb,http,http
0060,"web request blocked, forbidden category detected",SecureWeb,http,http
0061,"web request blocked, reputation limit",SecureWeb,http,http
0062,"web request blocked, forbidden url detected",SecureWeb,http,http
0064,"web request blocked, forbidden file extension detected",SecureWeb,http,http
0065,"web request blocked, forbidden mimetype detected",SecureWeb,http,http
0066,"web request blocked, forbidden application detected",SecureWeb,http,http
0067,"web request blocked, connection to forbidden country",SecureWeb,http,http
0068,"web request blocked, threat detected",SecureWeb,http,http
0070,"web request blocked, download exceeds maximum allowable size",SecureWeb,http,http
0071,"web request warned, forbidden category detected",SecureWeb,http,http
0072,Contentfilter warned and proceeded,SecureWeb,http,http
0073,"web request warned, forbidden file extension detected",SecureWeb,http,http
0074,File extension warned and proceeded,SecureWeb,http,http
0079,"web request quota-limited, forbidden category detected",SecureWeb,http,http
0080,Contentfilter quota-limited and proceeded,SecureWeb,http,http
0299,reverseproxy,,,reverseproxy
1000,email passed,SecureMail,smtp,smtp
1001,email quarantined,SecureMail,smtp,smtp
1100,email passed,SecureMail,pop3,pop3
1101,email quarantined,SecureMail,pop3,pop3
2001,Packet dropped,SecureNet,packetfilter,packetfilter
2002,Packet accepted,SecureNet,packetfilter,packetfilter
2003,Packet rejected,NULL,packetfilter,packetfilter
2009,ICMP redirect,SecureNet,packetfilter,packetfilter
2013,FTP data,SecureNet,packetfilter,packetfilter
2017,AFC Alert,SecureNet,packetfilter,afc
2019,AFC Block,SecureNet,packetfilter,afc
2021,Packet dropped (GEOIP),SecureNet,packetfilter,aptp
2022,Packet dropped (ATP),SecureNet,packetfilter,aptp
2101,Intrusion protection alert,SecureNet,ips,ips
2104,ICMP flood detected,SecureNet,ips,ips
2201,Connection started,SecureNet,vpn,html5vpn
2202,Connection stopped,SecureNet,vpn,html5vpn
3004,Login Success,System,auth,aua
3005,Login Processing,System,auth,aua
3006,Login Failure,System,auth,aua
3100,timeout: removing session,System,confd,confd
310a,object changed,System,confd,confd
310c,node changed,System,confd,confd
310n,applied changes,System,confd,confd
3701,Authentication successful,system,up2date,up2date
3707,Successfully synchronized fileset,system,up2date,up2date

When parsing the Web Filtering log file, I split the URL and the Referer URL into components (protocol, fqdn, path, querystring).   This has been very useful when doing log analysis.

Since there are so many web log records, I wanted to save space.

The web filtering file has these paired fields:

id and name

Appid and appplication

category and categoryname

To save space, I built a table with the code to text translation, which is automatically updated as I detect new values.  Then I drop the text from the parsed log file.

To save even more space, I replaced text with codes for these long fields, also using a code-to-text translation table:

Filter Action

Policy

UA (UserAgent)

This probably saves even more space than the paired fields.

When parsing the Web Filtering log file, I split the URL and the Referer URL into components (protocol, fqdn, path, querystring).   This has been very useful when doing log analysis.

Since there are so many web log records, I wanted to save space.

The web filtering file has these paired fields:

id and name

Appid and appplication

category and categoryname

To save space, I built a table with the code to text translation, which is automatically updated as I detect new values.  Then I drop the text from the parsed log file.

To save even more space, I replaced text with codes for these long fields, also using a code-to-text translation table:

Filter Action

Policy

UA (UserAgent)

This probably saves even more space than the paired fields.