Is there an official UTM logfile schema description?

Hi dtconnect,

Thank you for reaching out to Sophos Community.

"id=" in log event indicated message ID for specific log component.

Refer to the article below for more information on "Firewall Log Format".

support.sophos.com/.../KB-000037366

Hi Yash,

thank you for your response. I've seen that KB but I thought this was only valid for CR / XG ( It starts with a statement "Log ID is a Unique 12 characters code (c1c2c3c4c5c6c7c8c9c10c11c12) e.g. 0101011, 0102011" - that's not true for SG logs. But I've realized that the Table "Message ID" might be, what I am looking for.

But that only matches the packet filter.log. The Proxy Log has IDs like this one:

<142>2021:03:12-07:28:08 gateway httpd: id="0299" srcip="xxxxxxxxxxxxx" localip="xxxxxxxxxxxx" size="55886" user="-" host="xxxxxxxxxxxxxxxx" method="GET" statuscode="200" reason="-" extra="-" exceptions="-" time="114159" url="/" server="xxxxxxxxx" port="443" query="" referer="-" cookie="-" set-cookie="-" websocket_scheme="-" websocket_protocol="-" websocket_key="-" websocket_version="-" uid="xxxxxxxxxxxxxxxxxx"

I could not find ID 0299 in the KB doc. Can you help me on this, too?

Regards,

Daniel

Hallo Daniel,

Yeah, that Cyberoam documentation isn't very helpful for UTM.  The line you're showing isn't from Web Filtering, it's from the HTTP daemon log and I don't think I've ever seen a guide for that log.  I couldn't find a link to this old 2012 guide.  Here it is, but it doesn't cover the HTTP daemon.
Logfile_Guide-2012-09.pdf

Cheers - Bob

I have written tools to parse each of the useful log files into a SQL database.  (Splunk would probably have been a much better starting point, but it was not an available option.)  When doing so, I wrote pretty extensively on what I was learning and how I did it.   Most of those items are somewhere in "Recommended Reads"

As part of that effort, I built a master list of Item IDs.   In most cases, Item ID also determines the "name", "Sys", and "Sub" parameters, so these are included.   The final column indicates the log file where the Item ID can appear.   You should be able to drop this list into a CSV text file, then import it into Excel for better readability.

There will be omissions associated with the functions that I have never used, but at least you have a starting point. 

ItmId,Itmname,ItmSys,ItmSub,LogFile
0001,http access,SecureWeb,http,http
0002,web request blocked,SecureWeb,http,http
0003,http access.,SecureWeb,http,http
0056,"web request blocked, virus detected",SecureWeb,http,http
0057,"web request blocked, PUA detected",SecureWeb,http,http
0059,"web request blocked, forbidden tag detected",SecureWeb,http,http
0060,"web request blocked, forbidden category detected",SecureWeb,http,http
0061,"web request blocked, reputation limit",SecureWeb,http,http
0062,"web request blocked, forbidden url detected",SecureWeb,http,http
0063,Contentfilter override, , ,
0064,"web request blocked, forbidden file extension detected",SecureWeb,http,http
0065,"web request blocked, forbidden mimetype detected",SecureWeb,http,http
0066,"web request blocked, forbidden application detected",SecureWeb,http,http
0067,"web request blocked, connection to forbidden country",SecureWeb,http,http
0068,"web request blocked, threat detected",SecureWeb,http,http
0070,"web request blocked, download exceeds maximum allowable size",SecureWeb,http,http
0071,"web request warned, forbidden category detected",SecureWeb,http,http
0072,Contentfilter warned and proceeded,SecureWeb,http,http
0073,"web request warned, forbidden file extension detected",SecureWeb,http,http
0074,File extension warned and proceeded,SecureWeb,http,http
0079,"web request quota-limited, forbidden category detected",SecureWeb,http,http
0080,Contentfilter quota-limited and proceeded,SecureWeb,http,http
0299,reverseproxy,,,reverseproxy
1000,email passed,SecureMail,smtp,smtp
1001,email quarantined,SecureMail,smtp,smtp
1100,email passed,SecureMail,pop3,pop3
1101,email quarantined,SecureMail,pop3,pop3
2001,Packet dropped,SecureNet,packetfilter,packetfilter
2002,Packet accepted,SecureNet,packetfilter,packetfilter
2003,Packet rejected, ,packetfilter,packetfilter
2009,ICMP redirect,SecureNet,packetfilter,packetfilter
2013,FTP data,SecureNet,packetfilter,packetfilter
2017,AFC Alert,SecureNet,packetfilter,afc
2019,AFC Block,SecureNet,packetfilter,afc
2021,Packet dropped (GEOIP),SecureNet,packetfilter,aptp
2022,Packet dropped (ATP),SecureNet,packetfilter,aptp
2101,Intrusion protection alert,SecureNet,ips,ips
2104,ICMP flood detected,SecureNet,ips,ips
2201,Connection started,SecureNet,vpn,html5vpn
2202,Connection stopped,SecureNet,vpn,html5vpn
3004,Login Success,System,auth,aua
3005,Login Processing,System,auth,aua
3006,Login Failure,System,auth,aua
3100,timeout: removing session,System,confd,confd
310a,object changed,System,confd,confd
310c,node changed,System,confd,confd
310n,applied changes,System,confd,confd
3701,Authentication successful,system,up2date,up2date
3707,Successfully synchronized fileset,system,up2date,up2date

wow, very nice! Thanks for sharing!