We’re excited to announce a new search experience that makes it easier to investigate and hunt threats on the endpoint. This Early Access Program (EAP) is delivering the first major step towards reducing search complexity and enabling you to find the data you need faster, without needing to be an expert.
What’s Included?
This EAP includes a new Search functionality that leverages the Apache Lucene query language, which is more straight forward and less verbose than SQL. Notably, we’ve also begun to standardize field names on a common taxonomy, starting with Sophos Endpoint data stored in the Sophos Data Lake. The available fields are listed in the search page itself so that you don’t have to cross-reference documentation before using it.
How to Access and Use the New Search
This feature is available to members of the “New XDR Features” EAP. If you’re not yet a member, you can click your name in the top right corner of the console, and select “Early Access Programs.” From there, select the “New XDR Features” program. Be aware that you’ll need Admin level permissions to join the EAP.
Once you’re set up, navigate to the Threat Analysis Center and click “Search.”
This will bring to you to the page shown below. Key items for you to be aware of are numbered and annotated in yellow, with specific tips below.
1. Endpoint Data – During the EAP and in the first GA release, the data available to search will be Endpoint Data that has been hydrated into the Data Lake. In future releases, we intend to make detection data searchable through this experience as well.
2. Visible Columns – These commonly used fields are set by default as column headers for EAP. Depending on your search, you may want to change and/or rearrange these. Click the red minus icon to remove a column. Drag and drop the list of Visible Columns up and down to re-order them in the results table. At GA, the default columns will automatically update based which fields you search in order to show the most relevant fields and help reduce extra steps.
3. Available Columns – These are additional fields you can search against and optionally set as column headers in your results table. Click on any these field names to add these to Visible Columns.
4. Search Bar – This is where you’ll begin your query. Lucene queries are generally made up of Fieldnames and their Terms (values). For example, the below search looks for both (1) the field “title” where the field value is “The Fox and Hound” and (2) the field “message” where the value is “abcde”.
title:"The Fox and Hound" AND message:abcde
In the UI a user can search a few ways:
1. Free Text: This does not require prefacing your query with a specified field name. It is important to note that, by default, a free text search will look for exact matches. Alternatively, you can use the wildcard * within or at the end of the text to achieve a “starts with” search, such as:
ab*e
ab*
2. Prompted Lucene: Begin your query by typing @ to take advantage of autofill. When you press @ a quick-pick drop down list will appear so that you can quickly select a specific field without having to manually type it yourself.
3. Standard Lucene: If you’re familiar with Lucene syntax, you can simply type the Lucene structure directly into the search bar and go.
Tips and Tricks for Lucene
If you’re not familiar with the Lucene query language, here are a few tips and tricks:
To search for a specific field parameter, enter the field_name:value. Example:
username:jdoe
process_id:16188
Lucene requires that fielded searches use colons “:” between the fieldname and value. Equals signs “=” are not supported by Lucene and will return errors.
Lucene handles some characters differently and views them as special character dividers that will impact how your search is interpreted. The special characters to be aware of are:
+ - && || ! ( ) { } [ ] ^ " ~ * ? : \
If you want to search for data that typically involves any of the above characters, such as IP addresses, MAC addresses, or URLs, use quotations before and after your search values to prevent these characters from impacting the results. Example:
ip_address:“192.168.0.103”
url:“www.badguydomain.com/17fniowev9”
Lucene supports AND, OR, and NOT statements to better filter your results. The operators are case sensitive and must be in all caps. You can use these individually, or with parentheses to enable more complex searches.
Example: If you want to find out if there were any suspicious RDP connections from one specific device, you can search:
device_ip:“192.168.0.103” AND protocol:rdp
Example: If you want to find out if there were any suspicious RDP connections from a set of multiple devices, you can use parentheses to search the devices first, and then the protocol to ensure the full data set is captured:
(device_ip:“192.168.0.103” OR device_ip:“192.168.0.104” OR device_ip:“192.168.0.103”) AND protocol:rdp
Example: Because operators are case sensitive, the following search will not return the results you are looking for, and it may return an error.
device_ip:“192.168.0.103” and protocol:rdp
For additional tips, check out this Lucene overview from Apache for helpful background, as well as the Sophos Search Documentation.
Known EAP limitations
In this first version of Search, there are a few known limitations that we are actively working on:
Wildcards are currently enabled in the middle and at the end of search terms, but they are not enabled as the first character. Examples:
username:johnd* executes a search to look for all usernames that start with “johnd” but many contain additional characters at the end
username:j*doe executes a search for all usernames that starts with “j” and end with “doe”
username:*doe will return an error due to this current limitation
Null values can be filtered out using Lucene, but at the present moment users cannot search specifically for fields that are null (eg. do not contain values). For example, if I’m interested only in process activity on a specific machine, I can run a search for all activity on that machine, and then filter out everything where the “process_id” field contains no data (eg. is null). To do this, I can search first for the machine, AND then when process_id is null.
device_ip:”192.168.1.1” AND process_id NOT null
However, we do not presnetly support the inverse to search only for fields that are null. Example:
process_id:null will return an error
URLs are currently a known issue, and we are working to resolve this. The “”url” field is presently available as a searchable field, but URL fields are not presently searchable. This will be addressed in the coming weeks.
What else to know?
Throughout EAP, we’ll be making adjustments to improve results, fix bugs, and address your feedback. Reach out to us at GlobalSearchEAP@sophos.com to provide feedback.
