Hi,
I've been scheduling the two Live Discover DataLake Mitre queries that Karl posted as part of the March SophSkills. Initially these worked great, however, now that I have completed my rollout of the latest Core Agent to all Endpoints & Servers both queries consistently fail. I guess I have just short of 1800 devices in total now on the latest clients.
I see errors like this, my customer ID has been intentionally deleted:
"Invalid operation due to 'Query failed (#20210606_230037_01239_8dska): Error reading tail from s3://data-series-shared-eu-west-1-prod-xdr-datalake/mergedSymlinkFiles/xdr_data_global/customer_id=deleted-by-me/stream_ingest_date=2021-06-06/endpoint_type=computer/3ef6f2fe-b28d-4065-b2ce-7073f9ec8560_1623015062336_77658137.prq with length 16384'".
When these started failing I went through the queries and commented out a load of lines in the hope of getting them to run faster and successfully. Initially this worked, however, now that I have all my devices protected by latest Core Agent they are failing again. Looks like they fail between 10 to 15 minutes after starting.
If there anything I can do other than split these queries up into smaller batches? Are there any planned improvements coming that may allow me to run these queries successfully?
Anybody here with 2000+ clients successfully running these two community Mitre DataLake queries?
Many thanks.