heartbeat log: Cannot create ID for application, because appId range is exhausted. Application will be ignored.

This indicates all the 10k appids are already used. No more applications detected on endpoints will be reported by heartbeat. The cause could be a particular application that is being detected uniquely everytime. Heartbeat service should continue to run. However please confirm if heartbeat service is still running. To investigate further we would need access to the device.

Thank you.

We have some 3,5k static apps and a unknown number of synced apps. Can I count them somewhere?

Some of them are used very frequently as I can see from the counters.

30 apps per page would mean ~ 9780 apps with the 326 pages on GUI.

heartbeatd is running. We notice users sometimes lose their heartbeat / authentication on the firewall. maybe that is somehow in relation to that issue. normally a reboot brings them back in service.

Every new version / bin path of a program get's a new entry in the synced apps list. So it is exceptable that the list some day exceeds 10k AppIDs. Probably we need to delete the entire list and start to relearn the apps used.

Would be helpful to get a command to delete all at once or at least all not used after a specific date. Most of the entries are old stuff no longer used.

case 06348145 created for the issue.

 Hi LHerzog  Thanks for sharing the case ID. A note has been added to the case for this thread. Cc emmosophos 

thank you Vishal_R 

GES Support cleaned up old entries from the database but there is more to do because this did not solve the issue. Support is currently doing a good job on this topic.

I just find it irritating that this requires custom troubleshooting - there should be many customers with synchronized security that have seen more than 10k application paths during the operation time of their firewalls and endpoints. So I expected there should  be a clear guide how to resolve the issue.

From my opinion you should dynamically clean up the oldest applications when 10k limit is reached.

New entries overwrite the oldest. Probably in batches. Range full -> new app discovered -> clean the oldest 10 apps and resort them -> write the new app to db -> have 9 free slots

This should be mentioned in documentation.

Will post an update when the job is done.

There is a switch to cleanup old / unused apps: Do you have this enabled? 

That's nice! The feature already exists...

https://docs.sophos.com/nsg/sophos-firewall/19.5/Help/en-us/webhelp/onlinehelp/AdministratorHelp/SophosCentral/CentralSynchronizationSACOverview/index.html

Thank you Lucar Toni!

When we're done with the manual work, I'll enable this. Hopefully, a cleanup task does not restart heartbeat service each time...? Will this be synced to the Auxiliary node also?

As far as i can remember, it will cylce in the night and cleanup the database without restarting the database. 

So this issue is already known, therefore we build this option for bigger customers.

Because as you likely can image, 10k applications is not easily reachable for smaller appliances, but can grow or be reached by bigger customers. 

Cleanup is will not restart the heartbeat.

Yes, data sync expected to happen on Auxiliary node.