heartbeat log: Cannot create ID for application, because appId range is exhausted. Application will be ignored.

This indicates all the 10k appids are already used. No more applications detected on endpoints will be reported by heartbeat. The cause could be a particular application that is being detected uniquely everytime. Heartbeat service should continue to run. However please confirm if heartbeat service is still running. To investigate further we would need access to the device.

This indicates all the 10k appids are already used. No more applications detected on endpoints will be reported by heartbeat. The cause could be a particular application that is being detected uniquely everytime. Heartbeat service should continue to run. However please confirm if heartbeat service is still running. To investigate further we would need access to the device.

Thank you.

We have some 3,5k static apps and a unknown number of synced apps. Can I count them somewhere?

Some of them are used very frequently as I can see from the counters.

30 apps per page would mean ~ 9780 apps with the 326 pages on GUI.

heartbeatd is running. We notice users sometimes lose their heartbeat / authentication on the firewall. maybe that is somehow in relation to that issue. normally a reboot brings them back in service.

Every new version / bin path of a program get's a new entry in the synced apps list. So it is exceptable that the list some day exceeds 10k AppIDs. Probably we need to delete the entire list and start to relearn the apps used.

Would be helpful to get a command to delete all at once or at least all not used after a specific date. Most of the entries are old stuff no longer used.

case 06348145 created for the issue.

 Hi LHerzog  Thanks for sharing the case ID. A note has been added to the case for this thread. Cc emmosophos 

thank you Vishal_R 

GES Support cleaned up old entries from the database but there is more to do because this did not solve the issue. Support is currently doing a good job on this topic.

I just find it irritating that this requires custom troubleshooting - there should be many customers with synchronized security that have seen more than 10k application paths during the operation time of their firewalls and endpoints. So I expected there should  be a clear guide how to resolve the issue.

From my opinion you should dynamically clean up the oldest applications when 10k limit is reached.

New entries overwrite the oldest. Probably in batches. Range full -> new app discovered -> clean the oldest 10 apps and resort them -> write the new app to db -> have 9 free slots

This should be mentioned in documentation.

Will post an update when the job is done.

There is a switch to cleanup old / unused apps: Do you have this enabled? 

That's nice! The feature already exists...

https://docs.sophos.com/nsg/sophos-firewall/19.5/Help/en-us/webhelp/onlinehelp/AdministratorHelp/SophosCentral/CentralSynchronizationSACOverview/index.html

Thank you Lucar Toni!

When we're done with the manual work, I'll enable this. Hopefully, a cleanup task does not restart heartbeat service each time...? Will this be synced to the Auxiliary node also?

As far as i can remember, it will cylce in the night and cleanup the database without restarting the database. 

So this issue is already known, therefore we build this option for bigger customers.

Because as you likely can image, 10k applications is not easily reachable for smaller appliances, but can grow or be reached by bigger customers. 

Cleanup is will not restart the heartbeat.

Yes, data sync expected to happen on Auxiliary node.

Probably someday someone will look here for the resolution on the full appID range, so I write some more words:

There may be some problems with the automatic cleanup. in theory, it probably does what is written in the documentation.

But this has been discussed during our support case (06348145) and it could not be ruled out, that the cleanup really makes old, deleted  appID's usable. During our manual deletion, we found out, that deleted appID's are no used by heartbeat for new apps, because it simply wants to use the highest appID number. In the end, we need to flush multiple db tables now completely - so delete all ever learned apps and start from scratch. Not that we've used the apps for something, but other customers may. Anyway it is some kind of rough handling of an expectable issue.

Case notes:

Can you please confirm the automatic cleanup will work as intended and does not have the issues we're dealing with in this case?

From help: "Clean up application database: Sophos Firewall can automatically clear applications detected before a certain time period. It then runs a daily check for these applications and deletes them in batches of 100 every five minutes. Applications are also deleted from application filter policies if they were added individually."

To me that reads like exactly what we were doing, except removal from app filter policies.

 So let's imagine: we have a blank table tbleacapplications now. After 1 Year we have 1000 app id's in the table. Then we have automatic cleanup enabled to delete all older than 6 month. To make it simple, let's say it finds only 10 app id's older than 6 months. They are the app id's 100-105 and 200-205. They get deleted. Fine.

From your mails* I would now think, the next new application seen by the firewall will get id 1001, not 100 (which is free now)

*= The current design / code implementation simply checks what is the latest appid which is in use and created new appid entries after this until it reaches the max appid which is 19999.