We have been using Sophos successfully for over a year. However, we have recently noticed that workstations using Endpoint Security and Control v9.5 experience slowdowns once a day. We have our client update settings configured to get updates every 12 hours and it appears as though the workstation slowdowns are due to the client getting its updates. Has anyone else experienced this issue?
Sounds like the same issue Sophos have had all through it's life and one I've mentioned several times before.
Not 100% sure if 9.5 uses the same update procedure but I'm betting it does the following:
1. Match the cicsync.upd file checksum with the CID checksum file and if different start the update procedure else skip to end.
2. Clone the entire cache folder for the package to update to a tempsavxp folder (that's at least a 100mb copy)
3. Download and integrate the new ide file (which might only be a couple of kb's in size) into the tempsavxp folder.
4. hash checksum the entire tempsavxp folder to create a new cidsync.upd file
5. remove the existing savxp folder from the cache directory
6. Clone the entire 100+MB tempsavxp folder back into the savxp folder and remove the tempsavxp folder.
7. Start the msi installation from the savxp folder data (this step usually only take a few seconds and can be seen as the Sophos system tray shield going grey for a moment).
Problem with this somewhat antiquated way of doing things is that eventually, hard drives will become heavily fragmented. The more fragmentation you have, the slower that the copy in steps 2 and 6 takes and also the hashing in step 4. You can see all this in action on a machine if you want. Wait for your CID to get updated and then on a client machine, simply watch the folder c:\program files\sophos\AutoUpdate\Cache (win xp) or c:\ProgramData\Sophos\Autoupdate\Cache (Vista/7). Cause an update on the client and watch the folders grow and shrink. So much for the 'only a few kb each update statement' - humbug!
Been saying this for ages to Sophos and frustratingly, nothings changing except that the package size increases through each release causing machines to fragment more quickly. It would be considerably better if Sophos could find a way of downloading the few KB update data and integrate it directly into the cache\savxp folder directly rather than use the cloning method.
You'll get some relief for a short while running a full defragment on the client's HD. Hope this helps.
Matt
Matt,
Your breakdown of the updating sequence in the version 9.5 client is about right. Here’’’’s our description:
This approach was taken because it attempts to maintain a consistent file set on the machine that is “known good”. The checksum calculations are necessary to verify the integrity of the download.
We have been working on a couple of specific performance-related issues in the updating client. Recently, as part of our on-going maintenance cycle, we identified and fixed a potential problem where the http update process could use high CPU levels in certain circumstances during the download period. Also, our next major endpoint release is currently scheduled to include a change specifically to address some of the file-copying issues you mention. The approach we’’’’re using is to replace file copying with “hard links”, which means that the file isn’’’’t actually copied, but that two or more files shown in Windows Explorer actually refer to the same file on the disk. Using this technique, a file copy step actually just creates a new link to an existing file, which is quicker than copying the file. Also, since no actual copying is done, it should have less of an impact on disk fragmentation. We’’’’ve also made a slight improvement to the final file-move step, which is currently done one file at a time – now we do it in one step on the whole folder. Some of our tests suggest that this new version can reduce the CPU time taken by the download stage of the update cycle by about 40%. Here’’’’s an example:
Windows XP SP3, running on Pentium 4, 3Ghz, 512 Mb RAM.
Average time taken for an IDE update cycle using shipping product: 18.3 seconds.
Average time taken for an IDE update cycle using new client: 10.8 seconds.
Obviously, any improvement will depend on the details of the machine you’’’’re using, but it’’’’s our hope that you will notice an improvement in the next product release.
To answer one of your other points, we acknowledge that the overall package size has increased over recent releases. In fact, this is a trend that is likely to continue in forthcoming releases as we add significant new functionality into our endpoint product set. To help offset this, we are currently investigating options to enable use of http compression to reduce the size of downloads. However, we want to make sure that this doesn’’’’t add a performance hit on the client when the compressed files are unpacked. More fundamentally, we’’’’re also investigating options for moving our endpoint updating away from the existing file structure of the “Central Installation Directory”. We have an alternative format known as the Sophos Data Distribution System (SDDS), that uses a single file store where no copying is needed during an update cycle. It also brings other advantages such as being designed for use with http caching proxy servers to help reduce the network impact of file downloads, and finer control over which products and versions are updated.
Those are longer-term changes, but hopefully indicate to you that we are working on making improvements in this area of our product set.
Endpoint Product Management & Endpoint Deployment & Updating Team
Thanks Darren.
I would agree, hard links will solve the major slowness issues here. What people might not understand is that when disks spend time servicing requests, you WILL NOT see CPU usage rise and in fact, you may actually see it fall as the CPU idles waiting for IO requests to complete. When disks fragment, the service times increase dramatically and you end up with long request queues on the IO bus. The dead give away here that this is happening on your Sophos install is when you see the green bar flickering up and down on the Sophos shield and the hard drive light is light up solid. The transition of the release folder to temp and back again occurs during the 'Blue' phase of the shield and it's only when the shield goes grey that you're into the actuall installation phase of the update.
In my setup here, we're typically using P4 2.4Ghz machines runing XP SP2/3 and typically with 512MB or 1GB ram with IDE disks. We have update checks occuring every 30 minutes and when an update happens, a typical update will last around 30-60 seconds. When fragmented (say 2 to 3 weeks down the road on a 40GB disk with approx 15GB free), we will see update times meassured in minutes typically 90 to 180 seconds. Most noteable is the startup update which can take 5 to 10 minutes because the whole machine is dragged down by all the normal load up times and now the update cycle copying 100MB's of data back an forth.
Having almost no files move will cure most of the issues but I have to ask the obvious questions here: Hard links are just pointers to the original files, so why not just checksum the original files 'in place'? You have the new and old cidsync.upd files and therefore know which files are old and which are new, why not simply download new stuff to the temp folder and checksum files in current location then if all is good, move new stuff in and clear down anything not in the new cidsync.upd. This way you still guarantee 'known good' as much as using your current method but minimise all file system changes almost as much as possible?
Matt
