Disk fragmentation

Hi Detlav,

I agree. This is why I state clearly that my users should not fill beyond 60% capacity. In reality that's hitting 70 to 75%.

If you draw an arc of performance against usage, you'll see that at around 66% (of disk platter usage - not capacity), the curve rapidly increases exponentially until we hit around 98% then it tails off.

Unfortunately, NTFS and FAT aren't very good at preventing fragmentation so you get the problem even though the FS has no real need to fragment. This is mostly down to rapid movement of files and journalising (for NTFS - so the problem is worse on NTFS than FAT). You can see a disk that's only 20% full show very heavy fragmentation problems in relatively no time at all by copying a folder of around 100Mb's total size containing 100 to 200 files. If you copy this folder over and over back and forth between two directories, a 100GB disk with 20GB used copying 100MB's 300-400 times will show up as heavily fragmented. As you increase capacity, you'll see the fragmentation issue occurs much sooner reducing from 400 times at 20% to around 100 times at 60%. Once you hit around 30% fragmentation, you'll see that the entire disk structure is now utilised and the only space left is the gaps between files with no contiguous space available.

To prevent the above, you need to control the amount of copying going on and reduce it as much as possible. If the Sophos update can be rearranged such that the downloaded IDE gets integrated directly and safely into the install folder (that's the c:\program files\sophos\autoupdate\cache\SavXP folder) and the process doesn't rely on copying this entire folder out, integrating and copying back, then fragmentation caused by the 5-10 updates per day will dramatically reduce.

I have little else going on on the local users disk as all data is stored up on servers with roaming profiles so the users own movements of data are not causing problems. Looking carefully at the disk clusters I'm able to see that the SAV updates are the fragmentations. Once a disk is fragmented, the update cycle slows the machine down dramatically until it becomes basically unusable for around 2-3 minutes and the reason for the slowdown is the disk thrash that's now going on while it copies the cache folder, integrates the download and copies back. The actual install after that takes just a few moments.

Matt

Hi Detlav,

I agree. This is why I state clearly that my users should not fill beyond 60% capacity. In reality that's hitting 70 to 75%.

If you draw an arc of performance against usage, you'll see that at around 66% (of disk platter usage - not capacity), the curve rapidly increases exponentially until we hit around 98% then it tails off.

Unfortunately, NTFS and FAT aren't very good at preventing fragmentation so you get the problem even though the FS has no real need to fragment. This is mostly down to rapid movement of files and journalising (for NTFS - so the problem is worse on NTFS than FAT). You can see a disk that's only 20% full show very heavy fragmentation problems in relatively no time at all by copying a folder of around 100Mb's total size containing 100 to 200 files. If you copy this folder over and over back and forth between two directories, a 100GB disk with 20GB used copying 100MB's 300-400 times will show up as heavily fragmented. As you increase capacity, you'll see the fragmentation issue occurs much sooner reducing from 400 times at 20% to around 100 times at 60%. Once you hit around 30% fragmentation, you'll see that the entire disk structure is now utilised and the only space left is the gaps between files with no contiguous space available.

To prevent the above, you need to control the amount of copying going on and reduce it as much as possible. If the Sophos update can be rearranged such that the downloaded IDE gets integrated directly and safely into the install folder (that's the c:\program files\sophos\autoupdate\cache\SavXP folder) and the process doesn't rely on copying this entire folder out, integrating and copying back, then fragmentation caused by the 5-10 updates per day will dramatically reduce.

I have little else going on on the local users disk as all data is stored up on servers with roaming profiles so the users own movements of data are not causing problems. Looking carefully at the disk clusters I'm able to see that the SAV updates are the fragmentations. Once a disk is fragmented, the update cycle slows the machine down dramatically until it becomes basically unusable for around 2-3 minutes and the reason for the slowdown is the disk thrash that's now going on while it copies the cache folder, integrates the download and copies back. The actual install after that takes just a few moments.

Matt