Maximizing Windows File Systems Performance

1. Windows File Systems and Improving Performance APCUG 1/5/2008 Greg Hayes Raxco Software ghayes@raxco.com

2. Introduction • Company Background • Raxco founded 1978 • Largest provider of OpenVMS system management software • Spun off Axent Technologies (now part of Symantec) in 1996 • First Windows product released in 1998

3. Introduction • Company Background • Microsoft Gold Certified Partner • Microsoft ISV • Member of Microsoft Development Network • Software used by some of the largest companies and government organizations in the world

4. Introduction • Speaker background • System manager for 10+ years • Raxco customer for 10 years • Started at Raxco in 1996 • Manager, Technical Solutions • Microsoft MVP 2003-2007 - Windows File Systems • Online communities • Speaking/Education

5. Topic Background • Who remembers their first PC? • PC hardware performance continues to rise • CPU clock speeds > 2Ghz • 2GB RAM • 500GB+ capacity hard drives • Limiting factor is still disk I/O • Anything that speeds up access to disk improves performance

6. Windows File Systems • FAT16 • FAT32 • exFAT (new) • NTFS

7. Windows File Systems • FAT16/32 The File Allocation Table is created when you format a FAT drive and contains information about all of the files found on the drive and where they logically reside. There are actually 2 FATs per partition.

8. Windows File Systems – FAT16 • Benefits: • Faster performance on smaller partitions - less than 4GB • Best Performance for Pagefile • Drawbacks: • Little or no security • High rate of file system corruption • Limits on partition size (FAT16 – 4GB)

9. Windows File Systems – FAT16 • FAT16 • 4GB FAT16 partition has a cluster size of 64k. • Wastes space as even the smallest files take up a full cluster. • Maximum file size is 2GB • Files per volume 65,536

10. Windows File Systems – FAT32 • FAT32 • Maximum file size is 4GB • Allows smaller cluster sizes to be used – resulting in less space “waste” • Maximum partition size is 32GB (as formatted by XP/Vista/2K3) • Up to 2TB can be written/read (XP/Vista/2K3) • Files per volume ~4,177,920

11. Windows File Systems – exFAT • exFAT • “Extended FAT” • Designed for flash media – exchange media with PCs. • Free space bitmap for faster allocation • Maximum theoretical file size is 16EB • More than 1000 files per folder

12. Windows File Systems - NTFS • Journaling/Transaction Based File System • “Self Describing” File System • Metadata - data that describes data

13. Windows File Systems - NTFS • Benefits: • Better security • Better performance with large partitions – greater than 4GB • Better resistance to file system corruption • Drawbacks: • Can’t natively boot to MSDOS and get access to partition

14. Windows File Systems - NTFS • NTFS • Maximum file size is approx 16TB • Maximum partition size is 2TB – 256TB (depends on cluster size used) • Files per volume 4,294,967,295

15. Windows File Systems - NTFS • NTFS • Disk Quotas (Win2K and newer) • File/Folder compression built into the file system • Encryption built into the file system (Win2K and newer) • Volume Shadow Copy (VSS – WinXP and newer) • txNTFS

16. NTFS Metadata • $MFT - Master File Table (0) • The $MFT contains at least 1 record for each file that exists on the partition. • $MFTMirr – Master File Table Mirror (1) • Duplicate of the 1st 4 records of the $MFT • Used for redundancy/recoverability

17. NTFS Metadata • $LogFil – NTFS Transaction Log (2) • Updates to the file system first get posted to the transaction log • Gives NTFS its “self-healing” abilities • $Volume – Volume Information (3) • Volume Label/Version

18. NTFS Metadata • $AttrDef – Attribute Definitions (4) • Names, Numbers, Descriptions • $Root – (5) • Root folder on the drive. (I.e. C:\)

19. NTFS Metadata • $Bitmap – Cluster Bitmap (6) • Indicates whether a cluster is used or empty • Uses this to quickly find free space • $Boot – Partition Boot Sector (7) • Location of the partition boot sector • Bootstrap loader code if bootable drive

20. NTFS Metadata • $BadClus – Bad Cluster File (8) • Clusters the file system has reported as bad • Updated by CHKDSK • $Secure – Security Descriptors (9) • $Upcase – Upcase Table (10) • Translation of lowercase characters to their equivalent Unicode upper case characters

21. NTFS Metadata • $Extend – Optional Extensions (11) • User Change Journal • Quotas

22. Anatomy of a $MFT Record • Fixed size records – 1K • Attribute Records • $FILE_NAME • 8.3 file name and parent directory • Long File Name • $DATA • Extent information for file. Includes Logical Cluster Number (LCN) and Run Length • $DATA record contains information on file size

23. Anatomy of a $MFT Record • DiskEdit

24. MFT Reserved Zone • Created when partition is formatted • $MFT will “grow into” as needed • File system will keep free if possible • Defragmenters can not use free space inside of MFT Reserved Zone (NT4/Win2K)

25. MFT Reserved Zone • NT4 • Fixed Size • 12.5% by default • Can be increased up to 50% via registry key

26. MFT Reserved Zone • Windows 2003/2000/XP/Vista/2008 • Dynamically created every time partition is mounted • First record of $MFT to first non-free cluster - up to a default max of 12.5%

27. Logical vs Physical Clusters • Logical Clusters • File system level • Every partition starts at logical cluster 0 • No idea of hard drive technology in use • IDE • SCSI • RAIDx • # platters or read/write heads

28. Logical vs Physical Clusters • Physical Clusters • Hard drive level • Hard drive controller translates logical-to-physical and positions heads

30. Cluster Size and Performance • Smaller clusters • less wasted space • Worse performance – especially large files • Larger clusters • more wasted space • Better performance – especially large files

31. Conversion from FATx to NTFS • NT4/Win2K • Results in 512byte clusters • Not “the best” for performance - especially with video/image applications • WinXP/Vista • Will try to convert using 4k clusters. • Best for general file system performance.

32. OEM Vendors and Cluster Size • IBM/Dell/HP • Provide Software to install/configure Windows Server • Formats system drive with 512byte cluster • Absolute worse for system drive performance • Only way to convert is 3rd party tools and have server unavailable for extended period (HP SmartStart, Dell OpenManage, IBM ServerGuide)

33. Biggest Cause of Poor File System Performance • Fragmentation!

34. Fragmentation Causes • What causes fragmentation? • Occurs when files are created, extended or deleted • Happens regardless of how much free space is available (After XP/SP2 installation – 944 files/2943 fragments) • More than one Logical I/O request has to be made to the hard drive controller to access a file

35. Fragmentation Impacts • What does fragmentation do to my system? • Slows down access to files • Extra CPU/Memory/Disk resource usage • Some applications may not run • Slow system boot/shutdown • Audio/Video record/playback drops frames or “skips”

36. Measuring Impact of Fragmentation • Measuring the performance loss in reading a fragmented file

37. What Can I Do About Fragmentation? • You can’t stop fragmentation from happening (you can only slow it down)! • What you CAN do is to defragment

38. Defragmenting - Results • What does defragmenting do? • Locates logical pieces of a file and brings them together • Faster to access file and takes less resources • Improves read performance • Consolidates free space into larger pieces • New files get created in 1 piece • Improves write performance

39. Measuring Impact of Fragmentation • Measuring the performance difference in reading a contiguous file

40. Defragmenting - Issues to Consider • Safety, Safety, Safety! • No loss of data • No corruption of data

41. Defragmenting - Issues to Consider • Free Space • How much is enough? • Where is free space located? • Inside MFT Reserved Zone • Outside of MFT Reserved Zone • Consolidation of free space

42. Safety - Microsoft’s Defrag APIs • Provided as part of the operating system • Defragmenters do not actually move files • Integrated with caching system and Memory Manager • Performs all I/O synchronization • Allows even files in use to be defragmented

43. Safety - Microsoft’s Defrag APIs • Restrictions • Move Granularity • Free Space in MFT Reserved Zone • NTFS Cluster Size • Special OS files • Pagefile • Hibernate file • NTFS Metadata

44. Microsoft’s Defrag APIs • “Quirks” • Compressed files (NT4/Win2K) • File System Cache (NT4/Win2K) • Encrypted Files (Win2K) • NTFRS/DFS (Win2K) • VSS (Vista,Win2K3)

45. Free Space Consolidation • Allows new files to be created contiguously • Maintains file system performance longer • Requires less frequent defrag passes

46. Free Space Consolidation • Defragmenting files improves read performance • Free space consolidation improves write performance

47. Single Pass Defragmentation • As my dad always told me… • If you are going to do a job – do it right the first time • After defrag has completed, you don’t have to “wonder” if you need to run again • Fragmentation issue is solved – done!

48. File Placement Strategy • When you play chess, do you have a strategy or do you just start moving pieces around? • A good file placement strategy accomplishes several things: • Slows down re-fragmentation • Speeds up future defrag passes • Works with the operating system – not against

49. Windows Prefetch • What is Prefetching? • Windows monitors system boots and application launches • Uses information gathered to perform driver load optimization (speeds up boot process) • Uses information gathered to speed up application launches

50. Windows Prefetch • Windows stores this information in the \Windows\Prefetch folder • Layout.ini • Xxxx.pf • Every 3 days, Windows will automatically perform a “partial” defrag of the files indicated in layout.ini