Bye bye 1993 -- first major file system update to Windows in nearly a two decades lands.

In the latest post on its Building Windows blog, Microsoft Corp. (MSFT) shared some insight into how it's reinventing things on the storage side for Windows Server 8, the server-side counterpart to Windows 8.  Surendra Verma, Microsoft's Storage and File System team manager, broke down the improvements to the filesystem.

I. 1993 No More -- NTFS Gets Replaced

The biggest surprise is that Microsoft is drastically revamping the low-level New Technology File System (NTFS) code base, which was first introduced commercially with Windows NT back in 1993.  Since 1993, NTFS has grown in hodge podge fashion to its current form, incorporating the latest improvements.  Such an approach is inherently limited due to the burdens of legacy scraps of code intermingled throughout the code base.  

So Microsoft did the logical thing -- it went back to the drawing board, rewriting much of the low-level code.  The rewrite is significant enough that the engine ditches the "NTFS" moniker and adopts a new name, ReFS -- Resilient File System.  Its key goals are scalability, speed, and data integrity.

Microsoft describes the logic behind the madness, writing, "We didn’t start from scratch, but reimagined it where it made sense and built on the right parts of NTFS where that made sense. Above all, we are delivering this in a pragmatic manner consistent with the delivery of a major file system—something only Microsoft has done at this scale."

In other words the best parts of NTFS are still there, but the parts that were most loathed by Microsoft's engineers and customers are hopefully mostly gone and replaced with superior beasts.

ReFS keeps the NTFS API, but tosses much of NTFS's low level storage engine code.
[Image Source: Microsoft]

In its process of natural selection of its code base, Microsoft was careful to maintain API compatibility with the NTFS system's software/firmware API, so don't worry about the souped up engine being incompatible with your current generation hardware.

ReFS brings many improvements to the table.  It tries to space disk data in "stripes" on the memory space to improve access times on traditional disc-based hard drives.  It also supports checksumming for metadata -- which means that if a file gets corrupted, it's easier to identify and recover.

Initially the new file system will be available exclusively with Windows Server 8.  This follows with past file system introductions, which first came on the Server side, then crept into the main consumer Windows line.  

In other words, just to be clear -- Windows 8 will not have ReFS, Windows Server 8 will have ReFS.  Microsoft is also initially not supporting ReFS on external devices (USB sticks, external hard drives, etc.).  As with the consumer support, expect this to come with the next drop.

II. ReFS Handles Data Differently

The most fundamental change on the low level is that all on-system data is now stored in B+ trees (follow link for journal article, kindly provided by Microsoft for for the non-storage expert audience).  This data structure is very scalable and incorporates a new sort of key/value pair table (think STL map for you C-programming types).  The new system allows for greater code reuse and better data distribution, among other perks.
ReFS B+ Trees
ReFS adopts a new ubiquitous storage scheme -- B+ trees.  [Image Source: Microsoft]

Here's some specs about the limitations of data in Microsoft's ReFS implementation of B+:


Limit based on the on-disk format

Maximum size of a single file

2^64-1 bytes

Maximum size of a single volume

Format supports 2^78 bytes with 16KB cluster size (2^64 * 16 * 2^10). Windows stack addressing allows 2^64 bytes

Maximum number of files in a directory


Maximum number of directories in a volume


Maximum file name length

32K unicode characters

Maximum path length


Maximum size of any storage pool

4 PB

Maximum number of storage pools in a system

No limit

Maximum number of spaces in a storage pool

No limit

III. ReFS Fights Corruption

Another change is that the OS now writes metadata automatically, scrapping the journal-based "writing in place" of NTFS.  What this means is that if your system loses power corruption (so-called "torn writes" to metadata) is less likely to occur.

File checksumming is also added in a feature called "Integrity Streams". If your system loses power Windows 8 can peek at your file's checksums to see that it was corrupted and then restore the file to the original version that is allocated during the atomic write.  The write can then safely proceed with a decreased chance of corruption.

The downside of Integrity Streams is that data moves around on every file write, so for database users this is clearly not a good option.  So Microsoft has incorporated the ability to turn off its fancy new feature and revert to more traditional writes.

Other new features on the data integrity side include "scrubbing" to remove "bit rot".  Bit rot is something that can occur in NAND cells that have been written to too many times or even in traditional hard drives.  Since this form of corruption creeps in over time, alternate copies are typically already discarded by the time corruption is noticed.  To prevent this, a new lean system task will periodically run and hunt for bit rot.

Many of ReFS fancier anti-corruption features are incorporated into "Storage Spaces", Microsoft's storage-related virtualization utility.

III. Windows 8 and Windows Server 8 Almost Ready to Shake up the Market

Windows 8 (the consumer OS) is currently being tested in pre-beta form by developers and is set to land in finished form sometime late this year (like Oct. or Nov.).  A full-fledged beta is in the books for February (Microsoft is calling it a "Consumer Preview" this time around, but its essentially the same as Windows 7's beta).  

Windows Server 8 is currently in developer preview.  A beta build is suspected to have leaked on January 3, so a full corporate beta may soon be at hand.

Window 8 and Windows Server 8 have big shoes to fill as Windows 7 and Windows Server 2008 R2 were some of the most successful operating systems in Microsoft's history.  

In fact, Windows 7 sold faster than any operating system in history.  Sales numbers for Windows Server 2008 R2 are a bit harder to come by, but market research indicates that it held over 75 percent of server OS license sales, an impressive market share.

Past Building Windows blogs have revealed many Windows 8 features such as a less painful Windows Update processfaster bootsdecreased OS resource consumption, and improved file transfers, a streamlined upgrade process for the initial installation, switching to a primarily online sales distribution model, and the new "Windows Store" app store.

Both Windows 8 and Windows Server 8 feature support for the slick new tile-based user interface (UI) Metro UI.

Windows Server 8 is also shaking things up on the hardware front, continuing its predecessor's abandonment of support for 32-bit CPUs, and for the first time dropping support for Itanium CPUs.  Itaniums are a special kind of Intel Corp. (INTC) server CPU.  While they have traditionally sold at very small volume, they have been a high-margin, profitable endeavor for Intel.  Itanium has many fans who use the chips for mission-critical applications.

Intel Tukwila
Windows Server 8 nixes support for Itanium (pictured) and 32-bit CPUs. [Image Source: Intel]

While some swear there's nothing better than Itanium for certain applications, the low volume appears to be spelling the demise of the platform.  Both Microsoft and Red Hat Inc. (RHT) -- a top business Linux provider -- have thrown in the towel on Itanium, deciding that it's too unprofitable and expensive to support from a software side.

"I f***ing cannot play Halo 2 multiplayer. I cannot do it." -- Bungie Technical Lead Chris Butcher

Latest Blog Posts
T-Mobile Data Problems
Saimin Nidarson - Oct 20, 2016, 10:17 AM
IMEX America Trade Show
Saimin Nidarson - Oct 9, 2016, 10:00 AM

Copyright 2016 DailyTech LLC. - RSS Feed | Advertise | About Us | Ethics | FAQ | Terms, Conditions & Privacy Information | Kristopher Kubicki