Why 4k native drives haven't arrived

Until someone demonstrates otherwise I will continue to state that 4K native drives are still not on the open market. And the reason for this is quite logical if given a moment of thought.

Advanced Format 512e drives, drives with 4K native sectors but 512B sectors presented to the host, fully accomplished the goal of the spinning disk drive manufacturers. That goal was simply to pack more data per platter, on average about 11% more, by reducing the amount of bits consumed for ECC. I.e. they can sell a lager capacity drive using the same hardware as a native 512 byte/sector drive, or with some drive capacities, reduce the number of platters while maintaining the same capacity, thus reducing component and production cost, and hopefully retail price.

The physical sector size presented to the host is irrelevant to the drive manufacturers, given the singular goal above. Switching to a native 4K sector does not benefit the manufacturers. At the current time it actually will cause them tremendous problems.

If they were to put 4K native drives on the market today, many millions of Windows XP users would buy the drives, ignoring, or simply not reading far enough to find the "4K native" warnings. They then return the drives when they don't work, causing great ill will, bad reviews tarnishing manufacturers' reputations, and decreasing repeat business. Thus native 4K drives will not be on the open market until the manufacturers are comfortable that most legacy machines have been retired, eliminating the possibility of the scenario above.

The first units of native 4K drives will be, or possibly already have been, to OEMs who exercise control over which systems and disk arrays in which the drives will be installed. This prevents such a problem in the enterprise space, as purchasers typically rely on their vendors to do compatibility matching. Enterprise OEMs have long maintained such product compatibility databases.

Stan

I fully agree with your analysis. I am sure there are lots of BIOS, controllers etc that have lots of bugs with 4k native drives, so even if the OS and tools supports it, there will be other issues.

On the other hand, if the 512 byte sectors are properly aligned so that the 4k page size always are written as 8 contigous sectors, then what does native 4k sector really give us? Lower IRQ rates? This is probably an additional reason why they haven't arrived. It's a complete ecosystem and taking the first step has huge downsides.

Mikael Abrahamsson    email: [email protected]

I'm confused how Windows XP is relevant to the latest top-grade server hard-drives. And vice versa =)

I would have thought it's more likely for Windows XP users not to buy hardware components, given that clearly those users are not upgrade savy in any shape or form.
And even if one would buy a 4k native drive, i would have thought there'd be other obstacles in trying it out e.g. not having a SATA port =) considering it was created two years after Windows XP =) or since a 4k native drive is likely to be 2TB+ in size, GPT support is required to address the whole drive...

The target market of 4k native drives at the moment is not consumer market, where SSDs are dominating as "premium" components. Instead the target market for 4k native drives is server / enterprise with never-decreasing storage requirements.

FYI here is Microsoft policy with respect to native 4k support matrix:
http://support.microsoft.com/kb/2510009

Most Segate/Baracuda units of capacities close to 1TB and higher are 4k native. And since ~2010-2012 there has been an ever increasing amount of request for support of native 4k in linux installers which is as good indication of the uptake as it gets.

I guess eventually it will also tickle down to more conventional hardware setups.

Regards,

Dimitri.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Actually, 4k native drives can use MBR up to 16 TB. The limitation on the MBR partition table is 32 bits to address sectors, so larger sectors => higher addressable size. Some of the fakeraid controllers on the market take advantage of this fact and report a 1k or 2k physical sector size to allow MBR partitioning on larger arrays.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJS4Hw2AAoJEI5FoCIzSKrwX5YH/joOvhkO9wHpZOgbCfPhTvzC Zax4nNuv9m0jAHzx+D0jqWzq/maf6ju0Xu3Sc49Iyy1ZfCeNXe64KrLnXyF2He9W TMMqDoAe0pIqdSxMtLr/1G/kJa1pxIcaoYvtc2val3MPxz/dw32XFk60fyBBSGYC n5f2Hs1TB4H0uo1gE4ywtN+fLtAQoqhZpHYIb0rS26zdwMJ3ho3qKwJeONy4FkhQ pKh4TTJBBMKe/Iz4v4+5RP+yq3Jlx860c+CMHq8amtGHPiBmGBL4WpE6lul/MHGn WBBv0g/VNd5M7K9Q/g01sIIRwyFisEImMmvamC9ctsUlcFzMEtkRfRyM1C+2SfU= =bHlp
-----END PGP SIGNATURE-----

Stan,

Stan> Advanced Format 512e drives, drives with 4K native sectors but Stan> 512B sectors presented to the host,

Ignoring ECC, legacy/native drives have a 1:1 mapping between logical and physical block sizes (512/520/528 bytes).

512e drives have a 512-byte logical block size. That's what the host operating system uses for addressing purposes when filling out the command to the disk. Internally, they use 4096-byte physical blocks on media.

Drives with 4096-byte logical *and* physical blocks are slowly becoming available. These drives are referred to as 4Kn (4K native) drives. So be careful about using the term "native" when referring to the physical sector size.

Linux supports drives with logical block sizes up to the system page size. This means we support 4Kn drives and have for over a decade. DASD on the mainframe is 4Kn, for instance. And there are a bunch of SAN devices and SSDs out there that also report themselves as 4Kn. So devices absolutely exist and are available.

4Kn harddrives are harder to come by, however. SAS/FC drives are available formatted as 4Kn when you order them. Some 512n drives can be reformatted. But you won't find 4Kn formatted drives in retail.

4Kn SATA works fine in Linux as well but has failed to get any traction. Mainly because there is no win for the user. Just lots of pain.

Stan> The physical sector size presented to the host is irrelevant to Stan> the drive manufacturers, given the singular goal above. Switching Stan> to a native 4K sector does not benefit the manufacturers. At the Stan> current time it actually will cause them tremendous problems.

The drive vendors pushed 4Kn for years and years. The problem was that to the host there is no benefit whatsoever. Just lots of pain throughout the entire I/O stack (BIOS, OS, HBA ROMs, RAID controller firmware). And no win. None.

So the drive vendors begrudgingly did 512e as a transitional thing. But they would like nothing more than killing off read-modify-write handling in their firmware/ASICs.

We are sticking with 512-byte logical/physical blocks for server workloads for several reasons. First of all it's important to have predictable performance. The read-modify-write cycles for misaligned writes on 512e drives can severely impact performance.

The second reason is data integrity preservation. None of the consumer 512e drives feature protection against sibling block corruption during read-modify-write. The nasty thing here is that a partial block write can end up garbling logical blocks within the 4KB physical sector that were not part of the failed I/O request. This is an absolute no-go from a data integrity perspective.

Therefore server drives have two options: Native (512n up to a certain capacity point, 4Kn for larger drives), or 512e with flash, supercaps or other tech that'll allow the drive to complete a partial block write during power failure. Both are out there.

Stan> Thus native 4K drives will not be on the open market until the Stan> manufacturers are comfortable that most legacy machines have been Stan> retired, eliminating the possibility of the scenario above.

Actually, >2TB USB drives typically expose 4Kn to the host. For that reason there are already problems with XP and big drives.

PS. See also: https://oss.oracle.com/~mkp/docs/linux-advanced-storage.pdf

Martin K. Petersen	Oracle Linux Engineering

My exact statement above leaves no doubt. But your point is duly noted, and I'll make sure I use "logical" and "physical" in the future.

I agree for the most part, and I was a vocal critic of the whole 4K push when it started, and especially of 512e. But as the throughput of drives increases, 4K host transfers with 4Kn drives will pay some performance dividends, though obviously nothing dramatic. So it's not completely bleak.

I'm sure they would but is this a high priority? RMW handling was a small price to pay for the increased platter density they were after. And now that most modern OS partitioning tools align to 1MB this isn't a performance issue for the user. Does the RMW code occupy a huge amount of the firmware space on the drive, or continual sink of engineering dollars with each new drive model?

I'm of the impression than 512e is done, is taking no additional money out of the drive vendors' pockets, yet increasing net dollars due to larger drive capacities, and fewer platters needed on some smaller drives. This is why I said there is little motivation on the part of the drive vendors to continue pushing 4Kn drives.

Whether 512B or 4KB it's obviously preferable to everyone to have the logical and physical block sizes match--vendors, OS kernel programmers, users.

Note that I've presented counterpoints here simply for technical discussion. I'm no fan of 512e, never have been, never will be, have never used a 512e drive, and never will, not if I can void it. I'm still a 512n kinda guy.

http://support.microsoft.com/kb/2510009

Windows 7 doesn't support 4Kn drives either. Up to now I thought it was limited to XP. Since these two versions of Windows make up ~80% of the installed MS Windows base, putting 4Kn USB drives on the market *is* suicide.

Interesting read. Are the suggested IDENTIFY DEVICE responses simply a reprint of the ATA/SCSI standards, or are these return values Linux specific, as the paper seems to suggest? I assume they're standard, as the vendors would most likely code for Windows if it required different values.

Stan

Stan> I'm sure they would but is this a high priority? RMW handling was Stan> a small price to pay for the increased platter density they were Stan> after. And now that most modern OS partitioning tools align to Stan> 1MB this isn't a performance issue for the user. Does the RMW Stan> code occupy a huge amount of the firmware space on the drive, or Stan> continual sink of engineering dollars with each new drive model?

It certainly takes up resources and puts constraints on the inner workings of the drive firmware. They really don't want to be in the RMW business but I doubt that's going to change.

SMR drives are closer to tape and a big departure from decades of harddrive behavior. The current plan is to have cheap drive models that are essentially glorified tapes and where the OS and filesystem have to explicitly manage the zones. Misaligned or I/Os requiring RMW will simply be rejected.

More expensive and backwards-compatible SMR drives will be doing the RMW transparently in firmware. However, this comes at a much higher cost than for 512e. SMR zones are 256MB - 1GB. That's a big chunk of stuff to RMW!

My hunch is that in reality we'll land somewhere in-between the two approaches like we did for 512e. In Linux we essentially treat 512e drives as 4Kn in the I/O stack. And we are careful to get alignment right. But legacy applications that rely on 512-byte accesses (using direct I/O for instance) still work.

I think we'll see something similar with SMR. We'll query the drive topology, and filesystems that are conductive to the SMR approach like btrfs will properly align to the zones and only append to them. Whereas legacy filesystems will resort to letting the drive deal with the horrors.

Stan> This is why I said there is little motivation on the part of the Stan> drive vendors to continue pushing 4Kn drives.

They are pushing pretty hard, actually, but there is a lot of inertia and legacy hardware/software out there. The industry 4Kn transition was originally supposed to be completed around late 2006!

Stan> Windows 7 doesn't support 4Kn drives either. Up to now I thought Stan> it was limited to XP. Since these two versions of Windows make up Stan> ~80% of the installed MS Windows base, putting 4Kn USB drives on Stan> the market *is* suicide.

I believe Windows treats USB and SATA/SCSI differently. But I have no personal experience in the Windows department.

Off-the-shelf 4TB USB drive:

# lsscsi | grep sdc
[10:0:0:0]   disk    Seagate  Backup+ Desk     050B  /dev/sdc
# sg_readcap -l /dev/sdc | grep length
   Logical block length=4096 bytes

(I'm guessing it's actually a 512e drive and that it's the SATA-USB bridge that makes it look like 4Kn to the host. But who knows?).

Stan> Interesting read. Are the suggested IDENTIFY DEVICE responses Stan> simply a reprint of the ATA/SCSI standards, or are these return Stan> values Linux specific, as the paper seems to suggest?

The document describes the drive parameters I key off of in sd and libata. We only use a small subset of what T10/T13 allows in Linux. The document is what we share with drive vendors to make sure they implement the knobs Linux needs correctly.

Martin K. Petersen	Oracle Linux Engineering

[ ... ]

Some HW RAID host adapters can be configured to offer the host 4KiB sectors too.

But as mentioned later on they have been available for years. Also I remember buying some old generation of SATA WD Green drive that had 4096 logical sectors, quickly superceded by emulated drives in later updates, as major OS support some years ago was not quite there yet.

There are some obvious wins to addressing in 4KiB sectors, the most important of which is the avoidance of RMW as a possibility.

But there is another important short term win, as older storage protols and formats relate maximum disk or partition or filetree capacity to maximum number of sectors addressable, which is often fixed at below 2^32 or 2^31, so addresses in units of 4KiB sectors allow 8 times more capacity than 512B ones.

This is particularly important for drives partitioned using MBR partitions with FAT32 filetrees in them, which is why this is the case especially for USB drives:

As for example here:

http://kb.netgear.com/app/answers/detail/a_id/22039/~/readyshare-and-usb-hard-drives-with-4kb-sector-size

and ingeneral in several reports here:

http://www.google.com/search?btnG=SEARCH&num=100&as_epq=Logical+Sector+size+4096+bytes

Peter> There are some obvious wins to addressing in 4KiB sectors, the Peter> most important of which is the avoidance of RMW as a possibility.

There are also drives where you can disable RMW explicitly. I.e. the drive appears as 512e but will reject any misaligned non-multiple of 4K.

The downside to 4Kn is that it breaks applications using direct I/O that assume that the sector size is 512 bytes. Which essentially means all of them. Solaris opted to do RMW in software to overcome that issue. Whereas we are trying to get application writers to fix their code...

Peter> But there is another important short term win, as older storage Peter> protols and formats relate maximum disk or partition or filetree Peter> capacity to maximum number of sectors addressable, which is often Peter> fixed at below 2^32 or 2^31, so addresses in units of 4KiB Peter> sectors allow 8 times more capacity than 512B ones.

Yep. Although in Linux we use 512-byte sectors throughout the I/O stack regardless of the device's logical block size.

Martin K. Petersen	Oracle Linux Engineering