Skip to content

ntfs filesystem remake

Namjae Jeon linux-fsdevel lkml 11 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v9 Latest17 patches 4 repliesIn review
  2. v817 patches 0 repliesSuperseded
  3. v7 posting 2/214 of 17 patches 0 repliesSuperseded
Earlier versions · 8
VersionPostedPatchesRepliesStatus
v7 posting 1/217 patches1 replySuperseded
v614 of 16 patches22 repliesSuperseded
v514 patches35 repliesSuperseded
v414 patches6 repliesSuperseded
v3 posting 2/20 of 12 patches0 repliesSuperseded
v3 posting 1/20 of 12 patches4 repliesSuperseded
v211 patches34 repliesSuperseded
v111 patches14 repliesSuperseded

Changes in v9 from the cover letter

- Rebased onto 7.0-rc1 to follow changes during the merge window:
- kernel.h no longer includes hex.h.
- Use posix_acl_to_xattr() that allocates the buffer internally.
- Add generic_setlease to retain the ability to set leases.
- Stop using __getname().
- Fix failure to load attribute list on mount.
- Fix stale attribute pointer in MFT extension (Reported by Mobin Aydinfar).

Patches in v9 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/17Revert "fs: Remove NTFS classic"CREDITS Documentation/filesystems/ntfs.rst MAINTAINERS +44 more+29204 -50 replies
2/17fs: add generic FS_IOC_SHUTDOWN definitionsinclude/uapi/linux/fs.h+12 -02 replies
3/17ntfs: update in-memory, on-disk structures and headersfs/ntfs/aops.h fs/ntfs/attrib.h fs/ntfs/attrlist.h +22 more+2816 -27050 replies
4/17ntfs: update super block operationsfs/ntfs/super.c+1065 -14970 replies
5/17ntfs: update inode operationsfs/ntfs/inode.c fs/ntfs/namei.c+3638 -16170 replies
6/17ntfs: update mft operationsfs/ntfs/mft.c fs/ntfs/mst.c+1292 -12720 replies
7/17ntfs: update directory operationsfs/ntfs/dir.c fs/ntfs/index.c+2713 -13400 replies
8/17ntfs: update file operationsfs/ntfs/file.c+1022 -18580 replies
9/17ntfs: update iomap and address space operationsfs/ntfs/aops.c fs/ntfs/iomap.c+985 -15960 replies
10/17ntfs: update attrib operationsfs/ntfs/attrib.c fs/ntfs/attrlist.c fs/ntfs/compress.c+5248 -15312 replies
11/17ntfs: update runlist handling and cluster allocatorfs/ntfs/bitmap.c fs/ntfs/lcnalloc.c fs/ntfs/runlist.c+1310 -9820 replies
12/17ntfs: add reparse and ea operationsfs/ntfs/ea.c fs/ntfs/reparse.c+1515 -00 replies
13/17ntfs: update misc operationsfs/ntfs/bdev-io.c fs/ntfs/collate.c fs/ntfs/debug.c +8 more+775 -7050 replies
14/17ntfs3: remove legacy ntfs driver supportfs/ntfs3/Kconfig fs/ntfs3/dir.c fs/ntfs3/file.c +3 more+5 -1110 replies
15/17ntfs: add Kconfig and Makefilefs/Kconfig fs/Makefile fs/ntfs/Kconfig +2 more+60 -00 replies
16/17Documentation: filesystems: update NTFS driver documentationDocumentation/filesystems/index.rst Documentation/filesystems/ntfs.rst+129 -4350 replies
17/17MAINTAINERS: update ntfs filesystem entryCREDITS MAINTAINERS+7 -80 replies

Reviews trailers per patch

All 17 patches have a Reviewed-by or Acked-by.

PatchChristoph HellwigDarrick J. WongJan KaraAmir Goldstein
1/17A-b
2/17R-b R-b R-b
3/17R-b A-b
4/17R-b A-b
5/17A-b
6/17A-b
7/17A-b R-b
8/17R-b A-b
9/17A-b
10/17A-b
11/17A-b
12/17A-b
13/17A-b
14/17A-b
15/17R-b A-b R-b
16/17R-b A-b
17/17A-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 4 reviewers.

Read the v9 thread

Cover letter [PATCH v9 00/17] ntfs filesystem remake

v9:
- Rebased onto 7.0-rc1 to follow changes during the merge window:
- kernel.h no longer includes hex.h.
- Use posix_acl_to_xattr() that allocates the buffer internally.
- Add generic_setlease to retain the ability to set leases.
- Stop using __getname().
- Fix failure to load attribute list on mount.
- Fix stale attribute pointer in MFT extension (Reported by Mobin Aydinfar).

Introduction
============

The NTFS filesystem[1] still remains the default filesystem for Windows and The well-maintained NTFS driver in the Linux kernel enhances interoperability with Windows devices, making it easier for Linux users to work with NTFS-formatted drives. Currently, ntfs support in Linux was the long-neglected NTFS Classic (read-only), which has been removed from the Linux kernel, leaving the poorly maintained ntfs3. ntfs3 still has many problems and is poorly maintained, so users and distributions are still using the old legacy ntfs-3g.

The remade ntfs is an implementation that supports write and the essential requirements(iomap, no buffer-head, utilities, xfstests test result) based on read-only classic NTFS.
The old read-only ntfs code is much cleaner, with extensive comments, offers readability that makes understanding NTFS easier. This is why new ntfs was developed on old read-only NTFS base.
The target is to provide current trends(iomap, no buffer head, folio), enhanced performance, stable maintenance, utility support including fsck.

Key Features
============

- Write support:
Implement write support on classic read-only NTFS. Additionally, integrate delayed allocation to enhance write performance through multi-cluster allocation and minimized fragmentation of cluster bitmap.

- Switch to using iomap:
Use iomap for buffered IO writes, reads, direct IO, file extent mapping, readpages, writepages operations.

- Stop using the buffer head:
The use of buffer head in old ntfs and switched to use folio instead. As a result, CONFIG_BUFFER_HEAD option enable is removed in Kconfig also.

- Public utilities include fsck[2]:
While ntfs-3g includes ntfsprogs as a component, it notably lacks the fsck implementation. So we have launched a new ntfs utilitiies project called ntfsprogs-plus by forking from ntfs-3g after removing unnecessary ntfs fuse implementation. fsck.ntfs can be used for ntfs testing with xfstests as well as for recovering corrupted NTFS device.

- Performance Enhancements:

- ntfs vs. ntfs3:

* Performance was benchmarked using iozone with various chunk size.
- In single-thread(1T) write tests, ntfs show approximately 3~5% better performance.
- In multi-thread(4T) write tests, ntfs show approximately 35~110% better performance.
- Read throughput is identical for both ntfs implementations.

1GB file size:4096 size:16384 size:65536
MB/sec ntfs | ntfs3 ntfs | ntfs3 ntfs | ntfs3
─────────────────────────────────────────────────────────────────
read 399 | 399 426 | 424 429 | 430
─────────────────────────────────────────────────────────────────
write(1T) 291 | 276 325 | 305 333 | 317
write(4T) 105 | 50 113 | 78 114 | 99.6

* File list browsing performance. (about 12~14% faster)

files:100000 files:200000 files:400000
Sec ntfs | ntfs3 ntfs | ntfs3 ntfs | ntfs3
─────────────────────────────────────────────────────────────────
ls -lR 7.07 | 8.10 14.03 | 16.35 28.27 | 32.86

* mount time.

parti_size:1TB parti_size:2TB parti_size:4TB
Sec ntfs | ntfs3 ntfs | ntfs3 ntfs | ntfs3
─────────────────────────────────────────────────────────────────
mount 0.38 | 2.03 0.39 | 2.25 0.70 | 4.51

The following are the reasons why ntfs performance is higher
compared to ntfs3:
- Use iomap aops.
- Delayed allocation support.
- Optimize zero out for newly allocated clusters.
- Optimize runlist merge overhead with small chunck size.
- pre-load mft(inode) blocks and index(dentry) blocks to improve readdir + stat performance.
- Load lcn bitmap on background.

- Stability improvement:
a. Pass more xfstests tests:
ntfs passed 326 tests, significantly higher than ntfs3's 256. ntfs passed tests are a complete superset of the tests passed by ntfs3. ntfs implement fallocate, idmapped mount and permission, etc, resulting in a significantly high number of xfstests passing compared to ntfs3.
b. Bonnie++ issue[3]:
The Bonnie++ benchmark fails on ntfs3 with a "Directory not empty" error during file deletion. ntfs3 currently iterates directory entries by reading index blocks one by one. When entries are deleted concurrently, index block merging or entry relocation can cause readdir() to skip some entries, leaving files undeleted in workloads(bonnie++) that mix unlink and directory scans. ntfs implement leaf chain traversal in readdir to avoid entry skip on deletion.

- Journaling support:
ntfs3 does not provide full journaling support. It only implement journal replay[4], which in our testing did not function correctly. My next task after upstreaming will be to add full journal support to ntfs.

The feature comparison summary
==============================

Feature                               ntfs       ntfs3
===================================   ========   ===========
Write support                         Yes        Yes
iomap support                         Yes        No
No buffer head                        Yes        No
Public utilities(mkfs, fsck, etc.)    Yes        No
xfstests passed                       326        256
Idmapped mount                        Yes        No
Delayed allocation                    Yes        No
Bonnie++                              Pass       Fail
Journaling                            Planned    Inoperative
===================================   ========   ===========

References
==========
[1] https://en.wikipedia.org/wiki/NTFS
[2] https://github.com/ntfsprogs-plus/ntfsprogs-plus
[3] https://lore.kernel.org/ntfs3/CAOZgwEd7NDkGEpdF6UQTcbYuupDavaHBoj4WwTy3Qe4Bqm6V0g@mail.gmail.com/
[4] https://marc.info/?l=linux-fsdevel&m=161738417018673&q=mbox

Available in the Git repository at:
===================================
git://git.kernel.org/pub/scm/linux/kernel/git/linkinjeon/ntfs.git ntfs-next

v8:
- Acked-by from Christoph Hellwig for the whole series.
- Remove unneeded check for unsigned lowest_vcn found by smatch.
- Use LCN_HOLE instead of hardcoded -1 in runlist validation.

v7:
- Fix mount options indentation in Documentation.
- Add FMODE_CAN_ODIRECT in file open.
- Use -EOPNOTSUPP for unsupported EFS-encrypted I/O and add the comments.
- Remove lcn_seek_trunc field and O_TRUNC workaround codes.
- Move preallocated size trim to helper in file release.
- Use lowercase for reparse data structures.
- Split complex reparse validation into readable checks.
- Use early return in reparse tag validation.
- Consolidate memalloc_nofs_restore to single exit point.
- Remove useless headers from aops.c.
- Fix comment in ntfs_readahead.
- Rename mark_ntfs_record_dirty to ntfs_mft_mark_dirty.
- Move ntfs_mft_mark_dirty to mft.c.
- Remove empty aops.h.
- Simplify __ntfs_iomap_read_begin.
- Split ntfs_zero_range.
- Refactor __ntfs_write_iomap_begin, ntfs_write_iomap_end and ntfs_iomap_put_folio.

v6:
- Update ntfs.rst documentation.(Removed historical comparisons and temporal terms as per review feedback)
- Remove outdated Linux-NTFS project references.
- Add missing return comments for functions.
- Move comments above in structure.
- Replace wait_for_stable_page with filemap_write_and_wait_range.
- Dereference the ntfs_inode field instead of the casts.
- Remove malloc.h file and use kvmalloc and friends.
- Update help text for NTFS_FS_POSIX_ACL in Kconfig.
- Remove __always_unused directives.
- Change #ifndef to #ifdef in debug.
- Refactor ntfs_collate_ntofs_ulongs.
- Change type of key length for index.
- Remove verbose debug logs from ntfs_collate.
- Fix sparse warnings.
- Move initialization into declaration line.
- Don't type cast from pointer to void.
- Refactor ntfs_collate.
- Re-implement llseek using iomap.
- Don't allow acl mount option when config is disabled.
- Fix kerneldoc warnings.
- Update comments for magic constants.
- Use sizeof(unsigned char) instead of UCHAR_T_SIZE_BITS.
- Replace macros with inline helpers.
- Add/use generic FS_IOC_SHUTDOWN definitions.
- Remove flush_dcache_folio.
- Introduce address space operations for $MFT.
- Add comment for ntfs_bio_end_io.
- Add ntfs_get_locked_folio helper.
- Remove outdated ntfs_setattr comment.
- Remove unnecessary kernel-doc comments.
- Fix missing error handling for compressed/encrypted files in setattr.
- Refactor ATTR_SIZE handling into ntfs_setattr_size().
- Rely on iomap for direct I/O alignment state.
- Report advanced file attributes and DIO alignment in getattr.
- Move ntfs_attr_expand and ntfs_extend_initialized_size to ntfs iomap.
- Factor out ntfs_dio_write_iter from ntfs_write_iter().
- Remove unneeded IS_IMMUTABLE in ntfs_filemap_page_mkwrite.
- Remove regular file check in ntfs_fallocate.
- Change COMPRESS_CONTEXT to lowercase.
- Add the comment for hash multiplier.
- Fix potential deadlock when inode is freed.
- Fix generic/321 failure when acl is enabled.
- use bdev_rw_virt in ntfs_bdev_read.
- Remove mft_writepage from ntfs_writepages.
- let ntfs_mft_writepages call ntfs_write_mft_block instead of ntfs_mft_writepage.
- Move ntfs_write_mft_block into mft.c.
- Move ntfs_bdev_read/write into bdev-io.c
- Move ntfs_mft_writepages to mft.c.

v5:
- Update outdated comments to match implementation.
- Remove unused types.h and endians.h.
- Replace submit_bio_wait() with submit_bio().
- Fix lockdep warnings caused by the latest xfstets and scratch_mkfs_sized support.
- Rename ntfs_convert_folio_index_into_lcn() to lcn_from_index().
- Fix warnings reported by Smatch static checker.
- Fix typos patch description of MAINTAINERS.

v4:
- remove choice variable in fs/Kconfig and make ntfs and ntfs3 mutually exclusive in simpler way.
- Original revert commit includes MAINTAINERS and CREDITS and update ntfs entry in MAITAINERS and Anton's info in CREDITS.
- Original revert commit include documentation and update it instead of adding a new one.
- Fix generic/401 test failure and indicate that ntfs passed tests are a complete superset of those for ntfs3.
- Remove unnecessary comments and warning options from Makefile.
- Add patch description to original revert patch and the patch that remove legacy ntfs driver related codes in ntfs.
- Support timestamps prior to epoch (fix generic/258).
- Fix xfstests generic/683, 684, 686, 687, 688.

v3:
- Add generic helpers to convert cluster to folio index, cluster to byte, byte to sector, etc.
- Remove bio null check and ntfs_setup_bio().
- Remove unneeded extra handling from old ntfs leftover.
- Allow readahead for $MFT file.
- Change memcpy to memcpy_from_folio or memcpy_to_folio.
- Never switche between compressed and non-compressed for live inodes.
- Add the comments for iomap_valid and iomap_put_folio.
- Split the resident and non-resident cases into separate helpers.
- Use kmalloc instead of page allocation for iomap inline data.
- Use iomap_zero_range instead of ntfs_buffered_zero_clusters.
- Use blkdev_issue_zeroout instead of ntfs_zero_clusters.
- Remove 2TB limitation on 32-bit system.
- Rename ntfsplus to ntfs.
- Remove -EINTR handing for read_mapping_folio.
- Rename ntfs_iomap.c to iomap.c
- Revert alias for the legacy ntfs driver in ntfs3.
- Restrict built-in NTFS seclection to one driver, allow both as modules.
- Use static_assert() instead of the sizeof comments.
- Update the wrong iocharset comments in ntfs.rst.

v2:
- Add ntfs3-compatible mount options(sys_immutable, nohidden, hide_dot_files, nocase, acl, windows_names, disable_sparse, discard).
- Add iocharset mount option.
- Add ntfs3-compatible dos attribute and ntfs attribute load/store in setxattr/getattr().
- Add support for FS_IOC_{GET,SET}FSLABEL ioctl.
- Add support for FITRIM ioctl.
- Fix the warnings(duplicate symbol, __divdi3, etc) from kernel test robot.
- Prefix pr_xxx() with ntfsplus.
- Add support for $MFT File extension.
- Add Documentation/filesystems/ntfsplus.rst.
- Mark experimental.
- Remove BUG traps warnings from checkpatch.pl.

Appendix: xfstests Results on ntfs

Summary
- Total tests run: 787
- Passed: 326
- Failed: 38
- Not run / skipped: 423

Failed test cases and reasons
- Requires metadata journaling (34 tests)

    043, 056, 051, 057, 065, 066, 073, 104, 225, 335, 336, 341, 342, 343,
    348, 376, 388, 417, 475, 498, 502, 510, 526, 527, 530, 552,
    640, 690, 764, 771, 779, 782, 784, 785

- Others (4 tests)
094: Not supported by NTFS on-disk format (no unwritten extent concept). 563: cgroup v2 aware writeback accounting not supported. 631: RENAME_WHITEOUT support is required on ntfs.
787: NFS delegation test.

Namjae Jeon (17):
Revert "fs: Remove NTFS classic"
fs: add generic FS_IOC_SHUTDOWN definitions
ntfs: update in-memory, on-disk structures and headers
ntfs: update super block operations
ntfs: update inode operations
ntfs: update mft operations
ntfs: update directory operations
ntfs: update file operations
ntfs: update iomap and address space operations
ntfs: update attrib operations
ntfs: update runlist handling and cluster allocator
ntfs: add reparse and ea operations
ntfs: update misc operations
ntfs3: remove legacy ntfs driver support
ntfs: add Kconfig and Makefile
Documentation: filesystems: update NTFS driver documentation
MAINTAINERS: update ntfs filesystem entry

 CREDITS                             |    9 +-
 Documentation/filesystems/index.rst |    1 +
 Documentation/filesystems/ntfs.rst  |  159 +
 MAINTAINERS                         |    9 +
 fs/Kconfig                          |    1 +
 fs/Makefile                         |    1 +
 fs/ntfs/Kconfig                     |   47 +
 fs/ntfs/Makefile                    |   10 +
 fs/ntfs/aops.c                      |  263 ++
 fs/ntfs/attrib.c                    | 5425 +++++++++++++++++++++++++++
 fs/ntfs/attrib.h                    |  164 +
 fs/ntfs/attrlist.c                  |  289 ++
 fs/ntfs/attrlist.h                  |   20 +
 fs/ntfs/bdev-io.c                   |  117 +
 fs/ntfs/bitmap.c                    |  287 ++
 fs/ntfs/bitmap.h                    |  100 +
 fs/ntfs/collate.c                   |  146 +
 fs/ntfs/collate.h                   |   36 +
 fs/ntfs/compress.c                  | 1577 ++++++++
 fs/ntfs/debug.c                     |  171 +
 fs/ntfs/debug.h                     |   63 +
 fs/ntfs/dir.c                       | 1236 ++++++
 fs/ntfs/dir.h                       |   32 +
 fs/ntfs/ea.c                        |  942 +++++
 fs/ntfs/ea.h                        |   30 +
 fs/ntfs/file.c                      | 1161 ++++++
 fs/ntfs/index.c                     | 2117 +++++++++++
 fs/ntfs/index.h                     |  111 +
 fs/ntfs/inode.c                     | 3820 +++++++++++++++++++
 fs/ntfs/inode.h                     |  359 ++
 fs/ntfs/iomap.c                     |  870 +++++
 fs/ntfs/iomap.h                     |   23 +
 fs/ntfs/layout.h                    | 2346 ++++++++++++
 fs/ntfs/lcnalloc.c                  | 1047 ++++++
 fs/ntfs/lcnalloc.h                  |  134 +
 fs/ntfs/logfile.c                   |  778 ++++
 fs/ntfs/logfile.h                   |  245 ++
 fs/ntfs/malloc.h                    |   77 +
 fs/ntfs/mft.c                       | 2922 +++++++++++++++
 fs/ntfs/mft.h                       |   94 +
 fs/ntfs/mst.c                       |  194 +
 fs/ntfs/namei.c                     | 1695 +++++++++
 fs/ntfs/ntfs.h                      |  294 ++
 fs/ntfs/object_id.c                 |  158 +
 fs/ntfs/object_id.h                 |   14 +
 fs/ntfs/quota.c                     |   95 +
 fs/ntfs/quota.h                     |   15 +
 fs/ntfs/reparse.c                   |  573 +++
 fs/ntfs/reparse.h                   |   20 +
 fs/ntfs/runlist.c                   | 2066 ++++++++++
 fs/ntfs/runlist.h                   |   97 +
 fs/ntfs/super.c                     | 2770 ++++++++++++++
 fs/ntfs/sysctl.c                    |   55 +
 fs/ntfs/sysctl.h                    |   26 +
 fs/ntfs/time.h                      |   87 +
 fs/ntfs/unistr.c                    |  477 +++
 fs/ntfs/upcase.c                    |   70 +
 fs/ntfs/volume.h                    |  296 ++
 fs/ntfs3/Kconfig                    |   10 +-
 fs/ntfs3/dir.c                      |   10 -
 fs/ntfs3/file.c                     |   11 -
 fs/ntfs3/inode.c                    |   16 +-
 fs/ntfs3/ntfs_fs.h                  |   11 -
 fs/ntfs3/super.c                    |   59 +-
 include/uapi/linux/fs.h             |   12 +
 65 files changed, 36252 insertions(+), 118 deletions(-)
 create mode 100644 Documentation/filesystems/ntfs.rst
 create mode 100644 fs/ntfs/Kconfig
 create mode 100644 fs/ntfs/Makefile
 create mode 100644 fs/ntfs/aops.c
 create mode 100644 fs/ntfs/attrib.c
 create mode 100644 fs/ntfs/attrib.h
 create mode 100644 fs/ntfs/attrlist.c
 create mode 100644 fs/ntfs/attrlist.h
 create mode 100644 fs/ntfs/bdev-io.c
 create mode 100644 fs/ntfs/bitmap.c
 create mode 100644 fs/ntfs/bitmap.h
 create mode 100644 fs/ntfs/collate.c
 create mode 100644 fs/ntfs/collate.h
 create mode 100644 fs/ntfs/compress.c
 create mode 100644 fs/ntfs/debug.c
 create mode 100644 fs/ntfs/debug.h
 create mode 100644 fs/ntfs/dir.c
 create mode 100644 fs/ntfs/dir.h
 create mode 100644 fs/ntfs/ea.c
 create mode 100644 fs/ntfs/ea.h
 create mode 100644 fs/ntfs/file.c
 create mode 100644 fs/ntfs/index.c
 create mode 100644 fs/ntfs/index.h
 create mode 100644 fs/ntfs/inode.c
 create mode 100644 fs/ntfs/inode.h
 create mode 100644 fs/ntfs/iomap.c
 create mode 100644 fs/ntfs/iomap.h
 create mode 100644 fs/ntfs/layout.h
 create mode 100644 fs/ntfs/lcnalloc.c
 create mode 100644 fs/ntfs/lcnalloc.h
 create mode 100644 fs/ntfs/logfile.c
 create mode 100644 fs/ntfs/logfile.h
 create mode 100644 fs/ntfs/malloc.h
 create mode 100644 fs/ntfs/mft.c
 create mode 100644 fs/ntfs/mft.h
 create mode 100644 fs/ntfs/mst.c
 create mode 100644 fs/ntfs/namei.c
 create mode 100644 fs/ntfs/ntfs.h
 create mode 100644 fs/ntfs/object_id.c
 create mode 100644 fs/ntfs/object_id.h
 create mode 100644 fs/ntfs/quota.c
 create mode 100644 fs/ntfs/quota.h
 create mode 100644 fs/ntfs/reparse.c
 create mode 100644 fs/ntfs/reparse.h
 create mode 100644 fs/ntfs/runlist.c
 create mode 100644 fs/ntfs/runlist.h
 create mode 100644 fs/ntfs/super.c
 create mode 100644 fs/ntfs/sysctl.c
 create mode 100644 fs/ntfs/sysctl.h
 create mode 100644 fs/ntfs/time.h
 create mode 100644 fs/ntfs/unistr.c
 create mode 100644 fs/ntfs/upcase.c
 create mode 100644 fs/ntfs/volume.h
Signature · 1 line
2.25.1