ext4 changes for 3.15

Note: there will be a minor patch conflict since you included an earlier version of the"atomically set inode->i_flags in ext4_set_inode_flags()" in 3.14 bbefore you decided that set_mask_bits() wasn't a good interface to be exposing because people could too easily misuse it. The merge conflict is pretty obvious.

Let me know if you want me to send a separate cleanup patch which removes set_mask_bits(), or whether you just want to drop it from include/linux/bitops.h while fixing up the merge conflict.

Cheers,

- Ted

The following changes since commit 92e3b40537707001d17bbad800d150ab04e53bf4:

jbd2: fix use after free in jbd2_journal_start_reserved() (2014-02-17 20:33:01 -0500)

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git tags/ext4_for_linus

for you to fetch changes up to ad6599ab3ac98a4474544086e048ce86ec15a4d1:

ext4: fix premature freeing of partial clusters split across leaf blocks (2014-04-01 19:49:30 -0400)

----------------------------------------------------------------
Major changes for 3.14 include support for the newly added ZERO_RANGE and COLLAPSE_RANGE fallocate operations, and scalability improvements in the jbd2 layer and in xattr handling when the extended attributes spill over into an external block.

Other than that, the usual clean ups and minor bug fixes.

----------------------------------------------------------------
Dan Carpenter (1):
ext4: remove an unneeded check in mext_page_mkuptodate()

Darrick J. Wong (1):
ext4: merge uninitialized extents

Eric Sandeen (1):
ext4: remove unused ac_ex_scanned

Eric Whitney (5):
ext4: fix error return from ext4_ext_handle_uninitialized_extents()
ext4: silence warnings in extent status tree debugging code
ext4: delete path dealloc code in ext4_ext_handle_uninitialized_extents
ext4: fix partial cluster handling for bigalloc file systems
ext4: fix premature freeing of partial clusters split across leaf blocks

Fabian Frederick (1):
ext4: Add __init marking to init_inodecache

Jan Kara (1):
ext4: Speedup WB_SYNC_ALL pass called from sync(2)

Lukas Czerner (5):
ext4: translate fallocate mode bits to strings
ext4: Update inode i_size after the preallocation
ext4: refactor ext4_fallocate code
ext4: Introduce FALLOC_FL_ZERO_RANGE flag for fallocate
ext4: remove unneeded test of ret variable

Matthew Wilcox (2):
ext4: make ext4_block_zero_page_range static
ext4: fix comment typo

Maxim Patlasov (1):
ext4: avoid exposure of stale data in ext4_punch_hole()

Namjae Jeon (1):
ext4: Add support FALLOC_FL_COLLAPSE_RANGE for fallocate

Patrick Palka (1):
ext4: address a benign compiler warning

Rashika Kheria (1):
jbd2: mark file-local functions as static

T Makphaibulchoke (3):
fs/mbcache.c: change block and index hash chain to hlist_bl_node fs/mbcache.c: doucple the locking of local from global data
ext4: each filesystem creates and uses its own mb_cache

Theodore Ts'o (18):
ext4: clean up error handling in swap_inode_boot_loader()
ext4: add ext4_es_store_pblock_status()
ext4: don't calculate total xattr header size unless needed
ext4: make sure ex.fe_logical is initialized
ext4: avoid possible overflow in ext4_map_blocks()
jbd2: don't unplug after writing revoke records
jbd2: don't hold j_state_lock while calling wake_up()
jbd2: calculate statistics without holding j_state_lock and j_list_lock
jbd2: add transaction to checkpoint list earlier
jbd2: check jh->b_transaction without taking j_list_lock
jbd2: minimize region locked by j_list_lock in journal_get_create_access()
jbd2: minimize region locked by j_list_lock in jbd2_journal_forget()
jbd2: improve error messages for inconsistent journal heads
fs: push sync_filesystem() down to the file system's remount_fs()
ext4: only call sync_filesystm() when remounting read-only
ext4: kill i_version support for Hurd-castrated file systems
ext4: optimize Hurd tests when reading/writing inodes
ext4: atomically set inode->i_flags in ext4_set_inode_flags()

 fs/adfs/super.c             |   1 +
 fs/affs/super.c             |   1 +
 fs/befs/linuxvfs.c          |   1 +
 fs/btrfs/super.c            |   1 +
 fs/cifs/cifsfs.c            |   1 +
 fs/coda/inode.c             |   1 +
 fs/cramfs/inode.c           |   1 +
 fs/debugfs/inode.c          |   1 +
 fs/devpts/inode.c           |   1 +
 fs/efs/super.c              |   1 +
 fs/ext2/super.c             |   1 +
 fs/ext3/super.c             |   2 +
 fs/ext4/ext4.h              |  11 +-
 fs/ext4/ext4_jbd2.c         |  10 +
 fs/ext4/extents.c           | 818 ++++++++++++++++++++++++++++++++++++++++++++++++++--------
 fs/ext4/extents_status.c    |  28 +-
 fs/ext4/extents_status.h    |   9 +
 fs/ext4/inode.c             | 130 ++++++----
 fs/ext4/ioctl.c             |  24 +-
 fs/ext4/mballoc.c           |   7 +-
 fs/ext4/mballoc.h           |   4 +-
 fs/ext4/move_extent.c       |   5 +-
 fs/ext4/super.c             |  40 ++-
 fs/ext4/xattr.c             |  59 +++--
 fs/ext4/xattr.h             |   6 +-
 fs/f2fs/super.c             |   2 +
 fs/fat/inode.c              |   2 +
 fs/freevxfs/vxfs_super.c    |   1 +
 fs/fuse/inode.c             |   1 +
 fs/gfs2/super.c             |   2 +
 fs/hfs/super.c              |   1 +
 fs/hfsplus/super.c          |   1 +
 fs/hpfs/super.c             |   2 +
 fs/inode.c                  |  31 +++
 fs/isofs/inode.c            |   1 +
 fs/jbd2/commit.c            |  77 +++---
 fs/jbd2/journal.c           |  10 +-
 fs/jbd2/transaction.c       |  46 ++--
 fs/jffs2/super.c            |   1 +
 fs/jfs/super.c              |   1 +
 fs/mbcache.c                | 540 +++++++++++++++++++++++++++-----------
 fs/minix/inode.c            |   1 +
 fs/ncpfs/inode.c            |   1 +
 fs/nfs/super.c              |   2 +
 fs/nilfs2/super.c           |   1 +
 fs/ntfs/super.c             |   2 +
 fs/ocfs2/super.c            |   2 +
 fs/openpromfs/inode.c       |   1 +
 fs/proc/root.c              |   2 +
 fs/pstore/inode.c           |   1 +
 fs/qnx4/inode.c             |   1 +
 fs/qnx6/inode.c             |   1 +
 fs/reiserfs/super.c         |   1 +
 fs/romfs/super.c            |   1 +
 fs/squashfs/super.c         |   1 +
 fs/super.c                  |   2 -
 fs/sysv/inode.c             |   1 +
 fs/ubifs/super.c            |   1 +
 fs/udf/super.c              |   1 +
 fs/ufs/super.c              |   1 +
 fs/xfs/xfs_super.c          |   1 +
 include/linux/fs.h          |   3 +
 include/linux/mbcache.h     |  12 +-
 include/trace/events/ext4.h | 102 +++++---
 64 files changed, 1520 insertions(+), 505 deletions(-)

Btw, since I'm planning on getting to the filesystem pulls later today (or perhaps tomorrow), I wanted to check: are you ok with the ext4 parts of the cross-rename patches from Miklos?

They are currently at

git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git cross-rename

in case you want to refresh your memory.

I have this memory of having seen an acked-by from you last time they flew past, but looking at Miklos branch that isn't in the commit logs at least, so maybe my memory is crap. It wouldn't be the first time.

I think the concerns about renameat2() got all sorted out, and I'm planning on pulling them, but I want to make sure that there aren't any objections to them.

Al added to the cc for the same reason.

Linus

I've pulled in the cross-rename branch to my test branch and run a set of tests, and it passes. Unfortunately I don't believe Miklos contributed tests for renameat(2) to xfstests, so we don't have any on-point testing of renameat() and cross-rename, but it's at least not triggering any failures on the existing tests.

I've also reviewed the patches again, so:

- Ted

Actually Miklos has contributed xfstests tests. I'm not sure if they were
merged but I've seen them flying by.

Honza

Jan Kara <[email protected]>
SUSE Labs, CR

Yes:

http://marc.info/?l=linux-kernel&m=139523745403081&w=2

The following cases are tested:

1) plain rename (flags = 0)
2) cross rename
3) noreplace rename

For each the following are tested:

a) same directory rename
b) cross directory rename

And all combinations of source and dest being:

- negative
- regular file
- symlink
- empty dir
- nonempty dir

After the operation the file type of both source and dest are checked (or the error value returned).

Thanks,
Miklos

Ah yes, I had forgotten that you had sent those patches, thanks.

It looks like that since you worded it as "just RFC for now since they aren't in 3.15 yet", the xfstests folks never actually accepted your changes into xfstests, and so I never picked it up.

For future reference, the tests for COLLAPSE_RANGE and ZERO_RANGE were accepted into xfstests well before the merge window opened, and that was awfully convenience since we could pull the latest from the xfstests.git tree and do automated testing while those patches were in the ext4 and xfs trees.

So feel free to be a bit more insistent about asking for your xfstests to be merged upstream; you don't have to wait until the changes reach mainline. If it's clear that the patches are going to be accepted, and they are in the subsystem trees, that's a fine time to push to get the changes into xfstests.

Cheers,

- Ted

The xfstests for those features were merged into xfstests at the same time the the kernel code was pulled into the XFS tree. i.e. once the kernel code had been merged into a maintainer's tree. That's why they were there for the testing you needed to do with ext4.

There hasn't been any XFS patches written for renameat2 and the patches that were posted as "here's some tests, maybe we'll get renameat2 into 3.15" so there hasn't been any urgency indicated to the xfstests folks that they were needed. We've been pretty much out of the loop here....

Actually, we don't add tests to xfstests until the patches that they test are committed to an upstream repository somewhere. i.e. we need some guarantee that the code is actually accepted by a maintainer and is on it's way to mainline before we'll include the tests. We don't want to have to waste time on reviewing and committing tests for functionality that never goes into the mainline tree....

I asked whether this patchset is going to make 3.15 and reviewed the xfstests patches that had been posted earlier today. About 12 hours before I read this thread.... ;)

Cheers,

Dave.

Dave Chinner
[email protected]

I tried applying this patch on top of xfstests commit 3948694eb1, but running on ext4.git's test branch, which has the your cross-rename patches applied:

http://git.kernel.org/cgit/linux/kernel/git/tytso/ext4.git/commit/?h=test&id=b7bcc46d4f80139930c2e6bd04ff8ebbff121bc9

I see the following:

BEGIN TEST: Ext4 4k block Fri Apr 4 23:21:50 UTC 2014
Device: /dev/vdb
mk2fs options: -q
mount options: -o block_validity

FSTYP         -- ext4
PLATFORM      -- Linux/i686 candygram 3.14.0-00061-gb7bcc46
MKFS_OPTIONS  -- -q /dev/vdc

MOUNT_OPTIONS -- -o acl,user_xattr -o block_validity /dev/vdc /vdc

generic/323 [23:22:03] [23:22:04] [not run]
generic/323 -- kernel doesn't support renameat2 syscall
generic/324 [23:22:04] [23:22:05] [not run]
generic/324 -- kernel doesn't support renameat2 syscall
generic/325 [23:22:05] [23:22:06] [not run]
generic/325 -- kernel doesn't support renameat2 syscall
Not run: generic/323 generic/324 generic/325

Is there anything obvious that I might be doing wrong?

Thanks,

- Ted

I only wired up the syscall for x86_64. Who's responsible for adding
all the syscall tables for the various architectures?

Thanks,
Miklos

Ah, and I was testing with i386, not x86_64, so that it explains that.

It's been quite a while since I've worked to add a new system call, but my impressure is that in general the person who creates the new system call needs to reach out to the architecture maintainers (preferably with a patch :-), since otherwise the architecture maintainers would have no idea that a new syscall has been added.

- Ted

Preferably the creator of the new system call emails linux-arch. Patches are always nice to have, but they may cause conflicts w.r.t. syscall numbering.

If i386 has the new syscall, scripts/checksyscalls.sh will catch it and inform us about it during our next kernel build.

If you add it to x86_64 only, bad luck for anyone else ;-)

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

Also it would be nice if somebody would pick up the patch below as well :)

From: Heiko Carstens <[email protected]>
Date: Tue, 8 Apr 2014 12:55:46 +0200
Subject: [PATCH] include/linux/syscalls.h: add sys_renameat2() prototype

---
 include/linux/syscalls.h | 3 +++
 1 file changed, 3 insertions(+)

index 2aa8b749f13d..697ceb70a9a9 100644
@@ -748,6 +748,9 @@ asmlinkage long sys_linkat(int olddfd, const char __user *oldname,
 			   int newdfd, const char __user *newname, int flags);
 asmlinkage long sys_renameat(int olddfd, const char __user * oldname,
 			     int newdfd, const char __user * newname);
+asmlinkage long sys_renameat2(int olddfd, const char __user *oldname,
+			      int newdfd, const char __user *newname,
+			      unsigned int flags);
 asmlinkage long sys_futimesat(int dfd, const char __user *filename,
 			      struct timeval __user *utimes);
 asmlinkage long sys_faccessat(int dfd, const char __user *filename, int mode);

1.8.5.5

Picked up.

Linus

Maybe we should change scripts/checksyscalls.sh to check the x86_64 list of syscalls, and not i386?

It's been a long time since "all the world's an i386" --- these days, it's "all the world's an x86_64". :-)

- Ted

"Thou shalt foreswear, renounce, and abjure the vile heresy which claimeth that ``All the world's a VAX'', and have no commerce with the benighted heathens who cling to this barbarous belief, that the days of thy program may be long even though the days of thy current machine be short." -- The Ten Commandments for C Programmers

- http://www.lysator.liu.se/c/ten-commandments.html

Let the kbuild people (and their employers) fight over it...

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

I'm missing context here, but as an x86 maintainer I have no intention of allowing system calls that aren't x86-specific to be added to x86-64 only.

-hpa

Hi Peter,

commit 520c8b16505236fc82daa352e6c5e73cd9870cff

Author: Miklos Szeredi <[email protected]>
Date:   Tue Apr 1 17:08:42 2014 +0200

vfs: add renameat2 syscall

It was added to arch/x86/syscalls/syscall_64.tbl only.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

To be fair, part of the problem is that we don't have good documentation about best practices in what people should do if adding new system calls. (i.e., creating a man page and pulling in Michael Kerrisk, adding tests, wiring up both x86_64 and i386, sending mail to linux-arch, the scripts/checksyscalls.sh script, etc.)

I'll confess to being ignorant about the checksyscalls.sh script, and while I had known about the existence of the linux-arch list, I had forgotten about it, so if I had tried to add a new system call, it's likely I would have missed at one or more of these steps.

- Ted

It is quite frankly ridiculous how much work it takes to wire up a new system call. It should be possible to do it across architectures without all this crap.

Sent from my mobile phone.  Please pardon brevity and lack of formatting.

This change causes breakage:

fs/built-in.o: In function `__mb_cache_entry_release':
mbcache.c:(.text+0x56b3c): undefined reference to log2' mbcache.c:(.text+0x56b3c): relocation truncated to fit: R_MIPS_26 against log2' mbcache.c:(.text+0x56b74): undefined reference to __fixunsdfsi' mbcache.c:(.text+0x56b74): relocation truncated to fit: R_MIPS_26 against __fixunsdfsi' mbcache.c:(.text+0x56be4): undefined reference to log2' mbcache.c:(.text+0x56be4): relocation truncated to fit: R_MIPS_26 against log2' mbcache.c:(.text+0x56bf0): undefined reference to __fixunsdfsi' mbcache.c:(.text+0x56bf0): relocation truncated to fit: R_MIPS_26 against __fixunsdfsi'

and so on, and so on. It uses __builtin_log2() that is a floating-point function, its corresponding <math.h> prototype is:

-- Function: double log2 (double X)

(GCC builtins are of course implicitly prototyped). Please note that floating-point calculations are not allowed in the kernel and relying on the compiler to optimise them away is risky to say the least.

Can you please rewrite the fragment using __builtin_log2() so as to avoid
the floating-point calculation, using ffs() or suchlike perhaps?

Maciej

Sorry and thanks for notifying me about the problem.

I've submitted the fix, using ilog2, and could be found at,

https://lkml.org/lkml/2014/5/30/462

Please let me know if you have any comment.

Thanks,
Mak.

Thanks, that's even better, and the kernel builds and boots. I saw no
trace of your proposal going anywhere so I gave it my `Reviewed-by' tag in case that helps.

Maciej

Ted,
going through pending issues, and wondering if I'm expected to pick
this up directly? The original came through your tree (commit 1f3e55fe02d1: "fs/mbcache.c: doucple the locking of local from global data") and is in 3.15 too, so the patch should be marked for stable as well.

It's hopefully limited to just very bad gcc versions for odd architectures, so it's not exactly high-impact, but still..

Linus

Yes, I'll get something to you soon. Sorry, I was on travel to NYC earlier this week, so I'm a bit behind on the bug fix patches that I needed to send to you now that the merge window is closed.

- Ted