Skip to contentSkip to conversation

fs/9p: regression in 6.8-rc1

DISCUSSION Eric Van Hensbergen 4 participants 6 messages Jan 26, 2024 – Feb 18, 2024

Subsystems (2)
lore
Share & tools
lore

Eric Van Hensbergen

author maintainer
#0 lore raw

I caught a problem in the new netfs code when running in 9p when running with nocache mode. A regression sweep is turning up a:

 [ 1084.438387] netfs: Zero-sized write [R=1b6da]

when running my ldconfig test (included at the end of this)
it reports:
/sbin/ldconfig.real: Writing of cache extension data failed: Input/output error

I will try to dig into this later today if I have time, but not sure I'll get to it so I wanted to make other folks aware. I'm not sure how much other elements of my test harness are contributing to reproducing the problem.

-eric

--snip--

# setup chroot

mount -o bind /dev /mnt/9/dev
mount -o bind /sys /mnt/9/sys

# test mmap via ldconfig doesn't produce lots of errors

chroot /mnt/9 ldconfig -C /tmp/testldconfig.cache > $LOG.log 2>&1

if [ -s $LOG.log ]; then
echo "FAIL"
else
echo "SUCCESS"
fi

#1 lore raw

Eric Van Hensbergen wrote on Fri, Jan 26, 2024 at 02:21:39PM -0600:

Quoted 10 lines

I caught a problem in the new netfs code when running in 9p when running with nocache mode. A regression sweep is turning up a:

 [ 1084.438387] netfs: Zero-sized write [R=1b6da]

when running my ldconfig test (included at the end of this)
it reports:
/sbin/ldconfig.real: Writing of cache extension data failed: Input/output error

I will try to dig into this later today if I have time, but not sure I'll get to it so I wanted to make other folks aware. I'm not sure how much other elements of my test harness are contributing to reproducing the problem.

The syzbot report (refcount underflow[1]) is also probably related; I'll try to find some time to check a bit more this weekend

[1] https://lkml.kernel.org/r/[email protected]

Signature · 1 line
Dominique Martinet | Asmadeus
#2 lore raw

Dominique Martinet wrote on Sat, Jan 27, 2024 at 10:53:19AM +0900:

Quoted 11 lines

Eric Van Hensbergen wrote on Fri, Jan 26, 2024 at 02:21:39PM -0600:

Quoted 10 lines

I caught a problem in the new netfs code when running in 9p when running with nocache mode. A regression sweep is turning up a:

 [ 1084.438387] netfs: Zero-sized write [R=1b6da]

when running my ldconfig test (included at the end of this)
it reports:
/sbin/ldconfig.real: Writing of cache extension data failed: Input/output error

I will try to dig into this later today if I have time, but not sure I'll get to it so I wanted to make other folks aware. I'm not sure how much other elements of my test harness are contributing to reproducing the problem.

Didn't get much time, I can just confirm I can reproduce, it boils down to a 0-size write:

$ xfs_io -f -c 'pwrite 0 0' foo
(dmesg) netfs: Zero-sized write [R=fb5]
pwrite: Input/output error

I was going to say we probably need to filter it out - but it looks like that might be netfs' job given the call trace I get:

# retsnoop -T -e vfs_write -a :fs/9p/*.c -a :fs/netfs/*.c
FUNCTION CALL TRACE                 RESULT                  DURATION
---------------------------------   --------------------  ----------
→ vfs_write                                                         
    → netfs_unbuffered_write_iter                                   
        ↔ netfs_start_io_direct     [0]                      0.391us
        → netfs_alloc_request                                       
            ↔ v9fs_init_request     [0]                      0.431us
        ← netfs_alloc_request       [0xffff8b8a5584d600]     1.653us
        ↔ netfs_extract_user_iter   [0]                      0.671us
        → netfs_begin_write                                         
            ↔ v9fs_free_inode       [void]                  33.653us
            ↔ v9fs_free_inode       [void]                   0.511us
            ↔ v9fs_free_inode       [void]                   0.371us
            ↔ v9fs_free_inode       [void]                   0.350us
            ↔ v9fs_free_inode       [void]                   0.391us
            ↔ v9fs_free_inode       [void]                   0.361us
            ↔ v9fs_free_inode       [void]                   0.451us
            ↔ v9fs_free_inode       [void]                   0.391us
            ↔ v9fs_free_inode       [void]                   0.391us
            ↔ v9fs_free_inode       [void]                   0.451us
        ← netfs_begin_write         [-EIO]                1120.811us
        → netfs_free_request                                        
            ↔ v9fs_free_request     [void]                  28.062us
        ← netfs_free_request        [void]                  44.423us
        ↔ netfs_end_io_direct       [void]                   0.421us
    ← netfs_unbuffered_write_iter   [-EIO]                1207.784us
← vfs_write                         [-EIO]                1210.228us

David, where do you think we should catch that?
Can we leave that fix to you?

Quoted 4 lines

The syzbot report (refcount underflow[1]) is also probably related; I'll try to find some time to check a bit more this weekend

[1] https://lkml.kernel.org/r/[email protected]

So that one's not directly related to this, but given the timing I'd still bet something changed around cache... I didn't manage to reproduce it on a very quick workload but I didn't run all that much yet, will need to spend a bit more time on that another day...

Signature · 1 line
Dominique Martinet | Asmadeus
#3 lore raw
Quoted 8 lines · Dominique Martinet

I was going to say we probably need to filter it out - but it looks like that might be netfs' job given the call trace I get:

# retsnoop -T -e vfs_write -a :fs/9p/*.c -a :fs/netfs/*.c
FUNCTION CALL TRACE                 RESULT                  DURATION
---------------------------------   --------------------  ----------
→ vfs_write                                                         
    → netfs_unbuffered_write_iter                                   

Yeah - I should check to see if generic_write_checks() returned 0 and skip if it did (as does netfs_file_write_iter()).

It might also be worth doing the check before even taking any locks.

David

#4 lore raw
In reply to Eric Van Hensbergen
Quoted 10 lines · Eric Van Hensbergen

I caught a problem in the new netfs code when running in 9p when running with nocache mode. A regression sweep is turning up a:

 [ 1084.438387] netfs: Zero-sized write [R=1b6da]

when running my ldconfig test (included at the end of this)
it reports:
/sbin/ldconfig.real: Writing of cache extension data failed: Input/output error

I will try to dig into this later today if I have time, but not sure I'll get to it so I wanted to make other folks aware. I'm not sure how much other elements of my test harness are contributing to reproducing the problem.

To be sure the issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression tracking bot:

#regzbot ^introduced v6.7..v6.8-rc1
#regzbot title fs/9p: "netfs: Zero-sized write" failures during ldconfig test
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply and tell me -- ideally while also telling regzbot about it, as explained by the page listed in the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing to the report (the parent of this mail). See page linked in footer for details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

#5 lore raw
Quoted 8 lines · Linux regression tracking #adding (Thorsten Leemhuis)
Quoted 7 lines · Eric Van Hensbergen

I caught a problem in the new netfs code when running in 9p when running with nocache mode. A regression sweep is turning up a:

 [ 1084.438387] netfs: Zero-sized write [R=1b6da]

when running my ldconfig test (included at the end of this)
it reports:
/sbin/ldconfig.real: Writing of cache extension data failed: Input/output error

Quoted 8 lines
Quoted 3 lines

I will try to dig into this later today if I have time, but not sure I'll get to it so I wanted to make other folks aware. I'm not sure how much other elements of my test harness are contributing to reproducing the problem.

To be sure the issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression tracking bot:

#regzbot ^introduced v6.7..v6.8-rc1

#regzbot introduced 153a9961b551
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.