Reading from fuse pipe fails with EBADFD
Hi,
When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.
Does anyone have an idea what might cause this, and how to debug it further?
Unfortunately I can't tell if this happens upon read() or upon splice(). I've extended the code now, so the next time it happens I will be able to tell.
I am testing with kernel 4.15 from Ubuntu Bionic.
So far this has happened twice in about 3 hours of testing. For the testing, I am repeatedly running the same 3 workloads, without remounting. Each workload takes ~ 15-30 seconds.
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Hi,
Really no one any suggestion for debugging this?
(Adding some more people who recently worked on fs/fuse)
Best,
-Nikolaus
Quoted 25 lines · Nikolaus Rath
Hi,
When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.
Does anyone have an idea what might cause this, and how to debug it further?
Unfortunately I can't tell if this happens upon
read()or uponsplice(). I've extended the code now, so the next time it happens I will be able to tell.I am testing with kernel 4.15 from Ubuntu Bionic.
So far this has happened twice in about 3 hours of testing. For the testing, I am repeatedly running the same 3 workloads, without remounting. Each workload takes ~ 15-30 seconds.
Best,
-NikolausSignature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 15 lines · Nikolaus Rath
Hi,
Really no one any suggestion for debugging this?
(Adding some more people who recently worked on fs/fuse)
Best,
-NikolausQuoted 4 lines · Nikolaus Rath
Hi,
When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.
EBADFD or EBADF?
Thanks,
Miklos
Quoted 11 lines · Miklos Szeredi
Quoted 10 lines · Nikolaus Rath
Hi,
Really no one any suggestion for debugging this?
(Adding some more people who recently worked on fs/fuse)
Best,
-Nikolaus
Quoted 7 lines
Quoted 5 lines · Nikolaus Rath
Quoted 4 lines
Hi,
When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.
EBADFD or EBADF?
EBADF ("Bad file descriptor"), sorry.
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 14 lines · Nikolaus Rath
Quoted 12 lines · Miklos Szeredi
Quoted 11 lines · Nikolaus Rath
Hi,
Really no one any suggestion for debugging this?
(Adding some more people who recently worked on fs/fuse)
Best,
-NikolausOn Nov 27 2018, Nikolaus Rath <[email protected]> wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Hi,
When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.
EBADFD or EBADF?
EBADF ("Bad file descriptor"), sorry.
Can you run the thing with "strace -f ..."?
Thanks,
Miklos
Quoted 17 lines · Miklos Szeredi
Quoted 16 lines · Nikolaus Rath
Quoted 14 lines · Miklos Szeredi
Quoted 13 lines · Nikolaus Rath
Hi,
Really no one any suggestion for debugging this?
(Adding some more people who recently worked on fs/fuse)
Best,
-NikolausQuoted 2 lines · Nikolaus Rath
Hi,
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.
EBADFD or EBADF?
EBADF ("Bad file descriptor"), sorry.
Can you run the thing with "strace -f ..."?
Apologies for the delayed response. I have been trying to reproduce this but have instead run into another problem: the *client* getting spurious EBADF warnings. I am not sure if this is related or unrelated, and it has been hard to debug because it does not happen under strace:
$ find mnt > /dev/null find: ‘mnt/modules/4.18.0-0.bpo.3-amd64/kernel/net/8021q’: Bad file descriptor find: ‘mnt/modules/4.18.0-0.bpo.3-amd64/kernel/drivers/net/ethernet/natsemi’: Bad file descriptor
This happens in roughly 1 in 2 runs, and the affected directory entries are always different. Disabling readdirplus does not make a difference.
$ strace -o log find mnt > /dev/null
never gave in error in roughly 20 attempts.
Similarly, enabling fuse debug logging also makes the problem go away.
I also found some odd warnings in dmesg:
[24472.435256] fuse: trying to steal weird page [24472.435261] page=00000000b5b89670 index=0 flags=17fffc0000000ad, count=1, mapcount=0, map
..happens a lot (and I just wrote another email about it), and
[24473.170110] VFS: Lookup of '4.18.0-0.bpo.1-amd64' in fuse fuse would have caused loop
happened only a few times (and the fuse string is indeed duplicated). When this happens, find complains that:
$ find mnt > /dev/null find: File system loop detected; ‘mnt/modules/4.18.0-0.bpo.1-amd64/kernel/drivers/w1’ is part of the same file system loop as ‘mnt/modules’
Does this ring any bells with anyone?
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 5 lines · Nikolaus Rath
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.
EBADFD or EBADF?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
EBADF ("Bad file descriptor"), sorry.
Can you run the thing with "strace -f ..."?
Apologies for the delayed response. I have been trying to reproduce this but have instead run into another problem: the *client* getting spurious EBADF warnings. I am not sure if this is related or unrelated, and it has been hard to debug because it does not happen under strace:
[..]
I believe I have figured it out. I was mistakenly assuming that the bad file descriptor was the fuse pipe - but it was actually the target file descriptor (I managed to completely forget that splice works on two file descriptors).
The root cause was mistakenly closing a file descriptor twice and not checking the return value. Because of the missing return value check, the error went unnoticed almost all the time - except when a different thread managed to re-use the fd for different purposes between the first and second close.
Apologies for taking everyone's time for what was actually a filesystem bug.
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«