Skip to contentSkip to conversation

Reading from fuse pipe fails with EBADFD

DISCUSSION Nikolaus Rath 3 participants 7 messages Nov 28, 2018 – Dec 31, 2018 VFS

lore
Share & tools
lore
#0 lore raw

Hi,

When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.

Does anyone have an idea what might cause this, and how to debug it further?

Unfortunately I can't tell if this happens upon read() or upon splice(). I've extended the code now, so the next time it happens I will be able to tell.

I am testing with kernel 4.15 from Ubuntu Bionic.

So far this has happened twice in about 3 hours of testing. For the testing, I am repeatedly running the same 3 workloads, without remounting. Each workload takes ~ 15-30 seconds.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#1 lore raw

Hi,

Really no one any suggestion for debugging this?

(Adding some more people who recently worked on fs/fuse)

Best,
-Nikolaus

Quoted 25 lines · Nikolaus Rath

Hi,

When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.

Does anyone have an idea what might cause this, and how to debug it further?

Unfortunately I can't tell if this happens upon read() or upon splice(). I've extended the code now, so the next time it happens I will be able to tell.

I am testing with kernel 4.15 from Ubuntu Bionic.

So far this has happened twice in about 3 hours of testing. For the testing, I am repeatedly running the same 3 workloads, without remounting. Each workload takes ~ 15-30 seconds.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#2 lore raw
Quoted 15 lines · Nikolaus Rath

Hi,

Really no one any suggestion for debugging this?

(Adding some more people who recently worked on fs/fuse)

Best,
-Nikolaus

Quoted 4 lines · Nikolaus Rath

Hi,

When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.

EBADFD or EBADF?

Thanks,
Miklos

#3 lore raw
Quoted 11 lines · Miklos Szeredi
Quoted 10 lines · Nikolaus Rath

Hi,

Really no one any suggestion for debugging this?

(Adding some more people who recently worked on fs/fuse)

Best,
-Nikolaus

Quoted 7 lines
Quoted 5 lines · Nikolaus Rath
Quoted 4 lines

Hi,

When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.

EBADFD or EBADF?

EBADF ("Bad file descriptor"), sorry.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#4 lore raw
Quoted 14 lines · Nikolaus Rath
Quoted 12 lines · Miklos Szeredi
Quoted 11 lines · Nikolaus Rath

Hi,

Really no one any suggestion for debugging this?

(Adding some more people who recently worked on fs/fuse)

Best,
-Nikolaus

On Nov 27 2018, Nikolaus Rath <[email protected]> wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

Hi,

When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.

EBADFD or EBADF?

EBADF ("Bad file descriptor"), sorry.

Can you run the thing with "strace -f ..."?

Thanks,
Miklos

#5 lore raw
Quoted 17 lines · Miklos Szeredi
Quoted 16 lines · Nikolaus Rath
Quoted 14 lines · Miklos Szeredi
Quoted 13 lines · Nikolaus Rath

Hi,

Really no one any suggestion for debugging this?

(Adding some more people who recently worked on fs/fuse)

Best,
-Nikolaus

Quoted 2 lines · Nikolaus Rath

Hi,

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.

EBADFD or EBADF?

EBADF ("Bad file descriptor"), sorry.

Can you run the thing with "strace -f ..."?

Apologies for the delayed response. I have been trying to reproduce this but have instead run into another problem: the *client* getting spurious EBADF warnings. I am not sure if this is related or unrelated, and it has been hard to debug because it does not happen under strace:

$ find mnt > /dev/null
find: ‘mnt/modules/4.18.0-0.bpo.3-amd64/kernel/net/8021q’: Bad file descriptor
find: ‘mnt/modules/4.18.0-0.bpo.3-amd64/kernel/drivers/net/ethernet/natsemi’: Bad file descriptor

This happens in roughly 1 in 2 runs, and the affected directory entries are always different. Disabling readdirplus does not make a difference.

$ strace -o log find mnt > /dev/null

never gave in error in roughly 20 attempts.

Similarly, enabling fuse debug logging also makes the problem go away.

I also found some odd warnings in dmesg:

[24472.435256] fuse: trying to steal weird page
[24472.435261]   page=00000000b5b89670 index=0 flags=17fffc0000000ad, count=1, mapcount=0, map

..happens a lot (and I just wrote another email about it), and

[24473.170110] VFS: Lookup of '4.18.0-0.bpo.1-amd64' in fuse fuse would have caused loop

happened only a few times (and the fuse string is indeed duplicated). When this happens, find complains that:

$ find mnt > /dev/null
find: File system loop detected; ‘mnt/modules/4.18.0-0.bpo.1-amd64/kernel/drivers/w1’ is part of the same file system loop as ‘mnt/modules’

Does this ring any bells with anyone?

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#6 lore raw
Quoted 5 lines · Nikolaus Rath
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

When testing FUSE under heavy load, I am occasionally getting EBADFD errors when reading from the fuse pipe.

EBADFD or EBADF?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

EBADF ("Bad file descriptor"), sorry.

Can you run the thing with "strace -f ..."?

Apologies for the delayed response. I have been trying to reproduce this but have instead run into another problem: the *client* getting spurious EBADF warnings. I am not sure if this is related or unrelated, and it has been hard to debug because it does not happen under strace:

[..]

I believe I have figured it out. I was mistakenly assuming that the bad file descriptor was the fuse pipe - but it was actually the target file descriptor (I managed to completely forget that splice works on two file descriptors).

The root cause was mistakenly closing a file descriptor twice and not checking the return value. Because of the missing return value check, the error went unnoticed almost all the time - except when a different thread managed to re-use the fd for different purposes between the first and second close.

Apologies for taking everyone's time for what was actually a filesystem bug.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«