ETXTBSY when running Yarn (Node) since af5d68f
Subsystems (3)
#regzbot introduced: v6.8..v6.9-rc1
Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:
/test # yarn --offline install
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module warning [email protected]: "test" is also the name of a node core module [1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /test/node_modules/snyk: Command failed.
Exit code: 126
Command: node wrapper_dist/bootstrap.js exec
Arguments:
Directory: /test/node_modules/snyk
Output:/bin/sh: node: Text file busy
The commit was found by bisection with a simple initramfs that just runs 'yarn --offline install' with a test project and cached Yarn packages.
To reproduce:
npm install -g yarn
mkdir test
cd test
cat > package.json <<EOF
{
"name": "test",
"version": "1.0.0",
"main": "index.js",
"license": "MIT",
"dependencies": {
"snyk": "^1.1291.0" }
}
EOF
yarn install
Modern Yarn will give the same result but with slightly different output.
This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051
See also: https://bugs.gentoo.org/931942
--
Andrew
Jens Axboe
maintainer(removing stable as this isn't a stable regression, and the commit itself isn't marked for stable to begin with).
Quoted 39 lines · Andrew Udvare
#regzbot introduced: v6.8..v6.9-rc1
Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:
/test # yarn --offline install
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module warning [email protected]: "test" is also the name of a node core module [1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /test/node_modules/snyk: Command failed.
Exit code: 126
Command: node wrapper_dist/bootstrap.js exec
Arguments:
Directory: /test/node_modules/snyk
Output:/bin/sh: node: Text file busyThe commit was found by bisection with a simple initramfs that just runs 'yarn --offline install' with a test project and cached Yarn packages.
To reproduce:
npm install -g yarn
mkdir test
cd test
cat > package.json <<EOF{ "name": "test", "version": "1.0.0", "main": "index.js", "license": "MIT", "dependencies": { "snyk": "^1.1291.0" } }
Quoted 8 lines
EOF
yarn installModern Yarn will give the same result but with slightly different output.
This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051
See also: https://bugs.gentoo.org/931942
This looks like a timing alteration due to task_work being done differently, from a quick look and guess. For some reason SQPOLL is being used. I tried running it here, but it doesn't reproduce for me. Tried both current -git and 6.9 as released. I'll try on x86 as well to see if I can hit it.
Maybe someone can describe what is happening here? I'm assuming that something is racing with an io_uring operation done by SQPOLL, which is keeping the file open until done.
This may or may not be a kernel issue, depending on what assumptions are being made on when operations are begun and completed. Maybe those assumptions are correct and there is indeed a kernel regression here, or maybe the user side is just buggy in its assumptions.
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 50 lines · Jens Axboe
(removing stable as this isn't a stable regression, and the commit itself isn't marked for stable to begin with).
Quoted 46 lines · Andrew Udvare
#regzbot introduced: v6.8..v6.9-rc1
Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:
/test # yarn --offline install
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module warning [email protected]: "test" is also the name of a node core module [1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /test/node_modules/snyk: Command failed.
Exit code: 126
Command: node wrapper_dist/bootstrap.js exec
Arguments:
Directory: /test/node_modules/snyk
Output:/bin/sh: node: Text file busyThe commit was found by bisection with a simple initramfs that just runs 'yarn --offline install' with a test project and cached Yarn packages.
To reproduce:
npm install -g yarn
mkdir test
cd test
cat > package.json <<EOF{ "name": "test", "version": "1.0.0", "main": "index.js", "license": "MIT", "dependencies": { "snyk": "^1.1291.0" } }EOF
yarn installModern Yarn will give the same result but with slightly different output.
This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051
Quoted 7 lines
Quoted 1 line
See also: https://bugs.gentoo.org/931942
This looks like a timing alteration due to
task_workbeing done differently, from a quick look and guess. For some reason SQPOLL is being used. I tried running it here, but it doesn't reproduce for me. Tried both current -git and 6.9 as released. I'll try on x86 as well to see if I can hit it.
Reproduces on an x86-64 vm:
[root@archlinux test]# yarn install
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module
info No lockfile found.
warning [email protected]: "test" is also the name of a node core module
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /root/test/node_modules/snyk: Command failed.
Exit code: 126
Command: node wrapper_dist/bootstrap.js exec
Arguments:
Directory: /root/test/node_modules/snyk
Output:/bin/sh: /tmp/yarn--1716308447842-0.8496252120161716/node: /bin/sh: bad interpreter: Text file busy
I'll poke a bit.
Signature · 1 line
Jens Axboe
Quoted 9 lines · Jens Axboe
Quoted 9 lines · Andrew Udvare
Quoted 8 lines
#regzbot introduced: v6.8..v6.9-rc1
Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:
This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
See also: https://bugs.gentoo.org/931942
This looks like a timing alteration due to
task_workbeing done differently, from a quick look and guess. For some reason SQPOLL is being used. I tried running it here, but it doesn't reproduce for me. Tried both current -git and 6.9 as released. I'll try on x86 as well to see if I can hit it.
It seems like this also was a problem for libuv previously as somone noted in a comment on the Arch Linux Bugtracker[0]:
- https://github.com/libuv/libuv/commit/1752791c9ea89dbf54e2a20a9d9f899119a2d179
- https://github.com/libuv/libuv/blob/v1.48.0/src/unix/linux.c#L834
Cheers,
Chris
[0]: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/55#note_186921
#regzbot introduced: af5d68f8892f8ee8f137648b79ceb2abc153a19b
#regzbot link: https://bugs.gentoo.org/931942
#regzbot link: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/55
Jens Axboe
maintainerQuoted 19 lines · Christian Heusel
Quoted 9 lines · Jens Axboe
Quoted 9 lines · Andrew Udvare
Quoted 8 lines
#regzbot introduced: v6.8..v6.9-rc1
Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:
This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
See also: https://bugs.gentoo.org/931942
This looks like a timing alteration due to
task_workbeing done differently, from a quick look and guess. For some reason SQPOLL is being used. I tried running it here, but it doesn't reproduce for me. Tried both current -git and 6.9 as released. I'll try on x86 as well to see if I can hit it.
Quoted 5 lines
It seems like this also was a problem for libuv previously as somone noted in a comment on the Arch Linux Bugtracker[0]:
- https://github.com/libuv/libuv/commit/1752791c9ea89dbf54e2a20a9d9f899119a2d179
- https://github.com/libuv/libuv/blob/v1.48.0/src/unix/linux.c#L834
Just got that far in my local poking, it is indeed execve() getting ETXTBUSY. I'll see if I can make a test case for this. Presumably the application side is fine, eg it waits on the close completion before doing the execve, not just having it be issued. Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.
I'll be back with more info...
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 4 lines · Jens Axboe
Outside of that, only other thing I can think of is that the final close would be punted to
task_workbyfput(), which means there's also a dependency on the task having run its kerneltask_workbefore it's fully closed.
Yep I think that's it, the below should fix it.
io_uring/sqpoll.c +2 -0
@@ -241,6 +241,8 @@ static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)return count; max_entries -= count; } +if (task_work_pending (current)) +task_work_run (); *retry_list =tctx_task_work_run (tctx, max_entries, &count);return count;
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 7 lines · Jens Axboe
Quoted 4 lines · Jens Axboe
Outside of that, only other thing I can think of is that the final close would be punted to
task_workbyfput(), which means there's also a dependency on the task having run its kerneltask_workbefore it's fully closed.Yep I think that's it, the below should fix it.
Sent it out and also wrote a test case to catch this:
https://git.kernel.dk/cgit/liburing/commit/?id=06c22ef6637284ab1f31ee64f1ee48a829958816
just in case we ever regress in that manner again.
This will go into the 6.10-rc1 release, will ship it to Linus in a day or two. And then it'll land in the next stable release of 6.9 as well.
Thanks for the report!
Signature · 1 line
Jens Axboe
Christian Heusel
Tested-byQuoted 7 lines · Jens Axboe
Quoted 4 lines · Jens Axboe
Outside of that, only other thing I can think of is that the final close would be punted to
task_workbyfput(), which means there's also a dependency on the task having run its kerneltask_workbefore it's fully closed.Yep I think that's it, the below should fix it.
It indeed does! I applied the patch on top of Linus tree and the issue is now gone!
Cheers,
Chris
Jens Axboe
maintainerQuoted 7 lines · Christian Heusel
Quoted 6 lines · Jens Axboe
Quoted 4 lines · Jens Axboe
Outside of that, only other thing I can think of is that the final close would be punted to
task_workbyfput(), which means there's also a dependency on the task having run its kerneltask_workbefore it's fully closed.
Quoted 6 lines
Quoted 1 line
Yep I think that's it, the below should fix it.
It indeed does! I applied the patch on top of Linus tree and the issue is now gone!
Tested-by Christian Heusel <[email protected]>
Thanks for testing! I'll add your tag.
Signature · 1 line
Jens Axboe
Quoted 15 lines · Jens Axboe
Quoted 4 lines · Jens Axboe
Outside of that, only other thing I can think of is that the final close would be punted to
task_workbyfput(), which means there's also a dependency on the task having run its kerneltask_workbefore it's fully closed.Yep I think that's it, the below should fix it.
io_uring/sqpoll.c +0
-0
io_uring/sqpoll.c static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries) · line 248
Quoted 8 lines
io_uring/sqpoll.c +2
-0@@ -241,6 +241,8 @@ static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)return count; max_entries -= count; } +if (task_work_pending (current)) +task_work_run (); *retry_list =tctx_task_work_run (tctx, max_entries, &count);return count;
This patch works for me on 6.9.1.
$ yarn yarn install v1.22.22 warning package.json: "test" is also the name of a node core module info No lockfile found. warning [email protected]: "test" is also the name of a node core module [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... success Saved lockfile. Done in 3.32s.
$ uname -a Linux limelight 6.9.1-gentoo-limelight #2 SMP PREEMPT_DYNAMIC TKG Tue May 21 15:21:33 EDT 2024
--
Andrew
Jens Axboe
maintainerQuoted 38 lines · Andrew Udvare
Quoted 14 lines · Jens Axboe
Quoted 4 lines · Jens Axboe
Outside of that, only other thing I can think of is that the final close would be punted to
task_workbyfput(), which means there's also a dependency on the task having run its kerneltask_workbefore it's fully closed.Yep I think that's it, the below should fix it.
io_uring/sqpoll.c +0
-0io_uring/sqpoll.c static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries) · line 248
Quoted 8 lines
io_uring/sqpoll.c +2
-0@@ -241,6 +241,8 @@ static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)return count; max_entries -= count; } +if (task_work_pending (current)) +task_work_run (); *retry_list =tctx_task_work_run (tctx, max_entries, &count);return count;This patch works for me on 6.9.1.
$ yarn yarn install v1.22.22 warning package.json: "test" is also the name of a node core module info No lockfile found. warning [email protected]: "test" is also the name of a node core module [1/4] Resolving packages... [2/4] Fetching packages... [3/4] Linking dependencies... [4/4] Building fresh packages... success Saved lockfile. Done in 3.32s.
Quoted 2 lines
$ uname -a Linux limelight 6.9.1-gentoo-limelight #2 SMP PREEMPT_DYNAMIC TKG Tue May 21 15:21:33 EDT 2024
Great, thank you for testing!
Signature · 1 line
Jens Axboe
Andrew Udvare
Tested-byQuoted 7 lines · Jens Axboe
Quoted 4 lines · Jens Axboe
Outside of that, only other thing I can think of is that the final close would be punted to
task_workbyfput(), which means there's also a dependency on the task having run its kerneltask_workbefore it's fully closed.Yep I think that's it, the below should fix it.
Jens Axboe
maintainerQuoted 10 lines · Andrew Udvare
Quoted 7 lines · Jens Axboe
Quoted 4 lines · Jens Axboe
Outside of that, only other thing I can think of is that the final close would be punted to
task_workbyfput(), which means there's also a dependency on the task having run its kerneltask_workbefore it's fully closed.Yep I think that's it, the below should fix it.
Tested-by Andrew Udvare <[email protected]>
Added, thanks.
Signature · 1 line
Jens Axboe