Skip to contentSkip to conversation

ETXTBSY when running Yarn (Node) since af5d68f

REGRESSION Andrew Udvare 3 participants 13 messages May 21, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

#regzbot introduced: v6.8..v6.9-rc1

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=af5d68f8892f8ee8f137648b79ceb2abc153a19b

Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:

/test # yarn --offline install
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module warning [email protected]: "test" is also the name of a node core module [1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /test/node_modules/snyk: Command failed.
Exit code: 126
Command: node wrapper_dist/bootstrap.js exec
Arguments:
Directory: /test/node_modules/snyk
Output:
/bin/sh: node: Text file busy

The commit was found by bisection with a simple initramfs that just runs 'yarn --offline install' with a test project and cached Yarn packages.

To reproduce:

npm install -g yarn
mkdir test
cd test
cat > package.json <<EOF
{
"name": "test",
"version": "1.0.0",
"main": "index.js",
"license": "MIT",

   "dependencies": {
     "snyk": "^1.1291.0"

}
}
EOF
yarn install

Modern Yarn will give the same result but with slightly different output.

This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051

See also: https://bugs.gentoo.org/931942

--
Andrew

maintainer
#1 lore raw

(removing stable as this isn't a stable regression, and the commit itself isn't marked for stable to begin with).

Quoted 39 lines · Andrew Udvare

#regzbot introduced: v6.8..v6.9-rc1

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=af5d68f8892f8ee8f137648b79ceb2abc153a19b

Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:

/test # yarn --offline install
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module warning [email protected]: "test" is also the name of a node core module [1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /test/node_modules/snyk: Command failed.
Exit code: 126
Command: node wrapper_dist/bootstrap.js exec
Arguments:
Directory: /test/node_modules/snyk
Output:
/bin/sh: node: Text file busy

The commit was found by bisection with a simple initramfs that just runs 'yarn --offline install' with a test project and cached Yarn packages.

To reproduce:

npm install -g yarn
mkdir test
cd test
cat > package.json <<EOF

{
   "name": "test",
   "version": "1.0.0",
   "main": "index.js",
   "license": "MIT",
   "dependencies": {
     "snyk": "^1.1291.0"
   }
}
Quoted 8 lines

EOF
yarn install

Modern Yarn will give the same result but with slightly different output.

This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051

See also: https://bugs.gentoo.org/931942

This looks like a timing alteration due to task_work being done differently, from a quick look and guess. For some reason SQPOLL is being used. I tried running it here, but it doesn't reproduce for me. Tried both current -git and 6.9 as released. I'll try on x86 as well to see if I can hit it.

Maybe someone can describe what is happening here? I'm assuming that something is racing with an io_uring operation done by SQPOLL, which is keeping the file open until done.

This may or may not be a kernel issue, depending on what assumptions are being made on when operations are begun and completed. Maybe those assumptions are correct and there is indeed a kernel regression here, or maybe the user side is just buggy in its assumptions.

Signature · 1 line
Jens Axboe
maintainer
#2 lore raw
Quoted 50 lines · Jens Axboe

(removing stable as this isn't a stable regression, and the commit itself isn't marked for stable to begin with).

Quoted 46 lines · Andrew Udvare

#regzbot introduced: v6.8..v6.9-rc1

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=af5d68f8892f8ee8f137648b79ceb2abc153a19b

Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:

/test # yarn --offline install
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module warning [email protected]: "test" is also the name of a node core module [1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /test/node_modules/snyk: Command failed.
Exit code: 126
Command: node wrapper_dist/bootstrap.js exec
Arguments:
Directory: /test/node_modules/snyk
Output:
/bin/sh: node: Text file busy

The commit was found by bisection with a simple initramfs that just runs 'yarn --offline install' with a test project and cached Yarn packages.

To reproduce:

npm install -g yarn
mkdir test
cd test
cat > package.json <<EOF

{
   "name": "test",
   "version": "1.0.0",
   "main": "index.js",
   "license": "MIT",
   "dependencies": {
     "snyk": "^1.1291.0"
   }
}

EOF
yarn install

Modern Yarn will give the same result but with slightly different output.

This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051

Quoted 7 lines
Quoted 1 line

See also: https://bugs.gentoo.org/931942

This looks like a timing alteration due to task_work being done differently, from a quick look and guess. For some reason SQPOLL is being used. I tried running it here, but it doesn't reproduce for me. Tried both current -git and 6.9 as released. I'll try on x86 as well to see if I can hit it.

Reproduces on an x86-64 vm:

[root@archlinux test]# yarn install
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module
info No lockfile found.
warning [email protected]: "test" is also the name of a node core module
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /root/test/node_modules/snyk: Command failed.
Exit code: 126
Command: node wrapper_dist/bootstrap.js exec
Arguments:
Directory: /root/test/node_modules/snyk
Output:
/bin/sh: /tmp/yarn--1716308447842-0.8496252120161716/node: /bin/sh: bad interpreter: Text file busy

I'll poke a bit.

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 9 lines · Jens Axboe
Quoted 9 lines · Andrew Udvare
Quoted 8 lines

#regzbot introduced: v6.8..v6.9-rc1

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=af5d68f8892f8ee8f137648b79ceb2abc153a19b

Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:

This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

See also: https://bugs.gentoo.org/931942

This looks like a timing alteration due to task_work being done differently, from a quick look and guess. For some reason SQPOLL is being used. I tried running it here, but it doesn't reproduce for me. Tried both current -git and 6.9 as released. I'll try on x86 as well to see if I can hit it.

It seems like this also was a problem for libuv previously as somone noted in a comment on the Arch Linux Bugtracker[0]:

- https://github.com/libuv/libuv/commit/1752791c9ea89dbf54e2a20a9d9f899119a2d179
- https://github.com/libuv/libuv/blob/v1.48.0/src/unix/linux.c#L834

Cheers,
Chris

[0]: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/55#note_186921

#regzbot introduced: af5d68f8892f8ee8f137648b79ceb2abc153a19b
#regzbot link: https://bugs.gentoo.org/931942
#regzbot link: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/55

maintainer
#4 lore raw
Quoted 19 lines · Christian Heusel
Quoted 9 lines · Jens Axboe
Quoted 9 lines · Andrew Udvare
Quoted 8 lines

#regzbot introduced: v6.8..v6.9-rc1

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=af5d68f8892f8ee8f137648b79ceb2abc153a19b

Since the above commit present in 6.9+, Node running a Yarn installation that executes a subprocess always shows the following:

This also appears to affect node-gyp: https://github.com/nodejs/node/issues/53051

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

See also: https://bugs.gentoo.org/931942

This looks like a timing alteration due to task_work being done differently, from a quick look and guess. For some reason SQPOLL is being used. I tried running it here, but it doesn't reproduce for me. Tried both current -git and 6.9 as released. I'll try on x86 as well to see if I can hit it.

Quoted 5 lines

It seems like this also was a problem for libuv previously as somone noted in a comment on the Arch Linux Bugtracker[0]:

- https://github.com/libuv/libuv/commit/1752791c9ea89dbf54e2a20a9d9f899119a2d179
- https://github.com/libuv/libuv/blob/v1.48.0/src/unix/linux.c#L834

Just got that far in my local poking, it is indeed execve() getting ETXTBUSY. I'll see if I can make a test case for this. Presumably the application side is fine, eg it waits on the close completion before doing the execve, not just having it be issued. Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

I'll be back with more info...

Signature · 1 line
Jens Axboe
maintainer
#5 lore raw
Quoted 4 lines · Jens Axboe

Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

Yep I think that's it, the below should fix it.

io_uring/sqpoll.c +2 -0
index 554c7212aa46..68a3e3290411 100644
@@ -241,6 +241,8 @@ static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)
 			return count;
 		max_entries -= count;
 	}
+	if (task_work_pending(current))
+		task_work_run();
 
 	*retry_list = tctx_task_work_run(tctx, max_entries, &count);
 	return count;
Signature · 1 line
Jens Axboe
maintainer
#6 lore raw
Quoted 7 lines · Jens Axboe
Quoted 4 lines · Jens Axboe

Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

Yep I think that's it, the below should fix it.

Sent it out and also wrote a test case to catch this:

https://git.kernel.dk/cgit/liburing/commit/?id=06c22ef6637284ab1f31ee64f1ee48a829958816

just in case we ever regress in that manner again.

This will go into the 6.10-rc1 release, will ship it to Linus in a day or two. And then it'll land in the next stable release of 6.9 as well.

Thanks for the report!

Signature · 1 line
Jens Axboe
#7 lore raw
In reply to Jens Axboe
Quoted 7 lines · Jens Axboe
Quoted 4 lines · Jens Axboe

Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

Yep I think that's it, the below should fix it.

It indeed does! I applied the patch on top of Linus tree and the issue is now gone!

Cheers,
Chris

maintainer
#8 lore raw
Quoted 7 lines · Christian Heusel
Quoted 6 lines · Jens Axboe
Quoted 4 lines · Jens Axboe

Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

Quoted 6 lines
Quoted 1 line

Yep I think that's it, the below should fix it.

It indeed does! I applied the patch on top of Linus tree and the issue is now gone!

Thanks for testing! I'll add your tag.

Signature · 1 line
Jens Axboe
#9 lore raw
In reply to Jens Axboe
Quoted 15 lines · Jens Axboe
Quoted 4 lines · Jens Axboe

Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

Yep I think that's it, the below should fix it.

io_uring/sqpoll.c +0 -0
index 554c7212aa46..68a3e3290411 100644

io_uring/sqpoll.c static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries) · line 248

Quoted 8 lines
io_uring/sqpoll.c +2 -0
@@ -241,6 +241,8 @@ static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)
  			return count;
  		max_entries -= count;
  	}
+	if (task_work_pending(current))
+		task_work_run();
 
  	*retry_list = tctx_task_work_run(tctx, max_entries, &count);
  	return count;

This patch works for me on 6.9.1.

 $ yarn
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module
info No lockfile found.
warning [email protected]: "test" is also the name of a node core module
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 3.32s.
 $ uname -a
Linux limelight 6.9.1-gentoo-limelight #2 SMP PREEMPT_DYNAMIC TKG Tue May 21 15:21:33 EDT 2024

--
Andrew

maintainer
#10 lore raw
Quoted 38 lines · Andrew Udvare
Quoted 14 lines · Jens Axboe
Quoted 4 lines · Jens Axboe

Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

Yep I think that's it, the below should fix it.

io_uring/sqpoll.c +0 -0
index 554c7212aa46..68a3e3290411 100644

io_uring/sqpoll.c static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries) · line 248

Quoted 8 lines
io_uring/sqpoll.c +2 -0
@@ -241,6 +241,8 @@ static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries)
              return count;
          max_entries -= count;
      }
+    if (task_work_pending(current))
+        task_work_run();
        *retry_list = tctx_task_work_run(tctx, max_entries, &count);
      return count;
 

This patch works for me on 6.9.1.

 $ yarn
yarn install v1.22.22
warning package.json: "test" is also the name of a node core module
info No lockfile found.
warning [email protected]: "test" is also the name of a node core module
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 3.32s.
Quoted 2 lines
 $ uname -a
Linux limelight 6.9.1-gentoo-limelight #2 SMP PREEMPT_DYNAMIC TKG Tue May 21 15:21:33 EDT 2024

Great, thank you for testing!

Signature · 1 line
Jens Axboe
author Tested-by
#11 lore raw
In reply to Jens Axboe
Quoted 7 lines · Jens Axboe
Quoted 4 lines · Jens Axboe

Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

Yep I think that's it, the below should fix it.

maintainer
#12 lore raw
Quoted 10 lines · Andrew Udvare
Quoted 7 lines · Jens Axboe
Quoted 4 lines · Jens Axboe

Outside of that, only other thing I can think of is that the final close would be punted to task_work by fput(), which means there's also a dependency on the task having run its kernel task_work before it's fully closed.

Yep I think that's it, the below should fix it.

Added, thanks.

Signature · 1 line
Jens Axboe