Skip to contentSkip to conversation

pipe_read: don't wake up the writer if the pipe is still full

PATCH Applied Oleg Nesterov 10 participants 109 messages Jan 2, 2025 – Mar 11, 2025 1 message not in the archive

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 pipe_read: don't wake up the writer if the pipe is still full

wake_up(pipe->wr_wait) makes no sense if pipe_full() is still true after the reading, the writer sleeping in wait_event(wr_wait, pipe_writable()) will check the pipe_writable() == !pipe_full() condition and sleep again.

Only wake the writer if we actually released a pipe buf, and the pipe was full before we did so.

---
 fs/pipe.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
fs/pipe.c +10 -9
index 12b22c2723b7..82fede0f2111 100644
@@ -253,7 +253,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	size_t total_len = iov_iter_count(to);
 	struct file *filp = iocb->ki_filp;
 	struct pipe_inode_info *pipe = filp->private_data;
-	bool was_full, wake_next_reader = false;
+	bool wake_writer = false, wake_next_reader = false;
 	ssize_t ret;
 
 	/* Null read succeeds. */
@@ -264,14 +264,13 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	mutex_lock(&pipe->mutex);
 
 	/*
-	 * We only wake up writers if the pipe was full when we started
-	 * reading in order to avoid unnecessary wakeups.
+	 * We only wake up writers if the pipe was full when we started reading
+	 * and it is no longer full after reading to avoid unnecessary wakeups.
 	 *
 	 * But when we do wake up writers, we do so using a sync wakeup
 	 * (WF_SYNC), because we want them to get going and generate more
 	 * data for us.
 	 */
-	was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
 	for (;;) {
 		/* Read ->head with a barrier vs post_one_notification() */
 		unsigned int head = smp_load_acquire(&pipe->head);
@@ -340,8 +339,10 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 				buf->len = 0;
 			}
 
-			if (!buf->len)
+			if (!buf->len) {
+				wake_writer |= pipe_full(head, tail, pipe->max_usage);
 				tail = pipe_update_tail(pipe, buf, tail);
+			}
 			total_len -= chars;
 			if (!total_len)
 				break;	/* common path: read succeeded */
@@ -377,7 +378,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		 * _very_ unlikely case that the pipe was full, but we got
 		 * no data.
 		 */
-		if (unlikely(was_full))
+		if (unlikely(wake_writer))
 			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
 		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
 
@@ -390,15 +391,15 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
 			return -ERESTARTSYS;
 
-		mutex_lock(&pipe->mutex);
-		was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
+		wake_writer = false;
 		wake_next_reader = true;
+		mutex_lock(&pipe->mutex);
 	}
 	if (pipe_empty(pipe->head, pipe->tail))
 		wake_next_reader = false;
 	mutex_unlock(&pipe->mutex);
 
-	if (was_full)
+	if (wake_writer)
 		wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
 	if (wake_next_reader)
 		wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
Signature · 1 line
2.25.1.362.g51ebf55
#1 lore raw

[Adding some of my colleagues who were part of the original submission to the CC list for their information.]

Quoted 6 lines · Oleg Nesterov

wake_up(pipe->wr_wait) makes no sense if pipe_full() is still true after the reading, the writer sleeping in wait_event(wr_wait, pipe_writable()) will check the pipe_writable() == !pipe_full() condition and sleep again.

Only wake the writer if we actually released a pipe buf, and the pipe was full before we did so.

As Linus said, for fs/pipe, he "want any patches to be very clearly documented," perhaps we should include a link to the original discussion here.

Quoted 1 line

I'm happy to provide more test results for this patch if it's not too late.

Quoted 64 lines
---
  fs/pipe.c | 19 ++++++++++---------
  1 file changed, 10 insertions(+), 9 deletions(-)
fs/pipe.c +9 -8
index 12b22c2723b7..82fede0f2111 100644
@@ -253,7 +253,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  	size_t total_len = iov_iter_count(to);
  	struct file *filp = iocb->ki_filp;
  	struct pipe_inode_info *pipe = filp->private_data;
-	bool was_full, wake_next_reader = false;
+	bool wake_writer = false, wake_next_reader = false;
  	ssize_t ret;
 
  	/* Null read succeeds. */
@@ -264,14 +264,13 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  	mutex_lock(&pipe->mutex);
 
  	/*
-	 * We only wake up writers if the pipe was full when we started
-	 * reading in order to avoid unnecessary wakeups.
+	 * We only wake up writers if the pipe was full when we started reading
+	 * and it is no longer full after reading to avoid unnecessary wakeups.
  	 *
  	 * But when we do wake up writers, we do so using a sync wakeup
  	 * (WF_SYNC), because we want them to get going and generate more
  	 * data for us.
  	 */
-	was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
  	for (;;) {
  		/* Read ->head with a barrier vs post_one_notification() */
  		unsigned int head = smp_load_acquire(&pipe->head);
@@ -340,8 +339,10 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  				buf->len = 0;
  			}
 
-			if (!buf->len)
+			if (!buf->len) {
+				wake_writer |= pipe_full(head, tail, pipe->max_usage);
  				tail = pipe_update_tail(pipe, buf, tail);
+			}
  			total_len -= chars;
  			if (!total_len)
  				break;	/* common path: read succeeded */
@@ -377,7 +378,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  		 * _very_ unlikely case that the pipe was full, but we got
  		 * no data.
  		 */
-		if (unlikely(was_full))
+		if (unlikely(wake_writer))
  			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
  		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
 
@@ -390,15 +391,15 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
  			return -ERESTARTSYS;
 
-		mutex_lock(&pipe->mutex);
-		was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
+		wake_writer = false;
  		wake_next_reader = true;
+		mutex_lock(&pipe->mutex);

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 405

Quoted 10 lines
fs/pipe.c +1 -1
  	}
  	if (pipe_empty(pipe->head, pipe->tail))
  		wake_next_reader = false;
  	mutex_unlock(&pipe->mutex);
 
-	if (was_full)
+	if (wake_writer)
  		wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
  	if (wake_next_reader)
  		wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);

Hmm..
Initially, the sole purpose of our original patch was to simply check if there were any waiting processes in the process wait queue to avoid unnecessary wake-ups, for both reads and writes. And then, sincerely thank you all for taking the time to review it!
While your patch and ours share some little similarities, our primary goals may vary slightly. Do you have any suggestions on how we could better achieve our original objective?

Thanks,

Signature · 1 line
WangYuli
#2 lore raw
Quoted 3 lines · WangYuli

[Adding some of my colleagues who were part of the original submission to the CC list for their information.]

OK,

Quoted 3 lines

perhaps we should include a link to the original discussion

...

Quoted 1 line

WangYuli, this patch has nothing to do with your original patch and the discussion above.

Quoted 1 line

I'm happy to provide more test results for this patch if it's not too late.

Would be great, but I don't think this patch can make any difference performance-wise in practice. Short reads are not that common, I guess.

Quoted 4 lines

Hmm..
Initially, the sole purpose of our original patch was to simply check if there were any waiting processes in the process wait queue to avoid unnecessary wake-ups, for both reads and writes.

Exactly. So once again, this patch is orthogonal to the possible wq_has_sleeper() optimizations.

Quoted 2 lines

Do you have any suggestions on how we could better achieve our original objective?

See

	wakeup_pipe_readers/writers() && pipe_poll()
	https://lore.kernel.org/all/[email protected]/

Oleg.

maintainer Applied
#3 lore raw
In reply to Oleg Nesterov
Quoted 8 lines · Oleg Nesterov

wake_up(pipe->wr_wait) makes no sense if pipe_full() is still true after the reading, the writer sleeping in wait_event(wr_wait, pipe_writable()) will check the pipe_writable() == !pipe_full() condition and sleep again.

Only wake the writer if we actually released a pipe buf, and the pipe was full before we did so.

[...]

Applied to the vfs-6.14.misc branch of the vfs/vfs.git tree. Patches in the vfs-6.14.misc branch should appear in linux-next soon.

Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.

It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.

Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
branch: vfs-6.14.misc

[1/1] pipe_read: don't wake up the writer if the pipe is still full

      https://git.kernel.org/vfs/vfs/c/b004b4d254e7
#4 lore raw
In reply to Oleg Nesterov

Hello Oleg,

Quoted 6 lines · Oleg Nesterov

wake_up(pipe->wr_wait) makes no sense if pipe_full() is still true after the reading, the writer sleeping in wait_event(wr_wait, pipe_writable()) will check the pipe_writable() == !pipe_full() condition and sleep again.

Only wake the writer if we actually released a pipe buf, and the pipe was full before we did so.

I noticed a performance regression in perf bench sched messaging at higher utilization (larger number of groups) with this patch on the mainline kernel. For lower utilization, this patch yields good improvements but once the system is oversubscribed, the tale flips.

Following are the results from my testing on mainline at commit 05dbaf8dd8bf ("Merge tag 'x86-urgent-2025-01-28' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")
with and without this patch:

==================================================================

    Test          : sched-messaging
    cmdline       : perf bench sched messaging -p -t -l 100000 -g <groups>
    Units         : Normalized time in seconds
    Interpretation: Lower is better
    Statistic     : AMean
    ==================================================================
    Case:         mainline[pct imp](CV)    revert[pct imp](CV)
     1-groups     1.00 [ -0.00](12.29)     1.26 [-25.91]( 2.71)
     2-groups     1.00 [ -0.00]( 3.64)     1.39 [-38.53]( 0.89)
     4-groups     1.00 [ -0.00]( 3.33)     1.41 [-41.42]( 1.21)
     8-groups     1.00 [ -0.00]( 2.90)     1.10 [ -9.89]( 0.95)
    16-groups     1.00 [ -0.00]( 1.46)     0.66 [ 34.46]( 1.59)

On my 3rd Generation EPYC system (2 x 64C/128T), I see that on reverting the changes on the above mentioned commit, sched-messaging sees a regression up until the 8 group case which contains 320 tasks, however with 16 groups (640 tasks), the revert helps with performance.

Based on the trend in the performance, one can deduce that at lower utilization, sched-messaging benefits from not traversing the wake up path unnecessarily since wake_up_interruptible_sync_poll() acquires a lock before checking if the wait queue is empty or not thus saving on system time. However, at high utilization, there is likely a writer waiting to write to the pipe by the time the wait queue is inspected.

Following are the perf profile comparing the mainline with the revert:

o 1-group (4.604s [mainline] vs 8.163s [revert])

sudo ./perf record -C 0-7,64-127 -e ibs_op/cnt_ctl=1/ -- taskset -c 0-7,64-127 ./perf bench sched messaging -p -t -l 100000 -g 1

(sched-messaging was pinned to 1 CCX and only that CCX was profiled
using IBS to reduce noise)

mainline vs revert

Samples: 606K of event 'ibs_op/cnt_ctl=1/', Event count (approx.): 205972485144 Samples: 479K of event 'ibs_op/cnt_ctl=1/', Event count (approx.): 200365591518

Overhead  Command          Shared Object         Symbol                                                Overhead  Command          Shared Object         Symbol
   4.80%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_safe_ret                                  5.12%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_safe_ret
   4.10%  sched-messaging  [kernel.kallsyms]     [k] rep_movs_alternative                                 4.30%  sched-messaging  [kernel.kallsyms]     [k] rep_movs_alternative
   3.24%  sched-messaging  [kernel.kallsyms]     [k] osq_lock                                             3.42%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_return_thunk
   3.23%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_return_thunk                              3.31%  sched-messaging  [kernel.kallsyms]     [k] syscall_exit_to_user_mode
   3.13%  sched-messaging  [kernel.kallsyms]     [k] syscall_exit_to_user_mode                            2.71%  sched-messaging  [kernel.kallsyms]     [k] osq_lock
   2.44%  sched-messaging  [kernel.kallsyms]     [k] pipe_write                                           2.64%  sched-messaging  [kernel.kallsyms]     [k] pipe_write
   2.38%  sched-messaging  [kernel.kallsyms]     [k] pipe_read                                            2.34%  sched-messaging  [kernel.kallsyms]     [k] do_syscall_64
   2.23%  sched-messaging  [kernel.kallsyms]     [k] do_syscall_64                                        2.33%  sched-messaging  [kernel.kallsyms]     [k] pipe_read
   2.19%  sched-messaging  [kernel.kallsyms]     [k] mutex_spin_on_owner                                  2.10%  sched-messaging  [kernel.kallsyms]     [k] fdget_pos
   2.05%  swapper          [kernel.kallsyms]     [k] native_sched_clock                                   1.97%  sched-messaging  [kernel.kallsyms]     [k] vfs_write
   1.94%  sched-messaging  [kernel.kallsyms]     [k] fdget_pos                                            1.93%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSRETQ_unsafe_stack
   1.88%  sched-messaging  [kernel.kallsyms]     [k] vfs_read                                             1.91%  sched-messaging  [kernel.kallsyms]     [k] vfs_read
   1.87%  swapper          [kernel.kallsyms]     [k] psi_group_change                                     1.89%  sched-messaging  [kernel.kallsyms]     [k] mutex_spin_on_owner
   1.85%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSRETQ_unsafe_stack                           1.78%  sched-messaging  [kernel.kallsyms]     [k] current_time
   1.83%  sched-messaging  [kernel.kallsyms]     [k] vfs_write                                            1.77%  sched-messaging  [kernel.kallsyms]     [k] apparmor_file_permission
   1.68%  sched-messaging  [kernel.kallsyms]     [k] current_time                                         1.72%  sched-messaging  [kernel.kallsyms]     [k] aa_file_perm
   1.67%  sched-messaging  [kernel.kallsyms]     [k] apparmor_file_permission                             1.66%  sched-messaging  [kernel.kallsyms]     [k] rw_verify_area
   1.64%  sched-messaging  [kernel.kallsyms]     [k] aa_file_perm                                         1.59%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64_after_hwframe
   1.56%  sched-messaging  [kernel.kallsyms]     [k] rw_verify_area                                       1.38%  sched-messaging  [kernel.kallsyms]     [k] _copy_from_iter
   1.50%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64_after_hwframe                       1.38%  sched-messaging  [kernel.kallsyms]     [k] ktime_get_coarse_real_ts64_mg
   1.36%  sched-messaging  [kernel.kallsyms]     [k] ktime_get_coarse_real_ts64_mg                        1.37%  sched-messaging  [kernel.kallsyms]     [k] native_sched_clock
   1.33%  sched-messaging  [kernel.kallsyms]     [k] native_sched_clock                                   1.36%  swapper          [kernel.kallsyms]     [k] native_sched_clock
   1.29%  sched-messaging  libc.so.6             [.] read                                                 1.34%  sched-messaging  libc.so.6             [.] __GI___libc_write
   1.29%  sched-messaging  [kernel.kallsyms]     [k] _copy_from_iter                                      1.30%  sched-messaging  [kernel.kallsyms]     [k] _copy_to_iter
   1.28%  sched-messaging  [kernel.kallsyms]     [k] _copy_to_iter                                        1.29%  sched-messaging  libc.so.6             [.] read
   1.20%  sched-messaging  libc.so.6             [.] __GI___libc_write                                    1.23%  swapper          [kernel.kallsyms]     [k] psi_group_change
   1.19%  sched-messaging  [kernel.kallsyms]     [k] __mutex_lock.constprop.0                             1.10%  sched-messaging  [kernel.kallsyms]     [k] psi_group_change
   1.07%  swapper          [kernel.kallsyms]     [k] srso_alias_safe_ret                                  1.06%  sched-messaging  [kernel.kallsyms]     [k] atime_needs_update
   1.04%  sched-messaging  [kernel.kallsyms]     [k] atime_needs_update                                   1.00%  sched-messaging  [kernel.kallsyms]     [k] security_file_permission
   0.98%  sched-messaging  [kernel.kallsyms]     [k] security_file_permission                             0.97%  sched-messaging  [kernel.kallsyms]     [k] update_sd_lb_stats.constprop.0
   0.97%  sched-messaging  [kernel.kallsyms]     [k] psi_group_change                                     0.94%  sched-messaging  [kernel.kallsyms]     [k] copy_page_to_iter
   0.96%  sched-messaging  [kernel.kallsyms]     [k] copy_page_to_iter                                    0.93%  sched-messaging  [kernel.kallsyms]     [k] syscall_return_via_sysret
   0.88%  sched-messaging  [kernel.kallsyms]     [k] ksys_read                                            0.91%  sched-messaging  [kernel.kallsyms]     [k] copy_page_from_iter
   0.87%  sched-messaging  [kernel.kallsyms]     [k] syscall_return_via_sysret                            0.90%  sched-messaging  [kernel.kallsyms]     [k] ksys_write
   0.86%  sched-messaging  [kernel.kallsyms]     [k] copy_page_from_iter                                  0.89%  sched-messaging  [kernel.kallsyms]     [k] ksys_read
   0.85%  sched-messaging  [kernel.kallsyms]     [k] ksys_write                                           0.82%  sched-messaging  [kernel.kallsyms]     [k] fput
   0.79%  sched-messaging  [kernel.kallsyms]     [k] fsnotify_pre_content                                 0.82%  sched-messaging  [kernel.kallsyms]     [k] fsnotify_pre_content
   0.77%  sched-messaging  [kernel.kallsyms]     [k] fput                                                 0.78%  sched-messaging  [kernel.kallsyms]     [k] __mutex_lock.constprop.0
   0.71%  sched-messaging  [kernel.kallsyms]     [k] mutex_lock                                           0.78%  sched-messaging  [kernel.kallsyms]     [k] native_queued_spin_lock_slowpath
   0.71%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_unlock                                    0.75%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_unlock
   0.71%  swapper          [kernel.kallsyms]     [k] srso_alias_return_thunk                              0.73%  sched-messaging  [kernel.kallsyms]     [k] mutex_lock
   0.68%  sched-messaging  [kernel.kallsyms]     [k] x64_sys_call                                         0.70%  sched-messaging  [kernel.kallsyms]     [k] x64_sys_call
   0.68%  sched-messaging  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave                               0.69%  swapper          [kernel.kallsyms]     [k] srso_alias_safe_ret
   0.65%  swapper          [kernel.kallsyms]     [k] menu_select                                          0.59%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_lock
   0.57%  sched-messaging  [kernel.kallsyms]     [k] __schedule                                           0.59%  sched-messaging  [kernel.kallsyms]     [k] inode_needs_update_time.part.0
   0.56%  sched-messaging  [kernel.kallsyms]     [k] page_copy_sane                                       0.57%  sched-messaging  [kernel.kallsyms]     [k] page_copy_sane
   0.54%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_lock                                      0.52%  sched-messaging  [kernel.kallsyms]     [k] select_task_rq_fair
   0.53%  sched-messaging  [kernel.kallsyms]     [k] inode_needs_update_time.part.0                       0.51%  sched-messaging  libc.so.6             [.] __GI___pthread_enable_asynccancel
   0.48%  sched-messaging  libc.so.6             [.] __GI___pthread_enable_asynccancel                    0.50%  sched-messaging  [kernel.kallsyms]     [k] fpregs_assert_state_consistent
   0.48%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64                                     0.50%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64
   0.48%  swapper          [kernel.kallsyms]     [k] save_fpregs_to_fpstate                               0.49%  sched-messaging  [kernel.kallsyms]     [k] mutex_unlock
   0.48%  sched-messaging  [kernel.kallsyms]     [k] mutex_unlock                                         0.49%  sched-messaging  [kernel.kallsyms]     [k] __schedule
   0.47%  sched-messaging  [kernel.kallsyms]     [k] dequeue_entity                                       0.48%  sched-messaging  [kernel.kallsyms]     [k] update_load_avg
   0.46%  sched-messaging  [kernel.kallsyms]     [k] fpregs_assert_state_consistent                       0.48%  sched-messaging  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave
   0.46%  sched-messaging  [kernel.kallsyms]     [k] update_load_avg                                      0.48%  sched-messaging  [kernel.kallsyms]     [k] cpu_util
   0.46%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_se                                 0.47%  swapper          [kernel.kallsyms]     [k] srso_alias_return_thunk
   0.45%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_cfs_rq                             0.46%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_se
   0.37%  swapper          [kernel.kallsyms]     [k] __schedule                                           0.45%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_cfs_rq
   0.36%  swapper          [kernel.kallsyms]     [k] enqueue_entity                                       0.45%  swapper          [kernel.kallsyms]     [k] menu_select
   0.35%  sched-messaging  perf                  [.] sender                                               0.38%  sched-messaging  perf                  [.] sender
   0.34%  sched-messaging  [kernel.kallsyms]     [k] file_update_time                                     0.37%  sched-messaging  [kernel.kallsyms]     [k] file_update_time
   0.34%  swapper          [kernel.kallsyms]     [k] acpi_processor_ffh_cstate_enter                      0.36%  sched-messaging  [kernel.kallsyms]     [k] _find_next_and_bit
   0.33%  sched-messaging  perf                  [.] receiver                                             0.34%  sched-messaging  [kernel.kallsyms]     [k] dequeue_entity
   0.32%  sched-messaging  [kernel.kallsyms]     [k] __cond_resched                                       0.33%  sched-messaging  [kernel.kallsyms]     [k] update_curr

---

o 16-groups (11.895s [Mainline] vs 8.163s [revert])

    sudo ./perf record -a -e ibs_op/cnt_ctl=1/ -- ./perf bench sched messaging -p -t -l 100000 -g 1

    (Whole system was profiled since there are 640 tasks on a 256CPU
     setup)

							mainline			vs			revert

Samples: 10M of event 'ibs_op/cnt_ctl=1/', Event count (approx.): 3257434807546 Samples: 6M of event 'ibs_op/cnt_ctl=1/', Event count (approx.): 3115778240381

Overhead  Command          Shared Object         Symbol                                             Overhead  Command          Shared Object            Symbol
   5.07%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_safe_ret                               5.28%  sched-messaging  [kernel.kallsyms]        [k] srso_alias_safe_ret
   4.24%  sched-messaging  [kernel.kallsyms]     [k] rep_movs_alternative                              4.55%  sched-messaging  [kernel.kallsyms]        [k] rep_movs_alternative
   3.42%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_return_thunk                           3.56%  sched-messaging  [kernel.kallsyms]        [k] srso_alias_return_thunk
   3.26%  sched-messaging  [kernel.kallsyms]     [k] syscall_exit_to_user_mode                         3.44%  sched-messaging  [kernel.kallsyms]        [k] syscall_exit_to_user_mode
   2.55%  sched-messaging  [kernel.kallsyms]     [k] pipe_write                                        2.78%  sched-messaging  [kernel.kallsyms]        [k] pipe_write
   2.51%  sched-messaging  [kernel.kallsyms]     [k] osq_lock                                          2.48%  sched-messaging  [kernel.kallsyms]        [k] do_syscall_64
   2.38%  sched-messaging  [kernel.kallsyms]     [k] pipe_read                                         2.47%  sched-messaging  [kernel.kallsyms]        [k] pipe_read
   2.31%  sched-messaging  [kernel.kallsyms]     [k] do_syscall_64                                     2.15%  sched-messaging  [kernel.kallsyms]        [k] fdget_pos
   2.11%  sched-messaging  [kernel.kallsyms]     [k] mutex_spin_on_owner                               2.12%  sched-messaging  [kernel.kallsyms]        [k] vfs_write
   2.00%  sched-messaging  [kernel.kallsyms]     [k] fdget_pos                                         2.03%  sched-messaging  [kernel.kallsyms]        [k] entry_SYSRETQ_unsafe_stack
   1.93%  sched-messaging  [kernel.kallsyms]     [k] vfs_write                                         1.97%  sched-messaging  [kernel.kallsyms]        [k] vfs_read
   1.90%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSRETQ_unsafe_stack                        1.92%  sched-messaging  [kernel.kallsyms]        [k] native_sched_clock
   1.88%  sched-messaging  [kernel.kallsyms]     [k] vfs_read                                          1.87%  sched-messaging  [kernel.kallsyms]        [k] psi_group_change
   1.77%  sched-messaging  [kernel.kallsyms]     [k] native_sched_clock                                1.87%  sched-messaging  [kernel.kallsyms]        [k] current_time
   1.74%  sched-messaging  [kernel.kallsyms]     [k] current_time                                      1.83%  sched-messaging  [kernel.kallsyms]        [k] apparmor_file_permission
   1.70%  sched-messaging  [kernel.kallsyms]     [k] apparmor_file_permission                          1.79%  sched-messaging  [kernel.kallsyms]        [k] aa_file_perm
   1.67%  sched-messaging  [kernel.kallsyms]     [k] aa_file_perm                                      1.73%  sched-messaging  [kernel.kallsyms]        [k] rw_verify_area
   1.61%  sched-messaging  [kernel.kallsyms]     [k] rw_verify_area                                    1.66%  sched-messaging  [kernel.kallsyms]        [k] entry_SYSCALL_64_after_hwframe
   1.60%  sched-messaging  [kernel.kallsyms]     [k] psi_group_change                                  1.48%  sched-messaging  [kernel.kallsyms]        [k] ktime_get_coarse_real_ts64_mg
   1.56%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64_after_hwframe                    1.46%  sched-messaging  [kernel.kallsyms]        [k] _copy_from_iter
   1.38%  sched-messaging  [kernel.kallsyms]     [k] ktime_get_coarse_real_ts64_mg                     1.39%  sched-messaging  libc.so.6                [.] __GI___libc_write
   1.37%  sched-messaging  [kernel.kallsyms]     [k] _copy_from_iter                                   1.39%  sched-messaging  [kernel.kallsyms]        [k] _copy_to_iter
   1.31%  sched-messaging  [kernel.kallsyms]     [k] _copy_to_iter                                     1.37%  sched-messaging  libc.so.6                [.] read
   1.31%  sched-messaging  libc.so.6             [.] read                                              1.10%  sched-messaging  [kernel.kallsyms]        [k] atime_needs_update
   1.28%  sched-messaging  libc.so.6             [.] __GI___libc_write                                 1.07%  swapper          [kernel.kallsyms]        [k] native_sched_clock
   1.23%  swapper          [kernel.kallsyms]     [k] native_sched_clock                                1.05%  sched-messaging  [kernel.kallsyms]        [k] native_queued_spin_lock_slowpath
   1.04%  sched-messaging  [kernel.kallsyms]     [k] atime_needs_update                                1.05%  sched-messaging  [kernel.kallsyms]        [k] security_file_permission
   0.99%  sched-messaging  [kernel.kallsyms]     [k] security_file_permission                          1.00%  sched-messaging  [kernel.kallsyms]        [k] copy_page_to_iter
   0.99%  swapper          [kernel.kallsyms]     [k] psi_group_change                                  0.97%  sched-messaging  [kernel.kallsyms]        [k] copy_page_from_iter
   0.96%  sched-messaging  [kernel.kallsyms]     [k] copy_page_to_iter                                 0.97%  sched-messaging  [kernel.kallsyms]        [k] syscall_return_via_sysret
   0.91%  sched-messaging  [kernel.kallsyms]     [k] copy_page_from_iter                               0.96%  sched-messaging  [kernel.kallsyms]        [k] ksys_write
   0.90%  sched-messaging  [kernel.kallsyms]     [k] syscall_return_via_sysret                         0.95%  sched-messaging  [kernel.kallsyms]        [k] ksys_read
   0.90%  sched-messaging  [kernel.kallsyms]     [k] ksys_read                                         0.85%  sched-messaging  [kernel.kallsyms]        [k] fsnotify_pre_content
   0.90%  sched-messaging  [kernel.kallsyms]     [k] __mutex_lock.constprop.0                          0.84%  sched-messaging  [kernel.kallsyms]        [k] fput
   0.88%  sched-messaging  [kernel.kallsyms]     [k] ksys_write                                        0.82%  swapper          [kernel.kallsyms]        [k] psi_group_change
   0.80%  sched-messaging  [kernel.kallsyms]     [k] fput                                              0.80%  sched-messaging  [kernel.kallsyms]        [k] __rcu_read_unlock
   0.80%  sched-messaging  [kernel.kallsyms]     [k] fsnotify_pre_content                              0.76%  sched-messaging  [kernel.kallsyms]        [k] x64_sys_call
   0.74%  sched-messaging  [kernel.kallsyms]     [k] mutex_lock                                        0.76%  sched-messaging  [kernel.kallsyms]        [k] mutex_lock
   0.73%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_unlock                                 0.69%  sched-messaging  [kernel.kallsyms]        [k] __schedule
   0.70%  sched-messaging  [kernel.kallsyms]     [k] x64_sys_call                                      0.67%  sched-messaging  [kernel.kallsyms]        [k] osq_lock
   0.69%  swapper          [kernel.kallsyms]     [k] srso_alias_safe_ret                               0.64%  swapper          [kernel.kallsyms]        [k] srso_alias_safe_ret
   0.63%  sched-messaging  [kernel.kallsyms]     [k] __schedule                                        0.62%  sched-messaging  [kernel.kallsyms]        [k] __rcu_read_lock
   0.62%  sched-messaging  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave                            0.61%  sched-messaging  [kernel.kallsyms]        [k] inode_needs_update_time.part.0
   0.57%  sched-messaging  [kernel.kallsyms]     [k] page_copy_sane                                    0.61%  sched-messaging  [kernel.kallsyms]        [k] page_copy_sane
   0.57%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_lock                                   0.59%  sched-messaging  [kernel.kallsyms]        [k] select_task_rq_fair
   0.56%  sched-messaging  [kernel.kallsyms]     [k] inode_needs_update_time.part.0                    0.54%  sched-messaging  [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
   0.52%  sched-messaging  [kernel.kallsyms]     [k] update_sd_lb_stats.constprop.0                    0.53%  sched-messaging  libc.so.6                [.] __GI___pthread_enable_asynccancel
   0.49%  sched-messaging  [kernel.kallsyms]     [k] restore_fpregs_from_fpstate                       0.53%  sched-messaging  [kernel.kallsyms]        [k] update_load_avg
   0.49%  sched-messaging  libc.so.6             [.] __GI___pthread_enable_asynccancel                 0.52%  sched-messaging  [kernel.kallsyms]        [k] entry_SYSCALL_64
   0.49%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64                                  0.52%  sched-messaging  [kernel.kallsyms]        [k] fpregs_assert_state_consistent
   0.49%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_se                              0.52%  sched-messaging  [kernel.kallsyms]        [k] __update_load_avg_se
   0.49%  sched-messaging  [kernel.kallsyms]     [k] mutex_unlock                                      0.52%  sched-messaging  [kernel.kallsyms]        [k] mutex_spin_on_owner
   0.48%  sched-messaging  [kernel.kallsyms]     [k] update_load_avg                                   0.51%  sched-messaging  [kernel.kallsyms]        [k] mutex_unlock
   0.47%  sched-messaging  [kernel.kallsyms]     [k] fpregs_assert_state_consistent                    0.47%  sched-messaging  [kernel.kallsyms]        [k] __update_load_avg_cfs_rq
   0.46%  swapper          [kernel.kallsyms]     [k] srso_alias_return_thunk                           0.43%  swapper          [kernel.kallsyms]        [k] srso_alias_return_thunk
   0.46%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_cfs_rq                          0.41%  sched-messaging  [kernel.kallsyms]        [k] __mutex_lock.constprop.0
   0.43%  swapper          [kernel.kallsyms]     [k] menu_select                                       0.41%  sched-messaging  [kernel.kallsyms]        [k] dequeue_entity
   0.39%  sched-messaging  [kernel.kallsyms]     [k] dequeue_entity                                    0.40%  sched-messaging  [kernel.kallsyms]        [k] update_curr
   0.39%  sched-messaging  [kernel.kallsyms]     [k] native_queued_spin_lock_slowpath                  0.39%  sched-messaging  perf                     [.] sender
   0.38%  sched-messaging  [kernel.kallsyms]     [k] update_curr                                       0.39%  sched-messaging  [kernel.kallsyms]        [k] file_update_time
   0.37%  sched-messaging  perf                  [.] sender                                            0.37%  sched-messaging  [kernel.kallsyms]        [k] psi_task_switch
   0.35%  sched-messaging  [kernel.kallsyms]     [k] file_update_time                                  0.37%  swapper          [kernel.kallsyms]        [k] menu_select
   0.35%  sched-messaging  [kernel.kallsyms]     [k] select_task_rq_fair                               0.34%  sched-messaging  perf                     [.] receiver
   0.34%  sched-messaging  [kernel.kallsyms]     [k] psi_task_switch                                   0.32%  sched-messaging  [kernel.kallsyms]        [k] __calc_delta.constprop.0

---

For 1-groups I see "osq_lock" turning slightly hotter on mainline compared to the revert probably suggesting more optimistic spinning on the "pipe->mutex".

For the 16-client case, I see that "native_queued_spin_lock_slowpath" jumps up with the revert.

Adding --call-graph when profiling completely alters the profile but in case of the revert, I was able to see which paths lead to "native_queued_spin_lock_slowpath" with 16-groups case:

  Overhead  Command          Shared Object            Symbol
-    4.21%  sched-messaging  [kernel.kallsyms]        [k] native_queued_spin_lock_slowpath

- 2.77% native_queued_spin_lock_slowpath
- 1.52% _raw_spin_lock_irqsave
- 1.35% prepare_to_wait_event
- 1.34% pipe_write
vfs_write
ksys_write
do_syscall_64 entry_SYSCALL_64 __GI___libc_write
write (inlined) start_thread
- 1.25% _raw_spin_lock
- 1.25% raw_spin_rq_lock_nested
- 0.95% __task_rq_lock
- try_to_wake_up
- 0.95% autoremove_wake_function __wake_up_common __wake_up_sync_key
---

Quoted 58 lines
---
  fs/pipe.c | 19 ++++++++++---------
  1 file changed, 10 insertions(+), 9 deletions(-)
fs/pipe.c +7 -6
index 12b22c2723b7..82fede0f2111 100644
@@ -253,7 +253,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  	size_t total_len = iov_iter_count(to);
  	struct file *filp = iocb->ki_filp;
  	struct pipe_inode_info *pipe = filp->private_data;
-	bool was_full, wake_next_reader = false;
+	bool wake_writer = false, wake_next_reader = false;
  	ssize_t ret;
 
  	/* Null read succeeds. */
@@ -264,14 +264,13 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  	mutex_lock(&pipe->mutex);
 
  	/*
-	 * We only wake up writers if the pipe was full when we started
-	 * reading in order to avoid unnecessary wakeups.
+	 * We only wake up writers if the pipe was full when we started reading
+	 * and it is no longer full after reading to avoid unnecessary wakeups.
  	 *
  	 * But when we do wake up writers, we do so using a sync wakeup
  	 * (WF_SYNC), because we want them to get going and generate more
  	 * data for us.
  	 */
-	was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
  	for (;;) {
  		/* Read ->head with a barrier vs post_one_notification() */
  		unsigned int head = smp_load_acquire(&pipe->head);
@@ -340,8 +339,10 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  				buf->len = 0;
  			}
 
-			if (!buf->len)
+			if (!buf->len) {
+				wake_writer |= pipe_full(head, tail, pipe->max_usage);
  				tail = pipe_update_tail(pipe, buf, tail);
+			}
  			total_len -= chars;
  			if (!total_len)
  				break;	/* common path: read succeeded */
@@ -377,7 +378,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  		 * _very_ unlikely case that the pipe was full, but we got
  		 * no data.
  		 */
-		if (unlikely(was_full))
+		if (unlikely(wake_writer))
  			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
  		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
 

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 397

Quoted 9 lines
fs/pipe.c +2 -2
@@ -390,15 +391,15 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
  			return -ERESTARTSYS;
 
-		mutex_lock(&pipe->mutex);
-		was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
+		wake_writer = false;
  		wake_next_reader = true;
+		mutex_lock(&pipe->mutex);
  	}

Looking at the performance trend, I tried the following (possibly dumb) experiment on top of mainline:

fs/pipe.c +13 -0
index 82fede0f2111..43d827f99c55 100644
@@ -395,6 +395,19 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		wake_next_reader = true;
 		mutex_lock(&pipe->mutex);
 	}
+
+	if (!wake_writer && !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) {
+		/*
+		 * Proactively wake up writers if the pipe is not full.
+		 * This smp_mb() pairs with another barrier in ___wait_event(),
+		 * see more details in comments of waitqueue_active().
+		 */
+		smp_mb();
+
+		if (waitqueue_active(&pipe->wr_wait))
+			wake_writer = true;
+	}
+
 	if (pipe_empty(pipe->head, pipe->tail))
 		wake_next_reader = false;
 	mutex_unlock(&pipe->mutex);

base-commit: 05dbaf8dd8bf537d4b4eb3115ab42a5fb40ff1f5 --

and I see that the perfomance at lower utilization is closer to the mainline whereas at higher utlization, it is close to that with this patch reverted:

    ==================================================================
    Test          : sched-messaging
    Units         : Normalized time in seconds
    Interpretation: Lower is better
    Statistic     : AMean
    ==================================================================
    Case:         mainline[pct imp](CV)    revert[pct imp](CV)      patched[pct imp](CV)
     1-groups     1.00 [ -0.00](12.29)     1.26 [-25.91]( 2.71)     0.96 [  4.05]( 1.61)
     2-groups     1.00 [ -0.00]( 3.64)     1.39 [-38.53]( 0.89)     1.05 [ -5.26]( 0.93)
     4-groups     1.00 [ -0.00]( 3.33)     1.41 [-41.42]( 1.21)     1.04 [ -4.18]( 1.38)
     8-groups     1.00 [ -0.00]( 2.90)     1.10 [ -9.89]( 0.95)     0.84 [ 16.07]( 1.55)
    16-groups     1.00 [ -0.00]( 1.46)     0.66 [ 34.46]( 1.59)     0.50 [ 49.55]( 1.91)

The rationale was at higher utilization, perhaps there is a delay in wakeup of writers from the time tail was moved but looking at all the synchronization with "pipe->mutex", it is highly unlikely and I do not have a good explanation for why this helps (or if it is even correct)

Following are some system-wide aggregates of schedstats on each kernel running the 16-group variant collected using perf sched stats [0]:

sudo ./perf sched stats report #cord -- ./perf bench sched messaging -p -t -l 100000 -g 16

kernel                                                           :      mainline                         revert                          patched
runtime                                                          :     11.418s                           7.207s                           6.278s
sched_yield() count                                              :           0                                0                                0
Legacy counter can be ignored                                    :           0                                0                                0
schedule() called                                                :      402376                           403424                           172432
schedule() left the processor idle                               :      144622  (    35.94% )            142240  (    35.26% )             56732  (    32.90% )
try_to_wake_up() was called                                      :      237032                           241834                           101645
try_to_wake_up() was called to wake up the local cpu             :        1064  (     0.45% )             16656  (     6.89% )             12385  (    12.18% )
total runtime by tasks on this processor (in jiffies)            :  9072083005                       5516672721                       5105984838
total waittime by tasks on this processor (in jiffies)           :  4380309658  (    48.28% )        7304939649  (   132.42% )        6120940564  (   119.88% )
total timeslices run on this cpu                                 :      257644                           261129                           115628

[0] https://lore.kernel.org/lkml/[email protected]/

The trend seems to be higher local CPU wakeups albeit with more wait time but that diesn't seem to hurt the progress of sched-messaging.

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 405

Quoted 9 lines
fs/pipe.c +1 -1
  	if (pipe_empty(pipe->head, pipe->tail))
  		wake_next_reader = false;
  	mutex_unlock(&pipe->mutex);
 
-	if (was_full)
+	if (wake_writer)
  		wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
  	if (wake_next_reader)
  		wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);

If you need any more information from my test setup, please do let me know. All tests were run on a dual socket 3rd Generation EPYC system (2 x 64C/128T) running in NPS1 mode with C2 disabled and boost enabled.

Signature · 2 lines
Thanks and Regards,
Prateek
#5 lore raw

(Add Oliver and Mateusz)

Quoted 6 lines · K Prateek Nayak
Quoted 4 lines · Oleg Nesterov

wake_up(pipe->wr_wait) makes no sense if pipe_full() is still true after the reading, the writer sleeping in wait_event(wr_wait, pipe_writable()) will check the pipe_writable() == !pipe_full() condition and sleep again.

Quoted 7 lines
Quoted 2 lines

Only wake the writer if we actually released a pipe buf, and the pipe was full before we did so.

I noticed a performance regression in perf bench sched messaging at higher utilization (larger number of groups) with this patch on the mainline kernel. For lower utilization, this patch yields good improvements but once the system is oversubscribed, the tale flips.

Thanks a lot Prateek for your investigations.

I wasn't aware of tools/perf/bench/sched-messaging.c, but it seems to do the same thing as hackbench. So this was already reported, plus other "random" regressions and improvements caused by this patch. See
https://lore.kernel.org/all/[email protected]/

Yes, if the system is oversubscribed, then the early/unnecessary wakeup is not necessarily bad, but I still can't fully understand why this patch makes a noticeable difference in this case. I can't even understand why (with or without this patch) the readers sleep on rd_wait MUCH more often than the writers on wr_wait, may be because pipe_write() is generally slower than pipe_read() ...

I promised to (try to) investigate on the previous weekend, but I am a lazy dog, sorry! I'll try to do it on this weekend. Perhaps it would be better to simply revert this patch...

As for the change you propose... At first glance it doesn't look right to me, but this needs another discussion. At least it can be simplified afaics. As for the waitqueue_active() check, it probably makes sense (before wake_up) regardless, and this connects to another (confusing) discussion, please see
https://lore.kernel.org/all/[email protected]/

Thanks!

Oleg.

Quoted 377 lines

Following are the results from my testing on mainline at commit 05dbaf8dd8bf ("Merge tag 'x86-urgent-2025-01-28' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip")
with and without this patch:

   ==================================================================
    Test          : sched-messaging
    cmdline       : perf bench sched messaging -p -t -l 100000 -g <groups>
    Units         : Normalized time in seconds
    Interpretation: Lower is better
    Statistic     : AMean
    ==================================================================
    Case:         mainline[pct imp](CV)    revert[pct imp](CV)
     1-groups     1.00 [ -0.00](12.29)     1.26 [-25.91]( 2.71)
     2-groups     1.00 [ -0.00]( 3.64)     1.39 [-38.53]( 0.89)
     4-groups     1.00 [ -0.00]( 3.33)     1.41 [-41.42]( 1.21)
     8-groups     1.00 [ -0.00]( 2.90)     1.10 [ -9.89]( 0.95)
    16-groups     1.00 [ -0.00]( 1.46)     0.66 [ 34.46]( 1.59)

On my 3rd Generation EPYC system (2 x 64C/128T), I see that on reverting the changes on the above mentioned commit, sched-messaging sees a regression up until the 8 group case which contains 320 tasks, however with 16 groups (640 tasks), the revert helps with performance.

Based on the trend in the performance, one can deduce that at lower utilization, sched-messaging benefits from not traversing the wake up path unnecessarily since wake_up_interruptible_sync_poll() acquires a lock before checking if the wait queue is empty or not thus saving on system time. However, at high utilization, there is likely a writer waiting to write to the pipe by the time the wait queue is inspected.

Following are the perf profile comparing the mainline with the revert:

o 1-group (4.604s [mainline] vs 8.163s [revert])

sudo ./perf record -C 0-7,64-127 -e ibs_op/cnt_ctl=1/ -- taskset -c 0-7,64-127 ./perf bench sched messaging -p -t -l 100000 -g 1

(sched-messaging was pinned to 1 CCX and only that CCX was profiled
using IBS to reduce noise)

mainline vs revert

Samples: 606K of event 'ibs_op/cnt_ctl=1/', Event count (approx.): 205972485144 Samples: 479K of event 'ibs_op/cnt_ctl=1/', Event count (approx.): 200365591518

Overhead  Command          Shared Object         Symbol                                                Overhead  Command          Shared Object         Symbol
   4.80%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_safe_ret                                  5.12%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_safe_ret
   4.10%  sched-messaging  [kernel.kallsyms]     [k] rep_movs_alternative                                 4.30%  sched-messaging  [kernel.kallsyms]     [k] rep_movs_alternative
   3.24%  sched-messaging  [kernel.kallsyms]     [k] osq_lock                                             3.42%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_return_thunk
   3.23%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_return_thunk                              3.31%  sched-messaging  [kernel.kallsyms]     [k] syscall_exit_to_user_mode
   3.13%  sched-messaging  [kernel.kallsyms]     [k] syscall_exit_to_user_mode                            2.71%  sched-messaging  [kernel.kallsyms]     [k] osq_lock
   2.44%  sched-messaging  [kernel.kallsyms]     [k] pipe_write                                           2.64%  sched-messaging  [kernel.kallsyms]     [k] pipe_write
   2.38%  sched-messaging  [kernel.kallsyms]     [k] pipe_read                                            2.34%  sched-messaging  [kernel.kallsyms]     [k] do_syscall_64
   2.23%  sched-messaging  [kernel.kallsyms]     [k] do_syscall_64                                        2.33%  sched-messaging  [kernel.kallsyms]     [k] pipe_read
   2.19%  sched-messaging  [kernel.kallsyms]     [k] mutex_spin_on_owner                                  2.10%  sched-messaging  [kernel.kallsyms]     [k] fdget_pos
   2.05%  swapper          [kernel.kallsyms]     [k] native_sched_clock                                   1.97%  sched-messaging  [kernel.kallsyms]     [k] vfs_write
   1.94%  sched-messaging  [kernel.kallsyms]     [k] fdget_pos                                            1.93%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSRETQ_unsafe_stack
   1.88%  sched-messaging  [kernel.kallsyms]     [k] vfs_read                                             1.91%  sched-messaging  [kernel.kallsyms]     [k] vfs_read
   1.87%  swapper          [kernel.kallsyms]     [k] psi_group_change                                     1.89%  sched-messaging  [kernel.kallsyms]     [k] mutex_spin_on_owner
   1.85%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSRETQ_unsafe_stack                           1.78%  sched-messaging  [kernel.kallsyms]     [k] current_time
   1.83%  sched-messaging  [kernel.kallsyms]     [k] vfs_write                                            1.77%  sched-messaging  [kernel.kallsyms]     [k] apparmor_file_permission
   1.68%  sched-messaging  [kernel.kallsyms]     [k] current_time                                         1.72%  sched-messaging  [kernel.kallsyms]     [k] aa_file_perm
   1.67%  sched-messaging  [kernel.kallsyms]     [k] apparmor_file_permission                             1.66%  sched-messaging  [kernel.kallsyms]     [k] rw_verify_area
   1.64%  sched-messaging  [kernel.kallsyms]     [k] aa_file_perm                                         1.59%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64_after_hwframe
   1.56%  sched-messaging  [kernel.kallsyms]     [k] rw_verify_area                                       1.38%  sched-messaging  [kernel.kallsyms]     [k] _copy_from_iter
   1.50%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64_after_hwframe                       1.38%  sched-messaging  [kernel.kallsyms]     [k] ktime_get_coarse_real_ts64_mg
   1.36%  sched-messaging  [kernel.kallsyms]     [k] ktime_get_coarse_real_ts64_mg                        1.37%  sched-messaging  [kernel.kallsyms]     [k] native_sched_clock
   1.33%  sched-messaging  [kernel.kallsyms]     [k] native_sched_clock                                   1.36%  swapper          [kernel.kallsyms]     [k] native_sched_clock
   1.29%  sched-messaging  libc.so.6             [.] read                                                 1.34%  sched-messaging  libc.so.6             [.] __GI___libc_write
   1.29%  sched-messaging  [kernel.kallsyms]     [k] _copy_from_iter                                      1.30%  sched-messaging  [kernel.kallsyms]     [k] _copy_to_iter
   1.28%  sched-messaging  [kernel.kallsyms]     [k] _copy_to_iter                                        1.29%  sched-messaging  libc.so.6             [.] read
   1.20%  sched-messaging  libc.so.6             [.] __GI___libc_write                                    1.23%  swapper          [kernel.kallsyms]     [k] psi_group_change
   1.19%  sched-messaging  [kernel.kallsyms]     [k] __mutex_lock.constprop.0                             1.10%  sched-messaging  [kernel.kallsyms]     [k] psi_group_change
   1.07%  swapper          [kernel.kallsyms]     [k] srso_alias_safe_ret                                  1.06%  sched-messaging  [kernel.kallsyms]     [k] atime_needs_update
   1.04%  sched-messaging  [kernel.kallsyms]     [k] atime_needs_update                                   1.00%  sched-messaging  [kernel.kallsyms]     [k] security_file_permission
   0.98%  sched-messaging  [kernel.kallsyms]     [k] security_file_permission                             0.97%  sched-messaging  [kernel.kallsyms]     [k] update_sd_lb_stats.constprop.0
   0.97%  sched-messaging  [kernel.kallsyms]     [k] psi_group_change                                     0.94%  sched-messaging  [kernel.kallsyms]     [k] copy_page_to_iter
   0.96%  sched-messaging  [kernel.kallsyms]     [k] copy_page_to_iter                                    0.93%  sched-messaging  [kernel.kallsyms]     [k] syscall_return_via_sysret
   0.88%  sched-messaging  [kernel.kallsyms]     [k] ksys_read                                            0.91%  sched-messaging  [kernel.kallsyms]     [k] copy_page_from_iter
   0.87%  sched-messaging  [kernel.kallsyms]     [k] syscall_return_via_sysret                            0.90%  sched-messaging  [kernel.kallsyms]     [k] ksys_write
   0.86%  sched-messaging  [kernel.kallsyms]     [k] copy_page_from_iter                                  0.89%  sched-messaging  [kernel.kallsyms]     [k] ksys_read
   0.85%  sched-messaging  [kernel.kallsyms]     [k] ksys_write                                           0.82%  sched-messaging  [kernel.kallsyms]     [k] fput
   0.79%  sched-messaging  [kernel.kallsyms]     [k] fsnotify_pre_content                                 0.82%  sched-messaging  [kernel.kallsyms]     [k] fsnotify_pre_content
   0.77%  sched-messaging  [kernel.kallsyms]     [k] fput                                                 0.78%  sched-messaging  [kernel.kallsyms]     [k] __mutex_lock.constprop.0
   0.71%  sched-messaging  [kernel.kallsyms]     [k] mutex_lock                                           0.78%  sched-messaging  [kernel.kallsyms]     [k] native_queued_spin_lock_slowpath
   0.71%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_unlock                                    0.75%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_unlock
   0.71%  swapper          [kernel.kallsyms]     [k] srso_alias_return_thunk                              0.73%  sched-messaging  [kernel.kallsyms]     [k] mutex_lock
   0.68%  sched-messaging  [kernel.kallsyms]     [k] x64_sys_call                                         0.70%  sched-messaging  [kernel.kallsyms]     [k] x64_sys_call
   0.68%  sched-messaging  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave                               0.69%  swapper          [kernel.kallsyms]     [k] srso_alias_safe_ret
   0.65%  swapper          [kernel.kallsyms]     [k] menu_select                                          0.59%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_lock
   0.57%  sched-messaging  [kernel.kallsyms]     [k] __schedule                                           0.59%  sched-messaging  [kernel.kallsyms]     [k] inode_needs_update_time.part.0
   0.56%  sched-messaging  [kernel.kallsyms]     [k] page_copy_sane                                       0.57%  sched-messaging  [kernel.kallsyms]     [k] page_copy_sane
   0.54%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_lock                                      0.52%  sched-messaging  [kernel.kallsyms]     [k] select_task_rq_fair
   0.53%  sched-messaging  [kernel.kallsyms]     [k] inode_needs_update_time.part.0                       0.51%  sched-messaging  libc.so.6             [.] __GI___pthread_enable_asynccancel
   0.48%  sched-messaging  libc.so.6             [.] __GI___pthread_enable_asynccancel                    0.50%  sched-messaging  [kernel.kallsyms]     [k] fpregs_assert_state_consistent
   0.48%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64                                     0.50%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64
   0.48%  swapper          [kernel.kallsyms]     [k] save_fpregs_to_fpstate                               0.49%  sched-messaging  [kernel.kallsyms]     [k] mutex_unlock
   0.48%  sched-messaging  [kernel.kallsyms]     [k] mutex_unlock                                         0.49%  sched-messaging  [kernel.kallsyms]     [k] __schedule
   0.47%  sched-messaging  [kernel.kallsyms]     [k] dequeue_entity                                       0.48%  sched-messaging  [kernel.kallsyms]     [k] update_load_avg
   0.46%  sched-messaging  [kernel.kallsyms]     [k] fpregs_assert_state_consistent                       0.48%  sched-messaging  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave
   0.46%  sched-messaging  [kernel.kallsyms]     [k] update_load_avg                                      0.48%  sched-messaging  [kernel.kallsyms]     [k] cpu_util
   0.46%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_se                                 0.47%  swapper          [kernel.kallsyms]     [k] srso_alias_return_thunk
   0.45%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_cfs_rq                             0.46%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_se
   0.37%  swapper          [kernel.kallsyms]     [k] __schedule                                           0.45%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_cfs_rq
   0.36%  swapper          [kernel.kallsyms]     [k] enqueue_entity                                       0.45%  swapper          [kernel.kallsyms]     [k] menu_select
   0.35%  sched-messaging  perf                  [.] sender                                               0.38%  sched-messaging  perf                  [.] sender
   0.34%  sched-messaging  [kernel.kallsyms]     [k] file_update_time                                     0.37%  sched-messaging  [kernel.kallsyms]     [k] file_update_time
   0.34%  swapper          [kernel.kallsyms]     [k] acpi_processor_ffh_cstate_enter                      0.36%  sched-messaging  [kernel.kallsyms]     [k] _find_next_and_bit
   0.33%  sched-messaging  perf                  [.] receiver                                             0.34%  sched-messaging  [kernel.kallsyms]     [k] dequeue_entity
   0.32%  sched-messaging  [kernel.kallsyms]     [k] __cond_resched                                       0.33%  sched-messaging  [kernel.kallsyms]     [k] update_curr

---

o 16-groups (11.895s [Mainline] vs 8.163s [revert])

    sudo ./perf record -a -e ibs_op/cnt_ctl=1/ -- ./perf bench sched messaging -p -t -l 100000 -g 1

    (Whole system was profiled since there are 640 tasks on a 256CPU
     setup)

							mainline			vs			revert

Samples: 10M of event 'ibs_op/cnt_ctl=1/', Event count (approx.): 3257434807546 Samples: 6M of event 'ibs_op/cnt_ctl=1/', Event count (approx.): 3115778240381

Overhead  Command          Shared Object         Symbol                                             Overhead  Command          Shared Object            Symbol
   5.07%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_safe_ret                               5.28%  sched-messaging  [kernel.kallsyms]        [k] srso_alias_safe_ret
   4.24%  sched-messaging  [kernel.kallsyms]     [k] rep_movs_alternative                              4.55%  sched-messaging  [kernel.kallsyms]        [k] rep_movs_alternative
   3.42%  sched-messaging  [kernel.kallsyms]     [k] srso_alias_return_thunk                           3.56%  sched-messaging  [kernel.kallsyms]        [k] srso_alias_return_thunk
   3.26%  sched-messaging  [kernel.kallsyms]     [k] syscall_exit_to_user_mode                         3.44%  sched-messaging  [kernel.kallsyms]        [k] syscall_exit_to_user_mode
   2.55%  sched-messaging  [kernel.kallsyms]     [k] pipe_write                                        2.78%  sched-messaging  [kernel.kallsyms]        [k] pipe_write
   2.51%  sched-messaging  [kernel.kallsyms]     [k] osq_lock                                          2.48%  sched-messaging  [kernel.kallsyms]        [k] do_syscall_64
   2.38%  sched-messaging  [kernel.kallsyms]     [k] pipe_read                                         2.47%  sched-messaging  [kernel.kallsyms]        [k] pipe_read
   2.31%  sched-messaging  [kernel.kallsyms]     [k] do_syscall_64                                     2.15%  sched-messaging  [kernel.kallsyms]        [k] fdget_pos
   2.11%  sched-messaging  [kernel.kallsyms]     [k] mutex_spin_on_owner                               2.12%  sched-messaging  [kernel.kallsyms]        [k] vfs_write
   2.00%  sched-messaging  [kernel.kallsyms]     [k] fdget_pos                                         2.03%  sched-messaging  [kernel.kallsyms]        [k] entry_SYSRETQ_unsafe_stack
   1.93%  sched-messaging  [kernel.kallsyms]     [k] vfs_write                                         1.97%  sched-messaging  [kernel.kallsyms]        [k] vfs_read
   1.90%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSRETQ_unsafe_stack                        1.92%  sched-messaging  [kernel.kallsyms]        [k] native_sched_clock
   1.88%  sched-messaging  [kernel.kallsyms]     [k] vfs_read                                          1.87%  sched-messaging  [kernel.kallsyms]        [k] psi_group_change
   1.77%  sched-messaging  [kernel.kallsyms]     [k] native_sched_clock                                1.87%  sched-messaging  [kernel.kallsyms]        [k] current_time
   1.74%  sched-messaging  [kernel.kallsyms]     [k] current_time                                      1.83%  sched-messaging  [kernel.kallsyms]        [k] apparmor_file_permission
   1.70%  sched-messaging  [kernel.kallsyms]     [k] apparmor_file_permission                          1.79%  sched-messaging  [kernel.kallsyms]        [k] aa_file_perm
   1.67%  sched-messaging  [kernel.kallsyms]     [k] aa_file_perm                                      1.73%  sched-messaging  [kernel.kallsyms]        [k] rw_verify_area
   1.61%  sched-messaging  [kernel.kallsyms]     [k] rw_verify_area                                    1.66%  sched-messaging  [kernel.kallsyms]        [k] entry_SYSCALL_64_after_hwframe
   1.60%  sched-messaging  [kernel.kallsyms]     [k] psi_group_change                                  1.48%  sched-messaging  [kernel.kallsyms]        [k] ktime_get_coarse_real_ts64_mg
   1.56%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64_after_hwframe                    1.46%  sched-messaging  [kernel.kallsyms]        [k] _copy_from_iter
   1.38%  sched-messaging  [kernel.kallsyms]     [k] ktime_get_coarse_real_ts64_mg                     1.39%  sched-messaging  libc.so.6                [.] __GI___libc_write
   1.37%  sched-messaging  [kernel.kallsyms]     [k] _copy_from_iter                                   1.39%  sched-messaging  [kernel.kallsyms]        [k] _copy_to_iter
   1.31%  sched-messaging  [kernel.kallsyms]     [k] _copy_to_iter                                     1.37%  sched-messaging  libc.so.6                [.] read
   1.31%  sched-messaging  libc.so.6             [.] read                                              1.10%  sched-messaging  [kernel.kallsyms]        [k] atime_needs_update
   1.28%  sched-messaging  libc.so.6             [.] __GI___libc_write                                 1.07%  swapper          [kernel.kallsyms]        [k] native_sched_clock
   1.23%  swapper          [kernel.kallsyms]     [k] native_sched_clock                                1.05%  sched-messaging  [kernel.kallsyms]        [k] native_queued_spin_lock_slowpath
   1.04%  sched-messaging  [kernel.kallsyms]     [k] atime_needs_update                                1.05%  sched-messaging  [kernel.kallsyms]        [k] security_file_permission
   0.99%  sched-messaging  [kernel.kallsyms]     [k] security_file_permission                          1.00%  sched-messaging  [kernel.kallsyms]        [k] copy_page_to_iter
   0.99%  swapper          [kernel.kallsyms]     [k] psi_group_change                                  0.97%  sched-messaging  [kernel.kallsyms]        [k] copy_page_from_iter
   0.96%  sched-messaging  [kernel.kallsyms]     [k] copy_page_to_iter                                 0.97%  sched-messaging  [kernel.kallsyms]        [k] syscall_return_via_sysret
   0.91%  sched-messaging  [kernel.kallsyms]     [k] copy_page_from_iter                               0.96%  sched-messaging  [kernel.kallsyms]        [k] ksys_write
   0.90%  sched-messaging  [kernel.kallsyms]     [k] syscall_return_via_sysret                         0.95%  sched-messaging  [kernel.kallsyms]        [k] ksys_read
   0.90%  sched-messaging  [kernel.kallsyms]     [k] ksys_read                                         0.85%  sched-messaging  [kernel.kallsyms]        [k] fsnotify_pre_content
   0.90%  sched-messaging  [kernel.kallsyms]     [k] __mutex_lock.constprop.0                          0.84%  sched-messaging  [kernel.kallsyms]        [k] fput
   0.88%  sched-messaging  [kernel.kallsyms]     [k] ksys_write                                        0.82%  swapper          [kernel.kallsyms]        [k] psi_group_change
   0.80%  sched-messaging  [kernel.kallsyms]     [k] fput                                              0.80%  sched-messaging  [kernel.kallsyms]        [k] __rcu_read_unlock
   0.80%  sched-messaging  [kernel.kallsyms]     [k] fsnotify_pre_content                              0.76%  sched-messaging  [kernel.kallsyms]        [k] x64_sys_call
   0.74%  sched-messaging  [kernel.kallsyms]     [k] mutex_lock                                        0.76%  sched-messaging  [kernel.kallsyms]        [k] mutex_lock
   0.73%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_unlock                                 0.69%  sched-messaging  [kernel.kallsyms]        [k] __schedule
   0.70%  sched-messaging  [kernel.kallsyms]     [k] x64_sys_call                                      0.67%  sched-messaging  [kernel.kallsyms]        [k] osq_lock
   0.69%  swapper          [kernel.kallsyms]     [k] srso_alias_safe_ret                               0.64%  swapper          [kernel.kallsyms]        [k] srso_alias_safe_ret
   0.63%  sched-messaging  [kernel.kallsyms]     [k] __schedule                                        0.62%  sched-messaging  [kernel.kallsyms]        [k] __rcu_read_lock
   0.62%  sched-messaging  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave                            0.61%  sched-messaging  [kernel.kallsyms]        [k] inode_needs_update_time.part.0
   0.57%  sched-messaging  [kernel.kallsyms]     [k] page_copy_sane                                    0.61%  sched-messaging  [kernel.kallsyms]        [k] page_copy_sane
   0.57%  sched-messaging  [kernel.kallsyms]     [k] __rcu_read_lock                                   0.59%  sched-messaging  [kernel.kallsyms]        [k] select_task_rq_fair
   0.56%  sched-messaging  [kernel.kallsyms]     [k] inode_needs_update_time.part.0                    0.54%  sched-messaging  [kernel.kallsyms]        [k] _raw_spin_lock_irqsave
   0.52%  sched-messaging  [kernel.kallsyms]     [k] update_sd_lb_stats.constprop.0                    0.53%  sched-messaging  libc.so.6                [.] __GI___pthread_enable_asynccancel
   0.49%  sched-messaging  [kernel.kallsyms]     [k] restore_fpregs_from_fpstate                       0.53%  sched-messaging  [kernel.kallsyms]        [k] update_load_avg
   0.49%  sched-messaging  libc.so.6             [.] __GI___pthread_enable_asynccancel                 0.52%  sched-messaging  [kernel.kallsyms]        [k] entry_SYSCALL_64
   0.49%  sched-messaging  [kernel.kallsyms]     [k] entry_SYSCALL_64                                  0.52%  sched-messaging  [kernel.kallsyms]        [k] fpregs_assert_state_consistent
   0.49%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_se                              0.52%  sched-messaging  [kernel.kallsyms]        [k] __update_load_avg_se
   0.49%  sched-messaging  [kernel.kallsyms]     [k] mutex_unlock                                      0.52%  sched-messaging  [kernel.kallsyms]        [k] mutex_spin_on_owner
   0.48%  sched-messaging  [kernel.kallsyms]     [k] update_load_avg                                   0.51%  sched-messaging  [kernel.kallsyms]        [k] mutex_unlock
   0.47%  sched-messaging  [kernel.kallsyms]     [k] fpregs_assert_state_consistent                    0.47%  sched-messaging  [kernel.kallsyms]        [k] __update_load_avg_cfs_rq
   0.46%  swapper          [kernel.kallsyms]     [k] srso_alias_return_thunk                           0.43%  swapper          [kernel.kallsyms]        [k] srso_alias_return_thunk
   0.46%  sched-messaging  [kernel.kallsyms]     [k] __update_load_avg_cfs_rq                          0.41%  sched-messaging  [kernel.kallsyms]        [k] __mutex_lock.constprop.0
   0.43%  swapper          [kernel.kallsyms]     [k] menu_select                                       0.41%  sched-messaging  [kernel.kallsyms]        [k] dequeue_entity
   0.39%  sched-messaging  [kernel.kallsyms]     [k] dequeue_entity                                    0.40%  sched-messaging  [kernel.kallsyms]        [k] update_curr
   0.39%  sched-messaging  [kernel.kallsyms]     [k] native_queued_spin_lock_slowpath                  0.39%  sched-messaging  perf                     [.] sender
   0.38%  sched-messaging  [kernel.kallsyms]     [k] update_curr                                       0.39%  sched-messaging  [kernel.kallsyms]        [k] file_update_time
   0.37%  sched-messaging  perf                  [.] sender                                            0.37%  sched-messaging  [kernel.kallsyms]        [k] psi_task_switch
   0.35%  sched-messaging  [kernel.kallsyms]     [k] file_update_time                                  0.37%  swapper          [kernel.kallsyms]        [k] menu_select
   0.35%  sched-messaging  [kernel.kallsyms]     [k] select_task_rq_fair                               0.34%  sched-messaging  perf                     [.] receiver
   0.34%  sched-messaging  [kernel.kallsyms]     [k] psi_task_switch                                   0.32%  sched-messaging  [kernel.kallsyms]        [k] __calc_delta.constprop.0

---

For 1-groups I see "osq_lock" turning slightly hotter on mainline compared to the revert probably suggesting more optimistic spinning on the "pipe->mutex".

For the 16-client case, I see that "native_queued_spin_lock_slowpath" jumps up with the revert.

Adding --call-graph when profiling completely alters the profile but in case of the revert, I was able to see which paths lead to "native_queued_spin_lock_slowpath" with 16-groups case:

  Overhead  Command          Shared Object            Symbol
-    4.21%  sched-messaging  [kernel.kallsyms]        [k] native_queued_spin_lock_slowpath

- 2.77% native_queued_spin_lock_slowpath
- 1.52% _raw_spin_lock_irqsave
- 1.35% prepare_to_wait_event
- 1.34% pipe_write
vfs_write
ksys_write
do_syscall_64 entry_SYSCALL_64 __GI___libc_write
write (inlined) start_thread
- 1.25% _raw_spin_lock
- 1.25% raw_spin_rq_lock_nested
- 0.95% __task_rq_lock
- try_to_wake_up
- 0.95% autoremove_wake_function __wake_up_common __wake_up_sync_key
---

Quoted 54 lines
---
 fs/pipe.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
fs/pipe.c +7 -6
index 12b22c2723b7..82fede0f2111 100644
@@ -253,7 +253,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	size_t total_len = iov_iter_count(to);
 	struct file *filp = iocb->ki_filp;
 	struct pipe_inode_info *pipe = filp->private_data;
-	bool was_full, wake_next_reader = false;
+	bool wake_writer = false, wake_next_reader = false;
 	ssize_t ret;
 	/* Null read succeeds. */
@@ -264,14 +264,13 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	mutex_lock(&pipe->mutex);
 	/*
-	 * We only wake up writers if the pipe was full when we started
-	 * reading in order to avoid unnecessary wakeups.
+	 * We only wake up writers if the pipe was full when we started reading
+	 * and it is no longer full after reading to avoid unnecessary wakeups.
 	 *
 	 * But when we do wake up writers, we do so using a sync wakeup
 	 * (WF_SYNC), because we want them to get going and generate more
 	 * data for us.
 	 */
-	was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
 	for (;;) {
 		/* Read ->head with a barrier vs post_one_notification() */
 		unsigned int head = smp_load_acquire(&pipe->head);
@@ -340,8 +339,10 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 				buf->len = 0;
 			}
-			if (!buf->len)
+			if (!buf->len) {
+				wake_writer |= pipe_full(head, tail, pipe->max_usage);
 				tail = pipe_update_tail(pipe, buf, tail);
+			}
 			total_len -= chars;
 			if (!total_len)
 				break;	/* common path: read succeeded */
@@ -377,7 +378,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		 * _very_ unlikely case that the pipe was full, but we got
 		 * no data.
 		 */
-		if (unlikely(was_full))
+		if (unlikely(wake_writer))
 			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
 		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 396

Quoted 8 lines
fs/pipe.c +2 -2
@@ -390,15 +391,15 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
 			return -ERESTARTSYS;
-		mutex_lock(&pipe->mutex);
-		was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
+		wake_writer = false;
 		wake_next_reader = true;
+		mutex_lock(&pipe->mutex);
 	}

Looking at the performance trend, I tried the following (possibly dumb) experiment on top of mainline:

fs/pipe.c +13 -0
index 82fede0f2111..43d827f99c55 100644
@@ -395,6 +395,19 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		wake_next_reader = true;
 		mutex_lock(&pipe->mutex);
 	}
+
+	if (!wake_writer && !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) {
+		/*
+		 * Proactively wake up writers if the pipe is not full.
+		 * This smp_mb() pairs with another barrier in ___wait_event(),
+		 * see more details in comments of waitqueue_active().
+		 */
+		smp_mb();
+
+		if (waitqueue_active(&pipe->wr_wait))
+			wake_writer = true;
+	}
+
 	if (pipe_empty(pipe->head, pipe->tail))
 		wake_next_reader = false;
 	mutex_unlock(&pipe->mutex);

base-commit: 05dbaf8dd8bf537d4b4eb3115ab42a5fb40ff1f5 --

and I see that the perfomance at lower utilization is closer to the mainline whereas at higher utlization, it is close to that with this patch reverted:

    ==================================================================
    Test          : sched-messaging
    Units         : Normalized time in seconds
    Interpretation: Lower is better
    Statistic     : AMean
    ==================================================================
    Case:         mainline[pct imp](CV)    revert[pct imp](CV)      patched[pct imp](CV)
     1-groups     1.00 [ -0.00](12.29)     1.26 [-25.91]( 2.71)     0.96 [  4.05]( 1.61)
     2-groups     1.00 [ -0.00]( 3.64)     1.39 [-38.53]( 0.89)     1.05 [ -5.26]( 0.93)
     4-groups     1.00 [ -0.00]( 3.33)     1.41 [-41.42]( 1.21)     1.04 [ -4.18]( 1.38)
     8-groups     1.00 [ -0.00]( 2.90)     1.10 [ -9.89]( 0.95)     0.84 [ 16.07]( 1.55)
    16-groups     1.00 [ -0.00]( 1.46)     0.66 [ 34.46]( 1.59)     0.50 [ 49.55]( 1.91)

The rationale was at higher utilization, perhaps there is a delay in wakeup of writers from the time tail was moved but looking at all the synchronization with "pipe->mutex", it is highly unlikely and I do not have a good explanation for why this helps (or if it is even correct)

Following are some system-wide aggregates of schedstats on each kernel running the 16-group variant collected using perf sched stats [0]:

sudo ./perf sched stats report #cord -- ./perf bench sched messaging -p -t -l 100000 -g 16

kernel                                                           :      mainline                         revert                          patched
runtime                                                          :     11.418s                           7.207s                           6.278s
sched_yield() count                                              :           0                                0                                0
Legacy counter can be ignored                                    :           0                                0                                0
schedule() called                                                :      402376                           403424                           172432
schedule() left the processor idle                               :      144622  (    35.94% )            142240  (    35.26% )             56732  (    32.90% )
try_to_wake_up() was called                                      :      237032                           241834                           101645
try_to_wake_up() was called to wake up the local cpu             :        1064  (     0.45% )             16656  (     6.89% )             12385  (    12.18% )
total runtime by tasks on this processor (in jiffies)            :  9072083005                       5516672721                       5105984838
total waittime by tasks on this processor (in jiffies)           :  4380309658  (    48.28% )        7304939649  (   132.42% )        6120940564  (   119.88% )
total timeslices run on this cpu                                 :      257644                           261129                           115628

[0] https://lore.kernel.org/lkml/[email protected]/

The trend seems to be higher local CPU wakeups albeit with more wait time but that diesn't seem to hurt the progress of sched-messaging.

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 403

Quoted 8 lines
fs/pipe.c +1 -1
 	if (pipe_empty(pipe->head, pipe->tail))
 		wake_next_reader = false;
 	mutex_unlock(&pipe->mutex);
-	if (was_full)
+	if (wake_writer)
 		wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
 	if (wake_next_reader)
 		wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);

If you need any more information from my test setup, please do let me know. All tests were run on a dual socket 3rd Generation EPYC system (2 x 64C/128T) running in NPS1 mode with C2 disabled and boost enabled.

--
Thanks and Regards,
Prateek

#6 lore raw
In reply to K Prateek Nayak
Quoted 5 lines · K Prateek Nayak

On my 3rd Generation EPYC system (2 x 64C/128T), I see that on reverting the changes on the above mentioned commit, sched-messaging sees a regression up until the 8 group case which contains 320 tasks, however with 16 groups (640 tasks), the revert helps with performance.

I suspect that the extra wakeups just end up perturbing timing, and then you just randomly get better performance on that particular test-case and machine.

I'm not sure this is worth worrying about, unless there's a real load somewhere that shows this regression.

Linus

#7 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · K Prateek Nayak

On my 3rd Generation EPYC system (2 x 64C/128T), I see that on reverting the changes on the above mentioned commit, sched-messaging sees a regression up until the 8 group case which contains 320 tasks, however with 16 groups (640 tasks), the revert helps with performance.

Quoted 6 lines

I suspect that the extra wakeups just end up perturbing timing, and then you just randomly get better performance on that particular test-case and machine.

I'm not sure this is worth worrying about, unless there's a real load somewhere that shows this regression.

Well yes, but the problem is that people seem to believe that hackbench is the "real" workload, even in the "overloaded" case...

And if we do care about performance... Could you look at the trivial patch at the end? I don't think {a,c,m}time make any sense in the !fifo case, but as you explained before they are visible to fstat() so we probably shouldn't remove file_accessed/file_update_time unconditionally.

This patch does help if I change hackbench to uses pipe2(O_NOATIME) instead of pipe(). And in fact it helps even in the simplest case:

	static char buf[17 * 4096];

	static struct timeval TW, TR;

	int wr(int fd, int size)
	{
		int c, r;
		struct timeval t0, t1;

		gettimeofday(&t0, NULL);
		for (c = 0; (r = write(fd, buf, size)) > 0; c += r);
		gettimeofday(&t1, NULL);
		timeradd(&TW, &t1, &TW);
		timersub(&TW, &t0, &TW);

		return c;
	}

	int rd(int fd, int size)
	{
		int c, r;
		struct timeval t0, t1;

		gettimeofday(&t0, NULL);
		for (c = 0; (r = read(fd, buf, size)) > 0; c += r);
		gettimeofday(&t1, NULL);
		timeradd(&TR, &t1, &TR);
		timersub(&TR, &t0, &TR);

		return c;
	}

	int main(int argc, const char *argv[])
	{
		int fd[2], nb = 1, noat, loop, size;

		assert(argc == 4);
		noat = atoi(argv[1]) ? O_NOATIME : 0;
		loop = atoi(argv[2]);
		size = atoi(argv[3]);

		assert(pipe2(fd, noat) == 0);
		assert(ioctl(fd[0], FIONBIO, &nb) == 0);
		assert(ioctl(fd[1], FIONBIO, &nb) == 0);

		assert(size <= sizeof(buf));
		while (loop--)
			assert(wr(fd[1], size) == rd(fd[0], size));

		printf("TW = %lu.%03lu\n", TW.tv_sec, TW.tv_usec/1000);
		printf("TR = %lu.%03lu\n", TR.tv_sec, TR.tv_usec/1000);

		return 0;
	}

Now,

	/# for i in 1 2 3; do /host/tmp/test 0 10000 100; done
	TW = 7.692
	TR = 5.704
	TW = 7.930
	TR = 5.858
	TW = 7.685
	TR = 5.697
	/#
	/# for i in 1 2 3; do /host/tmp/test 1 10000 100; done
	TW = 6.432
	TR = 4.533
	TW = 6.612
	TR = 4.638
	TW = 6.409
	TR = 4.523

Oleg.
---

fs/pipe.c +5 -2
index a3f5fd7256e9..14b2c0f8b616 100644
@@ -1122,6 +1122,9 @@ int create_pipe_files(struct file **res, int flags)
 		}
 	}
 
+	if (flags & O_NOATIME)
+		inode->i_flags |= S_NOCMTIME;
+
 	f = alloc_file_pseudo(inode, pipe_mnt, "",
 				O_WRONLY | (flags & (O_NONBLOCK | O_DIRECT)),
 				&pipefifo_fops);
@@ -1134,7 +1137,7 @@ int create_pipe_files(struct file **res, int flags)
 	f->private_data = inode->i_pipe;
 	f->f_pipe = 0;
 
-	res[0] = alloc_file_clone(f, O_RDONLY | (flags & O_NONBLOCK),
+	res[0] = alloc_file_clone(f, O_RDONLY | (flags & (O_NONBLOCK | O_NOATIME)),
 				  &pipefifo_fops);
 	if (IS_ERR(res[0])) {
 		put_pipe_info(inode, inode->i_pipe);
@@ -1154,7 +1157,7 @@ static int __do_pipe_flags(int *fd, struct file **files, int flags)
 	int error;
 	int fdw, fdr;
 
-	if (flags & ~(O_CLOEXEC | O_NONBLOCK | O_DIRECT | O_NOTIFICATION_PIPE))
+	if (flags & ~(O_CLOEXEC | O_NONBLOCK | O_DIRECT | O_NOATIME | O_NOTIFICATION_PIPE))
 		return -EINVAL;
 
 	error = create_pipe_files(files, flags);
#8 lore raw
Quoted 5 lines · Oleg Nesterov

And if we do care about performance... Could you look at the trivial patch at the end? I don't think {a,c,m}time make any sense in the !fifo case, but as you explained before they are visible to fstat() so we probably shouldn't remove file_accessed/file_update_time unconditionally.

I dislike that patch because if we actually want to do this, I don't think you are going far enough.

Yeah, you may stop updating the time, but you still do that sb_start_write_trylock(), and you still call out to file_update_time(), and it's all fairly expensive.

So the short-circuiting happens too late, and it happens using a flag that is non-standard and only with a system call that almost nobody actually uses (ie 'pipe2()' rather than the normal 'pipe()').

Put another way: if we really care about this, we should just be a lot more aggressive.

Yes, the time is visible in fstat(). Yes, we've done this forever. But if the time update is such a big thing, let's go all in, and just see if anybody really notices?

For example, for tty's, a few years ago we intentionally started doing time updates only every few seconds, because it was leaking keyboard timing information (see tty_update_time()). Nobody ever complained.

So I'd actually favor a "let's just remove time updates entirely for unnamed pipes", and see if anybody notices. Simpler and more straightforward.

And yes, maybe somebody *does* notice, and we'll have to revisit.

IOW, if you care about this, I'd *much* rather try to do the big and simple approach _first_. Not do something small and timid that nobody will actually ever use and that complicates the code.

Linus

#9 lore raw
Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Oleg Nesterov

And if we do care about performance... Could you look at the trivial patch at the end? I don't think {a,c,m}time make any sense in the !fifo case, but as you explained before they are visible to fstat() so we probably shouldn't remove file_accessed/file_update_time unconditionally.

I dislike that patch because if we actually want to do this, I don't think you are going far enough.

...

Oh yes, yes, I agree, and for the same reasons, including the unnecessary sb_start_write_trylock() even if it is likely very cheap. Plus it doesn't look consistent in that "f_flags & O_NOATIME" can be changed by fcntl() but "i_flags & S_NOCMTIME" can't be changed. Not to mention that this "feature" will probably be never used.

In case it was not clear, I just tried to measure how much file_accessed/file_update_time hurt performance-wise. It turns out - a lot. And the ugly O_NOATIME knob simplifies the before/after testing.

However, yes I was worried about fstat(). But,

Quoted 3 lines

So I'd actually favor a "let's just remove time updates entirely for unnamed pipes", and see if anybody notices. Simpler and more straightforward.

OK, agreed. Will send the patch.

Oleg.

reviewer Tested-by
#11 lore raw
In reply to Oleg Nesterov

Hello Oleg,

Thank you for pointing me to the regression reports and the relevant upstream discussions on the parallel thread.

Quoted 101 lines · Oleg Nesterov
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · K Prateek Nayak

On my 3rd Generation EPYC system (2 x 64C/128T), I see that on reverting the changes on the above mentioned commit, sched-messaging sees a regression up until the 8 group case which contains 320 tasks, however with 16 groups (640 tasks), the revert helps with performance.

Quoted 6 lines

I suspect that the extra wakeups just end up perturbing timing, and then you just randomly get better performance on that particular test-case and machine.

I'm not sure this is worth worrying about, unless there's a real load somewhere that shows this regression.

Well yes, but the problem is that people seem to believe that hackbench is the "real" workload, even in the "overloaded" case...

And if we do care about performance... Could you look at the trivial patch at the end? I don't think {a,c,m}time make any sense in the !fifo case, but as you explained before they are visible to fstat() so we probably shouldn't remove file_accessed/file_update_time unconditionally.

This patch does help if I change hackbench to uses pipe2(O_NOATIME) instead of pipe(). And in fact it helps even in the simplest case:

	static char buf[17 * 4096];

	static struct timeval TW, TR;

	int wr(int fd, int size)
	{
		int c, r;
		struct timeval t0, t1;

		gettimeofday(&t0, NULL);
		for (c = 0; (r = write(fd, buf, size)) > 0; c += r);
		gettimeofday(&t1, NULL);
		timeradd(&TW, &t1, &TW);
		timersub(&TW, &t0, &TW);

		return c;
	}

	int rd(int fd, int size)
	{
		int c, r;
		struct timeval t0, t1;

		gettimeofday(&t0, NULL);
		for (c = 0; (r = read(fd, buf, size)) > 0; c += r);
		gettimeofday(&t1, NULL);
		timeradd(&TR, &t1, &TR);
		timersub(&TR, &t0, &TR);

		return c;
	}

	int main(int argc, const char *argv[])
	{
		int fd[2], nb = 1, noat, loop, size;

		assert(argc == 4);
		noat = atoi(argv[1]) ? O_NOATIME : 0;
		loop = atoi(argv[2]);
		size = atoi(argv[3]);

		assert(pipe2(fd, noat) == 0);
		assert(ioctl(fd[0], FIONBIO, &nb) == 0);
		assert(ioctl(fd[1], FIONBIO, &nb) == 0);

		assert(size <= sizeof(buf));
		while (loop--)
			assert(wr(fd[1], size) == rd(fd[0], size));

		printf("TW = %lu.%03lu\n", TW.tv_sec, TW.tv_usec/1000);
		printf("TR = %lu.%03lu\n", TR.tv_sec, TR.tv_usec/1000);

		return 0;
	}

Now,

	/# for i in 1 2 3; do /host/tmp/test 0 10000 100; done
	TW = 7.692
	TR = 5.704
	TW = 7.930
	TR = 5.858
	TW = 7.685
	TR = 5.697
	/#
	/# for i in 1 2 3; do /host/tmp/test 1 10000 100; done
	TW = 6.432
	TR = 4.533
	TW = 6.612
	TR = 4.638
	TW = 6.409
	TR = 4.523
Quoted 3 lines

Oleg.
---

With the below patch on mainline, I see more improvements for a modified version of sched-messaging (sched-messaging is same as hackbench as you noted on the parallel thread) that uses pipe2(O_NOATIME)

The original regression is still noticeable despite the improvements but if folks believe this is a corner case with the original changes exhibited by sched-messaging, I'll just continue further testing with the new baseline.

That said, following are the results for the below patch:

==================================================================

    Test          : sched-messaging
    cmdline       : perf bench sched messaging -p -t -l 100000 -g <groups>
    Units         : Normalized time in seconds
    Interpretation: Lower is better
    Statistic     : AMean
    ==================================================================
    Case:         mainline[pct imp](CV)    patched[pct imp](CV)
     1-groups     1.00 [ -0.00](12.29)     0.94 [  6.76]( 12.59)
     2-groups     1.00 [ -0.00]( 3.64)     0.95 [  5.16](  5.99)
     4-groups     1.00 [ -0.00]( 3.33)     0.99 [  1.03](  1.89)
     8-groups     1.00 [ -0.00]( 2.90)     1.00 [  0.16](  1.23)
    16-groups     1.00 [ -0.00]( 1.46)     0.98 [  2.01](  0.98)

Please feel free to add:

I'll give your generalized optimization a spin too when it comes out. Meanwhile, I'll go run a bunch of benchmarks to see if the original change has affected any other workload in my test bed.

Thank you for looking into this.

Signature · 2 lines
Thanks and Regards,
Prateek
Quoted 33 lines
fs/pipe.c +5 -2
index a3f5fd7256e9..14b2c0f8b616 100644
@@ -1122,6 +1122,9 @@ int create_pipe_files(struct file **res, int flags)
  		}
  	}
 
+	if (flags & O_NOATIME)
+		inode->i_flags |= S_NOCMTIME;
+
  	f = alloc_file_pseudo(inode, pipe_mnt, "",
  				O_WRONLY | (flags & (O_NONBLOCK | O_DIRECT)),
  				&pipefifo_fops);
@@ -1134,7 +1137,7 @@ int create_pipe_files(struct file **res, int flags)
  	f->private_data = inode->i_pipe;
  	f->f_pipe = 0;
 
-	res[0] = alloc_file_clone(f, O_RDONLY | (flags & O_NONBLOCK),
+	res[0] = alloc_file_clone(f, O_RDONLY | (flags & (O_NONBLOCK | O_NOATIME)),
  				  &pipefifo_fops);
  	if (IS_ERR(res[0])) {
  		put_pipe_info(inode, inode->i_pipe);
@@ -1154,7 +1157,7 @@ static int __do_pipe_flags(int *fd, struct file **files, int flags)
  	int error;
  	int fdw, fdr;
 
-	if (flags & ~(O_CLOEXEC | O_NONBLOCK | O_DIRECT | O_NOTIFICATION_PIPE))
+	if (flags & ~(O_CLOEXEC | O_NONBLOCK | O_DIRECT | O_NOATIME | O_NOTIFICATION_PIPE))
  		return -EINVAL;
 
  	error = create_pipe_files(files, flags);
#10 lore raw
In reply to Linus Torvalds
Quoted 33 lines · Linus Torvalds
Quoted 5 lines · Oleg Nesterov

And if we do care about performance... Could you look at the trivial patch at the end? I don't think {a,c,m}time make any sense in the !fifo case, but as you explained before they are visible to fstat() so we probably shouldn't remove file_accessed/file_update_time unconditionally.

I dislike that patch because if we actually want to do this, I don't think you are going far enough.

Yeah, you may stop updating the time, but you still do that sb_start_write_trylock(), and you still call out to file_update_time(), and it's all fairly expensive.

So the short-circuiting happens too late, and it happens using a flag that is non-standard and only with a system call that almost nobody actually uses (ie 'pipe2()' rather than the normal 'pipe()').

Put another way: if we really care about this, we should just be a lot more aggressive.

Yes, the time is visible in fstat(). Yes, we've done this forever. But if the time update is such a big thing, let's go all in, and just see if anybody really notices?

For example, for tty's, a few years ago we intentionally started doing time updates only every few seconds, because it was leaking keyboard timing information (see tty_update_time()). Nobody ever complained.

So I'd actually favor a "let's just remove time updates entirely for unnamed pipes", and see if anybody notices. Simpler and more straightforward.

Quoted 5 lines

And yes, maybe somebody *does* notice, and we'll have to revisit.

IOW, if you care about this, I'd *much* rather try to do the big and simple approach _first_. Not do something small and timid that nobody will actually ever use and that complicates the code.

Agreed.

#12 lore raw
In reply to K Prateek Nayak
Quoted 5 lines · K Prateek Nayak

With the below patch on mainline, I see more improvements for a modified version of sched-messaging (sched-messaging is same as hackbench as you noted on the parallel thread) that uses pipe2(O_NOATIME)

Thanks,

Quoted 4 lines

The original regression is still noticeable despite the improvements but if folks believe this is a corner case with the original changes exhibited by sched-messaging, I'll just continue further testing with the new baseline.

I still don't know if we should worry or not... But if we want to try to improve the wake_writer logic, then I think it makes sense to cleanup this code first.

IMO the (untested) patch below makes sense regardless, I am going to send it after I grep fs/splice.c a bit more.

a194dfe6e6f6f ("pipe: Rearrange sequence in pipe_write() to preallocate slot") changed pipe_write() to increment pipe->head in advance. IIUC to avoid the race with the post_one_notification()-like code which can add another buffer under pipe->rd_wait.lock without pipe->mutex.

This is no longer necessary after c73be61cede ("pipe: Add general notification queue support"), pipe_write() checks pipe_has_watch_queue() and returns -EXDEV at the start. And can't help in any case, pipe_write() no longer takes this spinlock.

Change pipe_write() to call copy_page_from_iter() first and do nothing if it fails. This way pipe_write() can't add a zero-sized bufer and we can simplify pipe_read() which currently has to handle this very unlikely case.

Oleg.

fs/pipe.c +9 -35
index baaa8c0817f1..0816070a5e7a 100644
@@ -312,6 +312,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 			size_t written;
 			int error;
 
+			WARN_ON_ONCE(chars == 0);
 			if (chars > total_len) {
 				if (buf->flags & PIPE_BUF_FLAG_WHOLE) {
 					if (ret == 0)
@@ -365,29 +366,9 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 			break;
 		}
 		mutex_unlock(&pipe->mutex);
-
 		/*
 		 * We only get here if we didn't actually read anything.
 		 *
-		 * However, we could have seen (and removed) a zero-sized
-		 * pipe buffer, and might have made space in the buffers
-		 * that way.
-		 *
-		 * You can't make zero-sized pipe buffers by doing an empty
-		 * write (not even in packet mode), but they can happen if
-		 * the writer gets an EFAULT when trying to fill a buffer
-		 * that already got allocated and inserted in the buffer
-		 * array.
-		 *
-		 * So we still need to wake up any pending writers in the
-		 * _very_ unlikely case that the pipe was full, but we got
-		 * no data.
-		 */
-		if (unlikely(wake_writer))
-			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
-		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
-
-		/*
 		 * But because we didn't read anything, at this point we can
 		 * just return directly with -ERESTARTSYS if we're interrupted,
 		 * since we've done any required wakeups and there's no need
@@ -396,7 +377,6 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
 			return -ERESTARTSYS;
 
-		wake_writer = false;
 		wake_next_reader = true;
 		mutex_lock(&pipe->mutex);
 	}
@@ -524,31 +504,25 @@ pipe_write(struct kiocb *iocb, struct iov_iter *from)
 				pipe->tmp_page = page;
 			}
 
-			/* Allocate a slot in the ring in advance and attach an
-			 * empty buffer.  If we fault or otherwise fail to use
-			 * it, either the reader will consume it or it'll still
-			 * be there for the next write.
-			 */
-			pipe->head = head + 1;
+			copied = copy_page_from_iter(page, 0, PAGE_SIZE, from);
+			if (unlikely(copied < PAGE_SIZE && iov_iter_count(from))) {
+				if (!ret)
+					ret = -EFAULT;
+				break;
+			}
 
+			pipe->head = head + 1;
+			pipe->tmp_page = NULL;
 			/* Insert it into the buffer array */
 			buf = &pipe->bufs[head & mask];
 			buf->page = page;
 			buf->ops = &anon_pipe_buf_ops;
 			buf->offset = 0;
-			buf->len = 0;
 			if (is_packetized(filp))
 				buf->flags = PIPE_BUF_FLAG_PACKET;
 			else
 				buf->flags = PIPE_BUF_FLAG_CAN_MERGE;
-			pipe->tmp_page = NULL;
 
-			copied = copy_page_from_iter(page, 0, PAGE_SIZE, from);
-			if (unlikely(copied < PAGE_SIZE && iov_iter_count(from))) {
-				if (!ret)
-					ret = -EFAULT;
-				break;
-			}
 			ret += copied;
 			buf->len = copied;
 
#13 lore raw
In reply to Oleg Nesterov

Hello Oleg,

Quoted 7 lines · Oleg Nesterov

wake_up(pipe->wr_wait) makes no sense if pipe_full() is still true after the reading, the writer sleeping in wait_event(wr_wait, pipe_writable()) will check the pipe_writable() == !pipe_full() condition and sleep again.

Only wake the writer if we actually released a pipe buf, and the pipe was full before we did so.

We saw hang in hackbench in our weekly regression testing on mainline kernel. The bisect pointed to this commit.

This patch avoids the unnecessary writer wakeup but I think there may be a subtle race due to which the writer is never woken up in certain cases.

On zen5 system with 2 sockets with 192C/384T each, I ran hackbench with 16 groups or 32 groups. In 1 out of 20 runs, the race condition is occurring where the writer is not getting woken up and the benchmarks hangs. I tried reverting this commit and it again started working fine.

I also tried with
https://lore.kernel.org/all/[email protected]/. After applying this patch, the frequency of hang is reduced to 1 in 100 times, but hang still exists.

Whenever I compare the case where was_full would have been set but wake_writer was not set, I see the following pattern:

ret = 100 (Read was successful)
pipe_full() = 1
total_len = 0
buf->len != 0

total_len is computed using iov_iter_count() while the buf->len is the length of the buffer corresponding to tail(pipe->bufs[tail & mask].len).
Looking at pipe_write(), there seems to be a case where the writer can make progress when (chars && !was_empty) which only looks at iov_iter_count(). Could it be the case that there is still room in the buffer but we are not waking up the writer?

Quoted 65 lines
---
  fs/pipe.c | 19 ++++++++++---------
  1 file changed, 10 insertions(+), 9 deletions(-)
fs/pipe.c +9 -8
index 12b22c2723b7..82fede0f2111 100644
@@ -253,7 +253,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  	size_t total_len = iov_iter_count(to);
  	struct file *filp = iocb->ki_filp;
  	struct pipe_inode_info *pipe = filp->private_data;
-	bool was_full, wake_next_reader = false;
+	bool wake_writer = false, wake_next_reader = false;
  	ssize_t ret;
 
  	/* Null read succeeds. */
@@ -264,14 +264,13 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  	mutex_lock(&pipe->mutex);
 
  	/*
-	 * We only wake up writers if the pipe was full when we started
-	 * reading in order to avoid unnecessary wakeups.
+	 * We only wake up writers if the pipe was full when we started reading
+	 * and it is no longer full after reading to avoid unnecessary wakeups.
  	 *
  	 * But when we do wake up writers, we do so using a sync wakeup
  	 * (WF_SYNC), because we want them to get going and generate more
  	 * data for us.
  	 */
-	was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
  	for (;;) {
  		/* Read ->head with a barrier vs post_one_notification() */
  		unsigned int head = smp_load_acquire(&pipe->head);
@@ -340,8 +339,10 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  				buf->len = 0;
  			}
 
-			if (!buf->len)
+			if (!buf->len) {
+				wake_writer |= pipe_full(head, tail, pipe->max_usage);
  				tail = pipe_update_tail(pipe, buf, tail);
+			}
  			total_len -= chars;
  			if (!total_len)
  				break;	/* common path: read succeeded */
@@ -377,7 +378,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  		 * _very_ unlikely case that the pipe was full, but we got
  		 * no data.
  		 */
-		if (unlikely(was_full))
+		if (unlikely(wake_writer))
  			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
  		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
 
@@ -390,15 +391,15 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
  			return -ERESTARTSYS;
 
-		mutex_lock(&pipe->mutex);
-		was_full = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
+		wake_writer = false;
  		wake_next_reader = true;
+		mutex_lock(&pipe->mutex);

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 405

Quoted 10 lines
fs/pipe.c +1 -1
  	}
  	if (pipe_empty(pipe->head, pipe->tail))
  		wake_next_reader = false;
  	mutex_unlock(&pipe->mutex);
 
-	if (was_full)
+	if (wake_writer)
  		wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
  	if (wake_next_reader)
  		wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);

--
Thanks and Regards,
Swapnil

#14 lore raw

Hi Sapkal,

Quoted 3 lines · Swapnil

We saw hang in hackbench in our weekly regression testing on mainline kernel. The bisect pointed to this commit.

OMG. This patch caused a lot of "hackbench performance degradation" reports, but hang??

Just in case, did you use

https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/hackbench/hackbench.c

?

OK, I gave up ;) I'll send the revert patch tomorrow (can't do this today) even if I still don't see how this patch can be wrong.

Quoted 8 lines

Whenever I compare the case where was_full would have been set but wake_writer was not set, I see the following pattern:

ret = 100 (Read was successful)
pipe_full() = 1
total_len = 0
buf->len != 0

Quoted 6 lines

total_len is computed using iov_iter_count() while the buf->len is the length of the buffer corresponding to tail(pipe->bufs[tail & mask].len). Looking at pipe_write(), there seems to be a case where the writer can make progress when (chars && !was_empty) which only looks at iov_iter_count(). Could it be the case that there is still room in the buffer but we are not waking up the writer?

I don't think so, but perhaps I am totally confused.

If the writer sleeps on pipe->wr_wait, it has already tried to write into the pipe->bufs[head - 1] buffer before the sleep.

Yes, the reader can read from that buffer, but this won't make it more "writable" for this particular writer, "PAGE_SIZE - buf->offset + buf->len" won't be changed. I even wrote the test-case, let me quote my old email below.

Thanks,

Oleg.
--------------------------------------------------------------------------------

Meanwhile I wrote a stupid test-case below.

Without the patch

State: S (sleeping)
voluntary_ctxt_switches: 74
nonvoluntary_ctxt_switches: 5
State: S (sleeping)
voluntary_ctxt_switches: 4169
nonvoluntary_ctxt_switches: 5
finally release the buffer
wrote next char!

With the patch

State: S (sleeping)
voluntary_ctxt_switches: 74
nonvoluntary_ctxt_switches: 3
State: S (sleeping)
voluntary_ctxt_switches: 74
nonvoluntary_ctxt_switches: 3
finally release the buffer
wrote next char!

As you can see, without this patch pipe_read() wakes the writer up 4095 times for no reason, the writer burns a bit of CPU and blocks again after wakeup until the last read(fd[0], &c, 1).

Oleg.

-------------------------------------------------------------------------------
#include <stdlib.h>
#include <unistd.h>
#include <assert.h>
#include <sys/ioctl.h>
#include <stdio.h>
#include <errno.h>

int main(void)

{
	int fd[2], nb, cnt;
	char cmd[1024], c;

	assert(pipe(fd) == 0);

	nb = 1; assert(ioctl(fd[1], FIONBIO, &nb) == 0);
	while (write(fd[1], &c, 1) == 1);
	assert(errno = -EAGAIN);
	nb = 0; assert(ioctl(fd[1], FIONBIO, &nb) == 0);

	// The pipe is full, the next write() will block.

	sprintf(cmd, "grep -e State -e ctxt_switches /proc/%d/status", getpid());

	if (!fork()) {
		// wait until the parent sleeps in pipe_write()
		usleep(10000);

		system(cmd);
		// trigger 4095 unnecessary wakeups
		for (cnt = 0; cnt < 4095; ++cnt) {
			assert(read(fd[0], &c, 1) == 1);
			usleep(1000);
		}
		system(cmd);

		// this should actually wake the writer
		printf("finally release the buffer\n");
		assert(read(fd[0], &c, 1) == 1);
		return 0;
	}

	assert(write(fd[1], &c, 1) == 1);
	printf("wrote next char!\n");

	return 0;
}
#15 lore raw
Quoted 3 lines · Oleg Nesterov

OK, I gave up ;) I'll send the revert patch tomorrow (can't do this today) even if I still don't see how this patch can be wrong.

Let's think about this a bit before reverting.

Because I think I see at least one possible issue..

With that commit aaec5a95d596 ("pipe_read: don't wake up the writer if the pipe is still full"), the rule for waking writers is pretty
simple: we only wake a writer if we update the tail pointer (so that we made a new slot available) _and_ the pipe was full before we did that.

And we do so while holding the pipe mutex, so we're guaranteed to be serialized with writers that are testing whether they can write (using the same pipe_full() logic).

Finally - we delay the actual wakeup until we actually sleep or are done with the read(), and we don't hold the mutex at that point any more, but we have updated the tail pointer and released the mutex, so the writer is guaranteed to have either seen the updates, or will see our wakeup.

All pretty simple and seems fool-proof, and the reader side logic would seem solid.

But I think I see a potential problem.

Because there's an additional subtlety: the pipe wakeup code not only wakes writers up only if it has freed an entry, it also does an EXCLUSIVE wakeup.

Which means that the reader will only wake up *one* writer on the wait queue.

And the *WRITER* side then will wake up any others when it has written, but *that* logic is

(a) wake up the next writer only if we were on the wait-queue (and
could thus have been the sole recipient of a wakeup)
(b) wake up the next writer only if the pipe isn't full

which also seems entirely sane. We must wake the next writer if we've "used up" the wakeup, but only when it makes sense.

However, I see at least one case where this exclusive wakeup seems broken:

/*
* But because we didn't read anything, at this point we can
* just return directly with -ERESTARTSYS if we're interrupted,
* since we've done any required wakeups and there's no need
* to mark anything accessed. And we've dropped the lock. */
if (wait_event_interruptible_exclusive(pipe->rd_wait,
pipe_readable(pipe)) < 0)

                        return -ERESTARTSYS;

and I'm wondering if the issue is that the *readers* got stuck, Because that "return -ERESTARTSYS" path now basically will by-pass the logic to wake up the next exclusive waiter.

Because that "return -ERESTARTSYS" is *after* the reader has been on the rd_wait queue - and possibly gotten the only wakeup that any of the readers will ever get - and now it returns without waking up any other reader.

So then the pipe stays full, because no readers are reading, even though there's potentially tons of them.

And maybe the "we had tons of extra write wakeups" meant that this was a pre-existing bug, but it was basically hidden by all the extra writers being woken up, and in turn waking up the readers that got missed.

I dunno. This feels wrong. And looking at the hackbench code, I don't see how it could actually be a problem on *that* load, because I don't see any signals that could cause that ERESTARTSYS case to happen, and if it did, the actual system call restart should get it all going again.

So I think that early return is actually buggy, and I think that comment is wrong (because "we didn't read anything" doesn't mean that we might not need to finish up), but I don't see how this could really cause the reported problems.

But maybe somebody sees some other subtle issue here.

The writer side does *not* have that early return case. It also does that wait_event_interruptible_exclusive() thing, but it will always end up doing the "wake_next_writer" logic if it got to that point.

The bug would have made more sense on the writer side.

But I basically do wonder if there's some bad interaction with the whole "exclusive wait" logic and the "we now only wake up one single time". The fact that I found *one* thing that smells bad to me makes me think maybe there's another that I didn't see.

Linus

#17 lore raw
In reply to Oleg Nesterov
Quoted 6 lines · Oleg Nesterov

Just in case, did you use

https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/hackbench/hackbench.c

?

Or did you use another version?

Exactly what parameters did you use?

If possible, please reproduce the hang again. How many threads/processes sleeping in pipe_read() or pipe_write() do you see? (you can look at /proc/$pid/stack).

Please pick one sleeping writer, and do

	$ strace -p pidof_that_write

this should wake this writer up. If a missed wakeup is the only problem, hackbench should continue.

The more info you can provide the better ;)

Oleg.

#16 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds

However, I see at least one case where this exclusive wakeup seems broken:

/*
* But because we didn't read anything, at this point we can
* just return directly with -ERESTARTSYS if we're interrupted,
* since we've done any required wakeups and there's no need
* to mark anything accessed. And we've dropped the lock. */
if (wait_event_interruptible_exclusive(pipe->rd_wait,
pipe_readable(pipe)) < 0)

Quoted 5 lines
                        return -ERESTARTSYS;

and I'm wondering if the issue is that the *readers* got stuck, Because that "return -ERESTARTSYS" path now basically will by-pass the logic to wake up the next exclusive waiter.

I think this is fine... lets denote this reader as R.

Quoted 4 lines

Because that "return -ERESTARTSYS" is *after* the reader has been on the rd_wait queue - and possibly gotten the only wakeup that any of the readers will ever get - and now it returns without waking up any other reader.

I think this can't happen. ___wait_event() does

	init_wait_entry(&__wq_entry, exclusive ? WQ_FLAG_EXCLUSIVE : 0);	\
	for (;;) {								\
		long __int = prepare_to_wait_event(&wq_head, &__wq_entry, state);\
										\
		if (condition)							\
			break;							\
										\
		if (___wait_is_interruptible(state) && __int) {			\
			__ret = __int;						\
			goto __out;						\
		}								\
										\
		cmd;								\
	}									\

and in this case condition == pipe_readable(pipe), cmd == schedule().

Suppose that R got that only wakeup, and wake_up() races with some signal so that signal_pending(R) is true.

In this case prepare_to_wait_event() will return -ERESTARTSYS, but ___wait_event() won't return this error code, it will check pipe_readable() and return 0.

After that R will restart the main loop with wake_next_reader = true, and whatever it does it should do wake_up(pipe->rd_wait) before return.

Note also that prepare_to_wait_event() removes the waiter from the wait_queue_head->head list, so another wake_up() can't pick this task.

Can ___wait_event() miss the pipe_readable() event in this case? No, both wake_up() and prepare_to_wait_event() take the same wq_head->lock.

What if pipe_readable() is actually false? Say, a spurios wakeup or, say, pipe_write() does wake_up(rd_wait) when another reader has already made the pipe_readable() condition false? This case looks "obviously fine" too.

So I am still confused.

I will wait for reply from Sapkal, then I'll try to make a debugging patch.

Oleg.

#18 lore raw
In reply to Oleg Nesterov

Hi Oleg,

Quoted 10 lines · Oleg Nesterov
Quoted 6 lines · Oleg Nesterov

Just in case, did you use

https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/hackbench/hackbench.c

?

Or did you use another version?

I am running hackbench using lkp-tests which downloads hackbench source from same rt-tests with version 2.8.

https://github.com/intel/lkp-tests.git
https://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-2.8.tar.gz

Quoted 2 lines

Exactly what parameters did you use?

Exact command with parameters is

	/usr/bin/hackbench -g 16 -f 20 --threads --pipe -l 100000 -s 100
Quoted 4 lines

If possible, please reproduce the hang again. How many threads/processes sleeping in pipe_read() or pipe_write() do you see? (you can look at /proc/$pid/stack).

In the latest hang, I saw 37 threads sleeping out of which 20 were sleeping in pipe_read() and 17 in pipe_write().

Main hackbench thread (which spawns the readers and writers) has the following stack trace:

[<0>] futex_wait_queue+0x6e/0x90
[<0>] __futex_wait+0x143/0x1c0
[<0>] futex_wait+0x69/0x110
[<0>] do_futex+0x147/0x1d0
[<0>] __x64_sys_futex+0x7c/0x1e0
[<0>] x64_sys_call+0x207a/0x2140
[<0>] do_syscall_64+0x6f/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

The readers have the following pipe_read stack trace:

[<0>] pipe_read+0x338/0x460
[<0>] vfs_read+0x308/0x350
[<0>] ksys_read+0xcc/0xe0
[<0>] __x64_sys_read+0x1d/0x30
[<0>] x64_sys_call+0x1b89/0x2140
[<0>] do_syscall_64+0x6f/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

The writers have the following pipe_write stack trace:

[<0>] pipe_write+0x370/0x630
[<0>] vfs_write+0x378/0x420
[<0>] ksys_write+0xcc/0xe0
[<0>] __x64_sys_write+0x1d/0x30
[<0>] x64_sys_call+0x16b3/0x2140
[<0>] do_syscall_64+0x6f/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
Quoted 7 lines

Please pick one sleeping writer, and do

	$ strace -p pidof_that_write

this should wake this writer up. If a missed wakeup is the only problem, hackbench should continue.

I tried waking one of the writer and the benchmark progressed and completed successfully.

Quoted 4 lines

The more info you can provide the better ;)

Oleg.

--
Thanks and Regards,
Swapnil

#19 lore raw

Thanks Sapkal!

I'll try to think. Meanwhile,

Quoted 4 lines · Swapnil

Exact command with parameters is

	/usr/bin/hackbench -g 16 -f 20 --threads --pipe -l 100000 -s 100

Can you reproduce with "--process" rather than "--threads" ?

Oleg.

#25 lore raw
In reply to Oleg Nesterov
Quoted 16 lines · Oleg Nesterov
Quoted 8 lines · Swapnil

Whenever I compare the case where was_full would have been set but wake_writer was not set, I see the following pattern:

ret = 100 (Read was successful)
pipe_full() = 1
total_len = 0
buf->len != 0

Quoted 6 lines

total_len is computed using iov_iter_count() while the buf->len is the length of the buffer corresponding to tail(pipe->bufs[tail & mask].len). Looking at pipe_write(), there seems to be a case where the writer can make progress when (chars && !was_empty) which only looks at iov_iter_count(). Could it be the case that there is still room in the buffer but we are not waking up the writer?

Quoted 7 lines

I don't think so, but perhaps I am totally confused.

If the writer sleeps on pipe->wr_wait, it has already tried to write into the pipe->bufs[head - 1] buffer before the sleep.

Yes, the reader can read from that buffer, but this won't make it more "writable" for this particular writer, "PAGE_SIZE - buf->offset + buf->len" won't be changed.

While I think the now-removed wakeup was indeed hiding a bug, I also think the write thing pointed out above is a fair point (orthogonal though).

The initial call to pipe_write allows for appending to an existing page.

However, should the pipe be full, the loop which follows it insists on allocating a new one and waits for a slot, even if ultimately *there is* space now.

The hackbench invocation used here passes around 100 bytes.

Both readers and writers do rounds over pipes issuing 100 byte-sized ops.

Suppose the pipe does not have space to hold the extra 100 bytes. The writer goes to sleep and waits for the tail to move. A reader shows up, reads 100 bytes (now there is space!) but since the current buf was not depleted it does not mess with the tail.

The bench spawns tons of threads, ensuring there is a lot of competition for the cpu time. The reader might get just enough time to largely deplete the pipe to a point where there is only one buf in there with space in it. Should pipe_write() be invoked now it would succeed appending to a page. But if the writer was already asleep, it is going to insist on allocating a new page.

As for the bug, I don't see anything obvious myself.

However, I think there are 2 avenues which warrant checking.

Sapkal, if you have time, can you please boot up the kernel which is more likely to run into the problem and then run hackbench as follows:

1. with 1 fd instead of 20:

/usr/bin/hackbench -g 16 -f 1 --threads --pipe -l 100000 -s 100

2. with a size which divides 4096 evenly (e.g., 128):

/usr/bin/hackbench -g 1 -f 20 --threads --pipe -l 100000 -s 128

#26 lore raw
Quoted 46 lines · Mateusz Guzik
Quoted 16 lines · Oleg Nesterov
Quoted 8 lines · Swapnil

Whenever I compare the case where was_full would have been set but wake_writer was not set, I see the following pattern:

ret = 100 (Read was successful)
pipe_full() = 1
total_len = 0
buf->len != 0

Quoted 6 lines

total_len is computed using iov_iter_count() while the buf->len is the length of the buffer corresponding to tail(pipe->bufs[tail & mask].len). Looking at pipe_write(), there seems to be a case where the writer can make progress when (chars && !was_empty) which only looks at iov_iter_count(). Could it be the case that there is still room in the buffer but we are not waking up the writer?

Quoted 7 lines

I don't think so, but perhaps I am totally confused.

If the writer sleeps on pipe->wr_wait, it has already tried to write into the pipe->bufs[head - 1] buffer before the sleep.

Yes, the reader can read from that buffer, but this won't make it more "writable" for this particular writer, "PAGE_SIZE - buf->offset + buf->len" won't be changed.

While I think the now-removed wakeup was indeed hiding a bug, I also think the write thing pointed out above is a fair point (orthogonal though).

The initial call to pipe_write allows for appending to an existing page.

However, should the pipe be full, the loop which follows it insists on allocating a new one and waits for a slot, even if ultimately *there is* space now.

The hackbench invocation used here passes around 100 bytes.

Both readers and writers do rounds over pipes issuing 100 byte-sized ops.

Suppose the pipe does not have space to hold the extra 100 bytes. The writer goes to sleep and waits for the tail to move. A reader shows up, reads 100 bytes (now there is space!) but since the current buf was not depleted it does not mess with the tail.

Quoted 6 lines

The bench spawns tons of threads, ensuring there is a lot of competition for the cpu time. The reader might get just enough time to largely deplete the pipe to a point where there is only one buf in there with space in it. Should pipe_write() be invoked now it would succeed appending to a page. But if the writer was already asleep, it is going to insist on allocating a new page.

Now that I sent the e-mail, I realized the page would have unread data after some offset, so there is no room to *append* to it, unless one wants to memmove everythiing back.

Please ignore this bit :P

However, the suggestion below stands:

Quoted 15 lines

As for the bug, I don't see anything obvious myself.

However, I think there are 2 avenues which warrant checking.

Sapkal, if you have time, can you please boot up the kernel which is more likely to run into the problem and then run hackbench as follows:

1. with 1 fd instead of 20:

/usr/bin/hackbench -g 16 -f 1 --threads --pipe -l 100000 -s 100

2. with a size which divides 4096 evenly (e.g., 128):

/usr/bin/hackbench -g 1 -f 20 --threads --pipe -l 100000 -s 128

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#27 lore raw
Quoted 6 lines · Mateusz Guzik
Quoted 1 line · Mateusz Guzik

Now that I sent the e-mail, I realized the page would have unread data after some offset, so there is no room to *append* to it, unless one wants to memmove everythiing back.

Yes, but... even "memmove everything back" won't help if pipe->ring_size > 1 (PIPE_DEF_BUFFERS == 16 by default).

Quoted 1 line

However, the suggestion below stands:

Agreed, any additional info can help.

Oleg.

Quoted 20 lines
Quoted 14 lines

As for the bug, I don't see anything obvious myself.

However, I think there are 2 avenues which warrant checking.

Sapkal, if you have time, can you please boot up the kernel which is more likely to run into the problem and then run hackbench as follows:

1. with 1 fd instead of 20:

/usr/bin/hackbench -g 16 -f 1 --threads --pipe -l 100000 -s 100

2. with a size which divides 4096 evenly (e.g., 128):

/usr/bin/hackbench -g 1 -f 20 --threads --pipe -l 100000 -s 128

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#20 lore raw
In reply to Oleg Nesterov

Hi Oleg,

Quoted 4 lines · Oleg Nesterov

Thanks Sapkal!

I'll try to think. Meanwhile,

Quoted 8 lines
Quoted 4 lines · Swapnil

Exact command with parameters is

	/usr/bin/hackbench -g 16 -f 20 --threads --pipe -l 100000 -s 100

Can you reproduce with "--process" rather than "--threads" ?

I was able to reproduce the issue with processes also. Total 33 processes were sleeping out of which 20 were readers and 13 were writers.

The stack trace for main hackbench process is as follows:

[<0>] do_wait+0xb5/0x110
[<0>] kernel_wait4+0xb2/0x150
[<0>] __do_sys_wait4+0x89/0xa0
[<0>] __x64_sys_wait4+0x20/0x30
[<0>] x64_sys_call+0x1bf7/0x2140
[<0>] do_syscall_64+0x6f/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

I am trying to reproduce the issue with suggestions by Mateusz.

Quoted 2 lines

Oleg.

--
Thanks and Regards,
Swapnil

#21 lore raw
Quoted 5 lines · Swapnil
Quoted 3 lines

Can you reproduce with "--process" rather than "--threads" ?

I was able to reproduce the issue with processes also. Total 33 processes were sleeping out of which 20 were readers and 13 were writers.

Thanks a lot. I am wondering what makes your machine (or .config?) special ;) A lot of people and robots tested this patch with these options.

Quoted 3 lines

The stack trace for main hackbench process is as follows:

[<0>] do_wait+0xb5/0x110

Yes, this is clear, the main thread/process is not interesting.

Quoted 1 line

I am trying to reproduce the issue with suggestions by Mateusz.

Great.

So far I still have no clue. Most probably I will ask you to do more testing after that, perhaps with some debugging patches.

Thanks again for your help,

Oleg.

#106 lore raw
In reply to Sapkal, Swapnil

Hmm...

Suppose that pipe is full, a writer W tries to write a single byte and sleeps on pipe->wr_wait.

A reader reads PAGE_SIZE bytes, updates pipe->tail, and wakes W up.

But, before the woken W takes pipe->mutex, another writer comes and writes 1 byte. This updates ->head and makes pipe_full() true again.

Now, W could happily merge its "small" write into the last buffer, but it will sleep again, despite the fact the last buffer has room for 4095 bytes.

Sapkal, I don't think this can explain the hang, receiver()->read() should wake this writer later anyway. But could you please retest with the patch below?

Thanks,

Oleg.
---

fs/pipe.c +2 -1
index b0641f75b1ba..222881559c30 100644
@@ -455,6 +455,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 	 * page-aligns the rest of the writes for large writes
 	 * spanning multiple pages.
 	 */
+again:
 	head = pipe->head;
 	was_empty = pipe_empty(head, pipe->tail);
 	chars = total_len & (PAGE_SIZE-1);
@@ -559,8 +560,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 		kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
 		wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
 		mutex_lock(&pipe->mutex);
-		was_empty = pipe_empty(pipe->head, pipe->tail);
 		wake_next_writer = true;
+		goto again;
 	}
 out:
 	if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
#107 lore raw

Forgot to mention...

Even if this patch is likely "offtopic", it probably makes sense. However, it is "incomplete" in that there are other scenarious.

Again, the pipe is full. A writer W1 tries to write 4096 bytes and sleeps. A writer W2 tries to write 1 byte and sleeps too.

A reader reads 4096 bytes, updates pipe->tail and wakes W1.

Another writer comes, writes 1 byte and "steals" the buffer released by the reader.

W1 sleeps again, this is correct. But nobody will wake W2 which could succeed.

This (and the previous more simple scenario) means that

	if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
		wake_next_writer = false;

before return in anon_pipe_write() is not really right in this sense.

Oleg.

Quoted 46 lines · Oleg Nesterov

Hmm...

Suppose that pipe is full, a writer W tries to write a single byte and sleeps on pipe->wr_wait.

A reader reads PAGE_SIZE bytes, updates pipe->tail, and wakes W up.

But, before the woken W takes pipe->mutex, another writer comes and writes 1 byte. This updates ->head and makes pipe_full() true again.

Now, W could happily merge its "small" write into the last buffer, but it will sleep again, despite the fact the last buffer has room for 4095 bytes.

Sapkal, I don't think this can explain the hang, receiver()->read() should wake this writer later anyway. But could you please retest with the patch below?

Thanks,

Oleg.
---

fs/pipe.c +2 -1
index b0641f75b1ba..222881559c30 100644
@@ -455,6 +455,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 	 * page-aligns the rest of the writes for large writes
 	 * spanning multiple pages.
 	 */
+again:
 	head = pipe->head;
 	was_empty = pipe_empty(head, pipe->tail);
 	chars = total_len & (PAGE_SIZE-1);
@@ -559,8 +560,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 		kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
 		wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
 		mutex_lock(&pipe->mutex);
-		was_empty = pipe_empty(pipe->head, pipe->tail);
 		wake_next_writer = true;
+		goto again;
 	}
 out:
 	if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
#108 lore raw
In reply to Oleg Nesterov
Quoted 38 lines · Oleg Nesterov

Hmm...

Suppose that pipe is full, a writer W tries to write a single byte and sleeps on pipe->wr_wait.

A reader reads PAGE_SIZE bytes, updates pipe->tail, and wakes W up.

But, before the woken W takes pipe->mutex, another writer comes and writes 1 byte. This updates ->head and makes pipe_full() true again.

Now, W could happily merge its "small" write into the last buffer, but it will sleep again, despite the fact the last buffer has room for 4095 bytes.

Sapkal, I don't think this can explain the hang, receiver()->read() should wake this writer later anyway. But could you please retest with the patch below?

Thanks,

Oleg.
---

fs/pipe.c +1 -0
index b0641f75b1ba..222881559c30 100644
@@ -455,6 +455,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
         * page-aligns the rest of the writes for large writes
         * spanning multiple pages.
         */
+again:
        head = pipe->head;
        was_empty = pipe_empty(head, pipe->tail);
        chars = total_len & (PAGE_SIZE-1);

fs/pipe.c anon_pipe_write(struct kiocb *iocb, struct iov_iter *from) · line 567

Quoted 9 lines
fs/pipe.c +1 -1
@@ -559,8 +560,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
                kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
                wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
                mutex_lock(&pipe->mutex);
-               was_empty = pipe_empty(pipe->head, pipe->tail);
                wake_next_writer = true;
+               goto again;
        }
 out:
        if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))

I think this is buggy.

You get wakeups also when the last reader goes away. The for loop you are jumping out of makes sure to check for the condition, same for the first mutex acquire. With this goto you can get a successful write instead of getting SIGPIPE. iow this should goto few lines higher.

I am not sure about the return value. The for loop bumps ret with each write, but the section you are jumping to overwrites it. So if the thread wrote some data within the loop, went to sleep and woke up to a state where it can do a write in the section you are jumping to, it is going to return the wrong number of bytes.

Unless I'm misreading something.

However, I do think something may be going on with the "split" ops, which is why I suggested going from 100 bytes where the bug was encountered to 128 for testing purposes. If that cleared it, that would be nice for sure. :>

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#28 lore raw
In reply to Mateusz Guzik

Hi Mateusz,

Quoted 52 lines · Mateusz Guzik
Quoted 16 lines · Oleg Nesterov
Quoted 8 lines · Swapnil

Whenever I compare the case where was_full would have been set but wake_writer was not set, I see the following pattern:

ret = 100 (Read was successful)
pipe_full() = 1
total_len = 0
buf->len != 0

Quoted 6 lines

total_len is computed using iov_iter_count() while the buf->len is the length of the buffer corresponding to tail(pipe->bufs[tail & mask].len). Looking at pipe_write(), there seems to be a case where the writer can make progress when (chars && !was_empty) which only looks at iov_iter_count(). Could it be the case that there is still room in the buffer but we are not waking up the writer?

Quoted 7 lines

I don't think so, but perhaps I am totally confused.

If the writer sleeps on pipe->wr_wait, it has already tried to write into the pipe->bufs[head - 1] buffer before the sleep.

Yes, the reader can read from that buffer, but this won't make it more "writable" for this particular writer, "PAGE_SIZE - buf->offset + buf->len" won't be changed.

While I think the now-removed wakeup was indeed hiding a bug, I also think the write thing pointed out above is a fair point (orthogonal though).

The initial call to pipe_write allows for appending to an existing page.

However, should the pipe be full, the loop which follows it insists on allocating a new one and waits for a slot, even if ultimately *there is* space now.

The hackbench invocation used here passes around 100 bytes.

Both readers and writers do rounds over pipes issuing 100 byte-sized ops.

Suppose the pipe does not have space to hold the extra 100 bytes. The writer goes to sleep and waits for the tail to move. A reader shows up, reads 100 bytes (now there is space!) but since the current buf was not depleted it does not mess with the tail.

The bench spawns tons of threads, ensuring there is a lot of competition for the cpu time. The reader might get just enough time to largely deplete the pipe to a point where there is only one buf in there with space in it. Should pipe_write() be invoked now it would succeed appending to a page. But if the writer was already asleep, it is going to insist on allocating a new page.

Quoted 7 lines

As for the bug, I don't see anything obvious myself.

However, I think there are 2 avenues which warrant checking.

Sapkal, if you have time, can you please boot up the kernel which is more likely to run into the problem and then run hackbench as follows:

I tried reproducing the issue with both the scenarios mentioned below.

Quoted 4 lines

1. with 1 fd instead of 20:

/usr/bin/hackbench -g 16 -f 1 --threads --pipe -l 100000 -s 100

With this I was not able to reproduce the issue. I tried almost 5000 iterations.

Quoted 3 lines

2. with a size which divides 4096 evenly (e.g., 128):

/usr/bin/hackbench -g 1 -f 20 --threads --pipe -l 100000 -s 128

I was not able to reproduce the issue with 1 group. But I thought you wanted to change only the message size to 128 bytes. When I retain the number of groups to 16 and change the message size to 128, it took me around 150 iterations to reproduce this issue (with 100 bytes it was 20 iterations). The exact command was

/usr/bin/hackbench -g 16 -f 20 --threads --pipe -l 100000 -s 128

I will try to sprinkle some trace_printk's in the code where the state of the pipe changes. I will report here if I find something.

--
Thanks and Regards,
Swapnil

#109 lore raw
In reply to Mateusz Guzik
Quoted 38 lines · Mateusz Guzik
Quoted 24 lines · Oleg Nesterov

Sapkal, I don't think this can explain the hang, receiver()->read() should wake this writer later anyway. But could you please retest with the patch below?

Thanks,

Oleg.
---

fs/pipe.c +1 -0
index b0641f75b1ba..222881559c30 100644
@@ -455,6 +455,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
         * page-aligns the rest of the writes for large writes
         * spanning multiple pages.
         */
+again:
        head = pipe->head;
        was_empty = pipe_empty(head, pipe->tail);
        chars = total_len & (PAGE_SIZE-1);

fs/pipe.c anon_pipe_write(struct kiocb *iocb, struct iov_iter *from) · line 567

Quoted 9 lines
fs/pipe.c +1 -1
@@ -559,8 +560,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
                kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
                wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
                mutex_lock(&pipe->mutex);
-               was_empty = pipe_empty(pipe->head, pipe->tail);
                wake_next_writer = true;
+               goto again;
        }
 out:
        if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))

I think this is buggy.

Quoted 4 lines

You get wakeups also when the last reader goes away. The for loop you are jumping out of makes sure to check for the condition, same for the first mutex acquire. With this goto you can get a successful write instead of getting SIGPIPE. iow this should goto few lines higher.

Yes, yes, and then we need to remove another pipe->readers check in the main loop.

Quoted 2 lines

I am not sure about the return value. The for loop bumps ret with each write, but the section you are jumping to overwrites it.

Ah, yes, thanks, I missed that.

OK, I'll make another one tomorrow, I need to run away.

Until then, it would be nice to test this patch with hackbench anyway.

Quoted 4 lines

However, I do think something may be going on with the "split" ops, which is why I suggested going from 100 bytes where the bug was encountered to 128 for testing purposes. If that cleared it, that would be nice for sure. :>

Yes, but note that the same scenario can happen with 128 bytes as well. It doesn't really matter how many bytes < PAGE_SIZE the sleeping writer needs to write, another writer can steal the buffer released by the last reader in any case.

Thanks!

Oleg.

#29 lore raw
In reply to Sapkal, Swapnil
Quoted 10 lines · Swapnil

I tried reproducing the issue with both the scenarios mentioned below.

Quoted 4 lines

1. with 1 fd instead of 20:

/usr/bin/hackbench -g 16 -f 1 --threads --pipe -l 100000 -s 100

With this I was not able to reproduce the issue. I tried almost 5000 iterations.

Ok, noted.

Quoted 6 lines
Quoted 3 lines

2. with a size which divides 4096 evenly (e.g., 128):

/usr/bin/hackbench -g 1 -f 20 --threads --pipe -l 100000 -s 128

I was not able to reproduce the issue with 1 group. But I thought you wanted to change only the message size to 128 bytes.

Yes indeed, thanks for catching the problem.

Quoted 9 lines

When I retain the number of groups to 16 and change the message size to 128, it took me around 150 iterations to reproduce this issue (with 100 bytes it was 20 iterations). The exact command was

/usr/bin/hackbench -g 16 -f 20 --threads --pipe -l 100000 -s 128

I will try to sprinkle some trace_printk's in the code where the state of the pipe changes. I will report here if I find something.

Thanks.

So to be clear, this is Oleg's bug, I am only looking from the side out of curiosity what's up. As it usually goes with these, after the dust settles I very much expect the fix will be roughly a one liner. :)

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#30 lore raw
In reply to Sapkal, Swapnil

Sapkal, first of all, thanks again!

Quoted 7 lines · Swapnil
Quoted 4 lines

1. with 1 fd instead of 20:

/usr/bin/hackbench -g 16 -f 1 --threads --pipe -l 100000 -s 100

With this I was not able to reproduce the issue. I tried almost 5000 iterations.

OK,

Quoted 1 line
Quoted 1 line

2. with a size which divides 4096 evenly (e.g., 128):

...

Quoted 6 lines

When I retain the number of
groups to 16 and change the message size to 128, it took me around 150 iterations to reproduce this issue (with 100 bytes it was 20 iterations). The exact command was

/usr/bin/hackbench -g 16 -f 20 --threads --pipe -l 100000 -s 128

Ah, good. This is good ;)

Quoted 2 lines

I will try to sprinkle some trace_printk's in the code where the state of the pipe changes. I will report here if I find something.

Great! but...

Sapkal, I was going to finish (and test! ;) the patch below tomorrow, after you test the previous debugging patch I sent in this thread. But since you are going to change the kernel...

For the moment, please forget about that (as Mateusz pointed buggy) patch. Could you apply the patch below and reproduce the problem ?

If yes, please do prctl(666) after the hang and send us the output from dmesg, between "DUMP START" and "DUMP END". You can just do

	$ perl -e 'syscall 157,666'

to call prctl(666) and trigger the dump.

Oleg.
---

fs/pipe.c +54 -1
index b0641f75b1ba..566c75a0ff81 100644
@@ -376,6 +376,8 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	}
 	if (pipe_empty(pipe->head, pipe->tail))
 		wake_next_reader = false;
+	if (ret > 0)
+		pipe->r_cnt++;
 	mutex_unlock(&pipe->mutex);
 
 	if (wake_writer)
@@ -565,6 +567,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 out:
 	if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
 		wake_next_writer = false;
+	if (ret > 0)
+		pipe->w_cnt++;
 	mutex_unlock(&pipe->mutex);
 
 	/*
@@ -695,6 +699,42 @@ pipe_poll(struct file *filp, poll_table *wait)
 	return mask;
 }
 
+static DEFINE_MUTEX(PI_MUTEX);
+static LIST_HEAD(PI_LIST);
+
+void pi_dump(void);
+void pi_dump(void)
+{
+	struct pipe_inode_info *pipe;
+
+	pr_crit("---------- DUMP START ----------\n");
+	mutex_lock(&PI_MUTEX);
+	list_for_each_entry(pipe, &PI_LIST, pi_list) {
+		unsigned head, tail;
+
+		mutex_lock(&pipe->mutex);
+		head = pipe->head;
+		tail = pipe->tail;
+		pr_crit("E=%d F=%d; W=%d R=%d\n",
+			pipe_empty(head, tail), pipe_full(head, tail, pipe->max_usage),
+			pipe->w_cnt, pipe->r_cnt);
+
+// INCOMPLETE
+pr_crit("RD=%d WR=%d\n", waitqueue_active(&pipe->rd_wait), waitqueue_active(&pipe->wr_wait));
+
+		for (; tail < head; tail++) {
+			struct pipe_buffer *buf = pipe_buf(pipe, tail);
+			WARN_ON(buf->ops != &anon_pipe_buf_ops);
+			pr_crit("buf: o=%d l=%d\n", buf->offset, buf->len);
+		}
+		pr_crit("\n");
+
+		mutex_unlock(&pipe->mutex);
+	}
+	mutex_unlock(&PI_MUTEX);
+	pr_crit("---------- DUMP END ------------\n");
+}
+
 static void put_pipe_info(struct inode *inode, struct pipe_inode_info *pipe)
 {
 	int kill = 0;
@@ -706,8 +746,14 @@ static void put_pipe_info(struct inode *inode, struct pipe_inode_info *pipe)
 	}
 	spin_unlock(&inode->i_lock);
 
-	if (kill)
+	if (kill) {
+		if (!list_empty(&pipe->pi_list)) {
+			mutex_lock(&PI_MUTEX);
+			list_del_init(&pipe->pi_list);
+			mutex_unlock(&PI_MUTEX);
+		}
 		free_pipe_info(pipe);
+	}
 }
 
 static int
@@ -790,6 +836,13 @@ struct pipe_inode_info *alloc_pipe_info(void)
 	if (pipe == NULL)
 		goto out_free_uid;
 
+	INIT_LIST_HEAD(&pipe->pi_list);
+	if (!strcmp(current->comm, "hackbench")) {
+		mutex_lock(&PI_MUTEX);
+		list_add_tail(&pipe->pi_list, &PI_LIST);
+		mutex_unlock(&PI_MUTEX);
+	}
+
 	if (pipe_bufs * PAGE_SIZE > max_size && !capable(CAP_SYS_RESOURCE))
 		pipe_bufs = max_size >> PAGE_SHIFT;
 
include/linux/pipe_fs_i.h +3 -0
index 8ff23bf5a819..48d9bf5171dc 100644
@@ -80,6 +80,9 @@ struct pipe_inode_info {
 #ifdef CONFIG_WATCH_QUEUE
 	struct watch_queue *watch_queue;
 #endif
+
+	struct list_head pi_list;
+	unsigned w_cnt, r_cnt;
 };
 
 /*
kernel/sys.c +5 -0
index 4efca8a97d62..a85e34861b2e 100644
@@ -2483,6 +2483,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 
 	error = 0;
 	switch (option) {
+	case 666: {
+		extern void pi_dump(void);
+		pi_dump();
+		break;
+	}
 	case PR_SET_PDEATHSIG:
 		if (!valid_signal(arg2)) {
 			error = -EINVAL;
#31 lore raw

Hi Oleg,

Quoted 27 lines · Oleg Nesterov

Sapkal, first of all, thanks again!

Quoted 7 lines · Swapnil
Quoted 4 lines

1. with 1 fd instead of 20:

/usr/bin/hackbench -g 16 -f 1 --threads --pipe -l 100000 -s 100

With this I was not able to reproduce the issue. I tried almost 5000 iterations.

OK,

Quoted 1 line
Quoted 1 line

2. with a size which divides 4096 evenly (e.g., 128):

...

Quoted 6 lines

When I retain the number of
groups to 16 and change the message size to 128, it took me around 150 iterations to reproduce this issue (with 100 bytes it was 20 iterations). The exact command was

/usr/bin/hackbench -g 16 -f 20 --threads --pipe -l 100000 -s 128

Ah, good. This is good ;)

Quoted 2 lines

I will try to sprinkle some trace_printk's in the code where the state of the pipe changes. I will report here if I find something.

Quoted 9 lines

Great! but...

Sapkal, I was going to finish (and test! ;) the patch below tomorrow, after you test the previous debugging patch I sent in this thread. But since you are going to change the kernel...

For the moment, please forget about that (as Mateusz pointed buggy) patch. Could you apply the patch below and reproduce the problem ?

Yes, I was able to reproduce the problem with the below patch.

Quoted 7 lines

If yes, please do prctl(666) after the hang and send us the output from dmesg, between "DUMP START" and "DUMP END". You can just do

	$ perl -e 'syscall 157,666'

to call prctl(666) and trigger the dump.

I found a case in the dump where the pipe is empty still both reader and writer are waiting on it.

[ 1397.829761] E=1 F=0; W=1719147 R=1719147
[ 1397.837843] RD=1 WR=1

Full dump is attached below.

Quoted 119 lines

Oleg.
---

fs/pipe.c +54 -1
index b0641f75b1ba..566c75a0ff81 100644
@@ -376,6 +376,8 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
  	}
  	if (pipe_empty(pipe->head, pipe->tail))
  		wake_next_reader = false;
+	if (ret > 0)
+		pipe->r_cnt++;
  	mutex_unlock(&pipe->mutex);
 
  	if (wake_writer)
@@ -565,6 +567,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
  out:
  	if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
  		wake_next_writer = false;
+	if (ret > 0)
+		pipe->w_cnt++;
  	mutex_unlock(&pipe->mutex);
 
  	/*
@@ -695,6 +699,42 @@ pipe_poll(struct file *filp, poll_table *wait)
  	return mask;
  }
 
+static DEFINE_MUTEX(PI_MUTEX);
+static LIST_HEAD(PI_LIST);
+
+void pi_dump(void);
+void pi_dump(void)
+{
+	struct pipe_inode_info *pipe;
+
+	pr_crit("---------- DUMP START ----------\n");
+	mutex_lock(&PI_MUTEX);
+	list_for_each_entry(pipe, &PI_LIST, pi_list) {
+		unsigned head, tail;
+
+		mutex_lock(&pipe->mutex);
+		head = pipe->head;
+		tail = pipe->tail;
+		pr_crit("E=%d F=%d; W=%d R=%d\n",
+			pipe_empty(head, tail), pipe_full(head, tail, pipe->max_usage),
+			pipe->w_cnt, pipe->r_cnt);
+
+// INCOMPLETE
+pr_crit("RD=%d WR=%d\n", waitqueue_active(&pipe->rd_wait), waitqueue_active(&pipe->wr_wait));
+
+		for (; tail < head; tail++) {
+			struct pipe_buffer *buf = pipe_buf(pipe, tail);
+			WARN_ON(buf->ops != &anon_pipe_buf_ops);
+			pr_crit("buf: o=%d l=%d\n", buf->offset, buf->len);
+		}
+		pr_crit("\n");
+
+		mutex_unlock(&pipe->mutex);
+	}
+	mutex_unlock(&PI_MUTEX);
+	pr_crit("---------- DUMP END ------------\n");
+}
+
  static void put_pipe_info(struct inode *inode, struct pipe_inode_info *pipe)
  {
  	int kill = 0;
@@ -706,8 +746,14 @@ static void put_pipe_info(struct inode *inode, struct pipe_inode_info *pipe)
  	}
  	spin_unlock(&inode->i_lock);
 
-	if (kill)
+	if (kill) {
+		if (!list_empty(&pipe->pi_list)) {
+			mutex_lock(&PI_MUTEX);
+			list_del_init(&pipe->pi_list);
+			mutex_unlock(&PI_MUTEX);
+		}
  		free_pipe_info(pipe);
+	}
  }
 
  static int
@@ -790,6 +836,13 @@ struct pipe_inode_info *alloc_pipe_info(void)
  	if (pipe == NULL)
  		goto out_free_uid;
 
+	INIT_LIST_HEAD(&pipe->pi_list);
+	if (!strcmp(current->comm, "hackbench")) {
+		mutex_lock(&PI_MUTEX);
+		list_add_tail(&pipe->pi_list, &PI_LIST);
+		mutex_unlock(&PI_MUTEX);
+	}
+
  	if (pipe_bufs * PAGE_SIZE > max_size && !capable(CAP_SYS_RESOURCE))
  		pipe_bufs = max_size >> PAGE_SHIFT;
 
include/linux/pipe_fs_i.h +3 -0
index 8ff23bf5a819..48d9bf5171dc 100644
@@ -80,6 +80,9 @@ struct pipe_inode_info {
  #ifdef CONFIG_WATCH_QUEUE
  	struct watch_queue *watch_queue;
  #endif
+
+	struct list_head pi_list;
+	unsigned w_cnt, r_cnt;
  };
 
  /*
kernel/sys.c +0 -0
index 4efca8a97d62..a85e34861b2e 100644
@@ -2483,6 +2483,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 

kernel/sys.c SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, · line 2493

Quoted 10 lines
kernel/sys.c +5 -0
  	error = 0;
  	switch (option) {
+	case 666: {
+		extern void pi_dump(void);
+		pi_dump();
+		break;
+	}
  	case PR_SET_PDEATHSIG:
  		if (!valid_signal(arg2)) {
  			error = -EINVAL;

--
Thanks and Regards,
Swapnil

[ 1394.383241] ---------- DUMP START ----------
[ 1394.388211] E=1 F=0; W=640 R=640
[ 1394.392001] RD=0 WR=0
[ 1394.396300] E=0 F=0; W=1 R=0
[ 1394.399625] RD=0 WR=0
[ 1394.402219] buf: o=0 l=1
[ 1394.406824] E=1 F=0; W=2000000 R=2000000
[ 1394.411322] RD=0 WR=0
[ 1394.415621] E=1 F=0; W=2000000 R=2000000
[ 1394.420108] RD=0 WR=0
[ 1394.424404] E=1 F=0; W=2000000 R=2000000
[ 1394.428898] RD=0 WR=0
[ 1394.433197] E=1 F=0; W=2000000 R=2000000
[ 1394.437692] RD=0 WR=0
[ 1394.441992] E=1 F=0; W=2000000 R=2000000
[ 1394.446490] RD=0 WR=0
[ 1394.450787] E=1 F=0; W=2000000 R=2000000
[ 1394.455282] RD=0 WR=0
[ 1394.459578] E=1 F=0; W=2000000 R=2000000
[ 1394.464073] RD=0 WR=0
[ 1394.468369] E=1 F=0; W=2000000 R=2000000
[ 1394.472866] RD=0 WR=0
[ 1394.477171] E=1 F=0; W=2000000 R=2000000
[ 1394.481666] RD=0 WR=0
[ 1394.485962] E=1 F=0; W=2000000 R=2000000
[ 1394.490460] RD=0 WR=0
[ 1394.494756] E=1 F=0; W=2000000 R=2000000
[ 1394.499252] RD=0 WR=0
[ 1394.503550] E=1 F=0; W=2000000 R=2000000
[ 1394.508044] RD=0 WR=0
[ 1394.512341] E=1 F=0; W=2000000 R=2000000
[ 1394.520567] RD=0 WR=0
[ 1394.532241] E=1 F=0; W=2000000 R=2000000
[ 1394.540298] RD=0 WR=0
[ 1394.551509] E=1 F=0; W=2000000 R=2000000
[ 1394.559501] RD=0 WR=0
[ 1394.570844] E=1 F=0; W=2000000 R=2000000
[ 1394.578883] RD=0 WR=0
[ 1394.590320] E=1 F=0; W=2000000 R=2000000
[ 1394.598359] RD=0 WR=0
[ 1394.609810] E=1 F=0; W=2000000 R=2000000
[ 1394.617850] RD=0 WR=0
[ 1394.629299] E=1 F=0; W=2000000 R=2000000
[ 1394.637334] RD=0 WR=0
[ 1394.648784] E=1 F=0; W=2000000 R=2000000
[ 1394.656837] RD=0 WR=0
[ 1394.668310] E=1 F=0; W=2000000 R=2000000
[ 1394.676365] RD=0 WR=0
[ 1394.687846] E=1 F=0; W=2000000 R=2000000
[ 1394.695906] RD=0 WR=0
[ 1394.707397] E=1 F=0; W=2000000 R=2000000
[ 1394.715461] RD=0 WR=0
[ 1394.726965] E=1 F=0; W=2000000 R=2000000
[ 1394.735036] RD=0 WR=0
[ 1394.746535] E=1 F=0; W=2000000 R=2000000
[ 1394.754609] RD=0 WR=0
[ 1394.766125] E=1 F=0; W=2000000 R=2000000
[ 1394.774200] RD=0 WR=0
[ 1394.785729] E=1 F=0; W=2000000 R=2000000
[ 1394.793813] RD=0 WR=0
[ 1394.805347] E=1 F=0; W=2000000 R=2000000
[ 1394.813433] RD=0 WR=0
[ 1394.824981] E=1 F=0; W=2000000 R=2000000
[ 1394.833074] RD=0 WR=0
[ 1394.844628] E=1 F=0; W=2000000 R=2000000
[ 1394.852720] RD=0 WR=0
[ 1394.864289] E=1 F=0; W=2000000 R=2000000
[ 1394.872388] RD=0 WR=0
[ 1394.883938] E=1 F=0; W=2000000 R=2000000
[ 1394.892030] RD=0 WR=0
[ 1394.903579] E=1 F=0; W=2000000 R=2000000
[ 1394.911673] RD=0 WR=0
[ 1394.923222] E=1 F=0; W=2000000 R=2000000
[ 1394.931321] RD=0 WR=0
[ 1394.942876] E=1 F=0; W=2000000 R=2000000
[ 1394.950972] RD=0 WR=0
[ 1394.962527] E=1 F=0; W=2000000 R=2000000
[ 1394.970623] RD=0 WR=0
[ 1394.982182] E=1 F=0; W=2000000 R=2000000
[ 1394.990279] RD=0 WR=0
[ 1395.001833] E=1 F=0; W=2000000 R=2000000
[ 1395.009942] RD=0 WR=0
[ 1395.021516] E=1 F=0; W=2000000 R=2000000
[ 1395.029623] RD=0 WR=0
[ 1395.041200] E=1 F=0; W=2000000 R=2000000
[ 1395.049310] RD=0 WR=0
[ 1395.060882] E=1 F=0; W=2000000 R=2000000
[ 1395.068986] RD=0 WR=0
[ 1395.080560] E=1 F=0; W=2000000 R=2000000
[ 1395.088667] RD=0 WR=0
[ 1395.100238] E=1 F=0; W=2000000 R=2000000
[ 1395.108348] RD=0 WR=0
[ 1395.119903] E=1 F=0; W=2000000 R=2000000
[ 1395.128002] RD=0 WR=0
[ 1395.139557] E=1 F=0; W=2000000 R=2000000
[ 1395.147646] RD=0 WR=0
[ 1395.159199] E=1 F=0; W=2000000 R=2000000
[ 1395.167295] RD=0 WR=0
[ 1395.178848] E=1 F=0; W=2000000 R=2000000
[ 1395.186942] RD=0 WR=0
[ 1395.198491] E=1 F=0; W=2000000 R=2000000
[ 1395.206587] RD=0 WR=0
[ 1395.218143] E=1 F=0; W=2000000 R=2000000
[ 1395.226241] RD=0 WR=0
[ 1395.237794] E=1 F=0; W=2000000 R=2000000
[ 1395.245893] RD=0 WR=0
[ 1395.257452] E=1 F=0; W=2000000 R=2000000
[ 1395.265548] RD=0 WR=0
[ 1395.277111] E=1 F=0; W=2000000 R=2000000
[ 1395.285212] RD=0 WR=0
[ 1395.296779] E=1 F=0; W=2000000 R=2000000
[ 1395.304873] RD=0 WR=0
[ 1395.316446] E=1 F=0; W=2000000 R=2000000
[ 1395.324541] RD=0 WR=0
[ 1395.336087] E=1 F=0; W=2000000 R=2000000
[ 1395.344163] RD=0 WR=0
[ 1395.355701] E=1 F=0; W=2000000 R=2000000
[ 1395.363787] RD=0 WR=0
[ 1395.375325] E=1 F=0; W=2000000 R=2000000
[ 1395.383409] RD=0 WR=0
[ 1395.394937] E=1 F=0; W=2000000 R=2000000
[ 1395.403015] RD=0 WR=0
[ 1395.414546] E=1 F=0; W=2000000 R=2000000
[ 1395.422630] RD=0 WR=0
[ 1395.434162] E=1 F=0; W=2000000 R=2000000
[ 1395.442248] RD=0 WR=0
[ 1395.453797] E=1 F=0; W=2000000 R=2000000
[ 1395.461886] RD=0 WR=0
[ 1395.473432] E=1 F=0; W=2000000 R=2000000
[ 1395.481526] RD=0 WR=0
[ 1395.493080] E=1 F=0; W=2000000 R=2000000
[ 1395.501175] RD=0 WR=0
[ 1395.512740] E=1 F=0; W=2000000 R=2000000
[ 1395.520835] RD=0 WR=0
[ 1395.532396] E=1 F=0; W=2000000 R=2000000
[ 1395.540497] RD=0 WR=0
[ 1395.552049] E=1 F=0; W=2000000 R=2000000
[ 1395.560149] RD=0 WR=0
[ 1395.571700] E=1 F=0; W=2000000 R=2000000
[ 1395.579792] RD=0 WR=0
[ 1395.591330] E=1 F=0; W=2000000 R=2000000
[ 1395.599416] RD=0 WR=0
[ 1395.610953] E=1 F=0; W=2000000 R=2000000
[ 1395.619041] RD=0 WR=0
[ 1395.630576] E=1 F=0; W=2000000 R=2000000
[ 1395.638666] RD=0 WR=0
[ 1395.650211] E=1 F=0; W=2000000 R=2000000
[ 1395.658295] RD=0 WR=0
[ 1395.669832] E=1 F=0; W=2000000 R=2000000
[ 1395.677915] RD=0 WR=0
[ 1395.689457] E=1 F=0; W=2000000 R=2000000
[ 1395.697543] RD=0 WR=0
[ 1395.709091] E=1 F=0; W=2000000 R=2000000
[ 1395.717172] RD=0 WR=0
[ 1395.728749] E=1 F=0; W=2000000 R=2000000
[ 1395.736846] RD=0 WR=0
[ 1395.748398] E=1 F=0; W=2000000 R=2000000
[ 1395.756488] RD=0 WR=0
[ 1395.768035] E=1 F=0; W=2000000 R=2000000
[ 1395.776117] RD=0 WR=0
[ 1395.787658] E=1 F=0; W=2000000 R=2000000
[ 1395.795746] RD=0 WR=0
[ 1395.807289] E=1 F=0; W=2000000 R=2000000
[ 1395.815363] RD=0 WR=0
[ 1395.826900] E=1 F=0; W=2000000 R=2000000
[ 1395.834983] RD=0 WR=0
[ 1395.846510] E=1 F=0; W=2000000 R=2000000
[ 1395.854584] RD=0 WR=0
[ 1395.866107] E=1 F=0; W=2000000 R=2000000
[ 1395.874203] RD=0 WR=0
[ 1395.885740] E=1 F=0; W=2000000 R=2000000
[ 1395.893826] RD=0 WR=0
[ 1395.905373] E=1 F=0; W=2000000 R=2000000
[ 1395.913469] RD=0 WR=0
[ 1395.925031] E=1 F=0; W=2000000 R=2000000
[ 1395.933115] RD=0 WR=0
[ 1395.944654] E=1 F=0; W=2000000 R=2000000
[ 1395.952751] RD=0 WR=0
[ 1395.964305] E=1 F=0; W=2000000 R=2000000
[ 1395.972399] RD=0 WR=0
[ 1395.983967] E=1 F=0; W=2000000 R=2000000
[ 1395.992063] RD=0 WR=0
[ 1396.003614] E=1 F=0; W=2000000 R=2000000
[ 1396.011715] RD=0 WR=0
[ 1396.023270] E=1 F=0; W=2000000 R=2000000
[ 1396.031359] RD=0 WR=0
[ 1396.042909] E=1 F=0; W=2000000 R=2000000
[ 1396.051001] RD=0 WR=0
[ 1396.062547] E=1 F=0; W=2000000 R=2000000
[ 1396.070642] RD=0 WR=0
[ 1396.082190] E=1 F=0; W=2000000 R=2000000
[ 1396.090283] RD=0 WR=0
[ 1396.101841] E=1 F=0; W=2000000 R=2000000
[ 1396.109930] RD=0 WR=0
[ 1396.121489] E=1 F=0; W=2000000 R=2000000
[ 1396.129588] RD=0 WR=0
[ 1396.141143] E=1 F=0; W=2000000 R=2000000
[ 1396.149239] RD=0 WR=0
[ 1396.160801] E=1 F=0; W=2000000 R=2000000
[ 1396.168902] RD=0 WR=0
[ 1396.180467] E=1 F=0; W=2000000 R=2000000
[ 1396.188569] RD=0 WR=0
[ 1396.200135] E=1 F=0; W=2000000 R=2000000
[ 1396.208238] RD=0 WR=0
[ 1396.219802] E=1 F=0; W=2000000 R=2000000
[ 1396.227898] RD=0 WR=0
[ 1396.239459] E=1 F=0; W=2000000 R=2000000
[ 1396.247559] RD=0 WR=0
[ 1396.259110] E=1 F=0; W=2000000 R=2000000
[ 1396.267204] RD=0 WR=0
[ 1396.278760] E=1 F=0; W=2000000 R=2000000
[ 1396.286859] RD=0 WR=0
[ 1396.298417] E=1 F=0; W=2000000 R=2000000
[ 1396.306514] RD=0 WR=0
[ 1396.318052] E=1 F=0; W=2000000 R=2000000
[ 1396.326137] RD=0 WR=0
[ 1396.337686] E=1 F=0; W=2000000 R=2000000
[ 1396.345771] RD=0 WR=0
[ 1396.357315] E=1 F=0; W=2000000 R=2000000
[ 1396.365403] RD=0 WR=0
[ 1396.376947] E=1 F=0; W=2000000 R=2000000
[ 1396.385032] RD=0 WR=0
[ 1396.396589] E=1 F=0; W=2000000 R=2000000
[ 1396.404677] RD=0 WR=0
[ 1396.416224] E=1 F=0; W=2000000 R=2000000
[ 1396.424312] RD=0 WR=0
[ 1396.435877] E=1 F=0; W=2000000 R=2000000
[ 1396.443971] RD=0 WR=0
[ 1396.455514] E=1 F=0; W=2000000 R=2000000
[ 1396.463602] RD=0 WR=0
[ 1396.475133] E=1 F=0; W=2000000 R=2000000
[ 1396.483219] RD=0 WR=0
[ 1396.494754] E=1 F=0; W=2000000 R=2000000
[ 1396.502841] RD=0 WR=0
[ 1396.514380] E=1 F=0; W=2000000 R=2000000
[ 1396.522466] RD=0 WR=0
[ 1396.533997] E=1 F=0; W=2000000 R=2000000
[ 1396.542082] RD=0 WR=0
[ 1396.553623] E=1 F=0; W=2000000 R=2000000
[ 1396.561716] RD=0 WR=0
[ 1396.573262] E=1 F=0; W=2000000 R=2000000
[ 1396.581351] RD=0 WR=0
[ 1396.592900] E=1 F=0; W=2000000 R=2000000
[ 1396.600991] RD=0 WR=0
[ 1396.612548] E=1 F=0; W=2000000 R=2000000
[ 1396.620643] RD=0 WR=0
[ 1396.632190] E=1 F=0; W=2000000 R=2000000
[ 1396.640275] RD=0 WR=0
[ 1396.651826] E=1 F=0; W=2000000 R=2000000
[ 1396.659919] RD=0 WR=0
[ 1396.671472] E=1 F=0; W=2000000 R=2000000
[ 1396.679572] RD=0 WR=0
[ 1396.691123] E=1 F=0; W=2000000 R=2000000
[ 1396.699220] RD=0 WR=0
[ 1396.710769] E=1 F=0; W=2000000 R=2000000
[ 1396.718862] RD=0 WR=0
[ 1396.730400] E=1 F=0; W=2000000 R=2000000
[ 1396.738487] RD=0 WR=0
[ 1396.750011] E=1 F=0; W=2000000 R=2000000
[ 1396.758092] RD=0 WR=0
[ 1396.769612] E=1 F=0; W=2000000 R=2000000
[ 1396.777695] RD=0 WR=0
[ 1396.789224] E=1 F=0; W=2000000 R=2000000
[ 1396.797305] RD=0 WR=0
[ 1396.808835] E=1 F=0; W=2000000 R=2000000
[ 1396.816914] RD=0 WR=0
[ 1396.828440] E=1 F=0; W=2000000 R=2000000
[ 1396.836522] RD=0 WR=0
[ 1396.848060] E=1 F=0; W=2000000 R=2000000
[ 1396.856140] RD=0 WR=0
[ 1396.867661] E=1 F=0; W=2000000 R=2000000
[ 1396.875742] RD=0 WR=0
[ 1396.887286] E=1 F=0; W=2000000 R=2000000
[ 1396.895367] RD=0 WR=0
[ 1396.906898] E=1 F=0; W=2000000 R=2000000
[ 1396.914978] RD=0 WR=0
[ 1396.926502] E=1 F=0; W=2000000 R=2000000
[ 1396.934587] RD=0 WR=0
[ 1396.946117] E=1 F=0; W=2000000 R=2000000
[ 1396.954214] RD=0 WR=0
[ 1396.965749] E=1 F=0; W=2000000 R=2000000
[ 1396.973834] RD=0 WR=0
[ 1396.985370] E=1 F=0; W=2000000 R=2000000
[ 1396.993458] RD=0 WR=0
[ 1397.005001] E=1 F=0; W=2000000 R=2000000
[ 1397.013098] RD=0 WR=0
[ 1397.024636] E=1 F=0; W=2000000 R=2000000
[ 1397.032727] RD=0 WR=0
[ 1397.044276] E=1 F=0; W=2000000 R=2000000
[ 1397.052374] RD=0 WR=0
[ 1397.063936] E=1 F=0; W=2000000 R=2000000
[ 1397.072036] RD=0 WR=0
[ 1397.083591] E=1 F=0; W=2000000 R=2000000
[ 1397.091689] RD=0 WR=0
[ 1397.103256] E=1 F=0; W=2000000 R=2000000
[ 1397.111359] RD=0 WR=0
[ 1397.122917] E=1 F=0; W=2000000 R=2000000
[ 1397.131013] RD=0 WR=0
[ 1397.142558] E=1 F=0; W=2000000 R=2000000
[ 1397.150649] RD=0 WR=0
[ 1397.162209] E=1 F=0; W=2000000 R=2000000
[ 1397.170301] RD=0 WR=0
[ 1397.181852] E=1 F=0; W=2000000 R=2000000
[ 1397.189943] RD=0 WR=0
[ 1397.201487] E=1 F=0; W=2000000 R=2000000
[ 1397.209580] RD=0 WR=0
[ 1397.221115] E=1 F=0; W=2000000 R=2000000
[ 1397.229204] RD=0 WR=0
[ 1397.240745] E=1 F=0; W=2000000 R=2000000
[ 1397.248839] RD=0 WR=0
[ 1397.260372] E=1 F=0; W=2000000 R=2000000
[ 1397.268463] RD=0 WR=0
[ 1397.280014] E=1 F=0; W=2000000 R=2000000
[ 1397.288103] RD=0 WR=0
[ 1397.299647] E=1 F=0; W=2000000 R=2000000
[ 1397.307734] RD=0 WR=0
[ 1397.319280] E=1 F=0; W=2000000 R=2000000
[ 1397.327370] RD=0 WR=0
[ 1397.338916] E=1 F=0; W=2000000 R=2000000
[ 1397.347001] RD=0 WR=0
[ 1397.358548] E=1 F=0; W=2000000 R=2000000
[ 1397.366632] RD=0 WR=0
[ 1397.378157] E=1 F=0; W=2000000 R=2000000
[ 1397.386246] RD=0 WR=0
[ 1397.397798] E=1 F=0; W=2000000 R=2000000
[ 1397.405888] RD=0 WR=0
[ 1397.417423] E=1 F=0; W=2000000 R=2000000
[ 1397.425510] RD=0 WR=0
[ 1397.437064] E=1 F=0; W=2000000 R=2000000
[ 1397.445155] RD=0 WR=0
[ 1397.456709] E=1 F=0; W=2000000 R=2000000
[ 1397.464800] RD=0 WR=0
[ 1397.476337] E=1 F=0; W=2000000 R=2000000
[ 1397.484430] RD=0 WR=0
[ 1397.495986] E=1 F=0; W=2000000 R=2000000
[ 1397.504081] RD=0 WR=0
[ 1397.515639] E=1 F=0; W=2000000 R=2000000
[ 1397.523737] RD=0 WR=0
[ 1397.535289] E=1 F=0; W=2000000 R=2000000
[ 1397.543372] RD=0 WR=0
[ 1397.554925] E=1 F=0; W=2000000 R=2000000
[ 1397.563021] RD=0 WR=0
[ 1397.574562] E=1 F=0; W=2000000 R=2000000
[ 1397.582654] RD=0 WR=0
[ 1397.594199] E=1 F=0; W=2000000 R=2000000
[ 1397.602287] RD=0 WR=0
[ 1397.613840] E=1 F=0; W=2000000 R=2000000
[ 1397.621934] RD=0 WR=0
[ 1397.633484] E=1 F=0; W=2000000 R=2000000
[ 1397.641570] RD=0 WR=0
[ 1397.653119] E=1 F=0; W=2000000 R=2000000
[ 1397.661211] RD=0 WR=0
[ 1397.672759] E=1 F=0; W=2000000 R=2000000
[ 1397.680850] RD=0 WR=0
[ 1397.692391] E=1 F=0; W=2000000 R=2000000
[ 1397.700474] RD=0 WR=0
[ 1397.712018] E=1 F=0; W=2000000 R=2000000
[ 1397.720110] RD=0 WR=0
[ 1397.731651] E=1 F=0; W=2000000 R=2000000
[ 1397.739739] RD=0 WR=0
[ 1397.751279] E=1 F=0; W=2000000 R=2000000
[ 1397.759370] RD=0 WR=0
[ 1397.770904] E=1 F=0; W=2000000 R=2000000
[ 1397.778992] RD=0 WR=0
[ 1397.790521] E=1 F=0; W=2000000 R=2000000
[ 1397.798607] RD=0 WR=0
[ 1397.810140] E=1 F=0; W=1719158 R=1719158
[ 1397.818226] RD=1 WR=0
[ 1397.829761] E=1 F=0; W=1719147 R=1719147
[ 1397.837843] RD=1 WR=1
[ 1397.849382] E=1 F=0; W=1719147 R=1719147
[ 1397.857465] RD=1 WR=0
[ 1397.869004] E=1 F=0; W=1719147 R=1719147
[ 1397.877090] RD=1 WR=0
[ 1397.888627] E=1 F=0; W=1719147 R=1719147
[ 1397.896715] RD=1 WR=0
[ 1397.908261] E=1 F=0; W=1719147 R=1719147
[ 1397.916348] RD=1 WR=0
[ 1397.927895] E=1 F=0; W=1719147 R=1719147
[ 1397.935981] RD=1 WR=0
[ 1397.947520] E=1 F=0; W=1719147 R=1719147
[ 1397.955600] RD=1 WR=0
[ 1397.967128] E=1 F=0; W=1719147 R=1719147
[ 1397.975211] RD=1 WR=0
[ 1397.986745] E=1 F=0; W=1719147 R=1719147
[ 1397.994830] RD=1 WR=0
[ 1398.006391] E=1 F=0; W=1719147 R=1719147
[ 1398.014480] RD=1 WR=0
[ 1398.026020] E=1 F=0; W=1719147 R=1719147
[ 1398.034104] RD=1 WR=0
[ 1398.045662] E=1 F=0; W=1719147 R=1719147
[ 1398.053754] RD=1 WR=0
[ 1398.065297] E=1 F=0; W=1719147 R=1719147
[ 1398.073385] RD=1 WR=0
[ 1398.084929] E=1 F=0; W=1719147 R=1719147
[ 1398.093022] RD=1 WR=0
[ 1398.104572] E=1 F=0; W=1719147 R=1719147
[ 1398.112666] RD=1 WR=0
[ 1398.124215] E=1 F=0; W=1719147 R=1719147
[ 1398.132303] RD=1 WR=0
[ 1398.143856] E=1 F=0; W=1719147 R=1719147
[ 1398.151945] RD=1 WR=0
[ 1398.163499] E=1 F=0; W=1719147 R=1719147
[ 1398.171592] RD=1 WR=0
[ 1398.183134] E=1 F=0; W=1719147 R=1719147
[ 1398.191225] RD=1 WR=0
[ 1398.202768] E=1 F=0; W=2000000 R=2000000
[ 1398.210859] RD=0 WR=0
[ 1398.222399] E=1 F=0; W=2000000 R=2000000
[ 1398.230485] RD=0 WR=0
[ 1398.242031] E=1 F=0; W=2000000 R=2000000
[ 1398.250122] RD=0 WR=0
[ 1398.261683] E=1 F=0; W=2000000 R=2000000
[ 1398.269786] RD=0 WR=0
[ 1398.281349] E=1 F=0; W=2000000 R=2000000
[ 1398.289447] RD=0 WR=0
[ 1398.301001] E=1 F=0; W=2000000 R=2000000
[ 1398.309100] RD=0 WR=0
[ 1398.320645] E=1 F=0; W=2000000 R=2000000
[ 1398.328735] RD=0 WR=0
[ 1398.340276] E=1 F=0; W=2000000 R=2000000
[ 1398.348363] RD=0 WR=0
[ 1398.359902] E=1 F=0; W=2000000 R=2000000
[ 1398.367988] RD=0 WR=0
[ 1398.379526] E=1 F=0; W=2000000 R=2000000
[ 1398.387609] RD=0 WR=0
[ 1398.399146] E=1 F=0; W=2000000 R=2000000
[ 1398.407231] RD=0 WR=0
[ 1398.418775] E=1 F=0; W=2000000 R=2000000
[ 1398.426858] RD=0 WR=0
[ 1398.438394] E=1 F=0; W=2000000 R=2000000
[ 1398.446475] RD=0 WR=0
[ 1398.458004] E=1 F=0; W=2000000 R=2000000
[ 1398.466084] RD=0 WR=0
[ 1398.477611] E=1 F=0; W=2000000 R=2000000
[ 1398.485695] RD=0 WR=0
[ 1398.497226] E=1 F=0; W=2000000 R=2000000
[ 1398.505308] RD=0 WR=0
[ 1398.516848] E=1 F=0; W=2000000 R=2000000
[ 1398.524934] RD=0 WR=0
[ 1398.536469] E=1 F=0; W=2000000 R=2000000
[ 1398.544554] RD=0 WR=0
[ 1398.556088] E=1 F=0; W=2000000 R=2000000
[ 1398.564171] RD=0 WR=0
[ 1398.575707] E=1 F=0; W=2000000 R=2000000
[ 1398.583794] RD=0 WR=0
[ 1398.595331] E=1 F=0; W=2000000 R=2000000
[ 1398.603421] RD=0 WR=0
[ 1398.614981] E=1 F=0; W=2000000 R=2000000
[ 1398.623073] RD=0 WR=0
[ 1398.634620] E=1 F=0; W=2000000 R=2000000
[ 1398.642716] RD=0 WR=0
[ 1398.654268] E=1 F=0; W=2000000 R=2000000
[ 1398.662367] RD=0 WR=0
[ 1398.673915] E=1 F=0; W=2000000 R=2000000
[ 1398.682011] RD=0 WR=0
[ 1398.693577] E=1 F=0; W=2000000 R=2000000
[ 1398.701673] RD=0 WR=0
[ 1398.713231] E=1 F=0; W=2000000 R=2000000
[ 1398.721325] RD=0 WR=0
[ 1398.732888] E=1 F=0; W=2000000 R=2000000
[ 1398.740983] RD=0 WR=0
[ 1398.752544] E=1 F=0; W=2000000 R=2000000
[ 1398.760640] RD=0 WR=0
[ 1398.772197] E=1 F=0; W=2000000 R=2000000
[ 1398.780294] RD=0 WR=0
[ 1398.791845] E=1 F=0; W=2000000 R=2000000
[ 1398.799943] RD=0 WR=0
[ 1398.811504] E=1 F=0; W=2000000 R=2000000
[ 1398.819602] RD=0 WR=0
[ 1398.831146] E=1 F=0; W=2000000 R=2000000
[ 1398.839237] RD=0 WR=0
[ 1398.850784] E=1 F=0; W=2000000 R=2000000
[ 1398.858874] RD=0 WR=0
[ 1398.870409] E=1 F=0; W=2000000 R=2000000
[ 1398.878494] RD=0 WR=0
[ 1398.890025] E=1 F=0; W=2000000 R=2000000
[ 1398.898108] RD=0 WR=0
[ 1398.909661] E=1 F=0; W=2000000 R=2000000
[ 1398.917746] RD=0 WR=0
[ 1398.929296] E=1 F=0; W=2000000 R=2000000
[ 1398.937384] RD=0 WR=0
[ 1398.948929] E=1 F=0; W=2000000 R=2000000
[ 1398.957019] RD=0 WR=0
[ 1398.968558] E=1 F=0; W=2000000 R=2000000
[ 1398.976647] RD=0 WR=0
[ 1398.988215] E=1 F=0; W=2000000 R=2000000
[ 1398.996319] RD=0 WR=0
[ 1399.007905] E=1 F=0; W=2000000 R=2000000
[ 1399.016001] RD=0 WR=0
[ 1399.027563] E=1 F=0; W=2000000 R=2000000
[ 1399.035657] RD=0 WR=0
[ 1399.047213] E=1 F=0; W=2000000 R=2000000
[ 1399.055307] RD=0 WR=0
[ 1399.066876] E=1 F=0; W=2000000 R=2000000
[ 1399.074979] RD=0 WR=0
[ 1399.086548] E=1 F=0; W=2000000 R=2000000
[ 1399.094652] RD=0 WR=0
[ 1399.106229] E=1 F=0; W=2000000 R=2000000
[ 1399.114341] RD=0 WR=0
[ 1399.125919] E=1 F=0; W=2000000 R=2000000
[ 1399.134026] RD=0 WR=0
[ 1399.145616] E=1 F=0; W=2000000 R=2000000
[ 1399.153720] RD=0 WR=0
[ 1399.165307] E=1 F=0; W=2000000 R=2000000
[ 1399.173423] RD=0 WR=0
[ 1399.185007] E=1 F=0; W=2000000 R=2000000
[ 1399.193113] RD=0 WR=0
[ 1399.204691] E=1 F=0; W=2000000 R=2000000
[ 1399.212812] RD=0 WR=0
[ 1399.224410] E=1 F=0; W=2000000 R=2000000
[ 1399.232525] RD=0 WR=0
[ 1399.244127] E=1 F=0; W=2000000 R=2000000
[ 1399.252246] RD=0 WR=0
[ 1399.263850] E=1 F=0; W=2000000 R=2000000
[ 1399.271971] RD=0 WR=0
[ 1399.283579] E=1 F=0; W=2000000 R=2000000
[ 1399.291701] RD=0 WR=0
[ 1399.303311] E=1 F=0; W=2000000 R=2000000
[ 1399.311441] RD=0 WR=0
[ 1399.323047] E=1 F=0; W=2000000 R=2000000
[ 1399.331165] RD=0 WR=0
[ 1399.342769] E=1 F=0; W=2000000 R=2000000
[ 1399.350887] RD=0 WR=0
[ 1399.362486] E=1 F=0; W=2000000 R=2000000
[ 1399.370593] RD=0 WR=0
[ 1399.382178] E=1 F=0; W=2000000 R=2000000
[ 1399.390291] RD=0 WR=0
[ 1399.401866] E=1 F=0; W=2000000 R=2000000
[ 1399.409971] RD=0 WR=0
[ 1399.421555] E=1 F=0; W=2000000 R=2000000
[ 1399.429669] RD=0 WR=0
[ 1399.441249] E=1 F=0; W=2000000 R=2000000
[ 1399.449361] RD=0 WR=0
[ 1399.460949] E=1 F=0; W=2000000 R=2000000
[ 1399.469068] RD=0 WR=0
[ 1399.480660] E=1 F=0; W=2000000 R=2000000
[ 1399.488778] RD=0 WR=0
[ 1399.500366] E=1 F=0; W=2000000 R=2000000
[ 1399.508475] RD=0 WR=0
[ 1399.520060] E=1 F=0; W=2000000 R=2000000
[ 1399.528170] RD=0 WR=0
[ 1399.539747] E=1 F=0; W=2000000 R=2000000
[ 1399.547850] RD=0 WR=0
[ 1399.559425] E=1 F=0; W=2000000 R=2000000
[ 1399.567530] RD=0 WR=0
[ 1399.579103] E=1 F=0; W=2000000 R=2000000
[ 1399.587207] RD=0 WR=0
[ 1399.598769] E=1 F=0; W=2000000 R=2000000
[ 1399.606865] RD=0 WR=0
[ 1399.618424] E=1 F=0; W=2000000 R=2000000
[ 1399.626520] RD=0 WR=0
[ 1399.638080] E=1 F=0; W=2000000 R=2000000
[ 1399.646175] RD=0 WR=0
[ 1399.657744] E=1 F=0; W=2000000 R=2000000
[ 1399.665842] RD=0 WR=0
[ 1399.677415] E=1 F=0; W=2000000 R=2000000
[ 1399.685518] RD=0 WR=0
[ 1399.697092] E=1 F=0; W=2000000 R=2000000
[ 1399.705197] RD=0 WR=0
[ 1399.716762] E=1 F=0; W=2000000 R=2000000
[ 1399.724865] RD=0 WR=0
[ 1399.736434] E=1 F=0; W=2000000 R=2000000
[ 1399.744534] RD=0 WR=0
[ 1399.756089] E=1 F=0; W=2000000 R=2000000
[ 1399.764183] RD=0 WR=0
[ 1399.775731] E=1 F=0; W=2000000 R=2000000
[ 1399.783818] RD=0 WR=0
[ 1399.795374] E=1 F=0; W=2000000 R=2000000
[ 1399.803458] RD=0 WR=0
[ 1399.814992] E=1 F=0; W=2000000 R=2000000
[ 1399.823076] RD=0 WR=0
[ 1399.834602] E=1 F=0; W=2000000 R=2000000
[ 1399.842684] RD=0 WR=0
[ 1399.854209] E=1 F=0; W=2000000 R=2000000
[ 1399.862290] RD=0 WR=0
[ 1399.873796] E=1 F=0; W=2000000 R=2000000
[ 1399.881873] RD=0 WR=0
[ 1399.893400] E=1 F=0; W=2000000 R=2000000
[ 1399.901480] RD=0 WR=0
[ 1399.912997] E=1 F=0; W=2000000 R=2000000
[ 1399.921074] RD=0 WR=0
[ 1399.932601] E=1 F=0; W=2000000 R=2000000
[ 1399.940682] RD=0 WR=0
[ 1399.952202] E=1 F=0; W=2000000 R=2000000
[ 1399.960281] RD=0 WR=0
[ 1399.971803] E=1 F=0; W=2000000 R=2000000
[ 1399.979874] RD=0 WR=0
[ 1399.991384] E=1 F=0; W=2000000 R=2000000
[ 1399.999464] RD=0 WR=0
[ 1400.011005] E=1 F=0; W=2000000 R=2000000
[ 1400.019094] RD=0 WR=0
[ 1400.030619] E=1 F=0; W=2000000 R=2000000
[ 1400.038705] RD=0 WR=0
[ 1400.050241] E=1 F=0; W=2000000 R=2000000
[ 1400.058336] RD=0 WR=0
[ 1400.069874] E=1 F=0; W=2000000 R=2000000
[ 1400.077959] RD=0 WR=0
[ 1400.089508] E=1 F=0; W=2000000 R=2000000
[ 1400.097597] RD=0 WR=0
[ 1400.109149] E=1 F=0; W=2000000 R=2000000
[ 1400.117240] RD=0 WR=0
[ 1400.128796] E=1 F=0; W=2000000 R=2000000
[ 1400.136891] RD=0 WR=0
[ 1400.148448] E=1 F=0; W=2000000 R=2000000
[ 1400.156539] RD=0 WR=0
[ 1400.168097] E=1 F=0; W=2000000 R=2000000
[ 1400.176193] RD=0 WR=0
[ 1400.187752] E=1 F=0; W=2000000 R=2000000
[ 1400.195848] RD=0 WR=0
[ 1400.207409] E=1 F=0; W=2000000 R=2000000
[ 1400.215509] RD=0 WR=0
[ 1400.227064] E=1 F=0; W=2000000 R=2000000
[ 1400.235158] RD=0 WR=0
[ 1400.246716] E=1 F=0; W=2000000 R=2000000
[ 1400.254812] RD=0 WR=0
[ 1400.266368] E=1 F=0; W=2000000 R=2000000
[ 1400.274461] RD=0 WR=0
[ 1400.286002] E=1 F=0; W=2000000 R=2000000
[ 1400.294091] RD=0 WR=0
[ 1400.305629] E=1 F=0; W=2000000 R=2000000
[ 1400.313722] RD=0 WR=0
[ 1400.325237] E=1 F=0; W=2000000 R=2000000
[ 1400.333316] RD=0 WR=0
[ 1400.344834] E=1 F=0; W=2000000 R=2000000
[ 1400.352912] RD=0 WR=0
[ 1400.364432] E=1 F=0; W=2000000 R=2000000
[ 1400.372514] RD=0 WR=0
[ 1400.384052] E=1 F=0; W=2000000 R=2000000
[ 1400.392134] RD=0 WR=0
[ 1400.403665] E=1 F=0; W=2000000 R=2000000
[ 1400.411755] RD=0 WR=0
[ 1400.423282] E=1 F=0; W=2000000 R=2000000
[ 1400.431365] RD=0 WR=0
[ 1400.442897] E=1 F=0; W=2000000 R=2000000
[ 1400.450984] RD=0 WR=0
[ 1400.462520] E=1 F=0; W=2000000 R=2000000
[ 1400.470607] RD=0 WR=0
[ 1400.482141] E=1 F=0; W=2000000 R=2000000
[ 1400.490225] RD=0 WR=0
[ 1400.501793] E=1 F=0; W=2000000 R=2000000
[ 1400.509886] RD=0 WR=0
[ 1400.521428] E=1 F=0; W=2000000 R=2000000
[ 1400.529517] RD=0 WR=0
[ 1400.541063] E=1 F=0; W=2000000 R=2000000
[ 1400.549149] RD=0 WR=0
[ 1400.560688] ---------- DUMP END ------------
#32 lore raw
Quoted 2 lines · Swapnil

Yes, I was able to reproduce the problem with the below patch.

...

Quoted 5 lines

I found a case in the dump where the pipe is empty still both reader and writer are waiting on it.

[ 1397.829761] E=1 F=0; W=1719147 R=1719147
[ 1397.837843] RD=1 WR=1

Thanks! and I see no more "WR=1" in the full dump.

This means that all live writes hang on the same pipe.

So maybe the trivial program below can too reproduce the problem on your machine??

Say, with GROUPS=16 and WRITERS=20 ... or maybe even with GROUPS=1 and WRITERS=320 ...

Oleg.

-------------------------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <assert.h>
#include <pthread.h>

static int GROUPS, WRITERS;
static volatile int ALIVE[1024];

void *group(void *arg)

{
	int fd[2], n, id = (long)arg;
	char buf[100];

	assert(pipe(fd) == 0);

	for (n = 0; n < WRITERS; ++n) {
		int pid = fork();
		assert(pid >= 0);
		if (pid)
			continue;

		close(fd[0]);
		for (;;)
			assert(write(fd[1], buf, sizeof(buf)) == sizeof(buf));
	}

	for (;;) {
		assert(read(fd[0], buf, sizeof(buf)) == sizeof(buf));
		ALIVE[id] = 1;
	}
}

int main(int argc, const char *argv[])

{
	pthread_t pt;
	int n;

	assert(argc == 3);
	GROUPS = atoi(argv[1]);
	WRITERS = atoi(argv[2]);
	assert(GROUPS <= 1024);

	for (n = 0; n < GROUPS; ++n)
		assert(pthread_create(&pt, NULL, group, (void*)(long)n) == 0);

	for (;;) {
		sleep(1);

		for (n = 0; n < GROUPS; ++n) {
			if (ALIVE[n] == 0)
				printf("!!! thread %d stuck?\n", n);
			ALIVE[n] = 0;
		}
	}
}
#33 lore raw

And... I know, I know you already hate me ;)

but if you have time, could you check if this patch (with or without the previous debugging patch) makes any difference? Just to be sure.

Oleg.
---

fs/pipe.c +12 -7
index 4336b8cccf84..524b8845523e 100644
@@ -445,7 +445,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 		return 0;
 
 	mutex_lock(&pipe->mutex);
-
+again:
 	if (!pipe->readers) {
 		send_sig(SIGPIPE, current, 0);
 		ret = -EPIPE;
@@ -467,20 +467,24 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 		unsigned int mask = pipe->ring_size - 1;
 		struct pipe_buffer *buf = &pipe->bufs[(head - 1) & mask];
 		int offset = buf->offset + buf->len;
+		int xxx;
 
 		if ((buf->flags & PIPE_BUF_FLAG_CAN_MERGE) &&
 		    offset + chars <= PAGE_SIZE) {
-			ret = pipe_buf_confirm(pipe, buf);
-			if (ret)
+			xxx = pipe_buf_confirm(pipe, buf);
+			if (xxx) {
+				if (!ret) ret = xxx;
 				goto out;
+			}
 
-			ret = copy_page_from_iter(buf->page, offset, chars, from);
-			if (unlikely(ret < chars)) {
-				ret = -EFAULT;
+			xxx = copy_page_from_iter(buf->page, offset, chars, from);
+			if (unlikely(xxx < chars)) {
+				if (!ret) ret = -EFAULT;
 				goto out;
 			}
 
-			buf->len += ret;
+			ret += xxx;
+			buf->len += xxx;
 			if (!iov_iter_count(from))
 				goto out;
 		}
@@ -567,6 +571,7 @@ atomic_inc(&WR_SLEEP);
 		mutex_lock(&pipe->mutex);
 		was_empty = pipe_empty(pipe->head, pipe->tail);
 		wake_next_writer = true;
+		goto again;
 	}
 out:
 	if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
#34 lore raw

Hi Oleg,

Quoted 2 lines · Oleg Nesterov

And... I know, I know you already hate me ;)

Not at all :)

Quoted 3 lines

but if you have time, could you check if this patch (with or without the previous debugging patch) makes any difference? Just to be sure.

Sure, I will give this a try.

But in the meanwhile me and Prateek tried some of the experiments in the weekend. We were able to reproduce this issue on a third generation EPYC system as well as on an Intel Emerald Rapids (2 X INTEL(R) XEON(R) PLATINUM 8592+).

We tried heavy hammered tracing approach over the weekend on top of your debug patch. I have attached the debug patch below. With tracing we found the following case for pipe_writable():

hackbench-118768 [206] ..... 1029.550601: pipe_write: 000000005eea28ff: 0: 37 38 16: 1

Here,

head = 37
tail = 38
max_usage = 16
pipe_full() returns 1.

Between reading of head and later the tail, the tail seems to have moved ahead of the head leading to wraparound. Applying the following changes I have not yet run into a hang on the original machine where I first saw it:

fs/pipe.c +12 -2
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_writable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
 	unsigned int max_usage = READ_ONCE(pipe->max_usage);
+	unsigned int head, tail;
+
+	tail = READ_ONCE(pipe->tail);
+	/*
+	 * Since the unsigned arithmetic in this lockless preemptible context
+	 * relies on the fact that the tail can never be ahead of head, read
+	 * the head after the tail to ensure we've not missed any updates to
+	 * the head. Reordering the reads can cause wraparounds and give the
+	 * illusion that the pipe is full.
+	 */
+	smp_rmb();
+	head = READ_ONCE(pipe->head);
 
 	return !pipe_full(head, tail, max_usage) ||
 		!READ_ONCE(pipe->readers);

---

smp_rmb() on x86 is a nop and even without the barrier we were not able to reproduce the hang even after 10000 iterations.

If you think this is a genuine bug fix, I will send a patch for this.

Thanks to Prateek who was actively involved in this debug.

--
Thanks and Regards,
Swapnil

Quoted 49 lines

Oleg.
---

fs/pipe.c +11 -7
index 4336b8cccf84..524b8845523e 100644
@@ -445,7 +445,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
  		return 0;
 
  	mutex_lock(&pipe->mutex);
-
+again:
  	if (!pipe->readers) {
  		send_sig(SIGPIPE, current, 0);
  		ret = -EPIPE;
@@ -467,20 +467,24 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
  		unsigned int mask = pipe->ring_size - 1;
  		struct pipe_buffer *buf = &pipe->bufs[(head - 1) & mask];
  		int offset = buf->offset + buf->len;
+		int xxx;
 
  		if ((buf->flags & PIPE_BUF_FLAG_CAN_MERGE) &&
  		    offset + chars <= PAGE_SIZE) {
-			ret = pipe_buf_confirm(pipe, buf);
-			if (ret)
+			xxx = pipe_buf_confirm(pipe, buf);
+			if (xxx) {
+				if (!ret) ret = xxx;
  				goto out;
+			}
 
-			ret = copy_page_from_iter(buf->page, offset, chars, from);
-			if (unlikely(ret < chars)) {
-				ret = -EFAULT;
+			xxx = copy_page_from_iter(buf->page, offset, chars, from);
+			if (unlikely(xxx < chars)) {
+				if (!ret) ret = -EFAULT;
  				goto out;
  			}
 
-			buf->len += ret;
+			ret += xxx;
+			buf->len += xxx;
  			if (!iov_iter_count(from))
  				goto out;
  		}

fs/pipe.c atomic_inc(&WR_SLEEP); · line 577

Quoted 7 lines
fs/pipe.c +1 -0
@@ -567,6 +571,7 @@ atomic_inc(&WR_SLEEP);
  		mutex_lock(&pipe->mutex);
  		was_empty = pipe_empty(pipe->head, pipe->tail);
  		wake_next_writer = true;
+		goto again;
  	}
  out:
  	if (pipe_full(pipe->head, pipe->tail, pipe->max_usage))
fs/pipe.c +94 -3

170 lines not on this page · raw message

include/linux/pipe_fs_i.h +3 -0
index 8ff23bf5a819..48d9bf5171dc 100644
@@ -80,6 +80,9 @@ struct pipe_inode_info {
 #ifdef CONFIG_WATCH_QUEUE
 	struct watch_queue *watch_queue;
 #endif
+
+	struct list_head pi_list;
+	unsigned w_cnt, r_cnt;
 };
 
 /*
include/linux/wait.h +1 -0
index 6d90ad974408..2c37517f6a05 100644
@@ -215,6 +215,7 @@ void __wake_up_locked_sync_key(struct wait_queue_head *wq_head, unsigned int mod
 void __wake_up_locked(struct wait_queue_head *wq_head, unsigned int mode, int nr);
 void __wake_up_sync(struct wait_queue_head *wq_head, unsigned int mode);
 void __wake_up_pollfree(struct wait_queue_head *wq_head);
+void __wait_queue_traverse_print_tasks(struct wait_queue_head *wq_head);
 
 #define wake_up(x)			__wake_up(x, TASK_NORMAL, 1, NULL)
 #define wake_up_nr(x, nr)		__wake_up(x, TASK_NORMAL, nr, NULL)
kernel/sched/wait.c +23 -0
index 51e38f5f4701..8f33da87a219 100644
@@ -174,6 +174,29 @@ void __wake_up_sync_key(struct wait_queue_head *wq_head, unsigned int mode,
 }
 EXPORT_SYMBOL_GPL(__wake_up_sync_key);
 
+void __wait_queue_traverse_print_tasks(struct wait_queue_head *wq_head)
+{
+	wait_queue_entry_t *curr, *next;
+	unsigned long flags;
+
+	if (unlikely(!wq_head))
+		return;
+
+	spin_lock_irqsave(&wq_head->lock, flags);
+	curr = list_first_entry(&wq_head->head, wait_queue_entry_t, entry);
+
+	if (&curr->entry == &wq_head->head)
+		return;
+
+	list_for_each_entry_safe_from(curr, next, &wq_head->head, entry) {
+		struct task_struct *tsk =  (struct task_struct *)curr->private;
+
+		pr_crit("%d(%s)\n", tsk->pid, tsk->comm);
+	}
+	spin_unlock_irqrestore(&wq_head->lock, flags);
+}
+EXPORT_SYMBOL_GPL(__wait_queue_traverse_print_tasks);
+
 /**
  * __wake_up_locked_sync_key - wake up a thread blocked on a locked waitqueue.
  * @wq_head: the waitqueue
kernel/sys.c +5 -0
index c4c701c6f0b4..676e623d491d 100644
@@ -2477,6 +2477,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 
 	error = 0;
 	switch (option) {
+	case 666: {
+		extern void pi_dump(void);
+		pi_dump();
+		break;
+	}
 	case PR_SET_PDEATHSIG:
 		if (!valid_signal(arg2)) {
 			error = -EINVAL;
#22 lore raw
In reply to Oleg Nesterov
Quoted 16 lines · Oleg Nesterov
Quoted 6 lines · Oleg Nesterov

Just in case, did you use

https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/hackbench/hackbench.c

?

Or did you use another version?

Exactly what parameters did you use?

If possible, please reproduce the hang again. How many threads/processes sleeping in pipe_read() or pipe_write() do you see? (you can look at /proc/$pid/stack).

Quoted 8 lines

Please pick one sleeping writer, and do

	$ strace -p pidof_that_write

this should wake this writer up. If a missed wakeup is the only problem, hackbench should continue.

The more info you can provide the better ;)

I was also able to reproduce the hackbench hang with the parameters mentioned earlier (threads and processes) on the kernel from master.

Signature · 1 line
Rgrds, legion
#35 lore raw
In reply to Sapkal, Swapnil
Quoted 8 lines · Swapnil

But in the meanwhile me and Prateek tried some of the experiments in the weekend. We were able to reproduce this issue on a third generation EPYC system as well as on an Intel Emerald Rapids (2 X INTEL(R) XEON(R) PLATINUM 8592+).

We tried heavy hammered tracing approach over the weekend on top of your debug patch. I have attached the debug patch below. With tracing we found the following case for pipe_writable():

Quoted 9 lines
   hackbench-118768  [206] .....  1029.550601: pipe_write: 000000005eea28ff: 0: 37 38 16: 1

Here,

head = 37
tail = 38
max_usage = 16
pipe_full() returns 1.

AFAICT the benchmark has one reader per fd, but multiple writers.

Maybe I'm misunderstanding something, but for such a case I think this is expected as a possible transient condition and while not ideal, it should not lead to the bug at hand.

Suppose there is only one reader and one writer and a wakeup-worthy condition showed up. Then both sides perform wakeups *after* dropping the pipe mutex, meaning their state is published before whoever they intend to wake up gets on CPU. At the same time any new arrivals which did not sleep start with taking the mutex.

Suppose there are two or more writers (one of which is blocked) and still one reader and the pipe transitions to no longer full. Before the woken up writer reaches pipe_writable() the pipe could have transitioned to any state an arbitrary number of times, but someone had to observe the correct state. In particular it is legitimate for a non-sleeping writer to sneak in and fill in the pipe and the reader to have time to get back empty it again etc.

Or to put it differently, if the patch does correct the bug, it needs to explain how everyone ends up blocked. Per the above, there always should be at least one writer and one reader who make progress -- this somehow breaks (hence the bug), but I don't believe the memory ordering explains it.

Consequently I think the patch happens to hide the bug, just like the now removed spurious wakeup used to do.

Quoted 37 lines

Between reading of head and later the tail, the tail seems to have moved ahead of the head leading to wraparound. Applying the following changes I have not yet run into a hang on the original machine where I first saw it:

fs/pipe.c +12 -3
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
  /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
  static inline bool pipe_writable(const struct pipe_inode_info *pipe)
  {
-       unsigned int head = READ_ONCE(pipe->head);
-       unsigned int tail = READ_ONCE(pipe->tail);
        unsigned int max_usage = READ_ONCE(pipe->max_usage);
+       unsigned int head, tail;
+
+       tail = READ_ONCE(pipe->tail);
+       /*
+        * Since the unsigned arithmetic in this lockless preemptible context
+        * relies on the fact that the tail can never be ahead of head, read
+        * the head after the tail to ensure we've not missed any updates to
+        * the head. Reordering the reads can cause wraparounds and give the
+        * illusion that the pipe is full.
+        */
+       smp_rmb();
+       head = READ_ONCE(pipe->head);
 
        return !pipe_full(head, tail, max_usage) ||
                !READ_ONCE(pipe->readers);
---

smp_rmb() on x86 is a nop and even without the barrier we were not able to reproduce the hang even after 10000 iterations.

If you think this is a genuine bug fix, I will send a patch for this.

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#36 lore raw
Quoted 40 lines · Mateusz Guzik
Quoted 8 lines · Swapnil

But in the meanwhile me and Prateek tried some of the experiments in the weekend. We were able to reproduce this issue on a third generation EPYC system as well as on an Intel Emerald Rapids (2 X INTEL(R) XEON(R) PLATINUM 8592+).

We tried heavy hammered tracing approach over the weekend on top of your debug patch. I have attached the debug patch below. With tracing we found the following case for pipe_writable():

Quoted 9 lines
   hackbench-118768  [206] .....  1029.550601: pipe_write: 000000005eea28ff: 0: 37 38 16: 1

Here,

head = 37
tail = 38
max_usage = 16
pipe_full() returns 1.

AFAICT the benchmark has one reader per fd, but multiple writers.

Maybe I'm misunderstanding something, but for such a case I think this is expected as a possible transient condition and while not ideal, it should not lead to the bug at hand.

Suppose there is only one reader and one writer and a wakeup-worthy condition showed up. Then both sides perform wakeups *after* dropping the pipe mutex, meaning their state is published before whoever they intend to wake up gets on CPU. At the same time any new arrivals which did not sleep start with taking the mutex.

Suppose there are two or more writers (one of which is blocked) and still one reader and the pipe transitions to no longer full. Before the woken up writer reaches pipe_writable() the pipe could have transitioned to any state an arbitrary number of times, but someone had to observe the correct state. In particular it is legitimate for a non-sleeping writer to sneak in and fill in the pipe and the reader to have time to get back empty it again etc.

Quoted 9 lines

Or to put it differently, if the patch does correct the bug, it needs to explain how everyone ends up blocked. Per the above, there always should be at least one writer and one reader who make progress -- this somehow breaks (hence the bug), but I don't believe the memory ordering explains it.

Consequently I think the patch happens to hide the bug, just like the now removed spurious wakeup used to do.

Now that I wrote the above, I had an epiphany and indeed there may be something to it. :)

Suppose the pipe is full, the reader drains one buffer and issues a wakeup on its way out. There is still several bytes stored to read.

Suppose the woken up writer is still trying to get on CPU.

On subsequent calls the reader keeps messing with the tail, *exposing* the possibility of the pipe_writable check being racy even if there is only one reader and one writer.

I'm gonna grab lunch and chew on it, but I think you guys are on the right track. But some more fences may be needed.

Quoted 41 lines
Quoted 32 lines

Between reading of head and later the tail, the tail seems to have moved ahead of the head leading to wraparound. Applying the following changes I have not yet run into a hang on the original machine where I first saw it:

fs/pipe.c +12 -3
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
  /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
  static inline bool pipe_writable(const struct pipe_inode_info *pipe)
  {
-       unsigned int head = READ_ONCE(pipe->head);
-       unsigned int tail = READ_ONCE(pipe->tail);
        unsigned int max_usage = READ_ONCE(pipe->max_usage);
+       unsigned int head, tail;
+
+       tail = READ_ONCE(pipe->tail);
+       /*
+        * Since the unsigned arithmetic in this lockless preemptible context
+        * relies on the fact that the tail can never be ahead of head, read
+        * the head after the tail to ensure we've not missed any updates to
+        * the head. Reordering the reads can cause wraparounds and give the
+        * illusion that the pipe is full.
+        */
+       smp_rmb();
+       head = READ_ONCE(pipe->head);
 
        return !pipe_full(head, tail, max_usage) ||
                !READ_ONCE(pipe->readers);
---
Quoted 5 lines

smp_rmb() on x86 is a nop and even without the barrier we were not able to reproduce the hang even after 10000 iterations.

If you think this is a genuine bug fix, I will send a patch for this.

--
Mateusz Guzik <mjguzik gmail.com>

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#37 lore raw

Hello Mateusz,

Quoted 54 lines · Mateusz Guzik
Quoted 40 lines · Mateusz Guzik
Quoted 8 lines · Swapnil

But in the meanwhile me and Prateek tried some of the experiments in the weekend. We were able to reproduce this issue on a third generation EPYC system as well as on an Intel Emerald Rapids (2 X INTEL(R) XEON(R) PLATINUM 8592+).

We tried heavy hammered tracing approach over the weekend on top of your debug patch. I have attached the debug patch below. With tracing we found the following case for pipe_writable():

Quoted 9 lines
    hackbench-118768  [206] .....  1029.550601: pipe_write: 000000005eea28ff: 0: 37 38 16: 1

Here,

head = 37
tail = 38
max_usage = 16
pipe_full() returns 1.

AFAICT the benchmark has one reader per fd, but multiple writers.

Maybe I'm misunderstanding something, but for such a case I think this is expected as a possible transient condition and while not ideal, it should not lead to the bug at hand.

Suppose there is only one reader and one writer and a wakeup-worthy condition showed up. Then both sides perform wakeups *after* dropping the pipe mutex, meaning their state is published before whoever they intend to wake up gets on CPU. At the same time any new arrivals which did not sleep start with taking the mutex.

Suppose there are two or more writers (one of which is blocked) and still one reader and the pipe transitions to no longer full. Before the woken up writer reaches pipe_writable() the pipe could have transitioned to any state an arbitrary number of times, but someone had to observe the correct state. In particular it is legitimate for a non-sleeping writer to sneak in and fill in the pipe and the reader to have time to get back empty it again etc.

Quoted 9 lines

Or to put it differently, if the patch does correct the bug, it needs to explain how everyone ends up blocked. Per the above, there always should be at least one writer and one reader who make progress -- this somehow breaks (hence the bug), but I don't believe the memory ordering explains it.

Consequently I think the patch happens to hide the bug, just like the now removed spurious wakeup used to do.

Now that I wrote the above, I had an epiphany and indeed there may be something to it. :)

Quoted 8 lines

Suppose the pipe is full, the reader drains one buffer and issues a wakeup on its way out. There is still several bytes stored to read.

Suppose the woken up writer is still trying to get on CPU.

On subsequent calls the reader keeps messing with the tail, *exposing* the possibility of the pipe_writable check being racy even if there is only one reader and one writer.

Yup. One possibility looking at the larger trace data, we may have a situation as follows:

Say:

pipe->head = 16
pipe->tail = 15
2 writers were waiting on a reader since pipe was full and action ...

        Reader                          Writer1                                  Writer2
        ======                          =======                                  =======

pipe->tail = tail + 1 (16)
(Wakes up writer 1) (!pipe_full() yay!)
done pipe->head = head + 1 (17)

                                (pipe is not full; wake writer2)        (yawn! I'm up)
                                done                                    head = READ_ONCE(pipe->head) (17)
                                                                        ... (interrupted)
(Guess who's back)
pipe->tail = tail + 1 (17)      (back again)
...                             pipe->head = head + 1 (18)
(reader's back)                 (I'm done mate!)                        ... (back)
pipe->tail = tail + 1 (18)                                              tail = READ_ONCE(pipe->tail) (18)
...                                                                     (u32)(17 - 18) >= 16? (Yup! Pipe is full)
(Sleeps until pipe has data)                                            (Sleep until pipe has room)

---

Now the above might be totally wrong and I might have missed a few intricacies of the wakeups in pipes but below is the trace snippet that led us to try rearranging the reads and test again:

    hackbench-118768  1029.549127: pipe_write: 000000005eea28ff: 0: 32 16 16: 1  (118768 - sleeps)
    ...
    hackbench-118766  1029.550592: pipe_write: 000000005eea28ff: h: 37 -> 38     (118766 - write succeeds)
    ...
    hackbench-118740  1029.550599: pipe_read:  000000005eea28ff: t: 37 -> 38     (118740 - read succeeds)
    ...
    hackbench-118740  1029.550599: pipe_read:  000000005eea28ff: 0: 38 38: 1     (118740 - sleeps)
    hackbench-118768  1029.550601: pipe_write: 000000005eea28ff: 0: 37 38 16: 1  (118768 - wakes up; finds tail (38) > head (37); sleeps)

The trace goes on but if at this point 118766 were to drop out, 118740 and 118768 would both indefinitely wait on each other. This is an uncharted territory for Swapnil and I so we are trying a bunch of stuff based on patterns we see - any and all advice is greatly appreciated.

Signature · 2 lines
Thanks and Regards,
Prateek
Quoted 48 lines

I'm gonna grab lunch and chew on it, but I think you guys are on the right track. But some more fences may be needed.

Quoted 41 lines
Quoted 32 lines

Between reading of head and later the tail, the tail seems to have moved ahead of the head leading to wraparound. Applying the following changes I have not yet run into a hang on the original machine where I first saw it:

fs/pipe.c +12 -3
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
   /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
   static inline bool pipe_writable(const struct pipe_inode_info *pipe)
   {
-       unsigned int head = READ_ONCE(pipe->head);
-       unsigned int tail = READ_ONCE(pipe->tail);
         unsigned int max_usage = READ_ONCE(pipe->max_usage);
+       unsigned int head, tail;
+
+       tail = READ_ONCE(pipe->tail);
+       /*
+        * Since the unsigned arithmetic in this lockless preemptible context
+        * relies on the fact that the tail can never be ahead of head, read
+        * the head after the tail to ensure we've not missed any updates to
+        * the head. Reordering the reads can cause wraparounds and give the
+        * illusion that the pipe is full.
+        */
+       smp_rmb();
+       head = READ_ONCE(pipe->head);
 
         return !pipe_full(head, tail, max_usage) ||
                 !READ_ONCE(pipe->readers);
---
Quoted 5 lines

smp_rmb() on x86 is a nop and even without the barrier we were not able to reproduce the hang even after 10000 iterations.

If you think this is a genuine bug fix, I will send a patch for this.

--
Mateusz Guzik <mjguzik gmail.com>

#23 lore raw
In reply to Alexey Gladkov

Hello Legion,

Quoted 21 lines · Alexey Gladkov
Quoted 20 lines · Oleg Nesterov
Quoted 6 lines · Oleg Nesterov

Just in case, did you use

https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/hackbench/hackbench.c

?

Or did you use another version?

Exactly what parameters did you use?

If possible, please reproduce the hang again. How many threads/processes sleeping in pipe_read() or pipe_write() do you see? (you can look at /proc/$pid/stack).

Please pick one sleeping writer, and do

	$ strace -p pidof_that_write
Quoted 8 lines
Quoted 4 lines

this should wake this writer up. If a missed wakeup is the only problem, hackbench should continue.

The more info you can provide the better ;)

I was also able to reproduce the hackbench hang with the parameters mentioned earlier (threads and processes) on the kernel from master.

Thank you for reporting your observations!

If you are able to reproduce it reliably, could you please give the below diff posted by Swapnil from the parallel thread [1] a try:

fs/pipe.c +12 -2
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_writable(const struct pipe_inode_info *pipe)
 {
-    unsigned int head = READ_ONCE(pipe->head);
-    unsigned int tail = READ_ONCE(pipe->tail);
     unsigned int max_usage = READ_ONCE(pipe->max_usage);
+    unsigned int head, tail;
+
+    tail = READ_ONCE(pipe->tail);
+    /*
+     * Since the unsigned arithmetic in this lockless preemptible context
+     * relies on the fact that the tail can never be ahead of head, read
+     * the head after the tail to ensure we've not missed any updates to
+     * the head. Reordering the reads can cause wraparounds and give the
+     * illusion that the pipe is full.
+     */
+    smp_rmb();
+    head = READ_ONCE(pipe->head);
 
     return !pipe_full(head, tail, max_usage) ||
         !READ_ONCE(pipe->readers);

---

We've been running hackbench for a while now with the above diff and we haven't run into a hang yet. Sorry for the troubles and thank you again.

[1] https://lore.kernel.org/all/[email protected]/

Signature · 2 lines
Thanks and Regards,
Prateek
#77 lore raw
In reply to Sapkal, Swapnil

Hi!

Quoted 5 lines · Swapnil
Quoted 3 lines

but if you have time, could you check if this patch (with or without the previous debugging patch) makes any difference? Just to be sure.

Sure, I will give this a try.

Forget ;)

[...snip...]

Quoted 13 lines
fs/pipe.c +4 -2
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_writable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
 	unsigned int max_usage = READ_ONCE(pipe->max_usage);
+	unsigned int head, tail;
+
+	tail = READ_ONCE(pipe->tail);
+	/*

fs/pipe.c static inline int is_packetized(struct file *file) · line 434

Quoted 10 lines
fs/pipe.c +8 -0
+	 * Since the unsigned arithmetic in this lockless preemptible context
+	 * relies on the fact that the tail can never be ahead of head, read
+	 * the head after the tail to ensure we've not missed any updates to
+	 * the head. Reordering the reads can cause wraparounds and give the
+	 * illusion that the pipe is full.
+	 */
+	smp_rmb();
+	head = READ_ONCE(pipe->head);
 	return !pipe_full(head, tail, max_usage) ||
 		!READ_ONCE(pipe->readers);

Ooh, thanks!!!

And sorry, can't work today. To be honest, I have some concerns, but probably I am wrong... I'll return tomorrow.

In any case, finally we have a hint. Thank you both!

(btw, please look at pipe_poll).

Oleg.

#24 lore raw
In reply to K Prateek Nayak
Quoted 66 lines · K Prateek Nayak

Hello Legion,

Quoted 21 lines · Alexey Gladkov
Quoted 20 lines · Oleg Nesterov
Quoted 6 lines · Oleg Nesterov

Just in case, did you use

https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git/tree/src/hackbench/hackbench.c

?

Or did you use another version?

Exactly what parameters did you use?

If possible, please reproduce the hang again. How many threads/processes sleeping in pipe_read() or pipe_write() do you see? (you can look at /proc/$pid/stack).

Please pick one sleeping writer, and do

	$ strace -p pidof_that_write
Quoted 8 lines
Quoted 4 lines

this should wake this writer up. If a missed wakeup is the only problem, hackbench should continue.

The more info you can provide the better ;)

I was also able to reproduce the hackbench hang with the parameters mentioned earlier (threads and processes) on the kernel from master.

Thank you for reporting your observations!

If you are able to reproduce it reliably, could you please give the below diff posted by Swapnil from the parallel thread [1] a try:

fs/pipe.c +12 -3
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
  /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
  static inline bool pipe_writable(const struct pipe_inode_info *pipe)
  {
-    unsigned int head = READ_ONCE(pipe->head);
-    unsigned int tail = READ_ONCE(pipe->tail);
      unsigned int max_usage = READ_ONCE(pipe->max_usage);
+    unsigned int head, tail;
+
+    tail = READ_ONCE(pipe->tail);
+    /*
+     * Since the unsigned arithmetic in this lockless preemptible context
+     * relies on the fact that the tail can never be ahead of head, read
+     * the head after the tail to ensure we've not missed any updates to
+     * the head. Reordering the reads can cause wraparounds and give the
+     * illusion that the pipe is full.
+     */
+    smp_rmb();
+    head = READ_ONCE(pipe->head);
 
      return !pipe_full(head, tail, max_usage) ||
          !READ_ONCE(pipe->readers);
---
Quoted 2 lines

We've been running hackbench for a while now with the above diff and we haven't run into a hang yet. Sorry for the troubles and thank you again.

No problem at all.

Along with the patch above, I tried reproducing the problem for about 40 minutes and no hangs found. Before that hackbench would hang within 5 minutes or so.

Signature · 1 line
Rgrds, legion
#38 lore raw
In reply to K Prateek Nayak

Can you guys try out the patch below?

It changes things up so that there is no need to read 2 different vars.

It is not the final version and I don't claim to be able to fully justify the thing at the moment either, but I would like to know if it fixes the problem.

If you don't have time that's fine, this is a quick jab. While I can't reproduce the bug myself even after inserting a delay by hand with msleep between the loads, I verified it does not outright break either. :P

fs/pipe.c +18 -13
index 19a7948ab234..e61ad589fc2c 100644
@@ -210,11 +210,21 @@ static const struct pipe_buf_operations anon_pipe_buf_ops = {
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_readable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
-	unsigned int writers = READ_ONCE(pipe->writers);
+	return !READ_ONCE(pipe->isempty) || !READ_ONCE(pipe->writers);
+}
+
+static inline void pipe_recalc_state(struct pipe_inode_info *pipe)
+{
+	pipe->isempty = pipe_empty(pipe->head, pipe->tail);
+	pipe->isfull = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
+	VFS_BUG_ON(pipe->isempty && pipe->isfull);
+}
 
-	return !pipe_empty(head, tail) || !writers;
+static inline void pipe_update_head(struct pipe_inode_info *pipe,
+				    unsigned int head)
+{
+	pipe->head = ++head;
+	pipe_recalc_state(pipe);
 }
 
 static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
@@ -244,6 +254,7 @@ static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
 	 * without the spinlock - the mutex is enough.
 	 */
 	pipe->tail = ++tail;
+	pipe_recalc_state(pipe);
 	return tail;
 }
 
@@ -403,12 +414,7 @@ static inline int is_packetized(struct file *file)
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_writable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
-	unsigned int max_usage = READ_ONCE(pipe->max_usage);
-
-	return !pipe_full(head, tail, max_usage) ||
-		!READ_ONCE(pipe->readers);
+	return !READ_ONCE(pipe->isfull) || !READ_ONCE(pipe->readers);
 }
 
 static ssize_t
@@ -512,7 +518,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 				break;
 			}
 
-			pipe->head = head + 1;
+			pipe_update_head(pipe, head);
 			pipe->tmp_page = NULL;
 			/* Insert it into the buffer array */
 			buf = &pipe->bufs[head & mask];
@@ -529,10 +535,9 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 
 			if (!iov_iter_count(from))
 				break;
-		}
 
-		if (!pipe_full(head, pipe->tail, pipe->max_usage))
 			continue;
+		}
 
 		/* Wait for buffer space to become available. */
 		if ((filp->f_flags & O_NONBLOCK) ||
include/linux/pipe_fs_i.h +2 -0
index 8ff23bf5a819..d4b7539399b5 100644
@@ -69,6 +69,8 @@ struct pipe_inode_info {
 	unsigned int r_counter;
 	unsigned int w_counter;
 	bool poll_usage;
+	bool isempty;
+	bool isfull;
 #ifdef CONFIG_WATCH_QUEUE
 	bool note_loss;
 #endif
#39 lore raw
Quoted 4 lines · Mateusz Guzik

Can you guys try out the patch below?

It changes things up so that there is no need to read 2 different vars.

No, please don't do it this way.

I think the memory ordering is interesting, and we ignored it - incorrectly - because all the "normal" cases are done either under the pipe lock (safe), or are done with "wait_event()" that will retry on wakeups.

And then we got the subtle issues with "was woken, but raced with order of operations" case got missed. This has probably been around forever (possibly since we got rid of the BKL).

But I don't like the "add separate full/empty fields that duplicate things", just to have those written always under the lock, and then loaded as one op.

I think there are better models.

So I think I'd prefer the "add the barrier" model.

We could also possibly just make head/tail be 16-bit fields, and then read things atomically by reading them as a single 32-bit word. That would expose the (existing) alpha issues more, since alpha doesn't have atomic 16-bit writes, but I can't find it in myself to care. I guess we could make it be two aligned 32-bit fields on alpha, and just use 64-bit reads.

We already treat those fields specially with the whole READ_ONCE() dance, so treating them even more specially would not be a noticeably different situation.

Hmm?

I just generally dislike redundant information in data structures. Then you get into nasty cases where some path forgets to update the redundant fields correctly. So I'd really just prefer the existing model, just with being careful about this case.

Linus

#75 lore raw
In reply to Mateusz Guzik

Hello Mateusz,

Quoted 7 lines · Mateusz Guzik

Can you guys try out the patch below?

It changes things up so that there is no need to read 2 different vars.

It is not the final version and I don't claim to be able to fully justify the thing at the moment either, but I would like to know if it fixes the problem.

Happy to help! We've queued the below patch for an overnight run, will report back once it is done.

Full disclaimer: We're testing on top of commit aaec5a95d596 ("pipe_read: don't wake up the writer if the pipe is still full") where the issue is more reproducible. I've replaced the VFS_BUG_ON() with a plain BUG_ON() based on [1] since v6.14-rc1 did not include the CONFIG_DEBUG_VFS bits. Hope that is alright.

[1] https://lore.kernel.org/lkml/[email protected]/

/off to get some shut eyes/

Signature · 2 lines
Thanks and Regards,
Prateek
Quoted 92 lines

If you don't have time that's fine, this is a quick jab. While I can't reproduce the bug myself even after inserting a delay by hand with msleep between the loads, I verified it does not outright break either. :P

fs/pipe.c +18 -13
index 19a7948ab234..e61ad589fc2c 100644
@@ -210,11 +210,21 @@ static const struct pipe_buf_operations anon_pipe_buf_ops = {
  /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
  static inline bool pipe_readable(const struct pipe_inode_info *pipe)
  {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
-	unsigned int writers = READ_ONCE(pipe->writers);
+	return !READ_ONCE(pipe->isempty) || !READ_ONCE(pipe->writers);
+}
+
+static inline void pipe_recalc_state(struct pipe_inode_info *pipe)
+{
+	pipe->isempty = pipe_empty(pipe->head, pipe->tail);
+	pipe->isfull = pipe_full(pipe->head, pipe->tail, pipe->max_usage);
+	VFS_BUG_ON(pipe->isempty && pipe->isfull);
+}
 
-	return !pipe_empty(head, tail) || !writers;
+static inline void pipe_update_head(struct pipe_inode_info *pipe,
+				    unsigned int head)
+{
+	pipe->head = ++head;
+	pipe_recalc_state(pipe);
  }
 
  static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
@@ -244,6 +254,7 @@ static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
  	 * without the spinlock - the mutex is enough.
  	 */
  	pipe->tail = ++tail;
+	pipe_recalc_state(pipe);
  	return tail;
  }
 
@@ -403,12 +414,7 @@ static inline int is_packetized(struct file *file)
  /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
  static inline bool pipe_writable(const struct pipe_inode_info *pipe)
  {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
-	unsigned int max_usage = READ_ONCE(pipe->max_usage);
-
-	return !pipe_full(head, tail, max_usage) ||
-		!READ_ONCE(pipe->readers);
+	return !READ_ONCE(pipe->isfull) || !READ_ONCE(pipe->readers);
  }
 
  static ssize_t
@@ -512,7 +518,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
  				break;
  			}
 
-			pipe->head = head + 1;
+			pipe_update_head(pipe, head);
  			pipe->tmp_page = NULL;
  			/* Insert it into the buffer array */
  			buf = &pipe->bufs[head & mask];
@@ -529,10 +535,9 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
 
  			if (!iov_iter_count(from))
  				break;
-		}
 
-		if (!pipe_full(head, pipe->tail, pipe->max_usage))
  			continue;
+		}
 
  		/* Wait for buffer space to become available. */
  		if ((filp->f_flags & O_NONBLOCK) ||
include/linux/pipe_fs_i.h +2 -0
index 8ff23bf5a819..d4b7539399b5 100644
@@ -69,6 +69,8 @@ struct pipe_inode_info {
  	unsigned int r_counter;
  	unsigned int w_counter;
  	bool poll_usage;
+	bool isempty;
+	bool isfull;
  #ifdef CONFIG_WATCH_QUEUE
  	bool note_loss;
  #endif
#40 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

But I don't like the "add separate full/empty fields that duplicate things", just to have those written always under the lock, and then loaded as one op.

I think there are better models.

So I think I'd prefer the "add the barrier" model.

I was trying to avoid having to reason about the fences, I would argue not having to worry about this makes future changes easier to make.

Quoted 6 lines

We could also possibly just make head/tail be 16-bit fields, and then read things atomically by reading them as a single 32-bit word. That would expose the (existing) alpha issues more, since alpha doesn't have atomic 16-bit writes, but I can't find it in myself to care. I guess we could make it be two aligned 32-bit fields on alpha, and just use 64-bit reads.

I admit I did not think of this, pretty obvious now that you mention it.

Perhaps either Prateek or Swapnil would be interested in coding this up?

Ultimately the crux of the issue is their finding.

Quoted 6 lines

I just generally dislike redundant information in data structures. Then you get into nasty cases where some path forgets to update the redundant fields correctly. So I'd really just prefer the existing model, just with being careful about this case.

The stock code already has a dedicated routine to advance the tail, adding one for head (instead of an ad-hoc increment) is borderline just clean up.

Then having both recalc the state imo does not add any bug-pronness [I did ignore the CONFIG_WATCH_QUEUE case though] afaics.

Even so, per the above, that's a side note.

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#41 lore raw
Quoted 4 lines · Mateusz Guzik

The stock code already has a dedicated routine to advance the tail, adding one for head (instead of an ad-hoc increment) is borderline just clean up.

There's currently a fair number of open-coded assignments:

    git grep -E 'pipe->((tail)|(head)).*=' fs/

and some of those are under specific locking rules together with other updates (ie the watch-queue 'note_loss' thing.

But hey, if some explicit empty/full flag is simpler, then it certainly does fit with our current model too, since we already do have those other flags (exactly like 'note_loss')

I do particularly hate seeing 'bool' in structures like this. On alpha it is either fundamentally racy, or it's 32-bit. On other architectures, it's typically 8 bits for a 1-bit value.

But we do have holes in that structure where it slots.

Linus

#42 lore raw
Quoted 18 lines · Linus Torvalds
Quoted 4 lines · Mateusz Guzik

The stock code already has a dedicated routine to advance the tail, adding one for head (instead of an ad-hoc increment) is borderline just clean up.

There's currently a fair number of open-coded assignments:

    git grep -E 'pipe->((tail)|(head)).*=' fs/

and some of those are under specific locking rules together with other updates (ie the watch-queue 'note_loss' thing.

But hey, if some explicit empty/full flag is simpler, then it certainly does fit with our current model too, since we already do have those other flags (exactly like 'note_loss')

Quoted 6 lines

I do particularly hate seeing 'bool' in structures like this. On alpha it is either fundamentally racy, or it's 32-bit. On other architectures, it's typically 8 bits for a 1-bit value.

But we do have holes in that structure where it slots.

I was thinking about just fs/pipe.c.

These helpers being used elsewhere is not something I was aware of (or thought would be a thing). The relevant git grep makes me self-nak that patch. :->

But that's some side stuff. Not only it is your call how to proceed, but per the previous e-mail I agree 16-byte head/tail and a 32-byte read would be best.

Hopefully the AMD guys will want to take a stab. Otherwise I'll hack it up.

Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
#43 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds

There's currently a fair number of open-coded assignments:

    git grep -E 'pipe->((tail)|(head)).*=' fs/

and some of those are under specific locking rules together with other updates (ie the watch-queue 'note_loss' thing.

Stupid question... but do we really need to change the code which update tail/head if we pack them into a single word?

I mean,

	-	unsigned int head;
	-	unsigned int tail;
	+	union {
	+		struct {
	+			u16 head, tail;
	+		}
	+
	+		__u32   head_tail;
	+	}

Now pipe_writebale() can read do READ_ONCE(pipe->head_tail) "atomically" without preemption and this is all we need, no?

Yes, pipe_writable() should take endianess into account, but this is simple...

Oleg

#44 lore raw
Quoted 3 lines · Oleg Nesterov

Stupid question... but do we really need to change the code which update tail/head if we pack them into a single word?

No. It's only the READ_ONCE() parts that need changing.

See this suggested patch, which does something very similar to what you were thinking of.

ENTIRELY UNTESTED, but it seems to generate ok code. It might even generate better code than what we have now.

Linus

 fs/pipe.c                 | 19 ++++++++-----------
 include/linux/pipe_fs_i.h | 39 +++++++++++++++++++++++++++++++++++++--
 2 files changed, 45 insertions(+), 13 deletions(-)
fs/pipe.c +8 -11
index ce1af7592780..97cc70572606 100644
@@ -210,11 +210,10 @@ static const struct pipe_buf_operations anon_pipe_buf_ops = {
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_readable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
+	union pipe_index idx = { READ_ONCE(pipe->head_tail) };
 	unsigned int writers = READ_ONCE(pipe->writers);
 
-	return !pipe_empty(head, tail) || !writers;
+	return !pipe_empty(idx.head, idx.tail) || !writers;
 }
 
 static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
@@ -417,11 +416,10 @@ static inline int is_packetized(struct file *file)
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_writable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
+	union pipe_index idx = { READ_ONCE(pipe->head_tail) };
 	unsigned int max_usage = READ_ONCE(pipe->max_usage);
 
-	return !pipe_full(head, tail, max_usage) ||
+	return !pipe_full(idx.head, idx.tail, max_usage) ||
 		!READ_ONCE(pipe->readers);
 }
 
@@ -659,7 +657,7 @@ pipe_poll(struct file *filp, poll_table *wait)
 {
 	__poll_t mask;
 	struct pipe_inode_info *pipe = filp->private_data;
-	unsigned int head, tail;
+	union pipe_index idx;
 
 	/* Epoll has some historical nasty semantics, this enables them */
 	WRITE_ONCE(pipe->poll_usage, true);
@@ -680,19 +678,18 @@ pipe_poll(struct file *filp, poll_table *wait)
 	 * if something changes and you got it wrong, the poll
 	 * table entry will wake you up and fix it.
 	 */
-	head = READ_ONCE(pipe->head);
-	tail = READ_ONCE(pipe->tail);
+	idx.head_tail = READ_ONCE(pipe->head_tail);
 
 	mask = 0;
 	if (filp->f_mode & FMODE_READ) {
-		if (!pipe_empty(head, tail))
+		if (!pipe_empty(idx.head, idx.tail))
 			mask |= EPOLLIN | EPOLLRDNORM;
 		if (!pipe->writers && filp->f_pipe != pipe->w_counter)
 			mask |= EPOLLHUP;
 	}
 
 	if (filp->f_mode & FMODE_WRITE) {
-		if (!pipe_full(head, tail, pipe->max_usage))
+		if (!pipe_full(idx.head, idx.tail, pipe->max_usage))
 			mask |= EPOLLOUT | EPOLLWRNORM;
 		/*
 		 * Most Unices do not set EPOLLERR for FIFOs but on Linux they
include/linux/pipe_fs_i.h +37 -2
index 8ff23bf5a819..b1a3b99f9ff8 100644
@@ -31,6 +31,33 @@ struct pipe_buffer {
 	unsigned long private;
 };
 
+/*
+ * Really only alpha needs 32-bit fields, but
+ * might as well do it for 64-bit architectures
+ * since that's what we've historically done,
+ * and it makes 'head_tail' always be a simple
+ * 'unsigned long'.
+ */
+#ifdef CONFIG_64BIT
+  typedef unsigned int pipe_index_t;
+#else
+  typedef unsigned short pipe_index_t;
+#endif
+
+/*
+ * We have to declare this outside 'struct pipe_inode_info',
+ * but then we can't use 'union pipe_index' for an anonymous
+ * union, so we end up having to duplicate this declaration
+ * below. Annoying.
+ */
+union pipe_index {
+	unsigned long head_tail;
+	struct {
+		pipe_index_t head;
+		pipe_index_t tail;
+	};
+};
+
 /**
  *	struct pipe_inode_info - a linux kernel pipe
  *	@mutex: mutex protecting the whole thing
@@ -58,8 +85,16 @@ struct pipe_buffer {
 struct pipe_inode_info {
 	struct mutex mutex;
 	wait_queue_head_t rd_wait, wr_wait;
-	unsigned int head;
-	unsigned int tail;
+
+	/* This has to match the 'union pipe_index' above */
+	union {
+		unsigned long head_tail;
+		struct {
+			pipe_index_t head;
+			pipe_index_t tail;
+		};
+	};
+
 	unsigned int max_usage;
 	unsigned int ring_size;
 	unsigned int nr_accounted;
#78 lore raw
In reply to Sapkal, Swapnil

On Mon, 3 Mar 2025 15:16:34 +0530 "Sapkal, Swapnil" <[email protected]>

Quoted 61 lines · Oleg Nesterov
Quoted 2 lines

And... I know, I know you already hate me ;)

Not at all :)

Quoted 3 lines

but if you have time, could you check if this patch (with or without the previous debugging patch) makes any difference? Just to be sure.

Sure, I will give this a try.

But in the meanwhile me and Prateek tried some of the experiments in the weekend. We were able to reproduce this issue on a third generation EPYC system as well as on an Intel Emerald Rapids (2 X INTEL(R) XEON(R) PLATINUM 8592+).

We tried heavy hammered tracing approach over the weekend on top of your debug patch. I have attached the debug patch below. With tracing we found the following case for pipe_writable():

   hackbench-118768  [206] .....  1029.550601: pipe_write: 000000005eea28ff: 0: 37 38 16: 1

Here,

head = 37
tail = 38
max_usage = 16
pipe_full() returns 1.

Between reading of head and later the tail, the tail seems to have moved ahead of the head leading to wraparound. Applying the following changes I have not yet run into a hang on the original machine where I first saw it:

fs/pipe.c +12 -3
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
  /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
  static inline bool pipe_writable(const struct pipe_inode_info *pipe)
  {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
  	unsigned int max_usage = READ_ONCE(pipe->max_usage);
+	unsigned int head, tail;
+
+	tail = READ_ONCE(pipe->tail);
+	/*
+	 * Since the unsigned arithmetic in this lockless preemptible context
+	 * relies on the fact that the tail can never be ahead of head, read
+	 * the head after the tail to ensure we've not missed any updates to
+	 * the head. Reordering the reads can cause wraparounds and give the
+	 * illusion that the pipe is full.
+	 */
+	smp_rmb();
+	head = READ_ONCE(pipe->head);
 
  	return !pipe_full(head, tail, max_usage) ||
  		!READ_ONCE(pipe->readers);
---
Quoted 3 lines

smp_rmb() on x86 is a nop and even without the barrier we were not able to reproduce the hang even after 10000 iterations.

My $.02 that changes the wait condition.
Not sure it makes sense for you.

fs/pipe.c +4 -2
@@ -430,7 +430,7 @@ pipe_write(struct kiocb *iocb, struct io
 {
 	struct file *filp = iocb->ki_filp;
 	struct pipe_inode_info *pipe = filp->private_data;
-	unsigned int head;
+	unsigned int head, tail;
 	ssize_t ret = 0;
 	size_t total_len = iov_iter_count(from);
 	ssize_t chars;
@@ -573,11 +573,13 @@ pipe_write(struct kiocb *iocb, struct io
 		 * after waiting we need to re-check whether the pipe
 		 * become empty while we dropped the lock.
 		 */
+		tail = pipe->tail;
 		mutex_unlock(&pipe->mutex);
 		if (was_empty)
 			wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
 		kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
-		wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
+		wait_event_interruptible_exclusive(pipe->wr_wait,
+				!READ_ONCE(pipe->readers) || tail != READ_ONCE(pipe->tail));
 		mutex_lock(&pipe->mutex);
 		was_empty = pipe_empty(pipe->head, pipe->tail);
 		wake_next_writer = true;

--

#76 lore raw
In reply to K Prateek Nayak

Hello Mateusz,

Quoted 5 lines · K Prateek Nayak

Hello Mateusz,

Quoted 2 lines · Mateusz Guzik

Can you guys try out the patch below?

Quoted 8 lines
Quoted 5 lines

It changes things up so that there is no need to read 2 different vars.

It is not the final version and I don't claim to be able to fully justify the thing at the moment either, but I would like to know if it fixes the problem.

Happy to help! We've queued the below patch for an overnight run, will report back once it is done.

Hackbench has been running for a few thousand iteration now without experiencing any hangs yet with your changes.

Quoted 11 lines

Full disclaimer: We're testing on top of commit aaec5a95d596 ("pipe_read: don't wake up the writer if the pipe is still full") where the issue is more reproducible. I've replaced the VFS_BUG_ON() with a plain BUG_ON() based on [1] since v6.14-rc1 did not include the CONFIG_DEBUG_VFS bits. Hope that is alright.

[1] https://lore.kernel.org/lkml/[email protected]/

/off to get some shut eyes/

Signature · 2 lines
Thanks and Regards,
Prateek
#45 lore raw
In reply to Linus Torvalds

Hello Linus,

Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Oleg Nesterov

Stupid question... but do we really need to change the code which update tail/head if we pack them into a single word?

Quoted 7 lines

No. It's only the READ_ONCE() parts that need changing.

See this suggested patch, which does something very similar to what you were thinking of.

ENTIRELY UNTESTED, but it seems to generate ok code. It might even generate better code than what we have now.

With the patch on top of commit aaec5a95d596 ("pipe_read: don't wake up the writer if the pipe is still full"), we've not seen any hangs yet with a few thousand iterations of short loops, and a few hundred iterations of larger loop sizes with hackbench.

If you can provide you S-o-b, we can send out an official patch with a commit log. We'll wait for Oleg's response in case he has any concerns.

Quoted 2 lines

Linus

Signature · 2 lines
Thanks and Regards,
Prateek
#79 lore raw
In reply to Hillf Danton

Hello Hillf,

Quoted 81 lines · Hillf Danton

On Mon, 3 Mar 2025 15:16:34 +0530 "Sapkal, Swapnil" <[email protected]>

Quoted 61 lines · Oleg Nesterov
Quoted 2 lines

And... I know, I know you already hate me ;)

Not at all :)

Quoted 3 lines

but if you have time, could you check if this patch (with or without the previous debugging patch) makes any difference? Just to be sure.

Sure, I will give this a try.

But in the meanwhile me and Prateek tried some of the experiments in the weekend. We were able to reproduce this issue on a third generation EPYC system as well as on an Intel Emerald Rapids (2 X INTEL(R) XEON(R) PLATINUM 8592+).

We tried heavy hammered tracing approach over the weekend on top of your debug patch. I have attached the debug patch below. With tracing we found the following case for pipe_writable():

    hackbench-118768  [206] .....  1029.550601: pipe_write: 000000005eea28ff: 0: 37 38 16: 1

Here,

head = 37
tail = 38
max_usage = 16
pipe_full() returns 1.

Between reading of head and later the tail, the tail seems to have moved ahead of the head leading to wraparound. Applying the following changes I have not yet run into a hang on the original machine where I first saw it:

fs/pipe.c +12 -3
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
   /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
   static inline bool pipe_writable(const struct pipe_inode_info *pipe)
   {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
   	unsigned int max_usage = READ_ONCE(pipe->max_usage);
+	unsigned int head, tail;
+
+	tail = READ_ONCE(pipe->tail);
+	/*
+	 * Since the unsigned arithmetic in this lockless preemptible context
+	 * relies on the fact that the tail can never be ahead of head, read
+	 * the head after the tail to ensure we've not missed any updates to
+	 * the head. Reordering the reads can cause wraparounds and give the
+	 * illusion that the pipe is full.
+	 */
+	smp_rmb();
+	head = READ_ONCE(pipe->head);
 
   	return !pipe_full(head, tail, max_usage) ||
   		!READ_ONCE(pipe->readers);
---
Quoted 3 lines

smp_rmb() on x86 is a nop and even without the barrier we were not able to reproduce the hang even after 10000 iterations.

My $.02 that changes the wait condition.
Not sure it makes sense for you.

fs/pipe.c +1 -1
@@ -430,7 +430,7 @@ pipe_write(struct kiocb *iocb, struct io
  {
  	struct file *filp = iocb->ki_filp;
  	struct pipe_inode_info *pipe = filp->private_data;
-	unsigned int head;
+	unsigned int head, tail;
  	ssize_t ret = 0;
  	size_t total_len = iov_iter_count(from);
  	ssize_t chars;
@@ -573,11 +573,13 @@ pipe_write(struct kiocb *iocb, struct io
  		 * after waiting we need to re-check whether the pipe

fs/pipe.c pipe_write(struct kiocb *iocb, struct io · line 582

Quoted 10 lines
fs/pipe.c +3 -1
  		 * become empty while we dropped the lock.
  		 */
+		tail = pipe->tail;
  		mutex_unlock(&pipe->mutex);
  		if (was_empty)
  			wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
  		kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
-		wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
+		wait_event_interruptible_exclusive(pipe->wr_wait,
+				!READ_ONCE(pipe->readers) || tail != READ_ONCE(pipe->tail));

That could work too for the case highlighted but in case the head too has moved by the time the writer wakes up, it'll lead to an extra wakeup.

Linus' diff seems cleaner and seems to cover all racy scenarios.

Quoted 4 lines
fs/pipe.c +0 -1
  		mutex_lock(&pipe->mutex);
  		was_empty = pipe_empty(pipe->head, pipe->tail);
  		wake_next_writer = true;
--
Signature · 2 lines
Thanks and Regards,
Prateek
#46 lore raw
In reply to K Prateek Nayak
Quoted 1 line · K Prateek Nayak

]> > ENTIRELY UNTESTED, but it seems to generate ok code. It might even

Quoted 9 lines
Quoted 1 line

generate better code than what we have now.

With the patch on top of commit aaec5a95d596 ("pipe_read: don't wake up the writer if the pipe is still full"), we've not seen any hangs yet with a few thousand iterations of short loops, and a few hundred iterations of larger loop sizes with hackbench.

If you can provide you S-o-b, we can send out an official patch with a commit log. We'll wait for Oleg's response in case he has any concerns.

Ack. With that testing background, please write a message and add my

and we'll get this all fixed up.

I assume this all goes back to commit 8cefc107ca54 ("pipe: Use head and tail pointers for the ring, not cursor and length") back in 2019.

Or possibly 85190d15f4ea ("pipe: don't use 'pipe_wait() for basic pipe IO")?

But it was all hidden by the fact that we used to just wake things up very aggressively and you'd never notice the race as a result, so then it got exposed by the more minimal wakeup changes.

Linus

#80 lore raw
In reply to K Prateek Nayak

On Tue, 4 Mar 2025 11:05:57 +0530 K Prateek Nayak <[email protected]>

Quoted 88 lines · Hillf Danton
Quoted 87 lines

On Mon, 3 Mar 2025 15:16:34 +0530 "Sapkal, Swapnil" <[email protected]>

Quoted 61 lines · Oleg Nesterov
Quoted 2 lines

And... I know, I know you already hate me ;)

Not at all :)

Quoted 3 lines

but if you have time, could you check if this patch (with or without the previous debugging patch) makes any difference? Just to be sure.

Sure, I will give this a try.

But in the meanwhile me and Prateek tried some of the experiments in the weekend. We were able to reproduce this issue on a third generation EPYC system as well as on an Intel Emerald Rapids (2 X INTEL(R) XEON(R) PLATINUM 8592+).

We tried heavy hammered tracing approach over the weekend on top of your debug patch. I have attached the debug patch below. With tracing we found the following case for pipe_writable():

    hackbench-118768  [206] .....  1029.550601: pipe_write: 000000005eea28ff: 0: 37 38 16: 1

Here,

head = 37
tail = 38
max_usage = 16
pipe_full() returns 1.

Between reading of head and later the tail, the tail seems to have moved ahead of the head leading to wraparound. Applying the following changes I have not yet run into a hang on the original machine where I first saw it:

fs/pipe.c +12 -3
index ce1af7592780..a1931c817822 100644
@@ -417,9 +417,19 @@ static inline int is_packetized(struct file *file)
   /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
   static inline bool pipe_writable(const struct pipe_inode_info *pipe)
   {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
   	unsigned int max_usage = READ_ONCE(pipe->max_usage);
+	unsigned int head, tail;
+
+	tail = READ_ONCE(pipe->tail);
+	/*
+	 * Since the unsigned arithmetic in this lockless preemptible context
+	 * relies on the fact that the tail can never be ahead of head, read
+	 * the head after the tail to ensure we've not missed any updates to
+	 * the head. Reordering the reads can cause wraparounds and give the
+	 * illusion that the pipe is full.
+	 */
+	smp_rmb();
+	head = READ_ONCE(pipe->head);
 
   	return !pipe_full(head, tail, max_usage) ||
   		!READ_ONCE(pipe->readers);
---
Quoted 3 lines

smp_rmb() on x86 is a nop and even without the barrier we were not able to reproduce the hang even after 10000 iterations.

My $.02 that changes the wait condition.
Not sure it makes sense for you.

fs/pipe.c +2 -1
@@ -430,7 +430,7 @@ pipe_write(struct kiocb *iocb, struct io
  {
  	struct file *filp = iocb->ki_filp;
  	struct pipe_inode_info *pipe = filp->private_data;
-	unsigned int head;
+	unsigned int head, tail;
  	ssize_t ret = 0;
  	size_t total_len = iov_iter_count(from);
  	ssize_t chars;
@@ -573,11 +573,13 @@ pipe_write(struct kiocb *iocb, struct io
  		 * after waiting we need to re-check whether the pipe
  		 * become empty while we dropped the lock.
  		 */
+		tail = pipe->tail;
  		mutex_unlock(&pipe->mutex);
  		if (was_empty)
  			wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
Quoted 9 lines

fs/pipe.c pipe_write(struct kiocb *iocb, struct io · line 582

Quoted 4 lines
fs/pipe.c +2 -1
  		kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
-		wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
+		wait_event_interruptible_exclusive(pipe->wr_wait,
+				!READ_ONCE(pipe->readers) || tail != READ_ONCE(pipe->tail));

That could work too for the case highlighted but in case the head too has moved by the time the writer wakes up, it'll lead to an extra wakeup.

Note wakeup can occur even if pipe is full, and more important, taking the pipe lock after wakeup is the price paid for curing the hang in question.

* So we still need to wake up any pending writers in the
* _very_ unlikely case that the pipe was full, but we got
* no data.
*/

Quoted 10 lines

Linus' diff seems cleaner and seems to cover all racy scenarios.

Quoted 4 lines
fs/pipe.c +0 -1
  		mutex_lock(&pipe->mutex);
  		was_empty = pipe_empty(pipe->head, pipe->tail);
  		wake_next_writer = true;
--
Signature · 2 lines
Thanks and Regards,
Prateek
#81 lore raw
Quoted 19 lines · Hillf Danton

On Tue, 4 Mar 2025 11:05:57 +0530 K Prateek Nayak <[email protected]>

Quoted 8 lines
Quoted 8 lines
fs/pipe.c +1 -0
@@ -573,11 +573,13 @@ pipe_write(struct kiocb *iocb, struct io
  		 * after waiting we need to re-check whether the pipe
  		 * become empty while we dropped the lock.
  		 */
+		tail = pipe->tail;
  		mutex_unlock(&pipe->mutex);
  		if (was_empty)
  			wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
Quoted 9 lines

fs/pipe.c pipe_write(struct kiocb *iocb, struct io · line 582

Quoted 4 lines
fs/pipe.c +2 -1
  		kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
-		wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
+		wait_event_interruptible_exclusive(pipe->wr_wait,
+				!READ_ONCE(pipe->readers) || tail != READ_ONCE(pipe->tail));

That could work too for the case highlighted but in case the head too has moved by the time the writer wakes up, it'll lead to an extra wakeup.

Quoted 1 line

Note wakeup can occur even if pipe is full,

Perhaps I misunderstood you, but I don't think pipe_read() can ever do wake_up(pipe->wr_wait) if pipe is full...

Quoted 4 lines

* So we still need to wake up any pending writers in the
* _very_ unlikely case that the pipe was full, but we got
* no data.
*/

Only if wake_writer is true,

		if (unlikely(wake_writer))
			wake_up_interruptible_sync_poll(...);

and in this case the pipe is no longer full. A zero-sized buffer was removed.

Of course this pipe can be full again when the woken writer checks the condition, but this is another story. And in this case, with your proposed change, the woken writer will take pipe->mutex for no reason.

Note also that the comment and code above was already removed by
https://lore.kernel.org/all/[email protected]/

Oleg.

author Reviewed-by
#47 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

ENTIRELY UNTESTED, but it seems to generate ok code. It might even generate better code than what we have now.

but I have another question...

fs/pipe.c static const struct pipe_buf_operations anon_pipe_buf_ops = { · line 213

Quoted 5 lines

static inline bool pipe_readable(const struct pipe_inode_info *pipe) {

fs/pipe.c +1 -2
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
+	union pipe_index idx = { READ_ONCE(pipe->head_tail) };

I thought this is wrong, but then I noticed that in your version ->head_tail is the 1st member in this union.

Still perhaps

	union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) };

will look more clear?

include/linux/pipe_fs_i.h struct pipe_buffer { · line 45

Quoted 12 lines
include/linux/pipe_fs_i.h +12 -0
+/*
+ * Really only alpha needs 32-bit fields, but
+ * might as well do it for 64-bit architectures
+ * since that's what we've historically done,
+ * and it makes 'head_tail' always be a simple
+ * 'unsigned long'.
+ */
+#ifdef CONFIG_64BIT
+  typedef unsigned int pipe_index_t;
+#else
+  typedef unsigned short pipe_index_t;
+#endif

I am just curious, why we can't use "unsigned short" unconditionally and avoid #ifdef ?

Is "unsigned int" more efficient on 64-bit?

Oleg.

#48 lore raw
Quoted 19 lines · Oleg Nesterov

include/linux/pipe_fs_i.h struct pipe_buffer { · line 45

Quoted 13 lines · Linus Torvalds
include/linux/pipe_fs_i.h +12 -0
+/*
+ * Really only alpha needs 32-bit fields, but
+ * might as well do it for 64-bit architectures
+ * since that's what we've historically done,
+ * and it makes 'head_tail' always be a simple
+ * 'unsigned long'.
+ */
+#ifdef CONFIG_64BIT
+  typedef unsigned int pipe_index_t;
+#else
+  typedef unsigned short pipe_index_t;
+#endif

I am just curious, why we can't use "unsigned short" unconditionally and avoid #ifdef ?

Quoted 1 line

Is "unsigned int" more efficient on 64-bit?

Ah, I guess I misread the comment...

So, the problem is that 64-bit alpha can't write u16 "atomically" ?

Oleg.

#53 lore raw

PATCH 1/1 fs/pipe: Read pipe->{head,tail} atomically outside pipe->mutex

In reply to Linus Torvalds

From: Linus Torvalds <[email protected]>

pipe_readable(), pipe_writable(), and pipe_poll() can read "pipe->head" and "pipe->tail" outside of "pipe->mutex" critical section. When the head and the tail are read individually in that order, there is a window for interruption between the two reads in which both the head and the tail can be updated by concurrent readers and writers.

One of the problematic scenarios observed with hackbench running multiple groups on a large server on a particular pipe inode is as follows:

    pipe->head = 36
    pipe->tail = 36

    hackbench-118762  [057] .....  1029.550548: pipe_write: *wakes up: pipe not full*
    hackbench-118762  [057] .....  1029.550548: pipe_write: head: 36 -> 37 [tail: 36]
    hackbench-118762  [057] .....  1029.550548: pipe_write: *wake up next reader 118740*
    hackbench-118762  [057] .....  1029.550548: pipe_write: *wake up next writer 118768*

    hackbench-118768  [206] .....  1029.55055X: pipe_write: *writer wakes up*
    hackbench-118768  [206] .....  1029.55055X: pipe_write: head = READ_ONCE(pipe->head) [37]
    ... CPU 206 interrupted (exact wakeup was not traced but 118768 did read head at 37 in traces)

    hackbench-118740  [057] .....  1029.550558: pipe_read:  *reader wakes up: pipe is not empty*
    hackbench-118740  [057] .....  1029.550558: pipe_read:  tail: 36 -> 37 [head = 37]
    hackbench-118740  [057] .....  1029.550559: pipe_read:  *pipe is empty; wakeup writer 118768*
    hackbench-118740  [057] .....  1029.550559: pipe_read:  *sleeps*

    hackbench-118766  [185] .....  1029.550592: pipe_write: *New writer comes in*
    hackbench-118766  [185] .....  1029.550592: pipe_write: head: 37 -> 38 [tail: 37]
    hackbench-118766  [185] .....  1029.550592: pipe_write: *wakes up reader 118766*

    hackbench-118740  [185] .....  1029.550598: pipe_read:  *reader wakes up; pipe not empty*
    hackbench-118740  [185] .....  1029.550599: pipe_read:  tail: 37 -> 38 [head: 38]
    hackbench-118740  [185] .....  1029.550599: pipe_read:  *pipe is empty*
    hackbench-118740  [185] .....  1029.550599: pipe_read:  *reader sleeps; wakeup writer 118768*

    ... CPU 206 switches back to writer
    hackbench-118768  [206] .....  1029.550601: pipe_write: tail = READ_ONCE(pipe->tail) [38]
    hackbench-118768  [206] .....  1029.550601: pipe_write: pipe_full()? (u32)(37 - 38) >= 16? Yes
    hackbench-118768  [206] .....  1029.550601: pipe_write: *writer goes back to sleep*

    [ Tasks 118740 and 118768 can then indefinitely wait on each other. ]

The unsigned arithmetic in pipe_occupancy() wraps around when "pipe->tail > pipe->head" leading to pipe_full() returning true despite the pipe being empty.

The case of genuine wraparound of "pipe->head" is handled since pipe buffer has data allowing readers to make progress until the pipe->tail wraps too after which the reader will wakeup a sleeping writer, however, mistaking the pipe to be full when it is in fact empty can lead to readers and writers waiting on each other indefinitely.

This issue became more problematic and surfaced as a hang in hackbench after the optimization in commit aaec5a95d596 ("pipe_read: don't wake up the writer if the pipe is still full") significantly reduced the number of spurious wakeups of writers that had previously helped mask the issue.

To avoid missing any updates between the reads of "pipe->head" and "pipe->write", unionize the two with a single unsigned long "pipe->head_tail" member that can be loaded atomically.

Using "pipe->head_tail" to read the head and the tail ensures the lockless checks do not miss any updates to the head or the tail and since those two are only updated under "pipe->mutex", it ensures that the head is always ahead of, or equal to the tail resulting in correct calculations.

[ prateek: commit log, testing on x86 platforms. ]

---
Changes are based on:

git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs-6.15.pipe

at commit commit ee5eda8ea595 ("pipe: change pipe_write() to never add a zero-sized buffer") but also applies cleanly on top of v6.14-rc5.

The diff from Linus is kept as is except for removing the whitespaces in front of the typedef that checkpatch complained about (the warning on usage of typedef itself has been ignored since I could not think of a better alternative other than #ifdef hackery in pipe_inode_info and the newly introduced pipe_index union.) and the suggestion from Oleg to explicitly initialize the "head_tail" with:

    union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) }

I went with commit 8cefc107ca54 ("pipe: Use head and tail pointers for the ring, not cursor and length") for the "Fixes:" tag since pipe_poll() added:

    unsigned int head = READ_ONCE(pipe->head);
    unsigned int tail = READ_ONCE(pipe->tail);

    poll_wait(filp, &pipe->wait, wait);

    BUG_ON(pipe_occupancy(head, tail) > pipe->ring_size);

and the race described can trigger that BUG_ON() but as Linus pointed out in [1] the commit 85190d15f4ea ("pipe: don't use 'pipe_wait() for basic pipe IO") is probably the one that can cause the writers to sleep on empty pipe since the pipe_wait() used prior did not drop the pipe lock until it called schedule() and prepare_to_wait() was called before pipe_unlock() ensuring no races.

[1] https://lore.kernel.org/all/CAHk-=wh804HX8H86VRUSKoJGVG0eBe8sPz8=E3d8LHftOCSqwQ@mail.gmail.com/

Please let me know if the patch requires any modifications and I'll jump right on it. The changes have been tested on both a 5th Generation AMD EPYC system and on a dual socket Intel Emerald Rapids system with multiple thousand iterations of hackbench for small and large loop counts. Thanks a ton to Swapnil for all the help.

---
 fs/pipe.c                 | 19 ++++++++-----------
 include/linux/pipe_fs_i.h | 39 +++++++++++++++++++++++++++++++++++++--
 2 files changed, 45 insertions(+), 13 deletions(-)
fs/pipe.c +8 -11
index b0641f75b1ba..780990f307ab 100644
@@ -210,11 +210,10 @@ static const struct pipe_buf_operations anon_pipe_buf_ops = {
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_readable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
+	union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) };
 	unsigned int writers = READ_ONCE(pipe->writers);
 
-	return !pipe_empty(head, tail) || !writers;
+	return !pipe_empty(idx.head, idx.tail) || !writers;
 }
 
 static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
@@ -403,11 +402,10 @@ static inline int is_packetized(struct file *file)
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_writable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
+	union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) };
 	unsigned int max_usage = READ_ONCE(pipe->max_usage);
 
-	return !pipe_full(head, tail, max_usage) ||
+	return !pipe_full(idx.head, idx.tail, max_usage) ||
 		!READ_ONCE(pipe->readers);
 }
 
@@ -649,7 +647,7 @@ pipe_poll(struct file *filp, poll_table *wait)
 {
 	__poll_t mask;
 	struct pipe_inode_info *pipe = filp->private_data;
-	unsigned int head, tail;
+	union pipe_index idx;
 
 	/* Epoll has some historical nasty semantics, this enables them */
 	WRITE_ONCE(pipe->poll_usage, true);
@@ -670,19 +668,18 @@ pipe_poll(struct file *filp, poll_table *wait)
 	 * if something changes and you got it wrong, the poll
 	 * table entry will wake you up and fix it.
 	 */
-	head = READ_ONCE(pipe->head);
-	tail = READ_ONCE(pipe->tail);
+	idx.head_tail = READ_ONCE(pipe->head_tail);
 
 	mask = 0;
 	if (filp->f_mode & FMODE_READ) {
-		if (!pipe_empty(head, tail))
+		if (!pipe_empty(idx.head, idx.tail))
 			mask |= EPOLLIN | EPOLLRDNORM;
 		if (!pipe->writers && filp->f_pipe != pipe->w_counter)
 			mask |= EPOLLHUP;
 	}
 
 	if (filp->f_mode & FMODE_WRITE) {
-		if (!pipe_full(head, tail, pipe->max_usage))
+		if (!pipe_full(idx.head, idx.tail, pipe->max_usage))
 			mask |= EPOLLOUT | EPOLLWRNORM;
 		/*
 		 * Most Unices do not set EPOLLERR for FIFOs but on Linux they
include/linux/pipe_fs_i.h +37 -2
index 8ff23bf5a819..3cc4f8eab853 100644
@@ -31,6 +31,33 @@ struct pipe_buffer {
 	unsigned long private;
 };
 
+/*
+ * Really only alpha needs 32-bit fields, but
+ * might as well do it for 64-bit architectures
+ * since that's what we've historically done,
+ * and it makes 'head_tail' always be a simple
+ * 'unsigned long'.
+ */
+#ifdef CONFIG_64BIT
+typedef unsigned int pipe_index_t;
+#else
+typedef unsigned short pipe_index_t;
+#endif
+
+/*
+ * We have to declare this outside 'struct pipe_inode_info',
+ * but then we can't use 'union pipe_index' for an anonymous
+ * union, so we end up having to duplicate this declaration
+ * below. Annoying.
+ */
+union pipe_index {
+	unsigned long head_tail;
+	struct {
+		pipe_index_t head;
+		pipe_index_t tail;
+	};
+};
+
 /**
  *	struct pipe_inode_info - a linux kernel pipe
  *	@mutex: mutex protecting the whole thing
@@ -58,8 +85,16 @@ struct pipe_buffer {
 struct pipe_inode_info {
 	struct mutex mutex;
 	wait_queue_head_t rd_wait, wr_wait;
-	unsigned int head;
-	unsigned int tail;
+
+	/* This has to match the 'union pipe_index' above */
+	union {
+		unsigned long head_tail;
+		struct {
+			pipe_index_t head;
+			pipe_index_t tail;
+		};
+	};
+
 	unsigned int max_usage;
 	unsigned int ring_size;
 	unsigned int nr_accounted;

base-commit: ee5eda8ea59546af2e8f192c060fbf29862d7cbd

Signature · 1 line
2.34.1
#49 lore raw
In reply to Oleg Nesterov
Quoted 3 lines · Oleg Nesterov

I thought this is wrong, but then I noticed that in your version ->head_tail is the 1st member in this union.

Yes. That was intentional, to make the code have much less extraneous noise.

The only reason to ever use that "union pipe_index" is for this whole "one word for everything", so I feel that making it compact is actually more legible than adding extra markers.

Quoted 15 lines

include/linux/pipe_fs_i.h struct pipe_buffer { · line 45

Quoted 11 lines

+ * Really only alpha needs 32-bit fields, but
+ * might as well do it for 64-bit architectures
+ * since that's what we've historically done,
+ * and it makes 'head_tail' always be a simple
+ * 'unsigned long'.
+ */

include/linux/pipe_fs_i.h +5 -0
+#ifdef CONFIG_64BIT
+  typedef unsigned int pipe_index_t;
+#else
+  typedef unsigned short pipe_index_t;
+#endif

I am just curious, why we can't use "unsigned short" unconditionally and avoid #ifdef ?

Quoted 1 line

Is "unsigned int" more efficient on 64-bit?

The main reason is that a "unsigned short" write on alpha isn't atomic
- it's a read-modify-write operation, and so it isn't safe to mix

        spin_lock_irq(&pipe->rd_wait.lock);
         ...
        pipe->tail = ++tail;
        ...
        spin_unlock_irq(&pipe->rd_wait.lock);

with

         mutex_lock(&pipe->mutex);
         ...
         pipe->head = head + 1;
         ...
         mutex_unlock(&pipe->mutex);

because while they are two different fields using two different
locks, on alpha the above only works if they are in separate words (because updating one will do a read-and-write-back of the other).

This is a fundamental alpha architecture bug. I was actually quite ready to just kill off alpha support entirely, because it's a dead architecture that is unfixably broken. But there's some crazy patches to make gcc generate horrific atomic code to make this all work on alpha by Maciej Rozycki, so one day we'll be in the situation that alpha can be considered "fixed", but we're not there yet.

Do we really care? Maybe not. The race is probably very hard to hit, so with the two remaining alpha users, we could just say "let's just make it use 16-bit ops".

But even on x86, 32-bit ops potentially generate just slightly better code due to lack of some prefix bytes.

And those fields *used* to be 32-bit, so my patch basically kept the status quo on 64-bit machines (and just turned it into 16-bit fields on 32-bit architectures).

Anyway, I wouldn't object to just unconditionally making it "two 16-bit indexes make a 32-bit head_tail" if it actually makes the structure smaller. It might not even matter on 64-bit because of alignment of fields around it - I didn't check. As mentioned, it was more of a combination of "alpha" plus "no change to relevant other architectures".

Linus

#54 lore raw

Re: [PATCH] fs/pipe: Read pipe->{head,tail} atomically outside pipe->mutex

In reply to K Prateek Nayak
Quoted 73 lines · K Prateek Nayak

From: Linus Torvalds <[email protected]>

pipe_readable(), pipe_writable(), and pipe_poll() can read "pipe->head" and "pipe->tail" outside of "pipe->mutex" critical section. When the head and the tail are read individually in that order, there is a window for interruption between the two reads in which both the head and the tail can be updated by concurrent readers and writers.

One of the problematic scenarios observed with hackbench running multiple groups on a large server on a particular pipe inode is as follows:

    pipe->head = 36
    pipe->tail = 36

    hackbench-118762  [057] .....  1029.550548: pipe_write: *wakes up: pipe not full*
    hackbench-118762  [057] .....  1029.550548: pipe_write: head: 36 -> 37 [tail: 36]
    hackbench-118762  [057] .....  1029.550548: pipe_write: *wake up next reader 118740*
    hackbench-118762  [057] .....  1029.550548: pipe_write: *wake up next writer 118768*

    hackbench-118768  [206] .....  1029.55055X: pipe_write: *writer wakes up*
    hackbench-118768  [206] .....  1029.55055X: pipe_write: head = READ_ONCE(pipe->head) [37]
    ... CPU 206 interrupted (exact wakeup was not traced but 118768 did read head at 37 in traces)

    hackbench-118740  [057] .....  1029.550558: pipe_read:  *reader wakes up: pipe is not empty*
    hackbench-118740  [057] .....  1029.550558: pipe_read:  tail: 36 -> 37 [head = 37]
    hackbench-118740  [057] .....  1029.550559: pipe_read:  *pipe is empty; wakeup writer 118768*
    hackbench-118740  [057] .....  1029.550559: pipe_read:  *sleeps*

    hackbench-118766  [185] .....  1029.550592: pipe_write: *New writer comes in*
    hackbench-118766  [185] .....  1029.550592: pipe_write: head: 37 -> 38 [tail: 37]
    hackbench-118766  [185] .....  1029.550592: pipe_write: *wakes up reader 118766*

    hackbench-118740  [185] .....  1029.550598: pipe_read:  *reader wakes up; pipe not empty*
    hackbench-118740  [185] .....  1029.550599: pipe_read:  tail: 37 -> 38 [head: 38]
    hackbench-118740  [185] .....  1029.550599: pipe_read:  *pipe is empty*
    hackbench-118740  [185] .....  1029.550599: pipe_read:  *reader sleeps; wakeup writer 118768*

    ... CPU 206 switches back to writer
    hackbench-118768  [206] .....  1029.550601: pipe_write: tail = READ_ONCE(pipe->tail) [38]
    hackbench-118768  [206] .....  1029.550601: pipe_write: pipe_full()? (u32)(37 - 38) >= 16? Yes
    hackbench-118768  [206] .....  1029.550601: pipe_write: *writer goes back to sleep*

    [ Tasks 118740 and 118768 can then indefinitely wait on each other. ]

The unsigned arithmetic in pipe_occupancy() wraps around when "pipe->tail > pipe->head" leading to pipe_full() returning true despite the pipe being empty.

The case of genuine wraparound of "pipe->head" is handled since pipe buffer has data allowing readers to make progress until the pipe->tail wraps too after which the reader will wakeup a sleeping writer, however, mistaking the pipe to be full when it is in fact empty can lead to readers and writers waiting on each other indefinitely.

This issue became more problematic and surfaced as a hang in hackbench after the optimization in commit aaec5a95d596 ("pipe_read: don't wake up the writer if the pipe is still full") significantly reduced the number of spurious wakeups of writers that had previously helped mask the issue.

To avoid missing any updates between the reads of "pipe->head" and "pipe->write", unionize the two with a single unsigned long "pipe->head_tail" member that can be loaded atomically.

Using "pipe->head_tail" to read the head and the tail ensures the lockless checks do not miss any updates to the head or the tail and since those two are only updated under "pipe->mutex", it ensures that the head is always ahead of, or equal to the tail resulting in correct calculations.

[ prateek: commit log, testing on x86 platforms. ]

Quoted 9 lines

With this patch, I'm also not reproducing the problem. Thanks!

Quoted 173 lines

---
Changes are based on:

git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs-6.15.pipe

at commit commit ee5eda8ea595 ("pipe: change pipe_write() to never add a zero-sized buffer") but also applies cleanly on top of v6.14-rc5.

The diff from Linus is kept as is except for removing the whitespaces in front of the typedef that checkpatch complained about (the warning on usage of typedef itself has been ignored since I could not think of a better alternative other than #ifdef hackery in pipe_inode_info and the newly introduced pipe_index union.) and the suggestion from Oleg to explicitly initialize the "head_tail" with:

    union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) }

I went with commit 8cefc107ca54 ("pipe: Use head and tail pointers for the ring, not cursor and length") for the "Fixes:" tag since pipe_poll() added:

    unsigned int head = READ_ONCE(pipe->head);
    unsigned int tail = READ_ONCE(pipe->tail);

    poll_wait(filp, &pipe->wait, wait);

    BUG_ON(pipe_occupancy(head, tail) > pipe->ring_size);

and the race described can trigger that BUG_ON() but as Linus pointed out in [1] the commit 85190d15f4ea ("pipe: don't use 'pipe_wait() for basic pipe IO") is probably the one that can cause the writers to sleep on empty pipe since the pipe_wait() used prior did not drop the pipe lock until it called schedule() and prepare_to_wait() was called before pipe_unlock() ensuring no races.

[1] https://lore.kernel.org/all/CAHk-=wh804HX8H86VRUSKoJGVG0eBe8sPz8=E3d8LHftOCSqwQ@mail.gmail.com/

Please let me know if the patch requires any modifications and I'll jump right on it. The changes have been tested on both a 5th Generation AMD EPYC system and on a dual socket Intel Emerald Rapids system with multiple thousand iterations of hackbench for small and large loop counts. Thanks a ton to Swapnil for all the help.

---
 fs/pipe.c                 | 19 ++++++++-----------
 include/linux/pipe_fs_i.h | 39 +++++++++++++++++++++++++++++++++++++--
 2 files changed, 45 insertions(+), 13 deletions(-)
fs/pipe.c +8 -11
index b0641f75b1ba..780990f307ab 100644
@@ -210,11 +210,10 @@ static const struct pipe_buf_operations anon_pipe_buf_ops = {
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_readable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
+	union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) };
 	unsigned int writers = READ_ONCE(pipe->writers);
 
-	return !pipe_empty(head, tail) || !writers;
+	return !pipe_empty(idx.head, idx.tail) || !writers;
 }
 
 static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
@@ -403,11 +402,10 @@ static inline int is_packetized(struct file *file)
 /* Done while waiting without holding the pipe lock - thus the READ_ONCE() */
 static inline bool pipe_writable(const struct pipe_inode_info *pipe)
 {
-	unsigned int head = READ_ONCE(pipe->head);
-	unsigned int tail = READ_ONCE(pipe->tail);
+	union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) };
 	unsigned int max_usage = READ_ONCE(pipe->max_usage);
 
-	return !pipe_full(head, tail, max_usage) ||
+	return !pipe_full(idx.head, idx.tail, max_usage) ||
 		!READ_ONCE(pipe->readers);
 }
 
@@ -649,7 +647,7 @@ pipe_poll(struct file *filp, poll_table *wait)
 {
 	__poll_t mask;
 	struct pipe_inode_info *pipe = filp->private_data;
-	unsigned int head, tail;
+	union pipe_index idx;
 
 	/* Epoll has some historical nasty semantics, this enables them */
 	WRITE_ONCE(pipe->poll_usage, true);
@@ -670,19 +668,18 @@ pipe_poll(struct file *filp, poll_table *wait)
 	 * if something changes and you got it wrong, the poll
 	 * table entry will wake you up and fix it.
 	 */
-	head = READ_ONCE(pipe->head);
-	tail = READ_ONCE(pipe->tail);
+	idx.head_tail = READ_ONCE(pipe->head_tail);
 
 	mask = 0;
 	if (filp->f_mode & FMODE_READ) {
-		if (!pipe_empty(head, tail))
+		if (!pipe_empty(idx.head, idx.tail))
 			mask |= EPOLLIN | EPOLLRDNORM;
 		if (!pipe->writers && filp->f_pipe != pipe->w_counter)
 			mask |= EPOLLHUP;
 	}
 
 	if (filp->f_mode & FMODE_WRITE) {
-		if (!pipe_full(head, tail, pipe->max_usage))
+		if (!pipe_full(idx.head, idx.tail, pipe->max_usage))
 			mask |= EPOLLOUT | EPOLLWRNORM;
 		/*
 		 * Most Unices do not set EPOLLERR for FIFOs but on Linux they
include/linux/pipe_fs_i.h +37 -2
index 8ff23bf5a819..3cc4f8eab853 100644
@@ -31,6 +31,33 @@ struct pipe_buffer {
 	unsigned long private;
 };
 
+/*
+ * Really only alpha needs 32-bit fields, but
+ * might as well do it for 64-bit architectures
+ * since that's what we've historically done,
+ * and it makes 'head_tail' always be a simple
+ * 'unsigned long'.
+ */
+#ifdef CONFIG_64BIT
+typedef unsigned int pipe_index_t;
+#else
+typedef unsigned short pipe_index_t;
+#endif
+
+/*
+ * We have to declare this outside 'struct pipe_inode_info',
+ * but then we can't use 'union pipe_index' for an anonymous
+ * union, so we end up having to duplicate this declaration
+ * below. Annoying.
+ */
+union pipe_index {
+	unsigned long head_tail;
+	struct {
+		pipe_index_t head;
+		pipe_index_t tail;
+	};
+};
+
 /**
  *	struct pipe_inode_info - a linux kernel pipe
  *	@mutex: mutex protecting the whole thing
@@ -58,8 +85,16 @@ struct pipe_buffer {
 struct pipe_inode_info {
 	struct mutex mutex;
 	wait_queue_head_t rd_wait, wr_wait;
-	unsigned int head;
-	unsigned int tail;
+
+	/* This has to match the 'union pipe_index' above */
+	union {
+		unsigned long head_tail;
+		struct {
+			pipe_index_t head;
+			pipe_index_t tail;
+		};
+	};
+
 	unsigned int max_usage;
 	unsigned int ring_size;
 	unsigned int nr_accounted;

base-commit: ee5eda8ea59546af2e8f192c060fbf29862d7cbd

Signature · 1 line
2.34.1
Signature · 1 line
Rgrds, legion
#55 lore raw

Re: [PATCH] fs/pipe: Read pipe->{head,tail} atomically outside pipe->mutex

In reply to K Prateek Nayak
Quoted 3 lines · K Prateek Nayak

pipe_readable(), pipe_writable(), and pipe_poll() can read "pipe->head" and "pipe->tail" outside of "pipe->mutex" critical section. [...]

Thanks, applied.

Linus

#50 lore raw
In reply to Linus Torvalds
Quoted 18 lines · Linus Torvalds
Quoted 16 lines · Oleg Nesterov

include/linux/pipe_fs_i.h struct pipe_buffer { · line 45

Quoted 12 lines

+ * Really only alpha needs 32-bit fields, but
+ * might as well do it for 64-bit architectures
+ * since that's what we've historically done,
+ * and it makes 'head_tail' always be a simple
+ * 'unsigned long'.
+ */

include/linux/pipe_fs_i.h +5 -0
+#ifdef CONFIG_64BIT
+  typedef unsigned int pipe_index_t;
+#else
+  typedef unsigned short pipe_index_t;
+#endif

I am just curious, why we can't use "unsigned short" unconditionally and avoid #ifdef ?

Quoted 3 lines
Quoted 1 line

Is "unsigned int" more efficient on 64-bit?

The main reason is that a "unsigned short" write on alpha isn't atomic

Yes, I have already realized this when I tried to actually read the comment, see my next email in reply to myself.

But thanks for your detailed explanation!

Oleg.

#82 lore raw
In reply to Oleg Nesterov

On Tue, 4 Mar 2025 13:34:57 +0100 Oleg Nesterov <[email protected]>

Quoted 34 lines · Hillf Danton
Quoted 18 lines

On Tue, 4 Mar 2025 11:05:57 +0530 K Prateek Nayak <[email protected]>

Quoted 8 lines
Quoted 8 lines
fs/pipe.c +1 -0
@@ -573,11 +573,13 @@ pipe_write(struct kiocb *iocb, struct io
  		 * after waiting we need to re-check whether the pipe
  		 * become empty while we dropped the lock.
  		 */
+		tail = pipe->tail;
  		mutex_unlock(&pipe->mutex);
  		if (was_empty)
  			wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
Quoted 9 lines

fs/pipe.c pipe_write(struct kiocb *iocb, struct io · line 582

Quoted 4 lines
fs/pipe.c +2 -1
  		kill_fasync(&pipe->fasync_readers, SIGIO, POLL_IN);
-		wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));
+		wait_event_interruptible_exclusive(pipe->wr_wait,
+				!READ_ONCE(pipe->readers) || tail != READ_ONCE(pipe->tail));

That could work too for the case highlighted but in case the head too has moved by the time the writer wakes up, it'll lead to an extra wakeup.

Quoted 1 line

Note wakeup can occur even if pipe is full,

Perhaps I misunderstood you, but I don't think pipe_read() can ever do wake_up(pipe->wr_wait) if pipe is full...

Quoted 4 lines

* So we still need to wake up any pending writers in the
* _very_ unlikely case that the pipe was full, but we got
* no data.
*/

Only if wake_writer is true,

		if (unlikely(wake_writer))
			wake_up_interruptible_sync_poll(...);
Quoted 7 lines

and in this case the pipe is no longer full. A zero-sized buffer was removed.

Of course this pipe can be full again when the woken writer checks the condition, but this is another story. And in this case, with your proposed change, the woken writer will take pipe->mutex for no reason.

See the following sequence,

1) waker makes full false
2) waker makes full true
3) waiter checks full
4) waker makes full false

waiter has no real idea of full without lock held, perhaps regardless the code cut below.

Quoted 4 lines

Note also that the comment and code above was already removed by
https://lore.kernel.org/all/[email protected]/

Oleg.

#83 lore raw
Quoted 25 lines · Hillf Danton

On Tue, 4 Mar 2025 13:34:57 +0100 Oleg Nesterov <[email protected]>

Quoted 16 lines
Quoted 2 lines
Quoted 1 line

Note wakeup can occur even if pipe is full,

Perhaps I misunderstood you, but I don't think pipe_read() can ever do wake_up(pipe->wr_wait) if pipe is full...

Quoted 4 lines

* So we still need to wake up any pending writers in the
* _very_ unlikely case that the pipe was full, but we got
* no data.
*/

Only if wake_writer is true,

		if (unlikely(wake_writer))
			wake_up_interruptible_sync_poll(...);
Quoted 7 lines

and in this case the pipe is no longer full. A zero-sized buffer was removed.

Of course this pipe can be full again when the woken writer checks the condition, but this is another story. And in this case, with your proposed change, the woken writer will take pipe->mutex for no reason.

Quoted 6 lines

See the following sequence,

1) waker makes full false
2) waker makes full true
3) waiter checks full
4) waker makes full false

I don't really understand this sequence, but

Quoted 2 lines

waiter has no real idea of full without lock held, perhaps regardless the code cut below.

Of course! Again, whatever the woken writer checks in pipe_writable() lockless, another writer can make pipe_full() true again.

But why do we care? Why do you think that the change you propose makes more sense than the fix from Prateek or the (already merged) Linus's fix?

Oleg.

#51 lore raw
In reply to Linus Torvalds

Hello Linus,

Quoted 28 lines · Linus Torvalds
Quoted 3 lines · Oleg Nesterov

I thought this is wrong, but then I noticed that in your version ->head_tail is the 1st member in this union.

Yes. That was intentional, to make the code have much less extraneous noise.

The only reason to ever use that "union pipe_index" is for this whole "one word for everything", so I feel that making it compact is actually more legible than adding extra markers.

Quoted 15 lines

include/linux/pipe_fs_i.h struct pipe_buffer { · line 45

Quoted 11 lines

+ * Really only alpha needs 32-bit fields, but
+ * might as well do it for 64-bit architectures
+ * since that's what we've historically done,
+ * and it makes 'head_tail' always be a simple
+ * 'unsigned long'.
+ */

include/linux/pipe_fs_i.h +5 -0
+#ifdef CONFIG_64BIT
+  typedef unsigned int pipe_index_t;
+#else
+  typedef unsigned short pipe_index_t;
+#endif

I am just curious, why we can't use "unsigned short" unconditionally and avoid #ifdef ?

Quoted 1 line

Is "unsigned int" more efficient on 64-bit?

Quoted 8 lines

The main reason is that a "unsigned short" write on alpha isn't atomic
- it's a read-modify-write operation, and so it isn't safe to mix

         spin_lock_irq(&pipe->rd_wait.lock);
          ...
         pipe->tail = ++tail;
         ...
         spin_unlock_irq(&pipe->rd_wait.lock);

From my understanding, this is still done with "pipe->mutex" held. Both anon_pipe_read() and pipe_resize_ring() will lock "pipe->mutex" first and then take the "pipe->rd_wait.lock" when updating "pipe->tail". "pipe->head" is always updated with "pipe->mutex" held.

Could that be enough to guaranteed that RMW on a 16-bit data on Alpha is safe since the updates to the two 16-bit fields are protected by the "pipe->mutex" or am I missing something?

Quoted 39 lines

with

          mutex_lock(&pipe->mutex);
          ...
          pipe->head = head + 1;
          ...
          mutex_unlock(&pipe->mutex);

because while they are two different fields using two different
locks, on alpha the above only works if they are in separate words (because updating one will do a read-and-write-back of the other).

This is a fundamental alpha architecture bug. I was actually quite ready to just kill off alpha support entirely, because it's a dead architecture that is unfixably broken. But there's some crazy patches to make gcc generate horrific atomic code to make this all work on alpha by Maciej Rozycki, so one day we'll be in the situation that alpha can be considered "fixed", but we're not there yet.

Do we really care? Maybe not. The race is probably very hard to hit, so with the two remaining alpha users, we could just say "let's just make it use 16-bit ops".

But even on x86, 32-bit ops potentially generate just slightly better code due to lack of some prefix bytes.

And those fields *used* to be 32-bit, so my patch basically kept the status quo on 64-bit machines (and just turned it into 16-bit fields on 32-bit architectures).

Anyway, I wouldn't object to just unconditionally making it "two 16-bit indexes make a 32-bit head_tail" if it actually makes the structure smaller. It might not even matter on 64-bit because of alignment of fields around it - I didn't check. As mentioned, it was more of a combination of "alpha" plus "no change to relevant other architectures".

Linus

Signature · 2 lines
Thanks and Regards,
Prateek
#52 lore raw
Quoted 11 lines · K Prateek Nayak
Quoted 6 lines
         spin_lock_irq(&pipe->rd_wait.lock);
          ...
         pipe->tail = ++tail;
         ...
         spin_unlock_irq(&pipe->rd_wait.lock);

From my understanding, this is still done with "pipe->mutex" held. Both
anon_pipe_read() and pipe_resize_ring() will lock "pipe->mutex" first and then take the "pipe->rd_wait.lock" when updating "pipe->tail". "pipe->head" is always updated with "pipe->mutex" held.

No, see the actual watch_queue code: post_one_notification() in fs/watch_queue.c.

It's isn't the exact sequence I posted, it looks like

        smp_store_release(&pipe->head, head + 1); /* vs pipe_read() */

instead, and it's pipe->head there vs pipe->tail in pipe_read().

And I do think we end up having exclusion thanks to pipe_update_tail() taking that spinlock if the pipe is actually a watchqueue thing, so it might all be ok on alpha too.

So *maybe* we can just make it all be two 16-bit words in a 32-bit thing, but somebody needs to walk through it all to make sure.

Linus

#84 lore raw
In reply to Oleg Nesterov

On Wed, 5 Mar 2025 00:49:09 +0100 Oleg Nesterov <[email protected]>

Quoted 5 lines

Of course! Again, whatever the woken writer checks in pipe_writable() lockless, another writer can make pipe_full() true again.

But why do we care? Why do you think that the change you propose makes

Because of the hang reported.

Quoted 2 lines

more sense than the fix from Prateek or the (already merged) Linus's fix?

See the loop in ___wait_event(),

	for (;;) {
		prepare_to_wait_event();

		// flip
		if (condition)
			break;

		schedule();
	}

After wakeup, waiter will sleep again if condition flips false on the waker side before waiter checks condition, even if condition is atomic, no?

#85 lore raw

Hi Hillf,

again, I am not sure we understand each other, at least me...

Quoted 9 lines · Hillf Danton

On Wed, 5 Mar 2025 00:49:09 +0100 Oleg Nesterov <[email protected]>

Quoted 5 lines

Of course! Again, whatever the woken writer checks in pipe_writable() lockless, another writer can make pipe_full() true again.

But why do we care? Why do you think that the change you propose makes

Because of the hang reported.

The hang happened because pipe_writable() could wrongly return false when the buffer is not full.

Afaics, the Prateek's or Linus's fix solve this problem, and this is all we need.

Quoted 4 lines
Quoted 2 lines

more sense than the fix from Prateek or the (already merged) Linus's fix?

See the loop in ___wait_event(),

Quoted 9 lines
	for (;;) {
		prepare_to_wait_event();

		// flip
		if (condition)
			break;

		schedule();
	}

I will assume that this "// flip" means the case I described above:
before this writer checks the condition, another writer comes and increments pipe->head.

Quoted 2 lines

After wakeup, waiter will sleep again if condition flips false on the waker side before waiter checks condition, even if condition is atomic, no?

Yes, but in this case pipe_full() == true is correct, this writer can safely sleep.

Even if flips again and becomes false right after the writer called pipe_writable().

Note that it checks the condition after set_current_state(INTERRUPTIBLE) and it is still on the pipe->wr_wait->head list.

The 2nd "flip" is only possible if some reader flushes another buffer and updates pipe->tail, and in this case it will wake this writer again.

So I still can't understand your concerns, sorry.

Oleg.

#56 lore raw
In reply to Linus Torvalds
Quoted 14 lines · Linus Torvalds
Quoted 3 lines · Oleg Nesterov

Stupid question... but do we really need to change the code which update tail/head if we pack them into a single word?

No. It's only the READ_ONCE() parts that need changing.

See this suggested patch, which does something very similar to what you were thinking of.

include/linux/pipe_fs_i.h +4 -0
+/*
+ * We have to declare this outside 'struct pipe_inode_info',
+ * but then we can't use 'union pipe_index' for an anonymous
+ * union, so we end up having to duplicate this declaration

include/linux/pipe_fs_i.h struct pipe_buffer { · line 60

Quoted 10 lines
include/linux/pipe_fs_i.h +10 -0
+ * below. Annoying.
+ */
+union pipe_index {
+	unsigned long head_tail;
+	struct {
+		pipe_index_t head;
+		pipe_index_t tail;
+	};
+};
+

-fms-extensions ? Willy wanted to add that for use in mm/ some years ago [*], and it has come up a few other times as well.

[*] https://lore.kernel.org/lkml/[email protected]/

Rasmus

#60 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

ENTIRELY UNTESTED, but it seems to generate ok code. It might even generate better code than what we have now.

Bah. This patch - which is now committed - was actually completely broken.

And the reason that complete breakage didn't show up in testing is that I suspect nobody really tested or thought about the 32-bit case.

That whole "use 16-bit indexes on 32-bit" is all fine and well, but I woke up in the middle of the night and realized that it doesn't actually work.

Because now "pipe_occupancy()" is getting *entirely* the wrong answers. It just does

        return head - tail;

but that only worked when the arithmetic was done modulo the size of the indexes. And now it isn't.

So I still haven't *tested* this, but at an absolute minimum, we need something like this:

--- a/include/linux/pipe_fs_i.h
+++ b/include/linux/pipe_fs_i.h
@@ -192,7 +192,7 @@

    */

static inline unsigned int pipe_occupancy(unsigned int head,
unsigned int tail)
{
- return head - tail;
+ return (pipe_index_t)(head - tail);
}

/**

and there might be other cases where the pipe_index_t size might matter.

For example, we should add a check to pipe_resize_ring() that the new size is smaller than the index size. Yes, in practice 'pipe_max_size' already ends up being that limit (the value is 256 pages), even for 16-bit indices, but we should do this properly.

And then, *while* looking at this, I also noticed that we had a very much related bug in this area that was pre-existing and not related to the 16-bit change: pipe_discard_from() is doing the wrong thing for overflows even in the old 'unsigned int' type, and the whole

        while (pipe->head > old_head)

is bogus, because 'pipe->head' may have wrapped around, so the whole "is it bigger" test doesn't work like that at all.

Of course, in practice it never hits (and would only hit more easily with the new 16-bit thing), but it's very very wrong and can result in a memory leak.

Are there other cases like this? I don't know. I've been looking around a bit, but those were the only ones I found immediately when I started thinking about the whole wrap-around issue.

I'd love it if other people tried to think about this too (and maybe even test the 32-bit case - gasp!)

Linus

#57 lore raw
In reply to Rasmus Villemoes
Quoted 21 lines · Rasmus Villemoes
Quoted 4 lines · Linus Torvalds
include/linux/pipe_fs_i.h +4 -0
+/*
+ * We have to declare this outside 'struct pipe_inode_info',
+ * but then we can't use 'union pipe_index' for an anonymous
+ * union, so we end up having to duplicate this declaration

include/linux/pipe_fs_i.h struct pipe_buffer { · line 60

Quoted 10 lines
include/linux/pipe_fs_i.h +10 -0
+ * below. Annoying.
+ */
+union pipe_index {
+     unsigned long head_tail;
+     struct {
+             pipe_index_t head;
+             pipe_index_t tail;
+     };
+};
+

-fms-extensions ? Willy wanted to add that for use in mm/ some years ago [*], and it has come up a few other times as well.

Quoted 1 line

[*] https://lore.kernel.org/lkml/[email protected]/

Oh, I was unaware of that extension, and yes, it would have been lovely here, avoiding that duplicate union declaration.

But it does require clang support - I see that clang has a '-fms-extensions' as well, so it's presumably there.

I don't know if it's worth it for the (small handful) of cases we'd have in the kernel, but it does seem useful.

Linus

#86 lore raw
In reply to Oleg Nesterov

On Wed, 5 Mar 2025 12:44:34 +0100 Oleg Nesterov <[email protected]>

Quoted 10 lines · Hillf Danton
Quoted 9 lines

See the loop in ___wait_event(),

	for (;;) {
		prepare_to_wait_event();

		// flip
		if (condition)
			break;
Quoted 9 lines
Quoted 5 lines
		schedule();
	}

After wakeup, waiter will sleep again if condition flips false on the waker side before waiter checks condition, even if condition is atomic, no?

Yes, but in this case pipe_full() == true is correct, this writer can safely sleep.

No, because no reader is woken up before sleep to make pipe not full.

#61 lore raw
In reply to Linus Torvalds
Quoted 34 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

ENTIRELY UNTESTED, but it seems to generate ok code. It might even generate better code than what we have now.

Bah. This patch - which is now committed - was actually completely broken.

And the reason that complete breakage didn't show up in testing is that I suspect nobody really tested or thought about the 32-bit case.

That whole "use 16-bit indexes on 32-bit" is all fine and well, but I woke up in the middle of the night and realized that it doesn't actually work.

Because now "pipe_occupancy()" is getting *entirely* the wrong answers. It just does

        return head - tail;

but that only worked when the arithmetic was done modulo the size of the indexes. And now it isn't.

So I still haven't *tested* this, but at an absolute minimum, we need something like this:

--- a/include/linux/pipe_fs_i.h
+++ b/include/linux/pipe_fs_i.h
@@ -192,7 +192,7 @@

    */

static inline unsigned int pipe_occupancy(unsigned int head,
unsigned int tail)
{
- return head - tail;

Quoted 6 lines
  +       return (pipe_index_t)(head - tail);
   }

   /**

and there might be other cases where the pipe_index_t size might matter.

Yeah, for example

      unsigned int count, head, tail, mask;

      case FIONREAD:
              mutex_lock(&pipe->mutex);
              count = 0;
              head = pipe->head;
              tail = pipe->tail;
              mask = pipe->ring_size - 1;

              while (tail != head) {
                      count += pipe->bufs[tail & mask].len;
                      tail++;
              }
              mutex_unlock(&pipe->mutex);

If head has already wrapped around, say it's 0 or 1, and tail is close to 65535, that loop is gonna take forever and of course produce the wrong result.

So yes, there are probably a lot more of these lurking.

There are probably not many tests that stuff 2^28 bytes through a pipe to try to trigger such corner cases. Perhaps we can help whatever automated tests are being done by initializing head and tail to something like (pipe_index_t)-2 when the pipe is created?

Rasmus

#63 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

Are there other cases like this? I don't know. I've been looking around a bit, but those were the only ones I found immediately when I started thinking about the whole wrap-around issue.

Without too much brainpower spent analyzing each case (i.e., some of these might actually be ok), I found these:

fs/fuse/dev.c
fuse_dev_splice_write()

unsigned int head, tail, mask, count;

  pipe_lock(pipe);

  head = pipe->head;
  tail = pipe->tail;
  mask = pipe->ring_size - 1;
  count = head - tail;

Open-coded pipe_occupancy(), so would be fixed by using that with your fixup.

A bit later in same function there's the same FIONREAD pattern:

  for (idx = tail; idx != head && rem < len; idx++)
          rem += pipe->bufs[idx & mask].len;

fs/pipe.c

We have pipe_update_tail() getting and returning an "unsigned int", and letting the compiler truncate the result written to pipe->tail:

      pipe->tail = ++tail;
      return tail;

pipe_update_tail() only has one caller, but a rather important one, pipe_read(), which uses the return value from pipe_update_tail as-is

                 tail = pipe_update_tail(pipe, buf, tail);
         }
         total_len -= chars;
         if (!total_len)
                 break;  /* common path: read succeeded */
         if (!pipe_empty(head, tail))    /* More to do? */
                 continue;

and pipe_empty() takes two "unsigned ints" and is just head==tail -- so if tail was incremented to 65536 while head is 0 that would break. Probably pipe_empty() should either take pipe_index_t arguments or cast to that internally, just as pipe_occupancy. Or, as pipe_full(), being spelled in terms of pipe_occupancy()==0.

With that fixed, maybe one could spell the FIONREAD-like patterns using pipe_empty(), i.e. using pipe_empty() to ask "have this tail index now caught up to this head index". So "idx != head" above would become "!pipe_empty(idx, head)".

Rasmus

#87 lore raw
In reply to Hillf Danton
Quoted 15 lines · Hillf Danton

On Wed, 5 Mar 2025 12:44:34 +0100 Oleg Nesterov <[email protected]>

Quoted 13 lines · Hillf Danton
Quoted 12 lines

See the loop in ___wait_event(),

	for (;;) {
		prepare_to_wait_event();

		// flip
		if (condition)
			break;

		schedule();
	}
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

After wakeup, waiter will sleep again if condition flips false on the waker side before waiter checks condition, even if condition is atomic, no?

Yes, but in this case pipe_full() == true is correct, this writer can safely sleep.

No, because no reader is woken up before sleep to make pipe not full.

Why the reader should be woken before this writer sleeps? Why the reader should be woken at all in this case (when pipe is full again) ?

We certainly can't understand each other.

Could your picture the exact scenario/sequence which can hang?

Oleg.

#58 lore raw
In reply to Linus Torvalds
Quoted 22 lines · Linus Torvalds
Quoted 21 lines · Rasmus Villemoes
Quoted 4 lines · Linus Torvalds
include/linux/pipe_fs_i.h +4 -0
+/*
+ * We have to declare this outside 'struct pipe_inode_info',
+ * but then we can't use 'union pipe_index' for an anonymous
+ * union, so we end up having to duplicate this declaration

include/linux/pipe_fs_i.h struct pipe_buffer { · line 60

Quoted 10 lines
include/linux/pipe_fs_i.h +10 -0
+ * below. Annoying.
+ */
+union pipe_index {
+     unsigned long head_tail;
+     struct {
+             pipe_index_t head;
+             pipe_index_t tail;
+     };
+};
+

-fms-extensions ? Willy wanted to add that for use in mm/ some years ago [*], and it has come up a few other times as well.

Quoted 7 lines
Quoted 1 line

[*] https://lore.kernel.org/lkml/[email protected]/

Oh, I was unaware of that extension, and yes, it would have been lovely here, avoiding that duplicate union declaration.

But it does require clang support - I see that clang has a '-fms-extensions' as well, so it's presumably there.

Yes, it seems they do have it, but for mysterious reasons saying -fms-extensions is not quite enough to convince clang that one does intend to use that MS extension, one also has to say -Wno-microsoft-anon-tag, or it complains

warning: anonymous unions are a Microsoft extension [-Wmicrosoft-anon-tag]

Also, the warning text is somewhat misleading; anon unions itself have certainly been a gcc extension since forever, and nowadays a C11 thing, and clang has a separate -Wpedantic warning for that when using -std=c99:

warning: anonymous unions are a C11 extension [-Wc11-extensions]

The -W flag name actually suggests an improvement to the warning "_tagged_ anonymous unions are a Microsoft extension", but I really wonder why -fms-extensions isn't sufficient to silence that in the first place. Also, the warning seems to be on by default; it's not some -Wextra or -Wpedantic thing.

cc += Nick

Rasmus

#64 lore raw

cover pipe: Convert pipe->{head,tail} to unsigned short

In reply to Linus Torvalds

Here is an attempt at converting pipe->{head,tail} to unsigned short members. All local variables storing the head and the tail have been modified to unsigned short too the best of my knowledge)

pipe_resize_ring() has added a check to make sure nr_slots can be contained within the limits of the pipe->{head,tail}. Building on that, pipe->{max_usage,ring_size} were also converted to unsigned short to catch any cases of incorrect unsigned arithmetic.

This has been tested for a few hours with anon pipes on a 5th Generation AMD EPYC System and on a dual socket Intel Granite Rapids system without experiencing any obvious issues.

pipe_write() was tagged with a debug trace_printk() on one of the test machines to make sure the head has indeed wrapped around behind the tail to ensure the wraparound scenarios are indeed happening.

Few pipe_occupancy() and pipe->max_usage based checks have been converted to use unsigned short based arithmetic in fs/fuse/dev.c, fs/splice.c, mm/filemap.c, and mm/filemap.c. Few of the observations from Rasmus on a parallel thread [1] has been folded into Patch 3 (thanks a ton for chasing them).

More eyes and testing is greatly appreciated. If my tests run into any issues, I'll report back on this thread. Series was tested with:

  hackbench -g 16 -f 20 --threads --pipe -l 10000000 -s 100 # Warp around
  stress-ng --oom-pipe 128 --oom-pipe-ops 100000 -t 600s # pipe resize
  stress-ng --splice 128 --splice-ops 100000000 -t 600s # splice
  stress-ng --vm-splice 128 --vm-splice-ops 100000000 -t 600s # splice

  stress-ng --tee 128 --tee-ops 100000000 -t 600s
  stress-ng --zlib 128 --zlib-ops 1000000 -t 600s
  stress-ng --sigpipe 128 -t 60s

stress-ng did not report any failure in my testing.

[1] https://lore.kernel.org/all/[email protected]/
--
K Prateek Nayak (3):
fs/pipe: Limit the slots in pipe_resize_ring() fs/splice: Atomically read pipe->{head,tail} in opipe_prep()
treewide: pipe: Convert all references to

    pipe->{head,tail,max_usage,ring_size} to unsigned short
 fs/fuse/dev.c             |  4 +++-
 fs/pipe.c                 | 33 +++++++++++++++-----------
 fs/splice.c               | 50 ++++++++++++++++++++-------------------
 include/linux/pipe_fs_i.h | 39 ++++++++++--------------------
 kernel/watch_queue.c      |  3 ++-
 mm/filemap.c              |  5 ++--
 mm/shmem.c                |  5 ++--
 7 files changed, 69 insertions(+), 70 deletions(-)

base-commit: 848e076317446f9c663771ddec142d7c2eb4cb43

Signature · 1 line
2.43.0
#65 lore raw

RFC PATCH 1/3 fs/pipe: Limit the slots in pipe_resize_ring()

Limit the number of slots in pipe_resize_ring() to the maximum value representable by pipe->{head,tail}. Values beyond the max limit can lead to incorrect pipe_occupancy() calculations where the pipe will never appear full.

Since nr_slots is always a power of 2 and the maximum size of pipe_index_t is 32 bits, BIT() is sufficient to represent the maximum value possible for nr_slots.

---
 fs/pipe.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/pipe.c +4 -0
index e8e6698f3698..3ca3103e1de7 100644
@@ -1272,6 +1272,10 @@ int pipe_resize_ring(struct pipe_inode_info *pipe, unsigned int nr_slots)
 	struct pipe_buffer *bufs;
 	unsigned int head, tail, mask, n;
 
+	/* nr_slots larger than limits of pipe->{head,tail} */
+	if (unlikely(nr_slots > BIT(BITS_PER_TYPE(pipe_index_t) - 1)))
+		return -EINVAL;
+
 	bufs = kcalloc(nr_slots, sizeof(*bufs),
 		       GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
 	if (unlikely(!bufs))

base-commit: 848e076317446f9c663771ddec142d7c2eb4cb43

Signature · 1 line
2.43.0
#68 lore raw

RFC PATCH 2/3 fs/splice: Atomically read pipe->{head,tail} in opipe_prep()

In reply to K Prateek Nayak

opipe_prep() checks pipe_full() before taking the "pipe->mutex". Use the newly introduced "pipe->head_tail" member to read the head and the tail atomically and not miss any updates between the reads.

---
 fs/splice.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/splice.c +2 -1
index 28cfa63aa236..e51f33aca032 100644
@@ -1682,13 +1682,14 @@ static int ipipe_prep(struct pipe_inode_info *pipe, unsigned int flags)
  */
 static int opipe_prep(struct pipe_inode_info *pipe, unsigned int flags)
 {
+	union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) };
 	int ret;
 
 	/*
 	 * Check pipe occupancy without the inode lock first. This function
 	 * is speculative anyways, so missing one is ok.
 	 */
-	if (!pipe_full(pipe->head, pipe->tail, pipe->max_usage))
+	if (!pipe_full(idx.head, idx.tail, READ_ONCE(pipe->max_usage)))
 		return 0;
 
 	ret = 0;
Signature · 1 line
2.43.0
#69 lore raw

RFC PATCH 3/3 treewide: pipe: Convert all references to pipe->{head,tail,max_usage,ring_size} to unsigned short

In reply to K Prateek Nayak

Use 16-bit head and tail to track the pipe buffer production and consumption.

Since "pipe->max_usage" and "pipe->ring_size" must fall between the head and the tail limits, convert them to unsigned short as well to catch any cases of unsigned arithmetic going wrong.

Part of fs/fuse/dev.c, fs/splice.c, mm/filemap.c, and mm/shmem.c were touched to accommodate the "unsigned short" based calculations of pipe_occupancy().

pipe->tail is incremented always with both "pipe->mutex" and "pipe->rd_wait.lock" held for pipes with watch queue. pipe_write() exits early if pipe has a watch queue but otherwise takes the "pipe->muxtex" before updating pipe->head. post_one_notification() holds the "pipe->rd_wait.lock" when updating pipe->head.

Updates to "pipe->head" and "pipe->tail" are always mutually exclusive, either guarded by "pipe->mutex" or by "pipe->rd_wait.lock". Even a RMW updates to the 16-bits fields should be safe because of those synchronization primitives on architectures that cannot do an atomic 16-bit store.

---
 fs/fuse/dev.c             |  7 +++---
 fs/pipe.c                 | 31 +++++++++++++-------------
 fs/splice.c               | 47 ++++++++++++++++++++-------------------
 include/linux/pipe_fs_i.h | 39 +++++++++++---------------------
 kernel/watch_queue.c      |  3 ++-
 mm/filemap.c              |  5 +++--
 mm/shmem.c                |  5 +++--
 7 files changed, 65 insertions(+), 72 deletions(-)
fs/fuse/dev.c +4 -3
index 2b2d1b755544..993e6dc24de1 100644
@@ -1440,6 +1440,7 @@ static ssize_t fuse_dev_splice_read(struct file *in, loff_t *ppos,
 	int page_nr = 0;
 	struct pipe_buffer *bufs;
 	struct fuse_copy_state cs;
+	unsigned short free_slots;
 	struct fuse_dev *fud = fuse_get_dev(in);
 
 	if (!fud)
@@ -1457,7 +1458,8 @@ static ssize_t fuse_dev_splice_read(struct file *in, loff_t *ppos,
 	if (ret < 0)
 		goto out;
 
-	if (pipe_occupancy(pipe->head, pipe->tail) + cs.nr_segs > pipe->max_usage) {
+	free_slots = pipe->max_usage - pipe_occupancy(pipe->head, pipe->tail);
+	if (free_slots < cs.nr_segs) {
 		ret = -EIO;
 		goto out;
 	}
@@ -2107,9 +2109,8 @@ static ssize_t fuse_dev_splice_write(struct pipe_inode_info *pipe,
 				     struct file *out, loff_t *ppos,
 				     size_t len, unsigned int flags)
 {
-	unsigned int head, tail, mask, count;
+	unsigned short head, tail, mask, count, idx;
 	unsigned nbuf;
-	unsigned idx;
 	struct pipe_buffer *bufs;
 	struct fuse_copy_state cs;
 	struct fuse_dev *fud;
fs/pipe.c +16 -15
index 3ca3103e1de7..b8d87eabff79 100644
@@ -216,9 +216,9 @@ static inline bool pipe_readable(const struct pipe_inode_info *pipe)
 	return !pipe_empty(idx.head, idx.tail) || !writers;
 }
 
-static inline unsigned int pipe_update_tail(struct pipe_inode_info *pipe,
-					    struct pipe_buffer *buf,
-					    unsigned int tail)
+static inline unsigned short pipe_update_tail(struct pipe_inode_info *pipe,
+					      struct pipe_buffer *buf,
+					      unsigned short tail)
 {
 	pipe_buf_release(pipe, buf);
 
@@ -272,9 +272,9 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	 */
 	for (;;) {
 		/* Read ->head with a barrier vs post_one_notification() */
-		unsigned int head = smp_load_acquire(&pipe->head);
-		unsigned int tail = pipe->tail;
-		unsigned int mask = pipe->ring_size - 1;
+		unsigned short head = smp_load_acquire(&pipe->head);
+		unsigned short tail = pipe->tail;
+		unsigned short mask = pipe->ring_size - 1;
 
 #ifdef CONFIG_WATCH_QUEUE
 		if (pipe->note_loss) {
@@ -417,7 +417,7 @@ static inline int is_packetized(struct file *file)
 static inline bool pipe_writable(const struct pipe_inode_info *pipe)
 {
 	union pipe_index idx = { .head_tail = READ_ONCE(pipe->head_tail) };
-	unsigned int max_usage = READ_ONCE(pipe->max_usage);
+	unsigned short max_usage = READ_ONCE(pipe->max_usage);
 
 	return !pipe_full(idx.head, idx.tail, max_usage) ||
 		!READ_ONCE(pipe->readers);
@@ -428,7 +428,7 @@ pipe_write(struct kiocb *iocb, struct iov_iter *from)
 {
 	struct file *filp = iocb->ki_filp;
 	struct pipe_inode_info *pipe = filp->private_data;
-	unsigned int head;
+	unsigned short head;
 	ssize_t ret = 0;
 	size_t total_len = iov_iter_count(from);
 	ssize_t chars;
@@ -471,7 +471,7 @@ pipe_write(struct kiocb *iocb, struct iov_iter *from)
 	was_empty = pipe_empty(head, pipe->tail);
 	chars = total_len & (PAGE_SIZE-1);
 	if (chars && !was_empty) {
-		unsigned int mask = pipe->ring_size - 1;
+		unsigned short mask = pipe->ring_size - 1;
 		struct pipe_buffer *buf = &pipe->bufs[(head - 1) & mask];
 		int offset = buf->offset + buf->len;
 
@@ -614,7 +614,8 @@ pipe_write(struct kiocb *iocb, struct iov_iter *from)
 static long pipe_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 {
 	struct pipe_inode_info *pipe = filp->private_data;
-	unsigned int count, head, tail, mask;
+	unsigned short head, tail, mask;
+	unsigned int count;
 
 	switch (cmd) {
 	case FIONREAD:
@@ -1270,10 +1271,10 @@ unsigned int round_pipe_size(unsigned int size)
 int pipe_resize_ring(struct pipe_inode_info *pipe, unsigned int nr_slots)
 {
 	struct pipe_buffer *bufs;
-	unsigned int head, tail, mask, n;
+	unsigned short head, tail, mask, n;
 
 	/* nr_slots larger than limits of pipe->{head,tail} */
-	if (unlikely(nr_slots > BIT(BITS_PER_TYPE(pipe_index_t) - 1)))
+	if (unlikely(nr_slots > USHRT_MAX))
 		return -EINVAL;
 
 	bufs = kcalloc(nr_slots, sizeof(*bufs),
@@ -1298,13 +1299,13 @@ int pipe_resize_ring(struct pipe_inode_info *pipe, unsigned int nr_slots)
 	 * and adjust the indices.
 	 */
 	if (n > 0) {
-		unsigned int h = head & mask;
-		unsigned int t = tail & mask;
+		unsigned short h = head & mask;
+		unsigned short t = tail & mask;
 		if (h > t) {
 			memcpy(bufs, pipe->bufs + t,
 			       n * sizeof(struct pipe_buffer));
 		} else {
-			unsigned int tsize = pipe->ring_size - t;
+			unsigned short tsize = pipe->ring_size - t;
 			if (h > 0)
 				memcpy(bufs + tsize, pipe->bufs,
 				       h * sizeof(struct pipe_buffer));
fs/splice.c +24 -23

117 lines not on this page · raw message

include/linux/pipe_fs_i.h +13 -26
index e572e6fc4f81..0997c028548c 100644
@@ -31,19 +31,6 @@ struct pipe_buffer {
 	unsigned long private;
 };
 
-/*
- * Really only alpha needs 32-bit fields, but
- * might as well do it for 64-bit architectures
- * since that's what we've historically done,
- * and it makes 'head_tail' always be a simple
- * 'unsigned long'.
- */
-#ifdef CONFIG_64BIT
-typedef unsigned int pipe_index_t;
-#else
-typedef unsigned short pipe_index_t;
-#endif
-
 /*
  * We have to declare this outside 'struct pipe_inode_info',
  * but then we can't use 'union pipe_index' for an anonymous
@@ -51,10 +38,10 @@ typedef unsigned short pipe_index_t;
  * below. Annoying.
  */
 union pipe_index {
-	unsigned long head_tail;
+	unsigned int head_tail;
 	struct {
-		pipe_index_t head;
-		pipe_index_t tail;
+		unsigned short head;
+		unsigned short tail;
 	};
 };
 
@@ -89,15 +76,15 @@ struct pipe_inode_info {
 
 	/* This has to match the 'union pipe_index' above */
 	union {
-		unsigned long head_tail;
+		unsigned int head_tail;
 		struct {
-			pipe_index_t head;
-			pipe_index_t tail;
+			unsigned short head;
+			unsigned short tail;
 		};
 	};
 
-	unsigned int max_usage;
-	unsigned int ring_size;
+	unsigned short max_usage;
+	unsigned short ring_size;
 	unsigned int nr_accounted;
 	unsigned int readers;
 	unsigned int writers;
@@ -181,7 +168,7 @@ static inline bool pipe_has_watch_queue(const struct pipe_inode_info *pipe)
  * @head: The pipe ring head pointer
  * @tail: The pipe ring tail pointer
  */
-static inline bool pipe_empty(unsigned int head, unsigned int tail)
+static inline bool pipe_empty(unsigned short head, unsigned short tail)
 {
 	return head == tail;
 }
@@ -191,9 +178,9 @@ static inline bool pipe_empty(unsigned int head, unsigned int tail)
  * @head: The pipe ring head pointer
  * @tail: The pipe ring tail pointer
  */
-static inline unsigned int pipe_occupancy(unsigned int head, unsigned int tail)
+static inline unsigned short pipe_occupancy(unsigned short head, unsigned short tail)
 {
-	return (pipe_index_t)(head - tail);
+	return head - tail;
 }
 
 /**
@@ -202,8 +189,8 @@ static inline unsigned int pipe_occupancy(unsigned int head, unsigned int tail)
  * @tail: The pipe ring tail pointer
  * @limit: The maximum amount of slots available.
  */
-static inline bool pipe_full(unsigned int head, unsigned int tail,
-			     unsigned int limit)
+static inline bool pipe_full(unsigned short head, unsigned short tail,
+			     unsigned short limit)
 {
 	return pipe_occupancy(head, tail) >= limit;
 }
kernel/watch_queue.c +2 -1
index 5267adeaa403..c76cfebf46c8 100644
@@ -101,7 +101,8 @@ static bool post_one_notification(struct watch_queue *wqueue,
 	struct pipe_inode_info *pipe = wqueue->pipe;
 	struct pipe_buffer *buf;
 	struct page *page;
-	unsigned int head, tail, mask, note, offset, len;
+	unsigned short head, tail, mask;
+	unsigned int note, offset, len;
 	bool done = false;
 
 	spin_lock_irq(&pipe->rd_wait.lock);
mm/filemap.c +3 -2
index d4564a79eb35..6007b2403471 100644
@@ -2943,9 +2943,10 @@ ssize_t filemap_splice_read(struct file *in, loff_t *ppos,
 {
 	struct folio_batch fbatch;
 	struct kiocb iocb;
-	size_t total_spliced = 0, used, npages;
+	size_t total_spliced = 0, npages;
 	loff_t isize, end_offset;
 	bool writably_mapped;
+	unsigned short used;
 	int i, error = 0;
 
 	if (unlikely(*ppos >= in->f_mapping->host->i_sb->s_maxbytes))
@@ -2956,7 +2957,7 @@ ssize_t filemap_splice_read(struct file *in, loff_t *ppos,
 
 	/* Work out how much data we can actually add into the pipe */
 	used = pipe_occupancy(pipe->head, pipe->tail);
-	npages = max_t(ssize_t, pipe->max_usage - used, 0);
+	npages = max_t(unsigned short, pipe->max_usage - used, 0);
 	len = min_t(size_t, len, npages * PAGE_SIZE);
 
 	folio_batch_init(&fbatch);
mm/shmem.c +3 -2
index 4ea6109a8043..339084e5a8a1 100644
@@ -3509,13 +3509,14 @@ static ssize_t shmem_file_splice_read(struct file *in, loff_t *ppos,
 	struct inode *inode = file_inode(in);
 	struct address_space *mapping = inode->i_mapping;
 	struct folio *folio = NULL;
-	size_t total_spliced = 0, used, npages, n, part;
+	size_t total_spliced = 0, npages, n, part;
+	unsigned short used;
 	loff_t isize;
 	int error = 0;
 
 	/* Work out how much data we can actually add into the pipe */
 	used = pipe_occupancy(pipe->head, pipe->tail);
-	npages = max_t(ssize_t, pipe->max_usage - used, 0);
+	npages = max_t(unsigned short, pipe->max_usage - used, 0);
 	len = min_t(size_t, len, npages * PAGE_SIZE);
 
 	do {
Signature · 1 line
2.43.0
#66 lore raw

Re: [RFC PATCH 1/3] fs/pipe: Limit the slots in pipe_resize_ring()

In reply to K Prateek Nayak

fs/pipe.c int pipe_resize_ring(struct pipe_inode_info *pipe, unsigned int nr_slots) · line 1276

Quoted 7 lines · K Prateek Nayak
fs/pipe.c +2 -0
@@ -1272,6 +1272,10 @@ int pipe_resize_ring(struct pipe_inode_info *pipe, unsigned int nr_slots)
 	struct pipe_buffer *bufs;
 	unsigned int head, tail, mask, n;
 
+	/* nr_slots larger than limits of pipe->{head,tail} */
+	if (unlikely(nr_slots > BIT(BITS_PER_TYPE(pipe_index_t) - 1)))

Hmm, perhaps

	if (nr_slots > (pipe_index_t)-1u)

is more clear?

Oleg.

#70 lore raw

Re: [RFC PATCH 3/3] treewide: pipe: Convert all references to pipe->{head,tail,max_usage,ring_size} to unsigned short

In reply to K Prateek Nayak

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 277

Quoted 11 lines · K Prateek Nayak
fs/pipe.c +3 -3
@@ -272,9 +272,9 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	 */
 	for (;;) {
 		/* Read ->head with a barrier vs post_one_notification() */
-		unsigned int head = smp_load_acquire(&pipe->head);
-		unsigned int tail = pipe->tail;
-		unsigned int mask = pipe->ring_size - 1;
+		unsigned short head = smp_load_acquire(&pipe->head);
+		unsigned short tail = pipe->tail;
+		unsigned short mask = pipe->ring_size - 1;

I dunno... but if we do this, perhaps we should
s/unsigned int/pipe_index_t instead?

At least this would be more grep friendly.

Oleg.

#71 lore raw

Re: [RFC PATCH 3/3] treewide: pipe: Convert all references to pipe->{head,tail,max_usage,ring_size} to unsigned short

Quoted 17 lines · Oleg Nesterov

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 277

Quoted 11 lines · K Prateek Nayak
fs/pipe.c +3 -3
@@ -272,9 +272,9 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	 */
 	for (;;) {
 		/* Read ->head with a barrier vs post_one_notification() */
-		unsigned int head = smp_load_acquire(&pipe->head);
-		unsigned int tail = pipe->tail;
-		unsigned int mask = pipe->ring_size - 1;
+		unsigned short head = smp_load_acquire(&pipe->head);
+		unsigned short tail = pipe->tail;
+		unsigned short mask = pipe->ring_size - 1;

I dunno... but if we do this, perhaps we should
s/unsigned int/pipe_index_t instead?

Quoted 1 line

At least this would be more grep friendly.

in any case, I think another cleanup before this change makes sense... pipe->ring_size is overused. pipe_read(), pipe_write() and much more users do not need "unsigned int mask", they can use pipe_buf(buf, slot) instead.

Oleg.

#74 lore raw

Re: [RFC PATCH 3/3] treewide: pipe: Convert all references to pipe->{head,tail,max_usage,ring_size} to unsigned short

In reply to Oleg Nesterov
Quoted 16 lines · Oleg Nesterov

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 277

Quoted 11 lines · K Prateek Nayak
fs/pipe.c +3 -3
@@ -272,9 +272,9 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 	 */
 	for (;;) {
 		/* Read ->head with a barrier vs post_one_notification() */
-		unsigned int head = smp_load_acquire(&pipe->head);
-		unsigned int tail = pipe->tail;
-		unsigned int mask = pipe->ring_size - 1;
+		unsigned short head = smp_load_acquire(&pipe->head);
+		unsigned short tail = pipe->tail;
+		unsigned short mask = pipe->ring_size - 1;

I dunno... but if we do this, perhaps we should
s/unsigned int/pipe_index_t instead?

Quoted 1 line

At least this would be more grep friendly.

Agreed. Also, while using u16 on all arches may be good for now to make sure everything is updated, it may also be that it ends up causing suboptimal code gen for 64 bit architectures, so even if we do change pipe_index_t now, perhaps we'd want to change it back to "half a ulong" at some point in the future.

Rasmus

#59 lore raw
In reply to Rasmus Villemoes
Quoted 46 lines · Rasmus Villemoes
Quoted 22 lines · Linus Torvalds
Quoted 21 lines · Rasmus Villemoes
Quoted 4 lines · Linus Torvalds
include/linux/pipe_fs_i.h +4 -0
+/*
+ * We have to declare this outside 'struct pipe_inode_info',
+ * but then we can't use 'union pipe_index' for an anonymous
+ * union, so we end up having to duplicate this declaration

include/linux/pipe_fs_i.h struct pipe_buffer { · line 60

Quoted 10 lines
include/linux/pipe_fs_i.h +10 -0
+ * below. Annoying.
+ */
+union pipe_index {
+     unsigned long head_tail;
+     struct {
+             pipe_index_t head;
+             pipe_index_t tail;
+     };
+};
+

-fms-extensions ? Willy wanted to add that for use in mm/ some years ago [*], and it has come up a few other times as well.

Quoted 7 lines
Quoted 1 line

[*] https://lore.kernel.org/lkml/[email protected]/

Oh, I was unaware of that extension, and yes, it would have been lovely here, avoiding that duplicate union declaration.

But it does require clang support - I see that clang has a '-fms-extensions' as well, so it's presumably there.

Yes, it seems they do have it, but for mysterious reasons saying -fms-extensions is not quite enough to convince clang that one does intend to use that MS extension, one also has to say -Wno-microsoft-anon-tag, or it complains

warning: anonymous unions are a Microsoft extension [-Wmicrosoft-anon-tag]

Also, the warning text is somewhat misleading; anon unions itself have certainly been a gcc extension since forever, and nowadays a C11 thing, and clang has a separate -Wpedantic warning for that when using -std=c99:

warning: anonymous unions are a C11 extension [-Wc11-extensions]

Quoted 7 lines

The -W flag name actually suggests an improvement to the warning "_tagged_ anonymous unions are a Microsoft extension", but I really wonder why -fms-extensions isn't sufficient to silence that in the first place. Also, the warning seems to be on by default; it's not some -Wextra or -Wpedantic thing.

cc += Nick

Gah, sorry, I wasn't aware that address didn't work anymore.

So Cc -= everything but the lists and Cc += Nick for real this time, hopefully.

Rasmus

#67 lore raw

Re: [RFC PATCH 1/3] fs/pipe: Limit the slots in pipe_resize_ring()

In reply to Oleg Nesterov

Hello Oleg,

Quoted 6 lines · Oleg Nesterov
Quoted 5 lines · K Prateek Nayak
fs/pipe.c +0 -0
@@ -1272,6 +1272,10 @@ int pipe_resize_ring(struct pipe_inode_info *pipe, unsigned int nr_slots)
  	struct pipe_buffer *bufs;
  	unsigned int head, tail, mask, n;
 
Quoted 8 lines

fs/pipe.c int pipe_resize_ring(struct pipe_inode_info *pipe, unsigned int nr_slots) · line 1276

Quoted 2 lines
fs/pipe.c +2 -0
+	/* nr_slots larger than limits of pipe->{head,tail} */
+	if (unlikely(nr_slots > BIT(BITS_PER_TYPE(pipe_index_t) - 1)))

Hmm, perhaps

	if (nr_slots > (pipe_index_t)-1u)

is more clear?

Indeed it is. I didn't even know we could do that! Thank you for pointing it out.

Quoted 3 lines

Oleg.

Signature · 2 lines
Thanks and Regards,
Prateek
#72 lore raw

Re: [RFC PATCH 3/3] treewide: pipe: Convert all references to pipe->{head,tail,max_usage,ring_size} to unsigned short

In reply to Oleg Nesterov

Hello Oleg, Rasmus,

Quoted 18 lines · Oleg Nesterov
Quoted 17 lines · Oleg Nesterov

fs/pipe.c pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 277

Quoted 11 lines · K Prateek Nayak
fs/pipe.c +3 -3
@@ -272,9 +272,9 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
  	 */
  	for (;;) {
  		/* Read ->head with a barrier vs post_one_notification() */
-		unsigned int head = smp_load_acquire(&pipe->head);
-		unsigned int tail = pipe->tail;
-		unsigned int mask = pipe->ring_size - 1;
+		unsigned short head = smp_load_acquire(&pipe->head);
+		unsigned short tail = pipe->tail;
+		unsigned short mask = pipe->ring_size - 1;

I dunno... but if we do this, perhaps we should
s/unsigned int/pipe_index_t instead?

Quoted 1 line
Quoted 1 line

At least this would be more grep friendly.

Ack. I'll leave the typedef untouched and convert these to use pipe_index_t. This was an experiment so see if anything breaks with u16 conversion just to get more testing on that scenario. As Rasmus mentioned, leaving the head and tail as u32 on 64bit will lead to better code generation.

Quoted 5 lines

in any case, I think another cleanup before this change makes sense... pipe->ring_size is overused. pipe_read(), pipe_write() and much more users do not need "unsigned int mask", they can use pipe_buf(buf, slot) instead.

Ack. I'll add a cleanup patch ahead of this conversion. Thank you both for taking a look.

Quoted 3 lines

Oleg.

Signature · 2 lines
Thanks and Regards,
Prateek
#62 lore raw
In reply to Rasmus Villemoes
Quoted 10 lines · Rasmus Villemoes
Quoted 2 lines · Linus Torvalds

and there might be other cases where the pipe_index_t size might matter.

Yeah, for example

      unsigned int count, head, tail, mask;

      case FIONREAD:

Thanks. I've hopefully fixed this (and the FUSE issue you also reported), and those should work correctly no on 32-bit. Knock wood.

Mind taking a look and double-checking the fixes?

Linus

#73 lore raw

Re: [RFC PATCH 3/3] treewide: pipe: Convert all references to pipe->{head,tail,max_usage,ring_size} to unsigned short

In reply to K Prateek Nayak
Quoted 11 lines · K Prateek Nayak
Quoted 5 lines
Quoted 5 lines

I dunno... but if we do this, perhaps we should
s/unsigned int/pipe_index_t instead?

At least this would be more grep friendly.

Ack. I'll leave the typedef untouched and convert these to use pipe_index_t. This was an experiment so see if anything breaks with u16 conversion just to get more testing on that scenario. As Rasmus mentioned, leaving the head and tail as u32 on 64bit will lead to better code generation.

Yes, I was going to say the same - please don't change to 'unsigned short'.

Judicious use of 'pipe_index_t' may be a good idea, but as I fixed some issues Rasmus found, I was also looking at the generated code, and on at least x86 where 16-bit generates extra instructions and prefixes, it seems marginally better to treat the values as 32-bit, and then only do the compares in 16 bits.

That only causes a few "movzwl" instructions (at load time), and then the occasional "cmpw" (empty check) and "movw" (store) etc.

But I only did a very quick "let's look at a few cases of x86-64 also using a 16-bit pipe_index_t".

So for testing purposes your patch looks fine, but not as something to apply.

If anything, I think we should actively try to remove as many direct accesses to these pipe fields as humanly possible. As Oleg said, a lot of them should just be cleaned up to use the helpers we already have.

Rasmus found a few cases of that already, like that FIONREAD case where it was just doing a lot of open-coding of things that shouldn't be open-coded.

I've fixed the two cases he pointed at up as obvious bugs, but it would be good to see where else issues like this might lurk.

Linus

#88 lore raw
In reply to Oleg Nesterov

On Thu, 6 Mar 2025 10:30:21 +0100 Oleg Nesterov <[email protected]>

Quoted 18 lines · Hillf Danton
Quoted 17 lines

On Wed, 5 Mar 2025 12:44:34 +0100 Oleg Nesterov <[email protected]>

Quoted 16 lines · Hillf Danton
Quoted 14 lines

See the loop in ___wait_event(),

	for (;;) {
		prepare_to_wait_event();

		// flip
		if (condition)
			break;

		schedule();
	}

After wakeup, waiter will sleep again if condition flips false on the waker side before waiter checks condition, even if condition is atomic, no?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Yes, but in this case pipe_full() == true is correct, this writer can safely sleep.

No, because no reader is woken up before sleep to make pipe not full.

Why the reader should be woken before this writer sleeps? Why the reader should be woken at all in this case (when pipe is full again) ?

"to make pipe not full" failed to prevent you asking questions like this one.

Quoted 4 lines

We certainly can't understand each other.

Could your picture the exact scenario/sequence which can hang?

If you think the scenario in commit 3d252160b818 [1] is correct, check the following one.

step-00

	pipe->head = 36
	pipe->tail = 36
	after 3d252160b818

step-01

	task-118762 writer
	pipe->head++;
	wakes up task-118740 and task-118768

step-02

	task-118768 writer
	makes pipe full;
	sleeps without waking up any reader as
	pipe was not empty after step-01

step-03

	task-118766 new reader
	makes pipe empty
	sleeps

step-04

	task-118740 reader
	sleeps as pipe is empty

[ Tasks 118740 and 118768 can then indefinitely wait on each other. ]

[1] https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/pipe.c?id=3d252160b818045f3a152b13756f6f37ca34639d

#89 lore raw

Hello Hiilf,

Quoted 47 lines · Hillf Danton

On Thu, 6 Mar 2025 10:30:21 +0100 Oleg Nesterov <[email protected]>

Quoted 18 lines · Hillf Danton
Quoted 17 lines

On Wed, 5 Mar 2025 12:44:34 +0100 Oleg Nesterov <[email protected]>

Quoted 16 lines · Hillf Danton
Quoted 14 lines

See the loop in ___wait_event(),

	for (;;) {
		prepare_to_wait_event();

		// flip
		if (condition)
			break;

		schedule();
	}

After wakeup, waiter will sleep again if condition flips false on the waker side before waiter checks condition, even if condition is atomic, no?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Yes, but in this case pipe_full() == true is correct, this writer can safely sleep.

No, because no reader is woken up before sleep to make pipe not full.

Why the reader should be woken before this writer sleeps? Why the reader should be woken at all in this case (when pipe is full again) ?

"to make pipe not full" failed to prevent you asking questions like this one.

Quoted 4 lines

We certainly can't understand each other.

Could your picture the exact scenario/sequence which can hang?

If you think the scenario in commit 3d252160b818 [1] is correct, check the following one.

step-00

	pipe->head = 36
	pipe->tail = 36
	after 3d252160b818

step-01

	task-118762 writer
	pipe->head++;
	wakes up task-118740 and task-118768

step-02

Quoted 8 lines
	task-118768 writer
	makes pipe full;
	sleeps without waking up any reader as
	pipe was not empty after step-01

step-03

	task-118766 new reader
	makes pipe empty

Reader seeing a pipe full should wake up a writer allowing 118768 to wakeup again and fill the pipe. Am I missing something?

Quoted 10 lines

sleeps

step-04

	task-118740 reader
	sleeps as pipe is empty

[ Tasks 118740 and 118768 can then indefinitely wait on each other. ]

[1] https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/pipe.c?id=3d252160b818045f3a152b13756f6f37ca34639d

Signature · 2 lines
Thanks and Regards,
Prateek
#90 lore raw

On Fri, 7 Mar 2025 11:54:56 +0530 K Prateek Nayak <[email protected]>

Quoted 52 lines · Hillf Danton
Quoted 52 lines

On Thu, 6 Mar 2025 10:30:21 +0100 Oleg Nesterov <[email protected]>

Quoted 18 lines · Hillf Danton
Quoted 17 lines

On Wed, 5 Mar 2025 12:44:34 +0100 Oleg Nesterov <[email protected]>

Quoted 16 lines · Hillf Danton
Quoted 14 lines

See the loop in ___wait_event(),

	for (;;) {
		prepare_to_wait_event();

		// flip
		if (condition)
			break;

		schedule();
	}

After wakeup, waiter will sleep again if condition flips false on the waker side before waiter checks condition, even if condition is atomic, no?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Yes, but in this case pipe_full() == true is correct, this writer can safely sleep.

No, because no reader is woken up before sleep to make pipe not full.

Why the reader should be woken before this writer sleeps? Why the reader should be woken at all in this case (when pipe is full again) ?

"to make pipe not full" failed to prevent you asking questions like this one.

Quoted 4 lines

We certainly can't understand each other.

Could your picture the exact scenario/sequence which can hang?

If you think the scenario in commit 3d252160b818 [1] is correct, check the following one.

step-00

	pipe->head = 36
	pipe->tail = 36
	after 3d252160b818

step-01

	task-118762 writer
	pipe->head++;
	wakes up task-118740 and task-118768

step-02

	task-118768 writer
	makes pipe full;
	sleeps without waking up any reader as
	pipe was not empty after step-01
Quoted 7 lines
Quoted 3 lines

step-03

	task-118766 new reader
	makes pipe empty

Reader seeing a pipe full should wake up a writer allowing 118768 to wakeup again and fill the pipe. Am I missing something?

Good catch, but that wakeup was cut off [2,3]

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

Quoted 14 lines
Quoted 10 lines

sleeps

step-04

	task-118740 reader
	sleeps as pipe is empty

[ Tasks 118740 and 118768 can then indefinitely wait on each other. ]

[1] https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/pipe.c?id=3d252160b818045f3a152b13756f6f37ca34639d

Signature · 2 lines
Thanks and Regards,
Prateek
#105 lore raw
In reply to Hillf Danton
Quoted 6 lines · Hillf Danton

On Thu, 6 Mar 2025 10:30:21 +0100 Oleg Nesterov <[email protected]>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Yes, but in this case pipe_full() == true is correct, this writer can safely sleep.

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

No, because no reader is woken up before sleep to make pipe not full.

Why the reader should be woken before this writer sleeps? Why the reader should be woken at all in this case (when pipe is full again) ?

"to make pipe not full" failed to prevent you asking questions like this one.

Hmm. I don't understand your "prevent you asking questions" reply.

If the pipe was full we do not need to wake the reader(s), the reader can only sleep if pipe_empty() == true.

Quoted 1 line
Quoted 1 line

We certainly can't understand each other.

Yes.

Quoted 16 lines

step-00

	pipe->head = 36
	pipe->tail = 36
	after 3d252160b818

step-01

	task-118762 writer
	pipe->head++;
	wakes up task-118740 and task-118768

step-02

	task-118768 writer
	makes pipe full;
	sleeps without waking up any reader as
	pipe was not empty after step-01
Quoted 4 lines

step-03

	task-118766 new reader
	makes pipe empty
	sleeps

but since the pipe was full, this reader should wake up the writer task-118768 once it updates the tail the 1st time during the read.

Quoted 3 lines

step-04

	task-118740 reader
	sleeps as pipe is empty

this is fine.

Quoted 1 line

[ Tasks 118740 and 118768 can then indefinitely wait on each other. ]

118768 should be woken at step 3 ?

Oleg.

#91 lore raw
In reply to Hillf Danton
Quoted 6 lines · Hillf Danton

On Fri, 7 Mar 2025 11:54:56 +0530 K Prateek Nayak <[email protected]>

Quoted 4 lines
Quoted 3 lines

step-03

	task-118766 new reader
	makes pipe empty
Quoted 7 lines
Quoted 3 lines

Reader seeing a pipe full should wake up a writer allowing 118768 to wakeup again and fill the pipe. Am I missing something?

Good catch, but that wakeup was cut off [2,3]

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

Why do you think

	[PATCH v2 1/1] pipe: change pipe_write() to never add a zero-sized buffer
	https://lore.kernel.org/all/[email protected]/

can make any difference ???

Where do you think a zero-sized buffer with ->len == 0 can come from?

Oleg.

#92 lore raw

In case I wasn't clear...

Quoted 4 lines · Oleg Nesterov
Quoted 2 lines · Hillf Danton

On Fri, 7 Mar 2025 11:54:56 +0530 K Prateek Nayak <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

step-03

	task-118766 new reader
	makes pipe empty

Reader seeing a pipe full should wake up a writer allowing 118768 to wakeup again and fill the pipe. Am I missing something?

Good catch, but that wakeup was cut off [2,3]

Please note that "that wakeup" was _not_ removed by the patch below.

"That wakeup" is another wakeup pipe_read() does before return:

	if (wake_writer)
		wake_up_interruptible_sync_poll(&pipe->wr_wait, ...);

And wake_writer must be true if this reader changed the pipe_full() condition from T to F.

Note also that pipe_read() won't sleep if it has read even one byte.

Quoted 11 lines
Quoted 2 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

Why do you think

	[PATCH v2 1/1] pipe: change pipe_write() to never add a zero-sized buffer
	https://lore.kernel.org/all/[email protected]/

can make any difference ???

Where do you think a zero-sized buffer with ->len == 0 can come from?

Oleg.

Tested-by K Prateek Nayak
#93 lore raw

On Fri, 7 Mar 2025 13:34:43 +0100 Oleg Nesterov <[email protected]>

Quoted 7 lines · Oleg Nesterov
Quoted 6 lines · Hillf Danton
Quoted 5 lines

On Fri, 7 Mar 2025 11:54:56 +0530 K Prateek Nayak <[email protected]>

Quoted 4 lines
Quoted 3 lines

step-03

	task-118766 new reader
	makes pipe empty
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Reader seeing a pipe full should wake up a writer allowing 118768 to wakeup again and fill the pipe. Am I missing something?

Good catch, but that wakeup was cut off [2,3]

Please note that "that wakeup" was _not_ removed by the patch below.

After another look, you did cut it.

---
 fs/pipe.c | 45 +++++++++------------------------------------
 1 file changed, 9 insertions(+), 36 deletions(-)
fs/pipe.c +0 -21
index 2ae75adfba64..b0641f75b1ba 100644
@@ -360,29 +360,9 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 			break;
 		}
 		mutex_unlock(&pipe->mutex);
-
 		/*
 		 * We only get here if we didn't actually read anything.
 		 *
-		 * However, we could have seen (and removed) a zero-sized
-		 * pipe buffer, and might have made space in the buffers
-		 * that way.
-		 *
-		 * You can't make zero-sized pipe buffers by doing an empty
-		 * write (not even in packet mode), but they can happen if
-		 * the writer gets an EFAULT when trying to fill a buffer
-		 * that already got allocated and inserted in the buffer
-		 * array.
-		 *
-		 * So we still need to wake up any pending writers in the
-		 * _very_ unlikely case that the pipe was full, but we got
-		 * no data.
-		 */
-		if (unlikely(wake_writer))
-			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
-		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
-
-		/*
 		 * But because we didn't read anything, at this point we can
 		 * just return directly with -ERESTARTSYS if we're interrupted,
 		 * since we've done any required wakeups and there's no need
@@ -391,7 +371,6 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
 			return -ERESTARTSYS;
 
-		wake_writer = false;
 		wake_next_reader = true;
 		mutex_lock(&pipe->mutex);
 	}
Quoted 8 lines

"That wakeup" is another wakeup pipe_read() does before return:

	if (wake_writer)
		wake_up_interruptible_sync_poll(&pipe->wr_wait, ...);

And wake_writer must be true if this reader changed the pipe_full() condition from T to F.

Could you read Prateek's comment again, then try to work out why he did so?

Quoted 15 lines

Note also that pipe_read() won't sleep if it has read even one byte.

Quoted 11 lines
Quoted 2 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

Why do you think

	[PATCH v2 1/1] pipe: change pipe_write() to never add a zero-sized buffer
	https://lore.kernel.org/all/[email protected]/

can make any difference ???

Where do you think a zero-sized buffer with ->len == 0 can come from?

Oleg.

#94 lore raw

Hello Hillf,

Quoted 13 lines · Hillf Danton

On Fri, 7 Mar 2025 13:34:43 +0100 Oleg Nesterov <[email protected]>

Quoted 12 lines · Oleg Nesterov
Quoted 10 lines · Hillf Danton
Quoted 9 lines

On Fri, 7 Mar 2025 11:54:56 +0530 K Prateek Nayak <[email protected]>

Quoted 7 lines
Quoted 3 lines

step-03

	task-118766 new reader
	makes pipe empty

Reader seeing a pipe full should wake up a writer allowing 118768 to wakeup again and fill the pipe. Am I missing something?

Good catch, but that wakeup was cut off [2,3]

Quoted 7 lines
Quoted 2 lines

Please note that "that wakeup" was _not_ removed by the patch below.

After another look, you did cut it.

So that is not problematic because pipe_write() no longer increments the head before a successful write. What also changed in that patch above is the order in which we do copy_page_from_iter() and head increment - now, the head is incremented only if copy_page_from_iter() actually manages to write data into the buffer which eliminates the need to do a wakeup if nothing was found in the buffer ...

fs/pipe.c anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)

Quoted 9 lines
---
  fs/pipe.c | 45 +++++++++------------------------------------
  1 file changed, 9 insertions(+), 36 deletions(-)
fs/pipe.c +0 -0
index 2ae75adfba64..b0641f75b1ba 100644
@@ -360,29 +360,9 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)

Above this, we have:

	if (!total_len)
		break;	/* common path: read succeeded */
	if (!pipe_empty(head, tail))	/* More to do? */
		continue;

And if a read is successful, one of them must be hit and with that reordering in pipe_write(), and the readers must be able to wake a writer if !pipe_empty() ...

Quoted 40 lines
fs/pipe.c +0 -21
  			break;
  		}
  		mutex_unlock(&pipe->mutex);
-
  		/*
  		 * We only get here if we didn't actually read anything.
  		 *
-		 * However, we could have seen (and removed) a zero-sized
-		 * pipe buffer, and might have made space in the buffers
-		 * that way.
-		 *
-		 * You can't make zero-sized pipe buffers by doing an empty
-		 * write (not even in packet mode), but they can happen if
-		 * the writer gets an EFAULT when trying to fill a buffer
-		 * that already got allocated and inserted in the buffer
-		 * array.
-		 *
-		 * So we still need to wake up any pending writers in the
-		 * _very_ unlikely case that the pipe was full, but we got
-		 * no data.
-		 */
-		if (unlikely(wake_writer))
-			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
-		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
-
-		/*
  		 * But because we didn't read anything, at this point we can
  		 * just return directly with -ERESTARTSYS if we're interrupted,
  		 * since we've done any required wakeups and there's no need
@@ -391,7 +371,6 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
  		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
  			return -ERESTARTSYS;
 
-		wake_writer = false;
  		wake_next_reader = true;
  		mutex_lock(&pipe->mutex);
  	}
Quoted 3 lines

"That wakeup" is another wakeup pipe_read() does before return:

Quoted 10 lines
Quoted 6 lines
	if (wake_writer)
		wake_up_interruptible_sync_poll(&pipe->wr_wait, ...);

And wake_writer must be true if this reader changed the pipe_full() condition from T to F.

Could you read Prateek's comment again, then try to work out why he did so?

Quoted 1 line

Note also that pipe_read() won't sleep if it has read even one byte.

and that is the key why Oleg's optimization that he highlighted above which is why it cannot cause a hang (with my imagination). Now, let us take a closer look at the whole sleep and wakeup mechanism. I'll expand the one for pipe_write() since that was the problematic bit we analyzed but you can do the same for pipe_read() side too. The

    wait_event_interruptible_exclusive(pipe->wr_wait, pipe_writable(pipe));

in pipe_write() will boil down to these bits (I'll only highlight the important ones):

    ret = 0;

    might_sleep()
    if (!pipe_writable()) { /* First line of defense */
        init_wait_entry(&__wq_entry, WQ_FLAG_EXCLUSIVE);

        for (;;) {
            /* Below is expansion of prepare_to_wait_event() success case*/
            spin_lock_irqsave(&wq_head->lock, flags);
            __add_wait_queue_entry_tail(wq_head, __wq_entry);
            set_current_state(TASK_INTERRUPTIBLE);
            spin_unlock_irqrestore(&wq_head->lock, flags);

            /* Second line of defense */
	    if (pipe_writable())
                break;

            schedule();
        }
        finish_wait(&wq_head, &__wq_entry);
    ]
    return 0;

The sequence for the writer wait is:

o Add on the wait queue.
o Set task state to TASK_INTERRUPTIBLE.
o Check if pipe_writable() one last time.
o Call schedule()

Now why can't we miss a wakeup? The reader increments the tail, and then does a wakeup of waiting writers in all cases it finds pipe_full(). Wakeup accesses a the wait queues which does:

    spin_lock_irqsave(&wq_head->lock, flags);
    __wake_up_common();
    spin_unlock_irqrestore(&wq_head->lock, flags);

so adding to wait queue and wakeup from wait queue are always done under the wait queue lock.

Now there are two non trivial cases:

o This is the simple case:

                 writer                                                 reader
                 ======                                                 ======
if (!pipe_writable()) /* True */ {
    for (;;) {
        ...                                            wake_writers = pipe_full(); /* True */
                                                       tail = tail + 1
                                                       wake_up_interruptible_sync_poll(&pipe->wr_wait) {
                                                           /*wr_wait is empty */
                                                       }
        ...
        /* Adds itself on the wait queue */
        writer->__state = TASK_INTERRUPTIBLE;
        
        if (pipe_writable()) /* True */ {
            break;
            /*
             * Goes and does finish_wait()
             */
        }
    }

    finish_wait() {
        p->__state = TASK_RUNNING;
    }

    /*
     * Goes and does a check under
     * pipe->mutex to be sure.
     */
}

O This is slightly complicated case:

                 writer                                                 reader
                 ======                                                 ======
if (!pipe_writable()) /* True */ {
    for (;;) {
        /* Adds itself on the wait queue */
        writer->__state = TASK_INTERRUPTIBLE;
        if (pipe_writable()) /* False */ {
            /* The break is not executed */
        }
        ...                                            wake_writers = pipe_full(); /* True */
                                                       tail = tail + 1
                                                       wake_up_interruptible_sync_poll(&pipe->wr_wait) {
                                                           default_wake_function() {
                                                               ttwu_runnable() {
                                                                   /* Calls ttwu_do_wakeup() which does */
                                                                   p->__state = TASK_RUNNING
                                                               }
							   }
                                                       }
        ...
        schedule() {
            if (!p->__state) /* False */ {
                /*
                 * Never called since p->__state
                 * is RUNNING
                 */
                try_to_block_task();
            }
        }

        ... repeat the loop
        if (pipe_writable()) /* True */ {
            break;
            /*
             * Goes and does finish_wait()
             */
        }
    }

    finish_wait() {
        p->__state = TASK_RUNNING;
    }

    /*
     * Goes and does a check under
     * pipe->mutex to be sure.
     */
}

All in all, the writer will wither see a pipe_writable() and never fully block, or it will be woken up it will add itself to the wit queue and will be woken by a reader after moving pipe->tail if pipe_full() returned true before. If I've missed something, please let me know and I'll try to go back and convince myself on how that situation can happen and get back to you but so far, my head cannot think of a situation where the pipe can hang after the recent set of fixes and optimizations.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Why do you think

	[PATCH v2 1/1] pipe: change pipe_write() to never add a zero-sized buffer
	https://lore.kernel.org/all/[email protected]/

can make any difference ???

Where do you think a zero-sized buffer with ->len == 0 can come from?

I audited the post_one_notification() code and the paths that lead up to it:

o remove_watch_from_object() will have the data of size watch_sizeof(n) so that beffer size is definitely !0 (unless (watch_sizeof(n) & 0x7f is zero but what are the odds of that?)

o Other is __post_watch_notification() which has a early return for

      ((n->info & WATCH_INFO_LENGTH) >> WATCH_INFO_LENGTH__SHIFT) == 0

and also a WARN_ON() so it'll not add an empty buffer and will instead scream at the users if someone tries to do it.

splice() cases too cannot do this based on my understanding so I don't think we can run into the issue but I'm limited by my own imagination :)

Quoted 2 lines
Quoted 2 lines

Oleg.

Signature · 2 lines
Thanks and Regards,
Prateek
#95 lore raw
In reply to Hillf Danton

Well. Prateek has already provide the lengthy/thorough explanation, but let me add anyway...

Quoted 9 lines · Hillf Danton

On Fri, 7 Mar 2025 13:34:43 +0100 Oleg Nesterov <[email protected]>

Quoted 7 lines · Oleg Nesterov
Quoted 6 lines · Hillf Danton
Quoted 5 lines

On Fri, 7 Mar 2025 11:54:56 +0530 K Prateek Nayak <[email protected]>

Quoted 4 lines
Quoted 3 lines

step-03

	task-118766 new reader
	makes pipe empty
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Reader seeing a pipe full should wake up a writer allowing 118768 to wakeup again and fill the pipe. Am I missing something?

Good catch, but that wakeup was cut off [2,3]

Please note that "that wakeup" was _not_ removed by the patch below.

After another look, you did cut it.

I still don't think so.

Quoted 1 line

...

Quoted 33 lines
fs/pipe.c +0 -20
@@ -360,29 +360,9 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 			break;
 		}
 		mutex_unlock(&pipe->mutex);
-
 		/*
 		 * We only get here if we didn't actually read anything.
 		 *
-		 * However, we could have seen (and removed) a zero-sized
-		 * pipe buffer, and might have made space in the buffers
-		 * that way.
-		 *
-		 * You can't make zero-sized pipe buffers by doing an empty
-		 * write (not even in packet mode), but they can happen if
-		 * the writer gets an EFAULT when trying to fill a buffer
-		 * that already got allocated and inserted in the buffer
-		 * array.
-		 *
-		 * So we still need to wake up any pending writers in the
-		 * _very_ unlikely case that the pipe was full, but we got
-		 * no data.
-		 */
-		if (unlikely(wake_writer))
-			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
-		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
-
-		/*
 		 * But because we didn't read anything, at this point we can
 		 * just return directly with -ERESTARTSYS if we're interrupted,
 		 * since we've done any required wakeups and there's no need

fs/pipe.c anon_pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 376

Quoted 7 lines
fs/pipe.c +0 -1
@@ -391,7 +371,6 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
 			return -ERESTARTSYS;
 
-		wake_writer = false;
 		wake_next_reader = true;
 		mutex_lock(&pipe->mutex);
 	}

Please note that in this particular case (hackbench testing) pipe_write() -> copy_page_from_iter() never fails. So wake_writer is never true before pipe_reader() calls wait_event(pipe->rd_wait).

So (again, in this particular case) we could apply the patch below on top of Linus's tree.

So, with or without these changes, the writer should be woken up at step-03 in your scenario.

Oleg.
---

fs/pipe.c +1 -21
@@ -360,27 +360,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		}
 		mutex_unlock(&pipe->mutex);
 
-		/*
-		 * We only get here if we didn't actually read anything.
-		 *
-		 * However, we could have seen (and removed) a zero-sized
-		 * pipe buffer, and might have made space in the buffers
-		 * that way.
-		 *
-		 * You can't make zero-sized pipe buffers by doing an empty
-		 * write (not even in packet mode), but they can happen if
-		 * the writer gets an EFAULT when trying to fill a buffer
-		 * that already got allocated and inserted in the buffer
-		 * array.
-		 *
-		 * So we still need to wake up any pending writers in the
-		 * _very_ unlikely case that the pipe was full, but we got
-		 * no data.
-		 */
-		if (unlikely(wake_writer))
-			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
-		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
-
+		BUG_ON(wake_writer);
 		/*
 		 * But because we didn't read anything, at this point we can
 		 * just return directly with -ERESTARTSYS if we're interrupted,
#96 lore raw

On Sun, 9 Mar 2025 18:02:55 +0100 Oleg Nesterov

Quoted 4 lines

Well. Prateek has already provide the lengthy/thorough explanation, but let me add anyway...

lengthy != correct

Quoted 58 lines · Hillf Danton
Quoted 8 lines

On Fri, 7 Mar 2025 13:34:43 +0100 Oleg Nesterov <[email protected]>

Quoted 7 lines · Oleg Nesterov
Quoted 6 lines · Hillf Danton
Quoted 5 lines

On Fri, 7 Mar 2025 11:54:56 +0530 K Prateek Nayak <[email protected]>

Quoted 4 lines
Quoted 3 lines

step-03

	task-118766 new reader
	makes pipe empty
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Reader seeing a pipe full should wake up a writer allowing 118768 to wakeup again and fill the pipe. Am I missing something?

Good catch, but that wakeup was cut off [2,3]

Please note that "that wakeup" was _not_ removed by the patch below.

After another look, you did cut it.

I still don't think so.

Quoted 1 line

...

Quoted 36 lines
fs/pipe.c +0 -20
@@ -360,29 +360,9 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 			break;
 		}
 		mutex_unlock(&pipe->mutex);
-
 		/*
 		 * We only get here if we didn't actually read anything.
 		 *
-		 * However, we could have seen (and removed) a zero-sized
-		 * pipe buffer, and might have made space in the buffers
-		 * that way.
-		 *
-		 * You can't make zero-sized pipe buffers by doing an empty
-		 * write (not even in packet mode), but they can happen if
-		 * the writer gets an EFAULT when trying to fill a buffer
-		 * that already got allocated and inserted in the buffer
-		 * array.
-		 *
-		 * So we still need to wake up any pending writers in the
-		 * _very_ unlikely case that the pipe was full, but we got
-		 * no data.
-		 */
-		if (unlikely(wake_writer))
-			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
-		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
-
-		/*
 		 * But because we didn't read anything, at this point we can
 		 * just return directly with -ERESTARTSYS if we're interrupted,
 		 * since we've done any required wakeups and there's no need
@@ -391,7 +371,6 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
 			return -ERESTARTSYS;
 
Quoted 9 lines

fs/pipe.c anon_pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 376

Quoted 4 lines
fs/pipe.c +0 -1
-		wake_writer = false;
 		wake_next_reader = true;
 		mutex_lock(&pipe->mutex);
 	}

Please note that in this particular case (hackbench testing) pipe_write() -> copy_page_from_iter() never fails. So wake_writer is never true before pipe_reader() calls wait_event(pipe->rd_wait).

Given never and the BUG_ON below, you accidentally prove that Prateek's comment is false, no?

Quoted 6 lines

So (again, in this particular case) we could apply the patch below on top of Linus's tree.

So, with or without these changes, the writer should be woken up at step-03 in your scenario.

Fine, before checking my scenario once more, feel free to pinpoint the line number where writer is woken up, with the change below applied.

Quoted 36 lines

Oleg.
---

fs/pipe.c +1 -21
@@ -360,27 +360,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		}
 		mutex_unlock(&pipe->mutex);
 
-		/*
-		 * We only get here if we didn't actually read anything.
-		 *
-		 * However, we could have seen (and removed) a zero-sized
-		 * pipe buffer, and might have made space in the buffers
-		 * that way.
-		 *
-		 * You can't make zero-sized pipe buffers by doing an empty
-		 * write (not even in packet mode), but they can happen if
-		 * the writer gets an EFAULT when trying to fill a buffer
-		 * that already got allocated and inserted in the buffer
-		 * array.
-		 *
-		 * So we still need to wake up any pending writers in the
-		 * _very_ unlikely case that the pipe was full, but we got
-		 * no data.
-		 */
-		if (unlikely(wake_writer))
-			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
-		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
-
+		BUG_ON(wake_writer);
 		/*
 		 * But because we didn't read anything, at this point we can
 		 * just return directly with -ERESTARTSYS if we're interrupted,
#97 lore raw
Quoted 11 lines · Hillf Danton

On Sun, 9 Mar 2025 18:02:55 +0100 Oleg Nesterov

Quoted 7 lines

So (again, in this particular case) we could apply the patch below on top of Linus's tree.

So, with or without these changes, the writer should be woken up at step-03 in your scenario.

Fine, before checking my scenario once more, feel free to pinpoint the line number where writer is woken up, with the change below applied.

381 if (wake_writer)

==> 382                  wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
    383          if (wake_next_reader)
    384                  wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
    385          kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
    386          if (ret > 0)
    387                  file_accessed(filp);
    388          return ret;

line 382, no?

Oleg.

#98 lore raw

On Mon, 10 Mar 2025 12:09:15 +0100 Oleg Nesterov

Quoted 21 lines · Hillf Danton
Quoted 10 lines

On Sun, 9 Mar 2025 18:02:55 +0100 Oleg Nesterov

Quoted 7 lines

So (again, in this particular case) we could apply the patch below on top of Linus's tree.

So, with or without these changes, the writer should be woken up at step-03 in your scenario.

Fine, before checking my scenario once more, feel free to pinpoint the line number where writer is woken up, with the change below applied.

381 if (wake_writer)

==> 382                  wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
    383          if (wake_next_reader)
    384                  wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
    385          kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
    386          if (ret > 0)
    387                  file_accessed(filp);
    388          return ret;
Quoted 2 lines

line 382, no?

Yes, but how is the wait loop at line-370 broken?

360 }
361 mutex_unlock(&pipe->mutex);
362
363 BUG_ON(wake_writer);
364 /*

 365                  * But because we didn't read anything, at this point we can
 366                  * just return directly with -ERESTARTSYS if we're interrupted,
 367                  * since we've done any required wakeups and there's no need
 368                  * to mark anything accessed. And we've dropped the lock.
 369                  */
 370                 if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
 371                         return -ERESTARTSYS;
 372
 373                 wake_writer = false;
 374                 wake_next_reader = true;
 375                 mutex_lock(&pipe->mutex);
#99 lore raw
Quoted 32 lines · Hillf Danton

On Mon, 10 Mar 2025 12:09:15 +0100 Oleg Nesterov

Quoted 21 lines · Hillf Danton
Quoted 10 lines

On Sun, 9 Mar 2025 18:02:55 +0100 Oleg Nesterov

Quoted 7 lines

So (again, in this particular case) we could apply the patch below on top of Linus's tree.

So, with or without these changes, the writer should be woken up at step-03 in your scenario.

Fine, before checking my scenario once more, feel free to pinpoint the line number where writer is woken up, with the change below applied.

381 if (wake_writer)

==> 382                  wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
    383          if (wake_next_reader)
    384                  wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
    385          kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
    386          if (ret > 0)
    387                  file_accessed(filp);
    388          return ret;
Quoted 2 lines

line 382, no?

Yes, but how is the wait loop at line-370 broken?

360 }
361 mutex_unlock(&pipe->mutex);
362
363 BUG_ON(wake_writer);
364 /*

Quoted 7 lines
 365                  * But because we didn't read anything, at this point we can
 366                  * just return directly with -ERESTARTSYS if we're interrupted,
 367                  * since we've done any required wakeups and there's no need
 368                  * to mark anything accessed. And we've dropped the lock.
 369                  */

370 if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
371 return -ERESTARTSYS;

Hmm. I don't understand you, again.

OK, once some writer writes at least one byte (this will make the pipe_empty() condition false) and wakes this reader up.

If you meant something else, say, if you referred to you previous scenario, please clarify your question.

Oleg.

#100 lore raw

On Mon, 10 Mar 2025 13:43:42 +0100 Oleg Nesterov

Quoted 40 lines · Hillf Danton
Quoted 31 lines

On Mon, 10 Mar 2025 12:09:15 +0100 Oleg Nesterov

Quoted 21 lines · Hillf Danton
Quoted 10 lines

On Sun, 9 Mar 2025 18:02:55 +0100 Oleg Nesterov

Quoted 7 lines

So (again, in this particular case) we could apply the patch below on top of Linus's tree.

So, with or without these changes, the writer should be woken up at step-03 in your scenario.

Fine, before checking my scenario once more, feel free to pinpoint the line number where writer is woken up, with the change below applied.

381 if (wake_writer)

==> 382                  wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
    383          if (wake_next_reader)
    384                  wake_up_interruptible_sync_poll(&pipe->rd_wait, EPOLLIN | EPOLLRDNORM);
    385          kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
    386          if (ret > 0)
    387                  file_accessed(filp);
    388          return ret;
Quoted 2 lines

line 382, no?

Yes, but how is the wait loop at line-370 broken?

360 }
361 mutex_unlock(&pipe->mutex);
362
363 BUG_ON(wake_writer);
364 /*

Quoted 7 lines
 365                  * But because we didn't read anything, at this point we can
 366                  * just return directly with -ERESTARTSYS if we're interrupted,
 367                  * since we've done any required wakeups and there's no need
 368                  * to mark anything accessed. And we've dropped the lock.
 369                  */

370 if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
371 return -ERESTARTSYS;

Quoted 8 lines

Hmm. I don't understand you, again.

OK, once some writer writes at least one byte (this will make the pipe_empty() condition false) and wakes this reader up.

If you meant something else, say, if you referred to you previous scenario, please clarify your question.

The step-03 in my scenario [1] shows a reader sleeps at line-370 after making the pipe empty, so after your change that cuts the chance for waking up writer, who will wake up the sleeping reader? Nobody.

Feel free to check my scenario again.

step-03

	task-118766 new reader
	makes pipe empty
	sleeps

[1] https://lore.kernel.org/lkml/[email protected]/

#101 lore raw
Quoted 4 lines · Hillf Danton

The step-03 in my scenario [1] shows a reader sleeps at line-370 after making the pipe empty, so after your change that cuts the chance for waking up writer, who will wake up the sleeping reader? Nobody.

But step-03 will wake the writer.

And no, nobody will wake readers, because the pipe is empty. Only the next writer that adds data to the pipe should wake any readers.

Note that the logic that sets "wake_writer" and "was_empty" is all protected by the pipe semaphore. So there are no races wrt figuring out "should we wake readers/writers".

So I really think you need to very explicitly point to what you think the problem is. Not point to some other email. Write out all out in full and explain.

Linus

#102 lore raw
In reply to Hillf Danton
Quoted 8 lines · Hillf Danton

On Mon, 10 Mar 2025 13:43:42 +0100 Oleg Nesterov

Quoted 6 lines

Hmm. I don't understand you, again.

OK, once some writer writes at least one byte (this will make the pipe_empty() condition false) and wakes this reader up.

Quoted 6 lines
Quoted 3 lines

If you meant something else, say, if you referred to you previous scenario, please clarify your question.

The step-03 in my scenario [1] shows a reader sleeps at line-370 after making the pipe empty, so after your change that cuts the chance for waking up writer,

We are circling.

Once again, in this case "wake_writer" can't be true when the reader does wait_event(rd_wait), this code can be replaced with BUG_ON(wake_writer). So that change cuts nothing. It simply has no effect in this case.

Quoted 10 lines

who will wake up the sleeping reader? Nobody.

Feel free to check my scenario again.

step-03

	task-118766 new reader
	makes pipe empty
	sleeps

[1] https://lore.kernel.org/lkml/[email protected]/

First of all, task-118766 won't sleep unless it calls read() again.

From https://lore.kernel.org/all/[email protected]/

Note also that pipe_read() won't sleep if it has read even one byte.

but this doesn't really matter.

From https://lore.kernel.org/all/[email protected]/

> step-03
> task-118766 new reader
> makes pipe empty
> sleeps

but since the pipe was full, this reader should wake up the writer task-118768 once it updates the tail the 1st time during the read.

> step-04
> task-118740 reader
> sleeps as pipe is empty

this is fine.

> [ Tasks 118740 and 118768 can then indefinitely wait on each other. ]

118768 should be woken at step 3

Now, when the writer task-118768 does write() it will wake the reader, task-118740.

Oleg.

#104 lore raw
In reply to a message not in the archive
Quoted 75 lines · Hillf Danton
Quoted 11 lines · On Mon, 10 Mar 2025 14:26:17 -1000 Linus Torvalds wrote:
Quoted 4 lines · Hillf Danton

The step-03 in my scenario [1] shows a reader sleeps at line-370 after making the pipe empty, so after your change that cuts the chance for waking up writer, who will wake up the sleeping reader? Nobody.

But step-03 will wake the writer.

And no, nobody will wake readers, because the pipe is empty. Only the next writer that adds data to the pipe should wake any readers.

Quoted 8 lines

Note that the logic that sets "wake_writer" and "was_empty" is all protected by the pipe semaphore. So there are no races wrt figuring out "should we wake readers/writers".

So I really think you need to very explicitly point to what you think the problem is. Not point to some other email. Write out all out in full and explain.

In the mainline tree, conditional wakeup [2] exists before a pipe writer takes a nap, so scenario can be constructed based on the one in commit 3d252160b818 to make pipe writer sleep with nobody woken up.

step-00

	pipe->head = 36
	pipe->tail = 36

step-01

	task-118762 is a writer
	pipe->head++;
	wakes up task-118740 and task-118768

step-02

	task-118768 is a writer
	makes pipe full;
	sleeps without waking up any reader as
	pipe was not empty after step-01

Conditional wakeup also exists on the reader side [3], but Oleg cut it off [4].

fs/pipe.c +1 -21
@@ -360,27 +360,7 @@ pipe_read(struct kiocb *iocb, struct iov_iter *to)
 		}
 		mutex_unlock(&pipe->mutex);
 
-		/*
-		 * We only get here if we didn't actually read anything.
-		 *
-		 * However, we could have seen (and removed) a zero-sized
-		 * pipe buffer, and might have made space in the buffers
-		 * that way.
-		 *
-		 * You can't make zero-sized pipe buffers by doing an empty
-		 * write (not even in packet mode), but they can happen if
-		 * the writer gets an EFAULT when trying to fill a buffer
-		 * that already got allocated and inserted in the buffer
-		 * array.
-		 *
-		 * So we still need to wake up any pending writers in the
-		 * _very_ unlikely case that the pipe was full, but we got
-		 * no data.
-		 */
-		if (unlikely(wake_writer))
-			wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM);
-		kill_fasync(&pipe->fasync_writers, SIGIO, POLL_OUT);
-
+		BUG_ON(wake_writer);
 		/*
 		 * But because we didn't read anything, at this point we can
 		 * just return directly with -ERESTARTSYS if we're interrupted,
Quoted 7 lines

step-03
task-118740 is reader
makes pipe empty
sleeps with no writer woken up

After step-03, both reader(task-118740) and writer (task-118768) sleep waiting for each other, with Oleg's change.

Well. I have already tried to explain this at least twice :/ Prateek too.

After step-03 task-118740 won't sleep. pipe_read() won't sleep if it has read even one byte. Since the pipe was full and this reader makes it empty, "wake_writer" must be true after the main loop before return from pipe_read(). This means that the reader(task-118740) will wake the writer(task-118768) before it returns from pipe_read().

Oleg.

Quoted 6 lines

PS Oleg, given no seperate reply to you, check the above scenario instead please.

[2] https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/pipe.c#n576
[3] https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/pipe.c#n381
[4] https://lore.kernel.org/lkml/[email protected]/