Skip to contentSkip to conversation

arch: patch_text: Fixup last cpu should be master

PATCH Superseded [email protected] 4 participants 6 messages Mar 13, 2022 – Mar 23, 2022

Subsystems (5)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

PATCH v2 1/1 arch: patch_text: Fixup last cpu should be master

From: Guo Ren <[email protected]>

These patch_text implementations are using stop_machine_cpuslocked infrastructure with atomic cpu_count. The original idea: When the master CPU patch_text, the others should wait for it. But current implementation is using the first CPU as master, which couldn't guarantee the remaining CPUs are waiting. This patch changes the last CPU as the master to solve the potential risk.

Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Palmer Dabbelt <[email protected]>
Cc Peter Zijlstra <[email protected]
Cc Masami Hiramatsu <[email protected]>
Cc Chris Zankel <[email protected]>
Cc Arnd Bergmann <[email protected]>

---
Changes in V2:
- Fixup last cpu should be num_online_cpus() by Max Filippov
- Fixup typos found by Max Filippov

---
 arch/arm64/kernel/patching.c      | 4 ++--
 arch/csky/kernel/probes/kprobes.c | 2 +-
 arch/riscv/kernel/patch.c         | 2 +-
 arch/xtensa/kernel/jump_label.c   | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)
arch/arm64/kernel/patching.c +2 -2
index 771f543464e0..33e0fabc0b79 100644
@@ -117,8 +117,8 @@ static int __kprobes aarch64_insn_patch_text_cb(void *arg)
 	int i, ret = 0;
 	struct aarch64_insn_patch *pp = arg;
 
-	/* The first CPU becomes master */
-	if (atomic_inc_return(&pp->cpu_count) == 1) {
+	/* The last CPU becomes master */
+	if (atomic_inc_return(&pp->cpu_count) == num_online_cpus()) {
 		for (i = 0; ret == 0 && i < pp->insn_cnt; i++)
 			ret = aarch64_insn_patch_text_nosync(pp->text_addrs[i],
 							     pp->new_insns[i]);
arch/csky/kernel/probes/kprobes.c +1 -1
index 42920f25e73c..34ba684d5962 100644
@@ -30,7 +30,7 @@ static int __kprobes patch_text_cb(void *priv)
 	struct csky_insn_patch *param = priv;
 	unsigned int addr = (unsigned int)param->addr;
 
-	if (atomic_inc_return(&param->cpu_count) == 1) {
+	if (atomic_inc_return(&param->cpu_count) == num_online_cpus()) {
 		*(u16 *) addr = cpu_to_le16(param->opcode);
 		dcache_wb_range(addr, addr + 2);
 		atomic_inc(&param->cpu_count);
arch/riscv/kernel/patch.c +1 -1
index 0b552873a577..765004b60513 100644
@@ -104,7 +104,7 @@ static int patch_text_cb(void *data)
 	struct patch_insn *patch = data;
 	int ret = 0;
 
-	if (atomic_inc_return(&patch->cpu_count) == 1) {
+	if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) {
 		ret =
 		    patch_text_nosync(patch->addr, &patch->insn,
 					    GET_INSN_LENGTH(patch->insn));
arch/xtensa/kernel/jump_label.c +1 -1
index 61cf6497a646..b67efcd7e32c 100644
@@ -40,7 +40,7 @@ static int patch_text_stop_machine(void *data)
 {
 	struct patch *patch = data;
 
-	if (atomic_inc_return(&patch->cpu_count) == 1) {
+	if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) {
 		local_patch_text(patch->addr, patch->data, patch->sz);
 		atomic_inc(&patch->cpu_count);
 	} else {
Signature · 1 line
2.25.1
maintainer Reviewed-by on v2 1/1
#1 lore raw
Quoted 5 lines · On Sun, Mar 13, 2022 at 09:22:21AM +0800, [email protected] wrote:

From: Guo Ren <[email protected]>

These patch_text implementations are using stop_machine_cpuslocked infrastructure with atomic cpu_count. The original idea: When the master CPU patch_text, the others should wait for it.

I couldn't find the original intent in the commit logs (at least not in the arm64 logs). Maybe the intention was for the CPUs to wait for the text patching to complete rather than the master CPU to wait for the others to enter the cpu_relax() loop before patching.

I think your patch makes sense anyway, the master CPU would wait for all the others to enter the cpu_relax() loop before patching and releasing them with another increment. You probably wouldn't see any issues in practice unless you insert probes in the multi_stop_cpu() function (or we could mark this function as __kprobes and get rid of the extra loops entirely).

arch/arm64/kernel/patching.c static int __kprobes aarch64_insn_patch_text_cb(void *arg) · line 124

Quoted 13 lines
arch/arm64/kernel/patching.c +2 -2
@@ -117,8 +117,8 @@ static int __kprobes aarch64_insn_patch_text_cb(void *arg)
 	int i, ret = 0;
 	struct aarch64_insn_patch *pp = arg;
 
-	/* The first CPU becomes master */
-	if (atomic_inc_return(&pp->cpu_count) == 1) {
+	/* The last CPU becomes master */
+	if (atomic_inc_return(&pp->cpu_count) == num_online_cpus()) {
 		for (i = 0; ret == 0 && i < pp->insn_cnt; i++)
 			ret = aarch64_insn_patch_text_nosync(pp->text_addrs[i],
 							     pp->new_insns[i]);

For arm64:

Signature · 1 line
Catalin
author maintainer
#2 lore raw
Quoted 13 lines · Catalin Marinas
Quoted 5 lines · On Sun, Mar 13, 2022 at 09:22:21AM +0800, [email protected] wrote:

From: Guo Ren <[email protected]>

These patch_text implementations are using stop_machine_cpuslocked infrastructure with atomic cpu_count. The original idea: When the master CPU patch_text, the others should wait for it.

I couldn't find the original intent in the commit logs (at least not in the arm64 logs). Maybe the intention was for the CPUs to wait for the text patching to complete rather than the master CPU to wait for the others to enter the cpu_relax() loop before patching.

Quoted 6 lines

I think your patch makes sense anyway, the master CPU would wait for all the others to enter the cpu_relax() loop before patching and releasing them with another increment. You probably wouldn't see any issues in practice unless you insert probes in the multi_stop_cpu() function (or we could mark this function as __kprobes and get rid of the extra loops entirely).

That could depend on micro-arch, trigger other harts' IPI is not guaranteed by hw.

Quoted 17 lines
Quoted 4 lines

arch/arm64/kernel/patching.c static int __kprobes aarch64_insn_patch_text_cb(void *arg) · line 124

Quoted 10 lines
arch/arm64/kernel/patching.c +2 -2
@@ -117,8 +117,8 @@ static int __kprobes aarch64_insn_patch_text_cb(void *arg)
      int i, ret = 0;
      struct aarch64_insn_patch *pp = arg;
 
-     /* The first CPU becomes master */
-     if (atomic_inc_return(&pp->cpu_count) == 1) {
+     /* The last CPU becomes master */
+     if (atomic_inc_return(&pp->cpu_count) == num_online_cpus()) {
              for (i = 0; ret == 0 && i < pp->insn_cnt; i++)
                      ret = aarch64_insn_patch_text_nosync(pp->text_addrs[i],
                                                           pp->new_insns[i]);

For arm64:

Quoted 1 line

Thx

Quoted 3 lines

--
Catalin

Signature · 4 lines
Best Regards
 Guo Ren

ML: https://lore.kernel.org/linux-csky/
maintainer Acked-by on v2 1/1
#3 lore raw
In reply to [email protected]
Quoted 64 lines · On Sat, 12 Mar 2022 17:22:21 PST (-0800), [email protected] wrote:

From: Guo Ren <[email protected]>

These patch_text implementations are using stop_machine_cpuslocked infrastructure with atomic cpu_count. The original idea: When the master CPU patch_text, the others should wait for it. But current implementation is using the first CPU as master, which couldn't guarantee the remaining CPUs are waiting. This patch changes the last CPU as the master to solve the potential risk.

Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Palmer Dabbelt <[email protected]>
Cc Peter Zijlstra <[email protected]
Cc Masami Hiramatsu <[email protected]>
Cc Chris Zankel <[email protected]>
Cc Arnd Bergmann <[email protected]>

---
Changes in V2:
- Fixup last cpu should be num_online_cpus() by Max Filippov
- Fixup typos found by Max Filippov

---
 arch/arm64/kernel/patching.c      | 4 ++--
 arch/csky/kernel/probes/kprobes.c | 2 +-
 arch/riscv/kernel/patch.c         | 2 +-
 arch/xtensa/kernel/jump_label.c   | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)
arch/arm64/kernel/patching.c +2 -2
index 771f543464e0..33e0fabc0b79 100644
@@ -117,8 +117,8 @@ static int __kprobes aarch64_insn_patch_text_cb(void *arg)
 	int i, ret = 0;
 	struct aarch64_insn_patch *pp = arg;
 
-	/* The first CPU becomes master */
-	if (atomic_inc_return(&pp->cpu_count) == 1) {
+	/* The last CPU becomes master */
+	if (atomic_inc_return(&pp->cpu_count) == num_online_cpus()) {
 		for (i = 0; ret == 0 && i < pp->insn_cnt; i++)
 			ret = aarch64_insn_patch_text_nosync(pp->text_addrs[i],
 							     pp->new_insns[i]);
arch/csky/kernel/probes/kprobes.c +1 -1
index 42920f25e73c..34ba684d5962 100644
@@ -30,7 +30,7 @@ static int __kprobes patch_text_cb(void *priv)
 	struct csky_insn_patch *param = priv;
 	unsigned int addr = (unsigned int)param->addr;
 
-	if (atomic_inc_return(&param->cpu_count) == 1) {
+	if (atomic_inc_return(&param->cpu_count) == num_online_cpus()) {
 		*(u16 *) addr = cpu_to_le16(param->opcode);
 		dcache_wb_range(addr, addr + 2);
 		atomic_inc(&param->cpu_count);
arch/riscv/kernel/patch.c +0 -0
index 0b552873a577..765004b60513 100644

arch/riscv/kernel/patch.c static int patch_text_cb(void *data) · line 110

Quoted 8 lines
arch/riscv/kernel/patch.c +1 -1
@@ -104,7 +104,7 @@ static int patch_text_cb(void *data)
 	struct patch_insn *patch = data;
 	int ret = 0;
 
-	if (atomic_inc_return(&patch->cpu_count) == 1) {
+	if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) {
 		ret =
 		    patch_text_nosync(patch->addr, &patch->insn,
 					    GET_INSN_LENGTH(patch->insn));

It's better if these sorts of things get split up: there's really no dependency between these diffs and having them together just makes for a merge/test headache for everyone.

I'm OK taking this through the RISC-V tree if other folks ack it, but for now I'm going to assume it's going to go in via somewhere else.

Quoted 13 lines
arch/xtensa/kernel/jump_label.c +1 -1
index 61cf6497a646..b67efcd7e32c 100644
@@ -40,7 +40,7 @@ static int patch_text_stop_machine(void *data)
 {
 	struct patch *patch = data;
 
-	if (atomic_inc_return(&patch->cpu_count) == 1) {
+	if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) {
 		local_patch_text(patch->addr, patch->data, patch->sz);
 		atomic_inc(&patch->cpu_count);
 	} else {
author maintainer
#4 lore raw
Quoted 77 lines · Palmer Dabbelt
Quoted 64 lines · On Sat, 12 Mar 2022 17:22:21 PST (-0800), [email protected] wrote:

From: Guo Ren <[email protected]>

These patch_text implementations are using stop_machine_cpuslocked infrastructure with atomic cpu_count. The original idea: When the master CPU patch_text, the others should wait for it. But current implementation is using the first CPU as master, which couldn't guarantee the remaining CPUs are waiting. This patch changes the last CPU as the master to solve the potential risk.

Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Palmer Dabbelt <[email protected]>
Cc Peter Zijlstra <[email protected]
Cc Masami Hiramatsu <[email protected]>
Cc Chris Zankel <[email protected]>
Cc Arnd Bergmann <[email protected]>

---
Changes in V2:
- Fixup last cpu should be num_online_cpus() by Max Filippov
- Fixup typos found by Max Filippov

---
 arch/arm64/kernel/patching.c      | 4 ++--
 arch/csky/kernel/probes/kprobes.c | 2 +-
 arch/riscv/kernel/patch.c         | 2 +-
 arch/xtensa/kernel/jump_label.c   | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)
arch/arm64/kernel/patching.c +2 -2
index 771f543464e0..33e0fabc0b79 100644
@@ -117,8 +117,8 @@ static int __kprobes aarch64_insn_patch_text_cb(void *arg)
      int i, ret = 0;
      struct aarch64_insn_patch *pp = arg;
 
-     /* The first CPU becomes master */
-     if (atomic_inc_return(&pp->cpu_count) == 1) {
+     /* The last CPU becomes master */
+     if (atomic_inc_return(&pp->cpu_count) == num_online_cpus()) {
              for (i = 0; ret == 0 && i < pp->insn_cnt; i++)
                      ret = aarch64_insn_patch_text_nosync(pp->text_addrs[i],
                                                           pp->new_insns[i]);
arch/csky/kernel/probes/kprobes.c +1 -1
index 42920f25e73c..34ba684d5962 100644
@@ -30,7 +30,7 @@ static int __kprobes patch_text_cb(void *priv)
      struct csky_insn_patch *param = priv;
      unsigned int addr = (unsigned int)param->addr;
 
-     if (atomic_inc_return(&param->cpu_count) == 1) {
+     if (atomic_inc_return(&param->cpu_count) == num_online_cpus()) {
              *(u16 *) addr = cpu_to_le16(param->opcode);
              dcache_wb_range(addr, addr + 2);
              atomic_inc(&param->cpu_count);
arch/riscv/kernel/patch.c +0 -0
index 0b552873a577..765004b60513 100644

arch/riscv/kernel/patch.c static int patch_text_cb(void *data) · line 110

Quoted 8 lines
arch/riscv/kernel/patch.c +1 -1
@@ -104,7 +104,7 @@ static int patch_text_cb(void *data)
      struct patch_insn *patch = data;
      int ret = 0;
 
-     if (atomic_inc_return(&patch->cpu_count) == 1) {
+     if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) {
              ret =
                  patch_text_nosync(patch->addr, &patch->insn,
                                          GET_INSN_LENGTH(patch->insn));
Quoted 3 lines

It's better if these sorts of things get split up: there's really no dependency between these diffs and having them together just makes for a merge/test headache for everyone.

I'm Okay with split patches, @Arnd Bergmann what's your opinion? We've got all arch vendors' agreements (arm64, csky, riscv, xtensa).

Quoted 3 lines

I'm OK taking this through the RISC-V tree if other folks ack it, but for now I'm going to assume it's going to go in via somewhere else.

Arnd has given some comments on unnecessary #error, maybe I need to update V2.

Quoted 14 lines
Quoted 14 lines
arch/xtensa/kernel/jump_label.c +1 -1
index 61cf6497a646..b67efcd7e32c 100644
@@ -40,7 +40,7 @@ static int patch_text_stop_machine(void *data)
 {
      struct patch *patch = data;
 
-     if (atomic_inc_return(&patch->cpu_count) == 1) {
+     if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) {
              local_patch_text(patch->addr, patch->data, patch->sz);
              atomic_inc(&patch->cpu_count);
      } else {
Signature · 4 lines
Best Regards
 Guo Ren

ML: https://lore.kernel.org/linux-csky/
Reviewed-by on v2 1/1
#5 lore raw
In reply to [email protected]
Quoted 9 lines · On Sun, 13 Mar 2022 09:22:21 +0800 [email protected] wrote:

From: Guo Ren <[email protected]>

These patch_text implementations are using stop_machine_cpuslocked infrastructure with atomic cpu_count. The original idea: When the master CPU patch_text, the others should wait for it. But current implementation is using the first CPU as master, which couldn't guarantee the remaining CPUs are waiting. This patch changes the last CPU as the master to solve the potential risk.

This looks good to me. And maybe we'd better backport to stable branch this since this can cause self-modification in wrong timing.

Thank you,

Quoted 79 lines
Cc Will Deacon <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Palmer Dabbelt <[email protected]>
Cc Peter Zijlstra <[email protected]
Cc Masami Hiramatsu <[email protected]>
Cc Chris Zankel <[email protected]>
Cc Arnd Bergmann <[email protected]>

---
Changes in V2:
- Fixup last cpu should be num_online_cpus() by Max Filippov
- Fixup typos found by Max Filippov

---
 arch/arm64/kernel/patching.c      | 4 ++--
 arch/csky/kernel/probes/kprobes.c | 2 +-
 arch/riscv/kernel/patch.c         | 2 +-
 arch/xtensa/kernel/jump_label.c   | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)
arch/arm64/kernel/patching.c +2 -2
index 771f543464e0..33e0fabc0b79 100644
@@ -117,8 +117,8 @@ static int __kprobes aarch64_insn_patch_text_cb(void *arg)
 	int i, ret = 0;
 	struct aarch64_insn_patch *pp = arg;
 
-	/* The first CPU becomes master */
-	if (atomic_inc_return(&pp->cpu_count) == 1) {
+	/* The last CPU becomes master */
+	if (atomic_inc_return(&pp->cpu_count) == num_online_cpus()) {
 		for (i = 0; ret == 0 && i < pp->insn_cnt; i++)
 			ret = aarch64_insn_patch_text_nosync(pp->text_addrs[i],
 							     pp->new_insns[i]);
arch/csky/kernel/probes/kprobes.c +1 -1
index 42920f25e73c..34ba684d5962 100644
@@ -30,7 +30,7 @@ static int __kprobes patch_text_cb(void *priv)
 	struct csky_insn_patch *param = priv;
 	unsigned int addr = (unsigned int)param->addr;
 
-	if (atomic_inc_return(&param->cpu_count) == 1) {
+	if (atomic_inc_return(&param->cpu_count) == num_online_cpus()) {
 		*(u16 *) addr = cpu_to_le16(param->opcode);
 		dcache_wb_range(addr, addr + 2);
 		atomic_inc(&param->cpu_count);
arch/riscv/kernel/patch.c +1 -1
index 0b552873a577..765004b60513 100644
@@ -104,7 +104,7 @@ static int patch_text_cb(void *data)
 	struct patch_insn *patch = data;
 	int ret = 0;
 
-	if (atomic_inc_return(&patch->cpu_count) == 1) {
+	if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) {
 		ret =
 		    patch_text_nosync(patch->addr, &patch->insn,
 					    GET_INSN_LENGTH(patch->insn));
arch/xtensa/kernel/jump_label.c +1 -1
index 61cf6497a646..b67efcd7e32c 100644
@@ -40,7 +40,7 @@ static int patch_text_stop_machine(void *data)
 {
 	struct patch *patch = data;
 
-	if (atomic_inc_return(&patch->cpu_count) == 1) {
+	if (atomic_inc_return(&patch->cpu_count) == num_online_cpus()) {
 		local_patch_text(patch->addr, patch->data, patch->sz);
 		atomic_inc(&patch->cpu_count);
 	} else {
Signature · 1 line
2.25.1
Signature · 1 line
Masami Hiramatsu <[email protected]>