Skip to contentSkip to conversation

s390x: Add multiboot and smp

PATCH Superseded Janosch Frank 3 participants 21 messages Sep 5, 2019 – Sep 19, 2019

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover s390x: Add multiboot and smp

Cross testing emulated instructions has in the past brought up some issues on all available IBM Z hypervisors. So let's upstream the last part of multiboot: sclp interrupts and line mode console.

SMP tests are a great way to excercise external interruptions, cpu resets and sigp. The smp library is always initialized and provides very rudimentary CPU management for now.

v2:
* Fixed cr0 masks
* Replaced gotos with loops
* Addressed other review comments

Janosch Frank (6):
s390x: Use interrupts in SCLP and add locking
s390x: Add linemode console
s390x: Add linemode buffer to fix newline on every print
s390x: Add initial smp code
s390x: Prepare for external calls
s390x: SMP test

 lib/s390x/asm/arch_def.h  |  13 ++
 lib/s390x/asm/interrupt.h |   5 +
 lib/s390x/asm/sigp.h      |  28 +++-
 lib/s390x/interrupt.c     |  28 +++-
 lib/s390x/io.c            |   5 +-
 lib/s390x/sclp-console.c  | 243 ++++++++++++++++++++++++++++++---
 lib/s390x/sclp.c          |  55 +++++++-
 lib/s390x/sclp.h          |  59 +++++++-
 lib/s390x/smp.c           | 276 ++++++++++++++++++++++++++++++++++++++
 lib/s390x/smp.h           |  51 +++++++
 s390x/Makefile            |   2 +
 s390x/cstart64.S          |   7 +
 s390x/smp.c               | 242 +++++++++++++++++++++++++++++++++
 s390x/unittests.cfg       |   4 +
 14 files changed, 991 insertions(+), 27 deletions(-)
 create mode 100644 lib/s390x/smp.c
 create mode 100644 lib/s390x/smp.h
 create mode 100644 s390x/smp.c
Signature · 1 line
2.17.0
#1 lore raw

PATCH v2 1/6 s390x: Use interrupts in SCLP and add locking

We need to properly implement interrupt handling for SCLP, because on z/VM and LPAR SCLP calls are not synchronous!

Also with smp CPUs have to compete for sclp. Let's add some locking, so they execute sclp calls in an orderly fashion and don't compete for the data buffer.

---
 lib/s390x/asm/interrupt.h |  2 ++
 lib/s390x/interrupt.c     | 12 +++++++--
 lib/s390x/sclp-console.c  |  2 ++
 lib/s390x/sclp.c          | 55 +++++++++++++++++++++++++++++++++++++--
 lib/s390x/sclp.h          |  3 +++
 5 files changed, 70 insertions(+), 4 deletions(-)
lib/s390x/asm/interrupt.h +2 -0
index 013709f..f485e96 100644
@@ -11,6 +11,8 @@
 #define _ASMS390X_IRQ_H_
 #include <asm/arch_def.h>
 
+#define EXT_IRQ_SERVICE_SIG	0x2401
+
 void handle_pgm_int(void);
 void handle_ext_int(void);
 void handle_mcck_int(void);
lib/s390x/interrupt.c +10 -2
index cf0a794..7832711 100644
@@ -12,6 +12,7 @@
 #include <libcflat.h>
 #include <asm/interrupt.h>
 #include <asm/barrier.h>
+#include <sclp.h>
 
 static bool pgm_int_expected;
 static struct lowcore *lc;
@@ -107,8 +108,15 @@ void handle_pgm_int(void)
 
 void handle_ext_int(void)
 {
-	report_abort("Unexpected external call interrupt: at %#lx",
-		     lc->ext_old_psw.addr);
+	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
+		report_abort("Unexpected external call interrupt: at %#lx",
+			     lc->ext_old_psw.addr);
+	} else {
+		lc->ext_old_psw.mask &= ~PSW_MASK_EXT;
+		lc->sw_int_cr0 &= ~(1UL << 9);
+		sclp_handle_ext();
+		lc->ext_int_code = 0;
+	}
 }
 
 void handle_mcck_int(void)
lib/s390x/sclp-console.c +2 -0
index bc01f41..a5ef45f 100644
@@ -17,6 +17,7 @@ static void sclp_set_write_mask(void)
 {
 	WriteEventMask *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventMask);
 	sccb->mask_length = sizeof(unsigned int);
 	sccb->receive_mask = SCLP_EVENT_MASK_MSG_ASCII;
@@ -37,6 +38,7 @@ void sclp_print(const char *str)
 	int len = strlen(str);
 	WriteEventData *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventData) + len;
 	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
 	sccb->ebh.length = sizeof(EventBufferHeader) + len;
lib/s390x/sclp.c +53 -2

94 lines not on this page · raw message

lib/s390x/sclp.h +3 -0
index 583c4e5..63cf609 100644
@@ -213,6 +213,9 @@ typedef struct ReadEventData {
 } __attribute__((packed)) ReadEventData;
 
 extern char _sccb[];
+void sclp_handle_ext(void);
+void sclp_wait_busy(void);
+void sclp_mark_busy(void);
 void sclp_console_setup(void);
 void sclp_print(const char *str);
 int sclp_service_call(unsigned int command, void *sccb);
Signature · 1 line
2.17.0
Reviewed-by on v2 1/6
#2 lore raw

Re: [kvm-unit-tests PATCH v2 1/6] s390x: Use interrupts in SCLP and add locking

Quoted 7 lines · Janosch Frank

We need to properly implement interrupt handling for SCLP, because on z/VM and LPAR SCLP calls are not synchronous!

Also with smp CPUs have to compete for sclp. Let's add some locking, so they execute sclp calls in an orderly fashion and don't compete for the data buffer.

Quoted 8 lines
---
 lib/s390x/asm/interrupt.h |  2 ++
 lib/s390x/interrupt.c     | 12 +++++++--
 lib/s390x/sclp-console.c  |  2 ++
 lib/s390x/sclp.c          | 55 +++++++++++++++++++++++++++++++++++++--
 lib/s390x/sclp.h          |  3 +++
 5 files changed, 70 insertions(+), 4 deletions(-)
#3 lore raw

Re: [kvm-unit-tests PATCH v2 1/6] s390x: Use interrupts in SCLP and add locking

In reply to Janosch Frank
Quoted 187 lines · Janosch Frank

We need to properly implement interrupt handling for SCLP, because on z/VM and LPAR SCLP calls are not synchronous!

Also with smp CPUs have to compete for sclp. Let's add some locking, so they execute sclp calls in an orderly fashion and don't compete for the data buffer.

---
 lib/s390x/asm/interrupt.h |  2 ++
 lib/s390x/interrupt.c     | 12 +++++++--
 lib/s390x/sclp-console.c  |  2 ++
 lib/s390x/sclp.c          | 55 +++++++++++++++++++++++++++++++++++++--
 lib/s390x/sclp.h          |  3 +++
 5 files changed, 70 insertions(+), 4 deletions(-)
lib/s390x/asm/interrupt.h +2 -0
index 013709f..f485e96 100644
@@ -11,6 +11,8 @@
 #define _ASMS390X_IRQ_H_
 #include <asm/arch_def.h>
 
+#define EXT_IRQ_SERVICE_SIG	0x2401
+
 void handle_pgm_int(void);
 void handle_ext_int(void);
 void handle_mcck_int(void);
lib/s390x/interrupt.c +10 -2
index cf0a794..7832711 100644
@@ -12,6 +12,7 @@
 #include <libcflat.h>
 #include <asm/interrupt.h>
 #include <asm/barrier.h>
+#include <sclp.h>
 
 static bool pgm_int_expected;
 static struct lowcore *lc;
@@ -107,8 +108,15 @@ void handle_pgm_int(void)
 
 void handle_ext_int(void)
 {
-	report_abort("Unexpected external call interrupt: at %#lx",
-		     lc->ext_old_psw.addr);
+	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
+		report_abort("Unexpected external call interrupt: at %#lx",
+			     lc->ext_old_psw.addr);
+	} else {
+		lc->ext_old_psw.mask &= ~PSW_MASK_EXT;
+		lc->sw_int_cr0 &= ~(1UL << 9);
+		sclp_handle_ext();
+		lc->ext_int_code = 0;
+	}
 }
 
 void handle_mcck_int(void)
lib/s390x/sclp-console.c +2 -0
index bc01f41..a5ef45f 100644
@@ -17,6 +17,7 @@ static void sclp_set_write_mask(void)
 {
 	WriteEventMask *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventMask);
 	sccb->mask_length = sizeof(unsigned int);
 	sccb->receive_mask = SCLP_EVENT_MASK_MSG_ASCII;
@@ -37,6 +38,7 @@ void sclp_print(const char *str)
 	int len = strlen(str);
 	WriteEventData *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventData) + len;
 	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
 	sccb->ebh.length = sizeof(EventBufferHeader) + len;
lib/s390x/sclp.c +53 -2
index b60f7a4..56fca0c 100644
@@ -14,6 +14,8 @@
 #include <asm/page.h>
 #include <asm/arch_def.h>
 #include <asm/interrupt.h>
+#include <asm/barrier.h>
+#include <asm/spinlock.h>
 #include "sclp.h"
 #include <alloc_phys.h>
 #include <alloc_page.h>
@@ -25,6 +27,8 @@ static uint64_t max_ram_size;
 static uint64_t ram_size;
 
 char _sccb[PAGE_SIZE] __attribute__((__aligned__(4096)));
+static volatile bool sclp_busy;
+static struct spinlock sclp_lock;
 
 static void mem_init(phys_addr_t mem_end)
 {
@@ -41,17 +45,62 @@ static void mem_init(phys_addr_t mem_end)
 	page_alloc_ops_enable();
 }
 
+static void sclp_setup_int(void)
+{
+	uint64_t mask;
+
+	ctl_set_bit(0, 9);
+
+	mask = extract_psw_mask();
+	mask |= PSW_MASK_EXT;
+	load_psw_mask(mask);
+}
+
+void sclp_handle_ext(void)
+{
+	ctl_clear_bit(0, 9);
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
+void sclp_wait_busy(void)
+{
+	while (sclp_busy)
+		mb();
+}
+
+void sclp_mark_busy(void)
+{
+	/*
+	 * With multiple CPUs we might need to wait for another CPU's
+	 * request before grabbing the busy indication.
+	 */
+	while (true) {
+		sclp_wait_busy();
+		spin_lock(&sclp_lock);
+		if (!sclp_busy) {
+			sclp_busy = true;
+			spin_unlock(&sclp_lock);
+			return;
+		}
+		spin_unlock(&sclp_lock);
+	}
+}
+
 static void sclp_read_scp_info(ReadInfo *ri, int length)
 {
 	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
 				    SCLP_CMDW_READ_SCP_INFO };
-	int i;
+	int i, cc;
 
 	for (i = 0; i < ARRAY_SIZE(commands); i++) {
+		sclp_mark_busy();
 		memset(&ri->h, 0, sizeof(ri->h));
 		ri->h.length = length;
 
-		if (sclp_service_call(commands[i], ri))
+		cc = sclp_service_call(commands[i], ri);
+		if (cc)
 			break;
 		if (ri->h.response_code == SCLP_RC_NORMAL_READ_COMPLETION)
 			return;
@@ -66,12 +115,14 @@ int sclp_service_call(unsigned int command, void *sccb)
 {
 	int cc;
 
+	sclp_setup_int();
 	asm volatile(
 		"       .insn   rre,0xb2200000,%1,%2\n"  /* servc %1,%2 */
 		"       ipm     %0\n"
 		"       srl     %0,28"
 		: "=&d" (cc) : "d" (command), "a" (__pa(sccb))
 		: "cc", "memory");
+	sclp_wait_busy();
 	if (cc == 3)
 		return -1;
 	if (cc == 2)
lib/s390x/sclp.h +0 -0
index 583c4e5..63cf609 100644

lib/s390x/sclp.h typedef struct ReadEventData { · line 221

Quoted 9 lines
lib/s390x/sclp.h +3 -0
@@ -213,6 +213,9 @@ typedef struct ReadEventData {
 } __attribute__((packed)) ReadEventData;
 
 extern char _sccb[];
+void sclp_handle_ext(void);
+void sclp_wait_busy(void);
+void sclp_mark_busy(void);
 void sclp_console_setup(void);
 void sclp_print(const char *str);
 int sclp_service_call(unsigned int command, void *sccb);

I was wondering whether it would make sense to enable sclp interrupts as default for all CPUs (once in a reasonable state after brought up), and simply let any CPU process the request. Initially, we could only let the boot CPU handle them.

You already decoupled sclp_mark_busy() and sclp_setup_int() already. The part would have to be moved to the CPU init stage and sclp_handle_ext() would simply not clear the interrupt-enable flag.

Opinions?

Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw

Re: [kvm-unit-tests PATCH v2 1/6] s390x: Use interrupts in SCLP and add locking

Quoted 203 lines · David Hildenbrand
Quoted 187 lines · Janosch Frank

We need to properly implement interrupt handling for SCLP, because on z/VM and LPAR SCLP calls are not synchronous!

Also with smp CPUs have to compete for sclp. Let's add some locking, so they execute sclp calls in an orderly fashion and don't compete for the data buffer.

---
 lib/s390x/asm/interrupt.h |  2 ++
 lib/s390x/interrupt.c     | 12 +++++++--
 lib/s390x/sclp-console.c  |  2 ++
 lib/s390x/sclp.c          | 55 +++++++++++++++++++++++++++++++++++++--
 lib/s390x/sclp.h          |  3 +++
 5 files changed, 70 insertions(+), 4 deletions(-)
lib/s390x/asm/interrupt.h +2 -0
index 013709f..f485e96 100644
@@ -11,6 +11,8 @@
 #define _ASMS390X_IRQ_H_
 #include <asm/arch_def.h>
 
+#define EXT_IRQ_SERVICE_SIG	0x2401
+
 void handle_pgm_int(void);
 void handle_ext_int(void);
 void handle_mcck_int(void);
lib/s390x/interrupt.c +10 -2
index cf0a794..7832711 100644
@@ -12,6 +12,7 @@
 #include <libcflat.h>
 #include <asm/interrupt.h>
 #include <asm/barrier.h>
+#include <sclp.h>
 
 static bool pgm_int_expected;
 static struct lowcore *lc;
@@ -107,8 +108,15 @@ void handle_pgm_int(void)
 
 void handle_ext_int(void)
 {
-	report_abort("Unexpected external call interrupt: at %#lx",
-		     lc->ext_old_psw.addr);
+	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
+		report_abort("Unexpected external call interrupt: at %#lx",
+			     lc->ext_old_psw.addr);
+	} else {
+		lc->ext_old_psw.mask &= ~PSW_MASK_EXT;
+		lc->sw_int_cr0 &= ~(1UL << 9);
+		sclp_handle_ext();
+		lc->ext_int_code = 0;
+	}
 }
 
 void handle_mcck_int(void)
lib/s390x/sclp-console.c +2 -0
index bc01f41..a5ef45f 100644
@@ -17,6 +17,7 @@ static void sclp_set_write_mask(void)
 {
 	WriteEventMask *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventMask);
 	sccb->mask_length = sizeof(unsigned int);
 	sccb->receive_mask = SCLP_EVENT_MASK_MSG_ASCII;
@@ -37,6 +38,7 @@ void sclp_print(const char *str)
 	int len = strlen(str);
 	WriteEventData *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventData) + len;
 	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
 	sccb->ebh.length = sizeof(EventBufferHeader) + len;
lib/s390x/sclp.c +53 -2
index b60f7a4..56fca0c 100644
@@ -14,6 +14,8 @@
 #include <asm/page.h>
 #include <asm/arch_def.h>
 #include <asm/interrupt.h>
+#include <asm/barrier.h>
+#include <asm/spinlock.h>
 #include "sclp.h"
 #include <alloc_phys.h>
 #include <alloc_page.h>
@@ -25,6 +27,8 @@ static uint64_t max_ram_size;
 static uint64_t ram_size;
 
 char _sccb[PAGE_SIZE] __attribute__((__aligned__(4096)));
+static volatile bool sclp_busy;
+static struct spinlock sclp_lock;
 
 static void mem_init(phys_addr_t mem_end)
 {
@@ -41,17 +45,62 @@ static void mem_init(phys_addr_t mem_end)
 	page_alloc_ops_enable();
 }
 
+static void sclp_setup_int(void)
+{
+	uint64_t mask;
+
+	ctl_set_bit(0, 9);
+
+	mask = extract_psw_mask();
+	mask |= PSW_MASK_EXT;
+	load_psw_mask(mask);
+}
+
+void sclp_handle_ext(void)
+{
+	ctl_clear_bit(0, 9);
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
+void sclp_wait_busy(void)
+{
+	while (sclp_busy)
+		mb();
+}
+
+void sclp_mark_busy(void)
+{
+	/*
+	 * With multiple CPUs we might need to wait for another CPU's
+	 * request before grabbing the busy indication.
+	 */
+	while (true) {
+		sclp_wait_busy();
+		spin_lock(&sclp_lock);
+		if (!sclp_busy) {
+			sclp_busy = true;
+			spin_unlock(&sclp_lock);
+			return;
+		}
+		spin_unlock(&sclp_lock);
+	}
+}
+
 static void sclp_read_scp_info(ReadInfo *ri, int length)
 {
 	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
 				    SCLP_CMDW_READ_SCP_INFO };
-	int i;
+	int i, cc;
 
 	for (i = 0; i < ARRAY_SIZE(commands); i++) {
+		sclp_mark_busy();
 		memset(&ri->h, 0, sizeof(ri->h));
 		ri->h.length = length;
 
-		if (sclp_service_call(commands[i], ri))
+		cc = sclp_service_call(commands[i], ri);
+		if (cc)
 			break;
 		if (ri->h.response_code == SCLP_RC_NORMAL_READ_COMPLETION)
 			return;
@@ -66,12 +115,14 @@ int sclp_service_call(unsigned int command, void *sccb)
 {
 	int cc;
 
+	sclp_setup_int();
 	asm volatile(
 		"       .insn   rre,0xb2200000,%1,%2\n"  /* servc %1,%2 */
 		"       ipm     %0\n"
 		"       srl     %0,28"
 		: "=&d" (cc) : "d" (command), "a" (__pa(sccb))
 		: "cc", "memory");
+	sclp_wait_busy();
 	if (cc == 3)
 		return -1;
 	if (cc == 2)
lib/s390x/sclp.h +0 -0
index 583c4e5..63cf609 100644

lib/s390x/sclp.h typedef struct ReadEventData { · line 221

Quoted 9 lines
lib/s390x/sclp.h +3 -0
@@ -213,6 +213,9 @@ typedef struct ReadEventData {
 } __attribute__((packed)) ReadEventData;
 
 extern char _sccb[];
+void sclp_handle_ext(void);
+void sclp_wait_busy(void);
+void sclp_mark_busy(void);
 void sclp_console_setup(void);
 void sclp_print(const char *str);
 int sclp_service_call(unsigned int command, void *sccb);

I was wondering whether it would make sense to enable sclp interrupts as default for all CPUs (once in a reasonable state after brought up), and simply let any CPU process the request. Initially, we could only let the boot CPU handle them.

Quoted 6 lines

You already decoupled sclp_mark_busy() and sclp_setup_int() already. The part would have to be moved to the CPU init stage and sclp_handle_ext() would simply not clear the interrupt-enable flag.

Opinions?

OTOH, the s390x-ccw bios enables interrupts on the single cpu after sending the request, and disables them again in the interrupt handler. I guess we should never get more than one interrupt per SCLP request?

Signature · 3 lines
Thanks,

David / dhildenb
#5 lore raw

Re: [kvm-unit-tests PATCH v2 1/6] s390x: Use interrupts in SCLP and add locking

Quoted 208 lines · David Hildenbrand
Quoted 207 lines · David Hildenbrand
Quoted 187 lines · Janosch Frank

We need to properly implement interrupt handling for SCLP, because on z/VM and LPAR SCLP calls are not synchronous!

Also with smp CPUs have to compete for sclp. Let's add some locking, so they execute sclp calls in an orderly fashion and don't compete for the data buffer.

---
 lib/s390x/asm/interrupt.h |  2 ++
 lib/s390x/interrupt.c     | 12 +++++++--
 lib/s390x/sclp-console.c  |  2 ++
 lib/s390x/sclp.c          | 55 +++++++++++++++++++++++++++++++++++++--
 lib/s390x/sclp.h          |  3 +++
 5 files changed, 70 insertions(+), 4 deletions(-)
lib/s390x/asm/interrupt.h +2 -0
index 013709f..f485e96 100644
@@ -11,6 +11,8 @@
 #define _ASMS390X_IRQ_H_
 #include <asm/arch_def.h>
 
+#define EXT_IRQ_SERVICE_SIG	0x2401
+
 void handle_pgm_int(void);
 void handle_ext_int(void);
 void handle_mcck_int(void);
lib/s390x/interrupt.c +10 -2
index cf0a794..7832711 100644
@@ -12,6 +12,7 @@
 #include <libcflat.h>
 #include <asm/interrupt.h>
 #include <asm/barrier.h>
+#include <sclp.h>
 
 static bool pgm_int_expected;
 static struct lowcore *lc;
@@ -107,8 +108,15 @@ void handle_pgm_int(void)
 
 void handle_ext_int(void)
 {
-	report_abort("Unexpected external call interrupt: at %#lx",
-		     lc->ext_old_psw.addr);
+	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
+		report_abort("Unexpected external call interrupt: at %#lx",
+			     lc->ext_old_psw.addr);
+	} else {
+		lc->ext_old_psw.mask &= ~PSW_MASK_EXT;
+		lc->sw_int_cr0 &= ~(1UL << 9);
+		sclp_handle_ext();
+		lc->ext_int_code = 0;
+	}
 }
 
 void handle_mcck_int(void)
lib/s390x/sclp-console.c +2 -0
index bc01f41..a5ef45f 100644
@@ -17,6 +17,7 @@ static void sclp_set_write_mask(void)
 {
 	WriteEventMask *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventMask);
 	sccb->mask_length = sizeof(unsigned int);
 	sccb->receive_mask = SCLP_EVENT_MASK_MSG_ASCII;
@@ -37,6 +38,7 @@ void sclp_print(const char *str)
 	int len = strlen(str);
 	WriteEventData *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventData) + len;
 	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
 	sccb->ebh.length = sizeof(EventBufferHeader) + len;
lib/s390x/sclp.c +53 -2
index b60f7a4..56fca0c 100644
@@ -14,6 +14,8 @@
 #include <asm/page.h>
 #include <asm/arch_def.h>
 #include <asm/interrupt.h>
+#include <asm/barrier.h>
+#include <asm/spinlock.h>
 #include "sclp.h"
 #include <alloc_phys.h>
 #include <alloc_page.h>
@@ -25,6 +27,8 @@ static uint64_t max_ram_size;
 static uint64_t ram_size;
 
 char _sccb[PAGE_SIZE] __attribute__((__aligned__(4096)));
+static volatile bool sclp_busy;
+static struct spinlock sclp_lock;
 
 static void mem_init(phys_addr_t mem_end)
 {
@@ -41,17 +45,62 @@ static void mem_init(phys_addr_t mem_end)
 	page_alloc_ops_enable();
 }
 
+static void sclp_setup_int(void)
+{
+	uint64_t mask;
+
+	ctl_set_bit(0, 9);
+
+	mask = extract_psw_mask();
+	mask |= PSW_MASK_EXT;
+	load_psw_mask(mask);
+}
+
+void sclp_handle_ext(void)
+{
+	ctl_clear_bit(0, 9);
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
+void sclp_wait_busy(void)
+{
+	while (sclp_busy)
+		mb();
+}
+
+void sclp_mark_busy(void)
+{
+	/*
+	 * With multiple CPUs we might need to wait for another CPU's
+	 * request before grabbing the busy indication.
+	 */
+	while (true) {
+		sclp_wait_busy();
+		spin_lock(&sclp_lock);
+		if (!sclp_busy) {
+			sclp_busy = true;
+			spin_unlock(&sclp_lock);
+			return;
+		}
+		spin_unlock(&sclp_lock);
+	}
+}
+
 static void sclp_read_scp_info(ReadInfo *ri, int length)
 {
 	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
 				    SCLP_CMDW_READ_SCP_INFO };
-	int i;
+	int i, cc;
 
 	for (i = 0; i < ARRAY_SIZE(commands); i++) {
+		sclp_mark_busy();
 		memset(&ri->h, 0, sizeof(ri->h));
 		ri->h.length = length;
 
-		if (sclp_service_call(commands[i], ri))
+		cc = sclp_service_call(commands[i], ri);
+		if (cc)
 			break;
 		if (ri->h.response_code == SCLP_RC_NORMAL_READ_COMPLETION)
 			return;
@@ -66,12 +115,14 @@ int sclp_service_call(unsigned int command, void *sccb)
 {
 	int cc;
 
+	sclp_setup_int();
 	asm volatile(
 		"       .insn   rre,0xb2200000,%1,%2\n"  /* servc %1,%2 */
 		"       ipm     %0\n"
 		"       srl     %0,28"
 		: "=&d" (cc) : "d" (command), "a" (__pa(sccb))
 		: "cc", "memory");
+	sclp_wait_busy();
 	if (cc == 3)
 		return -1;
 	if (cc == 2)
lib/s390x/sclp.h +0 -0
index 583c4e5..63cf609 100644

lib/s390x/sclp.h typedef struct ReadEventData { · line 221

Quoted 9 lines
lib/s390x/sclp.h +3 -0
@@ -213,6 +213,9 @@ typedef struct ReadEventData {
 } __attribute__((packed)) ReadEventData;
 
 extern char _sccb[];
+void sclp_handle_ext(void);
+void sclp_wait_busy(void);
+void sclp_mark_busy(void);
 void sclp_console_setup(void);
 void sclp_print(const char *str);
 int sclp_service_call(unsigned int command, void *sccb);

I was wondering whether it would make sense to enable sclp interrupts as default for all CPUs (once in a reasonable state after brought up), and simply let any CPU process the request. Initially, we could only let the boot CPU handle them.

You already decoupled sclp_mark_busy() and sclp_setup_int() already. The part would have to be moved to the CPU init stage and sclp_handle_ext() would simply not clear the interrupt-enable flag.

Quoted 7 lines
Quoted 2 lines

Opinions?

OTOH, the s390x-ccw bios enables interrupts on the single cpu after sending the request, and disables them again in the interrupt handler. I guess we should never get more than one interrupt per SCLP request?

Didn't old qemu versions do exactly that an we currently catch that in the kernel?

#6 lore raw

Re: [kvm-unit-tests PATCH v2 1/6] s390x: Use interrupts in SCLP and add locking

Quoted 212 lines · Janosch Frank
Quoted 211 lines · David Hildenbrand
Quoted 209 lines · David Hildenbrand
Quoted 187 lines · Janosch Frank

We need to properly implement interrupt handling for SCLP, because on z/VM and LPAR SCLP calls are not synchronous!

Also with smp CPUs have to compete for sclp. Let's add some locking, so they execute sclp calls in an orderly fashion and don't compete for the data buffer.

---
 lib/s390x/asm/interrupt.h |  2 ++
 lib/s390x/interrupt.c     | 12 +++++++--
 lib/s390x/sclp-console.c  |  2 ++
 lib/s390x/sclp.c          | 55 +++++++++++++++++++++++++++++++++++++--
 lib/s390x/sclp.h          |  3 +++
 5 files changed, 70 insertions(+), 4 deletions(-)
lib/s390x/asm/interrupt.h +2 -0
index 013709f..f485e96 100644
@@ -11,6 +11,8 @@
 #define _ASMS390X_IRQ_H_
 #include <asm/arch_def.h>
 
+#define EXT_IRQ_SERVICE_SIG	0x2401
+
 void handle_pgm_int(void);
 void handle_ext_int(void);
 void handle_mcck_int(void);
lib/s390x/interrupt.c +10 -2
index cf0a794..7832711 100644
@@ -12,6 +12,7 @@
 #include <libcflat.h>
 #include <asm/interrupt.h>
 #include <asm/barrier.h>
+#include <sclp.h>
 
 static bool pgm_int_expected;
 static struct lowcore *lc;
@@ -107,8 +108,15 @@ void handle_pgm_int(void)
 
 void handle_ext_int(void)
 {
-	report_abort("Unexpected external call interrupt: at %#lx",
-		     lc->ext_old_psw.addr);
+	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
+		report_abort("Unexpected external call interrupt: at %#lx",
+			     lc->ext_old_psw.addr);
+	} else {
+		lc->ext_old_psw.mask &= ~PSW_MASK_EXT;
+		lc->sw_int_cr0 &= ~(1UL << 9);
+		sclp_handle_ext();
+		lc->ext_int_code = 0;
+	}
 }
 
 void handle_mcck_int(void)
lib/s390x/sclp-console.c +2 -0
index bc01f41..a5ef45f 100644
@@ -17,6 +17,7 @@ static void sclp_set_write_mask(void)
 {
 	WriteEventMask *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventMask);
 	sccb->mask_length = sizeof(unsigned int);
 	sccb->receive_mask = SCLP_EVENT_MASK_MSG_ASCII;
@@ -37,6 +38,7 @@ void sclp_print(const char *str)
 	int len = strlen(str);
 	WriteEventData *sccb = (void *)_sccb;
 
+	sclp_mark_busy();
 	sccb->h.length = sizeof(WriteEventData) + len;
 	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
 	sccb->ebh.length = sizeof(EventBufferHeader) + len;
lib/s390x/sclp.c +53 -2
index b60f7a4..56fca0c 100644
@@ -14,6 +14,8 @@
 #include <asm/page.h>
 #include <asm/arch_def.h>
 #include <asm/interrupt.h>
+#include <asm/barrier.h>
+#include <asm/spinlock.h>
 #include "sclp.h"
 #include <alloc_phys.h>
 #include <alloc_page.h>
@@ -25,6 +27,8 @@ static uint64_t max_ram_size;
 static uint64_t ram_size;
 
 char _sccb[PAGE_SIZE] __attribute__((__aligned__(4096)));
+static volatile bool sclp_busy;
+static struct spinlock sclp_lock;
 
 static void mem_init(phys_addr_t mem_end)
 {
@@ -41,17 +45,62 @@ static void mem_init(phys_addr_t mem_end)
 	page_alloc_ops_enable();
 }
 
+static void sclp_setup_int(void)
+{
+	uint64_t mask;
+
+	ctl_set_bit(0, 9);
+
+	mask = extract_psw_mask();
+	mask |= PSW_MASK_EXT;
+	load_psw_mask(mask);
+}
+
+void sclp_handle_ext(void)
+{
+	ctl_clear_bit(0, 9);
+	spin_lock(&sclp_lock);
+	sclp_busy = false;
+	spin_unlock(&sclp_lock);
+}
+
+void sclp_wait_busy(void)
+{
+	while (sclp_busy)
+		mb();
+}
+
+void sclp_mark_busy(void)
+{
+	/*
+	 * With multiple CPUs we might need to wait for another CPU's
+	 * request before grabbing the busy indication.
+	 */
+	while (true) {
+		sclp_wait_busy();
+		spin_lock(&sclp_lock);
+		if (!sclp_busy) {
+			sclp_busy = true;
+			spin_unlock(&sclp_lock);
+			return;
+		}
+		spin_unlock(&sclp_lock);
+	}
+}
+
 static void sclp_read_scp_info(ReadInfo *ri, int length)
 {
 	unsigned int commands[] = { SCLP_CMDW_READ_SCP_INFO_FORCED,
 				    SCLP_CMDW_READ_SCP_INFO };
-	int i;
+	int i, cc;
 
 	for (i = 0; i < ARRAY_SIZE(commands); i++) {
+		sclp_mark_busy();
 		memset(&ri->h, 0, sizeof(ri->h));
 		ri->h.length = length;
 
-		if (sclp_service_call(commands[i], ri))
+		cc = sclp_service_call(commands[i], ri);
+		if (cc)
 			break;
 		if (ri->h.response_code == SCLP_RC_NORMAL_READ_COMPLETION)
 			return;
@@ -66,12 +115,14 @@ int sclp_service_call(unsigned int command, void *sccb)
 {
 	int cc;
 
+	sclp_setup_int();
 	asm volatile(
 		"       .insn   rre,0xb2200000,%1,%2\n"  /* servc %1,%2 */
 		"       ipm     %0\n"
 		"       srl     %0,28"
 		: "=&d" (cc) : "d" (command), "a" (__pa(sccb))
 		: "cc", "memory");
+	sclp_wait_busy();
 	if (cc == 3)
 		return -1;
 	if (cc == 2)
lib/s390x/sclp.h +0 -0
index 583c4e5..63cf609 100644

lib/s390x/sclp.h typedef struct ReadEventData { · line 221

Quoted 9 lines
lib/s390x/sclp.h +3 -0
@@ -213,6 +213,9 @@ typedef struct ReadEventData {
 } __attribute__((packed)) ReadEventData;
 
 extern char _sccb[];
+void sclp_handle_ext(void);
+void sclp_wait_busy(void);
+void sclp_mark_busy(void);
 void sclp_console_setup(void);
 void sclp_print(const char *str);
 int sclp_service_call(unsigned int command, void *sccb);

I was wondering whether it would make sense to enable sclp interrupts as default for all CPUs (once in a reasonable state after brought up), and simply let any CPU process the request. Initially, we could only let the boot CPU handle them.

You already decoupled sclp_mark_busy() and sclp_setup_int() already. The part would have to be moved to the CPU init stage and sclp_handle_ext() would simply not clear the interrupt-enable flag.

Opinions?

Quoted 8 lines
Quoted 4 lines

OTOH, the s390x-ccw bios enables interrupts on the single cpu after sending the request, and disables them again in the interrupt handler. I guess we should never get more than one interrupt per SCLP request?

Didn't old qemu versions do exactly that an we currently catch that in the kernel?

You mean, multiple interrupts? I remember that the old bios wouldn't wait for the sclp interrupt at all - meaning one could have remain pending for the kernel. But that was solved by always waiting for the single interrupt.

Signature · 3 lines
Thanks,

David / dhildenb
#7 lore raw

PATCH v2 2/6 s390x: Add linemode console

In reply to Janosch Frank

z/VM isn't fond of vt220, so we need line mode when running under z/VM.

---
 lib/s390x/sclp-console.c | 181 +++++++++++++++++++++++++++++++++++----
 lib/s390x/sclp.h         |  55 +++++++++++-
 2 files changed, 218 insertions(+), 18 deletions(-)
lib/s390x/sclp-console.c +164 -17
index a5ef45f..19416b5 100644
@@ -11,21 +11,165 @@
 #include <libcflat.h>
 #include <string.h>
 #include <asm/page.h>
+#include <asm/arch_def.h>
+#include <asm/io.h>
 #include "sclp.h"
 
+/*
+ * ASCII (IBM PC 437) -> EBCDIC 037
+ */
+static uint8_t _ascebc[256] = {
+ /*00 NUL   SOH   STX   ETX   EOT   ENQ   ACK   BEL */
+     0x00, 0x01, 0x02, 0x03, 0x37, 0x2D, 0x2E, 0x2F,
+ /*08  BS    HT    LF    VT    FF    CR    SO    SI */
+ /*              ->NL                               */
+     0x16, 0x05, 0x15, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F,
+ /*10 DLE   DC1   DC2   DC3   DC4   NAK   SYN   ETB */
+     0x10, 0x11, 0x12, 0x13, 0x3C, 0x3D, 0x32, 0x26,
+ /*18 CAN    EM   SUB   ESC    FS    GS    RS    US */
+ /*                               ->IGS ->IRS ->IUS */
+     0x18, 0x19, 0x3F, 0x27, 0x22, 0x1D, 0x1E, 0x1F,
+ /*20  SP     !     "     #     $     %     &     ' */
+     0x40, 0x5A, 0x7F, 0x7B, 0x5B, 0x6C, 0x50, 0x7D,
+ /*28   (     )     *     +     ,     -    .      / */
+     0x4D, 0x5D, 0x5C, 0x4E, 0x6B, 0x60, 0x4B, 0x61,
+ /*30   0     1     2     3     4     5     6     7 */
+     0xF0, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7,
+ /*38   8     9     :     ;     <     =     >     ? */
+     0xF8, 0xF9, 0x7A, 0x5E, 0x4C, 0x7E, 0x6E, 0x6F,
+ /*40   @     A     B     C     D     E     F     G */
+     0x7C, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5, 0xC6, 0xC7,
+ /*48   H     I     J     K     L     M     N     O */
+     0xC8, 0xC9, 0xD1, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6,
+ /*50   P     Q     R     S     T     U     V     W */
+     0xD7, 0xD8, 0xD9, 0xE2, 0xE3, 0xE4, 0xE5, 0xE6,
+ /*58   X     Y     Z     [     \     ]     ^     _ */
+     0xE7, 0xE8, 0xE9, 0xBA, 0xE0, 0xBB, 0xB0, 0x6D,
+ /*60   `     a     b     c     d     e     f     g */
+     0x79, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87,
+ /*68   h     i     j     k     l     m     n     o */
+     0x88, 0x89, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96,
+ /*70   p     q     r     s     t     u     v     w */
+     0x97, 0x98, 0x99, 0xA2, 0xA3, 0xA4, 0xA5, 0xA6,
+ /*78   x     y     z     {     |     }     ~    DL */
+     0xA7, 0xA8, 0xA9, 0xC0, 0x4F, 0xD0, 0xA1, 0x07,
+ /*80*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*88*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*90*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*98*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*A0*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*A8*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*B0*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*B8*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*C0*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*C8*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*D0*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*D8*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*E0        sz	*/
+     0x3F, 0x59, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*E8*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*F0*/
+     0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F, 0x3F,
+ /*F8*/
+     0x90, 0x3F, 0x3F, 0x3F, 0x3F, 0xEA, 0x3F, 0xFF
+};
+
+static void sclp_print_ascii(const char *str)
+{
+	int len = strlen(str);
+	WriteEventData *sccb = (void *)_sccb;
+
+	sclp_mark_busy();
+	memset(sccb, 0, sizeof(*sccb));
+	sccb->h.length = offsetof(WriteEventData, msg) + len;
+	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
+	sccb->ebh.length = sizeof(EventBufferHeader) + len;
+	sccb->ebh.type = SCLP_EVENT_ASCII_CONSOLE_DATA;
+	memcpy(&sccb->msg, str, len);
+
+	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
+}
+
+static void sclp_print_lm(const char *str)
+{
+	unsigned char *ptr, *end, ch;
+	unsigned int count, offset, len;
+	struct WriteEventData *sccb;
+	struct mdb *mdb;
+	struct mto *mto;
+	struct go *go;
+
+	sclp_mark_busy();
+	sccb = (struct WriteEventData *) _sccb;
+	end = (unsigned char *) sccb + 4096 - 1;
+	memset(sccb, 0, sizeof(*sccb));
+	ptr = (unsigned char *) &sccb->msg.mdb.mto;
+	len = strlen(str);
+	offset = 0;
+	do {
+		for (count = sizeof(*mto); offset < len; count++) {
+			ch = str[offset++];
+			if (ch == 0x0a || ptr + count > end)
+				break;
+			ptr[count] = _ascebc[ch];
+		}
+		mto = (struct mto *) ptr;
+		mto->length = count;
+		mto->type = 4;
+		mto->line_type_flags = LNTPFLGS_ENDTEXT;
+		ptr += count;
+	} while (offset < len && ptr + sizeof(*mto) <= end);
+	len = ptr - (unsigned char *) sccb;
+	sccb->h.length = len - offsetof(struct WriteEventData, h);
+	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
+	sccb->ebh.type = EVTYP_MSG;
+	sccb->ebh.length = len - offsetof(struct WriteEventData, ebh);
+	mdb = &sccb->msg.mdb;
+	mdb->header.type = 1;
+	mdb->header.tag = 0xD4C4C240;
+	mdb->header.revision_code = 1;
+	mdb->header.length = len - offsetof(struct WriteEventData, msg.mdb.header);
+	go = &mdb->go;
+	go->length = sizeof(*go);
+	go->type = 1;
+	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
+}
+
+/*
+ * SCLP needs to be initialized by setting a send and receive mask,
+ * indicating which messages the control program (we) want(s) to
+ * send/receive.
+ */
 static void sclp_set_write_mask(void)
 {
 	WriteEventMask *sccb = (void *)_sccb;
 
 	sclp_mark_busy();
+	memset(_sccb, 0, sizeof(*sccb));
 	sccb->h.length = sizeof(WriteEventMask);
-	sccb->mask_length = sizeof(unsigned int);
-	sccb->receive_mask = SCLP_EVENT_MASK_MSG_ASCII;
-	sccb->cp_receive_mask = SCLP_EVENT_MASK_MSG_ASCII;
-	sccb->send_mask = SCLP_EVENT_MASK_MSG_ASCII;
-	sccb->cp_send_mask = SCLP_EVENT_MASK_MSG_ASCII;
+	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
+	sccb->mask_length = sizeof(sccb_mask_t);
+
+	/* For now we don't process sclp input. */
+	sccb->cp_receive_mask = 0;
+	/* We send ASCII and line mode. */
+	sccb->cp_send_mask = SCLP_EVENT_MASK_MSG_ASCII | SCLP_EVENT_MASK_MSG;
 
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_MASK, sccb);
+	assert(sccb->h.response_code == SCLP_RC_NORMAL_COMPLETION);
 }
 
 void sclp_console_setup(void)
@@ -35,16 +179,19 @@ void sclp_console_setup(void)
 
 void sclp_print(const char *str)
 {
-	int len = strlen(str);
-	WriteEventData *sccb = (void *)_sccb;
-
-	sclp_mark_busy();
-	sccb->h.length = sizeof(WriteEventData) + len;
-	sccb->h.function_code = SCLP_FC_NORMAL_WRITE;
-	sccb->ebh.length = sizeof(EventBufferHeader) + len;
-	sccb->ebh.type = SCLP_EVENT_ASCII_CONSOLE_DATA;
-	sccb->ebh.flags = 0;
-	memcpy(sccb->data, str, len);
-
-	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
+	/*
+	 * z/VM advertises a vt220 console which is not functional:
+	 * (response code 05F0, "not active because of the state of
+	 * the machine"). Hence testing the masks would only work if
+	 * we also use stsi data to distinguish z/VM.
+	 *
+	 * Let's rather print on all available consoles.
+	 */
+	if (strlen(str) > (PAGE_SIZE / 2)) {
+		sclp_print_ascii("Warning: Printing is limited to 2KB of data.");
+		sclp_print_lm("Warning: Printing is limited to 2KB of data.");
+		return;
+	}
+	sclp_print_ascii(str);
+	sclp_print_lm(str);
 }
lib/s390x/sclp.h +54 -1

73 lines not on this page · raw message

Signature · 1 line
2.17.0
#8 lore raw

PATCH v2 3/6 s390x: Add linemode buffer to fix newline on every print

In reply to Janosch Frank

Linemode seems to add a newline for each sent message which makes reading rather hard. Hence we add a small buffer and only print if it's full or a newline is encountered. Except for when the string is longer than the buffer, then we flush the buffer and print directly.

---
 lib/s390x/sclp-console.c | 70 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 4 deletions(-)
lib/s390x/sclp-console.c +66 -4
index 19416b5..7397dc1 100644
@@ -13,6 +13,7 @@
 #include <asm/page.h>
 #include <asm/arch_def.h>
 #include <asm/io.h>
+#include <asm/spinlock.h>
 #include "sclp.h"
 
 /*
@@ -87,6 +88,10 @@ static uint8_t _ascebc[256] = {
      0x90, 0x3F, 0x3F, 0x3F, 0x3F, 0xEA, 0x3F, 0xFF
 };
 
+static char lm_buff[120];
+static unsigned char lm_buff_off;
+static struct spinlock lm_buff_lock;
+
 static void sclp_print_ascii(const char *str)
 {
 	int len = strlen(str);
@@ -103,10 +108,10 @@ static void sclp_print_ascii(const char *str)
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
 }
 
-static void sclp_print_lm(const char *str)
+static void lm_print(const char *buff, int len)
 {
 	unsigned char *ptr, *end, ch;
-	unsigned int count, offset, len;
+	unsigned int count, offset;
 	struct WriteEventData *sccb;
 	struct mdb *mdb;
 	struct mto *mto;
@@ -117,11 +122,10 @@ static void sclp_print_lm(const char *str)
 	end = (unsigned char *) sccb + 4096 - 1;
 	memset(sccb, 0, sizeof(*sccb));
 	ptr = (unsigned char *) &sccb->msg.mdb.mto;
-	len = strlen(str);
 	offset = 0;
 	do {
 		for (count = sizeof(*mto); offset < len; count++) {
-			ch = str[offset++];
+			ch = buff[offset++];
 			if (ch == 0x0a || ptr + count > end)
 				break;
 			ptr[count] = _ascebc[ch];
@@ -148,6 +152,64 @@ static void sclp_print_lm(const char *str)
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
 }
 
+
+/*
+ * In contrast to the ascii console, linemode produces a new
+ * line with every write of data. The report() function uses
+ * several printf() calls to generate a line of data which
+ * would all end up on different lines.
+ *
+ * Hence we buffer here until we encounter a \n or the buffer
+ * is full. That means that linemode output can look a bit
+ * different from ascii and that it takes a bit longer for
+ * lines to appear.
+ */
+static void sclp_print_lm(const char *str)
+{
+	int len;
+	char *nl;
+
+	spin_lock(&lm_buff_lock);
+
+	len = strlen(str);
+	/*
+	 * No use in copying into lm_buff, its time to flush the
+	 * buffer and print str until finished.
+	 */
+	if (len > sizeof(lm_buff)) {
+		if (lm_buff_off)
+			lm_print(lm_buff, lm_buff_off);
+		lm_print(str, len);
+		memset(lm_buff, 0 , sizeof(lm_buff));
+		lm_buff_off = 0;
+		goto out;
+	}
+
+fill:
+	len = len < (sizeof(lm_buff) - lm_buff_off) ? len : (sizeof(lm_buff) - lm_buff_off);
+	if ((lm_buff_off < sizeof(lm_buff) - 1)) {
+		memcpy(&lm_buff[lm_buff_off], str, len);
+		lm_buff_off += len;
+	}
+	/* Buffer not full and no newline */
+	nl = strchr(lm_buff, '\n');
+	if (lm_buff_off != sizeof(lm_buff) - 1 && !nl)
+		goto out;
+
+	lm_print(lm_buff, lm_buff_off);
+	memset(lm_buff, 0 , sizeof(lm_buff));
+	lm_buff_off = 0;
+
+	if (len < strlen(str)) {
+		str = &str[len];
+		len = strlen(str);
+		goto fill;
+	}
+
+out:
+	spin_unlock(&lm_buff_lock);
+}
+
 /*
  * SCLP needs to be initialized by setting a send and receive mask,
  * indicating which messages the control program (we) want(s) to
Signature · 1 line
2.17.0
Acked-by on v2 3/6
#9 lore raw

Re: [kvm-unit-tests PATCH v2 3/6] s390x: Add linemode buffer to fix newline on every print

Quoted 116 lines · Janosch Frank

Linemode seems to add a newline for each sent message which makes reading rather hard. Hence we add a small buffer and only print if it's full or a newline is encountered. Except for when the string is longer than the buffer, then we flush the buffer and print directly.

---
 lib/s390x/sclp-console.c | 70 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 4 deletions(-)
lib/s390x/sclp-console.c +59 -4
index 19416b5..7397dc1 100644
@@ -13,6 +13,7 @@
 #include <asm/page.h>
 #include <asm/arch_def.h>
 #include <asm/io.h>
+#include <asm/spinlock.h>
 #include "sclp.h"
 
 /*
@@ -87,6 +88,10 @@ static uint8_t _ascebc[256] = {
      0x90, 0x3F, 0x3F, 0x3F, 0x3F, 0xEA, 0x3F, 0xFF
 };
 
+static char lm_buff[120];
+static unsigned char lm_buff_off;
+static struct spinlock lm_buff_lock;
+
 static void sclp_print_ascii(const char *str)
 {
 	int len = strlen(str);
@@ -103,10 +108,10 @@ static void sclp_print_ascii(const char *str)
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
 }
 
-static void sclp_print_lm(const char *str)
+static void lm_print(const char *buff, int len)
 {
 	unsigned char *ptr, *end, ch;
-	unsigned int count, offset, len;
+	unsigned int count, offset;
 	struct WriteEventData *sccb;
 	struct mdb *mdb;
 	struct mto *mto;
@@ -117,11 +122,10 @@ static void sclp_print_lm(const char *str)
 	end = (unsigned char *) sccb + 4096 - 1;
 	memset(sccb, 0, sizeof(*sccb));
 	ptr = (unsigned char *) &sccb->msg.mdb.mto;
-	len = strlen(str);
 	offset = 0;
 	do {
 		for (count = sizeof(*mto); offset < len; count++) {
-			ch = str[offset++];
+			ch = buff[offset++];
 			if (ch == 0x0a || ptr + count > end)
 				break;
 			ptr[count] = _ascebc[ch];
@@ -148,6 +152,64 @@ static void sclp_print_lm(const char *str)
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
 }
 
+
+/*
+ * In contrast to the ascii console, linemode produces a new
+ * line with every write of data. The report() function uses
+ * several printf() calls to generate a line of data which
+ * would all end up on different lines.
+ *
+ * Hence we buffer here until we encounter a \n or the buffer
+ * is full. That means that linemode output can look a bit
+ * different from ascii and that it takes a bit longer for
+ * lines to appear.
+ */
+static void sclp_print_lm(const char *str)
+{
+	int len;
+	char *nl;
+
+	spin_lock(&lm_buff_lock);
+
+	len = strlen(str);
+	/*
+	 * No use in copying into lm_buff, its time to flush the
+	 * buffer and print str until finished.
+	 */
+	if (len > sizeof(lm_buff)) {
+		if (lm_buff_off)
+			lm_print(lm_buff, lm_buff_off);
+		lm_print(str, len);
+		memset(lm_buff, 0 , sizeof(lm_buff));
+		lm_buff_off = 0;
+		goto out;
+	}
+
+fill:
+	len = len < (sizeof(lm_buff) - lm_buff_off) ? len : (sizeof(lm_buff) - lm_buff_off);
+	if ((lm_buff_off < sizeof(lm_buff) - 1)) {
+		memcpy(&lm_buff[lm_buff_off], str, len);
+		lm_buff_off += len;
+	}
+	/* Buffer not full and no newline */
+	nl = strchr(lm_buff, '\n');
+	if (lm_buff_off != sizeof(lm_buff) - 1 && !nl)
+		goto out;
+
+	lm_print(lm_buff, lm_buff_off);
+	memset(lm_buff, 0 , sizeof(lm_buff));
+	lm_buff_off = 0;
+
+	if (len < strlen(str)) {
+		str = &str[len];
+		len = strlen(str);

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 215

Quoted 10 lines
lib/s390x/sclp-console.c +7 -0
+		goto fill;
+	}
+
+out:
+	spin_unlock(&lm_buff_lock);
+}
+
 /*
  * SCLP needs to be initialized by setting a send and receive mask,
  * indicating which messages the control program (we) want(s) to
Signature · 3 lines
Thanks,

David / dhildenb
#10 lore raw

Re: [kvm-unit-tests PATCH v2 3/6] s390x: Add linemode buffer to fix newline on every print

In reply to Janosch Frank
Quoted 24 lines · Janosch Frank

Linemode seems to add a newline for each sent message which makes reading rather hard. Hence we add a small buffer and only print if it's full or a newline is encountered. Except for when the string is longer than the buffer, then we flush the buffer and print directly.

---
 lib/s390x/sclp-console.c | 70 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 4 deletions(-)
lib/s390x/sclp-console.c +1 -0
index 19416b5..7397dc1 100644
@@ -13,6 +13,7 @@
 #include <asm/page.h>
 #include <asm/arch_def.h>
 #include <asm/io.h>
+#include <asm/spinlock.h>
 #include "sclp.h"
 
 /*

lib/s390x/sclp-console.c static uint8_t _ascebc[256] = { · line 91

Quoted 4 lines
lib/s390x/sclp-console.c +1 -0
@@ -87,6 +88,10 @@ static uint8_t _ascebc[256] = {
      0x90, 0x3F, 0x3F, 0x3F, 0x3F, 0xEA, 0x3F, 0xFF
 };
 
+static char lm_buff[120];

Just wondering, how did you come up with the 120? (my first guess would have been something around 80)

lib/s390x/sclp-console.c static void sclp_print_ascii(const char *str) · line 112

Quoted 13 lines
lib/s390x/sclp-console.c +4 -1
+static unsigned char lm_buff_off;
+static struct spinlock lm_buff_lock;
+
 static void sclp_print_ascii(const char *str)
 {
 	int len = strlen(str);
@@ -103,10 +108,10 @@ static void sclp_print_ascii(const char *str)
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
 }
 
-static void sclp_print_lm(const char *str)
+static void lm_print(const char *buff, int len)
 {

The rename of str->buff could have been avoided, however, the impact is rather small.

Quoted 33 lines
lib/s390x/sclp-console.c +12 -3
 	unsigned char *ptr, *end, ch;
-	unsigned int count, offset, len;
+	unsigned int count, offset;
 	struct WriteEventData *sccb;
 	struct mdb *mdb;
 	struct mto *mto;
@@ -117,11 +122,10 @@ static void sclp_print_lm(const char *str)
 	end = (unsigned char *) sccb + 4096 - 1;
 	memset(sccb, 0, sizeof(*sccb));
 	ptr = (unsigned char *) &sccb->msg.mdb.mto;
-	len = strlen(str);
 	offset = 0;
 	do {
 		for (count = sizeof(*mto); offset < len; count++) {
-			ch = str[offset++];
+			ch = buff[offset++];
 			if (ch == 0x0a || ptr + count > end)
 				break;
 			ptr[count] = _ascebc[ch];
@@ -148,6 +152,64 @@ static void sclp_print_lm(const char *str)
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
 }
 
+
+/*
+ * In contrast to the ascii console, linemode produces a new
+ * line with every write of data. The report() function uses
+ * several printf() calls to generate a line of data which
+ * would all end up on different lines.
+ *
+ * Hence we buffer here until we encounter a \n or the buffer
+ * is full. That means that linemode output can look a bit
+ * different from ascii and that it takes a bit longer for

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 174

Quoted 10 lines
lib/s390x/sclp-console.c +10 -0
+ * lines to appear.
+ */
+static void sclp_print_lm(const char *str)
+{
+	int len;
+	char *nl;
+
+	spin_lock(&lm_buff_lock);
+
+	len = strlen(str);

You could do that directly when declaring the variable, doesn't have to be under the lock.

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 179

Quoted 5 lines
lib/s390x/sclp-console.c +5 -0
+	/*
+	 * No use in copying into lm_buff, its time to flush the
+	 * buffer and print str until finished.
+	 */
+	if (len > sizeof(lm_buff)) {

I find ARRAY_SIZE(lm_buf) easier to understand than sizeof(lm_buff)

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 188

Quoted 9 lines
lib/s390x/sclp-console.c +9 -0
+		if (lm_buff_off)
+			lm_print(lm_buff, lm_buff_off);
+		lm_print(str, len);
+		memset(lm_buff, 0 , sizeof(lm_buff));
+		lm_buff_off = 0;
+		goto out;
+	}
+
+fill:

Is there a way to remove this goto by using a simple while loop?

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 190

Quoted 2 lines
lib/s390x/sclp-console.c +2 -0
+	len = len < (sizeof(lm_buff) - lm_buff_off) ? len : (sizeof(lm_buff) - lm_buff_off);
+	if ((lm_buff_off < sizeof(lm_buff) - 1)) {

Drop one set of ()

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 195

Quoted 5 lines
lib/s390x/sclp-console.c +5 -0
+		memcpy(&lm_buff[lm_buff_off], str, len);
+		lm_buff_off += len;
+	}
+	/* Buffer not full and no newline */
+	nl = strchr(lm_buff, '\n');

Why do we have to search? Shouldn't a newline be the last copied character only?

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 207

Quoted 12 lines
lib/s390x/sclp-console.c +12 -0
+	if (lm_buff_off != sizeof(lm_buff) - 1 && !nl)
+		goto out;
+
+	lm_print(lm_buff, lm_buff_off);
+	memset(lm_buff, 0 , sizeof(lm_buff));
+	lm_buff_off = 0;
+
+	if (len < strlen(str)) {
+		str = &str[len];
+		len = strlen(str);
+		goto fill;
+	}

This looks too complicated for my taste :) Or my caffeine level is low.

I think we have the following cases:

1. String contains newline
a) String fits into remaining buffer

	-> Copy into buffer, flush (last character is newline)
 b) String doesn't fit into remaining buffer
	-> Simply flush old buffer and print remaining string?
2. String doesn't contain newline.
 a) String fits into remaining buffer
	-> Copy into buffer, flush if full
 b) String doesn't fit into remaining buffer
	-> Simply flush old buffer and print remaining string?

Optimizing for 1b) or 2b) isn't really worth it I guess - unless we want to wrap *any* string at 120 characters. But then, your pre-loop handling would also have to be modified. I think this allow to simplify your code a lot.

(how often does it happen in our current tests that we exceed 120 characters?)

Quoted 9 lines
lib/s390x/sclp-console.c +5 -0
+
+out:
+	spin_unlock(&lm_buff_lock);
+}
+
 /*
  * SCLP needs to be initialized by setting a send and receive mask,
  * indicating which messages the control program (we) want(s) to
Signature · 3 lines
Thanks,

David / dhildenb
#11 lore raw

Re: [kvm-unit-tests PATCH v2 3/6] s390x: Add linemode buffer to fix newline on every print

Quoted 159 lines · David Hildenbrand
Quoted 24 lines · Janosch Frank

Linemode seems to add a newline for each sent message which makes reading rather hard. Hence we add a small buffer and only print if it's full or a newline is encountered. Except for when the string is longer than the buffer, then we flush the buffer and print directly.

---
 lib/s390x/sclp-console.c | 70 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 66 insertions(+), 4 deletions(-)
lib/s390x/sclp-console.c +1 -0
index 19416b5..7397dc1 100644
@@ -13,6 +13,7 @@
 #include <asm/page.h>
 #include <asm/arch_def.h>
 #include <asm/io.h>
+#include <asm/spinlock.h>
 #include "sclp.h"
 
 /*

lib/s390x/sclp-console.c static uint8_t _ascebc[256] = { · line 91

Quoted 4 lines
lib/s390x/sclp-console.c +1 -0
@@ -87,6 +88,10 @@ static uint8_t _ascebc[256] = {
      0x90, 0x3F, 0x3F, 0x3F, 0x3F, 0xEA, 0x3F, 0xFF
 };
 
+static char lm_buff[120];

Just wondering, how did you come up with the 120? (my first guess would have been something around 80)

lib/s390x/sclp-console.c static void sclp_print_ascii(const char *str) · line 112

Quoted 13 lines
lib/s390x/sclp-console.c +4 -1
+static unsigned char lm_buff_off;
+static struct spinlock lm_buff_lock;
+
 static void sclp_print_ascii(const char *str)
 {
 	int len = strlen(str);
@@ -103,10 +108,10 @@ static void sclp_print_ascii(const char *str)
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
 }
 
-static void sclp_print_lm(const char *str)
+static void lm_print(const char *buff, int len)
 {

The rename of str->buff could have been avoided, however, the impact is rather small.

Quoted 33 lines
lib/s390x/sclp-console.c +12 -3
 	unsigned char *ptr, *end, ch;
-	unsigned int count, offset, len;
+	unsigned int count, offset;
 	struct WriteEventData *sccb;
 	struct mdb *mdb;
 	struct mto *mto;
@@ -117,11 +122,10 @@ static void sclp_print_lm(const char *str)
 	end = (unsigned char *) sccb + 4096 - 1;
 	memset(sccb, 0, sizeof(*sccb));
 	ptr = (unsigned char *) &sccb->msg.mdb.mto;
-	len = strlen(str);
 	offset = 0;
 	do {
 		for (count = sizeof(*mto); offset < len; count++) {
-			ch = str[offset++];
+			ch = buff[offset++];
 			if (ch == 0x0a || ptr + count > end)
 				break;
 			ptr[count] = _ascebc[ch];
@@ -148,6 +152,64 @@ static void sclp_print_lm(const char *str)
 	sclp_service_call(SCLP_CMD_WRITE_EVENT_DATA, sccb);
 }
 
+
+/*
+ * In contrast to the ascii console, linemode produces a new
+ * line with every write of data. The report() function uses
+ * several printf() calls to generate a line of data which
+ * would all end up on different lines.
+ *
+ * Hence we buffer here until we encounter a \n or the buffer
+ * is full. That means that linemode output can look a bit
+ * different from ascii and that it takes a bit longer for

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 174

Quoted 10 lines
lib/s390x/sclp-console.c +10 -0
+ * lines to appear.
+ */
+static void sclp_print_lm(const char *str)
+{
+	int len;
+	char *nl;
+
+	spin_lock(&lm_buff_lock);
+
+	len = strlen(str);

You could do that directly when declaring the variable, doesn't have to be under the lock.

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 179

Quoted 5 lines
lib/s390x/sclp-console.c +5 -0
+	/*
+	 * No use in copying into lm_buff, its time to flush the
+	 * buffer and print str until finished.
+	 */
+	if (len > sizeof(lm_buff)) {

I find ARRAY_SIZE(lm_buf) easier to understand than sizeof(lm_buff)

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 188

Quoted 9 lines
lib/s390x/sclp-console.c +9 -0
+		if (lm_buff_off)
+			lm_print(lm_buff, lm_buff_off);
+		lm_print(str, len);
+		memset(lm_buff, 0 , sizeof(lm_buff));
+		lm_buff_off = 0;
+		goto out;
+	}
+
+fill:

Is there a way to remove this goto by using a simple while loop?

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 190

Quoted 2 lines
lib/s390x/sclp-console.c +2 -0
+	len = len < (sizeof(lm_buff) - lm_buff_off) ? len : (sizeof(lm_buff) - lm_buff_off);
+	if ((lm_buff_off < sizeof(lm_buff) - 1)) {

Drop one set of ()

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 195

Quoted 5 lines
lib/s390x/sclp-console.c +5 -0
+		memcpy(&lm_buff[lm_buff_off], str, len);
+		lm_buff_off += len;
+	}
+	/* Buffer not full and no newline */
+	nl = strchr(lm_buff, '\n');

Why do we have to search? Shouldn't a newline be the last copied character only?

lib/s390x/sclp-console.c static void sclp_print_lm(const char *str) · line 207

Quoted 12 lines
lib/s390x/sclp-console.c +12 -0
+	if (lm_buff_off != sizeof(lm_buff) - 1 && !nl)
+		goto out;
+
+	lm_print(lm_buff, lm_buff_off);
+	memset(lm_buff, 0 , sizeof(lm_buff));
+	lm_buff_off = 0;
+
+	if (len < strlen(str)) {
+		str = &str[len];
+		len = strlen(str);
+		goto fill;
+	}

This looks too complicated for my taste :) Or my caffeine level is low.

I think we have the following cases:

1. String contains newline
a) String fits into remaining buffer

	-> Copy into buffer, flush (last character is newline)
 b) String doesn't fit into remaining buffer
	-> Simply flush old buffer and print remaining string?
2. String doesn't contain newline.
 a) String fits into remaining buffer
	-> Copy into buffer, flush if full
 b) String doesn't fit into remaining buffer
	-> Simply flush old buffer and print remaining string?
Quoted 7 lines

Optimizing for 1b) or 2b) isn't really worth it I guess - unless we want to wrap *any* string at 120 characters. But then, your pre-loop handling would also have to be modified. I think this allow to simplify your code a lot.

(how often does it happen in our current tests that we exceed 120 characters?)

How about this?

 	char *nl;
        int len = strlen(str), i = 0;

        spin_lock(&lm_buff_lock);

	while (len) {
                lm_buff[lm_buff_off] = str[i];
                lm_buff_off++;
                len--;
                /* Buffer full or newline? */
                if (str[i] == '\n' || lm_buff_off == (sizeof(lm_buff) -
1)) {
                        lm_print(lm_buff, lm_buff_off);
                        memset(lm_buff, 0 , sizeof(lm_buff));
                        lm_buff_off = 0;
                }
                i++;
        }
        spin_unlock(&lm_buff_lock);
        return;
Quoted 12 lines
Quoted 10 lines
lib/s390x/sclp-console.c +5 -0
+
+out:
+	spin_unlock(&lm_buff_lock);
+}
+
 /*
  * SCLP needs to be initialized by setting a send and receive mask,
  * indicating which messages the control program (we) want(s) to
#12 lore raw

PATCH v2 4/6 s390x: Add initial smp code

In reply to Janosch Frank

Let's add a rudimentary SMP library, which will scan for cpus and has helper functions that manage the cpu state.

---
 lib/s390x/asm/arch_def.h |   8 ++
 lib/s390x/asm/sigp.h     |  28 +++-
 lib/s390x/io.c           |   5 +-
 lib/s390x/sclp.h         |   1 +
 lib/s390x/smp.c          | 276 +++++++++++++++++++++++++++++++++++++++
 lib/s390x/smp.h          |  51 ++++++++
 s390x/Makefile           |   1 +
 s390x/cstart64.S         |   7 +
 8 files changed, 371 insertions(+), 6 deletions(-)
 create mode 100644 lib/s390x/smp.c
 create mode 100644 lib/s390x/smp.h
lib/s390x/asm/arch_def.h +8 -0
index 5f8f45e..d5a7f51 100644
@@ -157,6 +157,14 @@ struct cpuid {
 	uint64_t reserved : 15;
 };
 
+static inline unsigned short stap(void)
+{
+	unsigned short cpu_address;
+
+	asm volatile("stap %0" : "=Q" (cpu_address));
+	return cpu_address;
+}
+
 static inline int tprot(unsigned long addr)
 {
 	int cc;
lib/s390x/asm/sigp.h +23 -5
index fbd94fc..2d52313 100644
@@ -46,14 +46,32 @@
 
 #ifndef __ASSEMBLER__
 
-static inline void sigp_stop(void)
+
+static inline int sigp(uint16_t addr, uint8_t order, unsigned long parm,
+		       uint32_t *status)
 {
-	register unsigned long status asm ("1") = 0;
-	register unsigned long cpu asm ("2") = 0;
+	register unsigned long reg1 asm ("1") = parm;
+	int cc;
 
 	asm volatile(
-		"	sigp %0,%1,0(%2)\n"
-		: "+d" (status)  : "d" (cpu), "d" (SIGP_STOP) : "cc");
+		"	sigp	%1,%2,0(%3)\n"
+		"	ipm	%0\n"
+		"	srl	%0,28\n"
+		: "=d" (cc), "+d" (reg1) : "d" (addr), "a" (order) : "cc");
+	if (status)
+		*status = reg1;
+	return cc;
+}
+
+static inline int sigp_retry(uint16_t addr, uint8_t order, unsigned long parm,
+			     uint32_t *status)
+{
+	int cc;
+
+	do {
+		cc = sigp(addr, order, parm, status);
+	} while (cc == 2);
+	return cc;
 }
 
 #endif /* __ASSEMBLER__ */
lib/s390x/io.c +4 -1
index becadfc..32f09b5 100644
@@ -16,6 +16,7 @@
 #include <asm/facility.h>
 #include <asm/sigp.h>
 #include "sclp.h"
+#include "smp.h"
 
 extern char ipl_args[];
 uint8_t stfl_bytes[NR_STFL_BYTES] __attribute__((aligned(8)));
@@ -37,12 +38,14 @@ void setup(void)
 	setup_facilities();
 	sclp_console_setup();
 	sclp_memory_setup();
+	smp_setup();
 }
 
 void exit(int code)
 {
+	smp_teardown();
 	printf("\nEXIT: STATUS=%d\n", ((code) << 1) | 1);
 	while (1) {
-		sigp_stop();
+		sigp(0, SIGP_STOP, 0, NULL);
 	}
 }
lib/s390x/sclp.h +1 -0
index 98c482a..4e69845 100644
@@ -19,6 +19,7 @@
 #define SCLP_CMD_CODE_MASK                      0xffff00ff
 
 /* SCLP command codes */
+#define SCLP_READ_CPU_INFO			0x00010001
 #define SCLP_CMDW_READ_SCP_INFO                 0x00020001
 #define SCLP_CMDW_READ_SCP_INFO_FORCED          0x00120001
 #define SCLP_READ_STORAGE_ELEMENT_INFO          0x00040001
lib/s390x/smp.c new file +276 -0

276 lines not on this page · raw message

lib/s390x/smp.h new file +51 -0

51 lines not on this page · raw message

s390x/Makefile +1 -0
index 96033dd..d83dd0b 100644
@@ -48,6 +48,7 @@ cflatobjs += lib/s390x/sclp.o
 cflatobjs += lib/s390x/sclp-console.o
 cflatobjs += lib/s390x/interrupt.o
 cflatobjs += lib/s390x/mmu.o
+cflatobjs += lib/s390x/smp.o
 
 OBJDIRS += lib/s390x
 
s390x/cstart64.S +7 -0
index 36f7cab..a45ea8f 100644
@@ -172,6 +172,13 @@ diag308_load_reset:
 	lhi	%r2, 1
 	br	%r14
 
+.globl smp_cpu_setup_state
+smp_cpu_setup_state:
+	xgr	%r1, %r1
+	lmg     %r0, %r15, 512(%r1)
+	lctlg   %c0, %c0, 776(%r1)
+	br	%r14
+
 pgm_int:
 	SAVE_REGS
 	brasl	%r14, handle_pgm_int
Signature · 1 line
2.17.0
#13 lore raw

Re: [kvm-unit-tests PATCH v2 4/6] s390x: Add initial smp code

Quoted 5 lines · Janosch Frank

Let's add a rudimentary SMP library, which will scan for cpus and has helper functions that manage the cpu state.

---

[...]

Quoted 13 lines
lib/s390x/smp.c new file +13 -0
+/*Expected to be called from boot cpu */
+extern uint64_t *stackptr;
+void smp_setup(void)
+{
+	int i = 0;
+	unsigned short cpu0_addr = stap();
+	struct ReadCpuInfo *info = (void *)cpu_info_buffer;
+
+	spin_lock(&lock);
+	sclp_mark_busy();
+	info->h.length = PAGE_SIZE;
+	sclp_service_call(SCLP_READ_CPU_INFO, cpu_info_buffer);
+

lib/s390x/smp.c line 270

Quoted 10 lines
lib/s390x/smp.c new file +10 -0
+	if (smp_query_num_cpus() > 1)
+		printf("SMP: Initializing, found %d cpus\n", info->nr_configured);
+
+	cpus = calloc(info->nr_configured, sizeof(cpus));
+	for (i = 0; i < info->nr_configured; i++) {
+		if (info->entries[i].address == cpu0_addr) {
+			cpu0 = &cpus[i];
+			cpu0->stack = stackptr;
+			cpu0->lowcore = (void *)0;
+			cpu0->active = true;

So here cpus[i].active gets set to true ...

lib/s390x/smp.c line 273

Quoted 3 lines
lib/s390x/smp.c new file +3 -0
+		}
+		cpus[i].addr = info->entries[i].address;
+		cpus[i].active = false;

... but here it is set back to false.

Maybe move the if-statement below this line?

Quoted 3 lines
lib/s390x/smp.c new file +3 -0
+	}
+	spin_unlock(&lock);
+}

[...]

s390x/cstart64.S diag308_load_reset: · line 178

Quoted 12 lines
s390x/cstart64.S +4 -0
index 36f7cab..a45ea8f 100644
@@ -172,6 +172,13 @@ diag308_load_reset:
 	lhi	%r2, 1
 	br	%r14
 
+.globl smp_cpu_setup_state
+smp_cpu_setup_state:
+	xgr	%r1, %r1
+	lmg     %r0, %r15, 512(%r1)

Can you use GEN_LC_SW_INT_GRS instead of 512?

s390x/cstart64.S diag308_load_reset: · line 179

Quoted 1 line
s390x/cstart64.S +1 -0
+	lctlg   %c0, %c0, 776(%r1)

... and here GEN_LC_SW_INT_CR0 instead of 776?

Apart from that, the patch looks fine to me.

Thomas

#14 lore raw

Re: [kvm-unit-tests PATCH v2 4/6] s390x: Add initial smp code

Quoted 60 lines · Thomas Huth
Quoted 5 lines · Janosch Frank

Let's add a rudimentary SMP library, which will scan for cpus and has helper functions that manage the cpu state.

---

[...]

Quoted 13 lines
lib/s390x/smp.c new file +13 -0
+/*Expected to be called from boot cpu */
+extern uint64_t *stackptr;
+void smp_setup(void)
+{
+	int i = 0;
+	unsigned short cpu0_addr = stap();
+	struct ReadCpuInfo *info = (void *)cpu_info_buffer;
+
+	spin_lock(&lock);
+	sclp_mark_busy();
+	info->h.length = PAGE_SIZE;
+	sclp_service_call(SCLP_READ_CPU_INFO, cpu_info_buffer);
+

lib/s390x/smp.c line 270

Quoted 10 lines
lib/s390x/smp.c new file +10 -0
+	if (smp_query_num_cpus() > 1)
+		printf("SMP: Initializing, found %d cpus\n", info->nr_configured);
+
+	cpus = calloc(info->nr_configured, sizeof(cpus));
+	for (i = 0; i < info->nr_configured; i++) {
+		if (info->entries[i].address == cpu0_addr) {
+			cpu0 = &cpus[i];
+			cpu0->stack = stackptr;
+			cpu0->lowcore = (void *)0;
+			cpu0->active = true;

So here cpus[i].active gets set to true ...

lib/s390x/smp.c line 273

Quoted 3 lines
lib/s390x/smp.c new file +3 -0
+		}
+		cpus[i].addr = info->entries[i].address;
+		cpus[i].active = false;

... but here it is set back to false.

Maybe move the if-statement below this line?

Quoted 3 lines
lib/s390x/smp.c new file +3 -0
+	}
+	spin_unlock(&lock);
+}

[...]

s390x/cstart64.S diag308_load_reset: · line 178

Quoted 12 lines
s390x/cstart64.S +4 -0
index 36f7cab..a45ea8f 100644
@@ -172,6 +172,13 @@ diag308_load_reset:
 	lhi	%r2, 1
 	br	%r14
 
+.globl smp_cpu_setup_state
+smp_cpu_setup_state:
+	xgr	%r1, %r1
+	lmg     %r0, %r15, 512(%r1)

Can you use GEN_LC_SW_INT_GRS instead of 512?

Quoted 5 lines

s390x/cstart64.S diag308_load_reset: · line 179

Quoted 1 line
s390x/cstart64.S +1 -0
+	lctlg   %c0, %c0, 776(%r1)

... and here GEN_LC_SW_INT_CR0 instead of 776?

Apart from that, the patch looks fine to me.

Thanks, just fixed that

#15 lore raw

Re: [kvm-unit-tests PATCH v2 4/6] s390x: Add initial smp code

In reply to Janosch Frank
Quoted 129 lines · Janosch Frank

Let's add a rudimentary SMP library, which will scan for cpus and has helper functions that manage the cpu state.

---
 lib/s390x/asm/arch_def.h |   8 ++
 lib/s390x/asm/sigp.h     |  28 +++-
 lib/s390x/io.c           |   5 +-
 lib/s390x/sclp.h         |   1 +
 lib/s390x/smp.c          | 276 +++++++++++++++++++++++++++++++++++++++
 lib/s390x/smp.h          |  51 ++++++++
 s390x/Makefile           |   1 +
 s390x/cstart64.S         |   7 +
 8 files changed, 371 insertions(+), 6 deletions(-)
 create mode 100644 lib/s390x/smp.c
 create mode 100644 lib/s390x/smp.h
lib/s390x/asm/arch_def.h +8 -0
index 5f8f45e..d5a7f51 100644
@@ -157,6 +157,14 @@ struct cpuid {
 	uint64_t reserved : 15;
 };
 
+static inline unsigned short stap(void)
+{
+	unsigned short cpu_address;
+
+	asm volatile("stap %0" : "=Q" (cpu_address));
+	return cpu_address;
+}
+
 static inline int tprot(unsigned long addr)
 {
 	int cc;
lib/s390x/asm/sigp.h +23 -5
index fbd94fc..2d52313 100644
@@ -46,14 +46,32 @@
 
 #ifndef __ASSEMBLER__
 
-static inline void sigp_stop(void)
+
+static inline int sigp(uint16_t addr, uint8_t order, unsigned long parm,
+		       uint32_t *status)
 {
-	register unsigned long status asm ("1") = 0;
-	register unsigned long cpu asm ("2") = 0;
+	register unsigned long reg1 asm ("1") = parm;
+	int cc;
 
 	asm volatile(
-		"	sigp %0,%1,0(%2)\n"
-		: "+d" (status)  : "d" (cpu), "d" (SIGP_STOP) : "cc");
+		"	sigp	%1,%2,0(%3)\n"
+		"	ipm	%0\n"
+		"	srl	%0,28\n"
+		: "=d" (cc), "+d" (reg1) : "d" (addr), "a" (order) : "cc");
+	if (status)
+		*status = reg1;
+	return cc;
+}
+
+static inline int sigp_retry(uint16_t addr, uint8_t order, unsigned long parm,
+			     uint32_t *status)
+{
+	int cc;
+
+	do {
+		cc = sigp(addr, order, parm, status);
+	} while (cc == 2);
+	return cc;
 }
 
 #endif /* __ASSEMBLER__ */
lib/s390x/io.c +4 -1
index becadfc..32f09b5 100644
@@ -16,6 +16,7 @@
 #include <asm/facility.h>
 #include <asm/sigp.h>
 #include "sclp.h"
+#include "smp.h"
 
 extern char ipl_args[];
 uint8_t stfl_bytes[NR_STFL_BYTES] __attribute__((aligned(8)));
@@ -37,12 +38,14 @@ void setup(void)
 	setup_facilities();
 	sclp_console_setup();
 	sclp_memory_setup();
+	smp_setup();
 }
 
 void exit(int code)
 {
+	smp_teardown();
 	printf("\nEXIT: STATUS=%d\n", ((code) << 1) | 1);
 	while (1) {
-		sigp_stop();
+		sigp(0, SIGP_STOP, 0, NULL);
 	}
 }
lib/s390x/sclp.h +1 -0
index 98c482a..4e69845 100644
@@ -19,6 +19,7 @@
 #define SCLP_CMD_CODE_MASK                      0xffff00ff
 
 /* SCLP command codes */
+#define SCLP_READ_CPU_INFO			0x00010001
 #define SCLP_CMDW_READ_SCP_INFO                 0x00020001
 #define SCLP_CMDW_READ_SCP_INFO_FORCED          0x00120001
 #define SCLP_READ_STORAGE_ELEMENT_INFO          0x00040001
lib/s390x/smp.c new file +5 -0
new file mode 100644
index 0000000..40c43ef
@@ -0,0 +1,276 @@
+/*
+ * s390x smp
+ * Based on Linux's arch/s390/kernel/smp.c and
+ * arch/s390/include/asm/sigp.h
+ *

lib/s390x/smp.c line 15

Quoted 10 lines
lib/s390x/smp.c new file +10 -0
+ * Copyright (c) 2019 IBM Corp
+ *
+ * Authors:
+ *  Janosch Frank <[email protected]>
+ *
+ * This code is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2.
+ */
+#include <libcflat.h>
+#include <errno.h>

This breaks building on Travis:

https://travis-ci.com/huth/kvm-unit-tests/jobs/232679058

I don't think that you can use the errno.h header from the system library in kvm-unit-tests in a save way. It's likely better if you define your own error codes instead.

Thomas

#16 lore raw

PATCH v2 5/6 s390x: Prepare for external calls

In reply to Janosch Frank

With SMP we also get new external interrupts like external call and emergency call. Let's make them known.

---
 lib/s390x/asm/arch_def.h  |  5 +++++
 lib/s390x/asm/interrupt.h |  3 +++
 lib/s390x/interrupt.c     | 24 ++++++++++++++++++++----
 3 files changed, 28 insertions(+), 4 deletions(-)
lib/s390x/asm/arch_def.h +5 -0
index d5a7f51..96cca2e 100644
@@ -19,6 +19,11 @@ struct psw {
 #define PSW_MASK_DAT			0x0400000000000000UL
 #define PSW_MASK_PSTATE			0x0001000000000000UL
 
+#define CR0_EXTM_SCLP			0X0000000000000200UL
+#define CR0_EXTM_EXTC			0X0000000000002000UL
+#define CR0_EXTM_EMGC			0X0000000000004000UL
+#define CR0_EXTM_MASK			0X0000000000006200UL
+
 struct lowcore {
 	uint8_t		pad_0x0000[0x0080 - 0x0000];	/* 0x0000 */
 	uint32_t	ext_int_param;			/* 0x0080 */
lib/s390x/asm/interrupt.h +3 -0
index f485e96..4cfade9 100644
@@ -11,6 +11,8 @@
 #define _ASMS390X_IRQ_H_
 #include <asm/arch_def.h>
 
+#define EXT_IRQ_EMERGENCY_SIG	0x1201
+#define EXT_IRQ_EXTERNAL_CALL	0x1202
 #define EXT_IRQ_SERVICE_SIG	0x2401
 
 void handle_pgm_int(void);
@@ -19,6 +21,7 @@ void handle_mcck_int(void);
 void handle_io_int(void);
 void handle_svc_int(void);
 void expect_pgm_int(void);
+void expect_ext_int(void);
 uint16_t clear_pgm_int(void);
 void check_pgm_int_code(uint16_t code);
 
lib/s390x/interrupt.c +20 -4
index 7832711..d4f279a 100644
@@ -15,6 +15,7 @@
 #include <sclp.h>
 
 static bool pgm_int_expected;
+static bool ext_int_expected;
 static struct lowcore *lc;
 
 void expect_pgm_int(void)
@@ -24,6 +25,13 @@ void expect_pgm_int(void)
 	mb();
 }
 
+void expect_ext_int(void)
+{
+	ext_int_expected = true;
+	lc->ext_int_code = 0;
+	mb();
+}
+
 uint16_t clear_pgm_int(void)
 {
 	uint16_t code;
@@ -108,15 +116,23 @@ void handle_pgm_int(void)
 
 void handle_ext_int(void)
 {
-	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
+	if (!ext_int_expected &&
+	    lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
 		report_abort("Unexpected external call interrupt: at %#lx",
 			     lc->ext_old_psw.addr);
-	} else {
-		lc->ext_old_psw.mask &= ~PSW_MASK_EXT;
+		return;
+	}
+
+	if (lc->ext_int_code == EXT_IRQ_SERVICE_SIG) {
 		lc->sw_int_cr0 &= ~(1UL << 9);
 		sclp_handle_ext();
-		lc->ext_int_code = 0;
 	}
+	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
+		ext_int_expected = false;
+	}
+
+	if (!(lc->sw_int_cr0 & CR0_EXTM_MASK))
+		lc->ext_old_psw.mask &= ~PSW_MASK_EXT;
 }
 
 void handle_mcck_int(void)
Signature · 1 line
2.17.0
Reviewed-by on v2 5/6
#17 lore raw

Re: [kvm-unit-tests PATCH v2 5/6] s390x: Prepare for external calls

Quoted 5 lines · Janosch Frank

With SMP we also get new external interrupts like external call and emergency call. Let's make them known.

---

[...]

Quoted 10 lines
lib/s390x/interrupt.c +2 -2
@@ -108,15 +116,23 @@ void handle_pgm_int(void)
 
 void handle_ext_int(void)
 {
-	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
+	if (!ext_int_expected &&
+	    lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {
 		report_abort("Unexpected external call interrupt: at %#lx",
 			     lc->ext_old_psw.addr);
-	} else {

lib/s390x/interrupt.c void handle_pgm_int(void) · line 130

Quoted 10 lines
lib/s390x/interrupt.c +5 -2
-		lc->ext_old_psw.mask &= ~PSW_MASK_EXT;
+		return;
+	}
+
+	if (lc->ext_int_code == EXT_IRQ_SERVICE_SIG) {
 		lc->sw_int_cr0 &= ~(1UL << 9);
 		sclp_handle_ext();
-		lc->ext_int_code = 0;
 	}
+	if (lc->ext_int_code != EXT_IRQ_SERVICE_SIG) {

I think you could simplify above line to "else {" ?

With that change:

#18 lore raw

PATCH v2 6/6 s390x: SMP test

In reply to Janosch Frank

Testing SIGP emulation for the following order codes:
* start
* stop
* restart
* set prefix
* store status
* stop and store status
* reset
* initial reset
* external call
* emegergency call

restart and set prefix are part of the library and needed to start other cpus.

---
 s390x/Makefile      |   1 +
 s390x/smp.c         | 242 ++++++++++++++++++++++++++++++++++++++++++++
 s390x/unittests.cfg |   4 +
 3 files changed, 247 insertions(+)
 create mode 100644 s390x/smp.c
s390x/Makefile +1 -0
index d83dd0b..3744372 100644
@@ -15,6 +15,7 @@ tests += $(TEST_DIR)/cpumodel.elf
 tests += $(TEST_DIR)/diag288.elf
 tests += $(TEST_DIR)/stsi.elf
 tests += $(TEST_DIR)/skrf.elf
+tests += $(TEST_DIR)/smp.elf
 tests_binary = $(patsubst %.elf,%.bin,$(tests))
 
 all: directories test_cases test_cases_binary
s390x/smp.c new file +242 -0

242 lines not on this page · raw message

s390x/unittests.cfg +4 -0
index cc79a4e..f1b07cd 100644
@@ -71,3 +71,7 @@ extra_params=-device diag288,id=watchdog0 --watchdog-action inject-nmi
 
 [stsi]
 file = stsi.elf
+
+[smp]
+file = smp.elf
+extra_params =-smp 2
Signature · 1 line
2.17.0
#19 lore raw

Re: [kvm-unit-tests PATCH v2 6/6] s390x: SMP test

Quoted 17 lines · Janosch Frank

Testing SIGP emulation for the following order codes:
* start
* stop
* restart
* set prefix
* store status
* stop and store status
* reset
* initial reset
* external call
* emegergency call

restart and set prefix are part of the library and needed to start other cpus.

---

[...]

Quoted 12 lines
s390x/smp.c new file +12 -0
+static void test_store_status(void)
+{
+	struct cpu_status *status = alloc_pages(1);
+	uint32_t r;
+
+	report_prefix_push("store status at address");
+	memset(status, 0, PAGE_SIZE * 2);
+
+	report_prefix_push("running");
+	smp_cpu_restart(1);
+	sigp(1, SIGP_STORE_STATUS_AT_ADDRESS, (uintptr_t)status, &r);
+	report("incorrect state", r == SIGP_STATUS_INCORRECT_STATE);

s390x/smp.c line 98

Quoted 10 lines
s390x/smp.c new file +10 -0
+	report("status not written", !memcmp(status, (void*)status + PAGE_SIZE, PAGE_SIZE));
+	report_prefix_pop();
+
+	memset(status, 0, PAGE_SIZE);
+	report_prefix_push("stopped");
+	smp_cpu_stop(1);
+	sigp(1, SIGP_STORE_STATUS_AT_ADDRESS, (uintptr_t)status, NULL);
+	while (!status->prefix) { mb(); }
+	report("status written", 1);
+	free_pages(status, PAGE_SIZE);

Shouldn't that be PAGE_SIZE * 2 instead?

s390x/smp.c line 102

Quoted 4 lines
s390x/smp.c new file +4 -0
+	report_prefix_pop();
+
+	report_prefix_pop();
+}

The remaining part of the patch looks fine to me.

Thomas

#20 lore raw

Re: [kvm-unit-tests PATCH v2 6/6] s390x: SMP test

Quoted 35 lines · Thomas Huth
Quoted 17 lines · Janosch Frank

Testing SIGP emulation for the following order codes:
* start
* stop
* restart
* set prefix
* store status
* stop and store status
* reset
* initial reset
* external call
* emegergency call

restart and set prefix are part of the library and needed to start other cpus.

---

[...]

Quoted 16 lines
s390x/smp.c new file +16 -0
+static void test_store_status(void)
+{
+	struct cpu_status *status = alloc_pages(1);
+	uint32_t r;
+
+	report_prefix_push("store status at address");
+	memset(status, 0, PAGE_SIZE * 2);
+
+	report_prefix_push("running");
+	smp_cpu_restart(1);
+	sigp(1, SIGP_STORE_STATUS_AT_ADDRESS, (uintptr_t)status, &r);
+	report("incorrect state", r == SIGP_STATUS_INCORRECT_STATE);
+	report("status not written", !memcmp(status, (void*)status + PAGE_SIZE, PAGE_SIZE));
+	report_prefix_pop();
+
+	memset(status, 0, PAGE_SIZE);
Quoted 8 lines

s390x/smp.c line 98

Quoted 6 lines
s390x/smp.c new file +6 -0
+	report_prefix_push("stopped");
+	smp_cpu_stop(1);
+	sigp(1, SIGP_STORE_STATUS_AT_ADDRESS, (uintptr_t)status, NULL);
+	while (!status->prefix) { mb(); }
+	report("status written", 1);
+	free_pages(status, PAGE_SIZE);

Shouldn't that be PAGE_SIZE * 2 instead?

Indeed

Quoted 7 lines

s390x/smp.c line 102

Quoted 5 lines
s390x/smp.c new file +4 -0
+	report_prefix_pop();
+
+	report_prefix_pop();
+}

The remaining part of the patch looks fine to me.

Thanks for having a look

Quoted 3 lines

Thomas