FAILED: patch "[PATCH] PCI: aardvark: Increase polling delay to 1.5s while waiting" failed to apply to 5.10-stable tree

PATCH [email protected] 2 participants 8 messages Sep 15, 2021 – Sep 27, 2021

Subsystems (4)

Stable branch PCI PCI native host bridge and endpoint drivers PCI driver for aardvark (Marvell Armada 3700)

lore

Share & tools

lore

[email protected]

author maintainer

2021-09-15 14:38 UTC #0 lore raw

The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From 02bcec3ea5591720114f586960490b04b093a09e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pali=20Roh=C3=A1r?= <[email protected]>
Date: Thu, 22 Jul 2021 16:40:39 +0200
Subject: [PATCH] PCI: aardvark: Increase polling delay to 1.5s while waiting
for PIO response
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Measurements in different conditions showed that aardvark hardware PIO response can take up to 1.44s. Increase wait timeout from 1ms to 1.5s to ensure that we do not miss responses from hardware. After 1.44s hardware returns errors (e.g. Completer abort).

The previous two patches fixed checking for PIO status, so now we can use it to also catch errors which are reported by hardware after 1.44s.

After applying this patch, kernel can detect and print PIO errors to dmesg:

    [    6.879999] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100004
    [    6.896436] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004
    [    6.913049] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100010
    [    6.929663] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100010
    [    6.953558] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100014
    [    6.970170] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100014
    [    6.994328] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004

Without this patch kernel prints only a generic error to dmesg:

    [    5.246847] advk-pcie d0070000.pcie: config read/write timed out

Link https://lore.kernel.org/r/[email protected] open here

Signed-off-by Pali Rohár <[email protected]>

Signed-off-by Lorenzo Pieralisi <[email protected]>

Reviewed-by Marek Behún <[email protected]>

Cc [email protected] # 7fbcb5da811b ("PCI: aardvark: Don't rely on jiffies while holding spinlock")

drivers/pci/controller/pci-aardvark.c +1 -1

index 8bd060e084f1..5b9e4e79c3ae 100644
@@ -167,7 +167,7 @@
 #define PCIE_CONFIG_WR_TYPE0			0xa
 #define PCIE_CONFIG_WR_TYPE1			0xb
 
-#define PIO_RETRY_CNT			500
+#define PIO_RETRY_CNT			750000 /* 1.5 s */
 #define PIO_RETRY_DELAY			2 /* 2 us*/
 
 #define LINK_WAIT_MAX_RETRIES		10

Pali Rohár

maintainer Reviewed-by Marek Behún

2021-09-15 16:52 UTC #1 lore raw

Quoted 4 lines · On Wednesday 15 September 2021 16:38:07 [email protected] wrote:

The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

Hello! Below is backport for 5.10 (and probably it should apply also for older versions):

From 4c801c70bdcd34ca0527d54206c0358a73154801 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pali=20Roh=C3=A1r?= <[email protected]>
Date: Thu, 22 Jul 2021 16:40:39 +0200
Subject: [PATCH] PCI: aardvark: Increase polling delay to 1.5s while waiting
for PIO response
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The previous two patches fixed checking for PIO status, so now we can use it to also catch errors which are reported by hardware after 1.44s.

After applying this patch, kernel can detect and print PIO errors to dmesg:

    [    6.879999] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100004
    [    6.896436] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004
    [    6.913049] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100010
    [    6.929663] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100010
    [    6.953558] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100014
    [    6.970170] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100014
    [    6.994328] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004

Without this patch kernel prints only a generic error to dmesg:

    [    5.246847] advk-pcie d0070000.pcie: config read/write timed out

Link https://lore.kernel.org/r/[email protected] open here

Signed-off-by Pali Rohár <[email protected]>

Signed-off-by Lorenzo Pieralisi <[email protected]>

Reviewed-by Marek Behún <[email protected]>

Cc [email protected] # 7fbcb5da811b ("PCI: aardvark: Don't rely on jiffies while holding spinlock")

---
 drivers/pci/controller/pci-aardvark.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/pci/controller/pci-aardvark.c +1 -1

index b1b41b61e0bd..11cf6f4e9775 100644
@@ -173,7 +173,7 @@
 	(PCIE_CONF_BUS(bus) | PCIE_CONF_DEV(PCI_SLOT(devfn))	| \
 	 PCIE_CONF_FUNC(PCI_FUNC(devfn)) | PCIE_CONF_REG(where))
 
-#define PIO_RETRY_CNT			500
+#define PIO_RETRY_CNT			750000 /* 1.5 s */
 #define PIO_RETRY_DELAY			2 /* 2 us*/
 
 #define LINK_WAIT_MAX_RETRIES		10

Signature · 1 line

2.20.1

Quoted 57 lines

thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From 02bcec3ea5591720114f586960490b04b093a09e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pali=20Roh=C3=A1r?= <[email protected]>
Date: Thu, 22 Jul 2021 16:40:39 +0200
Subject: [PATCH] PCI: aardvark: Increase polling delay to 1.5s while waiting
for PIO response
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Measurements in different conditions showed that aardvark hardware PIO response can take up to 1.44s. Increase wait timeout from 1ms to 1.5s to ensure that we do not miss responses from hardware. After 1.44s hardware returns errors (e.g. Completer abort).
The previous two patches fixed checking for PIO status, so now we can use it to also catch errors which are reported by hardware after 1.44s.
After applying this patch, kernel can detect and print PIO errors to dmesg:
    [    6.879999] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100004
    [    6.896436] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004
    [    6.913049] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100010
    [    6.929663] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100010
    [    6.953558] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100014
    [    6.970170] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100014
    [    6.994328] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004
Without this patch kernel prints only a generic error to dmesg:
    [    5.246847] advk-pcie d0070000.pcie: config read/write timed out
Link https://lore.kernel.org/r/[email protected] open here
Signed-off-by Pali Rohár <[email protected]>
Signed-off-by Lorenzo Pieralisi <[email protected]>
Reviewed-by Marek Behún <[email protected]>
Cc [email protected] # 7fbcb5da811b ("PCI: aardvark: Don't rely on jiffies while holding spinlock")
drivers/pci/controller/pci-aardvark.c +1 -1
index 8bd060e084f1..5b9e4e79c3ae 100644
@@ -167,7 +167,7 @@
 #define PCIE_CONFIG_WR_TYPE0			0xa
 #define PCIE_CONFIG_WR_TYPE1			0xb
 
-#define PIO_RETRY_CNT			500
+#define PIO_RETRY_CNT			750000 /* 1.5 s */
 #define PIO_RETRY_DELAY			2 /* 2 us*/
 
 #define LINK_WAIT_MAX_RETRIES		10

Pali Rohár

maintainer

2021-09-25 21:46 UTC #2 lore raw

Quoted 8 lines · On Wednesday 15 September 2021 18:52:43 Pali Rohár wrote:

Quoted 4 lines · On Wednesday 15 September 2021 16:38:07 [email protected] wrote:
The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
Hello! Below is backport for 5.10 (and probably it should apply also for older versions):

Hello Greg! Have you looked at this backport for 5.10?

Quoted 114 lines

From 4c801c70bdcd34ca0527d54206c0358a73154801 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pali=20Roh=C3=A1r?= <[email protected]>
Date: Thu, 22 Jul 2021 16:40:39 +0200
Subject: [PATCH] PCI: aardvark: Increase polling delay to 1.5s while waiting
for PIO response
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Measurements in different conditions showed that aardvark hardware PIO response can take up to 1.44s. Increase wait timeout from 1ms to 1.5s to ensure that we do not miss responses from hardware. After 1.44s hardware returns errors (e.g. Completer abort).
The previous two patches fixed checking for PIO status, so now we can use it to also catch errors which are reported by hardware after 1.44s.
After applying this patch, kernel can detect and print PIO errors to dmesg:
    [    6.879999] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100004
    [    6.896436] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004
    [    6.913049] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100010
    [    6.929663] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100010
    [    6.953558] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100014
    [    6.970170] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100014
    [    6.994328] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004
Without this patch kernel prints only a generic error to dmesg:
    [    5.246847] advk-pcie d0070000.pcie: config read/write timed out
Link https://lore.kernel.org/r/[email protected] open here
Signed-off-by Pali Rohár <[email protected]>
Signed-off-by Lorenzo Pieralisi <[email protected]>
Reviewed-by Marek Behún <[email protected]>
Cc [email protected] # 7fbcb5da811b ("PCI: aardvark: Don't rely on jiffies while holding spinlock")
---
 drivers/pci/controller/pci-aardvark.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/pci/controller/pci-aardvark.c +1 -1
index b1b41b61e0bd..11cf6f4e9775 100644
@@ -173,7 +173,7 @@
 	(PCIE_CONF_BUS(bus) | PCIE_CONF_DEV(PCI_SLOT(devfn))	| \
 	 PCIE_CONF_FUNC(PCI_FUNC(devfn)) | PCIE_CONF_REG(where))
 
-#define PIO_RETRY_CNT			500
+#define PIO_RETRY_CNT			750000 /* 1.5 s */
 #define PIO_RETRY_DELAY			2 /* 2 us*/
 
 #define LINK_WAIT_MAX_RETRIES		10
Signature · 1 line
2.20.1
Quoted 57 lines
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From 02bcec3ea5591720114f586960490b04b093a09e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pali=20Roh=C3=A1r?= <[email protected]>
Date: Thu, 22 Jul 2021 16:40:39 +0200
Subject: [PATCH] PCI: aardvark: Increase polling delay to 1.5s while waiting
for PIO response
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Measurements in different conditions showed that aardvark hardware PIO response can take up to 1.44s. Increase wait timeout from 1ms to 1.5s to ensure that we do not miss responses from hardware. After 1.44s hardware returns errors (e.g. Completer abort).
The previous two patches fixed checking for PIO status, so now we can use it to also catch errors which are reported by hardware after 1.44s.
After applying this patch, kernel can detect and print PIO errors to dmesg:
    [    6.879999] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100004
    [    6.896436] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004
    [    6.913049] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100010
    [    6.929663] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100010
    [    6.953558] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100014
    [    6.970170] advk-pcie d0070000.pcie: Non-posted PIO Response Status: CA, 0xe00 @ 0x100014
    [    6.994328] advk-pcie d0070000.pcie: Posted PIO Response Status: COMP_ERR, 0x804 @ 0x100004
Without this patch kernel prints only a generic error to dmesg:
    [    5.246847] advk-pcie d0070000.pcie: config read/write timed out
Link https://lore.kernel.org/r/[email protected] open here
Signed-off-by Pali Rohár <[email protected]>
Signed-off-by Lorenzo Pieralisi <[email protected]>
Reviewed-by Marek Behún <[email protected]>
Cc [email protected] # 7fbcb5da811b ("PCI: aardvark: Don't rely on jiffies while holding spinlock")
drivers/pci/controller/pci-aardvark.c +1 -1
index 8bd060e084f1..5b9e4e79c3ae 100644
@@ -167,7 +167,7 @@
 #define PCIE_CONFIG_WR_TYPE0			0xa
 #define PCIE_CONFIG_WR_TYPE1			0xb
 
-#define PIO_RETRY_CNT			500
+#define PIO_RETRY_CNT			750000 /* 1.5 s */
 #define PIO_RETRY_DELAY			2 /* 2 us*/
 
 #define LINK_WAIT_MAX_RETRIES		10

Greg KH

author maintainer

2021-09-26 12:19 UTC #3 lore raw

Quoted 11 lines · Pali Rohár

Quoted 8 lines · On Wednesday 15 September 2021 18:52:43 Pali Rohár wrote:
Quoted 4 lines · On Wednesday 15 September 2021 16:38:07 [email protected] wrote:
The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
Hello! Below is backport for 5.10 (and probably it should apply also for older versions):
Hello Greg! Have you looked at this backport for 5.10?

Ick, I somehow missed this for 5.10.y, thanks for catching it. I'll go queue it up now.

greg k-h

Pali Rohár

maintainer

2021-09-26 13:55 UTC #4 lore raw

Quoted 8 lines · On Sunday 26 September 2021 14:19:20 Greg KH wrote:

Quoted 7 lines · Pali Rohár
Quoted 6 lines · On Wednesday 15 September 2021 18:52:43 Pali Rohár wrote:
Quoted 4 lines · On Wednesday 15 September 2021 16:38:07 [email protected] wrote:
The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

Quoted 7 lines

Quoted 4 lines
Quoted 2 lines
Hello! Below is backport for 5.10 (and probably it should apply also for older versions):
Hello Greg! Have you looked at this backport for 5.10?
Ick, I somehow missed this for 5.10.y, thanks for catching it. I'll go queue it up now.

Ok!

Now I'm checking other aardvark patches and I found out that following commits marked with Cc: stable tags are not included in 4.14 tree yet:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8ceeac307a79f68c0d0c72d6e48b82fa424204ec
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb461e2bc8b83b7eaca20cb2221e8b940f2189c

And this in 4.19 stable tree:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb461e2bc8b83b7eaca20cb2221e8b940f2189c

With merge.renamelimit = 24506 these commits applies cleanly for 4.14 / 4.19 stable trees. Could you look at it, why there are missing?

Greg KH

author maintainer

2021-09-26 14:07 UTC #5 lore raw

Quoted 25 lines · Pali Rohár

Quoted 8 lines · On Sunday 26 September 2021 14:19:20 Greg KH wrote:
Quoted 7 lines · Pali Rohár
Quoted 6 lines · On Wednesday 15 September 2021 18:52:43 Pali Rohár wrote:
Quoted 4 lines · On Wednesday 15 September 2021 16:38:07 [email protected] wrote:
The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Hello! Below is backport for 5.10 (and probably it should apply also for older versions):
Hello Greg! Have you looked at this backport for 5.10?
Ick, I somehow missed this for 5.10.y, thanks for catching it. I'll go queue it up now.
Ok!
Now I'm checking other aardvark patches and I found out that following commits marked with Cc: stable tags are not included in 4.14 tree yet:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8ceeac307a79f68c0d0c72d6e48b82fa424204ec
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb461e2bc8b83b7eaca20cb2221e8b940f2189c

Quoted 6 lines

And this in 4.19 stable tree:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb461e2bc8b83b7eaca20cb2221e8b940f2189c
With merge.renamelimit = 24506 these commits applies cleanly for 4.14 / 4.19 stable trees. Could you look at it, why there are missing?

They are missing because I do not use renames when applying patches like this (we use quilt for the patch queue).

If you can send the updated patches, I will be glad to queue them up.

thanks,

greg k-h

Pali Rohár

maintainer

2021-09-26 14:26 UTC #6 lore raw

Quoted 30 lines · On Sunday 26 September 2021 16:07:50 Greg KH wrote:

Quoted 29 lines · Pali Rohár
Quoted 8 lines · On Sunday 26 September 2021 14:19:20 Greg KH wrote:
Quoted 7 lines · Pali Rohár
Quoted 6 lines · On Wednesday 15 September 2021 18:52:43 Pali Rohár wrote:
Quoted 4 lines · On Wednesday 15 September 2021 16:38:07 [email protected] wrote:
The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Hello! Below is backport for 5.10 (and probably it should apply also for older versions):
Hello Greg! Have you looked at this backport for 5.10?
Ick, I somehow missed this for 5.10.y, thanks for catching it. I'll go queue it up now.
Ok!
Now I'm checking other aardvark patches and I found out that following commits marked with Cc: stable tags are not included in 4.14 tree yet:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8ceeac307a79f68c0d0c72d6e48b82fa424204ec
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb461e2bc8b83b7eaca20cb2221e8b940f2189c
And this in 4.19 stable tree:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb461e2bc8b83b7eaca20cb2221e8b940f2189c

Quoted 7 lines

Quoted 2 lines
With merge.renamelimit = 24506 these commits applies cleanly for 4.14 / 4.19 stable trees. Could you look at it, why there are missing?
They are missing because I do not use renames when applying patches like this (we use quilt for the patch queue).
If you can send the updated patches, I will be glad to queue them up.

Ok. Now I have sent them to stable list.

Quoted 3 lines

thanks,
greg k-h

Greg KH

author maintainer

2021-09-27 11:06 UTC #7 lore raw

Quoted 34 lines · Pali Rohár

Quoted 33 lines · On Sunday 26 September 2021 16:07:50 Greg KH wrote:
Quoted 31 lines · Pali Rohár
Quoted 8 lines · On Sunday 26 September 2021 14:19:20 Greg KH wrote:
Quoted 7 lines · Pali Rohár
Quoted 6 lines · On Wednesday 15 September 2021 18:52:43 Pali Rohár wrote:
Quoted 4 lines · On Wednesday 15 September 2021 16:38:07 [email protected] wrote:
The patch below does not apply to the 5.10-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Hello! Below is backport for 5.10 (and probably it should apply also for older versions):
Hello Greg! Have you looked at this backport for 5.10?
Ick, I somehow missed this for 5.10.y, thanks for catching it. I'll go queue it up now.
Ok!
Now I'm checking other aardvark patches and I found out that following commits marked with Cc: stable tags are not included in 4.14 tree yet:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8ceeac307a79f68c0d0c72d6e48b82fa424204ec
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb461e2bc8b83b7eaca20cb2221e8b940f2189c
And this in 4.19 stable tree:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb461e2bc8b83b7eaca20cb2221e8b940f2189c
With merge.renamelimit = 24506 these commits applies cleanly for 4.14 / 4.19 stable trees. Could you look at it, why there are missing?

Quoted 6 lines

Quoted 4 lines
They are missing because I do not use renames when applying patches like this (we use quilt for the patch queue).
If you can send the updated patches, I will be glad to queue them up.
Ok. Now I have sent them to stable list.

Thanks, all now queued up.

greg k-h