Skip to contentSkip to conversation

vdpa: Return -EIO if device ack is VIRTIO_NET_ERR

PATCH In review Hawkins Jiawei 3 participants 10 messages Jul 4, 2023 – Aug 5, 2023 Networking

lore
Share & tools
lore
#1 lore raw

PATCH v3 1/3 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()

In reply to Hawkins Jiawei

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_mac() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patch fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Fixes f73c0c43ac ("vdpa: extract vhost_vdpa_net_load_mac from vhost_vdpa_net_load")

---
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2: https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

v1: https://lore.kernel.org/all/07a1133d6c989394b342e35d8202257771e76769.1686746406.git.yin31149@gmail.com/

 net/vhost-vdpa.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
net/vhost-vdpa.c +3 -2
index e19ab063fa..ee273c40ca 100644
@@ -646,8 +646,9 @@ static int vhost_vdpa_net_load_mac(VhostVDPAState *s, const VirtIONet *n)
         if (unlikely(dev_written < 0)) {
             return dev_written;
         }
-
-        return *s->status != VIRTIO_NET_OK;
+        if (*s->status != VIRTIO_NET_OK) {
+            return -EIO;
+        }
     }
 
     return 0;
Signature · 1 line
2.25.1
#2 lore raw

PATCH v3 2/3 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()

In reply to Hawkins Jiawei

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_mq() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patch fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Fixes f64c7cda69 ("vdpa: Add vhost_vdpa_net_load_mq")

---
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2: https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

v1: https://lore.kernel.org/all/07a1133d6c989394b342e35d8202257771e76769.1686746406.git.yin31149@gmail.com/

 net/vhost-vdpa.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/vhost-vdpa.c +4 -1
index ee273c40ca..03d87e85c8 100644
@@ -671,8 +671,11 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
     if (unlikely(dev_written < 0)) {
         return dev_written;
     }
+    if (*s->status != VIRTIO_NET_OK) {
+        return -EIO;
+    }
 
-    return *s->status != VIRTIO_NET_OK;
+    return 0;
 }
 
 static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
Signature · 1 line
2.25.1
#3 lore raw

PATCH v3 3/3 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()

In reply to Hawkins Jiawei

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_offloads() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patch fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Fixes 0b58d3686a ("vdpa: Add vhost_vdpa_net_load_offloads()")

---
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2: https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

v1: https://lore.kernel.org/all/07a1133d6c989394b342e35d8202257771e76769.1686746406.git.yin31149@gmail.com/

 net/vhost-vdpa.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
net/vhost-vdpa.c +4 -1
index 03d87e85c8..36aa2d7f8c 100644
@@ -712,8 +712,11 @@ static int vhost_vdpa_net_load_offloads(VhostVDPAState *s,
     if (unlikely(dev_written < 0)) {
         return dev_written;
     }
+    if (*s->status != VIRTIO_NET_OK) {
+        return -EIO;
+    }
 
-    return *s->status != VIRTIO_NET_OK;
+    return 0;
 }
 
 static int vhost_vdpa_net_load(NetClientState *nc)
Signature · 1 line
2.25.1
#0 lore raw

v3 cover vdpa: Return -EIO if device ack is VIRTIO_NET_ERR

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_x() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patchset fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Changelog
=========
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2: https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (3):
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()

 net/vhost-vdpa.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
Signature · 1 line
2.25.1
#4 lore raw

Hello Hawkins

QE can help test this series before it is merged into master, I would like to know what test steps can cover this series related scenario?

Thanks
Lei

Quoted 45 lines · Hawkins Jiawei

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_x() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patchset fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Changelog
=========
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2: https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (3):
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()

 net/vhost-vdpa.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

--
2.25.1

#5 lore raw
Quoted 5 lines · Lei Yang

Hello Hawkins

QE can help test this series before it is merged into master, I would like to know what test steps can cover this series related scenario?

Hi, I would like to suggest the following steps to test this patch series:

1. Modify the QEMU source code to make the device return a VIRTIO_NET_ERR for the CVQ command. Please apply the patch provided below:

net/vhost-vdpa.c +0 -0
index 373609216f..58ade6d4e0 100644
@@ -642,7 +642,7 @@ static int vhost_vdpa_net_load_mac(VhostVDPAState

*s, const VirtIONet *n)

      if (virtio_vdev_has_feature(&n->parent_obj,

VIRTIO_NET_F_CTRL_MAC_ADDR)) {
ssize_t dev_written = vhost_vdpa_net_load_cmd(s,
VIRTIO_NET_CTRL_MAC,

VIRTIO_NET_CTRL_MAC_ADDR_SET,
- n->mac, sizeof(n->mac));
+ n->mac,

sizeof(n->mac) - 1);
          if (unlikely(dev_written < 0)) {
              return dev_written;
          }

2. Start QEMU with the vdpa device in default state.
Without the patch series, QEMU should not trigger any errors or warnings. With the series applied, QEMU should trigger the warning like "qemu-system-x86_64: unable to start vhost net: 5: falling back on userspace virtio".

Thanks!

Quoted 50 lines

Thanks
Lei

Quoted 45 lines · Hawkins Jiawei

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_x() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patchset fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Changelog
=========
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2: https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (3):
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()

  net/vhost-vdpa.c | 15 +++++++++++----
  1 file changed, 11 insertions(+), 4 deletions(-)

--
2.25.1

Tested-by
#6 lore raw
Quoted 33 lines · Hawkins Jiawei
Quoted 5 lines · Lei Yang

Hello Hawkins

QE can help test this series before it is merged into master, I would like to know what test steps can cover this series related scenario?

Hi, I would like to suggest the following steps to test this patch series:

1. Modify the QEMU source code to make the device return a VIRTIO_NET_ERR for the CVQ command. Please apply the patch provided below:

net/vhost-vdpa.c +1 -1
index 373609216f..58ade6d4e0 100644
@@ -642,7 +642,7 @@ static int vhost_vdpa_net_load_mac(VhostVDPAState
 *s, const VirtIONet *n)
      if (virtio_vdev_has_feature(&n->parent_obj,
 VIRTIO_NET_F_CTRL_MAC_ADDR)) {
          ssize_t dev_written = vhost_vdpa_net_load_cmd(s,
 VIRTIO_NET_CTRL_MAC,
 
 VIRTIO_NET_CTRL_MAC_ADDR_SET,
-                                                  n->mac, sizeof(n->mac));
+                                                  n->mac,
sizeof(n->mac) - 1);
          if (unlikely(dev_written < 0)) {
              return dev_written;
          }
Quoted 5 lines

2. Start QEMU with the vdpa device in default state.
Without the patch series, QEMU should not trigger any errors or warnings. With the series applied, QEMU should trigger the warning like "qemu-system-x86_64: unable to start vhost net: 5: falling back on userspace virtio".

Based on the above steps, QE tests it without the above patch first, it will not trigger any errors or warnings. Then QE manually applied the above patch, boot guest again, it can trigger this warning:
qemu-system-x86_64: unable to start vhost net: 5: falling back on userspace virtio, this is an expected result.

BR
Lei

Quoted 55 lines

Thanks!

Quoted 50 lines

Thanks
Lei

Quoted 45 lines · Hawkins Jiawei

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_x() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patchset fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Changelog
=========
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2: https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (3):
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()

  net/vhost-vdpa.c | 15 +++++++++++----
  1 file changed, 11 insertions(+), 4 deletions(-)

--
2.25.1

#7 lore raw
In reply to Hawkins Jiawei
Quoted 30 lines · Hawkins Jiawei

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_x() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patchset fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Changelog
=========
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2: https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

Quoted 6 lines

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (3):
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()

Hi!

I don't remember why, but this patch series is marked as "check later" in my qemu-stable-to-apply email folder. Does it make sense to back-port this series to stable-8.0?

6f34807116 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads() f45fd95ec9 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq() b479bc3c9d vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()

Patch 6f34807116 also needs

b58d3686a0 vdpa: Add vhost_vdpa_net_load_offloads()

for 8.0.

Thanks,

/mjt

#8 lore raw
Quoted 41 lines · Michael Tokarev
Quoted 31 lines · Hawkins Jiawei

According to VirtIO standard, "The class, command and command-specific-data are set by the driver, and the device sets the ack byte.
There is little it can do except issue a diagnostic if ack is not VIRTIO_NET_OK."

Therefore, QEMU should stop sending the queued SVQ commands and cancel the device startup if the device's ack is not VIRTIO_NET_OK.

Yet the problem is that, vhost_vdpa_net_load_x() returns 1 based on *s->status != VIRTIO_NET_OK when the device's ack is VIRTIO_NET_ERR. As a result, net->nc->info->load() also returns 1, this makes vhost_net_start_one() incorrectly assume the device state is successfully loaded by vhost_vdpa_net_load() and return 0, instead of goto fail label to cancel the device startup, as vhost_net_start_one() only cancels the device startup when net->nc->info->load() returns a negative value.

This patchset fixes this problem by returning -EIO when the device's ack is not VIRTIO_NET_OK.

Changelog
=========
v3:
- split the fixes suggested by Eugenio
- return -EIO suggested by Michael

v2:
https://lore.kernel.org/all/69010e9ebb5e3729aef595ed92840f43e48e53e5.1687875592.git.yin31149@gmail.com/
- fix the same bug in vhost_vdpa_net_load_offloads()

Quoted 6 lines

v1: https://lore.kernel.org/all/[email protected]/

Hawkins Jiawei (3):
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()
vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()

Hi!

Quoted 9 lines

I don't remember why, but this patch series is marked as "check later" in my qemu-stable-to-apply email folder. Does it make sense to back-port this series to stable-8.0?

6f34807116 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()
f45fd95ec9 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq() b479bc3c9d vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()

Hi Michael,

Yes, this bug exists in stable-8.0, so it makes sense to back-port this series.

Commit f45fd95ec9("vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()") and
commit b479bc3c9d("vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()") can be back-ported directly.

Quoted 3 lines

Patch 6f34807116 also needs

b58d3686a0 vdpa: Add vhost_vdpa_net_load_offloads()

As you point out, patch 6f34807116("vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()") is a fix to the commit b58d3686a0("vdpa: Add vhost_vdpa_net_load_offloads()"), which was introduced by patch series "Vhost-vdpa Shadow Virtqueue Offloads support" at [1].

This mentioned patch series introduces a new feature for QEMU and has not been merged into stable-8.0 yet, so I think we do not need to apply the
patch 6f34807116("vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()") to stable-8.0.

Sorry for not mentioning this information in the cover letter.

Thanks!

[1]. https://lore.kernel.org/all/[email protected]/

Quoted 6 lines

for 8.0.

Thanks,

/mjt

#9 lore raw

05.08.2023 12:28, Hawkins Jiawei wrote:
..

Quoted 30 lines
Quoted 9 lines

I don't remember why, but this patch series is marked as "check later" in my qemu-stable-to-apply email folder. Does it make sense to back-port this series to stable-8.0?

6f34807116 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()
f45fd95ec9 vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq() b479bc3c9d vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()

Hi Michael,

Yes, this bug exists in stable-8.0, so it makes sense to back-port this series.

Commit f45fd95ec9("vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mq()") and
commit b479bc3c9d("vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_mac()") can be back-ported directly.

Quoted 3 lines

Patch 6f34807116 also needs

b58d3686a0 vdpa: Add vhost_vdpa_net_load_offloads()

As you point out, patch 6f34807116("vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()") is a fix to the commit b58d3686a0("vdpa: Add vhost_vdpa_net_load_offloads()"), which was introduced by patch series "Vhost-vdpa Shadow Virtqueue Offloads support" at [1].

Quoted 5 lines

This mentioned patch series introduces a new feature for QEMU and has not been merged into stable-8.0 yet, so I think we do not need to apply the
patch 6f34807116("vdpa: Return -EIO if device ack is VIRTIO_NET_ERR in _load_offloads()") to stable-8.0.

Ok, this makes sense, thank you for making it clear.

Quoted 1 line

Sorry for not mentioning this information in the cover letter.

That's okay. I too didn't pick it up in time, so fixing this now :

/mjt