vhost/migration: Fix incorrect size used in inflight->addr in VMSD
PATCH 1/1 vhost/migration: Fix incorrect size used in inflight->addr in VMSD
It was overlooked that VMSTATE_VBUFFER_UINT64() won't really work with an uint64_t, as vmstate core only treats the size as 32bits, and maximum INT32_MAX (see vmstate_size()).
Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue __size_32bits to be used only for VMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.
Remove VMSTATE_VBUFFER_UINT64() altogether, because it was never going to be used right. It means QEMU will only support 2G max for VMS_VBUFFER.
Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
PS2: Michael, we have just discussed what we should define as CVE for migration, and this one shouldn't fall into CVE category, please refer to:
https://lore.kernel.org/r/[email protected]
So I didn't yet attach CVE tag. Please correct if I'm wrong, thanks.
--- include/hw/virtio/vhost.h | 5 +++++ include/migration/vmstate.h | 10 ---------- hw/virtio/vhost.c | 19 ++++++++++++++----- 3 files changed, 19 insertions(+), 15 deletions(-)
include/hw/virtio/vhost.h +5 -0
@@ -17,6 +17,11 @@ struct vhost_inflight {int fd;void *addr;uint64_t size; +/* +* This is a temporary variable only used during migration loading to +* satisfy VMSTATE_VBUFFER_UINT32() typing. Please use @size otherwise. +*/ +uint32_t __size_32bits;uint64_t offset;uint16_t queue_size; };
include/migration/vmstate.h +0 -10
@@ -782,16 +782,6 @@ extern const VMStateInfo vmstate_info_g_byte_array; .offset = offsetof(_state, _field), \ } -#define VMSTATE_VBUFFER_UINT64 (_field, _state, _version, _test, _field_size) { \ - .name = (stringify(_field)), \ - .version_id = (_version), \ - .field_exists = (_test), \ - .size_offset = vmstate_offset_value(_state, _field_size, uint64_t),\ - .info = &vmstate_info_buffer, \ - .flags = VMS_VBUFFER | VMS_POINTER, \ - .offset = offsetof(_state, _field), \ -} -#define VMSTATE_VBUFFER_ALLOC_UINT32 (_field, _state, _version, \ _test, _field_size) { \ .name = (stringify(_field)), \
hw/virtio/vhost.c +14 -5
@@ -2022,15 +2022,24 @@ void vhost_get_features_ex(struct vhost_dev *hdev,static bool vhost_inflight_buffer_pre_load (void *opaque,Error **errp) {struct vhost_inflight *inflight = opaque; -int fd =-1 ; -void *addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, -F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , - &fd, errp); +void *addr; + +if (inflight->size >INT32_MAX ) { +error_setg (errp,"inflight size '%" PRIu64"' exceeds " +"migration limit '%" PRIu32"'" , inflight->size,INT32_MAX ); +return false; + } + + addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, +F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , + &fd, errp);if (!addr) {return false; } +/* Only used in VMSTATE_VBUFFER_UINT32() */ + inflight->__size_32bits = inflight->size; inflight->offset =0 ; inflight->addr = addr; inflight->fd = fd;
@@ -2042,7 +2051,7 @@ const VMStateDescription vmstate_vhost_inflight_region_buffer = { .name = "vhost-inflight-region/buffer", .pre_load_errp = vhost_inflight_buffer_pre_load, .fields = (const VMStateField []) { - VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size), + VMSTATE_VBUFFER_UINT32(addr, struct vhost_inflight, 0, NULL, __size_32bits),VMSTATE_END_OF_LIST () } };
Signature · 1 line
2.54.0
Quoted 65 lines · Peter Xu
It was overlooked that
VMSTATE_VBUFFER_UINT64()won't really work with anuint64_t, as vmstate core only treats the size as 32bits, and maximumINT32_MAX(seevmstate_size()).Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to
uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue__size_32bitsto be used only forVMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.Remove
VMSTATE_VBUFFER_UINT64()altogether, because it was never going to be used right. It means QEMU will only support 2G max forVMS_VBUFFER.Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
Reported-by 김승중 <[email protected]>Cc Alexandr Moshkov <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Fabiano Rosas <[email protected]>Fixes 3a80ff0721 ("vhost: add vmstate for inflight region with inner buffer")Signed-off-by Peter Xu <[email protected]>---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
PS2: Michael, we have just discussed what we should define as CVE for migration, and this one shouldn't fall into CVE category, please refer to:
https://lore.kernel.org/r/[email protected]
So I didn't yet attach CVE tag. Please correct if I'm wrong, thanks.
--- include/hw/virtio/vhost.h | 5 +++++ include/migration/vmstate.h | 10 ---------- hw/virtio/vhost.c | 19 ++++++++++++++----- 3 files changed, 19 insertions(+), 15 deletions(-)include/hw/virtio/vhost.h +5
-0@@ -17,6 +17,11 @@ struct vhost_inflight {int fd;void *addr;uint64_t size; +/* +* This is a temporary variable only used during migration loading to +* satisfy VMSTATE_VBUFFER_UINT32() typing. Please use @size otherwise. +*/ +uint32_t __size_32bits;uint64_t offset;uint16_t queue_size; };include/migration/vmstate.h +0
-0@@ -782,16 +782,6 @@ extern const VMStateInfo vmstate_info_g_byte_array; .offset = offsetof(_state, _field), \ }
include/migration/vmstate.h extern const VMStateInfo vmstate_info_g_byte_array; · line 793
Quoted 10 lines
include/migration/vmstate.h +0
-9-#define VMSTATE_VBUFFER_UINT64 (_field, _state, _version, _test, _field_size) { \ - .name = (stringify(_field)), \ - .version_id = (_version), \ - .field_exists = (_test), \ - .size_offset = vmstate_offset_value(_state, _field_size, uint64_t),\ - .info = &vmstate_info_buffer, \ - .flags = VMS_VBUFFER | VMS_POINTER, \ - .offset = offsetof(_state, _field), \ -}
Why don't you just leave this macro around, put a "do not use" comment above it and remove the type check for this one instance? You're already checking against INT32_MAX during pre_load.
We could even lift that check up into vmstate_pre_load and reject there globally, don't even touch the virtio code.
/rant
... what's even the point of having per-type variants of VMSTATE macros that exist just to type-check the extra offsets (.num_offset, .size_offset as opposed to .offset)?
For instance, look at vmstate_n_elems casting opaque data to 32 and sub-32bit size and assigning to an int! What do we gain from that? It's circular reasoning that does nothing aside from bothering the person writing the vmstate.
Right? Maybe I'm missing something, it's the end of the day already.
Quoted 46 lines
include/migration/vmstate.h +0
-1-#define VMSTATE_VBUFFER_ALLOC_UINT32 (_field, _state, _version, \ _test, _field_size) { \ .name = (stringify(_field)), \hw/virtio/vhost.c +14
-5@@ -2022,15 +2022,24 @@ void vhost_get_features_ex(struct vhost_dev *hdev,static bool vhost_inflight_buffer_pre_load (void *opaque,Error **errp) {struct vhost_inflight *inflight = opaque; -int fd =-1 ; -void *addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, -F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , - &fd, errp); +void *addr; + +if (inflight->size >INT32_MAX ) { +error_setg (errp,"inflight size '%" PRIu64"' exceeds " +"migration limit '%" PRIu32"'" , inflight->size,INT32_MAX ); +return false; + } + + addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, +F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , + &fd, errp);if (!addr) {return false; } +/* Only used in VMSTATE_VBUFFER_UINT32() */ + inflight->__size_32bits = inflight->size; inflight->offset =0 ; inflight->addr = addr; inflight->fd = fd;@@ -2042,7 +2051,7 @@ const VMStateDescription vmstate_vhost_inflight_region_buffer = { .name = "vhost-inflight-region/buffer", .pre_load_errp = vhost_inflight_buffer_pre_load, .fields = (const VMStateField []) { - VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size), + VMSTATE_VBUFFER_UINT32(addr, struct vhost_inflight, 0, NULL, __size_32bits),VMSTATE_END_OF_LIST () } };
Quoted 68 lines · Fabiano Rosas
Quoted 66 lines · Peter Xu
It was overlooked that
VMSTATE_VBUFFER_UINT64()won't really work with anuint64_t, as vmstate core only treats the size as 32bits, and maximumINT32_MAX(seevmstate_size()).Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to
uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue__size_32bitsto be used only forVMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.Remove
VMSTATE_VBUFFER_UINT64()altogether, because it was never going to be used right. It means QEMU will only support 2G max forVMS_VBUFFER.Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
Reported-by 김승중 <[email protected]>Cc Alexandr Moshkov <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Fabiano Rosas <[email protected]>Fixes 3a80ff0721 ("vhost: add vmstate for inflight region with inner buffer")Signed-off-by Peter Xu <[email protected]>---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
PS2: Michael, we have just discussed what we should define as CVE for migration, and this one shouldn't fall into CVE category, please refer to:
https://lore.kernel.org/r/[email protected]
So I didn't yet attach CVE tag. Please correct if I'm wrong, thanks.
--- include/hw/virtio/vhost.h | 5 +++++ include/migration/vmstate.h | 10 ---------- hw/virtio/vhost.c | 19 ++++++++++++++----- 3 files changed, 19 insertions(+), 15 deletions(-)include/hw/virtio/vhost.h +5
-0@@ -17,6 +17,11 @@ struct vhost_inflight {int fd;void *addr;uint64_t size; +/* +* This is a temporary variable only used during migration loading to +* satisfy VMSTATE_VBUFFER_UINT32() typing. Please use @size otherwise. +*/ +uint32_t __size_32bits;uint64_t offset;uint16_t queue_size; };include/migration/vmstate.h +0
-0@@ -782,16 +782,6 @@ extern const VMStateInfo vmstate_info_g_byte_array; .offset = offsetof(_state, _field), \ }
Quoted 13 lines
include/migration/vmstate.h extern const VMStateInfo vmstate_info_g_byte_array; · line 793
Quoted 9 lines
include/migration/vmstate.h +0
-9-#define VMSTATE_VBUFFER_UINT64 (_field, _state, _version, _test, _field_size) { \ - .name = (stringify(_field)), \ - .version_id = (_version), \ - .field_exists = (_test), \ - .size_offset = vmstate_offset_value(_state, _field_size, uint64_t),\ - .info = &vmstate_info_buffer, \ - .flags = VMS_VBUFFER | VMS_POINTER, \ - .offset = offsetof(_state, _field), \ -}Why don't you just leave this macro around, put a "do not use" comment above it and remove the type check for this one instance? You're already checking against
INT32_MAXduringpre_load.
What's the benefit of keeping this global macro if we only allow one user to use it?
Commenting to say "don't use" is not guaranteeing anything, at least we should also rename the macro to some weird name to make people notice. Keeping the macro name like this OTOH suggests abuse. But if we think it should only be used in 1 place, fixing the one place might be better?
Quoted 3 lines
We could even lift that check up into
vmstate_pre_loadand reject there globally, don't even touch the virtio code.
Personally, I don't like the idea leaking VBUFFER impl details into vmstate_pre_load() which is so far only a wrapper, especially only for this one instance.. which we do not suggest future users to use.
If we go this route, I'd rather merge Michael's version to support u64, even if we don't need a u64 size. But I really don't want to introduce yet another VMS flag just for this... we'll have no real use if we have noticed this problem when the vhost inflight patch was reviewed. It will be a uint32_t or int32_t already. I just can't come up with some users need size >2G.
The recent AI reports just make such feeling stronger: we just used the reason "max 2G, should be fine" when AI reports that unlimited size allocation problem, now we will need to wait for another AI report which says "it's not 2G anymore, 1<<64-1 that is", if we don't come up with a proper limit for VMSD field allocations.
Quoted 7 lines
/rant
... what's even the point of having per-type variants of VMSTATE macros that exist just to type-check the extra offsets (.num_offset, .size_offset as opposed to .offset)?
Quoted 6 lines
For instance, look at
vmstate_n_elemscasting opaque data to 32 and sub-32bit size and assigning to an int! What do we gain from that? It's circular reasoning that does nothing aside from bothering the person writing the vmstate.Right? Maybe I'm missing something, it's the end of the day already.
I can also only guess, which is.. I believe Michael was right, the initial vmstate code was simply broken where it should have considered the type of size fields of all kinds, but forgot, and it just worked because nobody needed u64 for a size field.
Said that, I still want to see if we can avoid introducing that at all. To me, I still prefer this patch (after fixing pre_save()..). But let me know if I didn't convince any of you.. we can discuss.
Thanks,
Quoted 48 lines
Quoted 47 lines
include/migration/vmstate.h +0
-1-#define VMSTATE_VBUFFER_ALLOC_UINT32 (_field, _state, _version, \ _test, _field_size) { \ .name = (stringify(_field)), \hw/virtio/vhost.c +14
-5@@ -2022,15 +2022,24 @@ void vhost_get_features_ex(struct vhost_dev *hdev,static bool vhost_inflight_buffer_pre_load (void *opaque,Error **errp) {struct vhost_inflight *inflight = opaque; -int fd =-1 ; -void *addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, -F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , - &fd, errp); +void *addr; + +if (inflight->size >INT32_MAX ) { +error_setg (errp,"inflight size '%" PRIu64"' exceeds " +"migration limit '%" PRIu32"'" , inflight->size,INT32_MAX ); +return false; + } + + addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, +F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , + &fd, errp);if (!addr) {return false; } +/* Only used in VMSTATE_VBUFFER_UINT32() */ + inflight->__size_32bits = inflight->size; inflight->offset =0 ; inflight->addr = addr; inflight->fd = fd;@@ -2042,7 +2051,7 @@ const VMStateDescription vmstate_vhost_inflight_region_buffer = { .name = "vhost-inflight-region/buffer", .pre_load_errp = vhost_inflight_buffer_pre_load, .fields = (const VMStateField []) { - VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size), + VMSTATE_VBUFFER_UINT32(addr, struct vhost_inflight, 0, NULL, __size_32bits),VMSTATE_END_OF_LIST () } };
Signature · 1 line
Peter Xu
Quoted 99 lines · Peter Xu
Quoted 68 lines · Fabiano Rosas
Quoted 66 lines · Peter Xu
It was overlooked that
VMSTATE_VBUFFER_UINT64()won't really work with anuint64_t, as vmstate core only treats the size as 32bits, and maximumINT32_MAX(seevmstate_size()).Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to
uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue__size_32bitsto be used only forVMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.Remove
VMSTATE_VBUFFER_UINT64()altogether, because it was never going to be used right. It means QEMU will only support 2G max forVMS_VBUFFER.Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
Reported-by 김승중 <[email protected]>Cc Alexandr Moshkov <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Fabiano Rosas <[email protected]>Fixes 3a80ff0721 ("vhost: add vmstate for inflight region with inner buffer")Signed-off-by Peter Xu <[email protected]>---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
PS2: Michael, we have just discussed what we should define as CVE for migration, and this one shouldn't fall into CVE category, please refer to:
https://lore.kernel.org/r/[email protected]
So I didn't yet attach CVE tag. Please correct if I'm wrong, thanks.
--- include/hw/virtio/vhost.h | 5 +++++ include/migration/vmstate.h | 10 ---------- hw/virtio/vhost.c | 19 ++++++++++++++----- 3 files changed, 19 insertions(+), 15 deletions(-)include/hw/virtio/vhost.h +5
-0@@ -17,6 +17,11 @@ struct vhost_inflight {int fd;void *addr;uint64_t size; +/* +* This is a temporary variable only used during migration loading to +* satisfy VMSTATE_VBUFFER_UINT32() typing. Please use @size otherwise. +*/ +uint32_t __size_32bits;uint64_t offset;uint16_t queue_size; };include/migration/vmstate.h +0
-0@@ -782,16 +782,6 @@ extern const VMStateInfo vmstate_info_g_byte_array; .offset = offsetof(_state, _field), \ }Quoted 13 lines
include/migration/vmstate.h extern const VMStateInfo vmstate_info_g_byte_array; · line 793
Quoted 9 lines
include/migration/vmstate.h +0
-9-#define VMSTATE_VBUFFER_UINT64 (_field, _state, _version, _test, _field_size) { \ - .name = (stringify(_field)), \ - .version_id = (_version), \ - .field_exists = (_test), \ - .size_offset = vmstate_offset_value(_state, _field_size, uint64_t),\ - .info = &vmstate_info_buffer, \ - .flags = VMS_VBUFFER | VMS_POINTER, \ - .offset = offsetof(_state, _field), \ -}Why don't you just leave this macro around, put a "do not use" comment above it and remove the type check for this one instance? You're already checking against
INT32_MAXduringpre_load.What's the benefit of keeping this global macro if we only allow one user to use it?
Commenting to say "don't use" is not guaranteeing anything, at least we should also rename the macro to some weird name to make people notice. Keeping the macro name like this OTOH suggests abuse. But if we think it should only be used in 1 place, fixing the one place might be better?
Quoted 3 lines
We could even lift that check up into
vmstate_pre_loadand reject there globally, don't even touch the virtio code.Personally, I don't like the idea leaking VBUFFER impl details into
vmstate_pre_load()which is so far only a wrapper, especially only for this one instance.. which we do not suggest future users to use.
Quoted 6 lines
If we go this route, I'd rather merge Michael's version to support u64, even if we don't need a u64 size. But I really don't want to introduce yet another VMS flag just for this... we'll have no real use if we have noticed this problem when the vhost inflight patch was reviewed. It will be a
uint32_torint32_talready. I just can't come up with some users need size >2G.
It's not that size needs to be >2G. But it might be nice for some code to keep it as u64 for its own reasons to avoid issues like overruns during math ops.
Quoted 26 lines
The recent AI reports just make such feeling stronger: we just used the reason "max 2G, should be fine" when AI reports that unlimited size allocation problem, now we will need to wait for another AI report which says "it's not 2G anymore, 1<<64-1 that is", if we don't come up with a proper limit for VMSD field allocations.
Quoted 7 lines
/rant
... what's even the point of having per-type variants of VMSTATE macros that exist just to type-check the extra offsets (.num_offset, .size_offset as opposed to .offset)?Quoted 6 lines
For instance, look at
vmstate_n_elemscasting opaque data to 32 and sub-32bit size and assigning to an int! What do we gain from that? It's circular reasoning that does nothing aside from bothering the person writing the vmstate.Right? Maybe I'm missing something, it's the end of the day already.
I can also only guess, which is.. I believe Michael was right, the initial vmstate code was simply broken where it should have considered the type of size fields of all kinds, but forgot, and it just worked because nobody needed u64 for a size field.
Quoted 5 lines
Said that, I still want to see if we can avoid introducing that at all. To me, I still prefer this patch (after fixing
pre_save()..). But let me know if I didn't convince any of you.. we can discuss.Thanks,
The issue is that 2G is not a sane limit either.
So what we really want is a macro that supplies
a size limit. That would be an improvement.
Quoted 51 lines
Quoted 48 lines
Quoted 47 lines
include/migration/vmstate.h +0
-1-#define VMSTATE_VBUFFER_ALLOC_UINT32 (_field, _state, _version, \ _test, _field_size) { \ .name = (stringify(_field)), \hw/virtio/vhost.c +14
-5@@ -2022,15 +2022,24 @@ void vhost_get_features_ex(struct vhost_dev *hdev,static bool vhost_inflight_buffer_pre_load (void *opaque,Error **errp) {struct vhost_inflight *inflight = opaque; -int fd =-1 ; -void *addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, -F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , - &fd, errp); +void *addr; + +if (inflight->size >INT32_MAX ) { +error_setg (errp,"inflight size '%" PRIu64"' exceeds " +"migration limit '%" PRIu32"'" , inflight->size,INT32_MAX ); +return false; + } + + addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, +F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , + &fd, errp);if (!addr) {return false; } +/* Only used in VMSTATE_VBUFFER_UINT32() */ + inflight->__size_32bits = inflight->size; inflight->offset =0 ; inflight->addr = addr; inflight->fd = fd;@@ -2042,7 +2051,7 @@ const VMStateDescription vmstate_vhost_inflight_region_buffer = { .name = "vhost-inflight-region/buffer", .pre_load_errp = vhost_inflight_buffer_pre_load, .fields = (const VMStateField []) { - VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size), + VMSTATE_VBUFFER_UINT32(addr, struct vhost_inflight, 0, NULL, __size_32bits),VMSTATE_END_OF_LIST () } };Signature · 1 line
Peter Xu
Quoted 83 lines · Peter Xu
Quoted 68 lines · Fabiano Rosas
Quoted 66 lines · Peter Xu
It was overlooked that
VMSTATE_VBUFFER_UINT64()won't really work with anuint64_t, as vmstate core only treats the size as 32bits, and maximumINT32_MAX(seevmstate_size()).Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to
uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue__size_32bitsto be used only forVMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.Remove
VMSTATE_VBUFFER_UINT64()altogether, because it was never going to be used right. It means QEMU will only support 2G max forVMS_VBUFFER.Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
Reported-by 김승중 <[email protected]>Cc Alexandr Moshkov <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Fabiano Rosas <[email protected]>Fixes 3a80ff0721 ("vhost: add vmstate for inflight region with inner buffer")Signed-off-by Peter Xu <[email protected]>---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
PS2: Michael, we have just discussed what we should define as CVE for migration, and this one shouldn't fall into CVE category, please refer to:
https://lore.kernel.org/r/[email protected]
So I didn't yet attach CVE tag. Please correct if I'm wrong, thanks.
--- include/hw/virtio/vhost.h | 5 +++++ include/migration/vmstate.h | 10 ---------- hw/virtio/vhost.c | 19 ++++++++++++++----- 3 files changed, 19 insertions(+), 15 deletions(-)include/hw/virtio/vhost.h +5
-0@@ -17,6 +17,11 @@ struct vhost_inflight {int fd;void *addr;uint64_t size; +/* +* This is a temporary variable only used during migration loading to +* satisfy VMSTATE_VBUFFER_UINT32() typing. Please use @size otherwise. +*/ +uint32_t __size_32bits;uint64_t offset;uint16_t queue_size; };include/migration/vmstate.h +0
-0@@ -782,16 +782,6 @@ extern const VMStateInfo vmstate_info_g_byte_array; .offset = offsetof(_state, _field), \ }Quoted 13 lines
include/migration/vmstate.h extern const VMStateInfo vmstate_info_g_byte_array; · line 793
Quoted 9 lines
include/migration/vmstate.h +0
-9-#define VMSTATE_VBUFFER_UINT64 (_field, _state, _version, _test, _field_size) { \ - .name = (stringify(_field)), \ - .version_id = (_version), \ - .field_exists = (_test), \ - .size_offset = vmstate_offset_value(_state, _field_size, uint64_t),\ - .info = &vmstate_info_buffer, \ - .flags = VMS_VBUFFER | VMS_POINTER, \ - .offset = offsetof(_state, _field), \ -}Why don't you just leave this macro around, put a "do not use" comment above it and remove the type check for this one instance? You're already checking against
INT32_MAXduringpre_load.
Quoted 8 lines
What's the benefit of keeping this global macro if we only allow one user to use it?
Commenting to say "don't use" is not guaranteeing anything, at least we should also rename the macro to some weird name to make people notice. Keeping the macro name like this OTOH suggests abuse. But if we think it should only be used in 1 place, fixing the one place might be better?
I was thinking of fixing this on the vmstate size and not changing the virtio code. But yes, it's better to fix properly.
Quoted 8 lines
Quoted 3 lines
We could even lift that check up into
vmstate_pre_loadand reject there globally, don't even touch the virtio code.Personally, I don't like the idea leaking VBUFFER impl details into
vmstate_pre_load()which is so far only a wrapper, especially only for this one instance.. which we do not suggest future users to use.
Hm, but isn't it better to have a global cap anyway? So we don't need every device code to change with similar checks.
Quoted 7 lines
If we go this route, I'd rather merge Michael's version to support u64, even if we don't need a u64 size. But I really don't want to introduce yet another VMS flag just for this... we'll have no real use if we have noticed this problem when the vhost inflight patch was reviewed. It will be a
uint32_torint32_talready. I just can't come up with some users need size >2G.
I agree with making all u64. I think we can actually remove all the extra VMS_VARRAY_* and VBUFFER_* flags and instead doa single type-check of "int <= 64bit". Give me a couple of hours and I will post an RFC.
Quoted 79 lines
The recent AI reports just make such feeling stronger: we just used the reason "max 2G, should be fine" when AI reports that unlimited size allocation problem, now we will need to wait for another AI report which says "it's not 2G anymore, 1<<64-1 that is", if we don't come up with a proper limit for VMSD field allocations.
Quoted 7 lines
/rant
... what's even the point of having per-type variants of VMSTATE macros that exist just to type-check the extra offsets (.num_offset, .size_offset as opposed to .offset)?Quoted 6 lines
For instance, look at
vmstate_n_elemscasting opaque data to 32 and sub-32bit size and assigning to an int! What do we gain from that? It's circular reasoning that does nothing aside from bothering the person writing the vmstate.Right? Maybe I'm missing something, it's the end of the day already.
I can also only guess, which is.. I believe Michael was right, the initial vmstate code was simply broken where it should have considered the type of size fields of all kinds, but forgot, and it just worked because nobody needed u64 for a size field.
Said that, I still want to see if we can avoid introducing that at all. To me, I still prefer this patch (after fixing
pre_save()..). But let me know if I didn't convince any of you.. we can discuss.Thanks,
Quoted 48 lines
Quoted 47 lines
include/migration/vmstate.h +0
-1-#define VMSTATE_VBUFFER_ALLOC_UINT32 (_field, _state, _version, \ _test, _field_size) { \ .name = (stringify(_field)), \hw/virtio/vhost.c +14
-5@@ -2022,15 +2022,24 @@ void vhost_get_features_ex(struct vhost_dev *hdev,static bool vhost_inflight_buffer_pre_load (void *opaque,Error **errp) {struct vhost_inflight *inflight = opaque; -int fd =-1 ; -void *addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, -F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , - &fd, errp); +void *addr; + +if (inflight->size >INT32_MAX ) { +error_setg (errp,"inflight size '%" PRIu64"' exceeds " +"migration limit '%" PRIu32"'" , inflight->size,INT32_MAX ); +return false; + } + + addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, +F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , + &fd, errp);if (!addr) {return false; } +/* Only used in VMSTATE_VBUFFER_UINT32() */ + inflight->__size_32bits = inflight->size; inflight->offset =0 ; inflight->addr = addr; inflight->fd = fd;@@ -2042,7 +2051,7 @@ const VMStateDescription vmstate_vhost_inflight_region_buffer = { .name = "vhost-inflight-region/buffer", .pre_load_errp = vhost_inflight_buffer_pre_load, .fields = (const VMStateField []) { - VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size), + VMSTATE_VBUFFER_UINT32(addr, struct vhost_inflight, 0, NULL, __size_32bits),VMSTATE_END_OF_LIST () } };
Quoted 11 lines · Fabiano Rosas
Hm, but isn't it better to have a global cap anyway? So we don't need every device code to change with similar checks.
Quoted 7 lines
If we go this route, I'd rather merge Michael's version to support u64, even if we don't need a u64 size. But I really don't want to introduce yet another VMS flag just for this... we'll have no real use if we have noticed this problem when the vhost inflight patch was reviewed. It will be a
uint32_torint32_talready. I just can't come up with some users need size >2G.
Quoted 3 lines
I agree with making all u64. I think we can actually remove all the extra VMS_VARRAY_* and VBUFFER_* flags and instead doa single type-check of "int <= 64bit". Give me a couple of hours and I will post an RFC.
I confess I don't yet get what is the global cap you mentioned.. but sure, I'll wait and read the RFC first.
Signature · 1 line
Peter Xu
Quoted 12 lines · Peter Xu
Quoted 11 lines · Fabiano Rosas
Hm, but isn't it better to have a global cap anyway? So we don't need every device code to change with similar checks.
Quoted 7 lines
If we go this route, I'd rather merge Michael's version to support u64, even if we don't need a u64 size. But I really don't want to introduce yet another VMS flag just for this... we'll have no real use if we have noticed this problem when the vhost inflight patch was reviewed. It will be a
uint32_torint32_talready. I just can't come up with some users need size >2G.
Quoted 6 lines
Quoted 3 lines
I agree with making all u64. I think we can actually remove all the extra VMS_VARRAY_* and VBUFFER_* flags and instead doa single type-check of "int <= 64bit". Give me a couple of hours and I will post an RFC.
I confess I don't yet get what is the global cap you mentioned.. but sure, I'll wait and read the RFC first.
I was thinking the vmstate code could define a limit to the size (of anything) and always enforce it. The device code can then use some custom macros (not yet existent) to limit even further.
Quoted 18 lines · Fabiano Rosas
Quoted 16 lines · Peter Xu
Quoted 14 lines · Fabiano Rosas
Hm, but isn't it better to have a global cap anyway? So we don't need every device code to change with similar checks.
Quoted 7 lines
If we go this route, I'd rather merge Michael's version to support u64, even if we don't need a u64 size. But I really don't want to introduce yet another VMS flag just for this... we'll have no real use if we have noticed this problem when the vhost inflight patch was reviewed. It will be a
uint32_torint32_talready. I just can't come up with some users need size >2G.I agree with making all u64. I think we can actually remove all the extra VMS_VARRAY_* and VBUFFER_* flags and instead doa single type-check of "int <= 64bit". Give me a couple of hours and I will post an RFC.
Quoted 6 lines
Quoted 2 lines
I confess I don't yet get what is the global cap you mentioned.. but sure, I'll wait and read the RFC first.
I was thinking the vmstate code could define a limit to the size (of anything) and always enforce it. The device code can then use some custom macros (not yet existent) to limit even further.
Ah I see.
With 32bit limit (let's assume we don't have that further "it's int32_t" problem..), it's already throttled to 32bit max. I added it into vhost code only because in reality it was a 64bit internally. None of the rest users should need it since they should be sticking with 32bit fields.
Signature · 1 line
Peter Xu
Quoted 3 lines · Fabiano Rosas
I was thinking the vmstate code could define a limit to the size (of anything) and always enforce it. The device code can then use some custom macros (not yet existent) to limit even further.
PS: one more thing to mention on "further limit":
To me, that was not a real thing that will help anything for real users. My guess is, it will only be something we can use to "convince" people stop opening security tickets.. and say "hey, see, we have an upper bound".
IMHO no limit is really fine for QEMU's complex use case, it's because QEMU is a complex userapp, if it's a daemon and more accessible we may need to be more careful, but it's not: it's a hypervisor, here migration protocol (as one of the network consumer) is very dedicated, perf / maintenance / .. matter more than that upper limit. There're other things that might be more risky, like if we have NBD server listening and it's more of an attack surface. Even if so, I still think a VM userapp is very special.
If I design migration from scratch, I would allow some migration command to dynamically create ramblocks, for example, and obviously that can really be any size: limiting it to 4T will stop working when one wants to create a 12T VM. So to me, it's much simpler we say migration stream must be trusted, and I expect dest QEMU can allocate any buffer it needs, until it eats the whole system memory. I really don't see much real risk..
That's partly why personally I want us to spend less time on all these limits.. because then we only handle trusted data stream, we can keep in mind whenever we want to be safer, but it's not 100% required and it's different from adhoc network servers. When the protocol allows "arbitrary size" I think we should just let it happen, and IMHO it avoids unnecessary work.
Signature · 1 line
Peter Xu
Quoted 3 lines · Peter Xu
So to me, it's much simpler we say migration stream must be trusted, and I expect dest QEMU can allocate any buffer it needs, until it eats the whole system memory. I really don't see much real risk..
It's not risk due to migration, specifically. But making qemu drink up terabytes from the guest would be problematic, right?
Putting qemu in a cgroup with restricted total memory would be one way to prevent this class of security issue, and a robust one.
But that, in turn, is impossible if qemu insists on allocating unlimited memory at the drop of a hat.
However, migration is hardly the worst offender here, and I am not at all sure we need to start with it if we even want to address this limitation.
Signature · 1 line
MST
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Xu
So to me, it's much simpler we say migration stream must be trusted, and I expect dest QEMU can allocate any buffer it needs, until it eats the whole system memory. I really don't see much real risk..
It's not risk due to migration, specifically. But making qemu drink up terabytes from the guest would be problematic, right?
Quoted 6 lines
Putting qemu in a cgroup with restricted total memory would be one way to prevent this class of security issue, and a robust one.
But that, in turn, is impossible if qemu insists on allocating unlimited memory at the drop of a hat.
Just to clarify at least one thing.. we have two attack surfaces here and they're very different IMHO:
(1) guest behavior caused memory allocation, or,
(2) migration stream caused memory allocation.
AFAIU, (1) is more severe. All my points only apply to (2).
Quoted 4 lines
However, migration is hardly the worst offender here, and I am not at all sure we need to start with it if we even want to address this limitation.
Agree.
Thanks,
Signature · 1 line
Peter Xu
Quoted 16 lines · Peter Xu
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Xu
So to me, it's much simpler we say migration stream must be trusted, and I expect dest QEMU can allocate any buffer it needs, until it eats the whole system memory. I really don't see much real risk..
It's not risk due to migration, specifically. But making qemu drink up terabytes from the guest would be problematic, right?
Quoted 6 lines
Putting qemu in a cgroup with restricted total memory would be one way to prevent this class of security issue, and a robust one.
But that, in turn, is impossible if qemu insists on allocating unlimited memory at the drop of a hat.
Quoted 8 lines
Just to clarify at least one thing.. we have two attack surfaces here and they're very different IMHO:
(1) guest behavior caused memory allocation, or,
(2) migration stream caused memory allocation.
AFAIU, (1) is more severe. All my points only apply to (2).
Absolutely. Yet without fixing 2 we can't mitigate 1 with OS level protections.
Quoted 12 lines
Quoted 5 lines
However, migration is hardly the worst offender here, and I am not at all sure we need to start with it if we even want to address this limitation.
Agree.
Thanks,
Signature · 1 line
Peter Xu
Quoted 22 lines · Michael S. Tsirkin
Quoted 21 lines · Peter Xu
Quoted 8 lines · Michael S. Tsirkin
Quoted 3 lines · Peter Xu
So to me, it's much simpler we say migration stream must be trusted, and I expect dest QEMU can allocate any buffer it needs, until it eats the whole system memory. I really don't see much real risk..
It's not risk due to migration, specifically. But making qemu drink up terabytes from the guest would be problematic, right?
Quoted 6 lines
Putting qemu in a cgroup with restricted total memory would be one way to prevent this class of security issue, and a robust one.
But that, in turn, is impossible if qemu insists on allocating unlimited memory at the drop of a hat.
Just to clarify at least one thing.. we have two attack surfaces here and they're very different IMHO:
(1) guest behavior caused memory allocation, or,
Quoted 7 lines
Quoted 3 lines
(2) migration stream caused memory allocation.
AFAIU, (1) is more severe. All my points only apply to (2).
Absolutely. Yet without fixing 2 we can't mitigate 1 with OS level protections.
Nowadays most of issues around migration stream can cause allocations are about what we have already persisted internally to QEMU to maintain guest states.
Takeing a GTree as example.
In guest context, one concrete example is GTree can contain unlimited number of elements for a vIOMMU device to keep the mappings, before migration we should better make sure the mapping isn't too much to eat all host memory and get QEMU OOM killed.
In case of migration, it's about when migrating a GTree we will migrate exactly whatever it is there already on src to dest, then a malicious stream may cause unlimited allocations. That's one of the security reports, we have similar ones for qlist, etc.
IOW, I think yes if we stick with "migration stream trusted" all issues should be non-issue, and we should not worry about (2) too much, because we really should majorly need to worry (1).. which is real, since guest is never trusted..
Meanwhile, migration should still make sure it won't allocate anything else than what has already been there for source QEMU. If we need such temp allocation for migration only, that's the real part where a migration security issue may reside, but so far none of the reports is about that... I also can't think of a lot that migration does allocation on its own for things that can occupy a lot of memory, some might be relevant I can still think of is bitmaps all over, that's unfortunate, we need them for various reasons, either on src/dst.. say, kvm also has bitmaps of such, only allocated during migrations, not easily avoidable.
Another example is QEMU_VM_VMDESCRIPTION that is definitely migration specific (not part of src QEMU), but dest is already careful there, in qemu_loadvm_state():
if (ret == 0 && should_send_vmdesc()) {
...
if (section_type != QEMU_VM_VMDESCRIPTION) {
...
} else {
buf = g_malloc(0x1000);
size = qemu_get_be32(f);
while (size > 0) {
uint32_t read_chunk = MIN(size, 0x1000);
qemu_get_buffer(f, buf, read_chunk);
size -= read_chunk;
}
g_free(buf);
}I also remember VFIO has some internal buffering only used in migration, I also remember when I reviewed it I tried to point out the buffer limitation issue I hope it was properly settled..
I hope we're always careful on those otherwise, but these cases should be rare.
Thanks,
Signature · 1 line
Peter Xu
Quoted 30 lines · Peter Xu
It was overlooked that
VMSTATE_VBUFFER_UINT64()won't really work with anuint64_t, as vmstate core only treats the size as 32bits, and maximumINT32_MAX(seevmstate_size()).Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to
uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue__size_32bitsto be used only forVMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.Remove
VMSTATE_VBUFFER_UINT64()altogether, because it was never going to be used right. It means QEMU will only support 2G max forVMS_VBUFFER.Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
Reported-by 김승중 <[email protected]>Cc Alexandr Moshkov <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Fabiano Rosas <[email protected]>Fixes 3a80ff0721 ("vhost: add vmstate for inflight region with inner buffer")Signed-off-by Peter Xu <[email protected]>---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
Quoted 6 lines
PS2: Michael, we have just discussed what we should define as CVE for migration, and this one shouldn't fall into CVE category, please refer to:
https://lore.kernel.org/r/[email protected]
So I didn't yet attach CVE tag. Please correct if I'm wrong, thanks.
I agree.
Quoted 87 lines
--- include/hw/virtio/vhost.h | 5 +++++ include/migration/vmstate.h | 10 ---------- hw/virtio/vhost.c | 19 ++++++++++++++----- 3 files changed, 19 insertions(+), 15 deletions(-)include/hw/virtio/vhost.h +5
-0@@ -17,6 +17,11 @@ struct vhost_inflight {int fd;void *addr;uint64_t size; +/* +* This is a temporary variable only used during migration loading to +* satisfy VMSTATE_VBUFFER_UINT32() typing. Please use @size otherwise. +*/ +uint32_t __size_32bits;uint64_t offset;uint16_t queue_size; };include/migration/vmstate.h +0
-10@@ -782,16 +782,6 @@ extern const VMStateInfo vmstate_info_g_byte_array; .offset = offsetof(_state, _field), \ } -#define VMSTATE_VBUFFER_UINT64 (_field, _state, _version, _test, _field_size) { \ - .name = (stringify(_field)), \ - .version_id = (_version), \ - .field_exists = (_test), \ - .size_offset = vmstate_offset_value(_state, _field_size, uint64_t),\ - .info = &vmstate_info_buffer, \ - .flags = VMS_VBUFFER | VMS_POINTER, \ - .offset = offsetof(_state, _field), \ -} -#define VMSTATE_VBUFFER_ALLOC_UINT32 (_field, _state, _version, \ _test, _field_size) { \ .name = (stringify(_field)), \hw/virtio/vhost.c +14
-5@@ -2022,15 +2022,24 @@ void vhost_get_features_ex(struct vhost_dev *hdev,static bool vhost_inflight_buffer_pre_load (void *opaque,Error **errp) {struct vhost_inflight *inflight = opaque; -int fd =-1 ; -void *addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, -F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , - &fd, errp); +void *addr; + +if (inflight->size >INT32_MAX ) { +error_setg (errp,"inflight size '%" PRIu64"' exceeds " +"migration limit '%" PRIu32"'" , inflight->size,INT32_MAX ); +return false; + } + + addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, +F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , + &fd, errp);if (!addr) {return false; } +/* Only used in VMSTATE_VBUFFER_UINT32() */ + inflight->__size_32bits = inflight->size; inflight->offset =0 ; inflight->addr = addr; inflight->fd = fd;@@ -2042,7 +2051,7 @@ const VMStateDescription vmstate_vhost_inflight_region_buffer = { .name = "vhost-inflight-region/buffer", .pre_load_errp = vhost_inflight_buffer_pre_load, .fields = (const VMStateField []) { - VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size), + VMSTATE_VBUFFER_UINT32(addr, struct vhost_inflight, 0, NULL, __size_32bits),VMSTATE_END_OF_LIST () } };Signature · 1 line
2.54.0
Quoted 23 lines · Peter Xu
It was overlooked that
VMSTATE_VBUFFER_UINT64()won't really work with anuint64_t, as vmstate core only treats the size as 32bits, and maximumINT32_MAX(seevmstate_size()).Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to
uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue__size_32bitsto be used only forVMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.Remove
VMSTATE_VBUFFER_UINT64()altogether, because it was never going to be used right. It means QEMU will only support 2G max forVMS_VBUFFER.Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
Reported-by 김승중 <[email protected]>Cc Alexandr Moshkov <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Fabiano Rosas <[email protected]>Fixes 3a80ff0721 ("vhost: add vmstate for inflight region with inner buffer")Signed-off-by Peter Xu <[email protected]>
Quoted 6 lines
---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
It looks like inflight migration is broken:
qemu-system-x86_64: Missing section footer for 0000:00:02.0:00.0:00.0/vhost-user-blk migrate_error error=load of migration failed: Invalid argument: Section footer error, section_id: 53
qemu-system-x86_64: load of migration failed: Invalid argument: Section footer error, section_id: 53
As far as I understand this happen because __size_32bits not initialized on source - it's never set. So vmstate_size() reads 0, and zero-lenght buffer is written into migration stream. The destination then allocates the correct buffer but reads 0 bytes from the stream, leaving unconsumed data and causing the section footer check to fail.
It can be fixed with adding pre_save (or pre_save_errp) to vmstate_vhost_inflight_region_buffer that initialize __size_32bits from size:
static int vhost_inflight_buffer_pre_save(void *opaque) {
struct vhost_inflight *inflight = opaque; /* Only used in VMSTATE_VBUFFER_UINT32() */ inflight->__size_32bits = inflight->size; return 0;
}
Quoted 92 lines
PS2: Michael, we have just discussed what we should define as CVE for migration, and this one shouldn't fall into CVE category, please refer to:
https://lore.kernel.org/r/[email protected]
So I didn't yet attach CVE tag. Please correct if I'm wrong, thanks.
--- include/hw/virtio/vhost.h | 5 +++++ include/migration/vmstate.h | 10 ---------- hw/virtio/vhost.c | 19 ++++++++++++++----- 3 files changed, 19 insertions(+), 15 deletions(-)include/hw/virtio/vhost.h +5
-0@@ -17,6 +17,11 @@ struct vhost_inflight {int fd;void *addr;uint64_t size; +/* +* This is a temporary variable only used during migration loading to +* satisfy VMSTATE_VBUFFER_UINT32() typing. Please use @size otherwise. +*/ +uint32_t __size_32bits;uint64_t offset;uint16_t queue_size; };include/migration/vmstate.h +0
-10@@ -782,16 +782,6 @@ extern const VMStateInfo vmstate_info_g_byte_array; .offset = offsetof(_state, _field), \ } -#define VMSTATE_VBUFFER_UINT64 (_field, _state, _version, _test, _field_size) { \ - .name = (stringify(_field)), \ - .version_id = (_version), \ - .field_exists = (_test), \ - .size_offset = vmstate_offset_value(_state, _field_size, uint64_t),\ - .info = &vmstate_info_buffer, \ - .flags = VMS_VBUFFER | VMS_POINTER, \ - .offset = offsetof(_state, _field), \ -} -#define VMSTATE_VBUFFER_ALLOC_UINT32 (_field, _state, _version, \ _test, _field_size) { \ .name = (stringify(_field)), \hw/virtio/vhost.c +14
-5@@ -2022,15 +2022,24 @@ void vhost_get_features_ex(struct vhost_dev *hdev,static bool vhost_inflight_buffer_pre_load (void *opaque,Error **errp) {struct vhost_inflight *inflight = opaque; -int fd =-1 ; -void *addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, -F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , - &fd, errp); +void *addr; + +if (inflight->size >INT32_MAX ) { +error_setg (errp,"inflight size '%" PRIu64"' exceeds " +"migration limit '%" PRIu32"'" , inflight->size,INT32_MAX ); +return false; + } + + addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, +F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , + &fd, errp);if (!addr) {return false; } +/* Only used in VMSTATE_VBUFFER_UINT32() */ + inflight->__size_32bits = inflight->size; inflight->offset =0 ; inflight->addr = addr; inflight->fd = fd;@@ -2042,7 +2051,7 @@ const VMStateDescription vmstate_vhost_inflight_region_buffer = { .name = "vhost-inflight-region/buffer", .pre_load_errp = vhost_inflight_buffer_pre_load, .fields = (const VMStateField []) { - VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size), + VMSTATE_VBUFFER_UINT32(addr, struct vhost_inflight, 0, NULL, __size_32bits),VMSTATE_END_OF_LIST () } };
Quoted 52 lines · Alexandr Moshkov
Quoted 23 lines · Peter Xu
It was overlooked that
VMSTATE_VBUFFER_UINT64()won't really work with anuint64_t, as vmstate core only treats the size as 32bits, and maximumINT32_MAX(seevmstate_size()).Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to
uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue__size_32bitsto be used only forVMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.Remove
VMSTATE_VBUFFER_UINT64()altogether, because it was never going to be used right. It means QEMU will only support 2G max forVMS_VBUFFER.Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
Reported-by 김승중 <[email protected]>Cc Alexandr Moshkov <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Fabiano Rosas <[email protected]>Fixes 3a80ff0721 ("vhost: add vmstate for inflight region with inner buffer")Signed-off-by Peter Xu <[email protected]>Quoted 6 lines
---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
It looks like inflight migration is broken:
qemu-system-x86_64: Missing section footer for 0000:00:02.0:00.0:00.0/vhost-user-blk
migrate_errorerror=load of migration failed: Invalid argument: Section footer error,section_id: 53
qemu-system-x86_64: load of migration failed: Invalid argument: Section footer error,section_id: 53As far as I understand this happen because
__size_32bitsnot initialized on source - it's never set. Sovmstate_size()reads 0, and zero-lenght buffer is written into migration stream.
The destination then allocates the correct buffer but reads 0 bytes from the stream, leaving unconsumed data and causing the section footer check to fail.It can be fixed with adding
pre_save(orpre_save_errp) tovmstate_vhost_inflight_region_bufferthat initialize__size_32bitsfrom size:
Quoted 7 lines
static int
vhost_inflight_buffer_pre_save(void *opaque){
structvhost_inflight*inflight = opaque; /* Only used inVMSTATE_VBUFFER_UINT32()*/inflight->__size_32bits=inflight->size; return 0;
}
At which point I ask whether open coding all this mess is so much better.
Way I look at it, vmstate machinery had an API of storing size in u64 that it failed to implement correctly. Why not fix it?
Quoted 90 lines
Quoted 90 lines
PS2: Michael, we have just discussed what we should define as CVE for migration, and this one shouldn't fall into CVE category, please refer to:
https://lore.kernel.org/r/[email protected]
So I didn't yet attach CVE tag. Please correct if I'm wrong, thanks.
--- include/hw/virtio/vhost.h | 5 +++++ include/migration/vmstate.h | 10 ---------- hw/virtio/vhost.c | 19 ++++++++++++++----- 3 files changed, 19 insertions(+), 15 deletions(-)include/hw/virtio/vhost.h +5
-0@@ -17,6 +17,11 @@ struct vhost_inflight {int fd;void *addr;uint64_t size; +/* +* This is a temporary variable only used during migration loading to +* satisfy VMSTATE_VBUFFER_UINT32() typing. Please use @size otherwise. +*/ +uint32_t __size_32bits;uint64_t offset;uint16_t queue_size; };include/migration/vmstate.h +0
-10@@ -782,16 +782,6 @@ extern const VMStateInfo vmstate_info_g_byte_array; .offset = offsetof(_state, _field), \ } -#define VMSTATE_VBUFFER_UINT64 (_field, _state, _version, _test, _field_size) { \ - .name = (stringify(_field)), \ - .version_id = (_version), \ - .field_exists = (_test), \ - .size_offset = vmstate_offset_value(_state, _field_size, uint64_t),\ - .info = &vmstate_info_buffer, \ - .flags = VMS_VBUFFER | VMS_POINTER, \ - .offset = offsetof(_state, _field), \ -} -#define VMSTATE_VBUFFER_ALLOC_UINT32 (_field, _state, _version, \ _test, _field_size) { \ .name = (stringify(_field)), \hw/virtio/vhost.c +14
-5@@ -2022,15 +2022,24 @@ void vhost_get_features_ex(struct vhost_dev *hdev,static bool vhost_inflight_buffer_pre_load (void *opaque,Error **errp) {struct vhost_inflight *inflight = opaque; -int fd =-1 ; -void *addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, -F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , - &fd, errp); +void *addr; + +if (inflight->size >INT32_MAX ) { +error_setg (errp,"inflight size '%" PRIu64"' exceeds " +"migration limit '%" PRIu32"'" , inflight->size,INT32_MAX ); +return false; + } + + addr =qemu_memfd_alloc ("vhost-inflight" , inflight->size, +F_SEAL_GROW |F_SEAL_SHRINK |F_SEAL_SEAL , + &fd, errp);if (!addr) {return false; } +/* Only used in VMSTATE_VBUFFER_UINT32() */ + inflight->__size_32bits = inflight->size; inflight->offset =0 ; inflight->addr = addr; inflight->fd = fd;@@ -2042,7 +2051,7 @@ const VMStateDescription vmstate_vhost_inflight_region_buffer = { .name = "vhost-inflight-region/buffer", .pre_load_errp = vhost_inflight_buffer_pre_load, .fields = (const VMStateField []) { - VMSTATE_VBUFFER_UINT64(addr, struct vhost_inflight, 0, NULL, size), + VMSTATE_VBUFFER_UINT32(addr, struct vhost_inflight, 0, NULL, __size_32bits),VMSTATE_END_OF_LIST () } };
Quoted 53 lines · Michael S. Tsirkin
Quoted 52 lines · Alexandr Moshkov
Quoted 23 lines · Peter Xu
It was overlooked that
VMSTATE_VBUFFER_UINT64()won't really work with anuint64_t, as vmstate core only treats the size as 32bits, and maximumINT32_MAX(seevmstate_size()).Considering that we do not need real 64bits for the size, stick with the 2G limit, converting the size field into 32bits.
Since we can't touch the wire protocol on migration from an old QEMU, we can't directly modify the type of size to
uint32_t. Instead, we need to introduce a temporary variable for this extremely rare issue__size_32bitsto be used only forVMSTATE_VBUFFER_UINT32(). Document it and name it weird enough so people won't get confused on having two size variables.Remove
VMSTATE_VBUFFER_UINT64()altogether, because it was never going to be used right. It means QEMU will only support 2G max forVMS_VBUFFER.Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/3675
Reported-by 김승중 <[email protected]>Cc Alexandr Moshkov <[email protected]>Cc Michael S. Tsirkin <[email protected]>Cc Fabiano Rosas <[email protected]>Fixes 3a80ff0721 ("vhost: add vmstate for inflight region with inner buffer")Signed-off-by Peter Xu <[email protected]>Quoted 6 lines
---
PS1: I only did smoke test as I'm not fluent with vhost inflight feature. Please kindly try it out if possible. In general, migrations from older QEMU should work even after applied. One can also treat this as partly-RFC from that.
It looks like inflight migration is broken:
qemu-system-x86_64: Missing section footer for 0000:00:02.0:00.0:00.0/vhost-user-blk
migrate_errorerror=load of migration failed: Invalid argument: Section footer error,section_id: 53
qemu-system-x86_64: load of migration failed: Invalid argument: Section footer error,section_id: 53As far as I understand this happen because
__size_32bitsnot initialized on source - it's never set. Sovmstate_size()reads 0, and zero-lenght buffer is written into migration stream.
The destination then allocates the correct buffer but reads 0 bytes from the stream, leaving unconsumed data and causing the section footer check to fail.It can be fixed with adding
pre_save(orpre_save_errp) tovmstate_vhost_inflight_region_bufferthat initialize__size_32bitsfrom size:
Quoted 7 lines
Quoted 7 lines
static int
vhost_inflight_buffer_pre_save(void *opaque){
structvhost_inflight*inflight = opaque; /* Only used inVMSTATE_VBUFFER_UINT32()*/inflight->__size_32bits=inflight->size; return 0;
}
Thanks for the testing and report, Alexandr. Obviously I only kept in mind of the cross-binary case.. I'll see if I'll respin with the fix or something different.
Quoted 7 lines
At which point I ask whether open coding all this mess is so much better.
Way I look at it, vmstate machinery had an API of storing size in u64 that it failed to implement correctly. Why not fix it?
I explained in my reply to Fabiano:
https://lore.kernel.org/qemu-devel/[email protected]/
Thanks,
Signature · 1 line
Peter Xu