vfio/pci: guest error recovery proposal
Quoted 6 lines
1) We need to do the right thing for the guest, I don't think we should be presuming that different reset types are equivalent, leaving gaps where we expect the guest/host to do a reset and don't follow through on other reset requests, and we need to notify the guest immediately for the error.
c> 2) We need to do the right thing for the host, that means we should
Quoted 3 lines
not give the user the opportunity to leave a device in a state where we haven't at least performed a bus reset on link error (this may be our current state and if so we should fix it).
Ok so here is a concrete proposal for improving guest device error recovery (1). This is not trying to fix current bugs for 2, but also does not lock us into not fixing them.
I'll write up proposal for (2) but I feel we can't properly fix host without fixing (1) first and without breaking compatibility.
Background:
non-fatal errors:
- These errors are due to data link problems.
The problem is that a transaction was lost, so driver and device are out of sync. Device reset is in theory enough to recover from these, in practice some drivers might try to do link level reset instead.
fatal errors:
- These errors are due to physical problems.
The problem is that a transaction was lost, so driver and device are out of sync. Link reset might be necessary to recover from these, sometimes device reset might be enough for very simple devices. If a link above the device reports errors, device might have went away, link reset is the only thing that might being it back.
current behaviour:
- vfio will always report that it recovered function from an error.
- whether link reset will trigger depends on whether any other function on the same link has a host driver that reports an error.
- also, if there's a host driver that can't handle errors, link reset will never trigger
proposed enhancement:
1- allow userspace to request reporting non fatal/fatal errors separately 2- report errors on monitor as events as well
3- forward correct error type to guest
4- set link error flag in userspace (this is optional, used for 5 below) 5- if guest requests link reset, and error flag is set,
stop vm (I hope we can distinguish this
from resets that happen on reboot here.
if yes we might not need error flag in 4 above)
Results:
The advantage of this is that we don't need to manage any state at all. Most drivers will handle non fatal errors by FLR and will recover fine. Drivers that attempt link reset will get vmstop which is not worse than what we have now.
I don't see how this can break any reasonable configuration that is not already broken, but we might want a flag
to suppress aer reports to guest and just do vmstop unconditionally.
Alternatively, management can pause vm itself when it sees the error.
Pls remember to Cc qemu list on discussion, not just kvm.
Signature · 1 line
MST
vfio/pci: host error recovery proposal
Quoted 64 lines · Michael S. Tsirkin
Quoted 6 lines
1) We need to do the right thing for the guest, I don't think we should be presuming that different reset types are equivalent, leaving gaps where we expect the guest/host to do a reset and don't follow through on other reset requests, and we need to notify the guest immediately for the error.
c> 2) We need to do the right thing for the host, that means we should
Quoted 3 lines
not give the user the opportunity to leave a device in a state where we haven't at least performed a bus reset on link error (this may be our current state and if so we should fix it).
Ok so here is a concrete proposal for improving guest device error recovery (1). This is not trying to fix current bugs for 2, but also does not lock us into not fixing them.
I'll write up proposal for (2) but I feel we can't properly fix host without fixing (1) first and without breaking compatibility.
Background:
non-fatal errors:
- These errors are due to data link problems.
The problem is that a transaction was lost, so driver and device are out of sync. Device reset is in theory enough to recover from these, in practice some drivers might try to do link level reset instead.fatal errors:
- These errors are due to physical problems.
The problem is that a transaction was lost, so driver and device are out of sync. Link reset might be necessary to recover from these, sometimes device reset might be enough for very simple devices. If a link above the device reports errors, device might have went away, link reset is the only thing that might being it back.current behaviour:
- vfio will always report that it recovered function from an error.
- whether link reset will trigger depends on whether any other function on the same link has a host driver that reports an error.
- also, if there's a host driver that can't handle errors, link reset will never triggerproposed enhancement:
1- allow userspace to request reporting non fatal/fatal errors separately 2- report errors on monitor as events as well
3- forward correct error type to guest
4- set link error flag in userspace (this is optional, used for 5 below) 5- if guest requests link reset, and error flag is set,
stop vm (I hope we can distinguish this
from resets that happen on reboot here.
if yes we might not need error flag in 4 above)Results:
The advantage of this is that we don't need to manage any state at all. Most drivers will handle non fatal errors by FLR and will recover fine. Drivers that attempt link reset will get vmstop which is not worse than what we have now.
Quoted 8 lines
I don't see how this can break any reasonable configuration that is not already broken, but we might want a flag
to suppress aer reports to guest and just do vmstop unconditionally.
Alternatively, management can pause vm itself when it sees the error.Pls remember to Cc qemu list on discussion, not just kvm.
Proposed enhancement to improve host level error handling. This might need more work.
The issue we are trying to address is that devices might need link reset.
1. detect slot (possibly link - tbd) reset
2. report to guest that link has been lost
3. at this point we can allow guest to continue,
converting reset to flr
What's difficult is deciding whether link reset on host is required.
Three possible ideas:
1. we can whitelist devices based on ID
2. For a generic device, we can have an option to assume that non fatal errors do not need link resets. Can be an option for devices linux does not know about
3. we can add a PV interface to tell host that 2 above can be used for this device
Quoted 2 lines
Signature · 1 line
MST
Quoted 48 lines · Michael S. Tsirkin
Quoted 6 lines
1) We need to do the right thing for the guest, I don't think we should be presuming that different reset types are equivalent, leaving gaps where we expect the guest/host to do a reset and don't follow through on other reset requests, and we need to notify the guest immediately for the error.
c> 2) We need to do the right thing for the host, that means we should
Quoted 3 lines
not give the user the opportunity to leave a device in a state where we haven't at least performed a bus reset on link error (this may be our current state and if so we should fix it).
Ok so here is a concrete proposal for improving guest device error recovery (1). This is not trying to fix current bugs for 2, but also does not lock us into not fixing them.
I'll write up proposal for (2) but I feel we can't properly fix host without fixing (1) first and without breaking compatibility.
Background:
non-fatal errors:
- These errors are due to data link problems.
The problem is that a transaction was lost, so driver and device are out of sync. Device reset is in theory enough to recover from these, in practice some drivers might try to do link level reset instead.fatal errors:
- These errors are due to physical problems.
The problem is that a transaction was lost, so driver and device are out of sync. Link reset might be necessary to recover from these, sometimes device reset might be enough for very simple devices. If a link above the device reports errors, device might have went away, link reset is the only thing that might being it back.current behaviour:
- vfio will always report that it recovered function from an error.
- whether link reset will trigger depends on whether any other function on the same link has a host driver that reports an error.
- also, if there's a host driver that can't handle errors, link reset will never triggerproposed enhancement:
Quoted 9 lines
1- allow userspace to request reporting non fatal/fatal errors separately 2- report errors on monitor as events as well
3- forward correct error type to guest
4- set link error flag in userspace (this is optional, used for 5 below) 5- if guest requests link reset, and error flag is set,
stop vm (I hope we can distinguish this
from resets that happen on reboot here.
if yes we might not need error flag in 4 above)
Hi,
I have a question about vm stop on fatal error.
Recently, When test my patches, I often saw fatal error(Malformed TLP Status) happens, which disturbed my test. So I am wondering: why vm stop is a better choice than qdev_unplug? Although we told user "Please collect any data possible and then kill the guest", I still don't know how to save any possible data. For example, if user is editing document, vm_stop caused by a device fatal error will destroy user's effort.
Signature · 2 lines
Sincerely, Cao jin
Quoted 16 lines
Results:
The advantage of this is that we don't need to manage any state at all. Most drivers will handle non fatal errors by FLR and will recover fine. Drivers that attempt link reset will get vmstop which is not worse than what we have now.I don't see how this can break any reasonable configuration that is not already broken, but we might want a flag
to suppress aer reports to guest and just do vmstop unconditionally.
Alternatively, management can pause vm itself when it sees the error.Pls remember to Cc qemu list on discussion, not just kvm.
Quoted 63 lines · Cao jin
Quoted 48 lines · Michael S. Tsirkin
Quoted 6 lines
1) We need to do the right thing for the guest, I don't think we should be presuming that different reset types are equivalent, leaving gaps where we expect the guest/host to do a reset and don't follow through on other reset requests, and we need to notify the guest immediately for the error.
c> 2) We need to do the right thing for the host, that means we should
Quoted 3 lines
not give the user the opportunity to leave a device in a state where we haven't at least performed a bus reset on link error (this may be our current state and if so we should fix it).
Ok so here is a concrete proposal for improving guest device error recovery (1). This is not trying to fix current bugs for 2, but also does not lock us into not fixing them.
I'll write up proposal for (2) but I feel we can't properly fix host without fixing (1) first and without breaking compatibility.
Background:
non-fatal errors:
- These errors are due to data link problems.
The problem is that a transaction was lost, so driver and device are out of sync. Device reset is in theory enough to recover from these, in practice some drivers might try to do link level reset instead.fatal errors:
- These errors are due to physical problems.
The problem is that a transaction was lost, so driver and device are out of sync. Link reset might be necessary to recover from these, sometimes device reset might be enough for very simple devices. If a link above the device reports errors, device might have went away, link reset is the only thing that might being it back.current behaviour:
- vfio will always report that it recovered function from an error.
- whether link reset will trigger depends on whether any other function on the same link has a host driver that reports an error.
- also, if there's a host driver that can't handle errors, link reset will never triggerproposed enhancement:
Quoted 9 lines
1- allow userspace to request reporting non fatal/fatal errors separately 2- report errors on monitor as events as well
3- forward correct error type to guest
4- set link error flag in userspace (this is optional, used for 5 below) 5- if guest requests link reset, and error flag is set,
stop vm (I hope we can distinguish this
from resets that happen on reboot here.
if yes we might not need error flag in 4 above)Hi,
Quoted 11 lines
I have a question about vm stop on fatal error.
Recently, When test my patches, I often saw fatal error(Malformed TLP Status) happens, which disturbed my test. So I am wondering: why vm stop is a better choice thanqdev_unplug? Although we told user "Please collect any data possible and then kill the guest", I still don't know how to save any possible data. For example, if user is editing document,vm_stopcaused by a device fatal error will destroy user's effort.Signature · 2 lines
Sincerely, Cao jin
Why vm stop might not always be the right thing to do it happens to be what we already do.
This patchset isn't making any progress for a long time.
Focusing on incremental enhancements with minimal changes at each step is probably the only
chance there is to make meaningful progress.
Quoted 19 lines
Quoted 16 lines
Results:
The advantage of this is that we don't need to manage any state at all. Most drivers will handle non fatal errors by FLR and will recover fine. Drivers that attempt link reset will get vmstop which is not worse than what we have now.I don't see how this can break any reasonable configuration that is not already broken, but we might want a flag
to suppress aer reports to guest and just do vmstop unconditionally.
Alternatively, management can pause vm itself when it sees the error.Pls remember to Cc qemu list on discussion, not just kvm.