vfio multiple processes
Nabarro
Can't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all /dev/vfio/* device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.
Have any ideas?
(processes are started in normal, non-multiprocess mode)
Thanks in advance
Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development
Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Nabarro
Resolved by adding pci whitelist into spdk_env_opts. Adding whitelist into nvme.conf was not sufficient.
Regards,
Tom Nabarro - DCG/ESAD
M: +44 (0)7786 260986
Skype: tom.nabarro
From: Nabarro, Tom
Sent: Friday, January 31, 2020 12:42 PM
To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
Subject: vfio multiple processes
Can't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all /dev/vfio/* device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.
Have any ideas?
(processes are started in normal, non-multiprocess mode)
Thanks in advance
Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development
Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Stojaczyk
TBH I'm not sure the whitelist should be required. Sounds like a bug, I'll take a look.
D.
Quoted 51 lines
-----Original Message-----
From: Nabarro, Tom <tom.nabarro(a)intel.com>
Sent: Saturday, February 1, 2020 3:00 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vfio multiple processesResolved by adding pci whitelist into
spdk_env_opts. Adding whitelist into nvme.conf was not sufficient.Regards,
Tom Nabarro - DCG/ESAD
M: +44 (0)7786 260986
Skype: tom.nabarroFrom: Nabarro, Tom
Sent: Friday, January 31, 2020 12:42 PM
To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
Subject: vfio multiple processesCan't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all
/dev/vfio/*device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.Have any ideas?
(processes are started in normal, non-multiprocess mode)Thanks in advance
Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development
Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
Stojaczyk
Ah, that's a limitation I introduced together with DPDK 18.11 support. SPDK keeps vfio handles open because they may be required for a secondary process. There's room for improvement, but the current state is probably going to stay for a while.
D.
Quoted 65 lines
-----Original Message-----
From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
Sent: Monday, February 3, 2020 10:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vfio multiple processesTBH I'm not sure the whitelist should be required. Sounds like a bug, I'll take a look.
D.
Quoted 44 lines
-----Original Message-----
From: Nabarro, Tom <tom.nabarro(a)intel.com>
Sent: Saturday, February 1, 2020 3:00 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vfio multiple processesResolved by adding pci whitelist into
spdk_env_opts. Adding whitelist into nvme.conf was not sufficient.Regards,
Tom Nabarro - DCG/ESAD
M: +44 (0)7786 260986
Skype: tom.nabarroFrom: Nabarro, Tom
Sent: Friday, January 31, 2020 12:42 PM
To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
Subject: vfio multiple processesCan't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all
/dev/vfio/*device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.Have any ideas?
(processes are started in normal, non-multiprocess mode)Thanks in advance
Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development
Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47Quoted 7 lines
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
Nabarro
That's useful info, we used to have the primary process (which performed device provisioning) launch the secondary I/O processes (after device provisioning) therefore we could always do discovery through the primary process on devices in use by the secondaries.
Now we want to run our main processes as non-root so switch to vfio and we do device provisioning through a privileged helper binary (which runs setuid root and not in multiprocess mode) and call out to it from our main control process (not a SPDK process). The control process then forks our I/O processes which run in primary mode (as non-root) so that when we want to interrogate devices being used by our I/O processes we can attach as secondary rather than have to implement some form of IPC.
Regards,
Tom Nabarro - DCG/ESAD
M: +44 (0)7786 260986
Skype: tom.nabarro
-----Original Message-----
From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
Sent: Tuesday, February 4, 2020 12:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vfio multiple processes
Ah, that's a limitation I introduced together with DPDK 18.11 support. SPDK keeps vfio handles open because they may be required for a secondary process. There's room for improvement, but the current state is probably going to stay for a while.
D.
Quoted 62 lines
-----Original Message-----
From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
Sent: Monday, February 3, 2020 10:14 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vfio multiple processesTBH I'm not sure the whitelist should be required. Sounds like a bug, I'll take a look.
D.
Quoted 44 lines
-----Original Message-----
From: Nabarro, Tom <tom.nabarro(a)intel.com>
Sent: Saturday, February 1, 2020 3:00 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vfio multiple processesResolved by adding pci whitelist into
spdk_env_opts. Adding whitelist into nvme.conf was not sufficient.Regards,
Tom Nabarro - DCG/ESAD
M: +44 (0)7786 260986
Skype: tom.nabarroFrom: Nabarro, Tom
Sent: Friday, January 31, 2020 12:42 PM
To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
Subject: vfio multiple processesCan't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all
/dev/vfio/*device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.Have any ideas?
(processes are started in normal, non-multiprocess mode)Thanks in advance
Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro--------------------------------------------------------------------
-
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47Quoted 7 lines
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org
Quoted 3 lines
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.