Skip to contentSkip to conversation

Controller memory buffer - P2PDMA question

DISCUSSION Heitke, Kenneth 3 participants 3 messages Apr 23, 2019 NVMe

lore
Share & tools
lore
#0 lore raw

For the controller memory buffer, the read-data (RDS) and write-data (WDS) support isn't required to symmetrical since there are individual bits for each. The code below requires that both be supported before the P2P memory will be published

nvme_map_cmb:nvme/host/pci.c

         if ((dev->cmbsz & (NVME_CMBSZ_WDS | NVME_CMBSZ_RDS)) ==
                         (NVME_CMBSZ_WDS | NVME_CMBSZ_RDS))
                 pci_p2pmem_publish(pdev, true);

Would it make sense for P2PDMA to support the idea of directionality? Should a device be able to publish its controller memory buffer just for TO_DEVICE and/or FROM_DEVICE?

maintainer
#1 lore raw
Quoted 7 lines · Kenneth

For the controller memory buffer, the read-data (RDS) and write-data (WDS) support isn't required to symmetrical since there are individual bits for each. The code below requires that both be supported before the P2P memory will be published

nvme_map_cmb:nvme/host/pci.c

Quoted 7 lines
        if ((dev->cmbsz & (NVME_CMBSZ_WDS | NVME_CMBSZ_RDS)) ==
                        (NVME_CMBSZ_WDS | NVME_CMBSZ_RDS))
                pci_p2pmem_publish(pdev, true);

Would it make sense for P2PDMA to support the idea of directionality? Should a device be able to publish its controller memory buffer just for TO_DEVICE and/or FROM_DEVICE?

P2P needs both directions for registered device memory. It indiscriminately selects peer destinations, which are later used as sources.

#2 lore raw
Quoted 12 lines · Keith Busch
Quoted 11 lines · Kenneth

For the controller memory buffer, the read-data (RDS) and write-data (WDS) support isn't required to symmetrical since there are individual bits for each. The code below requires that both be supported before the P2P memory will be published

nvme_map_cmb:nvme/host/pci.c

        if ((dev->cmbsz & (NVME_CMBSZ_WDS | NVME_CMBSZ_RDS)) ==
                        (NVME_CMBSZ_WDS | NVME_CMBSZ_RDS))
                pci_p2pmem_publish(pdev, true);
Quoted 7 lines
Quoted 3 lines

Would it make sense for P2PDMA to support the idea of directionality? Should a device be able to publish its controller memory buffer just for TO_DEVICE and/or FROM_DEVICE?

P2P needs both directions for registered device memory. It indiscriminately selects peer destinations, which are later used as sources.

Yes, though in theory the p2pdma code could track this. It really comes down to the fact that we can add that functionality/complexity if/when there are actually users that can make use of it.

Logan