Skip to contentSkip to conversation

What's brewing in CXL?

LSF/MM/BPF TOPIC Dan Williams 2 participants 2 messages Apr 8

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

For the summit I offered to organize a discussion on what has been happening in CXL since Plumbers and a summary of the monthly CXL calls since. The session goal is present top challenges, address concerns / questions from the room, and perhaps introduce the CXL topics that have their own session later in the day.

Please do comment, question, ack/nak, or add to the themes below to highlight interest and help focus on the few topics we can cover in the limited time.

* CXL vs MM:

* Much of the angst of "CXL vs MM" has been harnessed by Gregory in his Private Memory Nodes proposal [1]. The meta discussion for this session would be questions like "how many former device-dax use cases can be subsumed by a mechanism like this?". In general, the game here is how to properly isolate memory that does not behave like locally attached DRAM.

* The granularity of CXL hotplug vs memory_blocks vs various distribution hotplug policies stimulated a proposal to support full "region" hotplug [2].

* CXL vs Platform Firmware (ACPI/EFI/BIOS):

* Attempts to use software interleaving to amortize the surprise of "memory that does not behave like locally attached DRAM", introduce firmware dependencies. The firmware descriptions of the performance need to be complete and match a shared understanding of the requirements. Surprise, sometimes this does not line up. [3].

* Firmware, in trying to be helpful to pre-CXL aware OSes, pre-map CXL memory (whether public generic expansion or private belonging to an accelerator) into the system address map. This causes problems for a subsystem that wants to support hot remove and re-assignment of host-bridge resources. Lack of a specified protocol and the resulting problems it causes for accelerator reset and driver reload, need more thought.

* CXL vs PCI:

* While CXL capabilities are enumerated over PCI, to software it is an additional optional protocol. The subsystem supports being built as a module. This has led to a design for error handling that sees the PCI core minimally involved to forward events over kfifo. This arrangement is raising ongoing questions for uAPI like PCI reset and vfio_pci that expect to be able to manage device with PCI core services alone [4].

* CXL vs Tooling / RAS:

* Error injection, tests, and usability have continued to improve. For folks looking to deploy CXL in production what is cxl-cli missing?

* While CXL Error isolation has a hard time justifying its utility for general purpose memory expansion, the accelerator use case at least has a reasonable chance to construct a recoverable scenario.

[1]: https://lore.kernel.org/linux-cxl/abwRu1FNqI3dVyqL@gourry-fedora-PF4VCD3F/
[2]: https://lore.kernel.org/linux-cxl/[email protected]/
[3]: http://lore.kernel.org/[email protected]
[4]: http://lore.kernel.org/[email protected]

"CXL, making MM problems worse since 2021..." -- CXL subsystem tagline

Acked-by
#1 lore raw
Quoted 10 lines · Dan Williams

For the summit I offered to organize a discussion on what has been happening in CXL since Plumbers and a summary of the monthly CXL calls since. The session goal is present top challenges, address concerns / questions from the room, and perhaps introduce the CXL topics that have their own session later in the day.

Please do comment, question, ack/nak, or add to the themes below to highlight interest and help focus on the few topics we can cover in the limited time.

... snip ...

Quoted 2 lines

"CXL, making MM problems worse since 2021..." -- CXL subsystem tagline

;P

Obviously I have interest in this space, and the private nodes discussion is semi-orthogonal as I have been applying it to non-cxl use cases - but i do think the framing of it replacing some DAX patterns is a decent one.

~Gregory