Skip to contentSkip to conversation

Deadlock with CMA and CPU hotplug

DISCUSSION Laura Abbott 2 participants 2 messages Oct 22, 2014 – Oct 28, 2014 Memory management

lore
Share & tools
lore

Laura Abbott

author
#0 lore raw

Hi,

We've run into a AB/BA deadlock situation involving a driver lock and the CPU hotplug lock on a 3.10 based kernel. The situation is this:

CPU 0 CPU 1
----- ----
Start CPU hotplug
mutex_lock(&cpu_hotplug.lock)
Run CPU hotplug notifier
data for driver comes in mutex_lock(&driver_lock) driver calls dma_alloc_coherent alloc_contig_range lru_add_drain_all get_online_cpus() mutex_lock(&cpu_hotplug.lock)

Driver hotplug notifier runs
mutex_lock(&driver_lock)

The driver itself is out of tree right now[1] and we're looking at ways to rework the driver. The best option for rework right now though might result in some performance penalties. The size that's being allocated can't easily be converted to an atomic allocation either It seems like this might be a limitation of where CMA/ dma_alloc_coherent could potentially be used and make drivers unnecessarily aware of CPU hotplug locking.

Does this seem like an actual problem that needs to be fixed or is trying to use CMA in a CPU hotplug notifier path just asking for trouble?

Thanks,
Laura

[1] For reference, the driver is a version of
https://lkml.org/lkml/2014/10/7/495 although that particular posted version allocates memory at probe instead of runtime and probably doesn't have the deadlock.

Signature · 2 lines
Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
#1 lore raw

Hello,

Quoted 27 lines · Laura Abbott

We've run into a AB/BA deadlock situation involving a driver lock and the CPU hotplug lock on a 3.10 based kernel. The situation is this:

CPU 0 CPU 1
----- ----
Start CPU hotplug
mutex_lock(&cpu_hotplug.lock)
Run CPU hotplug notifier
data for driver comes in
mutex_lock(&driver_lock)
driver calls dma_alloc_coherent
alloc_contig_range
lru_add_drain_all
get_online_cpus()
mutex_lock(&cpu_hotplug.lock)

Driver hotplug notifier runs
mutex_lock(&driver_lock)

The driver itself is out of tree right now[1] and we're looking at ways to rework the driver. The best option for rework right now though might result in some performance penalties. The size that's being allocated can't easily be converted to an atomic allocation either It seems like this might be a limitation of where CMA/ dma_alloc_coherent could potentially be used and make drivers unnecessarily aware of CPU hotplug locking.

Quoted 3 lines

Does this seem like an actual problem that needs to be fixed or is trying to use CMA in a CPU hotplug notifier path just asking for trouble?

IMHO doing any allocation without GFP_ATOMIC from a notifier is asking for problems. I always considered notifiers as callbacks that might be called directly from i.e. interrupts. I don't know much about your code, but maybe it would be possible to move the problematic code from a notifier to a separate worker or thread?

Best regards

Signature · 8 lines
Marek Szyprowski, PhD
Samsung R&D Institute Poland

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>