Skip to contentSkip to conversation

How to make mmap'ed kernel buffer non-cacheable

DISCUSSION Bhuvan Kumar MITTAL 3 participants 8 messages Apr 20, 2007 – May 2, 2007

lore
Share & tools
lore
#0 lore raw

Hi,
I am working on an audio device driver development on Linux. I have a kernel buffer which I have mapped to user space using mmap call from user space. My problem is that the data which comes to the kernel buffer is getting dropped in user space and I get only 50-60% of the data which is randomly ordered. The user to kernel level buffer address translation code is fine and I suspect this data dropping is occurring coz the kernel buffer is cacheable. Please suggest me some way of making the entire buffer non cacheable. I am stuck on this for quite a while now.

Regards,

Bhuvan

#1 lore raw
Quoted 2 lines · On Fri, 20 Apr 2007 18:30:25 +0530 Bhuvan Kumar MITTAL <[email protected]>…

Hi,
I am working on an audio device driver development on Linux. I have a kernel buffer which I have mapped to user space using mmap call from user space. My problem is that the data which comes to the kernel buffer is getting dropped in user space and I get only 50-60% of the data which is randomly ordered. The user to kernel level buffer address translation code is fine and I suspect this data dropping is occurring coz the kernel buffer is cacheable. Please suggest me some way of making the entire buffer non cacheable. I am stuck on this for quite a while now.

The dma mapping API (or the PCI equivalent) provide the neccessary behaviours for DMA receive, DMA send and consistent memory space in a portable fashion.

That may not be done using uncachable memory in all cases as not all processors even support uncacheable memory spaces.

If you are using the ALSA core routines (snd_dma_alloc_coherent) then ALSA already uses dma_alloc_coherent to ensure the memory is allocated for the appropriate use and will be kernel marked uncached.

#2 lore raw

Hi Alan,

I believe that dma_alloc_coherent will mark the kernel buffer as uncached at alocation time.
But that is not my intention. I have mapped some user space memory to the kernel buffer and I wish to ensure that the contents of both are coherent and correctly ordered.

In other words I wish to flush the contents of the kernel buffer to user space as soon as new data is available in my kernel buffer. How to do that? Will doing mysnc from the user space help?

Rather than flushing everytime (or msyncing) I intend to make my user-to-kernel mapping as non cacheable so that multiple flushing can be avoided.

Bhuvan

Quoted 2 lines

Hi,
I am working on an audio device driver development on Linux. I have a kernel buffer which I have mapped to user space using mmap call from user space. My problem is that the data which comes to the kernel buffer is getting dropped in user space and I get only 50-60% of the data which is randomly ordered. The user to kernel level buffer address translation code is fine and I suspect this data dropping is occurring coz the kernel buffer is cacheable. Please suggest me some way of making the entire buffer non cacheable. I am stuck on this for quite a while now.

The dma mapping API (or the PCI equivalent) provide the neccessary behaviours for DMA receive, DMA send and consistent memory space in a portable fashion.

That may not be done using uncachable memory in all cases as not all processors even support uncacheable memory spaces.

If you are using the ALSA core routines (snd_dma_alloc_coherent) then ALSA already uses dma_alloc_coherent to ensure the memory is allocated for the appropriate use and will be kernel marked uncached.

#3 lore raw
Quoted 5 lines · Bhuvan Kumar MITTAL

Hi Alan,

I believe that dma_alloc_coherent will mark the kernel buffer as uncached at alocation time. But that is not my intention. I have mapped some user space memory to the kernel buffer and I wish to ensure that the contents of both are coherent and correctly ordered.

In other words I wish to flush the contents of the kernel buffer to user space as soon as new data is available in my kernel buffer. How to do that? Will doing mysnc from the user space help?

msync is only for pagecache. If you modify user mapped RAM from the kernel, or wish to read user modified RAM from the kernel, you should issue a flush_dcache_page after and before, respectively. See Documentation/cachetlb.h.

Does that fix it? What are the details of your platform?

Signature · 1 line
SUSE Labs, Novell Inc.
#4 lore raw

Hi Alan,Nick

Thanks for your inputs. I was able to solve the problem of mapping kernel buffer to user space. Just FYI, I am working on a sh4 based architecture proprietary board with Linux kernel 2.6.17.3

Initially I was adopting the following approach:
1.The kernel thread was copying some data and waiting on a semaphore for the user task to make an ioctl into the kernel space and relieve the wait.
2. Then the user task was getting spawned and taking the data from kernel.

---> This was resulting in some initial data drop coz the kernel task was overwhelming and outspeeding the user task (Maybe someone can explain how)

How I solved the problem?

Instead of making the kernel wait for the user task, I reversed the sequence and did not let the kernel task be activated till the user task was spawned and waiting in kernel space(after making ioctl) for the kernel task. Then I started the kernel task and the problem of data dropping was resolved.

In continuation to this, I have another query:
As mentioned above, If there is a kernel task waiting on a semaphore (maybe even in hung state) and I wish to kill the kernel task from user space by making an ioctl call into the kernel, then how is it possible? Is it feasible at all?

Regards,

Bhuvan

Quoted 9 lines · Bhuvan Kumar MITTAL

Hi,
I am working on an audio device driver development on Linux. I have
a kernel buffer which I have mapped to user space using mmap call from user space. My problem is that the data which comes to the kernel
buffer is getting dropped in user space and I get only 50-60% of the
data which is randomly ordered. The user to kernel level buffer
address translation code is fine and I suspect this data dropping is occurring coz the kernel buffer is cacheable. Please suggest me some
way of making the entire buffer non cacheable. I am stuck on this for quite a while now.

You can use rdmsr() in your driver to check if the page attribute table MSR is available, then find and/or add the right entry in the table to set in each page's flags. This is documented in the Intel IA-32 Intel Architecture Software Developer's Manuals at intel.com.
-----Original Message-----
From: Nick Piggin [mailto:[email protected]]
Sent: Monday, April 23, 2007 9:56 AM
To: Bhuvan Kumar MITTAL
Cc: 'Alan Cox'; [email protected]
Subject: Re: How to make mmap'ed kernel buffer non-cacheable

Quoted 6 lines · Bhuvan Kumar MITTAL

Hi Alan,

I believe that dma_alloc_coherent will mark the kernel buffer as uncached at alocation time.
But that is not my intention. I have mapped some user space memory to the kernel buffer and I wish to ensure that the contents of both are coherent and correctly ordered.

In other words I wish to flush the contents of the kernel buffer to user space as soon as new data is available in my kernel buffer. How to do that? Will doing mysnc from the user space help?

msync is only for pagecache. If you modify user mapped RAM from the kernel, or wish to read user modified RAM from the kernel, you should issue a flush_dcache_page after and before, respectively. See Documentation/cachetlb.h.

Does that fix it? What are the details of your platform?

Signature · 1 line
SUSE Labs, Novell Inc.
#5 lore raw
Quoted 3 lines · Bhuvan Kumar MITTAL

Hi Alan,Nick

Thanks for your inputs. I was able to solve the problem of mapping kernel buffer to user space. Just FYI, I am working on a sh4 based architecture proprietary board with Linux kernel 2.6.17.3

So... you used flush_dcache_page? Or what?

Quoted 8 lines

Initially I was adopting the following approach: 1.The kernel thread was copying some data and waiting on a semaphore for the user task to make an ioctl into the kernel space and relieve the wait.
2. Then the user task was getting spawned and taking the data from kernel.

---> This was resulting in some initial data drop coz the kernel task was overwhelming and outspeeding the user task (Maybe someone can explain how)

How I solved the problem?

Instead of making the kernel wait for the user task, I reversed the sequence and did not let the kernel task be activated till the user task was spawned and waiting in kernel space(after making ioctl) for the kernel task. Then I started the kernel task and the problem of data dropping was resolved.

OK. Seems like a straightfoward producer-consumer problem.

I don't know why you should be dropping data if your kernel task blocks when its output buffer is full and your user task blocks when it is empty. But anyway, it seems you solved it. Good :)

Quoted 2 lines

In continuation to this, I have another query:
As mentioned above, If there is a kernel task waiting on a semaphore (maybe even in hung state) and I wish to kill the kernel task from user space by making an ioctl call into the kernel, then how is it possible? Is it feasible at all?

If the kernel task is hung because of some bug in your code, then it isn't really possible in general. You could always just "up" the semaphore and hope, but things might be corrupted.

If you just want to tell the kernel task to quit, I guess you'd just introduce some quit ioctl message, and have the kernel thread respond to that.

Signature · 1 line
SUSE Labs, Novell Inc.
#6 lore raw

I'll rephrase the problem as follow:

I have a userthread which makes ioctl calls to the kernel and once it reaches inside the kernel it waits on a semaphore. It then does some work inside the kernel and continuously keeps looping between the kernel and user space in an endless while loop.

I want to kill that thread from another user thread while the former is waiting on a semaphore in kernel (which it will never get) coz I want a clean exit of my driver. Please suggest me a way. I did try the use of pthread_cancel, but that is possible only if the target thread is executing in user space and making use of a pthread_testcancel call (with cancelability enabled ofcourse).

Kindly suggest something.

Quoted 2 lines

In continuation to this, I have another query:
As mentioned above, If there is a kernel task waiting on a semaphore (maybe even in hung state) and I wish to kill the kernel task from user space by making an ioctl call into the kernel, then how is it possible? Is it feasible at all?

If the kernel task is hung because of some bug in your code, then it isn't really possible in general. You could always just "up" the semaphore and hope, but things might be corrupted.

If you just want to tell the kernel task to quit, I guess you'd just introduce some quit ioctl message, and have the kernel thread respond to that.

#7 lore raw

You can do down_interruptible to make the down interruptible.

Quoted 22 lines · Bhuvan Kumar MITTAL

I'll rephrase the problem as follow:

I have a userthread which makes ioctl calls to the kernel and once it reaches inside the kernel it waits on a semaphore. It then does some work inside the kernel and continuously keeps looping between the kernel and user space in an endless while loop.

I want to kill that thread from another user thread while the former is waiting on a semaphore in kernel (which it will never get) coz I want a clean exit of my driver. Please suggest me a way. I did try the use of pthread_cancel, but that is possible only if the target thread is executing in user space and making use of a pthread_testcancel call (with cancelability enabled ofcourse).

Kindly suggest something.

Quoted 2 lines

In continuation to this, I have another query:
As mentioned above, If there is a kernel task waiting on a semaphore (maybe even in hung state) and I wish to kill the kernel task from user space by making an ioctl call into the kernel, then how is it possible? Is it feasible at all?

If the kernel task is hung because of some bug in your code, then it isn't really possible in general. You could always just "up" the semaphore and hope, but things might be corrupted.

If you just want to tell the kernel task to quit, I guess you'd just introduce some quit ioctl message, and have the kernel thread respond to that.

Signature · 1 line
SUSE Labs, Novell Inc.