Skip to contentSkip to conversation

Random process lockup on ARM board: alsa-lib-1.0.25, FUTEX_WAIT_PRIVATE

DISCUSSION Jonathan Andrews 5 participants 28 messages Feb 16, 2012 – Mar 14, 2012 6 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Sorry all me again.

I've written a program that picks up UDP broadcast packets containing audio from a number of sources, mixes it together and presents it to alsa for playback.

I'm experiencing random freezes. I cant reproduce then running the process under strace as the process on the ARM board slows down to the point it underruns with ALSA all the time and the symptom doesn't show.

If I attach strace after the process has frozen I get this and nothing else:

# strace -p 443
Process 443 attached - interrupt to quit
futex(0x4030e144, FUTEX_WAIT_PRIVATE, 2, NULL

This stop happens at random at any point typically after minutes but run time can be over an hour.

I need to work around this as I need the board for a demo.

Im using, alsa-lib-1.0.25
kernel 2.6.36.4
BusyBox v1.19.2

I also compiled strace for arm, but this and a busybox shell is the only environment I have on the board.

The problem showed when I starting using this function to poll alsa:

The same code on my PC dual core AMD does not seem to suffer the problem : alsa-lib-1.0.23, Linux jonspc 2.6.32.26-175.fc12.i686 #1 SMP

Before anyone points it out I want to poll as alsa is not the only sound output supported by the code.

int sound_system_needs_data_now()

{
        int framesleft;
        snd_pcm_status_t *status;
        snd_pcm_status_alloca( &status );

        // If the alsa playback has stalled then it definately needs data
        snd_pcm_status(playback_handle,status);
        if( snd_pcm_status_get_state( status ) != SND_PCM_STATE_RUNNING )
                return(TRUE);

        framesleft=snd_pcm_avail (playback_handle);
	//printf("FL=%d\n",framesleft); fflush(stdout);

        if (framesleft>1000)
                return(TRUE);
        else
                return(FALSE);
}

Stuck process

# cat /proc/443/status 
Name:   udp-many-way-au
State:  S (sleeping)
Tgid:   443
Pid:    443
PPid:   392
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 32
Groups: 
VmPeak:     4356 kB
VmSize:     4292 kB
VmLck:       132 kB
VmHWM:      1788 kB
VmRSS:      1788 kB
VmData:      396 kB
VmStk:       136 kB
VmExe:        28 kB
VmLib:      3348 kB
VmPTE:        12 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/471
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000002000
SigIgn: 0000000000000006
SigCgt: 0000000180002000
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: ffffffffffffffff
CapBnd: ffffffffffffffff
Cpus_allowed:   1
Cpus_allowed_list:      0
Mems_allowed:   1
Mems_allowed_list:      0
voluntary_ctxt_switches:        5908
nonvoluntary_ctxt_switches:     1766

Any useful suggestions welcome.

Thanks,
Jon

#1 lore raw

Reposting this:

# strace -p 443
Quoted 3 lines

Process 443 attached - interrupt to quit
futex(0x4030e144, FUTEX_WAIT_PRIVATE, 2, NULL

locks up. Stops, stalls ... fails to continue ....

Now tried several alsa-lib versions. A clue would be nice? What do I replace to fix this ?

Anyone ?

Thanks,
Jon

Quoted 112 lines · Jonathan Andrews

Sorry all me again.

I've written a program that picks up UDP broadcast packets containing audio from a number of sources, mixes it together and presents it to alsa for playback.

I'm experiencing random freezes. I cant reproduce then running the process under strace as the process on the ARM board slows down to the point it underruns with ALSA all the time and the symptom doesn't show.

If I attach strace after the process has frozen I get this and nothing else:

# strace -p 443
Process 443 attached - interrupt to quit
futex(0x4030e144, FUTEX_WAIT_PRIVATE, 2, NULL

This stop happens at random at any point typically after minutes but run time can be over an hour.

I need to work around this as I need the board for a demo.

Im using, alsa-lib-1.0.25
kernel 2.6.36.4
BusyBox v1.19.2

I also compiled strace for arm, but this and a busybox shell is the only environment I have on the board.

The problem showed when I starting using this function to poll alsa:

The same code on my PC dual core AMD does not seem to suffer the problem : alsa-lib-1.0.23, Linux jonspc 2.6.32.26-175.fc12.i686 #1 SMP

Before anyone points it out I want to poll as alsa is not the only sound output supported by the code.

int sound_system_needs_data_now()

{
        int framesleft;
        snd_pcm_status_t *status;
        snd_pcm_status_alloca( &status );

        // If the alsa playback has stalled then it definately needs data
        snd_pcm_status(playback_handle,status);
        if( snd_pcm_status_get_state( status ) != SND_PCM_STATE_RUNNING )
                return(TRUE);

        framesleft=snd_pcm_avail (playback_handle);
	//printf("FL=%d\n",framesleft); fflush(stdout);

        if (framesleft>1000)
                return(TRUE);
        else
                return(FALSE);
}

Stuck process

# cat /proc/443/status 
Name:   udp-many-way-au
State:  S (sleeping)
Tgid:   443
Pid:    443
PPid:   392
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 32
Groups: 
VmPeak:     4356 kB
VmSize:     4292 kB
VmLck:       132 kB
VmHWM:      1788 kB
VmRSS:      1788 kB
VmData:      396 kB
VmStk:       136 kB
VmExe:        28 kB
VmLib:      3348 kB
VmPTE:        12 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/471
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000002000
SigIgn: 0000000000000006
SigCgt: 0000000180002000
CapInh: 0000000000000000
CapPrm: ffffffffffffffff
CapEff: ffffffffffffffff
CapBnd: ffffffffffffffff
Cpus_allowed:   1
Cpus_allowed_list:      0
Mems_allowed:   1
Mems_allowed_list:      0
voluntary_ctxt_switches:        5908
nonvoluntary_ctxt_switches:     1766

Any useful suggestions welcome.

Thanks,
Jon

_______________________________________________
Alsa-devel mailing list
[email protected]
http://mailman.alsa-project.org/mailman/listinfo/alsa-devel

#2 lore raw
Quoted 12 lines · Jonathan Andrews

Reposting this:

# strace -p 443
Quoted 3 lines

Process 443 attached - interrupt to quit
futex(0x4030e144, FUTEX_WAIT_PRIVATE, 2, NULL

locks up. Stops, stalls ... fails to continue ....

Now tried several alsa-lib versions. A clue would be nice? What do I replace to fix this ?

Now tried 2 kernels, 3 versions of alsalib still stalls.

If I remove alsa code it works forever.

Thanks to everyone who helped (no one at all).

FUCK OPEN SOURCE !!!!!!!!!!!

#3 lore raw
Quoted 10 lines · Jonathan Andrews
Quoted 8 lines · Jonathan Andrews
# strace -p 443
Quoted 1 line

futex(0x4030e144, FUTEX_WAIT_PRIVATE, 2, NULL

locks up. Stops, stalls ... fails to continue ....

Now tried several alsa-lib versions. A clue would be nice? What do I replace to fix this ?

Now tried 2 kernels, 3 versions of alsalib still stalls.

The synchronization calls in alsa-lib shouldn't block unless you're actually using several threads, and even then you wouldn't be able to reproduce it so reliably.

Do you have a stack trace to find out where this is called from?

Can you confirm that pthread mutexes work on your board?

Regards,
Clemens

A message that is not in the archive (sent off-list or not yet received): 1329603022.1089.57.camel@jonspc

#5 lore raw
Quoted 2 lines · Jonathan Andrews

I've tried building alsalib in many ways, at the moment its :

$ ./configure --target=arm-linux --host=i686-linux --enable-shared --disable-static --without-pthread

Try adding --disable-ucm and --disable-python.

Regards,
Clemens

A message that is not in the archive (sent off-list or not yet received): 1329926198.22918.10.camel@jonspc

#7 lore raw
Quoted 8 lines · Jonathan Andrews
Quoted 5 lines · Clemens Ladisch
Quoted 2 lines · Jonathan Andrews

I've tried building alsalib in many ways, at the moment its :

$ ./configure --target=arm-linux --host=i686-linux --enable-shared --disable-static --without-pthread

Try adding --disable-ucm and --disable-python.

Err ok, alsa-lib still builds ok.

Now it doesn't use any pthread functions.

Quoted 2 lines

My application runs for longer, but still freezes in the end. With the same futex locking problem.

I'd guess that this is not an alsa-lib problem.
Now you really need a stack trace.

Regards,
Clemens

#8 lore raw

Using kernel 3.2.5 with alsa-lib 1.0.25, all compiled with generic Debian arm-linux-gnueabi toolchain.

arm-linux-gnueabi-gcc (Debian 4.3.2-1.1) 4.3.2 Was used to build kernel, alsa-lib and application.

Changing gcc version, kernel version or alsa-lib version makes the problem worse or better, but ALL versions seem to suffer this problem. I have also seen it once on Intel (but only once so far).

Something seeks broken at a lower layer than im using. I simply don't have the skill to debug it.

The hardware is a USB cm109 audio adapter, but the problem seems to show on more than this one driver.

The audio application writing to alsa will freezes at random intervals, infrequent at the moment, last one was after runtime 20H 37M 29S. Two processes are running, one reading from the sound device and one writing to the sound device. I am not using threading or anything very clever just generic alsa functions.

This is the only diagnostic I can generate so far as running the application under strace slows it to the point it no longer functions enough to generate the problem.

ARM / # strace -p 417
Process 417 attached - interrupt to quit futex(0x175734, FUTEX_WAIT_PRIVATE, 2, NULL^C <unfinished ...> Process 417 detached

ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux ARM / # uptime
22:36:19 up 22:36, 0 users, load average: 0.15, 0.16, 0.18
ARM / # cat /proc/cpuinfo
Processor : ARM920T rev 0 (v4l)
BogoMIPS : 199.06
Features : swp half thumb crunch
CPU implementer : 0x41
CPU architecture: 4T
CPU variant : 0x1
CPU part : 0x920
CPU revision : 0

Any help welcome.

Thanks,
Jon

#9 lore raw

Hi ,

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

BR
Huang Shijie

Quoted 57 lines · Jonathan Andrews

Using kernel 3.2.5 with alsa-lib 1.0.25, all compiled with generic Debian arm-linux-gnueabi toolchain.

arm-linux-gnueabi-gcc (Debian 4.3.2-1.1) 4.3.2 Was used to build kernel, alsa-lib and application.

Changing gcc version, kernel version or alsa-lib version makes the problem worse or better, but ALL versions seem to suffer this problem. I have also seen it once on Intel (but only once so far).

Something seeks broken at a lower layer than im using.  I simply don't have the skill to debug it.

The hardware is a USB cm109 audio adapter, but the problem seems to show on more than this one driver.

The audio application writing to alsa will freezes at random intervals, infrequent at the moment, last one was after runtime 20H 37M 29S.  Two processes are running, one reading from the sound device and one writing to the sound device. I am not using threading or anything very clever just generic alsa functions.

This is the only diagnostic I can generate so far as running the application under strace slows it to the point it no longer functions enough to generate the problem.

ARM / # strace -p 417
Process 417 attached - interrupt to quit futex(0x175734, FUTEX_WAIT_PRIVATE, 2, NULL^C <unfinished ...> Process 417 detached

ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux ARM / # uptime
 22:36:19 up 22:36,  0 users,  load average: 0.15, 0.16, 0.18 ARM / # cat /proc/cpuinfo
Processor       : ARM920T rev 0 (v4l)
BogoMIPS        : 199.06
Features        : swp half thumb crunch
CPU implementer : 0x41
CPU architecture: 4T
CPU variant     : 0x1
CPU part        : 0x920
CPU revision    : 0

Any help welcome.

Thanks,
Jon

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

#10 lore raw
Quoted 5 lines · Huang Shijie

Hi ,

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

Are either of you able to bisect the kernel? At the very least can you find two kernels where it works and where it does not?

Hanging on FUTEX_WAIT_PRIVATE can be the symptom for higher level problems including userspace locking issues and race conditions.

Huang, are you also on ARM?

--
Darren

Quoted 67 lines

BR
Huang Shijie

Quoted 52 lines · Jonathan Andrews

Using kernel 3.2.5 with alsa-lib 1.0.25, all compiled with generic Debian arm-linux-gnueabi toolchain.

arm-linux-gnueabi-gcc (Debian 4.3.2-1.1) 4.3.2 Was used to build kernel, alsa-lib and application.

Changing gcc version, kernel version or alsa-lib version makes the problem worse or better, but ALL versions seem to suffer this problem. I have also seen it once on Intel (but only once so far).

Something seeks broken at a lower layer than im using. I simply don't have the skill to debug it.

The hardware is a USB cm109 audio adapter, but the problem seems to show on more than this one driver.

The audio application writing to alsa will freezes at random intervals, infrequent at the moment, last one was after runtime 20H 37M 29S. Two processes are running, one reading from the sound device and one writing to the sound device. I am not using threading or anything very clever just generic alsa functions.

This is the only diagnostic I can generate so far as running the application under strace slows it to the point it no longer functions enough to generate the problem.

ARM / # strace -p 417
Process 417 attached - interrupt to quit futex(0x175734, FUTEX_WAIT_PRIVATE, 2, NULL^C <unfinished ...> Process 417 detached

ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux ARM / # uptime
22:36:19 up 22:36, 0 users, load average: 0.15, 0.16, 0.18
ARM / # cat /proc/cpuinfo
Processor : ARM920T rev 0 (v4l)
BogoMIPS : 199.06
Features : swp half thumb crunch
CPU implementer : 0x41
CPU architecture: 4T
CPU variant : 0x1
CPU part : 0x920
CPU revision : 0

Any help welcome.

Thanks,
Jon

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
#11 lore raw
Quoted 9 lines · Darren Hart
Quoted 5 lines · Huang Shijie

Hi ,

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

Are either of you able to bisect the kernel?

I'm not a kernel hacker what do you mean ?

Quoted 5 lines

At the very least can you
find two kernels where it works and where it does not?

Hanging on FUTEX_WAIT_PRIVATE can be the symptom for higher level problems including userspace locking issues and race conditions.

My workload is UDP network audio. I have compiled my code with and without ALSA support. The version without ALSA seems to run forever, the version with ALSA works on ARM for between a few minutes and a few hours. On Intel the same futex stall problem occurs, but it may take runtime of days.

I have two processes running. One RX process that takes UDP packets from the network mixes them and presents them to ALSA as an audio stream, the second process takes audio from the sound device and transmits it as a UDP audio stream. The two processes are independent.

My workload is atypical as I need to both transmit and receive audio via UDP on a 27/7 basis.

So far I have experienced the problem on 3 kernels, but I have tried only 3 kernels it may be all 2.6 kernels that suffer.

My development PC is "Linux jonspc 2.6.32.26-175.fc12.i686 #1 SMP Wed Dec 1 21:52:04 UTC 2010 i686 athlon i386 GNU/Linux"

My ARM board target:
ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux

And my ARM target running its older kernel was (2.6.36).

I have an strace of the process running and stalling on the PC. The file is 2GB, its not a fast link sorry.

http://www.jonshouse.co.uk/download/a_stop.txt

Many thanks,
Jon

#12 lore raw
Quoted 11 lines · Jonathan Andrews
Quoted 9 lines · Darren Hart
Quoted 5 lines · Huang Shijie

Hi ,

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

Are either of you able to bisect the kernel?

I'm not a kernel hacker what do you mean ?

Google for "git bisect". It's a way to divide-and-conquer and find exactly where the kernel stops working for you. You would need a known working kernel though. If you don't have a known working kernel, then this is either a long standing bug (not a regression) or, I suspect more likely, a locking bug or race in the userspace code.

When you see your thread stuck in FUTEX_WAIT_PRIVATE, that in and of itself is not indication of a problem. It's waiting there to be woken (FUTEX_WAKE_PRIVATE) by another thread in your application. So the questions you should be asking are:

Why is it blocked?
pthread_cond_wait()?
pthread_mutex_lock()?

Why isn't it being woken up?
Did you call pthread_cond_broadcast() without taking the mutex first?

There are other API that use futexes under the covers. You should be able to sort out where in your application your threads are blocked and thus determine which API you are using - and from there how you were expecting that to get woken up. These calls may be being made from within ALSA, in which case you may need to elicit the help of the ALSA developers.

--
Darren

Quoted 45 lines
Quoted 6 lines

At the very least can you
find two kernels where it works and where it does not?

Hanging on FUTEX_WAIT_PRIVATE can be the symptom for higher level problems including userspace locking issues and race conditions.

My workload is UDP network audio. I have compiled my code with and without ALSA support. The version without ALSA seems to run forever, the version with ALSA works on ARM for between a few minutes and a few hours. On Intel the same futex stall problem occurs, but it may take runtime of days.

I have two processes running. One RX process that takes UDP packets from the network mixes them and presents them to ALSA as an audio stream, the second process takes audio from the sound device and transmits it as a UDP audio stream. The two processes are independent.

My workload is atypical as I need to both transmit and receive audio via UDP on a 27/7 basis.

So far I have experienced the problem on 3 kernels, but I have tried only 3 kernels it may be all 2.6 kernels that suffer.

My development PC is "Linux jonspc 2.6.32.26-175.fc12.i686 #1 SMP Wed Dec 1 21:52:04 UTC 2010 i686 athlon i386 GNU/Linux"

My ARM board target:
ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux

And my ARM target running its older kernel was (2.6.36).

I have an strace of the process running and stalling on the PC. The file is 2GB, its not a fast link sorry.

http://www.jonshouse.co.uk/download/a_stop.txt

Many thanks,
Jon

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
#13 lore raw
In reply to Darren Hart

Hi,

Quoted 10 lines · Darren Hart
Quoted 5 lines · Huang Shijie

Hi ,

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

Are either of you able to bisect the kernel? At the very least can you

I finially found my arch/arm/include/asm/futex.h is not the latest, so i updated the header.

And the futex issue is gone now. But a dataAbort issue appears, I am not sure whether it caused by the futex patch.
I am debugging it now.

BR
Huang Shijie

Quoted 6 lines

find two kernels where it works and where it does not?

Hanging on FUTEX_WAIT_PRIVATE can be the symptom for higher level problems including userspace locking issues and race conditions.

Huang, are you also on ARM?

Yes, FREESCALE imx6q platform.

BR
Huang Shijie

Quoted 76 lines

--
Darren

Quoted 62 lines

BR
Huang Shijie

Quoted 52 lines · Jonathan Andrews

Using kernel 3.2.5 with alsa-lib 1.0.25, all compiled with generic Debian arm-linux-gnueabi toolchain.

arm-linux-gnueabi-gcc (Debian 4.3.2-1.1) 4.3.2 Was used to build kernel, alsa-lib and application.

Changing gcc version, kernel version or alsa-lib version makes the problem worse or better, but ALL versions seem to suffer this problem. I have also seen it once on Intel (but only once so far).

Something seeks broken at a lower layer than im using.  I simply don't have the skill to debug it.

The hardware is a USB cm109 audio adapter, but the problem seems to show on more than this one driver.

The audio application writing to alsa will freezes at random intervals, infrequent at the moment, last one was after runtime 20H 37M 29S.  Two processes are running, one reading from the sound device and one writing to the sound device. I am not using threading or anything very clever just generic alsa functions.

This is the only diagnostic I can generate so far as running the application under strace slows it to the point it no longer functions enough to generate the problem.

ARM / # strace -p 417
Process 417 attached - interrupt to quit futex(0x175734, FUTEX_WAIT_PRIVATE, 2, NULL^C <unfinished ...> Process 417 detached

ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux ARM / # uptime
 22:36:19 up 22:36,  0 users,  load average: 0.15, 0.16, 0.18 ARM / # cat /proc/cpuinfo
Processor       : ARM920T rev 0 (v4l)
BogoMIPS        : 199.06
Features        : swp half thumb crunch
CPU implementer : 0x41
CPU architecture: 4T
CPU variant     : 0x1
CPU part        : 0x920
CPU revision    : 0

Any help welcome.

Thanks,
Jon

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

#14 lore raw
Quoted 8 lines · Huang Shijie

Hi,

Quoted 5 lines · Darren Hart
Quoted 2 lines · Huang Shijie

Hi ,

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

Are either of you able to bisect the kernel? At the very least can you

I finially found my arch/arm/include/asm/futex.h is not the latest, so i updated the header.

Just make sure it matches your kernel version.

Quoted 4 lines

And the futex issue is gone now. But a dataAbort issue appears, I am not sure whether it caused by the futex patch.
I am debugging it now.

Which APIs are you using that make the futex syscall?

--
Darren

Quoted 91 lines

BR
Huang Shijie

Quoted 6 lines

find two kernels where it works and where it does not?

Hanging on FUTEX_WAIT_PRIVATE can be the symptom for higher level problems including userspace locking issues and race conditions.

Huang, are you also on ARM?

Yes, FREESCALE imx6q platform.

BR
Huang Shijie

Quoted 76 lines

--
Darren

Quoted 62 lines

BR
Huang Shijie

Quoted 52 lines · Jonathan Andrews

Using kernel 3.2.5 with alsa-lib 1.0.25, all compiled with generic Debian arm-linux-gnueabi toolchain.

arm-linux-gnueabi-gcc (Debian 4.3.2-1.1) 4.3.2 Was used to build kernel, alsa-lib and application.

Changing gcc version, kernel version or alsa-lib version makes the problem worse or better, but ALL versions seem to suffer this problem. I have also seen it once on Intel (but only once so far).

Something seeks broken at a lower layer than im using. I simply don't have the skill to debug it.

The hardware is a USB cm109 audio adapter, but the problem seems to show on more than this one driver.

The audio application writing to alsa will freezes at random intervals, infrequent at the moment, last one was after runtime 20H 37M 29S. Two processes are running, one reading from the sound device and one writing to the sound device. I am not using threading or anything very clever just generic alsa functions.

This is the only diagnostic I can generate so far as running the application under strace slows it to the point it no longer functions enough to generate the problem.

ARM / # strace -p 417
Process 417 attached - interrupt to quit futex(0x175734, FUTEX_WAIT_PRIVATE, 2, NULL^C <unfinished ...> Process 417 detached

ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux ARM / # uptime
22:36:19 up 22:36, 0 users, load average: 0.15, 0.16, 0.18
ARM / # cat /proc/cpuinfo
Processor : ARM920T rev 0 (v4l)
BogoMIPS : 199.06
Features : swp half thumb crunch
CPU implementer : 0x41
CPU architecture: 4T
CPU variant : 0x1
CPU part : 0x920
CPU revision : 0

Any help welcome.

Thanks,
Jon

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
#15 lore raw

hi,

Quoted 20 lines · Darren Hart
Quoted 8 lines · Huang Shijie

Hi,

Quoted 5 lines · Darren Hart
Quoted 2 lines · Huang Shijie

Hi ,

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

Are either of you able to bisect the kernel? At the very least can you

I finially found my arch/arm/include/asm/futex.h is not the latest, so i updated the header.

Quoted 9 lines

Just make sure it matches your kernel version.

Quoted 4 lines

And the futex issue is gone now. But a dataAbort issue appears, I am not sure whether it caused by the futex patch.
I am debugging it now.

Which APIs are you using that make the futex syscall?

futex_wait().

I cherry-pick the
" df77abc ARM: 7099/1: futex: preserve oldval in SMP __futex_atomic_op" to the arch/arm/include/asm/futex.h.

BR
Huang Shijie

Quoted 99 lines

--
Darren

Quoted 9 lines

BR
Huang Shijie

Quoted 5 lines

find two kernels where it works and where it does not?

Hanging on FUTEX_WAIT_PRIVATE can be the symptom for higher level problems including userspace locking issues and race conditions.

Quoted 82 lines
Quoted 1 line

Huang, are you also on ARM?

Yes, FREESCALE imx6q platform.

BR
Huang Shijie

Quoted 76 lines

--
Darren

Quoted 62 lines

BR
Huang Shijie

Quoted 52 lines · Jonathan Andrews

Using kernel 3.2.5 with alsa-lib 1.0.25, all compiled with generic Debian arm-linux-gnueabi toolchain.

arm-linux-gnueabi-gcc (Debian 4.3.2-1.1) 4.3.2 Was used to build kernel, alsa-lib and application.

Changing gcc version, kernel version or alsa-lib version makes the problem worse or better, but ALL versions seem to suffer this problem. I have also seen it once on Intel (but only once so far).

Something seeks broken at a lower layer than im using.  I simply don't have the skill to debug it.

The hardware is a USB cm109 audio adapter, but the problem seems to show on more than this one driver.

The audio application writing to alsa will freezes at random intervals, infrequent at the moment, last one was after runtime 20H 37M 29S.  Two processes are running, one reading from the sound device and one writing to the sound device. I am not using threading or anything very clever just generic alsa functions.

This is the only diagnostic I can generate so far as running the application under strace slows it to the point it no longer functions enough to generate the problem.

ARM / # strace -p 417
Process 417 attached - interrupt to quit futex(0x175734, FUTEX_WAIT_PRIVATE, 2, NULL^C <unfinished ...> Process 417 detached

ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux ARM / # uptime
 22:36:19 up 22:36,  0 users,  load average: 0.15, 0.16, 0.18 ARM / # cat /proc/cpuinfo
Processor       : ARM920T rev 0 (v4l)
BogoMIPS        : 199.06
Features        : swp half thumb crunch
CPU implementer : 0x41
CPU architecture: 4T
CPU variant     : 0x1
CPU part        : 0x920
CPU revision    : 0

Any help welcome.

Thanks,
Jon

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

#16 lore raw
Quoted 26 lines · Huang Shijie

hi,

Quoted 23 lines · Darren Hart
Quoted 8 lines · Huang Shijie

Hi,

Quoted 5 lines · Darren Hart
Quoted 2 lines · Huang Shijie

Hi ,

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

Are either of you able to bisect the kernel? At the very least can you

I finially found my arch/arm/include/asm/futex.h is not the latest, so i updated the header.

Just make sure it matches your kernel version.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

And the futex issue is gone now. But a dataAbort issue appears, I am not sure whether it caused by the futex patch.
I am debugging it now.

Which APIs are you using that make the futex syscall?

futex_wait().

You are calling the futex syscall directly from your application?

--
Darren

Quoted 109 lines

I cherry-pick the
" df77abc ARM: 7099/1: futex: preserve oldval in SMP __futex_atomic_op" to the arch/arm/include/asm/futex.h.

BR
Huang Shijie

Quoted 99 lines

--
Darren

Quoted 9 lines

BR
Huang Shijie

Quoted 5 lines

find two kernels where it works and where it does not?

Hanging on FUTEX_WAIT_PRIVATE can be the symptom for higher level problems including userspace locking issues and race conditions.

Quoted 82 lines
Quoted 1 line

Huang, are you also on ARM?

Yes, FREESCALE imx6q platform.

BR
Huang Shijie

Quoted 76 lines

--
Darren

Quoted 62 lines

BR
Huang Shijie

Quoted 52 lines · Jonathan Andrews

Using kernel 3.2.5 with alsa-lib 1.0.25, all compiled with generic Debian arm-linux-gnueabi toolchain.

arm-linux-gnueabi-gcc (Debian 4.3.2-1.1) 4.3.2 Was used to build kernel, alsa-lib and application.

Changing gcc version, kernel version or alsa-lib version makes the problem worse or better, but ALL versions seem to suffer this problem. I have also seen it once on Intel (but only once so far).

Something seeks broken at a lower layer than im using. I simply don't have the skill to debug it.

The hardware is a USB cm109 audio adapter, but the problem seems to show on more than this one driver.

The audio application writing to alsa will freezes at random intervals, infrequent at the moment, last one was after runtime 20H 37M 29S. Two processes are running, one reading from the sound device and one writing to the sound device. I am not using threading or anything very clever just generic alsa functions.

This is the only diagnostic I can generate so far as running the application under strace slows it to the point it no longer functions enough to generate the problem.

ARM / # strace -p 417
Process 417 attached - interrupt to quit futex(0x175734, FUTEX_WAIT_PRIVATE, 2, NULL^C <unfinished ...> Process 417 detached

ARM / # uname -a
Linux (none) 3.2.5 #2 Wed Feb 22 17:11:52 GMT 2012 armv4tl GNU/Linux ARM / # uptime
22:36:19 up 22:36, 0 users, load average: 0.15, 0.16, 0.18
ARM / # cat /proc/cpuinfo
Processor : ARM920T rev 0 (v4l)
BogoMIPS : 199.06
Features : swp half thumb crunch
CPU implementer : 0x41
CPU architecture: 4T
CPU variant : 0x1
CPU part : 0x920
CPU revision : 0

Any help welcome.

Thanks,
Jon

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
#17 lore raw
Quoted 33 lines · Darren Hart
Quoted 30 lines · Huang Shijie

hi,

Quoted 27 lines · Darren Hart
Quoted 8 lines · Huang Shijie

Hi,

Quoted 5 lines · Darren Hart
Quoted 2 lines · Huang Shijie

Hi ,

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I meet a similar problem with the latest futex code.

I play the video and the processes will hang at the futex.

Are either of you able to bisect the kernel? At the very least can you

I finially found my arch/arm/include/asm/futex.h is not the latest, so i updated the header.

Just make sure it matches your kernel version.

Quoted 4 lines

And the futex issue is gone now. But a dataAbort issue appears, I am not sure whether it caused by the futex patch.
I am debugging it now.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Which APIs are you using that make the futex syscall?

futex_wait().

You are calling the futex syscall directly from your application?

The application call futex(), but i ever hung at futex_wait().

Huang Shijie

A message that is not in the archive (sent off-list or not yet received): 1329933773.24266.44.camel@jonspc

#19 lore raw
Quoted 5 lines · Jonathan Andrews
Quoted 1 line · Clemens Ladisch

Now it doesn't use any pthread functions.

Thanks, use or require ? Still complains if I static compile my application but don't link with pthreads.

Please try with the patch below; this should remove the last occurence of pthreads in alsa-lib.

Regards,
Clemens

configure.in +1 -0
@@ -512,6 +512,7 @@ if test "$HAVE_LIBDL" != "yes"; then
 fi
 
 if test "$HAVE_LIBPTHREAD" != "yes"; then
+  build_pcm_meter="no"
   build_pcm_share="no"
 fi

A message that is not in the archive (sent off-list or not yet received): 1331199625.22455.7.camel@jonspc

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 1331203724.22455.44.camel@jonspc

#23 lore raw
Quoted 8 lines · Jonathan Andrews
Quoted 5 lines · Clemens Ladisch
Quoted 2 lines · Jonathan Andrews

I'm using this script to build both static and dynamic versions of the alsalib. Can you check the configure arguments are correct.

They look correct.

Tried to compile my application, still fails asking for pthreads.

The configure.in file contains the patch, but the generated configure script does not have it (although its time stamp is newer). Please ensure that you've run autoconf (although make should do this automatically).

Regards,
Clemens

#24 lore raw
Quoted 10 lines · Clemens Ladisch
Quoted 8 lines · Jonathan Andrews
Quoted 5 lines · Clemens Ladisch
Quoted 2 lines · Jonathan Andrews

I'm using this script to build both static and dynamic versions of the alsalib. Can you check the configure arguments are correct.

They look correct.

Tried to compile my application, still fails asking for pthreads.

Quoted 8 lines

The configure.in file contains the patch, but the generated configure script does not have it (although its time stamp is newer). Please ensure that you've run autoconf (although make should do this automatically).

Regards,
Clemens

aclocal.m4:16: warning: this file was generated for autoconf 2.68. You have another version of autoconf. It may work, but is not guaranteed to.
If you have problems, you may need to regenerate the build system entirely.
To do so, use the procedure documented by the package, typically `autoreconf'.
configure.in:15: error: Autoconf version 2.62 or higher is required aclocal.m4:6823: AM_INIT_AUTOMAKE is expanded from... configure.in:15: the top level
autom4te: /usr/bin/m4 failed with exit status: 63

Thanks for finding that, seems its a version problem. My machine has version 2.61

Upgrading now.

Thanks,
Jon

#25 lore raw
In reply to Clemens Ladisch
Quoted 6 lines · Clemens Ladisch
Quoted 5 lines · Jonathan Andrews
Quoted 4 lines · Clemens Ladisch
Quoted 2 lines · Jonathan Andrews

I'm using this script to build both static and dynamic versions of the alsalib. Can you check the configure arguments are correct.

Quoted 10 lines
Quoted 3 lines
Quoted 1 line

They look correct.

Tried to compile my application, still fails asking for pthreads.

The configure.in file contains the patch, but the generated configure script does not have it (although its time stamp is newer). Please ensure that you've run autoconf (although make should do this automatically).

Getting closer, alsalib now builds without the need for pthreads but seems to insist on dlopen dlclose even in static lib ?

Compiling udp-many-way-audio-rx.c -> udp-many-way-audio-rx_alsa_arms (static)
/usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(pcm_dshare.o): In function `snd_pcm_dshare_sync_ptr':
pcm_dshare.c:(.text+0x6a4): warning: Warning: snd_pcm_hwsync() is deprecated, consider to use snd_pcm_avail() /usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(pcm_direct.o): In function `snd1_pcm_direct_parse_open_conf':
pcm_direct.c:(.text+0x58c8): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(pcm_shm.o): In function `_snd_pcm_shm_open':
pcm_shm.c:(.text+0x27dc): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking /usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(dlmisc.o): In function `snd_dlopen':
dlmisc.c:(.text+0x38): undefined reference to dlopen' /usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(dlmisc.o): In function snd_dlclose':
dlmisc.c:(.text+0x90): undefined reference to dlclose' /usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(dlmisc.o): In function snd_dlsym_verify':
dlmisc.c:(.text+0x190): undefined reference to dlsym' /usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(dlmisc.o): In function snd_dlsym':
dlmisc.c:(.text+0x2f8): undefined reference to dlsym' /usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(pcm_ladspa.o): In function snd_pcm_ladspa_free_plugins':
pcm_ladspa.c:(.text+0x3cc): undefined reference to dlclose' /usr/lib/gcc/arm-linux-gnueabi/4.3.2/../../../../arm-linux-gnueabi/lib/libasound.a(pcm_ladspa.o): In function snd_pcm_ladspa_check_file':
pcm_ladspa.c:(.text+0x41c4): undefined reference to `dlopen'
pcm_ladspa.c:(.text+0x41e4): undefined reference to `dlsym'
pcm_ladspa.c:(.text+0x4250): undefined reference to `dlclose'
pcm_ladspa.c:(.text+0x4344): undefined reference to `dlclose'
pcm_ladspa.c:(.text+0x43b0): undefined reference to `dlclose'
collect2: ld returned 1 exit status
~

#26 lore raw
Quoted 2 lines · Jonathan Andrews

Getting closer, alsalib now builds without the need for pthreads but seems to insist on dlopen dlclose even in static lib ?

Add --without-libdl.

Regards,
Clemens

#27 lore raw
Quoted 6 lines · Clemens Ladisch
Quoted 2 lines · Jonathan Andrews

Getting closer, alsalib now builds without the need for pthreads but seems to insist on dlopen dlclose even in static lib ?

Add --without-libdl.

Ok, that fixed it thanks.

A couple of comments.

1) A lot of my trouble was this :
configure.in:15: error: Autoconf version 2.62 or higher is required

If autoconf exists, is executed, but does not complete should configure not exit with an error ?

I checked the output of the configure/make carefully. Autoconf was silently failing, gives no clues at all. As the datetime is in the binary created each library had a unique file hash so I (reasonably) assumed it to be doing the job :-(

2) "Add --without-libdl",
ok ... should "--static" not turn this on then ?

Thanks,
Jon

#28 lore raw

At Thu, 08 Mar 2012 15:14:50 +0000,

Quoted 11 lines · Jonathan Andrews
Quoted 6 lines · Clemens Ladisch
Quoted 2 lines · Jonathan Andrews

Getting closer, alsalib now builds without the need for pthreads but seems to insist on dlopen dlclose even in static lib ?

Add --without-libdl.

Ok, that fixed it thanks.

Quoted 7 lines

A couple of comments.

1) A lot of my trouble was this :
configure.in:15: error: Autoconf version 2.62 or higher is required

If autoconf exists, is executed, but does not complete should configure not exit with an error ?

Blame your autoconf :)

Quoted 7 lines

I checked the output of the configure/make carefully. Autoconf was silently failing, gives no clues at all. As the datetime is in the binary created each library had a unique file hash so I (reasonably) assumed it to be doing the job :-(

2) "Add --without-libdl",
ok ... should "--static" not turn this on then ?

dlopen() can be still used for taking other objects like LADSPA or external io or pcm plugins. There is no reason to prohibit dlopen() in a static library.

Takashi

#29 lore raw
Quoted 21 lines · Takashi Iwai

At Thu, 08 Mar 2012 15:14:50 +0000,

Quoted 18 lines · Jonathan Andrews
Quoted 6 lines · Clemens Ladisch
Quoted 2 lines · Jonathan Andrews

Getting closer, alsalib now builds without the need for pthreads but seems to insist on dlopen dlclose even in static lib ?

Add --without-libdl.

Ok, that fixed it thanks.

A couple of comments.

1) A lot of my trouble was this :
configure.in:15: error: Autoconf version 2.62 or higher is required

If autoconf exists, is executed, but does not complete should configure not exit with an error ?

Blame your autoconf :)
Hmmmm .... true, but still not convinced this could not have been prevented by reading the errorlevel and crapping out with a nice fluffy error message for me !

Quoted 11 lines
Quoted 7 lines

I checked the output of the configure/make carefully. Autoconf was silently failing, gives no clues at all. As the datetime is in the binary created each library had a unique file hash so I (reasonably) assumed it to be doing the job :-(

2) "Add --without-libdl",
ok ... should "--static" not turn this on then ?

dlopen() can be still used for taking other objects like LADSPA or external io or pcm plugins. There is no reason to prohibit dlopen() in a static library.

Ok, makes sense.

Thanks,
Jon

#30 lore raw
In reply to Takashi Iwai
Quoted 10 lines · Takashi Iwai

At Thu, 08 Mar 2012 15:14:50 +0000,

Quoted 8 lines · Jonathan Andrews
Quoted 6 lines · Clemens Ladisch
Quoted 2 lines · Jonathan Andrews

Getting closer, alsalib now builds without the need for pthreads but seems to insist on dlopen dlclose even in static lib ?

Add --without-libdl.

Using the static libasound without pthreads now.

Process still stalls in same way:

ARM / # strace -p 408
Process 408 attached - interrupt to quit
futex(0x15a684, FUTEX_WAIT, 2, NULL
^C <unfinished ...>
Process 408 detached

Took 3 hours 1 min to stall. It was decoding 8 udp audio streams, a second tx process reading from the same sound device was also running.

An strace of a stall (intel this time) is here. Its 2GB and on a slow link (sorry).
http://www.jonshouse.co.uk/download/a_stop.txt

futex(0xae5320, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---

The futex address 0xae5320 occurs only once at the end of the text file so I have no idea what sets it up.

Thanks,
Jon

#31 lore raw
Quoted 7 lines · Jonathan Andrews

Using the static libasound without pthreads now.

Process still stalls in same way:

ARM / # strace -p 408
Process 408 attached - interrupt to quit
futex(0x15a684, FUTEX_WAIT, 2, NULL

So it's not a call from alsa-lib.

Quoted 3 lines

An strace of a stall (intel this time) is here. Its 2GB and on a slow link (sorry).
http://www.jonshouse.co.uk/download/a_stop.txt

If only there were a tool to compress files ...

Quoted 2 lines

The futex address 0xae5320 occurs only once at the end of the text file so I have no idea what sets it up.

Can't you attach gdb to your process?
Or at least log where in your program you are?
(If strace has too much overhead, write to a variable, and run a thread that waits for keyboard input and prints the variable.)

Regards,
Clemens

#32 lore raw
Quoted 10 lines · Clemens Ladisch
Quoted 7 lines · Jonathan Andrews

Using the static libasound without pthreads now.

Process still stalls in same way:

ARM / # strace -p 408
Process 408 attached - interrupt to quit
futex(0x15a684, FUTEX_WAIT, 2, NULL

So it's not a call from alsa-lib.

The alsa code is a conditional compile, as is that for the /dev/dsp device. Only the alsa version fails, the version that just takes data from the network and mixes it does not fail.

cc udp-many-way-audio-rx.c -DARM -DALSA -lasound -lrt -lm -o udp-many-way-audio-rx_alsa_arms --static Fails randomly

cc udp-many-way-audio-rx.c -DARM -DDUMMY -DSTATIC -o udp-many-way-audio-rx_dummy_arms --static Dummy version, never fails. Still has network code and my audio mixer.

Its not as if I'm linking in a lot of libs...

Quoted 5 lines
Quoted 3 lines

An strace of a stall (intel this time) is here. Its 2GB and on a slow link (sorry).
http://www.jonshouse.co.uk/download/a_stop.txt

If only there were a tool to compress files ...

Ok, i'm pretty tired I admit it .... 30MB
http://www.jonshouse.co.uk/download/a_stop.txt.gz

Quoted 5 lines
Quoted 3 lines

The futex address 0xae5320 occurs only once at the end of the text file so I have no idea what sets it up.

Can't you attach gdb to your process?

I need to build an arm version, will work on that next.

Quoted 1 line

Or at least log where in your program you are?

I did that, best I can tell it enters one of these two functions but never exists. Makes sense as these are the two conditional compiles wrapped around #ifdef ALSA in the main loop.
http://www.jonshouse.co.uk/download/udp-many-way-audio-playsample-alsa.c

Quoted 2 lines

(If strace has too much overhead, write to a variable, and run a thread that waits for keyboard input and prints the variable.)

The strace (now gzipped) is the same process as the arm running (and then stalling) on the PC. It still stalls, just takes (on average) longer to stall on the faster machine ......

Thanks,
Jon