ALSA: hda: PCM streams are suspended while opening

From: Jonathan Hunter <[email protected]>

This issue is not observed on the latest mainline kernel since the 'ALSA: PCM suspend cleanup' series has been applied and the snd_pcm_suspend_all() function call in the HDA codec driver has been removed. However, this issue impacts stable kernel branches and so I am trying to find a solution for these branches.

When stress testing audio playback across multiple HDA streams simultaneously, the following failure is sometimes observed when starting playback ...

Unable to set hw params for playback: File descriptor in bad state

The problem is caused because ...
1. Playback is starting for one HDA codec and so the chip->open_mutex in azx_pcm_open() has been acquired.
2. Playback for another HDA codec is starting but is blocked waiting to acquire the chip->open_mutex in azx_pcm_open().
3. For the HDA codec that is blocked, its runtime-pm status is still enabled from a previous playback session that has since completed and been closed, however its autosuspend timeout has not expired yet.
4. For the HDA codec that is blocked, the runtime-pm autosuspend timeout now occurs calling the runtime-pm suspend callback and this calls snd_pcm_suspend_all() placing all PCM streams in the suspended state.
5. The block HDA codec is now unblocked and fails to set the HW params because the PCM stream is in the suspended state.

The above has been observed on Tegra platforms where the autosuspend delay is set to 1 second and there is a delay to an AZX command. This highlights that there is a window of time where autosuspend can place the HDA PCM stream in the suspended state while opening the stream and cause playback to fail.

Looking at commit 3d21ef0b49f8 ('ALSA: pcm: Suspend streams globally via device type PM ops') it appears that the call to snd_pcm_suspend_all() has now been moved to the to the suspend handler for PCM streams and so I am wondering if it would be equivalent to make the following change to the HDA codec driver to achieve the same behaviour but only for the HDA driver. So far the issue has not been observed with this change.

Please note that I am not sending this as a formal patch as I wanted to get some feedback/comments on the above.

Quoted 42 lines · Jon Hunter

From: Jonathan Hunter <[email protected]>

This issue is not observed on the latest mainline kernel since the 'ALSA: PCM suspend cleanup' series has been applied and the snd_pcm_suspend_all() function call in the HDA codec driver has been removed. However, this issue impacts stable kernel branches and so I am trying to find a solution for these branches.

When stress testing audio playback across multiple HDA streams simultaneously, the following failure is sometimes observed when starting playback ...

Unable to set hw params for playback: File descriptor in bad state

The problem is caused because ...
1. Playback is starting for one HDA codec and so the chip->open_mutex in azx_pcm_open() has been acquired.
2. Playback for another HDA codec is starting but is blocked waiting to acquire the chip->open_mutex in azx_pcm_open().
3. For the HDA codec that is blocked, its runtime-pm status is still enabled from a previous playback session that has since completed and been closed, however its autosuspend timeout has not expired yet.
4. For the HDA codec that is blocked, the runtime-pm autosuspend timeout now occurs calling the runtime-pm suspend callback and this calls snd_pcm_suspend_all() placing all PCM streams in the suspended state.
5. The block HDA codec is now unblocked and fails to set the HW params because the PCM stream is in the suspended state.

The above has been observed on Tegra platforms where the autosuspend delay is set to 1 second and there is a delay to an AZX command. This highlights that there is a window of time where autosuspend can place the HDA PCM stream in the suspended state while opening the stream and cause playback to fail.

Looking at commit 3d21ef0b49f8 ('ALSA: pcm: Suspend streams globally via device type PM ops') it appears that the call to snd_pcm_suspend_all() has now been moved to the to the suspend handler for PCM streams and so I am wondering if it would be equivalent to make the following change to the HDA codec driver to achieve the same behaviour but only for the HDA driver. So far the issue has not been observed with this change.

Quoted 2 lines

Please note that I am not sending this as a formal patch as I wanted to get some feedback/comments on the above.

I guess just backporting 3d21ef0b49f8 should be OK even for older kernels. This assures the PCM stream gets suspended before entering any parent device suspend call.

The remaining changes (except for the one for atiixp) are merely cleanup of superfluous calls, and keeping them are harmless.

Could you check whether my theory above correct?

Quoted 54 lines · Takashi Iwai

Quoted 42 lines · Jon Hunter

From: Jonathan Hunter <[email protected]>

This issue is not observed on the latest mainline kernel since the 'ALSA: PCM suspend cleanup' series has been applied and the snd_pcm_suspend_all() function call in the HDA codec driver has been removed. However, this issue impacts stable kernel branches and so I am trying to find a solution for these branches.

When stress testing audio playback across multiple HDA streams simultaneously, the following failure is sometimes observed when starting playback ...

Unable to set hw params for playback: File descriptor in bad state

The problem is caused because ...
1. Playback is starting for one HDA codec and so the chip->open_mutex in azx_pcm_open() has been acquired.
2. Playback for another HDA codec is starting but is blocked waiting to acquire the chip->open_mutex in azx_pcm_open().
3. For the HDA codec that is blocked, its runtime-pm status is still enabled from a previous playback session that has since completed and been closed, however its autosuspend timeout has not expired yet.
4. For the HDA codec that is blocked, the runtime-pm autosuspend timeout now occurs calling the runtime-pm suspend callback and this calls snd_pcm_suspend_all() placing all PCM streams in the suspended state.
5. The block HDA codec is now unblocked and fails to set the HW params because the PCM stream is in the suspended state.

The above has been observed on Tegra platforms where the autosuspend delay is set to 1 second and there is a delay to an AZX command. This highlights that there is a window of time where autosuspend can place the HDA PCM stream in the suspended state while opening the stream and cause playback to fail.

Looking at commit 3d21ef0b49f8 ('ALSA: pcm: Suspend streams globally via device type PM ops') it appears that the call to snd_pcm_suspend_all() has now been moved to the to the suspend handler for PCM streams and so I am wondering if it would be equivalent to make the following change to the HDA codec driver to achieve the same behaviour but only for the HDA driver. So far the issue has not been observed with this change.

Quoted 2 lines

Please note that I am not sending this as a formal patch as I wanted to get some feedback/comments on the above.

I guess just backporting 3d21ef0b49f8 should be OK even for older kernels. This assures the PCM stream gets suspended before entering any parent device suspend call.

The remaining changes (except for the one for atiixp) are merely cleanup of superfluous calls, and keeping them are harmless.

Quoted 1 line

Could you check whether my theory above correct?

I believe that you will need that change and 17bc4815de58 because we need to prevent snd_pcm_suspend_all() being called in the pm-runtime suspend callback. Applying only 3d21ef0b49f8 will means that at runtime snd_pcm_suspend_all() can still be called from within the HDA codec driver when the autosuspend timeout occurs. I will test this and confirm. Maybe both could be backported?

Quoted 56 lines · Jon Hunter

Quoted 54 lines · Takashi Iwai

Quoted 42 lines · Jon Hunter

From: Jonathan Hunter <[email protected]>

This issue is not observed on the latest mainline kernel since the 'ALSA: PCM suspend cleanup' series has been applied and the snd_pcm_suspend_all() function call in the HDA codec driver has been removed. However, this issue impacts stable kernel branches and so I am trying to find a solution for these branches.

When stress testing audio playback across multiple HDA streams simultaneously, the following failure is sometimes observed when starting playback ...

Unable to set hw params for playback: File descriptor in bad state

The problem is caused because ...
1. Playback is starting for one HDA codec and so the chip->open_mutex in azx_pcm_open() has been acquired.
2. Playback for another HDA codec is starting but is blocked waiting to acquire the chip->open_mutex in azx_pcm_open().
3. For the HDA codec that is blocked, its runtime-pm status is still enabled from a previous playback session that has since completed and been closed, however its autosuspend timeout has not expired yet.
4. For the HDA codec that is blocked, the runtime-pm autosuspend timeout now occurs calling the runtime-pm suspend callback and this calls snd_pcm_suspend_all() placing all PCM streams in the suspended state.
5. The block HDA codec is now unblocked and fails to set the HW params because the PCM stream is in the suspended state.

The above has been observed on Tegra platforms where the autosuspend delay is set to 1 second and there is a delay to an AZX command. This highlights that there is a window of time where autosuspend can place the HDA PCM stream in the suspended state while opening the stream and cause playback to fail.

Looking at commit 3d21ef0b49f8 ('ALSA: pcm: Suspend streams globally via device type PM ops') it appears that the call to snd_pcm_suspend_all() has now been moved to the to the suspend handler for PCM streams and so I am wondering if it would be equivalent to make the following change to the HDA codec driver to achieve the same behaviour but only for the HDA driver. So far the issue has not been observed with this change.

Quoted 2 lines

Please note that I am not sending this as a formal patch as I wanted to get some feedback/comments on the above.

I guess just backporting 3d21ef0b49f8 should be OK even for older kernels. This assures the PCM stream gets suspended before entering any parent device suspend call.

The remaining changes (except for the one for atiixp) are merely cleanup of superfluous calls, and keeping them are harmless.

Quoted 8 lines

Quoted 1 line

Could you check whether my theory above correct?

I believe that you will need that change and 17bc4815de58 because we need to prevent snd_pcm_suspend_all() being called in the pm-runtime suspend callback. Applying only 3d21ef0b49f8 will means that at runtime snd_pcm_suspend_all() can still be called from within the HDA codec driver when the autosuspend timeout occurs. I will test this and confirm. Maybe both could be backported?

I confirmed today that with just 3d21ef0b49f8 the issue can occur, but with both 3d21ef0b49f8 and 17bc4815de58 the issue is not seen.

Do you think that it would be appropriate to include these in the stable branches? They apply cleanly to v5.0, but we would probably need to back-port for early kernels such as v4.19, v4.9, etc.

Quoted 68 lines · Jon Hunter

Quoted 56 lines · Jon Hunter

Quoted 54 lines · Takashi Iwai

Quoted 42 lines · Jon Hunter

From: Jonathan Hunter <[email protected]>

This issue is not observed on the latest mainline kernel since the 'ALSA: PCM suspend cleanup' series has been applied and the snd_pcm_suspend_all() function call in the HDA codec driver has been removed. However, this issue impacts stable kernel branches and so I am trying to find a solution for these branches.

When stress testing audio playback across multiple HDA streams simultaneously, the following failure is sometimes observed when starting playback ...

Unable to set hw params for playback: File descriptor in bad state

The problem is caused because ...
1. Playback is starting for one HDA codec and so the chip->open_mutex in azx_pcm_open() has been acquired.
2. Playback for another HDA codec is starting but is blocked waiting to acquire the chip->open_mutex in azx_pcm_open().
3. For the HDA codec that is blocked, its runtime-pm status is still enabled from a previous playback session that has since completed and been closed, however its autosuspend timeout has not expired yet.
4. For the HDA codec that is blocked, the runtime-pm autosuspend timeout now occurs calling the runtime-pm suspend callback and this calls snd_pcm_suspend_all() placing all PCM streams in the suspended state.
5. The block HDA codec is now unblocked and fails to set the HW params because the PCM stream is in the suspended state.

The above has been observed on Tegra platforms where the autosuspend delay is set to 1 second and there is a delay to an AZX command. This highlights that there is a window of time where autosuspend can place the HDA PCM stream in the suspended state while opening the stream and cause playback to fail.

Looking at commit 3d21ef0b49f8 ('ALSA: pcm: Suspend streams globally via device type PM ops') it appears that the call to snd_pcm_suspend_all() has now been moved to the to the suspend handler for PCM streams and so I am wondering if it would be equivalent to make the following change to the HDA codec driver to achieve the same behaviour but only for the HDA driver. So far the issue has not been observed with this change.

Quoted 2 lines

Please note that I am not sending this as a formal patch as I wanted to get some feedback/comments on the above.

I guess just backporting 3d21ef0b49f8 should be OK even for older kernels. This assures the PCM stream gets suspended before entering any parent device suspend call.

The remaining changes (except for the one for atiixp) are merely cleanup of superfluous calls, and keeping them are harmless.

Quoted 8 lines

Quoted 1 line

Could you check whether my theory above correct?

I believe that you will need that change and 17bc4815de58 because we need to prevent snd_pcm_suspend_all() being called in the pm-runtime suspend callback. Applying only 3d21ef0b49f8 will means that at runtime snd_pcm_suspend_all() can still be called from within the HDA codec driver when the autosuspend timeout occurs. I will test this and confirm. Maybe both could be backported?

Quoted 6 lines

I confirmed today that with just 3d21ef0b49f8 the issue can occur, but with both 3d21ef0b49f8 and 17bc4815de58 the issue is not seen.

Do you think that it would be appropriate to include these in the stable branches? They apply cleanly to v5.0, but we would probably need to back-port for early kernels such as v4.19, v4.9, etc.

I reconsidered the problem again, and noticed that the very same problem may appear with the system PM, not only with runtime PM. The suspend can happen at any time, so even a stream in OPEN state may go to suspend, and you'll hit the same problem. It's just a corner case so no one really cared much.

So, I think the patch like below should fix the problem. This can be easily backported to all stable trees, and it alone should work without backporting the else intrusive changes.

Could you check whether my theory is correct?

Quoted 77 lines · Takashi Iwai

Quoted 68 lines · Jon Hunter

Quoted 56 lines · Jon Hunter

Quoted 54 lines · Takashi Iwai

Quoted 42 lines · Jon Hunter

From: Jonathan Hunter <[email protected]>

This issue is not observed on the latest mainline kernel since the 'ALSA: PCM suspend cleanup' series has been applied and the snd_pcm_suspend_all() function call in the HDA codec driver has been removed. However, this issue impacts stable kernel branches and so I am trying to find a solution for these branches.

When stress testing audio playback across multiple HDA streams simultaneously, the following failure is sometimes observed when starting playback ...

Unable to set hw params for playback: File descriptor in bad state

The problem is caused because ...
1. Playback is starting for one HDA codec and so the chip->open_mutex in azx_pcm_open() has been acquired.
2. Playback for another HDA codec is starting but is blocked waiting to acquire the chip->open_mutex in azx_pcm_open().
3. For the HDA codec that is blocked, its runtime-pm status is still enabled from a previous playback session that has since completed and been closed, however its autosuspend timeout has not expired yet.
4. For the HDA codec that is blocked, the runtime-pm autosuspend timeout now occurs calling the runtime-pm suspend callback and this calls snd_pcm_suspend_all() placing all PCM streams in the suspended state.
5. The block HDA codec is now unblocked and fails to set the HW params because the PCM stream is in the suspended state.

The above has been observed on Tegra platforms where the autosuspend delay is set to 1 second and there is a delay to an AZX command. This highlights that there is a window of time where autosuspend can place the HDA PCM stream in the suspended state while opening the stream and cause playback to fail.

Looking at commit 3d21ef0b49f8 ('ALSA: pcm: Suspend streams globally via device type PM ops') it appears that the call to snd_pcm_suspend_all() has now been moved to the to the suspend handler for PCM streams and so I am wondering if it would be equivalent to make the following change to the HDA codec driver to achieve the same behaviour but only for the HDA driver. So far the issue has not been observed with this change.

Quoted 2 lines

Please note that I am not sending this as a formal patch as I wanted to get some feedback/comments on the above.

I guess just backporting 3d21ef0b49f8 should be OK even for older kernels. This assures the PCM stream gets suspended before entering any parent device suspend call.

The remaining changes (except for the one for atiixp) are merely cleanup of superfluous calls, and keeping them are harmless.

Quoted 8 lines

Quoted 1 line

Could you check whether my theory above correct?

I believe that you will need that change and 17bc4815de58 because we need to prevent snd_pcm_suspend_all() being called in the pm-runtime suspend callback. Applying only 3d21ef0b49f8 will means that at runtime snd_pcm_suspend_all() can still be called from within the HDA codec driver when the autosuspend timeout occurs. I will test this and confirm. Maybe both could be backported?

Quoted 6 lines

I confirmed today that with just 3d21ef0b49f8 the issue can occur, but with both 3d21ef0b49f8 and 17bc4815de58 the issue is not seen.

Do you think that it would be appropriate to include these in the stable branches? They apply cleanly to v5.0, but we would probably need to back-port for early kernels such as v4.19, v4.9, etc.

Quoted 5 lines

I reconsidered the problem again, and noticed that the very same problem may appear with the system PM, not only with runtime PM. The suspend can happen at any time, so even a stream in OPEN state may go to suspend, and you'll hit the same problem. It's just a corner case so no one really cared much.

Yes I was not sure if it could also happen in the system suspend case or not.

Quoted 5 lines

So, I think the patch like below should fix the problem. This can be easily backported to all stable trees, and it alone should work without backporting the else intrusive changes.

Could you check whether my theory is correct?

Do you want me to test this on the same stable branch I have been testing with where commits 3d21ef0b49f8 and 17bc4815de58 are not present?

I reconsidered the problem again, and noticed that the very same problem may appear with the system PM, not only with runtime PM. The suspend can happen at any time, so even a stream in OPEN state may go to suspend, and you'll hit the same problem. It's just a corner case so no one really cared much.

So, I think the patch like below should fix the problem. This can be easily backported to all stable trees, and it alone should work without backporting the else intrusive changes.

Could you check whether my theory is correct?

thanks,

Takashi

---
From: Takashi Iwai <[email protected]>
Subject: [PATCH] ALSA: pcm: Don't suspend stream in unrecoverable PCM state

Currently PCM core sets each opened stream forcibly to SUSPENDED state via snd_pcm_suspend_all() call, and the user-space is responsible for re-triggering the resume manually either via snd_pcm_resume() or prepare call. The scheme works fine usually, but there are corner cases where the stream can't be resumed by that call: the streams still in OPEN state before finishing hw_params. When they are suspended, user-space cannot perform resume or prepare because they haven't been set up yet. The only possible recovery is to re-open the device, which isn't nice at all. Similarly, when a stream is in DISCONNECTED state, it makes no sense to change it to SUSPENDED state. Ditto for in SETUP state; which you can re-prepare directly.

So, this patch addresses these issues by filtering the PCM streams to be suspended by checking the PCM state. When a stream is in either OPEN, SETUP or DISCONNECTED as well as already SUSPENDED, the suspend action is skipped.

To be noted, this problem was originally reported for the PCM runtime PM on HD-audio. And, the runtime PM problem itself was already addressed (although not intended) by the code refactoring commits 3d21ef0b49f8 ("ALSA: pcm: Suspend streams globally via device type PM ops") and 17bc4815de58 ("ALSA: pci: Remove superfluous snd_pcm_suspend*() calls"). These commits eliminated the snd_pcm_suspend*() calls from the runtime PM suspend callback code path, hence the racy OPEN state won't appear while runtime PM. (FWIW, the race window is between snd_pcm_open_substream() and the first power up in azx_pcm_open().)

Although the runtime PM issue was already "fixed", the same problem is still present for the system PM, hence this patch is still needed. And for stable trees, this patch alone should suffice for fixing the runtime PM problem, too.

Reported-by Jonathan Hunter <[email protected]>

Cc <[email protected]>

Signed-off-by Takashi Iwai <[email protected]>

---
 sound/core/pcm_native.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

sound/core/pcm_native.c +2 -1

index f731f904e8cc..1d8452912b14 100644
@@ -1445,8 +1445,15 @@ static int snd_pcm_pause(struct snd_pcm_substream *substream, int push)
 static int snd_pcm_pre_suspend(struct snd_pcm_substream *substream, int state)
 {
 	struct snd_pcm_runtime *runtime = substream->runtime;
-	if (runtime->status->state == SNDRV_PCM_STATE_SUSPENDED)
+	switch (runtime->status->state) {
+	case SNDRV_PCM_STATE_SUSPENDED:

On 25/03/2019 09:59, Takashi Iwai wrote:

...

Quoted 72 lines

I reconsidered the problem again, and noticed that the very same problem may appear with the system PM, not only with runtime PM. The suspend can happen at any time, so even a stream in OPEN state may go to suspend, and you'll hit the same problem. It's just a corner case so no one really cared much.

So, I think the patch like below should fix the problem. This can be easily backported to all stable trees, and it alone should work without backporting the else intrusive changes.

Could you check whether my theory is correct?

thanks,

Takashi

---
From: Takashi Iwai <[email protected]>
Subject: [PATCH] ALSA: pcm: Don't suspend stream in unrecoverable PCM state

Currently PCM core sets each opened stream forcibly to SUSPENDED state via snd_pcm_suspend_all() call, and the user-space is responsible for re-triggering the resume manually either via snd_pcm_resume() or prepare call. The scheme works fine usually, but there are corner cases where the stream can't be resumed by that call: the streams still in OPEN state before finishing hw_params. When they are suspended, user-space cannot perform resume or prepare because they haven't been set up yet. The only possible recovery is to re-open the device, which isn't nice at all. Similarly, when a stream is in DISCONNECTED state, it makes no sense to change it to SUSPENDED state. Ditto for in SETUP state; which you can re-prepare directly.

So, this patch addresses these issues by filtering the PCM streams to be suspended by checking the PCM state. When a stream is in either OPEN, SETUP or DISCONNECTED as well as already SUSPENDED, the suspend action is skipped.

To be noted, this problem was originally reported for the PCM runtime PM on HD-audio. And, the runtime PM problem itself was already addressed (although not intended) by the code refactoring commits 3d21ef0b49f8 ("ALSA: pcm: Suspend streams globally via device type PM ops") and 17bc4815de58 ("ALSA: pci: Remove superfluous snd_pcm_suspend*() calls"). These commits eliminated the snd_pcm_suspend*() calls from the runtime PM suspend callback code path, hence the racy OPEN state won't appear while runtime PM. (FWIW, the race window is between snd_pcm_open_substream() and the first power up in azx_pcm_open().)

Although the runtime PM issue was already "fixed", the same problem is still present for the system PM, hence this patch is still needed. And for stable trees, this patch alone should suffice for fixing the runtime PM problem, too.

Reported-by Jonathan Hunter <[email protected]>

Cc <[email protected]>

Signed-off-by Takashi Iwai <[email protected]>

---
 sound/core/pcm_native.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

sound/core/pcm_native.c +2 -1

index f731f904e8cc..1d8452912b14 100644
@@ -1445,8 +1445,15 @@ static int snd_pcm_pause(struct snd_pcm_substream *substream, int push)
 static int snd_pcm_pre_suspend(struct snd_pcm_substream *substream, int state)
 {
 	struct snd_pcm_runtime *runtime = substream->runtime;
-	if (runtime->status->state == SNDRV_PCM_STATE_SUSPENDED)
+	switch (runtime->status->state) {
+	case SNDRV_PCM_STATE_SUSPENDED:

sound/core/pcm_native.c static int snd_pcm_pause(struct snd_pcm_substream *substream, int push) · line 1459

Quoted 10 lines

sound/core/pcm_native.c +6 -0

 		return -EBUSY;
+	/* unresumable PCM state; return -EBUSY for skipping suspend */
+	case SNDRV_PCM_STATE_OPEN:
+	case SNDRV_PCM_STATE_SETUP:
+	case SNDRV_PCM_STATE_DISCONNECTED:
+		return -EBUSY;
+	}
 	runtime->trigger_master = substream;
 	return 0;
 }

Thanks, this works for me! I have tested this on stable branch linux-5.0.y with Tegra194. So feel free to add my ...

Tested-by Jon Hunter <[email protected]>