Skip to contentSkip to conversation

media: synopsys: hdmirx: add HDMI audio capture support

PATCH In review Igor Paunovic 5 participants 20 messages Jul 21 – Sep 23

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v4 cover media: synopsys: hdmirx: add HDMI audio capture support

This series adds audio capture support to the Synopsys DesignWare HDMI RX controller used on the Rockchip RK3588: the controller's audio FIFO is exposed through an ASoC hdmi-codec device, with a periodic worker that keeps the local audio clock locked to the incoming stream by tracking the FIFO fill level. Together with the two dts patches this yields a capture-only ALSA card fed by the HDMI input.

Validated on the Orange Pi 5 Plus against multiple HDMI sources:
capture follows the source sample rate (44.1/48 kHz switches included), survives repeated stream start/stop cycles and SIGKILL of the capturing process, stays free of FIFO under/overruns via the clock tracking, and now also survives system suspend with an active capture stream: audio resumes flowing on its own after the cycle.

Changes in v4:
- Driver: support system suspend instead of the -EBUSY guard discussed on the v3 thread (Dmitry Osipenko's preference). The worker is stopped before the controller clocks are gated; resume fully resets the controller, so the audio path is re-programmed from the last known sample rate and the worker re-armed while a capture stream is active, through a hdmirx_audio_setup() helper shared with hw_params(). Validated with an active capture stream across the suspend/resume cycle.
- Driver: use swab32() for the ACR word reversal; readl() already abstracts bus endianness, so the reversal must be unconditional (automated review, valid finding)
- dts: the sound card is now a shared, disabled-by-default "RK3588 HDMI-IN" node in rk3588-extra.dtsi, mirroring the HDMI TX sound cards; the board patch only flips it on (Sebastian Reichel)
- binding: reference dai-common.yaml, switch to
unevaluatedProperties: false, and drop the consumer comment from the example (Krzysztof Kozlowski)

Known limitation, noted on the v3 thread and under investigation as a
follow-up: a second suspend/resume cycle in the same boot can leave the audio datapath silent until reboot. It reproduces with no-stream suspend cycles as well, so it is independent of the suspend handling added here, and looks related to the cold-start behavior described in the v3 testing notes.

Link to v3: https://lore.kernel.org/linux-media/[email protected]/
Link to v2: https://lore.kernel.org/linux-media/[email protected]/
Link to v1 (RFC): https://lore.kernel.org/linux-media/[email protected]/

Igor Paunovic (4):
dt-bindings: media: snps,dw-hdmi-rx: add #sound-dai-cells
media: synopsys: hdmirx: add HDMI audio capture support
arm64: dts: rockchip: add HDMI RX audio on RK3588
arm64: dts: rockchip: enable HDMI RX audio capture on Orange Pi 5 Plus

 .../bindings/media/snps,dw-hdmi-rx.yaml       |  13 +-
 .../arm64/boot/dts/rockchip/rk3588-extra.dtsi |  18 ++
 .../dts/rockchip/rk3588-orangepi-5-plus.dts   |   8 +
 .../platform/synopsys/hdmirx/snps_hdmirx.c    | 306 ++++++++++++++++++
 .../platform/synopsys/hdmirx/snps_hdmirx.h    |   8 +
 5 files changed, 352 insertions(+), 1 deletion(-)
Signature · 1 line
2.53.0
#1 lore raw

PATCH v4 1/4 dt-bindings: media: snps,dw-hdmi-rx: add #sound-dai-cells

The HDMI RX controller can expose the audio embedded in the incoming HDMI stream as an ALSA capture device. Document the #sound-dai-cells property so that a sound card can reference the HDMI RX audio DAI.

---
Changes in v4 (both suggested by Krzysztof Kozlowski):
- reference dai-common.yaml and switch additionalProperties to
unevaluatedProperties: false
- drop the consumer comment from the binding example

Changes in v3:
- no changes

Changes in v2 (all suggested by Sebastian Reichel):
- reworded the commit message to describe the hardware
- property description now documents the DAI indexes (0 = I2S, 1 = S/PDIF)
- moved the consumer reference into the binding example

 .../devicetree/bindings/media/snps,dw-hdmi-rx.yaml  | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
Documentation/devicetree/bindings/media/snps,dw-hdmi-rx.yaml +12 -1
index b7f6c87..b80660d 100644
@@ -78,6 +78,13 @@ properties:
       The phandle of the syscon node for the Video Output GRF register
       to enable EDID transfer through SDAIN and SCLIN.
 
+  "#sound-dai-cells":
+    const: 1
+    description:
+      The HDMI RX controller has two digital audio interfaces, one for
+      I2S and one for S/PDIF. The DAI cell selects the interface, 0 for
+      I2S and 1 for S/PDIF.
+
 required:
   - compatible
   - reg
@@ -90,7 +97,10 @@ required:
   - pinctrl-0
   - hpd-gpios
 
-additionalProperties: false
+allOf:
+  - $ref: /schemas/sound/dai-common.yaml#
+
+unevaluatedProperties: false
 
 examples:
   - |
@@ -129,4 +139,5 @@ examples:
       pinctrl-0 = <&hdmim1_rx_cec &hdmim1_rx_hpdin &hdmim1_rx_scl &hdmim1_rx_sda &hdmirx_5v_detection>;
       pinctrl-names = "default";
       hpd-gpios = <&gpio1 22 GPIO_ACTIVE_LOW>;
+      #sound-dai-cells = <1>;
     };
Signature · 1 line
2.53.0
#5 lore raw

PATCH v4 2/4 media: synopsys: hdmirx: add HDMI audio capture support

In reply to Igor Paunovic

The Synopsys DesignWare HDMI RX controller extracts the audio stream embedded in the incoming HDMI signal and feeds it to an on-SoC I2S controller. Expose it as an ALSA capture device by registering the generic hdmi-codec as a child of the controller, so that a simple-audio-card in the device tree can bind the HDMI RX audio DAI.

The sample rate is recovered from the ACR N/CTS values together with the measured TMDS character rate. A periodic worker keeps the local audio reference clock locked to the source by nudging it in small ppm steps to hold the audio FIFO fill level near its target, which avoids FIFO under/overflow and the resulting dropped samples.

Across system suspend the worker is stopped before the controller clocks are gated. Resume fully resets the controller, wiping the audio configuration, so when a capture stream is active it re-programs the audio path from the last known sample rate and re-arms the worker, whose rate change and FIFO error paths resynchronize once the source delivers audio again.

---
Changes in v4:
- use swab32() for the ACR word reversal: readl() already abstracts bus endianness, so the reversal must be unconditional (be32_to_cpu is a no-op on big-endian)
- support system suspend: stop the worker in suspend before the clocks are gated; resume fully resets the controller, so re-program the audio path and re-arm the worker while a capture stream is active, through a hdmirx_audio_setup() helper shared with hw_params() (Dmitry Osipenko; validated with an active capture stream across suspend/resume)

Changes in v3:
- restore the v1 audio teardown in remove(): audio_shutdown() already stops the worker when the stream closes, so the extra flag clear and trailing cancel added in v2 were redundant (Dmitry Osipenko)
- rename the ACR read locals and add a comment documenting the register byte packing
- drop the get_dai_id stub so OF-graph cards resolve the DAI index from the reg property

Changes in v2:
- register the S/PDIF DAI so the indexes match the binding and reject it with -EOPNOTSUPP until wired up (Sebastian Reichel)
- use platform_device_register_data() and drop the fixed 32-bit DMA mask (Dmitry Osipenko)
- don't leave an ERR_PTR in audio_pdev on registration failure
- fix teardown ordering in remove()
- stop the worker before reprogramming shared state in hw_params()
- look up the "audio" clock by name instead of indexing clks[1]
- keep the worker on system_unbound_wq when re-arming

 .../platform/synopsys/hdmirx/snps_hdmirx.c    | 306 ++++++++++++++++++
 .../platform/synopsys/hdmirx/snps_hdmirx.h    |   8 +
 2 files changed, 314 insertions(+)
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +306 -0
index 9cceffa..99ad621 100644
@@ -41,6 +41,8 @@
 #include <media/videobuf2-dma-contig.h>
 #include <media/videobuf2-v4l2.h>
 
+#include <sound/hdmi-codec.h>
+
 #include "snps_hdmirx.h"
 #include "snps_hdmirx_cec.h"
 
@@ -132,6 +134,13 @@ struct snps_hdmirx_dev {
 	struct delayed_work delayed_work_hotplug;
 	struct delayed_work delayed_work_res_change;
 	struct hdmirx_cec *cec;
+	struct platform_device *audio_pdev;
+	struct clk *audio_clk;
+	struct delayed_work audio_work;
+	u32 audio_clkrate;
+	u32 audio_fs;
+	int audio_pre_state;
+	bool audio_streaming;
 	struct mutex phy_rw_lock; /* to protect phy r/w configuration */
 	struct mutex stream_lock; /* to lock video stream capture */
 	struct mutex work_lock; /* to lock the critical section of hotplug event */
@@ -2283,6 +2292,13 @@ static int hdmirx_parse_dt(struct snps_hdmirx_dev *hdmirx_dev)
 	if (hdmirx_dev->num_clks < 1)
 		return -ENODEV;
 
+	for (int i = 0; i < hdmirx_dev->num_clks; i++) {
+		if (!strcmp(hdmirx_dev->clks[i].id, "audio")) {
+			hdmirx_dev->audio_clk = hdmirx_dev->clks[i].clk;
+			break;
+		}
+	}
+
 	hdmirx_dev->resets[HDMIRX_RST_A].id = "axi";
 	hdmirx_dev->resets[HDMIRX_RST_P].id = "apb";
 	hdmirx_dev->resets[HDMIRX_RST_REF].id = "ref";
@@ -2527,10 +2543,19 @@ static void hdmirx_enable_irq(struct device *dev)
 			   msecs_to_jiffies(110));
 }
 
+static void hdmirx_audio_setup(struct snps_hdmirx_dev *hdmirx_dev, u32 fs);
+
 static __maybe_unused int hdmirx_suspend(struct device *dev)
 {
 	struct snps_hdmirx_dev *hdmirx_dev = dev_get_drvdata(dev);
 
+	/*
+	 * Stop the audio worker before the controller clocks are gated;
+	 * the audio path is restored and the worker re-armed from
+	 * resume() while a capture stream is active.
+	 */
+	cancel_delayed_work_sync(&hdmirx_dev->audio_work);
+
 	hdmirx_disable_irq(dev);
 
 	/* TODO store CEC HW state */
@@ -2553,6 +2578,19 @@ static __maybe_unused int hdmirx_resume(struct device *dev)
 		hdmirx_hpd_ctrl(hdmirx_dev, true);
 	}
 
+	/*
+	 * hdmirx_enable() fully reset the controller, wiping the audio
+	 * configuration. If a capture stream is active across suspend,
+	 * re-program the audio path with the last known sample rate and
+	 * restart the worker; its rate change and FIFO error paths
+	 * resynchronize once the source delivers audio again.
+	 */
+	if (READ_ONCE(hdmirx_dev->audio_streaming)) {
+		hdmirx_audio_setup(hdmirx_dev, hdmirx_dev->audio_fs);
+		mod_delayed_work(system_unbound_wq, &hdmirx_dev->audio_work,
+				 msecs_to_jiffies(200));
+	}
+
 	/* TODO restore CEC HW state */
 	enable_irq(hdmirx_dev->cec->irq);
 
@@ -2650,6 +2688,266 @@ static int hdmirx_register_cec(struct snps_hdmirx_dev *hdmirx_dev,
 	return 0;
 }
 
+#define HDMIRX_AUDIO_INIT_FIFO_STATE	128
+#define HDMIRX_AUDIO_INIT_STATE		(HDMIRX_AUDIO_INIT_FIFO_STATE * 4)
+
+static const int hdmirx_supported_fs[] = {
+	32000, 44100, 48000, 88200, 96000, 176400, 192000, 768000, -1
+};
+
+static int hdmirx_audio_closest_fs(int fs)
+{
+	int i = 0, fs_t = hdmirx_supported_fs[0];
+
+	while (fs_t > 0) {
+		if (abs(fs - fs_t) <= 2000)
+			return fs_t;
+		fs_t = hdmirx_supported_fs[++i];
+	}
+	return 0;
+}
+
+/* Recover the incoming audio sample rate from the ACR N/CTS + TMDS clock. */
+static u32 hdmirx_audio_fs(struct snps_hdmirx_dev *hdmirx_dev)
+{
+	u64 tmds_clk, fs_audio = 0;
+	u32 acr_cts, acr_n, tmdsqpclk_freq;
+	u32 acr_pb3_0, acr_pb7_4;
+
+	tmdsqpclk_freq = hdmirx_readl(hdmirx_dev, CMU_TMDSQPCLK_FREQ);
+	hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PH2_1);
+	acr_pb3_0 = hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PB3_0);
+	acr_pb7_4 = hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PB7_4);
+	/*
+	 * The packet decoder stores the ACR subpacket bytes with packet byte
+	 * 0 in register bits [7:0], so byte-reverse each word to line the
+	 * bytes up: CTS is packet bytes 1-3 (PKTDEC_ACR_PB3_0) and N is
+	 * packet bytes 4-6 (PKTDEC_ACR_PB7_4), 20 bits each. readl()
+	 * already abstracts the bus endianness, so the reversal is
+	 * unconditional.
+	 */
+	acr_cts = swab32(acr_pb3_0) & 0xfffff;
+	acr_n = (swab32(acr_pb7_4) & 0x0fffff00) >> 8;
+	tmds_clk = tmdsqpclk_freq * 4 * 1000U;
+	if (acr_cts != 0) {
+		fs_audio = div_u64((tmds_clk * acr_n), acr_cts);
+		fs_audio /= 128;
+		fs_audio = hdmirx_audio_closest_fs(fs_audio);
+	}
+	return (u32)fs_audio;
+}
+
+/* Nudge the audio reference clock by +/- ppm to keep the FIFO balanced. */
+static void hdmirx_audio_clk_ppm_inc(struct snps_hdmirx_dev *hdmirx_dev, int ppm)
+{
+	int delta, inc;
+	long rate = hdmirx_dev->audio_clkrate;
+
+	if (ppm < 0) {
+		ppm = -ppm;
+		inc = -1;
+	} else {
+		inc = 1;
+	}
+	delta = (int)div64_u64((u64)rate * ppm + 500000, 1000000);
+	delta *= inc;
+	rate = hdmirx_dev->audio_clkrate + delta;
+	clk_set_rate(hdmirx_dev->audio_clk, rate);
+	hdmirx_dev->audio_clkrate = rate;
+}
+
+static int hdmirx_audio_clk_adjust(struct snps_hdmirx_dev *hdmirx_dev,
+				   int total_offset, int single_offset)
+{
+	int schedule_time = 500;
+	int ppm = 10;
+	u32 offset_abs = abs(total_offset);
+
+	if (offset_abs > 200) {
+		ppm += 200;
+		schedule_time -= 100;
+	}
+	if (offset_abs > 100) {
+		ppm += 200;
+		schedule_time -= 100;
+	}
+	if (offset_abs > 32) {
+		ppm += 20;
+		schedule_time -= 100;
+	}
+	if (offset_abs > 16)
+		ppm += 20;
+	if (total_offset > 16 && single_offset > 0)
+		hdmirx_audio_clk_ppm_inc(hdmirx_dev, ppm);
+	else if (total_offset < -16 && single_offset < 0)
+		hdmirx_audio_clk_ppm_inc(hdmirx_dev, -ppm);
+	return schedule_time;
+}
+
+static void hdmirx_audio_fifo_reinit(struct snps_hdmirx_dev *hdmirx_dev)
+{
+	hdmirx_writel(hdmirx_dev, AUDIO_FIFO_CONTROL, 1);
+	usleep_range(200, 210);
+	hdmirx_writel(hdmirx_dev, AUDIO_FIFO_CONTROL, 0);
+}
+
+/*
+ * Program the audio clock, FIFO thresholds and enables for the given
+ * sample rate. Shared by hw_params and system resume: the controller is
+ * fully reset on resume, so the whole configuration must be re-applied.
+ */
+static void hdmirx_audio_setup(struct snps_hdmirx_dev *hdmirx_dev, u32 fs)
+{
+	hdmirx_dev->audio_fs = fs;
+	hdmirx_dev->audio_clkrate = fs * 128;
+	clk_set_rate(hdmirx_dev->audio_clk, hdmirx_dev->audio_clkrate);
+
+	hdmirx_audio_fifo_reinit(hdmirx_dev);
+	hdmirx_writel(hdmirx_dev, AUDIO_FIFO_THR_PASS, HDMIRX_AUDIO_INIT_FIFO_STATE);
+	hdmirx_writel(hdmirx_dev, AUDIO_FIFO_THR,
+		      AFIFO_THR_LOW_QST(0x20) | AFIFO_THR_HIGH_QST(0x160));
+	hdmirx_writel(hdmirx_dev, AUDIO_FIFO_MUTE_THR,
+		      AFIFO_THR_MUTE_LOW_QST(0x8) | AFIFO_THR_MUTE_HIGH_QST(0x178));
+
+	hdmirx_update_bits(hdmirx_dev, AUDIO_PROC_CONFIG0, I2S_EN, I2S_EN);
+	hdmirx_update_bits(hdmirx_dev, GLOBAL_SWENABLE, AUDIO_ENABLE, AUDIO_ENABLE);
+
+	hdmirx_dev->audio_pre_state = 0;
+}
+
+/*
+ * Periodic worker that locks the local audio clock to the source by keeping
+ * the audio FIFO fill level close to its target, avoiding under/overflow.
+ */
+static void hdmirx_audio_work(struct work_struct *work)
+{
+	struct snps_hdmirx_dev *hdmirx_dev =
+		container_of(to_delayed_work(work), struct snps_hdmirx_dev, audio_work);
+	unsigned long delay = 200;
+	int cur, total, single;
+	u32 fifo, fs;
+
+	fs = hdmirx_audio_fs(hdmirx_dev);
+	fifo = hdmirx_readl(hdmirx_dev, AUDIO_FIFO_STATUS2);
+
+	if (fifo & (AFIFO_UNDERFLOW_ST | AFIFO_OVERFLOW_ST)) {
+		if (fs) {
+			clk_set_rate(hdmirx_dev->audio_clk, fs * 128);
+			hdmirx_dev->audio_clkrate = fs * 128;
+			hdmirx_dev->audio_fs = fs;
+		}
+		hdmirx_audio_fifo_reinit(hdmirx_dev);
+		hdmirx_dev->audio_pre_state = 0;
+		goto out;
+	}
+
+	cur = fifo & 0xffff;
+	total = cur - HDMIRX_AUDIO_INIT_STATE;
+	single = cur - hdmirx_dev->audio_pre_state;
+
+	if (fs && abs((int)fs - (int)hdmirx_dev->audio_fs) > 1000) {
+		clk_set_rate(hdmirx_dev->audio_clk, fs * 128);
+		hdmirx_dev->audio_clkrate = fs * 128;
+		hdmirx_dev->audio_fs = fs;
+		hdmirx_audio_fifo_reinit(hdmirx_dev);
+		hdmirx_dev->audio_pre_state = 0;
+		goto out;
+	}
+
+	if (cur != 0)
+		delay = hdmirx_audio_clk_adjust(hdmirx_dev, total, single);
+	hdmirx_dev->audio_pre_state = cur;
+out:
+	/* Only re-arm while streaming; avoids a self-reschedule race with
+	 * the cancel_delayed_work_sync() callers (hw_params and
+	 * audio_shutdown).
+	 */
+	if (READ_ONCE(hdmirx_dev->audio_streaming))
+		queue_delayed_work(system_unbound_wq, &hdmirx_dev->audio_work,
+				   msecs_to_jiffies(delay));
+}
+
+static int hdmirx_audio_hw_params(struct device *dev, void *data,
+				  struct hdmi_codec_daifmt *fmt,
+				  struct hdmi_codec_params *hparms)
+{
+	struct snps_hdmirx_dev *hdmirx_dev = dev_get_drvdata(dev);
+	u32 fs;
+
+	/* Only the I2S interface (DAI 0) is wired up so far. */
+	if (fmt->fmt == HDMI_SPDIF)
+		return -EOPNOTSUPP;
+
+	/*
+	 * Stop the worker before touching the shared audio state; it is
+	 * re-armed below once the new parameters are in place.
+	 */
+	WRITE_ONCE(hdmirx_dev->audio_streaming, false);
+	cancel_delayed_work_sync(&hdmirx_dev->audio_work);
+
+	fs = hdmirx_audio_fs(hdmirx_dev);
+	if (!fs)
+		fs = hparms ? hparms->sample_rate : 48000;
+	if (!fs)
+		fs = 48000;
+
+	hdmirx_audio_setup(hdmirx_dev, fs);
+
+	WRITE_ONCE(hdmirx_dev->audio_streaming, true);
+	mod_delayed_work(system_unbound_wq, &hdmirx_dev->audio_work,
+			 msecs_to_jiffies(200));
+
+	dev_dbg(dev, "audio hw_params: fs=%u\n", fs);
+	return 0;
+}
+
+static void hdmirx_audio_shutdown(struct device *dev, void *data)
+{
+	struct snps_hdmirx_dev *hdmirx_dev = dev_get_drvdata(dev);
+
+	WRITE_ONCE(hdmirx_dev->audio_streaming, false);
+	cancel_delayed_work_sync(&hdmirx_dev->audio_work);
+	hdmirx_update_bits(hdmirx_dev, GLOBAL_SWENABLE, AUDIO_ENABLE, 0);
+}
+
+static const struct hdmi_codec_ops hdmirx_audio_codec_ops = {
+	.hw_params = hdmirx_audio_hw_params,
+	.audio_shutdown = hdmirx_audio_shutdown,
+};
+
+static int hdmirx_register_audio_device(struct snps_hdmirx_dev *hdmirx_dev)
+{
+	struct hdmi_codec_pdata codec_data = {
+		.ops = &hdmirx_audio_codec_ops,
+		.i2s = 1,
+		.no_i2s_playback = 1,
+		.max_i2s_channels = 8,
+		/*
+		 * The controller also has an S/PDIF audio interface (DAI 1 in
+		 * the binding). Register it so DAI indexes match the binding,
+		 * but reject its use in hw_params() until it is wired up.
+		 */
+		.spdif = 1,
+		.no_spdif_playback = 1,
+		.data = hdmirx_dev,
+	};
+	struct platform_device *audio_pdev;
+
+	if (!hdmirx_dev->audio_clk)
+		return -ENODEV;
+
+	audio_pdev = platform_device_register_data(hdmirx_dev->dev,
+						   HDMI_CODEC_DRV_NAME,
+						   PLATFORM_DEVID_AUTO,
+						   &codec_data, sizeof(codec_data));
+	if (IS_ERR(audio_pdev))
+		return PTR_ERR(audio_pdev);
+
+	hdmirx_dev->audio_pdev = audio_pdev;
+
+	return 0;
+}
+
 static int hdmirx_probe(struct platform_device *pdev)
 {
 	struct snps_hdmirx_dev *hdmirx_dev;
@@ -2701,6 +2999,7 @@ static int hdmirx_probe(struct platform_device *pdev)
 			  hdmirx_delayed_work_hotplug);
 	INIT_DELAYED_WORK(&hdmirx_dev->delayed_work_res_change,
 			  hdmirx_delayed_work_res_change);
+	INIT_DELAYED_WORK(&hdmirx_dev->audio_work, hdmirx_audio_work);
 
 	hdmirx_dev->cur_fmt_fourcc = V4L2_PIX_FMT_BGR24;
 	hdmirx_dev->timings = cea640x480;
@@ -2769,6 +3068,10 @@ static int hdmirx_probe(struct platform_device *pdev)
 						       V4L2_DEBUGFS_IF_AVI, hdmirx_dev,
 						       hdmirx_debugfs_if_read);
 
+	ret = hdmirx_register_audio_device(hdmirx_dev);
+	if (ret)
+		dev_warn(dev, "failed to register HDMI audio codec: %d\n", ret);
+
 	return 0;
 
 err_unreg_video_dev:
@@ -2788,6 +3091,9 @@ static void hdmirx_remove(struct platform_device *pdev)
 	struct device *dev = &pdev->dev;
 	struct snps_hdmirx_dev *hdmirx_dev = dev_get_drvdata(dev);
 
+	if (hdmirx_dev->audio_pdev)
+		platform_device_unregister(hdmirx_dev->audio_pdev);
+
 	v4l2_debugfs_if_free(hdmirx_dev->infoframes);
 	debugfs_remove_recursive(hdmirx_dev->debugfs_dir);
 
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.h +8 -0
index 31b887e..a99f54f 100644
@@ -81,6 +81,7 @@
 #define DATAPATH_ENABLE				BIT(12)
 #define PKTFIFO_ENABLE				BIT(11)
 #define AVPUNIT_ENABLE				BIT(8)
+#define AUDIO_ENABLE				BIT(9)
 #define MAIN_ENABLE				BIT(0)
 #define GLOBAL_TIMER_REF_BASE			0x0028
 #define CORE_CONFIG				0x0050
@@ -177,20 +178,27 @@
 #define VPROC_FMT_OVR_VALUE(x)			UPDATE(x, 6, 4)
 #define VPROC_FMT_OVR_EN			BIT(0)
 
+#define AUDIO_FIFO_CONFIG			0x0460
 #define AFIFO_FILL_RESTART			BIT(0)
+#define AUDIO_FIFO_CONTROL			0x0464
 #define AFIFO_INIT_P				BIT(0)
+#define AUDIO_FIFO_THR_PASS			0x0468
+#define AUDIO_FIFO_THR				0x046c
 #define AFIFO_THR_LOW_QST_MASK			GENMASK(25, 16)
 #define AFIFO_THR_LOW_QST(x)			UPDATE(x, 25, 16)
 #define AFIFO_THR_HIGH_QST_MASK			GENMASK(9, 0)
 #define AFIFO_THR_HIGH_QST(x)			UPDATE(x, 9, 0)
+#define AUDIO_FIFO_MUTE_THR			0x0470
 #define AFIFO_THR_MUTE_LOW_QST_MASK		GENMASK(25, 16)
 #define AFIFO_THR_MUTE_LOW_QST(x)		UPDATE(x, 25, 16)
 #define AFIFO_THR_MUTE_HIGH_QST_MASK		GENMASK(9, 0)
 #define AFIFO_THR_MUTE_HIGH_QST(x)		UPDATE(x, 9, 0)
 
+#define AUDIO_FIFO_STATUS2			0x0478
 #define AFIFO_UNDERFLOW_ST			BIT(25)
 #define AFIFO_OVERFLOW_ST			BIT(24)
 
+#define AUDIO_PROC_CONFIG0			0x0480
 #define SPEAKER_ALLOC_OVR_EN			BIT(16)
 #define I2S_BPCUV_EN				BIT(4)
 #define SPDIF_EN				BIT(2)
Signature · 1 line
2.53.0
#10 lore raw

PATCH v4 3/4 arm64: dts: rockchip: add HDMI RX audio on RK3588

In reply to Igor Paunovic

The Synopsys HDMI RX controller exposes two digital audio interfaces, one for I2S (0) and one for S/PDIF (1). Add the #sound-dai-cells property so audio cards can reference them as a codec, as documented by the snps,dw-hdmi-rx binding.

Add a shared simple-audio-card routing the receiver I2S DAI to i2s7_8ch, the receive-only I2S interface dedicated to HDMI RX, with the receiver as bitclock and frame master. The card is named "RK3588 HDMI-IN", which is what users see in alsamixer and what ALSA UCM matches on. It mirrors the existing HDMI TX sound cards in this file and is disabled by default, since not every RK3588 board routes HDMI RX audio; boards that do only need to flip its status.

---
Changes in v4 (both suggested by Sebastian Reichel):
- define the whole sound card here as a shared, disabled-by-default node mirroring the HDMI TX cards, instead of per board
- name the card "RK3588 HDMI-IN": the name is user-visible in alsamixer and matched by ALSA UCM

New in v3 (as "add #sound-dai-cells to the RK3588 HDMI receiver").

 arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +18 -0
index a264001..e0dce15 100644
@@ -23,6 +23,23 @@
 		};
 	};
 
+	hdmi_receiver_sound: hdmi-receiver-sound {
+		compatible = "simple-audio-card";
+		simple-audio-card,format = "i2s";
+		simple-audio-card,name = "RK3588 HDMI-IN";
+		simple-audio-card,bitclock-master = <&hdmiin_codec>;
+		simple-audio-card,frame-master = <&hdmiin_codec>;
+		status = "disabled";
+
+		hdmiin_codec: simple-audio-card,codec {
+			sound-dai = <&hdmi_receiver 0>;
+		};
+
+		simple-audio-card,cpu {
+			sound-dai = <&i2s7_8ch>;
+		};
+	};
+
 	reserved-memory {
 		#address-cells = <2>;
 		#size-cells = <2>;
@@ -338,6 +355,7 @@
 		reset-names = "axi", "apb", "ref", "biu";
 		rockchip,grf = <&sys_grf>;
 		rockchip,vo1-grf = <&vo1_grf>;
+		#sound-dai-cells = <1>;
 		status = "disabled";
 	};
 
Signature · 1 line
2.53.0
#14 lore raw

PATCH v4 4/4 arm64: dts: rockchip: enable HDMI RX audio capture on Orange Pi 5 Plus

In reply to Igor Paunovic

Enable the shared HDMI RX sound card and the i2s7_8ch interface it uses. Together with the audio capture support in the snps_hdmirx driver this exposes a capture-only ALSA card fed by the HDMI input.

Tested on the Orange Pi 5 Plus with multiple HDMI sources: capture follows the source sample rate and stays in sync via the FIFO-level clock tracking in the driver.

---
Changes in v4:
- only enable the shared hdmi_receiver_sound card and i2s7_8ch, now that the card definition lives in rk3588-extra.dtsi (Sebastian Reichel)

New in v3.

 arch/arm64/boot/dts/rockchip/rk3588-orangepi-5-plus.dts | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-orangepi-5-plus.dts +8 -0
index 9950d11..9f9a303 100644
@@ -171,6 +171,10 @@
 	status = "okay";
 };
 
+&hdmi_receiver_sound {
+	status = "okay";
+};
+
 &hdptxphy0 {
 	status = "okay";
 };
@@ -247,6 +251,10 @@
 	status = "okay";
 };
 
+&i2s7_8ch {
+	status = "okay";
+};
+
 &led_blue_gpio {
 	gpios = <&gpio3 RK_PA6 GPIO_ACTIVE_HIGH>;
 	status = "okay";
Signature · 1 line
2.53.0
#11 lore raw

Re: [PATCH v4 3/4] arm64: dts: rockchip: add HDMI RX audio on RK3588

In reply to Igor Paunovic
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but the DAPM widget for the HDMI RX capture source is incorrectly defined as an OUTPUT widget, breaking the DAPM graph for the capture path. --

--- Patch [3]: [PATCH v4 3/4] arm64: dts: rockchip: add HDMI RX audio on RK3588 --- commit e5a9a8ba89a4a45c730c1b44e28e899087346b1f
Author: Igor Paunovic <[email protected]>

arm64: dts: rockchip: add HDMI RX audio on RK3588

This commit exposes the HDMI RX controller's digital audio interfaces by adding the #sound-dai-cells property. It also adds a shared simple-audio-card routing the receiver I2S DAI to i2s7_8ch, mirroring existing HDMI TX sound cards.

Quoted 4 lines
arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +0 -0
index a2640014ee042..e0dce15643997 100644

[ ... ]

Quoted 5 lines
arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +1 -0
@@ -23,6 +23,23 @@ simple-audio-card,cpu {
 		};
 	};
 
+	hdmi_receiver_sound: hdmi-receiver-sound {

arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi simple-audio-card,cpu { · line 36

Quoted 10 lines
arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +10 -0
+		compatible = "simple-audio-card";
+		simple-audio-card,format = "i2s";
+		simple-audio-card,name = "RK3588 HDMI-IN";
+		simple-audio-card,bitclock-master = <&hdmiin_codec>;
+		simple-audio-card,frame-master = <&hdmiin_codec>;
+		status = "disabled";
+
+		hdmiin_codec: simple-audio-card,codec {
+			sound-dai = <&hdmi_receiver 0>;
+		};

[Severity: High]
This isn't a bug introduced by this patch, but will this HDMI RX capture path work correctly with DAPM given how the underlying codec is currently defined?

Looking at the hdmi-codec driver used for this receiver:

sound/soc/codecs/hdmi-codec.c:hdmi_widgets[] {
    ...
    SND_SOC_DAPM_OUTPUT("RX"),
    ...
}

The DAPM widget for the HDMI RX capture source is incorrectly defined as an OUTPUT (sink) instead of an INPUT (source).

When a capture stream starts, DAPM walks backward from the active CPU DAI sink to find a source, but terminates at "RX" without finding an EP_SOURCE. Does this prevent DAPM-managed components on the capture path (such as SoC FIFOs, regulators, or clock supplies) from powering on, causing silent failures during capture?

Quoted 5 lines
arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +5 -0
+
+		simple-audio-card,cpu {
+			sound-dai = <&i2s7_8ch>;
+		};
+	};
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#6 lore raw
In reply to Igor Paunovic
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] The use of swab32() when extracting the 20-bit ACR N and CTS values incorrectly scrambles their bits. Because the packet bytes are stored contiguously from the LSB up (PB0 in [7:0], PB1 in [15:8], etc.), reversing the 32-bit word using swab32() places the LSB of CTS (PB1) into the MSB of the extracted value, and the MSB of CTS (PB3) into the LSB. Masking this result produces a completely invalid sample rate.

Pre-existing issues:
- [High] The V4L2 video_device is embedded in struct snps_hdmirx_dev, which is allocated using devm_kzalloc. The driver sets vdev.release = video_device_release_empty. If a user space process keeps the /dev/videoX file descriptor open during device unbind, the devm subsystem will free the snps_hdmirx_dev memory. When the process later closes the file descriptor, the V4L2 core will access the freed video_device to decrement its refcount and call the release callback, causing a use-after-free vulnerability. --

--- Patch [2]: [PATCH v4 2/4] media: synopsys: hdmirx: add HDMI audio capture support --- commit 19c9c4db1394f1dc0fc1cb44d51317198a87b078

Author: Igor Paunovic <[email protected]>
Date:   Tue Jul 21 06:41:13 2026 +0000

media: synopsys: hdmirx: add HDMI audio capture support

This patch adds HDMI audio capture support to the Synopsys DesignWare HDMI RX controller. It extracts the embedded audio stream and exposes it as an ALSA capture device via the generic hdmi-codec. It also introduces a periodic worker to keep the local audio reference clock locked to the source.

Quoted 4 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +0 -0
index 25f8ca0d6d946..1b4a7fda9e195 100644

[ ... ]

Quoted 10 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +10 -0
+static u32 hdmirx_audio_fs(struct snps_hdmirx_dev *hdmirx_dev)
+{
+	u64 tmds_clk, fs_audio = 0;
+	u32 acr_cts, acr_n, tmdsqpclk_freq;
+	u32 acr_pb3_0, acr_pb7_4;
+
+	tmdsqpclk_freq = hdmirx_readl(hdmirx_dev, CMU_TMDSQPCLK_FREQ);
+	hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PH2_1);
+	acr_pb3_0 = hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PB3_0);
+	acr_pb7_4 = hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PB7_4);

drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c static int hdmirx_register_cec(struct snps_hdmirx_dev *hdmirx_dev, · line 2730

Quoted 10 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +10 -0
+	/*
+	 * The packet decoder stores the ACR subpacket bytes with packet byte
+	 * 0 in register bits [7:0], so byte-reverse each word to line the
+	 * bytes up: CTS is packet bytes 1-3 (PKTDEC_ACR_PB3_0) and N is
+	 * packet bytes 4-6 (PKTDEC_ACR_PB7_4), 20 bits each. readl()
+	 * already abstracts the bus endianness, so the reversal is
+	 * unconditional.
+	 */
+	acr_cts = swab32(acr_pb3_0) & 0xfffff;
+	acr_n = (swab32(acr_pb7_4) & 0x0fffff00) >> 8;

[Severity: High]
Does this byte reversal using swab32() incorrectly scramble the N and CTS values?

Because the packet bytes are stored contiguously from the LSB up (with packet byte 0 in [7:0], packet byte 1 in [15:8], etc.), reversing the 32-bit word using swab32() places the LSB of CTS (from PB1) into the MSB of the extracted value, and the MSB of CTS (from PB3) into the LSB.

Will this result in an invalid sample rate being calculated, causing the driver's fallback logic to silently ignore the broken rate and use 48kHz? This could cause any audio stream not natively at 48kHz to be played back at the wrong rate.

Quoted 3 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +3 -0
+	tmds_clk = tmdsqpclk_freq * 4 * 1000U;
+	if (acr_cts != 0) {
+		fs_audio = div_u64((tmds_clk * acr_n), acr_cts);

[ ... ]

drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c static int hdmirx_probe(struct platform_device *pdev) · line 3071

Quoted 9 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +4 -0
@@ -2765,6 +3064,10 @@ static int hdmirx_probe(struct platform_device *pdev)
 						       V4L2_DEBUGFS_IF_AVI, hdmirx_dev,
 						       hdmirx_debugfs_if_read);
 
+	ret = hdmirx_register_audio_device(hdmirx_dev);
+	if (ret)
+		dev_warn(dev, "failed to register HDMI audio codec: %d\n", ret);
+
 	return 0;

[Severity: High]
This isn't a bug introduced by this patch, but is there a use-after-free risk during device unbind in this driver?

The V4L2 video_device is embedded in struct snps_hdmirx_dev, which is allocated using devm_kzalloc() earlier in hdmirx_probe(). However, the driver sets the release callback to video_device_release_empty in hdmirx_register_stream_vdev().

If a user space process keeps the /dev/videoX file descriptor open during device unbind (such as during driver unbind or module removal), the devm subsystem will free the snps_hdmirx_dev memory. When the process later closes the file descriptor, will the V4L2 core access the freed video_device memory to decrement its refcount and call the release callback?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#7 lore raw
Quoted 2 lines

Does this byte reversal using swab32() incorrectly scramble the N and CTS values?

No. This is the third review of this finding -- it was raised on v2 and v3 in the be32_to_cpu() spelling and answered both times, and the comment above the code documents the layout. The premise doesn't match the ACR subpacket format: CTS is not stored LSB-first from PB1. Per the HDMI spec, subpacket byte 0 is reserved, CTS[19:16] sits in PB1[3:0], CTS[15:8] in PB2 and CTS[7:0] in PB3 -- MSB first -- with N likewise in PB4..PB6. The packet decoder stores PB0 in register bits [7:0], so reversing the word is precisely what lines the bytes up before the 20-bit masks. swab32() and the previous be32_to_cpu()/cpu_to_be32 spelling produce the same value on both endiannesses; the v4 change only makes the reversal unconditional.

It also remains the path validated on hardware, as noted in the cover letter: a 44.1 kHz source is detected as 44.1 kHz rather than falling back to the 48 kHz default, and 44.1/48 kHz switches are followed. Neither would work if the extraction produced garbage.

Quoted 3 lines

is there a use-after-free risk during device unbind in this driver? [video_device embedded in devm-allocated struct, release set to video_device_release_empty]

This one looks structurally plausible, but as the report itself notes it predates this series: it concerns the video_device lifetime on the V4L2 side and is independent of the audio support added here (the audio teardown path was exercised on unbind with an active stream during v3 review, see the v3 thread). Addressing it would be a separate fix for the existing driver -- happy to leave that call to Dmitry.

Igor

Quoted 97 lines · On Tue, Jul 21, 2026 at 9:00 AM <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] The use of swab32() when extracting the 20-bit ACR N and CTS values incorrectly scrambles their bits. Because the packet bytes are stored contiguously from the LSB up (PB0 in [7:0], PB1 in [15:8], etc.), reversing the 32-bit word using swab32() places the LSB of CTS (PB1) into the MSB of the extracted value, and the MSB of CTS (PB3) into the LSB. Masking this result produces a completely invalid sample rate.

Pre-existing issues:
- [High] The V4L2 video_device is embedded in struct snps_hdmirx_dev, which is allocated using devm_kzalloc. The driver sets vdev.release = video_device_release_empty. If a user space process keeps the /dev/videoX file descriptor open during device unbind, the devm subsystem will free the snps_hdmirx_dev memory. When the process later closes the file descriptor, the V4L2 core will access the freed video_device to decrement its refcount and call the release callback, causing a use-after-free vulnerability. --

--- Patch [2]: [PATCH v4 2/4] media: synopsys: hdmirx: add HDMI audio capture support --- commit 19c9c4db1394f1dc0fc1cb44d51317198a87b078

Author: Igor Paunovic <[email protected]>
Date:   Tue Jul 21 06:41:13 2026 +0000

media: synopsys: hdmirx: add HDMI audio capture support

This patch adds HDMI audio capture support to the Synopsys DesignWare HDMI RX controller. It extracts the embedded audio stream and exposes it as an ALSA capture device via the generic hdmi-codec. It also introduces a periodic worker to keep the local audio reference clock locked to the source.

Quoted 4 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +0 -0
index 25f8ca0d6d946..1b4a7fda9e195 100644

[ ... ]

Quoted 10 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +10 -0
+static u32 hdmirx_audio_fs(struct snps_hdmirx_dev *hdmirx_dev)
+{
+     u64 tmds_clk, fs_audio = 0;
+     u32 acr_cts, acr_n, tmdsqpclk_freq;
+     u32 acr_pb3_0, acr_pb7_4;
+
+     tmdsqpclk_freq = hdmirx_readl(hdmirx_dev, CMU_TMDSQPCLK_FREQ);
+     hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PH2_1);
+     acr_pb3_0 = hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PB3_0);
+     acr_pb7_4 = hdmirx_readl(hdmirx_dev, PKTDEC_ACR_PB7_4);

drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c static int hdmirx_register_cec(struct snps_hdmirx_dev *hdmirx_dev, · line 2730

Quoted 10 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +10 -0
+     /*
+      * The packet decoder stores the ACR subpacket bytes with packet byte
+      * 0 in register bits [7:0], so byte-reverse each word to line the
+      * bytes up: CTS is packet bytes 1-3 (PKTDEC_ACR_PB3_0) and N is
+      * packet bytes 4-6 (PKTDEC_ACR_PB7_4), 20 bits each. readl()
+      * already abstracts the bus endianness, so the reversal is
+      * unconditional.
+      */
+     acr_cts = swab32(acr_pb3_0) & 0xfffff;
+     acr_n = (swab32(acr_pb7_4) & 0x0fffff00) >> 8;

[Severity: High]
Does this byte reversal using swab32() incorrectly scramble the N and CTS values?

Because the packet bytes are stored contiguously from the LSB up (with packet byte 0 in [7:0], packet byte 1 in [15:8], etc.), reversing the 32-bit word using swab32() places the LSB of CTS (from PB1) into the MSB of the extracted value, and the MSB of CTS (from PB3) into the LSB.

Will this result in an invalid sample rate being calculated, causing the driver's fallback logic to silently ignore the broken rate and use 48kHz? This could cause any audio stream not natively at 48kHz to be played back at the wrong rate.

Quoted 3 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +3 -0
+     tmds_clk = tmdsqpclk_freq * 4 * 1000U;
+     if (acr_cts != 0) {
+             fs_audio = div_u64((tmds_clk * acr_n), acr_cts);

[ ... ]

drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c static int hdmirx_probe(struct platform_device *pdev) · line 3071

Quoted 9 lines
drivers/media/platform/synopsys/hdmirx/snps_hdmirx.c +4 -0
@@ -2765,6 +3064,10 @@ static int hdmirx_probe(struct platform_device *pdev)
                                                     V4L2_DEBUGFS_IF_AVI, hdmirx_dev,
                                                     hdmirx_debugfs_if_read);
 
+     ret = hdmirx_register_audio_device(hdmirx_dev);
+     if (ret)
+             dev_warn(dev, "failed to register HDMI audio codec: %d\n", ret);
+
      return 0;

[Severity: High]
This isn't a bug introduced by this patch, but is there a use-after-free risk during device unbind in this driver?

The V4L2 video_device is embedded in struct snps_hdmirx_dev, which is allocated using devm_kzalloc() earlier in hdmirx_probe(). However, the driver sets the release callback to video_device_release_empty in hdmirx_register_stream_vdev().

If a user space process keeps the /dev/videoX file descriptor open during device unbind (such as during driver unbind or module removal), the devm subsystem will free the snps_hdmirx_dev memory. When the process later closes the file descriptor, will the V4L2 core access the freed video_device memory to decrement its refcount and call the release callback?

--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2

Reviewed-by on v4 1/4
#2 lore raw

Re: [PATCH v4 1/4] dt-bindings: media: snps,dw-hdmi-rx: add #sound-dai-cells

In reply to Igor Paunovic

Hi,

Quoted 6 lines · Igor Paunovic

The HDMI RX controller can expose the audio embedded in the incoming HDMI stream as an ALSA capture device. Document the #sound-dai-cells property so that a sound card can reference the HDMI RX audio DAI.

---

-- Sebastian

Quoted 55 lines

Changes in v4 (both suggested by Krzysztof Kozlowski):
- reference dai-common.yaml and switch additionalProperties to
unevaluatedProperties: false
- drop the consumer comment from the binding example

Changes in v3:
- no changes

Changes in v2 (all suggested by Sebastian Reichel):
- reworded the commit message to describe the hardware
- property description now documents the DAI indexes (0 = I2S, 1 = S/PDIF)
- moved the consumer reference into the binding example

 .../devicetree/bindings/media/snps,dw-hdmi-rx.yaml  | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
Documentation/devicetree/bindings/media/snps,dw-hdmi-rx.yaml +12 -1
index b7f6c87..b80660d 100644
@@ -78,6 +78,13 @@ properties:
       The phandle of the syscon node for the Video Output GRF register
       to enable EDID transfer through SDAIN and SCLIN.
 
+  "#sound-dai-cells":
+    const: 1
+    description:
+      The HDMI RX controller has two digital audio interfaces, one for
+      I2S and one for S/PDIF. The DAI cell selects the interface, 0 for
+      I2S and 1 for S/PDIF.
+
 required:
   - compatible
   - reg
@@ -90,7 +97,10 @@ required:
   - pinctrl-0
   - hpd-gpios
 
-additionalProperties: false
+allOf:
+  - $ref: /schemas/sound/dai-common.yaml#
+
+unevaluatedProperties: false
 
 examples:
   - |
@@ -129,4 +139,5 @@ examples:
       pinctrl-0 = <&hdmim1_rx_cec &hdmim1_rx_hpdin &hdmim1_rx_scl &hdmim1_rx_sda &hdmirx_5v_detection>;
       pinctrl-names = "default";
       hpd-gpios = <&gpio1 22 GPIO_ACTIVE_LOW>;
+      #sound-dai-cells = <1>;
     };
Signature · 1 line
2.53.0
Reviewed-by on v4 3/4
#12 lore raw

Re: [PATCH v4 3/4] arm64: dts: rockchip: add HDMI RX audio on RK3588

In reply to Igor Paunovic

Hi,

Quoted 13 lines · Igor Paunovic

The Synopsys HDMI RX controller exposes two digital audio interfaces, one for I2S (0) and one for S/PDIF (1). Add the #sound-dai-cells property so audio cards can reference them as a codec, as documented by the snps,dw-hdmi-rx binding.

Add a shared simple-audio-card routing the receiver I2S DAI to i2s7_8ch, the receive-only I2S interface dedicated to HDMI RX, with the receiver as bitclock and frame master. The card is named "RK3588 HDMI-IN", which is what users see in alsamixer and what ALSA UCM matches on. It mirrors the existing HDMI TX sound cards in this file and is disabled by default, since not every RK3588 board routes HDMI RX audio; boards that do only need to flip its status.

Quoted 2 lines

---

-- Sebastian

Quoted 49 lines

Changes in v4 (both suggested by Sebastian Reichel):
- define the whole sound card here as a shared, disabled-by-default node mirroring the HDMI TX cards, instead of per board
- name the card "RK3588 HDMI-IN": the name is user-visible in alsamixer and matched by ALSA UCM

New in v3 (as "add #sound-dai-cells to the RK3588 HDMI receiver").

 arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +18 -0
index a264001..e0dce15 100644
@@ -23,6 +23,23 @@
 		};
 	};
 
+	hdmi_receiver_sound: hdmi-receiver-sound {
+		compatible = "simple-audio-card";
+		simple-audio-card,format = "i2s";
+		simple-audio-card,name = "RK3588 HDMI-IN";
+		simple-audio-card,bitclock-master = <&hdmiin_codec>;
+		simple-audio-card,frame-master = <&hdmiin_codec>;
+		status = "disabled";
+
+		hdmiin_codec: simple-audio-card,codec {
+			sound-dai = <&hdmi_receiver 0>;
+		};
+
+		simple-audio-card,cpu {
+			sound-dai = <&i2s7_8ch>;
+		};
+	};
+
 	reserved-memory {
 		#address-cells = <2>;
 		#size-cells = <2>;
@@ -338,6 +355,7 @@
 		reset-names = "axi", "apb", "ref", "biu";
 		rockchip,grf = <&sys_grf>;
 		rockchip,vo1-grf = <&vo1_grf>;
+		#sound-dai-cells = <1>;
 		status = "disabled";
 	};
 
Signature · 1 line
2.53.0
Reviewed-by on v4 4/4
#15 lore raw

Re: [PATCH v4 4/4] arm64: dts: rockchip: enable HDMI RX audio capture on Orange Pi 5 Plus

In reply to Igor Paunovic

Hi,

Quoted 10 lines · Igor Paunovic

Enable the shared HDMI RX sound card and the i2s7_8ch interface it uses. Together with the audio capture support in the snps_hdmirx driver this exposes a capture-only ALSA card fed by the HDMI input.

Tested on the Orange Pi 5 Plus with multiple HDMI sources: capture follows the source sample rate and stays in sync via the FIFO-level clock tracking in the driver.

---

-- Sebastian

Quoted 38 lines

Changes in v4:
- only enable the shared hdmi_receiver_sound card and i2s7_8ch, now that the card definition lives in rk3588-extra.dtsi (Sebastian Reichel)

New in v3.

 arch/arm64/boot/dts/rockchip/rk3588-orangepi-5-plus.dts | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-orangepi-5-plus.dts +8 -0
index 9950d11..9f9a303 100644
@@ -171,6 +171,10 @@
 	status = "okay";
 };
 
+&hdmi_receiver_sound {
+	status = "okay";
+};
+
 &hdptxphy0 {
 	status = "okay";
 };
@@ -247,6 +251,10 @@
 	status = "okay";
 };
 
+&i2s7_8ch {
+	status = "okay";
+};
+
 &led_blue_gpio {
 	gpios = <&gpio3 RK_PA6 GPIO_ACTIVE_HIGH>;
 	status = "okay";
Signature · 1 line
2.53.0
Reviewed-by on v4 1/4
#3 lore raw

Re: [PATCH v4 1/4] dt-bindings: media: snps,dw-hdmi-rx: add #sound-dai-cells

In reply to Igor Paunovic
Quoted 5 lines · Igor Paunovic

The HDMI RX controller can expose the audio embedded in the incoming HDMI stream as an ALSA capture device. Document the #sound-dai-cells property so that a sound card can reference the HDMI RX audio DAI.

Best regards,
Krzysztof

maintainer Reviewed-by Tested-by on v4 2/4
#9 lore raw
In reply to Igor Paunovic
Quoted 51 lines · Igor Paunovic

The Synopsys DesignWare HDMI RX controller extracts the audio stream embedded in the incoming HDMI signal and feeds it to an on-SoC I2S controller. Expose it as an ALSA capture device by registering the generic hdmi-codec as a child of the controller, so that a simple-audio-card in the device tree can bind the HDMI RX audio DAI.

The sample rate is recovered from the ACR N/CTS values together with the measured TMDS character rate. A periodic worker keeps the local audio reference clock locked to the source by nudging it in small ppm steps to hold the audio FIFO fill level near its target, which avoids FIFO under/overflow and the resulting dropped samples.

Across system suspend the worker is stopped before the controller clocks are gated. Resume fully resets the controller, wiping the audio configuration, so when a capture stream is active it re-programs the audio path from the last known sample rate and re-arms the worker, whose rate change and FIFO error paths resynchronize once the source delivers audio again.

---
Changes in v4:
- use swab32() for the ACR word reversal: readl() already abstracts bus endianness, so the reversal must be unconditional (be32_to_cpu is a no-op on big-endian)
- support system suspend: stop the worker in suspend before the clocks are gated; resume fully resets the controller, so re-program the audio path and re-arm the worker while a capture stream is active, through a hdmirx_audio_setup() helper shared with hw_params() (Dmitry Osipenko; validated with an active capture stream across suspend/resume)

Changes in v3:
- restore the v1 audio teardown in remove(): audio_shutdown() already stops the worker when the stream closes, so the extra flag clear and trailing cancel added in v2 were redundant (Dmitry Osipenko)
- rename the ACR read locals and add a comment documenting the register byte packing
- drop the get_dai_id stub so OF-graph cards resolve the DAI index from the reg property

Changes in v2:
- register the S/PDIF DAI so the indexes match the binding and reject it with -EOPNOTSUPP until wired up (Sebastian Reichel)
- use platform_device_register_data() and drop the fixed 32-bit DMA mask (Dmitry Osipenko)
- don't leave an ERR_PTR in audio_pdev on registration failure
- fix teardown ordering in remove()
- stop the worker before reprogramming shared state in hw_params()
- look up the "audio" clock by name instead of indexing clks[1]
- keep the worker on system_unbound_wq when re-arming

Quoted 3 lines
 .../platform/synopsys/hdmirx/snps_hdmirx.c    | 306 ++++++++++++++++++
 .../platform/synopsys/hdmirx/snps_hdmirx.h    |   8 +
 2 files changed, 314 insertions(+)

Note: Current variant of audio support requires userspace to manually select appropriate audio freq on capture. This needs to be improved later on for regular userspace apps by dynamically registering audio CODEC on HDMI cable plug event and reading out actual audio freq on the wire, restricting the CODEC's rate, which might require extension of the hdmi_codec_ops.

Signature · 2 lines
Best regards,
Dmitry
maintainer
#8 lore raw
In reply to Igor Paunovic
Quoted 7 lines · Igor Paunovic

This one looks structurally plausible, but as the report itself notes it predates this series: it concerns the video_device lifetime on the V4L2 side and is independent of the audio support added here (the audio teardown path was exercised on unbind with an active stream during v3 review, see the v3 thread). Addressing it would be a separate fix for the existing driver -- happy to leave that call to Dmitry.

No need to worry about it for this series. It is a niche bug if it's valid.

Signature · 2 lines
Best regards,
Dmitry
maintainer Reviewed-by on v4 1/4
#4 lore raw

Re: [PATCH v4 1/4] dt-bindings: media: snps,dw-hdmi-rx: add #sound-dai-cells

In reply to Igor Paunovic
Quoted 53 lines · Igor Paunovic

The HDMI RX controller can expose the audio embedded in the incoming HDMI stream as an ALSA capture device. Document the #sound-dai-cells property so that a sound card can reference the HDMI RX audio DAI.

---
Changes in v4 (both suggested by Krzysztof Kozlowski):
- reference dai-common.yaml and switch additionalProperties to
unevaluatedProperties: false
- drop the consumer comment from the binding example

Changes in v3:
- no changes

Changes in v2 (all suggested by Sebastian Reichel):
- reworded the commit message to describe the hardware
- property description now documents the DAI indexes (0 = I2S, 1 = S/PDIF)
- moved the consumer reference into the binding example

 .../devicetree/bindings/media/snps,dw-hdmi-rx.yaml  | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
Documentation/devicetree/bindings/media/snps,dw-hdmi-rx.yaml +11 -1
index b7f6c87..b80660d 100644
@@ -78,6 +78,13 @@ properties:
       The phandle of the syscon node for the Video Output GRF register
       to enable EDID transfer through SDAIN and SCLIN.
 
+  "#sound-dai-cells":
+    const: 1
+    description:
+      The HDMI RX controller has two digital audio interfaces, one for
+      I2S and one for S/PDIF. The DAI cell selects the interface, 0 for
+      I2S and 1 for S/PDIF.
+
 required:
   - compatible
   - reg
@@ -90,7 +97,10 @@ required:
   - pinctrl-0
   - hpd-gpios
 
-additionalProperties: false
+allOf:
+  - $ref: /schemas/sound/dai-common.yaml#
+
+unevaluatedProperties: false
 
 examples:
   - |

Documentation/devicetree/bindings/media/snps,dw-hdmi-rx.yaml examples: · line 143

Quoted 5 lines
Documentation/devicetree/bindings/media/snps,dw-hdmi-rx.yaml +1 -0
@@ -129,4 +139,5 @@ examples:
       pinctrl-0 = <&hdmim1_rx_cec &hdmim1_rx_hpdin &hdmim1_rx_scl &hdmim1_rx_sda &hdmirx_5v_detection>;
       pinctrl-names = "default";
       hpd-gpios = <&gpio1 22 GPIO_ACTIVE_LOW>;
+      #sound-dai-cells = <1>;
     };
Signature · 2 lines
Best regards,
Dmitry
maintainer Reviewed-by on v4 3/4
#13 lore raw

Re: [PATCH v4 3/4] arm64: dts: rockchip: add HDMI RX audio on RK3588

In reply to Igor Paunovic
Quoted 54 lines · Igor Paunovic

The Synopsys HDMI RX controller exposes two digital audio interfaces, one for I2S (0) and one for S/PDIF (1). Add the #sound-dai-cells property so audio cards can reference them as a codec, as documented by the snps,dw-hdmi-rx binding.

Add a shared simple-audio-card routing the receiver I2S DAI to i2s7_8ch, the receive-only I2S interface dedicated to HDMI RX, with the receiver as bitclock and frame master. The card is named "RK3588 HDMI-IN", which is what users see in alsamixer and what ALSA UCM matches on. It mirrors the existing HDMI TX sound cards in this file and is disabled by default, since not every RK3588 board routes HDMI RX audio; boards that do only need to flip its status.

---
Changes in v4 (both suggested by Sebastian Reichel):
- define the whole sound card here as a shared, disabled-by-default node mirroring the HDMI TX cards, instead of per board
- name the card "RK3588 HDMI-IN": the name is user-visible in alsamixer and matched by ALSA UCM

New in v3 (as "add #sound-dai-cells to the RK3588 HDMI receiver").

 arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +17 -0
index a264001..e0dce15 100644
@@ -23,6 +23,23 @@
 		};
 	};
 
+	hdmi_receiver_sound: hdmi-receiver-sound {
+		compatible = "simple-audio-card";
+		simple-audio-card,format = "i2s";
+		simple-audio-card,name = "RK3588 HDMI-IN";
+		simple-audio-card,bitclock-master = <&hdmiin_codec>;
+		simple-audio-card,frame-master = <&hdmiin_codec>;
+		status = "disabled";
+
+		hdmiin_codec: simple-audio-card,codec {
+			sound-dai = <&hdmi_receiver 0>;
+		};
+
+		simple-audio-card,cpu {
+			sound-dai = <&i2s7_8ch>;
+		};
+	};
+
 	reserved-memory {
 		#address-cells = <2>;
 		#size-cells = <2>;

arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi line 361

Quoted 7 lines
arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi +1 -0
@@ -338,6 +355,7 @@
 		reset-names = "axi", "apb", "ref", "biu";
 		rockchip,grf = <&sys_grf>;
 		rockchip,vo1-grf = <&vo1_grf>;
+		#sound-dai-cells = <1>;
 		status = "disabled";
 	};
 
Signature · 7 lines
Best regards,
Dmitry

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip
maintainer Reviewed-by on v4 4/4
#16 lore raw

Re: [PATCH v4 4/4] arm64: dts: rockchip: enable HDMI RX audio capture on Orange Pi 5 Plus

In reply to Igor Paunovic
Quoted 35 lines · Igor Paunovic

Enable the shared HDMI RX sound card and the i2s7_8ch interface it uses. Together with the audio capture support in the snps_hdmirx driver this exposes a capture-only ALSA card fed by the HDMI input.

Tested on the Orange Pi 5 Plus with multiple HDMI sources: capture follows the source sample rate and stays in sync via the FIFO-level clock tracking in the driver.

---
Changes in v4:
- only enable the shared hdmi_receiver_sound card and i2s7_8ch, now that the card definition lives in rk3588-extra.dtsi (Sebastian Reichel)

New in v3.

 arch/arm64/boot/dts/rockchip/rk3588-orangepi-5-plus.dts | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/arm64/boot/dts/rockchip/rk3588-orangepi-5-plus.dts +4 -0
index 9950d11..9f9a303 100644
@@ -171,6 +171,10 @@
 	status = "okay";
 };
 
+&hdmi_receiver_sound {
+	status = "okay";
+};
+
 &hdptxphy0 {
 	status = "okay";
 };

arch/arm64/boot/dts/rockchip/rk3588-orangepi-5-plus.dts line 260

Quoted 10 lines
arch/arm64/boot/dts/rockchip/rk3588-orangepi-5-plus.dts +4 -0
@@ -247,6 +251,10 @@
 	status = "okay";
 };
 
+&i2s7_8ch {
+	status = "okay";
+};
+
 &led_blue_gpio {
 	gpios = <&gpio3 RK_PA6 GPIO_ACTIVE_HIGH>;
 	status = "okay";
Signature · 7 lines
Best regards,
Dmitry

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip
#17 lore raw
In reply to Igor Paunovic

Hi Dmitry, Mauro,

First, belated thanks to Dmitry, Sebastian and Krzysztof for the reviews and testing across this series - much appreciated.

A gentle ping: all four patches carry Reviewed-by tags (2/4 also has Dmitry's Tested-by) and no changes have been requested since 23 July. Is there anything needed from my side for the series to be picked up into the media tree?

Thanks,
Igor

maintainer
#18 lore raw
Quoted 9 lines · Igor Paunovic

Hi Dmitry, Mauro,

First, belated thanks to Dmitry, Sebastian and Krzysztof for the reviews and testing across this series - much appreciated.

A gentle ping: all four patches carry Reviewed-by tags (2/4 also has Dmitry's Tested-by) and no changes have been requested since 23 July. Is there anything needed from my side for the series to be picked up into the media tree?

You can always send follow up patches separately, referencing to this series in cover letter. It may take significant time before patches will land to kernel tree. No need to wait.

Signature · 2 lines
Best regards,
Dmitry
#19 lore raw
In reply to Igor Paunovic

Hi Heiko,

patches 1/4 (binding) and 2/4 (driver) of this series are queued in the media tree (media-committers next, ee6a964dca65 and d667a3c4d003) and are in linux-next. 3/4 and 4/4 both carry Reviewed-by from Sebastian and Dmitry, and both still apply to your for-next (4/4 with an offset).

Could you pick up 3/4 and 4/4, or would you rather wait for the binding to reach mainline? I can resend them at that point if that is easier.

An LLM assistant helped with these checks and with this mail.

Igor