Skip to contentSkip to conversation

Add LTR controls

RFC PATCH In review Dikshita Agarwal 3 participants 11 messages Jun 11, 2020 – Aug 7, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Add LTR controls

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past and stored in the DPB buffer list to be used as reference to encode future frames. One usage of LTR encoding is to reduce error propagation for video transmission in packet lossy networks. For example, encoder may want to specify some key frames as LTR pictures and use them as reference frames for encoding. With extra protection selectively on these LTR frames or synchronization with the receiver of reception of the LTR frames during transmission, decoder can receive reference frames more reliably than other non-reference frames. As a result, transmission error can be effectively restricted within certain frames rather than propagated to future frames.

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
a. This is used to query or configure the number of LTR frames.
This is a static control and is controlled by the client.
b. The LTR index varies from 0 to the max LTR-1.
c. If LTR Count is more than max supported LTR count (max LTR) by driver, it will be rejected. d. Auto Marking : If LTR count is non zero,
1) first LTR count frames would be mark as LTR automatically after
every IDR frame (inclusive).
2) For multilayer encoding: first LTR count base layer reference frames starting after every IDR frame (inclusive) in encoding order would be marked as LTR frames by the encoder.
3) Auto marking of LTR due to IDR should consider following conditions:
1. The frame is not already set to be marked as LTR.
2. The frame is part of the base layer in the hierarchical layer case.
3. The number of frames currently marked as LTR is less than the maximum LTR frame index plus 1.
e. Encoder needs to handle explicit Mark/Use command when encoder is still doing "auto" marking

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
a. This signals to mark the current frame as LTR frame. It is a dynamic control and also provide the LTR index to be used. b. the LTR index provided by this control should never exceed the max LTR-1. Else it will be rejected.

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
a. This specifies the LTR frame(s) to be used for encoding the current frame. This is a dynamic control. b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N LSB bits of this field are valid,
where N is the maximum number of LTRs supported. All the other bits are invalid and should be rejected. The LSB corresponds to the LTR index 0. Bit N-1 from the LSB corresponds to the LTR index max LTR-1.

Dikshita Agarwal (1):
media: v4l2-ctrls: add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++
 2 files changed, 10 insertions(+)
Signature · 1 line
1.9.1
#1 lore raw

RFC PATCH 1/1 media: v4l2-ctrls: add control for ltr

Add v4l2 controls for ltr

---
 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++
 2 files changed, 10 insertions(+)
drivers/media/v4l2-core/v4l2-ctrls.c +6 -0
index 93d33d1..205863e 100644
@@ -977,6 +977,9 @@ const char *v4l2_ctrl_get_name(u32 id)
 	case V4L2_CID_MPEG_VIDEO_HEVC_SLICE_PARAMS:		return "HEVC Slice Parameters";
 	case V4L2_CID_MPEG_VIDEO_HEVC_DECODE_MODE:		return "HEVC Decode Mode";
 	case V4L2_CID_MPEG_VIDEO_HEVC_START_CODE:		return "HEVC Start Code";
+	case V4L2_CID_MPEG_VIDEO_LTRCOUNT: 			return "LTR Count";
+	case V4L2_CID_MPEG_VIDEO_MARKLTRFRAME: 			return "Mark LTR";
+	case V4L2_CID_MPEG_VIDEO_USELTRFRAME:			return "Use LTR";
 
 	/* CAMERA controls */
 	/* Keep the order of the 'case's the same as in v4l2-controls.h! */
@@ -1208,6 +1211,9 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
 		break;
 	case V4L2_CID_MPEG_VIDEO_MV_H_SEARCH_RANGE:
 	case V4L2_CID_MPEG_VIDEO_MV_V_SEARCH_RANGE:
+	case V4L2_CID_MPEG_VIDEO_LTRCOUNT:
+	case V4L2_CID_MPEG_VIDEO_MARKLTRFRAME:
+	case V4L2_CID_MPEG_VIDEO_USELTRFRAME:
 		*type = V4L2_CTRL_TYPE_INTEGER;
 		break;
 	case V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME:
include/uapi/linux/v4l2-controls.h +4 -0
index 1a58d7c..88d24e1 100644
@@ -737,6 +737,10 @@ enum v4l2_cid_mpeg_video_hevc_size_of_length_field {
 #define V4L2_CID_MPEG_VIDEO_HEVC_HIER_CODING_L6_BR	(V4L2_CID_MPEG_BASE + 642)
 #define V4L2_CID_MPEG_VIDEO_REF_NUMBER_FOR_PFRAMES	(V4L2_CID_MPEG_BASE + 643)
 #define V4L2_CID_MPEG_VIDEO_PREPEND_SPSPPS_TO_IDR	(V4L2_CID_MPEG_BASE + 644)
+#define V4L2_CID_MPEG_VIDEO_LTRCOUNT			(V4L2_CID_MPEG_BASE + 645)
+#define V4L2_CID_MPEG_VIDEO_MARKLTRFRAME		(V4L2_CID_MPEG_BASE + 646)
+#define V4L2_CID_MPEG_VIDEO_USELTRFRAME		(V4L2_CID_MPEG_BASE + 647)
+
 
 /*  MPEG-class control IDs specific to the CX2341x driver as defined by V4L2 */
 #define V4L2_CID_MPEG_CX2341X_BASE				(V4L2_CTRL_CLASS_MPEG | 0x1000)
Signature · 1 line
1.9.1
#2 lore raw
In reply to Dikshita Agarwal
Quoted 10 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past and stored in the DPB buffer list to be used as reference to encode future frames. One usage of LTR encoding is to reduce error propagation for video transmission in packet lossy networks. For example, encoder may want to specify some key frames as LTR pictures and use them as reference frames for encoding. With extra protection selectively on these LTR frames or synchronization with the receiver of reception of the LTR frames during transmission, decoder can receive reference frames more reliably than other non-reference frames. As a result, transmission error can be effectively restricted within certain frames rather than propagated to future frames.

Quoted 16 lines

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
a. This is used to query or configure the number of LTR frames.
This is a static control and is controlled by the client.
b. The LTR index varies from 0 to the max LTR-1.
c. If LTR Count is more than max supported LTR count (max LTR) by driver, it will be rejected. d. Auto Marking : If LTR count is non zero,
1) first LTR count frames would be mark as LTR automatically after
every IDR frame (inclusive).
2) For multilayer encoding: first LTR count base layer reference frames starting after every IDR frame (inclusive) in encoding order would be marked as LTR frames by the encoder.
3) Auto marking of LTR due to IDR should consider following conditions:
1. The frame is not already set to be marked as LTR.
2. The frame is part of the base layer in the hierarchical layer case.
3. The number of frames currently marked as LTR is less than the maximum LTR frame index plus 1.
e. Encoder needs to handle explicit Mark/Use command when encoder is still doing "auto" marking

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you add more _).

Quoted 4 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
a. This signals to mark the current frame as LTR frame. It is a dynamic control and also provide the LTR index to be used. b. the LTR index provided by this control should never exceed the max LTR-1. Else it will be rejected.

The "current" frame seems a bit loose. Perhaps you wanted to use buffer flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

Quoted 6 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
a. This specifies the LTR frame(s) to be used for encoding the current frame. This is a dynamic control. b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N LSB bits of this field are valid,
where N is the maximum number of LTRs supported. All the other bits are invalid and should be rejected. The LSB corresponds to the LTR index 0. Bit N-1 from the LSB corresponds to the LTR index max LTR-1.

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

As per all other generic encoder controls, we need to make sure it will be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
media: v4l2-ctrls: add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++
 2 files changed, 10 insertions(+)
#3 lore raw

Hi Dikshita, Nicolas,

Quoted 11 lines · Nicolas Dufresne
Quoted 10 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past and stored in the DPB buffer list to be used as reference to encode future frames. One usage of LTR encoding is to reduce error propagation for video transmission in packet lossy networks. For example, encoder may want to specify some key frames as LTR pictures and use them as reference frames for encoding. With extra protection selectively on these LTR frames or synchronization with the receiver of reception of the LTR frames during transmission, decoder can receive reference frames more reliably than other non-reference frames. As a result, transmission error can be effectively restricted within certain frames rather than propagated to future frames.

Quoted 16 lines
Quoted 8 lines

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
a. This is used to query or configure the number of LTR frames.
This is a static control and is controlled by the client.
b. The LTR index varies from 0 to the max LTR-1.
c. If LTR Count is more than max supported LTR count (max LTR) by driver, it will be rejected. d. Auto Marking : If LTR count is non zero,
1) first LTR count frames would be mark as LTR automatically after

Quoted 8 lines

every IDR frame (inclusive).
2) For multilayer encoding: first LTR count base layer reference frames starting after every IDR frame (inclusive) in encoding order would be marked as LTR frames by the encoder.
3) Auto marking of LTR due to IDR should consider following conditions:
1. The frame is not already set to be marked as LTR.
2. The frame is part of the base layer in the hierarchical layer case.
3. The number of frames currently marked as LTR is less than the maximum LTR frame index plus 1.
e. Encoder needs to handle explicit Mark/Use command when encoder is still doing "auto" marking

I don't follow this, quite possibly due to lack of experience with encoders.

I kind of would expect to see two modes: either automatic where encoders can mark up to LTR_COUNT frames as long term reference, and userspace just sets LTR_COUNT and doesn't have to do anything else.

Or it is manual mode where userspace explicitly marks long term reference frames.

From the proposal above it looks like you can mix auto and manual modes.

BTW, how do you 'unmark' long term reference frames?

This feature is for stateful encoders, right?

Quoted 9 lines

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you add more _).

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
a. This signals to mark the current frame as LTR frame. It is a dynamic control and also provide the LTR index to be used. b. the LTR index provided by this control should never exceed the max LTR-1. Else it will be rejected.

The "current" frame seems a bit loose. Perhaps you wanted to use buffer flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

I was thinking the same thing. Using a control for this doesn't seem right.

Quoted 7 lines
Quoted 7 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
a. This specifies the LTR frame(s) to be used for encoding the current frame. This is a dynamic control. b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N LSB bits of this field are valid,
where N is the maximum number of LTRs supported. All the other bits are invalid and should be rejected. The LSB corresponds to the LTR index 0. Bit N-1 from the LSB corresponds to the LTR index max LTR-1.

How would userspace know this? Esp. with auto marking since userspace would have to predict how auto marking works (if I understand this correctly).

For which HW encoder is this meant?

Quoted 10 lines

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

As per all other generic encoder controls, we need to make sure it will be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

I agree with Nicolas.

Regards,

Hans

Quoted 13 lines

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
media: v4l2-ctrls: add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++
 2 files changed, 10 insertions(+)
#4 lore raw

Hi Hans, Nicolas,

Thanks for your comments.

Quoted 45 lines · Hans Verkuil

Hi Dikshita, Nicolas,

Quoted 27 lines · Nicolas Dufresne
Quoted 26 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past and stored in the DPB buffer list to be used as reference to encode future frames. One usage of LTR encoding is to reduce error propagation for video transmission in packet lossy networks. For example, encoder may want to specify some key frames as LTR pictures and use them as reference frames for encoding. With extra protection selectively on these LTR frames or synchronization with the receiver of reception of the LTR frames during transmission, decoder can receive reference frames more reliably than other non-reference frames. As a result, transmission error can be effectively restricted within certain frames rather than propagated to future frames.

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
a. This is used to query or configure the number of LTR frames.
This is a static control and is controlled by the client.
b. The LTR index varies from 0 to the max LTR-1.
c. If LTR Count is more than max supported LTR count (max LTR) by
driver, it will be rejected.
d. Auto Marking : If LTR count is non zero,
1) first LTR count frames would be mark as LTR automatically
after
every IDR frame (inclusive).
2) For multilayer encoding: first LTR count base layer
reference frames starting after
every IDR frame (inclusive) in encoding order would be
marked as LTR frames by the encoder.
3) Auto marking of LTR due to IDR should consider following

Quoted 8 lines
Quoted 8 lines

conditions:
1. The frame is not already set to be marked as LTR.
2. The frame is part of the base layer in the
hierarchical layer case.
3. The number of frames currently marked as LTR is less
than the maximum LTR frame index plus 1.
e. Encoder needs to handle explicit Mark/Use command when encoder
is still doing "auto" marking

I don't follow this, quite possibly due to lack of experience with encoders.

I kind of would expect to see two modes: either automatic where encoders can mark up to LTR_COUNT frames as long term reference, and userspace just sets LTR_COUNT and doesn't have to do anything else.

Quoted 17 lines

Or it is manual mode where userspace explicitly marks long term reference frames.

From the proposal above it looks like you can mix auto and manual modes.

BTW, how do you 'unmark' long term reference frames?

This feature is for stateful encoders, right?

Quoted 8 lines

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you add more _).

Userspace sets LTR count which signifies the number of LTR frames encoder needs to generate or keep.
The encoder has to build-up its internal buffer reference list (aka DBP list or recon buffer list).
In order to achieve that encoder will fill It's LTR (long term references) list and STR (short term references) list by auto marking n frames as LTR frames(n is equal to LTR count) based on auto-marking dictated by the encoder spec. The client then can replace those automatically marked frames with new frames using V4L2_CID_MPEG_VIDEO_MARKLTRFRAME and can ask encoder to refer the newly marked frame for encoding the next frame using V4L2_CID_MPEG_VIDEO_USELTRFRAME.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
a. This signals to mark the current frame as LTR frame. It is a
dynamic control and also provide the LTR index to be used.
b. the LTR index provided by this control should never exceed the
max LTR-1. Else it will be rejected.

Quoted 6 lines
Quoted 3 lines

The "current" frame seems a bit loose. Perhaps you wanted to use buffer flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

I was thinking the same thing. Using a control for this doesn't seem right.

the client sets this to replace automatically marked frames by the encoder with a particular frame. this provides an index that ranges from 0 to LTR count-1 and then the particular frame will be marked with that index. this can be achieved through request by associating this control with a specific buffer to make it synchronized.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
a. This specifies the LTR frame(s) to be used for encoding the
current frame. This is a dynamic control.
b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N
LSB bits of this field are valid,
where N is the maximum number of LTRs supported. All the other
bits are invalid and should be rejected.
The LSB corresponds to the LTR index 0. Bit N-1 from the LSB
corresponds to the LTR index max LTR-1.

Quoted 3 lines

How would userspace know this? Esp. with auto marking since userspace would have to predict how auto marking works (if I understand this correctly).

Client sets LTR count which tells about the number of LTR frames automatically marked by the encoder.
so client can use LTR index (0 to LTR count -1) to ask encoder to refer any particular
frame (marked automatically by driver or marked by client with V4L2_CID_MPEG_VIDEO_MARKLTRFRAME) as a reference to encode the next frame.

Quoted 2 lines

For which HW encoder is this meant?

This is primarily meant for H264 and HEVC.

Thanks,
Dikshita

Quoted 30 lines
Quoted 10 lines

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

As per all other generic encoder controls, we need to make sure it will be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

I agree with Nicolas.

Regards,

Hans

Quoted 13 lines

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
media: v4l2-ctrls: add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++
 2 files changed, 10 insertions(+)
#5 lore raw

Hi,

A gentle reminder for the review.

Thanks,
Dikshita

Quoted 163 lines · On 2020-06-17 00:11, [email protected] wrote:

Hi Hans, Nicolas,

Thanks for your comments.

Quoted 45 lines · Hans Verkuil

Hi Dikshita, Nicolas,

Quoted 27 lines · Nicolas Dufresne
Quoted 26 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past and stored in the DPB buffer list to be used as reference to encode future frames. One usage of LTR encoding is to reduce error propagation for video transmission in packet lossy networks. For example, encoder may want to specify some key frames as LTR pictures and use them as reference frames for encoding. With extra protection selectively on these LTR frames or synchronization with the receiver of reception of the LTR frames during transmission, decoder can receive reference frames more reliably than other non-reference frames. As a result, transmission error can be effectively restricted within certain frames rather than propagated to future frames.

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
a. This is used to query or configure the number of LTR frames.
This is a static control and is controlled by the client.
b. The LTR index varies from 0 to the max LTR-1.
c. If LTR Count is more than max supported LTR count (max LTR)
by driver, it will be rejected.
d. Auto Marking : If LTR count is non zero,
1) first LTR count frames would be mark as LTR automatically
after
every IDR frame (inclusive).
2) For multilayer encoding: first LTR count base layer
reference frames starting after
every IDR frame (inclusive) in encoding order would be
marked as LTR frames by the encoder.
3) Auto marking of LTR due to IDR should consider following

Quoted 8 lines
Quoted 8 lines

conditions:
1. The frame is not already set to be marked as LTR.
2. The frame is part of the base layer in the
hierarchical layer case.
3. The number of frames currently marked as LTR is less
than the maximum LTR frame index plus 1.
e. Encoder needs to handle explicit Mark/Use command when
encoder is still doing "auto" marking

I don't follow this, quite possibly due to lack of experience with encoders.

I kind of would expect to see two modes: either automatic where encoders can mark up to LTR_COUNT frames as long term reference, and userspace just sets LTR_COUNT and doesn't have to do anything else.

Quoted 17 lines

Or it is manual mode where userspace explicitly marks long term reference frames.

From the proposal above it looks like you can mix auto and manual modes.

BTW, how do you 'unmark' long term reference frames?

This feature is for stateful encoders, right?

Quoted 8 lines

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you add more _).

Userspace sets LTR count which signifies the number of LTR frames encoder needs to generate or keep.
The encoder has to build-up its internal buffer reference list (aka DBP list or recon buffer list).
In order to achieve that encoder will fill It's LTR (long term references) list and STR (short term references) list
by auto marking n frames as LTR frames(n is equal to LTR count) based on auto-marking dictated by the encoder spec.
The client then can replace those automatically marked frames with new frames using V4L2_CID_MPEG_VIDEO_MARKLTRFRAME and can ask encoder to refer the newly marked frame for encoding the next frame using V4L2_CID_MPEG_VIDEO_USELTRFRAME.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
a. This signals to mark the current frame as LTR frame. It is a
dynamic control and also provide the LTR index to be used.
b. the LTR index provided by this control should never exceed
the max LTR-1. Else it will be rejected.

Quoted 6 lines
Quoted 3 lines

The "current" frame seems a bit loose. Perhaps you wanted to use buffer flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

I was thinking the same thing. Using a control for this doesn't seem right.

the client sets this to replace automatically marked frames by the encoder with a particular frame.
this provides an index that ranges from 0 to LTR count-1 and then the particular frame will be marked with that index.
this can be achieved through request by associating this control with a specific buffer to make it synchronized.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
a. This specifies the LTR frame(s) to be used for encoding the
current frame. This is a dynamic control.
b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N
LSB bits of this field are valid,
where N is the maximum number of LTRs supported. All the
other bits are invalid and should be rejected.
The LSB corresponds to the LTR index 0. Bit N-1 from the LSB
corresponds to the LTR index max LTR-1.

Quoted 3 lines

How would userspace know this? Esp. with auto marking since userspace would have to predict how auto marking works (if I understand this correctly).

Client sets LTR count which tells about the number of LTR frames automatically marked by the encoder.
so client can use LTR index (0 to LTR count -1) to ask encoder to refer any particular
frame (marked automatically by driver or marked by client with V4L2_CID_MPEG_VIDEO_MARKLTRFRAME) as a reference to encode the next frame.

Quoted 2 lines

For which HW encoder is this meant?

This is primarily meant for H264 and HEVC.

Thanks,
Dikshita

Quoted 30 lines
Quoted 10 lines

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

As per all other generic encoder controls, we need to make sure it will be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

I agree with Nicolas.

Regards,

Hans

Quoted 13 lines

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
media: v4l2-ctrls: add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++
 2 files changed, 10 insertions(+)
#6 lore raw
In reply to [email protected]
Quoted 144 lines · On 16/06/2020 20:41, [email protected] wrote:

Hi Hans, Nicolas,

Thanks for your comments.

Quoted 61 lines · Hans Verkuil

Hi Dikshita, Nicolas,

Quoted 36 lines · Nicolas Dufresne
Quoted 35 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past
and stored in the DPB buffer list to be used as reference to encode future frames.
One usage of LTR encoding is to reduce error propagation for video transmission
in packet lossy networks. For example, encoder may want to specify some key frames as
LTR pictures and use them as reference frames for encoding. With extra protection
selectively on these LTR frames or synchronization with the receiver of reception of
the LTR frames during transmission, decoder can receive reference frames more reliably
than other non-reference frames. As a result, transmission error can be effectively
restricted within certain frames rather than propagated to future frames.

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
a. This is used to query or configure the number of LTR frames.
This is a static control and is controlled by the client.
b. The LTR index varies from 0 to the max LTR-1.
c. If LTR Count is more than max supported LTR count (max LTR) by
driver, it will be rejected.
d. Auto Marking : If LTR count is non zero,
1) first LTR count frames would be mark as LTR automatically
after
every IDR frame (inclusive).
2) For multilayer encoding: first LTR count base layer
reference frames starting after
every IDR frame (inclusive) in encoding order would be
marked as LTR frames by the encoder.
3) Auto marking of LTR due to IDR should consider following

Quoted 8 lines
Quoted 8 lines

conditions:
1. The frame is not already set to be marked as LTR.
2. The frame is part of the base layer in the
hierarchical layer case.
3. The number of frames currently marked as LTR is less
than the maximum LTR frame index plus 1.
e. Encoder needs to handle explicit Mark/Use command when encoder
is still doing "auto" marking

I don't follow this, quite possibly due to lack of experience with encoders.

I kind of would expect to see two modes: either automatic where encoders can
mark up to LTR_COUNT frames as long term reference, and userspace just sets
LTR_COUNT and doesn't have to do anything else.

Or it is manual mode where userspace explicitly marks long term reference
frames.

Quoted 16 lines

From the proposal above it looks like you can mix auto and manual modes.

BTW, how do you 'unmark' long term reference frames?

This feature is for stateful encoders, right?

Quoted 9 lines

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you
add more _).

Userspace sets LTR count which signifies the number of LTR frames encoder needs to generate or keep.
The encoder has to build-up its internal buffer reference list (aka DBP list or recon buffer list).
In order to achieve that encoder will fill It's LTR (long term references) list and STR (short term references) list
by auto marking n frames as LTR frames(n is equal to LTR count) based on auto-marking dictated by the encoder spec.
The client then can replace those automatically marked frames with new frames using V4L2_CID_MPEG_VIDEO_MARKLTRFRAME and can ask encoder to refer the newly marked frame for encoding the next frame using V4L2_CID_MPEG_VIDEO_USELTRFRAME.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
a. This signals to mark the current frame as LTR frame. It is a
dynamic control and also provide the LTR index to be used.
b. the LTR index provided by this control should never exceed the
max LTR-1. Else it will be rejected.

Quoted 8 lines
Quoted 4 lines

The "current" frame seems a bit loose. Perhaps you wanted to use buffer
flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

I was thinking the same thing. Using a control for this doesn't seem right.

the client sets this to replace automatically marked frames by the encoder with a particular frame.
this provides an index that ranges from 0 to LTR count-1 and then the particular frame will be marked with that index.
this can be achieved through request by associating this control with a specific buffer to make it synchronized.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
a. This specifies the LTR frame(s) to be used for encoding the
current frame. This is a dynamic control.
b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N
LSB bits of this field are valid,
where N is the maximum number of LTRs supported. All the other
bits are invalid and should be rejected.
The LSB corresponds to the LTR index 0. Bit N-1 from the LSB
corresponds to the LTR index max LTR-1.

Quoted 4 lines

How would userspace know this? Esp. with auto marking since userspace would have
to predict how auto marking works (if I understand this correctly).

Client sets LTR count which tells about the number of LTR frames automatically marked by the encoder.
so client can use LTR index (0 to LTR count -1) to ask encoder to refer any particular
frame (marked automatically by driver or marked by client with V4L2_CID_MPEG_VIDEO_MARKLTRFRAME) as a reference to encode the next frame.

Quoted 3 lines
Quoted 2 lines

For which HW encoder is this meant?

This is primarily meant for H264 and HEVC.

The venus encoder?

Some more questions:

1) How many LTR frames do h.264 and hevc allow?
2) Given N LTR frames, is there a ordering of those frames? E.g. the LTR frame with index 0 is processed/used differently from LTR frame with index 1? Or are they all equal in that it is just a pool of LTR frames that the encoder can use as it wishes?

Regards,

Hans

Quoted 36 lines

Thanks,
Dikshita

Quoted 32 lines
Quoted 4 lines

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

Quoted 8 lines

As per all other generic encoder controls, we need to make sure it will
be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this
RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

I agree with Nicolas.

Regards,

Hans

Quoted 13 lines

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
media: v4l2-ctrls: add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++
 2 files changed, 10 insertions(+)
#7 lore raw
Quoted 150 lines · Hans Verkuil
Quoted 149 lines · On 16/06/2020 20:41, [email protected] wrote:

Hi Hans, Nicolas,

Thanks for your comments.

Quoted 64 lines · Hans Verkuil

Hi Dikshita, Nicolas,

Quoted 39 lines · Nicolas Dufresne
Quoted 38 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past
and stored in the DPB buffer list to be used as reference to encode future frames.
One usage of LTR encoding is to reduce error propagation for video transmission
in packet lossy networks. For example, encoder may want to specify some key frames as
LTR pictures and use them as reference frames for encoding. With extra protection
selectively on these LTR frames or synchronization with the receiver of reception of
the LTR frames during transmission, decoder can receive reference frames more reliably
than other non-reference frames. As a result, transmission error can be effectively
restricted within certain frames rather than propagated to future frames.

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
a. This is used to query or configure the number of LTR frames.
This is a static control and is controlled by the client.
b. The LTR index varies from 0 to the max LTR-1.
c. If LTR Count is more than max supported LTR count (max LTR)
by
driver, it will be rejected.
d. Auto Marking : If LTR count is non zero,
1) first LTR count frames would be mark as LTR
automatically
after
every IDR frame (inclusive).
2) For multilayer encoding: first LTR count base layer
reference frames starting after
every IDR frame (inclusive) in encoding order would be
marked as LTR frames by the encoder.
3) Auto marking of LTR due to IDR should consider following
conditions:

Quoted 8 lines
Quoted 8 lines

1. The frame is not already set to be marked as LTR.
2. The frame is part of the base layer in the
hierarchical layer case.
3. The number of frames currently marked as LTR is less
than the maximum LTR frame index plus 1.
e. Encoder needs to handle explicit Mark/Use command when
encoder
is still doing "auto" marking

I don't follow this, quite possibly due to lack of experience with encoders.

I kind of would expect to see two modes: either automatic where encoders can
mark up to LTR_COUNT frames as long term reference, and userspace just sets
LTR_COUNT and doesn't have to do anything else.

Or it is manual mode where userspace explicitly marks long term reference
frames.

Quoted 16 lines

From the proposal above it looks like you can mix auto and manual modes.

BTW, how do you 'unmark' long term reference frames?

This feature is for stateful encoders, right?

Quoted 9 lines

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you
add more _).

Userspace sets LTR count which signifies the number of LTR frames encoder needs to generate or keep.
The encoder has to build-up its internal buffer reference list (aka DBP list or recon buffer list).
In order to achieve that encoder will fill It's LTR (long term references) list and STR (short term references) list
by auto marking n frames as LTR frames(n is equal to LTR count) based on auto-marking dictated by the encoder spec.
The client then can replace those automatically marked frames with new frames using V4L2_CID_MPEG_VIDEO_MARKLTRFRAME and can ask encoder to refer the newly marked frame for encoding the next frame using V4L2_CID_MPEG_VIDEO_USELTRFRAME.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
a. This signals to mark the current frame as LTR frame. It is a
dynamic control and also provide the LTR index to be used.
b. the LTR index provided by this control should never exceed
the
max LTR-1. Else it will be rejected.

Quoted 8 lines
Quoted 4 lines

The "current" frame seems a bit loose. Perhaps you wanted to use buffer
flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

I was thinking the same thing. Using a control for this doesn't seem right.

the client sets this to replace automatically marked frames by the encoder with a particular frame.
this provides an index that ranges from 0 to LTR count-1 and then the particular frame will be marked with that index.
this can be achieved through request by associating this control with a specific buffer to make it synchronized.

Quoted 13 lines
Quoted 12 lines
Quoted 12 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
a. This specifies the LTR frame(s) to be used for encoding the
current frame. This is a dynamic control.
b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N
LSB bits of this field are valid,
where N is the maximum number of LTRs supported. All the
other
bits are invalid and should be rejected.
The LSB corresponds to the LTR index 0. Bit N-1 from the LSB
corresponds to the LTR index max LTR-1.

Quoted 4 lines

How would userspace know this? Esp. with auto marking since userspace would have
to predict how auto marking works (if I understand this correctly).

Client sets LTR count which tells about the number of LTR frames automatically marked by the encoder.
so client can use LTR index (0 to LTR count -1) to ask encoder to refer any particular
frame (marked automatically by driver or marked by client with V4L2_CID_MPEG_VIDEO_MARKLTRFRAME) as a reference to encode the next frame.

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

For which HW encoder is this meant?

This is primarily meant for H264 and HEVC.

The venus encoder?

yes

Quoted 4 lines

Some more questions:

1) How many LTR frames do h.264 and hevc allow?

According to spec, MAX LTR allowed by H.264 is 15 and HEVC allows max 32 LTR frames.

Quoted 5 lines

2) Given N LTR frames, is there a ordering of those frames? E.g. the LTR frame with index 0 is processed/used differently from LTR frame with index 1? Or are they all equal in that it is just a
pool
of LTR frames that the encoder can use as it wishes?

they are different frames marked to be used as LTR and stored in available indices. Userspace notifies encoder which LTR frame index to use via USE LTR control.

Thanks,
Dikshita

Quoted 41 lines

Regards,

Hans

Quoted 36 lines

Thanks,
Dikshita

Quoted 32 lines
Quoted 4 lines

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

Quoted 8 lines

As per all other generic encoder controls, we need to make sure it will
be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this
RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

I agree with Nicolas.

Regards,

Hans

Quoted 13 lines

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
media: v4l2-ctrls: add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++
 2 files changed, 10 insertions(+)
#8 lore raw
Quoted 155 lines · On 20/07/2020 16:33, [email protected] wrote:
Quoted 145 lines · Hans Verkuil
Quoted 144 lines · On 16/06/2020 20:41, [email protected] wrote:

Hi Hans, Nicolas,

Thanks for your comments.

Quoted 61 lines · Hans Verkuil

Hi Dikshita, Nicolas,

Quoted 36 lines · Nicolas Dufresne
Quoted 35 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past
and stored in the DPB buffer list to be used as reference to encode future frames.
One usage of LTR encoding is to reduce error propagation for video transmission
in packet lossy networks.  For example, encoder may want to specify some key frames as
LTR pictures and use them as reference frames for encoding. With extra protection
selectively on these LTR frames or synchronization with the receiver of reception of
the LTR frames during transmission, decoder can receive reference frames more reliably
than other non-reference frames. As a result, transmission error can be effectively
restricted within certain frames rather than propagated to future frames.

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
    a. This is used to query or configure the number of LTR frames. This is a static control and is controlled by the client. b. The LTR index varies from 0 to the max LTR-1.
    c. If LTR Count is more than max supported LTR count (max LTR) by driver, it will be rejected.
    d. Auto Marking : If LTR count is non zero,
        1) first LTR count frames would be mark as LTR automatically after
             every IDR frame (inclusive).
        2) For multilayer encoding: first LTR count base layer reference frames starting after
           every IDR frame (inclusive) in encoding order would be marked as LTR frames by the encoder.
        3) Auto marking of LTR due to IDR should consider following

Quoted 8 lines
Quoted 8 lines

conditions:
            1. The frame is not already set to be marked as LTR.
            2. The frame is part of the base layer in the hierarchical layer case.
            3. The number of frames currently marked as LTR is less than the maximum LTR frame index plus 1.
    e. Encoder needs to handle explicit Mark/Use command when encoder is still doing "auto" marking

I don't follow this, quite possibly due to lack of experience with encoders.

I kind of would expect to see two modes: either automatic where encoders can
mark up to LTR_COUNT frames as long term reference, and userspace just sets
LTR_COUNT and doesn't have to do anything else.

Or it is manual mode where userspace explicitly marks long term reference
frames.

Quoted 16 lines

From the proposal above it looks like you can mix auto and manual modes.

BTW, how do you 'unmark' long term reference frames?

This feature is for stateful encoders, right?

Quoted 9 lines

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you
add more _).

Userspace sets LTR count which signifies the number of LTR frames encoder needs to generate or keep.
The encoder has to build-up its internal buffer reference list (aka DBP list or recon buffer list).
In order to achieve that encoder will fill It's LTR (long term references) list and STR (short term references) list
by auto marking n frames as LTR frames(n is equal to LTR count) based on auto-marking dictated by the encoder spec.
The client then can replace those automatically marked frames with new frames using V4L2_CID_MPEG_VIDEO_MARKLTRFRAME and can ask encoder to refer the newly marked frame for encoding the next frame using V4L2_CID_MPEG_VIDEO_USELTRFRAME.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
    a. This signals to mark the current frame as LTR frame. It is a dynamic control and also provide the LTR index to be used. b. the LTR index provided by this control should never exceed the max LTR-1. Else it will be rejected.

Quoted 8 lines
Quoted 4 lines

The "current" frame seems a bit loose. Perhaps you wanted to use buffer
flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

I was thinking the same thing. Using a control for this doesn't seem right.

the client sets this to replace automatically marked frames by the encoder with a particular frame.
this provides an index that ranges from 0 to LTR count-1 and then the particular frame will be marked with that index.
this can be achieved through request by associating this control with a specific buffer to make it synchronized.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
    a. This specifies the LTR frame(s) to be used for encoding the current frame. This is a dynamic control.
    b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N LSB bits of this field are valid,
       where N is the maximum number of LTRs supported. All the other bits are invalid and should be rejected.
       The LSB corresponds to the LTR index 0. Bit N-1 from the LSB corresponds to the LTR index max LTR-1.

Quoted 4 lines

How would userspace know this? Esp. with auto marking since userspace would have
to predict how auto marking works (if I understand this correctly).

Client sets LTR count which tells about the number of LTR frames automatically marked by the encoder.
so client can use LTR index (0 to LTR count -1) to ask encoder to refer any particular
frame (marked automatically by driver or marked by client with V4L2_CID_MPEG_VIDEO_MARKLTRFRAME) as a reference to encode the next frame.

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

For which HW encoder is this meant?

This is primarily meant for H264 and HEVC.

The venus encoder?

yes

Quoted 3 lines

Some more questions:

Quoted 8 lines
Quoted 1 line

1) How many LTR frames do h.264 and hevc allow?

According to spec, MAX LTR allowed by H.264 is 15 and HEVC allows max 32 LTR frames.

Quoted 4 lines

2) Given N LTR frames, is there a ordering of those frames? E.g. the LTR frame with index 0 is processed/used differently from LTR frame with index 1? Or are they all equal in that it is just a pool of LTR frames that the encoder can use as it wishes?

they are different frames marked to be used as LTR and stored in available indices. Userspace notifies encoder which LTR frame index to use via USE LTR control.

One more question: if a frame is marked as a LTR frame, does that mean that userspace can't reuse the buffer containing that frame? I assume that's the case.

One thing that I don't like about this API is that it introduces the LTR indices. Would it be possible to instead use buffer indices? (i.e. the v4l2_buffer index field). Then you can use a buffer flag to indicate that a buffer should be an LTR buffer, or (for automarking) the driver can mark a buffer as an LTR buffer by setting this flag before returning the buffer to userspace.

You would still need a V4L2_CID_MPEG_VIDEO_USELTRFRAME, but that can be a bitmask using the buffer index.

This would avoid introducing a second method of referring to buffers (LTR index), and I think that V4L2_CID_MPEG_VIDEO_MARKLTRFRAME can be dropped.

Regards,

Hans

Quoted 44 lines

Thanks,
Dikshita

Quoted 41 lines

Regards,

    Hans

Quoted 36 lines

Thanks,
Dikshita

Quoted 32 lines
Quoted 4 lines

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

Quoted 8 lines

As per all other generic encoder controls, we need to make sure it will
be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this
RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

I agree with Nicolas.

Regards,

    Hans

Quoted 13 lines

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
  media: v4l2-ctrls:  add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++

 2 files changed, 10 insertions(+)
#9 lore raw
Quoted 183 lines · Hans Verkuil
Quoted 155 lines · On 20/07/2020 16:33, [email protected] wrote:
Quoted 145 lines · Hans Verkuil
Quoted 144 lines · On 16/06/2020 20:41, [email protected] wrote:

Hi Hans, Nicolas,

Thanks for your comments.

Quoted 61 lines · Hans Verkuil

Hi Dikshita, Nicolas,

Quoted 36 lines · Nicolas Dufresne
Quoted 35 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past
and stored in the DPB buffer list to be used as reference to encode future frames.
One usage of LTR encoding is to reduce error propagation for video transmission
in packet lossy networks.  For example, encoder may want to specify some key frames as
LTR pictures and use them as reference frames for encoding. With extra protection
selectively on these LTR frames or synchronization with the receiver of reception of
the LTR frames during transmission, decoder can receive reference frames more reliably
than other non-reference frames. As a result, transmission error can be effectively
restricted within certain frames rather than propagated to future frames.

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
    a. This is used to query or configure the number of LTR frames. This is a static control and is controlled by the client. b. The LTR index varies from 0 to the max LTR-1.
    c. If LTR Count is more than max supported LTR count (max LTR) by driver, it will be rejected.
    d. Auto Marking : If LTR count is non zero,
        1) first LTR count frames would be mark as LTR automatically after
             every IDR frame (inclusive).
        2) For multilayer encoding: first LTR count base layer reference frames starting after
           every IDR frame (inclusive) in encoding order would be marked as LTR frames by the encoder.
        3) Auto marking of LTR due to IDR should consider following

Quoted 8 lines
Quoted 8 lines

conditions:
            1. The frame is not already set to be marked as LTR.
            2. The frame is part of the base layer in the hierarchical layer case.
            3. The number of frames currently marked as LTR is less than the maximum LTR frame index plus 1.
    e. Encoder needs to handle explicit Mark/Use command when encoder is still doing "auto" marking

I don't follow this, quite possibly due to lack of experience with encoders.

I kind of would expect to see two modes: either automatic where encoders can
mark up to LTR_COUNT frames as long term reference, and userspace just sets
LTR_COUNT and doesn't have to do anything else.

Or it is manual mode where userspace explicitly marks long term reference
frames.

Quoted 16 lines

From the proposal above it looks like you can mix auto and manual modes.

BTW, how do you 'unmark' long term reference frames?

This feature is for stateful encoders, right?

Quoted 9 lines

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you
add more _).

Userspace sets LTR count which signifies the number of LTR frames encoder needs to generate or keep.
The encoder has to build-up its internal buffer reference list (aka DBP list or recon buffer list).
In order to achieve that encoder will fill It's LTR (long term references) list and STR (short term references) list
by auto marking n frames as LTR frames(n is equal to LTR count) based on auto-marking dictated by the encoder spec.
The client then can replace those automatically marked frames with new frames using V4L2_CID_MPEG_VIDEO_MARKLTRFRAME and can ask encoder to refer the newly marked frame for encoding the next frame using V4L2_CID_MPEG_VIDEO_USELTRFRAME.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
    a. This signals to mark the current frame as LTR frame. It is a dynamic control and also provide the LTR index to be used. b. the LTR index provided by this control should never exceed the max LTR-1. Else it will be rejected.

Quoted 8 lines
Quoted 4 lines

The "current" frame seems a bit loose. Perhaps you wanted to use buffer
flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

I was thinking the same thing. Using a control for this doesn't seem right.

the client sets this to replace automatically marked frames by the encoder with a particular frame.
this provides an index that ranges from 0 to LTR count-1 and then the particular frame will be marked with that index.
this can be achieved through request by associating this control with a specific buffer to make it synchronized.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
    a. This specifies the LTR frame(s) to be used for encoding the current frame. This is a dynamic control.
    b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N LSB bits of this field are valid,
       where N is the maximum number of LTRs supported. All the other bits are invalid and should be rejected.
       The LSB corresponds to the LTR index 0. Bit N-1 from the LSB corresponds to the LTR index max LTR-1.

Quoted 4 lines

How would userspace know this? Esp. with auto marking since userspace would have
to predict how auto marking works (if I understand this correctly).

Client sets LTR count which tells about the number of LTR frames automatically marked by the encoder.
so client can use LTR index (0 to LTR count -1) to ask encoder to refer any particular
frame (marked automatically by driver or marked by client with V4L2_CID_MPEG_VIDEO_MARKLTRFRAME) as a reference to encode the next frame.

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

For which HW encoder is this meant?

This is primarily meant for H264 and HEVC.

The venus encoder?

yes

Quoted 3 lines

Some more questions:

Quoted 8 lines
Quoted 1 line

1) How many LTR frames do h.264 and hevc allow?

According to spec, MAX LTR allowed by H.264 is 15 and HEVC allows max 32 LTR frames.

Quoted 4 lines

2) Given N LTR frames, is there a ordering of those frames? E.g. the LTR frame with index 0 is processed/used differently from LTR frame with index 1? Or are they all equal in that it is just a pool of LTR frames that the encoder can use as it wishes?

they are different frames marked to be used as LTR and stored in available indices. Userspace notifies encoder which LTR frame index to use via USE LTR control.

One more question: if a frame is marked as a LTR frame, does that mean that userspace
can't reuse the buffer containing that frame? I assume that's the case.

One thing that I don't like about this API is that it introduces the LTR indices.
Would it be possible to instead use buffer indices? (i.e. the v4l2_buffer index field).
Then you can use a buffer flag to indicate that a buffer should be an LTR buffer,
or (for automarking) the driver can mark a buffer as an LTR buffer by setting this flag
before returning the buffer to userspace.

You would still need a V4L2_CID_MPEG_VIDEO_USELTRFRAME, but that can be a bitmask
using the buffer index.

Quoted 7 lines

This would avoid introducing a second method of referring to buffers (LTR index),
and I think that V4L2_CID_MPEG_VIDEO_MARKLTRFRAME can be dropped.

Regards,

Hans

Hi Hans,
LTR indices are part of the spec, we are not introducing anything new.
Also, there is no relation between v4l2 buffer index and LTR index. So we can't use V4l2 index as LTR index.

below are some details about LTR from spec:

Regarding LTR count:
max_num_ref_frames specifies the maximum number of short-term and long-term reference frames, complementary reference field pairs, and non-paired reference fields that may be used by the decoding process for inter prediction of any picture in the coded video sequence. The value of max_num_ref_frames shall be in the range of 0 to MaxDpbFrames (as specified in clause A.3.1 or A.3.2), inclusive. max_long_term_frame_idx_plus1 minus 1 specifies the maximum value of long-term frame index allowed for long-term reference pictures (until receipt of another value of max_long_term_frame_idx_plus1). The value of max_long_term_frame_idx_plus1 shall be in the range of 0 to max_num_ref_frames, inclusive.

Regarding LTR index which will be specified by V4L2_CID_MPEG_VIDEO_MARKLTRFRAM :
long_term_pic_num specifies the long-term picture number of the picture being moved to the current index in the list. When decoding a coded frame, long_term_pic_num shall be equal to a LongTermPicNum assigned to one of the reference frames or complementary reference field pairs marked as "used for long-term reference". long_term_frame_idx is used (with memory_management_control_operation equal to 3 or 6) to assign a long-term frame index to a picture. When the associated memory_management_control_operation is processed by the decoding process, the value of long_term_frame_idx shall be in the range of 0 to MaxLongTermFrameIdx, inclusive.

Also As there can be multiple LTR present. We can instruct encoder to use one and/or multiple of them using V4L2_CID_MPEG_VIDEO_USELTRFRAME with the bitmask.

Thanks,
Dikshita

Quoted 45 lines
Quoted 45 lines

Thanks,
Dikshita

Quoted 41 lines

Regards,

    Hans

Quoted 36 lines

Thanks,
Dikshita

Quoted 32 lines
Quoted 4 lines

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

Quoted 8 lines

As per all other generic encoder controls, we need to make sure it will
be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this
RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

I agree with Nicolas.

Regards,

    Hans

Quoted 13 lines

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
  media: v4l2-ctrls:  add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++

 2 files changed, 10 insertions(+)
#10 lore raw
Quoted 188 lines · On 06/08/2020 13:05, [email protected] wrote:
Quoted 187 lines · Hans Verkuil
Quoted 155 lines · On 20/07/2020 16:33, [email protected] wrote:
Quoted 145 lines · Hans Verkuil
Quoted 144 lines · On 16/06/2020 20:41, [email protected] wrote:

Hi Hans, Nicolas,

Thanks for your comments.

Quoted 61 lines · Hans Verkuil

Hi Dikshita, Nicolas,

Quoted 36 lines · Nicolas Dufresne
Quoted 35 lines · Dikshita Agarwal

LTR (Long Term Reference) frames are the frames that are encoded sometime in the past
and stored in the DPB buffer list to be used as reference to encode future frames.
One usage of LTR encoding is to reduce error propagation for video transmission
in packet lossy networks.  For example, encoder may want to specify some key frames as
LTR pictures and use them as reference frames for encoding. With extra protection
selectively on these LTR frames or synchronization with the receiver of reception of
the LTR frames during transmission, decoder can receive reference frames more reliably
than other non-reference frames. As a result, transmission error can be effectively
restricted within certain frames rather than propagated to future frames.

We are introducing below V4l2 Controls for this feature
1. V4L2_CID_MPEG_VIDEO_LTRCOUNT
    a. This is used to query or configure the number of LTR frames. This is a static control and is controlled by the client. b. The LTR index varies from 0 to the max LTR-1.
    c. If LTR Count is more than max supported LTR count (max LTR) by driver, it will be rejected.
    d. Auto Marking : If LTR count is non zero,
        1) first LTR count frames would be mark as LTR automatically after
             every IDR frame (inclusive).
        2) For multilayer encoding: first LTR count base layer reference frames starting after
           every IDR frame (inclusive) in encoding order would be marked as LTR frames by the encoder.
        3) Auto marking of LTR due to IDR should consider following

Quoted 8 lines
Quoted 8 lines

conditions:
            1. The frame is not already set to be marked as LTR.
            2. The frame is part of the base layer in the hierarchical layer case.
            3. The number of frames currently marked as LTR is less than the maximum LTR frame index plus 1.
    e. Encoder needs to handle explicit Mark/Use command when encoder is still doing "auto" marking

I don't follow this, quite possibly due to lack of experience with encoders.

I kind of would expect to see two modes: either automatic where encoders can
mark up to LTR_COUNT frames as long term reference, and userspace just sets
LTR_COUNT and doesn't have to do anything else.

Or it is manual mode where userspace explicitly marks long term reference
frames.

Quoted 16 lines

From the proposal above it looks like you can mix auto and manual modes.

BTW, how do you 'unmark' long term reference frames?

This feature is for stateful encoders, right?

Quoted 9 lines

Perhaps we are missing a LONG_TERM_REFERENCE_MODE ? I bet some encoder can select by themself long term references and even some encoders may not let the user decide.

(not huge han of LTR acronyme, but that could be fine too, assuming you
add more _).

Userspace sets LTR count which signifies the number of LTR frames encoder needs to generate or keep.
The encoder has to build-up its internal buffer reference list (aka DBP list or recon buffer list).
In order to achieve that encoder will fill It's LTR (long term references) list and STR (short term references) list
by auto marking n frames as LTR frames(n is equal to LTR count) based on auto-marking dictated by the encoder spec.
The client then can replace those automatically marked frames with new frames using V4L2_CID_MPEG_VIDEO_MARKLTRFRAME and can ask encoder to refer the newly marked frame for encoding the next frame using V4L2_CID_MPEG_VIDEO_USELTRFRAME.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

2. V4L2_CID_MPEG_VIDEO_MARKLTRFRAME :
    a. This signals to mark the current frame as LTR frame. It is a dynamic control and also provide the LTR index to be used. b. the LTR index provided by this control should never exceed the max LTR-1. Else it will be rejected.

Quoted 8 lines
Quoted 4 lines

The "current" frame seems a bit loose. Perhaps you wanted to use buffer
flags ? A bit like what we have to signal TOP/BOTTOM fields in alternate interlacing.

I was thinking the same thing. Using a control for this doesn't seem right.

the client sets this to replace automatically marked frames by the encoder with a particular frame.
this provides an index that ranges from 0 to LTR count-1 and then the particular frame will be marked with that index.
this can be achieved through request by associating this control with a specific buffer to make it synchronized.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

3. V4L2_CID_MPEG_VIDEO_USELTRFRAME :
    a. This specifies the LTR frame(s) to be used for encoding the current frame. This is a dynamic control.
    b. LTR Use Bitmap : this consists of bits [0, 15]. A total of N LSB bits of this field are valid,
       where N is the maximum number of LTRs supported. All the other bits are invalid and should be rejected.
       The LSB corresponds to the LTR index 0. Bit N-1 from the LSB corresponds to the LTR index max LTR-1.

Quoted 4 lines

How would userspace know this? Esp. with auto marking since userspace would have
to predict how auto marking works (if I understand this correctly).

Client sets LTR count which tells about the number of LTR frames automatically marked by the encoder.
so client can use LTR index (0 to LTR count -1) to ask encoder to refer any particular
frame (marked automatically by driver or marked by client with V4L2_CID_MPEG_VIDEO_MARKLTRFRAME) as a reference to encode the next frame.

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

For which HW encoder is this meant?

This is primarily meant for H264 and HEVC.

The venus encoder?

yes

Quoted 3 lines

Some more questions:

Quoted 8 lines
Quoted 1 line

1) How many LTR frames do h.264 and hevc allow?

According to spec, MAX LTR allowed by H.264 is 15 and HEVC allows max 32 LTR frames.

Quoted 4 lines

2) Given N LTR frames, is there a ordering of those frames? E.g. the LTR frame with index 0 is processed/used differently from LTR frame with index 1? Or are they all equal in that it is just a pool of LTR frames that the encoder can use as it wishes?

they are different frames marked to be used as LTR and stored in available indices. Userspace notifies encoder which LTR frame index to use via USE LTR control.

One more question: if a frame is marked as a LTR frame, does that mean that userspace
can't reuse the buffer containing that frame? I assume that's the case.

One thing that I don't like about this API is that it introduces the LTR indices.
Would it be possible to instead use buffer indices? (i.e. the v4l2_buffer index field).
Then you can use a buffer flag to indicate that a buffer should be an LTR buffer,
or (for automarking) the driver can mark a buffer as an LTR buffer by setting this flag
before returning the buffer to userspace.

You would still need a V4L2_CID_MPEG_VIDEO_USELTRFRAME, but that can be a bitmask
using the buffer index.

This would avoid introducing a second method of referring to buffers (LTR index),
and I think that V4L2_CID_MPEG_VIDEO_MARKLTRFRAME can be dropped.

Quoted 7 lines
Quoted 3 lines

Regards,

    Hans

Hi Hans,
LTR indices are part of the spec, we are not introducing anything new.
Also, there is no relation between v4l2 buffer index and LTR index. So we can't use V4l2 index as LTR index.

Ah, I think I start to understand the cause of my confusion: I thought that the buffers containing LTR frames had to be kept around by userspace, but instead the driver/HW makes copies and userspace does not need to do anything special with those buffers.

OK, can you post a v2?

Document the controls in Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst

Regards,

Hans

Quoted 67 lines

below are some details about LTR from spec:

Regarding LTR count:
max_num_ref_frames specifies the maximum number of short-term and long-term reference frames, complementary reference field pairs, and non-paired reference fields that may be used by the decoding process for inter prediction of any picture in the coded video sequence. The value of max_num_ref_frames shall be in the range of 0 to MaxDpbFrames (as specified in clause A.3.1 or A.3.2), inclusive. max_long_term_frame_idx_plus1 minus 1 specifies the maximum value of long-term frame index allowed for long-term reference pictures (until receipt of another value of max_long_term_frame_idx_plus1). The value of max_long_term_frame_idx_plus1 shall be in the range of 0 to max_num_ref_frames, inclusive.

Regarding LTR index which will be specified by V4L2_CID_MPEG_VIDEO_MARKLTRFRAM :
long_term_pic_num specifies the long-term picture number of the picture being moved to the current index in the list. When decoding a coded frame, long_term_pic_num shall be equal to a LongTermPicNum assigned to one of the reference frames or complementary reference field pairs marked as "used for long-term reference".
long_term_frame_idx is used (with memory_management_control_operation equal to 3 or 6) to assign a long-term frame index to a picture. When the associated memory_management_control_operation is processed by the decoding process, the value of long_term_frame_idx shall be in the range of 0 to MaxLongTermFrameIdx, inclusive.

Also As there can be multiple LTR present. We can instruct encoder to use one and/or multiple of them using V4L2_CID_MPEG_VIDEO_USELTRFRAME with the bitmask.

Thanks,
Dikshita

Quoted 45 lines
Quoted 45 lines

Thanks,
Dikshita

Quoted 41 lines

Regards,

    Hans

Quoted 36 lines

Thanks,
Dikshita

Quoted 32 lines
Quoted 4 lines

Note, I haven't captured very well the userspace control flow, perhaps this could be enhanced through writing some documentation.

Quoted 8 lines

As per all other generic encoder controls, we need to make sure it will
be usable and flexible enough for multiple HW blocks, as it can be tedious to extend later otherwise. It is important that along with this
RFC you provide some comparisons with with other HW / SW APIs in order to help justify the design decisions. I also think there should be link made V4L2_CID_MPEG_VIDEO_GOP_* , number of B-Frames etc.

I agree with Nicolas.

Regards,

    Hans

Quoted 13 lines

regards,
Nicolas

Quoted 8 lines

Dikshita Agarwal (1):
  media: v4l2-ctrls:  add control for ltr

 drivers/media/v4l2-core/v4l2-ctrls.c | 6 ++++++
 include/uapi/linux/v4l2-controls.h   | 4 ++++

 2 files changed, 10 insertions(+)