Skip to contentSkip to conversation

mctp i2c: check packet length before marking flow active

PATCH Superseded William A. Kennington III 4 participants 12 messages Apr 23 – May 7

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mctp i2c: check packet length before marking flow active

Currently, mctp_i2c_get_tx_flow_state() is called before the packet length sanity check. This function marks a new flow as active in the MCTP core.

If the sanity check fails, mctp_i2c_xmit() returns early without calling mctp_i2c_lock_nest(). This results in a mismatched locking state: the flow is active, but the I2C bus lock was never acquired for it.

When the flow is later released, mctp_i2c_release_flow() will see the active state and queue an unlock marker. The TX thread will then decrement midev->i2c_lock_count from 0, causing it to underflow to -1.

This underflow permanently breaks the driver's locking logic, allowing future transmissions to occur without holding the I2C bus lock, leading to bus collisions and potential hardware hangs.

Move the mctp_i2c_get_tx_flow_state() call to after the length sanity check to ensure we only transition the flow state if we are actually going to proceed with the transmission and locking.

Fixes f5b8abf9fc3d ("mctp i2c: MCTP I2C binding driver")
---
 drivers/net/mctp/mctp-i2c.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/mctp/mctp-i2c.c +2 -2
index 15fe4d1163c1..ee2913758e54 100644
@@ -496,8 +496,6 @@ static void mctp_i2c_xmit(struct mctp_i2c_dev *midev, struct sk_buff *skb)
 	u8 *pecp;
 	int rc;
 
-	fs = mctp_i2c_get_tx_flow_state(midev, skb);
-
 	hdr = (void *)skb_mac_header(skb);
 	/* Sanity check that packet contents matches skb length,
 	 * and can't exceed MCTP_I2C_BUFSZ
@@ -509,6 +507,8 @@ static void mctp_i2c_xmit(struct mctp_i2c_dev *midev, struct sk_buff *skb)
 		return;
 	}
 
+	fs = mctp_i2c_get_tx_flow_state(midev, skb);
+
 	if (skb_tailroom(skb) >= 1) {
 		/* Linear case with space, we can just append the PEC */
 		skb_put(skb, 1);
Signature · 1 line
2.54.0.rc2.533.g4f5dca5207-goog
maintainer Acked-by on v1 1/1
#1 lore raw

Hi William,

Quoted 3 lines

Move the mctp_i2c_get_tx_flow_state() call to after the length sanity check to ensure we only transition the flow state if we are actually going to proceed with the transmission and locking.

Good catch, thanks!

Quoted 1 line

Subject: [PATCH] mctp i2c: check packet length before marking flow active

You'll want to indicate that this is for the net tree, rather than net-next, so something like:

Subject: [PATCH net] net: mctp i2c: check packet length [...]

With that change:

Out of curiosity though, how did you hit the hdr_byte_count mismatch in the first place?

Cheers,

Jeremy

#2 lore raw
Quoted 5 lines · Jeremy Kerr

Hi William,

Quoted 3 lines

Move the mctp_i2c_get_tx_flow_state() call to after the length sanity check to ensure we only transition the flow state if we are actually going to proceed with the transmission and locking.

Quoted 7 lines

Good catch, thanks!

Quoted 1 line

Subject: [PATCH] mctp i2c: check packet length before marking flow active

You'll want to indicate that this is for the net tree, rather than net-next, so something like:

Subject: [PATCH net] net: mctp i2c: check packet length [...]

Thanks, forgot the convention...

Quoted 7 lines

With that change:

Out of curiosity though, how did you hit the hdr_byte_count mismatch in the first place?

Our current theory is that we have known buggy firmware on our NVME MCTP devices and we are seeing some kind of corruption on the bus that we are going to fix in on the firmware side. We started also seeing kernel crashes along with the bad firmware symptoms, walked through ~110 kdumps and found i2c locks that were held by 2 owners (eeprom reading and the MCTP TX queue). This ended up causing deadlocks in the i2c stack that result in panics. We noticed in many of these cases that we had 10K+ BERs and 800+ NACKs in the i2c driver stats struct at the time of crash.

Quoted 5 lines

Cheers,

Jeremy

#7 lore raw

PATCH v2 1/1 net: mctp i2c: check length before marking flow active

In reply to William A. Kennington III

Currently, mctp_i2c_get_tx_flow_state() is called before the packet length sanity check. This function marks a new flow as active in the MCTP core.

If the sanity check fails, mctp_i2c_xmit() returns early without calling mctp_i2c_lock_nest(). This results in a mismatched locking state: the flow is active, but the I2C bus lock was never acquired for it.

When the flow is later released, mctp_i2c_release_flow() will see the active state and queue an unlock marker. The TX thread will then decrement midev->i2c_lock_count from 0, causing it to underflow to -1.

This underflow permanently breaks the driver's locking logic, allowing future transmissions to occur without holding the I2C bus lock, leading to bus collisions and potential hardware hangs.

Move the mctp_i2c_get_tx_flow_state() call to after the length sanity check to ensure we only transition the flow state if we are actually going to proceed with the transmission and locking.

Fixes f5b8abf9fc3d ("mctp i2c: MCTP I2C binding driver")
---
 drivers/net/mctp/mctp-i2c.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/mctp/mctp-i2c.c +2 -2
index 15fe4d1163c1..ee2913758e54 100644
@@ -496,8 +496,6 @@ static void mctp_i2c_xmit(struct mctp_i2c_dev *midev, struct sk_buff *skb)
 	u8 *pecp;
 	int rc;
 
-	fs = mctp_i2c_get_tx_flow_state(midev, skb);
-
 	hdr = (void *)skb_mac_header(skb);
 	/* Sanity check that packet contents matches skb length,
 	 * and can't exceed MCTP_I2C_BUFSZ
@@ -509,6 +507,8 @@ static void mctp_i2c_xmit(struct mctp_i2c_dev *midev, struct sk_buff *skb)
 		return;
 	}
 
+	fs = mctp_i2c_get_tx_flow_state(midev, skb);
+
 	if (skb_tailroom(skb) >= 1) {
 		/* Linear case with space, we can just append the PEC */
 		skb_put(skb, 1);
Signature · 1 line
2.54.0.545.g6539524ca2-goog
maintainer
#3 lore raw
In reply to William A. Kennington III

Hi William,

Quoted 6 lines
Quoted 2 lines

Out of curiosity though, how did you hit the hdr_byte_count mismatch in the first place?

Our current theory is that we have known buggy firmware on our NVME MCTP devices and we are seeing some kind of corruption on the bus that we are going to fix in on the firmware side.

OK, sounds good for the overall fix, but I don't think that would be causing the path that you're addressing here. The fix is definitely valid, but can't be hit through any RX data corruption (we're in the TX path).

The header byte count is populated during header construction, so a mismatch here would indicate modification of the skb between that point at the actual xmit. Do you see the "Bad TX len" warning in these cases?

Quoted 4 lines

We started also seeing kernel
crashes along with the bad firmware symptoms, walked through ~110 kdumps and found i2c locks that were held by 2 owners (eeprom reading and the MCTP TX queue).

Just to clarify my understanding of the state: "being held by two owners" would indicate a violation of the lock itself. Or is it that there are two threads blocked waiting to acquire the mutex?

For NVMe-MI, you're likely using manual tag allocation, where the tag allocation (and hence flow state) is entirely controlled by userspace. It may be that the NVMe protocol-level errors are causing that tags to be held for long durations, perhaps?

Cheers,

Jeremy

maintainer
#8 lore raw

Re: [PATCH net v2] net: mctp i2c: check length before marking flow active

In reply to William A. Kennington III
Quoted 15 lines · William A. Kennington III

Currently, mctp_i2c_get_tx_flow_state() is called before the packet length sanity check. This function marks a new flow as active in the MCTP core.

If the sanity check fails, mctp_i2c_xmit() returns early without calling mctp_i2c_lock_nest(). This results in a mismatched locking state: the flow is active, but the I2C bus lock was never acquired for it.

When the flow is later released, mctp_i2c_release_flow() will see the active state and queue an unlock marker. The TX thread will then decrement midev->i2c_lock_count from 0, causing it to underflow to -1.

This underflow permanently breaks the driver's locking logic, allowing future transmissions to occur without holding the I2C bus lock, leading to bus collisions and potential hardware hangs.

Quoted 6 lines

Move the mctp_i2c_get_tx_flow_state() call to after the length sanity check to ensure we only transition the flow state if we are actually going to proceed with the transmission and locking.

Fixes f5b8abf9fc3d ("mctp i2c: MCTP I2C binding driver")

Note that you should have included Jeremy's ack, and you should have avoided reposting before the 24h grace period. In this specific case, you could have avoided a repost entirely

/P

#9 lore raw

Re: [PATCH net v2] net: mctp i2c: check length before marking flow active

In reply to William A. Kennington III
This patch was applied to netdev/net.git (main) by Paolo Abeni <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Paolo Abeni <[email protected]>:

Quoted 8 lines · On Thu, 23 Apr 2026 00:46:52 -0700 you wrote:

Currently, mctp_i2c_get_tx_flow_state() is called before the packet length sanity check. This function marks a new flow as active in the MCTP core.

If the sanity check fails, mctp_i2c_xmit() returns early without calling mctp_i2c_lock_nest(). This results in a mismatched locking state: the flow is active, but the I2C bus lock was never acquired for it.

[...]

Here is the summary with links:
- [net,v2] net: mctp i2c: check length before marking flow active

    https://git.kernel.org/netdev/net/c/4ca07b9239bd

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
maintainer
#10 lore raw

Re: [PATCH net v2] net: mctp i2c: check length before marking flow active

Hi Pablo,

Quoted 5 lines

Here is the summary with links:
  - [net,v2] net: mctp i2c: check length before marking flow active
    https://git.kernel.org/netdev/net/c/4ca07b9239bd

You are awesome, thank you!

4ca07b9239bd seems to have acquired an unrelated change:

   $ git show 4ca07b9239bd | diffstat
    drivers/net/mctp/mctp-i2c.c |    4 ++--
    net/sched/cls_flower.c      |    4 +++-
    2 files changed, 5 insertions(+), 3 deletions(-)

While the commit structure is probably not intentional, are the Flower changes acceptable for the net tree?

Cheers,

Jeremy

maintainer
#11 lore raw

Re: [PATCH net v2] net: mctp i2c: check length before marking flow active

Quoted 10 lines · Jeremy Kerr

Hi Pablo,

Quoted 5 lines

Here is the summary with links:
  - [net,v2] net: mctp i2c: check length before marking flow active
    https://git.kernel.org/netdev/net/c/4ca07b9239bd

You are awesome, thank you!

4ca07b9239bd seems to have acquired an unrelated change:

Quoted 7 lines
   $ git show 4ca07b9239bd | diffstat
    drivers/net/mctp/mctp-i2c.c |    4 ++--
    net/sched/cls_flower.c      |    4 +++-
    2 files changed, 5 insertions(+), 3 deletions(-)

While the commit structure is probably not intentional, are the Flower changes acceptable for the net tree?

Thanks for checking!

That is a serious error on my side, I had an unclean working tree when I applied the patch that ended-up including random staff. Absolutely not intended for the net tree. I'll send a fixup patch.

Thank you again for the head-up!

Paolo

#4 lore raw
In reply to Jeremy Kerr
Quoted 11 lines · Jeremy Kerr

Hi William,

Quoted 5 lines
Quoted 2 lines

Out of curiosity though, how did you hit the hdr_byte_count mismatch in the first place?

Our current theory is that we have known buggy firmware on our NVME MCTP devices and we are seeing some kind of corruption on the bus that we are going to fix in on the firmware side.

OK, sounds good for the overall fix, but I don't think that would be causing the path that you're addressing here. The fix is definitely valid, but can't be hit through any RX data corruption (we're in the TX path).

Yeah I think you might be right, the hard part is reproducing this is so infrequent for us that it takes a long time to iterate on testing these changes.

Quoted 4 lines

The header byte count is populated during header construction, so a mismatch here would indicate modification of the skb between that point at the actual xmit. Do you see the "Bad TX len" warning in these cases?

I double checked and so far I can’t find evidence of it. Probably we still want to keep this change, but it’s not the root of our problems.

Quoted 7 lines
Quoted 4 lines

We started also seeing kernel
crashes along with the bad firmware symptoms, walked through ~110 kdumps and found i2c locks that were held by 2 owners (eeprom reading and the MCTP TX queue).

Just to clarify my understanding of the state: "being held by two owners" would indicate a violation of the lock itself. Or is it that there are two threads blocked waiting to acquire the mutex?

I think it’s actually this, 2 threads are waiting on acquiring the lock. There was a theory that it was a lock underflow that allowed 2 threads to acquire the lock that lead to this patch.

Quoted 4 lines

For NVMe-MI, you're likely using manual tag allocation, where the tag allocation (and hence flow state) is entirely controlled by userspace. It may be that the NVMe protocol-level errors are causing that tags to be held for long durations, perhaps?

Yeah, this is very plausible given the device(s) stop responding correctly. I imagine we are getting stuck with manual allocations and not releasing locks. Can we reset the state machine back to NEW instead of holding the lock?

Quoted 5 lines

Cheers,

Jeremy

maintainer
#5 lore raw

Hi William,

Quoted 4 lines
Quoted 3 lines

Just to clarify my understanding of the state: "being held by two owners" would indicate a violation of the lock itself. Or is it that there are two threads blocked waiting to acquire the mutex?

I think it’s actually this, 2 threads are waiting on acquiring the lock.

OK, that's good news!

Quoted 8 lines

There was a theory that it was a lock underflow that allowed 2 threads to acquire the lock that lead to this patch.

Quoted 4 lines

For NVMe-MI, you're likely using manual tag allocation, where the tag allocation (and hence flow state) is entirely controlled by userspace. It may be that the NVMe protocol-level errors are causing that tags to be held for long durations, perhaps?

Quoted 4 lines

Yeah, this is very plausible given the device(s) stop responding correctly. I imagine we are getting stuck with manual allocations and not releasing locks. Can we reset the state machine back to NEW instead of holding the lock?

Not sure what you're referring to here; if the userspace application is not releasing the tag, we have to keep the i2c bus locked, otherwise we may not receive a response from the device.

The one case I can think of (in upstream infrastructure, at least) is that this might be triggered by the device reporting a long MPRT value, and then a response gets lost. libnvme is respecting the MPRT, and not releasing the tag for that (excessive) duration.

However, the tag -> i2c lock associations are only useful if you have muxes in the i2c topology. Is that the case on your platform? If not, perhaps we could elide all the bus locking when we can detect that...

Cheers,

Jeremy

#6 lore raw
Quoted 15 lines · Jeremy Kerr

Hi William,

Quoted 4 lines
Quoted 3 lines

Just to clarify my understanding of the state: "being held by two owners" would indicate a violation of the lock itself. Or is it that there are two threads blocked waiting to acquire the mutex?

I think it’s actually this, 2 threads are waiting on acquiring the lock.

OK, that's good news!

Quoted 7 lines

There was a theory that it was a lock underflow that allowed 2 threads to acquire the lock that lead to this patch.

Quoted 4 lines

For NVMe-MI, you're likely using manual tag allocation, where the tag allocation (and hence flow state) is entirely controlled by userspace. It may be that the NVMe protocol-level errors are causing that tags to be held for long durations, perhaps?

Quoted 7 lines
Quoted 4 lines

Yeah, this is very plausible given the device(s) stop responding correctly. I imagine we are getting stuck with manual allocations and not releasing locks. Can we reset the state machine back to NEW instead of holding the lock?

Not sure what you're referring to here; if the userspace application is not releasing the tag, we have to keep the i2c bus locked, otherwise we may not receive a response from the device.

Isn't this inherently an approach asking for trouble, where a potentially buggy userspace can starve out other applications which need to access the bus. For us we have FRU devices on the that are periodically rescanned or accessed for various reasons alongside the MCTP endpoint on the NVME device.

Quoted 4 lines

The one case I can think of (in upstream infrastructure, at least) is that this might be triggered by the device reporting a long MPRT value, and then a response gets lost. libnvme is respecting the MPRT, and not releasing the tag for that (excessive) duration.

Yeah, I'll have to look at the specific firmware bug more but I don't think it's been oot caused it fully yet.

Quoted 3 lines

However, the tag -> i2c lock associations are only useful if you have muxes in the i2c topology. Is that the case on your platform? If not, perhaps we could elide all the bus locking when we can detect that...

We have at least 1 layer of mux before each NVME and FRU device.

Quoted 4 lines

Cheers,

Jeremy