Skip to contentSkip to conversation

NFS4.1 migration: in-flight RPCs retried with stale session credentials to new server

BUG Wonju Ka 1 participant 1 message Jan 7

Subsystems (2)
lore
Share & tools
lore

Wonju Ka

author
#0 lore raw

Hi,

I've encountered a kernel panic during NFS4.1 transparent state migration where in-flight RPC requests are retransmitted with stale session credentials to the new (target) server after migration completes.

Environment:
- Kernel: 5.15.0-121-generic (Ubuntu)
- Protocol: NFSv4.1
- Scenario: Transparent state migration with heavy I/O load

Problem Description:
-------------------
During migration, when many WRITE operations are in-flight (16 slots fully utilized), the following sequence occurs:

1. Client detects LEASE_MOVED flag from source server
2. Client performs migration recovery (FS_LOCATIONS, CREATE_SESSION on target)
3. Migration completes - nfs_server now points to target server's nfs_client
4. rpc_xprt is switched to target server connection
5. In-flight RPCs on source connection receive NFS4ERR_BADSESSION
6. These RPCs are retried using the NEW transport (target server) but with OLD session credentials (source server's session ID)
7. Target server receives requests with foreign session ID -> "client not found"
8. Eventually leads to use-after-free panic

The core issue is that when migration switches the transport via rpc_switch_client_transport(), the pending RPC tasks retain their original session credentials. When these tasks are retried after the source connection fails, they're sent to the target server with the source server's session ID.

Timeline from logs:
------------------

07:28:57.069  LEASE_MOVED flag detected (source server)
07:28:58.251  CREATE_SESSION on target (new ClientID: 2:1767652138)
07:28:58.252  RECLAIM_COMPLETE
07:28:58.5xx  WRITEs still processing on source (session 5:1767650393:0:1)
07:28:58.5xx  New WRITEs on target (session 2:1767652138:0:1)
07:28:59.248  Target receives request with SOURCE session ID!
              ERROR: client not found {"clientId": "5:1767650393",
                                       "sessionId": "5:1767650393:0:1"}

07:28:59.489 PANIC: "refcount_t: addition on 0; use-after-free"

Evidence from target server logs:
--------------------------------
The target server (ClientID 2:1767652138) received requests with the source server's session ID (5:1767650393:0:1):

  2026-01-06 07:28:59.248 ERROR client not found
      {"clientId": "5:1767650393", "sessionId": "5:1767650393:0:1"}
  2026-01-06 07:28:59.650 ERROR client not found
      {"clientId": "5:1767650393", "sessionId": "5:1767650393:0:2"}

These requests arrived on port 59244 (target connection), not port 56768 (source connection), confirming that the transport was updated but session credentials were not.

Kernel panic trace:
------------------
refcount_t: addition on 0; use-after-free.
WARNING: CPU: 7 PID: 33665 at lib/refcount.c:25 refcount_warn_saturate+0x9f/0x150
Workqueue: rpciod rpc_async_schedule [sunrpc]
Call Trace:
 nfs4_schedule_state_manager+0x...
 nfs4_schedule_session_recovery+0x...
 nfs41_sequence_process+0x...
 nfs4_sequence_done+0x...
 nfs4_write_done+0x...

Analysis:
--------
In fs/nfs/nfs4state.c, nfs4_try_migration() switches the nfs_server to point to the new nfs_client and updates the RPC transport. However, RPC tasks that are already in-flight retain references to the old session.

When these tasks fail on the old connection and are retried, they use:
- New transport (target server) - correct
- Old session credentials - incorrect

The session ID in the SEQUENCE operation comes from the rpc_task's credentials which were set up before migration occurred.

Possible fix directions:
-----------------------
1. Drain all in-flight RPCs before completing migration
2. When retrying failed RPCs after migration, refresh session credentials
3. Mark pending RPCs for re-authentication after transport switch

This issue is related to the previously reported problem:
https://lore.kernel.org/all/CAHrBo4v_arroQ5pC2xS9PK2xsAm4O-X75zinDgKSBDSitbWuYw@mail.gmail.com/T/

I can provide full logs and additional details if needed.

Thanks,
Wonjun