Skip to contentSkip to conversation

sunrpc: prevent out-of-bounds read in __cache_seq_start()

PATCH In review Chuck Lever 4 participants 4 messages Apr 21 – May 26

Subsystems (3)
lore
Share & tools
lore

Chuck Lever

author maintainer
#0 lore raw

PATCH 1/1 sunrpc: prevent out-of-bounds read in __cache_seq_start()

From: Chuck Lever <[email protected]>

Commit 7b546bd89975 ("sunrpc/cache: improve RCU safety in cache_list walking.") changed the tail of __cache_seq_start() to unconditionally store

	*pos = ((long long)hash << 32) + 1

before returning, dropping a prior "hash >= cd->hash_size" guard. When the while loop exits because every remaining bucket was empty, hash equals cd->hash_size, so the stored *pos is one position past the table's last valid bucket. seq_read_iter() caches that index in m->index. A subsequent pread(2) at the same file offset skips traverse() and hands the stored index back to __cache_seq_start(), which decodes hash = cd->hash_size and dereferences cd->hash_table[cd->hash_size] -- one hlist_head past the end of the kzalloc'd table.

KASAN reports an eight-byte slab-out-of-bounds read at the tail of the 2048-byte hash_table allocation for the NFSD export cache (EXPORT_HASHMAX * sizeof(struct hlist_head) == 256 * 8).

Reject an input hash that is out of range before touching the hash table. cache_seq_next() already bounds-checks its own loop; the start routine needs to be symmetric.

Reported-by [email protected]
Fixes 7b546bd89975 ("sunrpc/cache: improve RCU safety in cache_list walking.")
---
 net/sunrpc/cache.c | 3 +++
 1 file changed, 3 insertions(+)
net/sunrpc/cache.c +3 -0
index 305c6e67f052..391037f15292 100644
@@ -1352,6 +1352,9 @@ static void *__cache_seq_start(struct seq_file *m, loff_t *pos)
 	hash = n >> 32;
 	entry = n & ((1LL<<32) - 1);
 
+	if (hash >= cd->hash_size)
+		return NULL;
+
 	hlist_for_each_entry_rcu(ch, &cd->hash_table[hash], cache_list)
 		if (!entry--)
 			return ch;
Signature · 1 line
2.53.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 24 lines · Chuck Lever

From: Chuck Lever <[email protected]>

Commit 7b546bd89975 ("sunrpc/cache: improve RCU safety in cache_list walking.") changed the tail of __cache_seq_start() to unconditionally store

	*pos = ((long long)hash << 32) + 1

before returning, dropping a prior "hash >= cd->hash_size" guard. When the while loop exits because every remaining bucket was empty, hash equals cd->hash_size, so the stored *pos is one position past the table's last valid bucket. seq_read_iter() caches that index in m->index. A subsequent pread(2) at the same file offset skips traverse() and hands the stored index back to __cache_seq_start(), which decodes hash = cd->hash_size and dereferences cd->hash_table[cd->hash_size] -- one hlist_head past the end of the kzalloc'd table.

KASAN reports an eight-byte slab-out-of-bounds read at the tail of the 2048-byte hash_table allocation for the NFSD export cache (EXPORT_HASHMAX * sizeof(struct hlist_head) == 256 * 8).

Quoted 8 lines

Reject an input hash that is out of range before touching the hash table. cache_seq_next() already bounds-checks its own loop; the start routine needs to be symmetric.

Reported-by [email protected]
Fixes 7b546bd89975 ("sunrpc/cache: improve RCU safety in cache_list walking.")

Thanks!!
We were hunting for this one. You'll probably get a Tested-by shortly.

Ben

Reviewed-by on v1 1/1
#2 lore raw
In reply to Chuck Lever
Quoted 24 lines · Chuck Lever

From: Chuck Lever <[email protected]>

Commit 7b546bd89975 ("sunrpc/cache: improve RCU safety in cache_list walking.") changed the tail of __cache_seq_start() to unconditionally store

	*pos = ((long long)hash << 32) + 1

before returning, dropping a prior "hash >= cd->hash_size" guard. When the while loop exits because every remaining bucket was empty, hash equals cd->hash_size, so the stored *pos is one position past the table's last valid bucket. seq_read_iter() caches that index in m->index. A subsequent pread(2) at the same file offset skips traverse() and hands the stored index back to __cache_seq_start(), which decodes hash = cd->hash_size and dereferences cd->hash_table[cd->hash_size] -- one hlist_head past the end of the kzalloc'd table.

KASAN reports an eight-byte slab-out-of-bounds read at the tail of the 2048-byte hash_table allocation for the NFSD export cache (EXPORT_HASHMAX * sizeof(struct hlist_head) == 256 * 8).

Quoted 8 lines

Reject an input hash that is out of range before touching the hash table. cache_seq_next() already bounds-checks its own loop; the start routine needs to be symmetric.

Reported-by [email protected]
Fixes 7b546bd89975 ("sunrpc/cache: improve RCU safety in cache_list walking.")

Thanks for finding and fixing this.
We could of course avoid ever storing a too-large pos but adding back a test for hash at the end of __cache_seq_start() but I prefer the approach you took as it is more robust.

Thanks,
NeilBrown

Quoted 22 lines
---
 net/sunrpc/cache.c | 3 +++
 1 file changed, 3 insertions(+)
net/sunrpc/cache.c +3 -0
index 305c6e67f052..391037f15292 100644
@@ -1352,6 +1352,9 @@ static void *__cache_seq_start(struct seq_file *m, loff_t *pos)
 	hash = n >> 32;
 	entry = n & ((1LL<<32) - 1);
 
+	if (hash >= cd->hash_size)
+		return NULL;
+
 	hlist_for_each_entry_rcu(ch, &cd->hash_table[hash], cache_list)
 		if (!entry--)
 			return ch;
Signature · 1 line
2.53.0
Tested-by Harshit Mogalapalli Tested-by on v1 1/1
#3 lore raw
In reply to Chuck Lever
Quoted 24 lines

From: Chuck Lever <[email protected]>

Commit 7b546bd89975 ("sunrpc/cache: improve RCU safety in cache_list walking.") changed the tail of __cache_seq_start() to unconditionally store

	*pos = ((long long)hash << 32) + 1

before returning, dropping a prior "hash >= cd->hash_size" guard. When the while loop exits because every remaining bucket was empty, hash equals cd->hash_size, so the stored *pos is one position past the table's last valid bucket. seq_read_iter() caches that index in m->index. A subsequent pread(2) at the same file offset skips traverse() and hands the stored index back to __cache_seq_start(), which decodes hash = cd->hash_size and dereferences cd->hash_table[cd->hash_size] -- one hlist_head past the end of the kzalloc'd table.

KASAN reports an eight-byte slab-out-of-bounds read at the tail of the 2048-byte hash_table allocation for the NFSD export cache (EXPORT_HASHMAX * sizeof(struct hlist_head) == 256 * 8).

Quoted 8 lines

Reject an input hash that is out of range before touching the hash table. cache_seq_next() already bounds-checks its own loop; the start routine needs to be symmetric.

Reported-by [email protected]
Fixes 7b546bd89975 ("sunrpc/cache: improve RCU safety in cache_list walking.")

This looks good to me, Chuck. Fixes the frequent sosreport crashes in cache_check_rcu we've been seeing, even with Erkun's revert applied.

thanks very much.

cheers,
c.

Quoted 21 lines
---
 net/sunrpc/cache.c | 3 +++
 1 file changed, 3 insertions(+)
net/sunrpc/cache.c +3 -0
index 305c6e67f052..391037f15292 100644
@@ -1352,6 +1352,9 @@ static void *__cache_seq_start(struct seq_file *m, loff_t *pos)
 	hash = n >> 32;
 	entry = n & ((1LL<<32) - 1);
 
+	if (hash >= cd->hash_size)
+		return NULL;
+
 	hlist_for_each_entry_rcu(ch, &cd->hash_table[hash], cache_list)
 		if (!entry--)
 			return ch;
Signature · 1 line
2.53.0