Skip to contentSkip to conversation

nft: cache: Fix iptables-save segfault under stress

PATCH In review Phil Sutter 2 participants 2 messages Mar 13, 2020 – Mar 15, 2020 netfilter

lore
Share & tools
lore

Phil Sutter

author reviewer
#0 lore raw

PATCH 1/1 nft: cache: Fix iptables-save segfault under stress

If kernel ruleset is constantly changing, code called by nft_is_table_compatible() may crash: For each item in table's chain list, nft_is_chain_compatible() is called. This in turn calls nft_build_cache() to fetch chain's rules. Though if kernel genid has changed meanwhile, cache is flushed and rebuilt from scratch, thereby freeing table's chain list - the foreach loop in nft_is_table_compatible() then operates on freed memory.

A simple reproducer (may need a few calls):

| RULESET='*filter
| :INPUT ACCEPT [10517:1483527]
| :FORWARD ACCEPT [0:0]
| :OUTPUT ACCEPT [1714:105671]
| COMMIT
| '
|
| for ((i = 0; i < 100; i++)); do
|         iptables-nft-restore <<< "$RULESET" &
| done &
| iptables-nft-save

To fix the problem, basically revert commit ab1cd3b510fa5 ("nft: ensure cache consistency") so that __nft_build_cache() no longer flushes the cache. Instead just record kernel's genid when fetching for the first time. If kernel rule set changes until the changes are committed, the commit simply fails and local cache is being rebuilt.

---
Pablo,

This is a similar situation as with commit c550c81fd373e ("nft: cache:
Fix nft_release_cache() under stress"): Your caching rewrite avoids this problem as well, but kills some of my performance improvements. I'm currently working on restoring them with your approach but that's not a quick "fix". Can we go with this patch until I'm done?

Cheers, Phil

---
 iptables/nft-cache.c | 16 ++--------------
 1 file changed, 2 insertions(+), 14 deletions(-)
iptables/nft-cache.c +2 -14
index e3c9655739187..a0c76705c848e 100644
@@ -449,15 +449,11 @@ __nft_build_cache(struct nft_handle *h, enum nft_cache_level level,
 		  const struct builtin_table *t, const char *set,
 		  const char *chain)
 {
-	uint32_t genid_start, genid_stop;
-
 	if (level <= h->cache_level)
 		return;
-retry:
-	mnl_genid_get(h, &genid_start);
 
-	if (h->cache_level && genid_start != h->nft_genid)
-		flush_chain_cache(h, NULL);
+	if (!h->nft_genid)
+		mnl_genid_get(h, &h->nft_genid);
 
 	switch (h->cache_level) {
 	case NFT_CL_NONE:
@@ -486,18 +482,10 @@ retry:
 		break;
 	}
 
-	mnl_genid_get(h, &genid_stop);
-	if (genid_start != genid_stop) {
-		flush_chain_cache(h, NULL);
-		goto retry;
-	}
-
 	if (!t && !chain)
 		h->cache_level = level;
 	else if (h->cache_level < NFT_CL_TABLES)
 		h->cache_level = NFT_CL_TABLES;
-
-	h->nft_genid = genid_start;
 }
 
 void nft_build_cache(struct nft_handle *h, struct nftnl_chain *c)
Signature · 1 line
2.25.1
#1 lore raw
Quoted 29 lines · Phil Sutter

If kernel ruleset is constantly changing, code called by nft_is_table_compatible() may crash: For each item in table's chain list, nft_is_chain_compatible() is called. This in turn calls nft_build_cache() to fetch chain's rules. Though if kernel genid has changed meanwhile, cache is flushed and rebuilt from scratch, thereby freeing table's chain list - the foreach loop in nft_is_table_compatible() then operates on freed memory.

A simple reproducer (may need a few calls):

| RULESET='*filter
| :INPUT ACCEPT [10517:1483527]
| :FORWARD ACCEPT [0:0]
| :OUTPUT ACCEPT [1714:105671]
| COMMIT
| '
|
| for ((i = 0; i < 100; i++)); do
|         iptables-nft-restore <<< "$RULESET" &
| done &
| iptables-nft-save

To fix the problem, basically revert commit ab1cd3b510fa5 ("nft: ensure cache consistency") so that __nft_build_cache() no longer flushes the cache. Instead just record kernel's genid when fetching for the first time. If kernel rule set changes until the changes are committed, the commit simply fails and local cache is being rebuilt.

Quoted 8 lines

---
Pablo,

This is a similar situation as with commit c550c81fd373e ("nft: cache:
Fix nft_release_cache() under stress"): Your caching rewrite avoids this problem as well, but kills some of my performance improvements. I'm currently working on restoring them with your approach but that's not a quick "fix". Can we go with this patch until I'm done?

Fair enough.