ipset swap to nftables set
hi,
i'm using in production ipset swap with ~3000 ip addr
i.e.
- change coming from customer (new ips, old ips deprecated)
- generation of new list /etc/sysconfig/ipset-new
- swap ipset-new with ipset
transaction is near real-time
now i'm moving from RHEL7 to RHEL9 (rocky, kernel 5.14, nftables 1.0.4) and trying move from ipset to nftables sets
nft add set ip filter blackhole { type ipv4_addr\; comment \"drop all packets from these hosts\" \; } fill the set
now the new flow is
- change coming from customer (new ips, old ips deprecated)
- nft flush set ip filter blackhole
- nft add element ip filter blackhole { X } (bash for loop)
it tooks 10sec
can you recommend better way? (performance similar to ipset swap)
thanks
Marek
Quoted 3 lines · marek
- nft flush set ip filter blackhole
- nft add element ip filter blackhole { X } (bash for loop)
Hi Marek,
Do that last step atomically and it should be plenty fast.
nft flush set ip filter blackhole
nft add element ip filter blackhole {\
1.0.0.1, \
1.0.0.2, \
1.0.0.3, \
1.1.1.1, \
1.1.1.2, \
1.1.1.3, \
1.193.146.35, \
...
99.99.99.99, \
}Eric
Am 29.09.23 um 16:30 schrieb Eric:
Quoted 11 lines · marek
Quoted 3 lines
- nft flush set ip filter blackhole
- nft add element ip filter blackhole { X } (bash for loop)
Hi Marek,
Do that last step atomically and it should be plenty fast.
nft flush set ip filter blackhole
nft add element ip filter blackhole {\
Quoted 10 lines
1.0.0.1, \
1.0.0.2, \
1.0.0.3, \
1.1.1.1, \
1.1.1.2, \
1.1.1.3, \
1.193.146.35, \
...
99.99.99.99, \
}still horrible ciomoared to ipset swap beause it is *not* atomically
Quoted 23 lines · Reindl Harald
Am 29.09.23 um 16:30 schrieb Eric:
Quoted 11 lines · marek
Quoted 3 lines
- nft flush set ip filter blackhole
- nft add element ip filter blackhole { X } (bash for loop)
Hi Marek,
Do that last step atomically and it should be plenty fast.
nft flush set ip filter blackhole
nft add element ip filter blackhole {\Quoted 10 lines
1.0.0.1, \ 1.0.0.2, \ 1.0.0.3, \ 1.1.1.1, \ 1.1.1.2, \ 1.1.1.3, \ 1.193.146.35, \ ... 99.99.99.99, \ }
Quoted 1 line
still horrible ciomoared to ipset swap beause it is *not* atomically
It should be atomic, provided that the command stream conveyed to a single invocation of nft includes both the flush and add command. That's how I've been going about it for years now.
Here is an example, in which a list of newline-delimited IPv4 addresses is consumed from a file named "ipv4-list.txt".
{
set="ip filter blackhole"
echo "flush set $set"
echo "add element $set {"
sed -e 's/$/,/'
echo "}"
} < ipv4-list.txt | nft -f -
Signature · 1 line
Kerin Millar
Quoted 22 lines · marek
hi,
i'm using in production ipset swap with ~3000 ip addr
i.e.
- change coming from customer (new ips, old ips deprecated)
- generation of new list
/etc/sysconfig/ipset-new- swap ipset-new with ipset
transaction is near real-time
now i'm moving from RHEL7 to RHEL9 (rocky, kernel 5.14, nftables 1.0.4) and trying move from ipset to nftables sets
nft add set ip filter blackhole { type ipv4_addr\; comment \"drop all packets from these hosts\" \; }
fill the set
Quoted 7 lines
now the new flow is
- change coming from customer (new ips, old ips deprecated)
- nft flush set ip filter blackhole
- nft add element ip filter blackhole { X } (bash for loop)
Executing nft repeatedly will certainly be slow, if that is what you are currently doing. I'd be interested to know what you are using for to iterate over precisely because there's a fair chance that your use of for is an anti-pattern to begin with.
Quoted 5 lines
it tooks 10sec
can you recommend better way? (performance similar to ipset swap)
Yes, definitely. However, I would prefer to see some of the existing code first. This will make it easier to provide clear guidance as to how to modify your script. In particular, please make it clear where the input data comes from, and the precise format of this data.
Signature · 1 line
Kerin Millar
Dne 2023-09-29 v 16:41 Kerin Millar napsal(a):
Quoted 29 lines · marek
Quoted 28 lines
hi,
i'm using in production ipset swap with ~3000 ip addr
i.e.
- change coming from customer (new ips, old ips deprecated)
- generation of new list
/etc/sysconfig/ipset-new- swap ipset-new with ipset
transaction is near real-time
now i'm moving from RHEL7 to RHEL9 (rocky, kernel 5.14, nftables 1.0.4) and trying move from ipset to nftables sets
nft add set ip filter blackhole { type ipv4_addr\; comment \"drop all packets from these hosts\" \; }
fill the setnow the new flow is
- change coming from customer (new ips, old ips deprecated)
- nft flush set ip filter blackhole
Quoted 8 lines
Quoted 1 line
- nft add element ip filter blackhole { X } (bash for loop)
Executing nft repeatedly will certainly be slow, if that is what you are currently doing. I'd be interested to know what you are using for to iterate over precisely because there's a fair chance that your use of for is an anti-pattern to begin with.
Quoted 4 lines
it tooks 10sec
can you recommend better way? (performance similar to ipset swap)
Yes, definitely. However, I would prefer to see some of the existing code first. This will make it easier to provide clear guidance as to how to modify your script. In particular, please make it clear where the input data comes from, and the precise format of this data.
i'll try tip from previous mail
current version is very simple loop like
cat file | while read i
do
nft add element ip filter blackhole { $i }
done
"i" is ipv4 /32 ip address
Marek
Am 29.09.23 um 21:47 schrieb marek:
Quoted 47 lines
Dne 2023-09-29 v 16:41 Kerin Millar napsal(a):
Quoted 29 lines · marek
Quoted 28 lines
hi,
i'm using in production ipset swap with ~3000 ip addr
i.e.
- change coming from customer (new ips, old ips deprecated)
- generation of new list
/etc/sysconfig/ipset-new- swap ipset-new with ipset
transaction is near real-time
now i'm moving from RHEL7 to RHEL9 (rocky, kernel 5.14, nftables 1.0.4) and trying move from ipset to nftables sets
nft add set ip filter blackhole { type ipv4_addr\; comment \"drop all packets from these hosts\" \; }
fill the setnow the new flow is
- change coming from customer (new ips, old ips deprecated)
- nft flush set ip filter blackhole
Quoted 8 lines
Quoted 1 line
- nft add element ip filter blackhole { X } (bash for loop)
Executing nft repeatedly will certainly be slow, if that is what you are currently doing. I'd be interested to know what you are using for to iterate over precisely because there's a fair chance that your use of for is an anti-pattern to begin with.
Quoted 4 lines
it tooks 10sec
can you recommend better way? (performance similar to ipset swap)
Yes, definitely. However, I would prefer to see some of the existing code first. This will make it easier to provide clear guidance as to how to modify your script. In particular, please make it clear where the input data comes from, and the precise format of this data.
i'll try tip from previous mail
current version is very simple loop like
cat file | while read i
Quoted 7 lines
do
nft add element ip filter blackhole { $i }
done
"i" is ipv4 /32 ip address
this is always a problem - with a list of 10k records you call a binary 10k times - you have the same difference between "iptables restore" and call iptables thousands of times
i disklike nftables in general and especially because oyu can't have you ipsets completly seperate and load the atomically at every point in time
hopefully iptables-nft will last forever