nft equivalent of "ipset test"
The "ipset" commandline tool has the "test" command
for testing whether a given item (ie. an IP) is in a given set. Is there an equivalent for the "nft" commandline tool of nftables? I unfortunately couldn't find the answer in the manpage of nft.
Thx
Florian Westphal
maintainerQuoted 4 lines · U.Mutlu
The "ipset" commandline tool has the "test" command
for testing whether a given item (ie. an IP) is in a given set. Is there an equivalent for the "nft" commandline tool of nftables? I unfortunately couldn't find the answer in the manpage of nft.
nft "get element inet tablename setname { 1.2.3.4 }"
Florian Westphal wrote on 10/17/23 23:35:
Quoted 7 lines
Quoted 4 lines · U.Mutlu
The "ipset" commandline tool has the "test" command
for testing whether a given item (ie. an IP) is in a given set. Is there an equivalent for the "nft" commandline tool of nftables? I unfortunately couldn't find the answer in the manpage of nft.nft "get element inet tablename setname { 1.2.3.4 }"
But isn't that printing the whole item on stdout? I just need to quickly test it only,
ie. need just a return code of 0 or 1, or so, for use in a shell script (bash).
Florian Westphal
maintainerQuoted 7 lines · U.Mutlu
Florian Westphal wrote on 10/17/23 23:35:
Quoted 6 lines
Quoted 4 lines · U.Mutlu
The "ipset" commandline tool has the "test" command
for testing whether a given item (ie. an IP) is in a given set. Is there an equivalent for the "nft" commandline tool of nftables? I unfortunately couldn't find the answer in the manpage of nft.
Quoted 6 lines
Quoted 1 line
nft "get element inet tablename setname { 1.2.3.4 }"
But isn't that printing the whole item on stdout? I just need to quickly test it only,
ie. need just a return code of 0 or 1, or so, for use in a shell script (bash).
?
nft "get element inet t s { 1.2.3.4 }" > /dev/null 2>&1; echo $? 1
nft "add element inet t s { 1.2.3.4 }"
nft "get element inet t s { 1.2.3.4 }" > /dev/null 2>&1; echo $? 0
Florian Westphal wrote on 10/18/23 00:05:
Quoted 15 lines
Quoted 7 lines · U.Mutlu
Florian Westphal wrote on 10/17/23 23:35:
Quoted 6 lines
Quoted 4 lines · U.Mutlu
The "ipset" commandline tool has the "test" command
for testing whether a given item (ie. an IP) is in a given set. Is there an equivalent for the "nft" commandline tool of nftables? I unfortunately couldn't find the answer in the manpage of nft.Quoted 6 lines
Quoted 1 line
nft "get element inet tablename setname { 1.2.3.4 }"
But isn't that printing the whole item on stdout? I just need to quickly test it only,
ie. need just a return code of 0 or 1, or so, for use in a shell script (bash).
Quoted 7 lines
?
nft "get element inet t s { 1.2.3.4 }" >
/dev/null2>&1; echo $? 1
nft "add element inet t s { 1.2.3.4 }"
nft "get element inet t s { 1.2.3.4 }" >/dev/null2>&1; echo $? 0
Actualy I need to do this monster: :-)
IP="1.2.3.4"
! nft "get element inet mytable myset { $IP }" > /dev/null 2>&1 && \
! nft "get element inet mytable myset2 { $IP }" > /dev/null 2>&1 && \
nft "add element inet mytable myset { $IP }"
Ie. add it to the set myset only if it's not already present in any of myset and myset2.
A true "test" command w/o any output, much like in "ipset test", would be a better method, IMO.
I've not switched yet to nftables, just (dry-) evaluating it.
Quoted 7 lines · On Wed, 18 Oct 2023 00:36:37 +0200 "U.Mutlu" <[email protected]> wrote:
...
Actualy I need to do this monster: :-)IP="1.2.3.4"
! nft "get element inet mytable myset { $IP }" >/dev/null2>&1 && \
! nft "get element inet mytable myset2 { $IP }" >/dev/null2>&1 && \
nft "add element inet mytable myset { $IP }"
Try using '||', akin to:
----
mkdir aaaa; cd aaaa
touch a b
(
ls a || \
ls b
) >/dev/null 2>&1 || \
echo "not found"
(
ls c || \
ls b
) >/dev/null 2>&1 || \
echo "not found"
(
ls a || \
ls d
) >/dev/null 2>&1 || \
echo "not found"
(
ls c || \
ls d
) >/dev/null 2>&1 || \
echo "not found"
cd ..
rm -rf aaaa
----
Only if neither file is found will the echo execute.
So this should do the trick for you:
----
(
nft "get element inet mytable myset { $IP }" || \
nft "get element inet mytable myset2 { $IP }"
) >/dev/null 2>&1 || \
nft "add element inet mytable myset { $IP }"
----
N
Pablo Neira Ayuso
maintainerQuoted 24 lines · U.Mutlu
Florian Westphal wrote on 10/18/23 00:05:
Quoted 15 lines
Quoted 7 lines · U.Mutlu
Florian Westphal wrote on 10/17/23 23:35:
Quoted 6 lines
Quoted 4 lines · U.Mutlu
The "ipset" commandline tool has the "test" command
for testing whether a given item (ie. an IP) is in a given set. Is there an equivalent for the "nft" commandline tool of nftables? I unfortunately couldn't find the answer in the manpage of nft.Quoted 6 lines
Quoted 1 line
nft "get element inet tablename setname { 1.2.3.4 }"
But isn't that printing the whole item on stdout? I just need to quickly test it only,
ie. need just a return code of 0 or 1, or so, for use in a shell script (bash).Quoted 7 lines
?
nft "get element inet t s { 1.2.3.4 }" >
/dev/null2>&1; echo $? 1
nft "add element inet t s { 1.2.3.4 }"
nft "get element inet t s { 1.2.3.4 }" >/dev/null2>&1; echo $? 0
Quoted 6 lines
Actualy I need to do this monster: :-)
IP="1.2.3.4"
! nft "get element inet mytable myset { $IP }" >/dev/null2>&1 && \
! nft "get element inet mytable myset2 { $IP }" >/dev/null2>&1 && \
nft "add element inet mytable myset { $IP }"
Use 'nft create element' if you want to fail if element already exists.
Quoted 7 lines
Ie. add it to the set myset only if it's not already present in any of myset and myset2.
A true "test" command w/o any output, much like in "ipset test", would be a better method, IMO.
I've not switched yet to nftables, just (dry-) evaluating it.
Pablo Neira Ayuso
maintainerQuoted 6 lines · On Tue, Oct 17, 2023 at 08:00:57PM -0400, [email protected] wrote:
Quoted 3 lines · On Wed, 18 Oct 2023 00:36:37 +0200 "U.Mutlu" <[email protected]> wrote:
...
Actualy I need to do this monster: :-)
Quoted 6 lines
Quoted 4 lines
IP="1.2.3.4"
! nft "get element inet mytable myset { $IP }" >/dev/null2>&1 && \
! nft "get element inet mytable myset2 { $IP }" >/dev/null2>&1 && \
nft "add element inet mytable myset { $IP }"Try using '||', akin to:
Please, use 'nft create' for this, no need for an explicit test and then add from command line.
The idiom above is an antipattern, because it is not atomic, the 'create' command provides a way to first test if the element exists (if so it fails) then add it.
Pablo Neira Ayuso wrote on 10/18/23 11:36:
Quoted 12 lines · On Tue, Oct 17, 2023 at 08:00:57PM -0400, [email protected] wrote:
Quoted 11 lines
Quoted 7 lines · On Wed, 18 Oct 2023 00:36:37 +0200 "U.Mutlu" <[email protected]> wrote:
...
Actualy I need to do this monster: :-)IP="1.2.3.4"
! nft "get element inet mytable myset { $IP }" >/dev/null2>&1 && \
! nft "get element inet mytable myset2 { $IP }" >/dev/null2>&1 && \
nft "add element inet mytable myset { $IP }"
Quoted 8 lines
Quoted 1 line
Try using '||', akin to:
Please, use 'nft create' for this, no need for an explicit test and then add from command line.
The idiom above is an antipattern, because it is not atomic, the 'create' command provides a way to first test if the element exists (if so it fails) then add it.
Pablo, unfortunately your solution with 'create' cannot be used in my above said special use-case of testing first in BOTH sets...
I just don't understand why the author cannot simply add a real 'test' function to the nft tool... The logic is already in 'get element' and also in your 'create' method.
PS: I'm not yet subscribed to netfilter-dev, so CC may fail...
Pablo Neira Ayuso
maintainerQuoted 18 lines · U.Mutlu
Pablo Neira Ayuso wrote on 10/18/23 11:36:
Quoted 17 lines · On Tue, Oct 17, 2023 at 08:00:57PM -0400, [email protected] wrote:
Quoted 6 lines
Quoted 3 lines · On Wed, 18 Oct 2023 00:36:37 +0200 "U.Mutlu" <[email protected]> wrote:
...
Actualy I need to do this monster: :-)Quoted 6 lines
Quoted 4 lines
IP="1.2.3.4"
! nft "get element inet mytable myset { $IP }" >/dev/null2>&1 && \
! nft "get element inet mytable myset2 { $IP }" >/dev/null2>&1 && \
nft "add element inet mytable myset { $IP }"Try using '||', akin to:
Please, use 'nft create' for this, no need for an explicit test and then add from command line.
Quoted 6 lines
Quoted 3 lines
The idiom above is an antipattern, because it is not atomic, the 'create' command provides a way to first test if the element exists (if so it fails) then add it.
Pablo, unfortunately your solution with 'create' cannot be used in my above said special use-case of testing first in BOTH sets...
'ipset test' also requires a set to be specified.
Quoted 2 lines
I just don't understand why the author cannot simply add a real 'test' function to the nft tool...
I just don't understand your usecase :-), why do you need this atomic check on two different sets?
Could you explain your ruleset in more detail?
Quoted 1 line
The logic is already in 'get element' and also in your 'create' method.
Pablo Neira Ayuso wrote on 10/18/23 12:00:
Quoted 19 lines · U.Mutlu
Quoted 18 lines
Pablo Neira Ayuso wrote on 10/18/23 11:36:
Quoted 17 lines · On Tue, Oct 17, 2023 at 08:00:57PM -0400, [email protected] wrote:
Quoted 6 lines
Quoted 3 lines · On Wed, 18 Oct 2023 00:36:37 +0200 "U.Mutlu" <[email protected]> wrote:
...
Actualy I need to do this monster: :-)Quoted 6 lines
Quoted 4 lines
IP="1.2.3.4"
! nft "get element inet mytable myset { $IP }" >/dev/null2>&1 && \
! nft "get element inet mytable myset2 { $IP }" >/dev/null2>&1 && \
nft "add element inet mytable myset { $IP }"Try using '||', akin to:
Please, use 'nft create' for this, no need for an explicit test and then add from command line.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The idiom above is an antipattern, because it is not atomic, the 'create' command provides a way to first test if the element exists (if so it fails) then add it.
Pablo, unfortunately your solution with 'create' cannot be used in my above said special use-case of testing first in BOTH sets...
'ipset test' also requires a set to be specified.
Of course, what else! :-) And I haven't said anything different.
Quoted 7 lines
Quoted 2 lines
I just don't understand why the author cannot simply add a real 'test' function to the nft tool...
I just don't understand your usecase :-), why do you need this atomic check on two different sets?
Could you explain your ruleset in more detail?
It's maybe complicated: I've a restrictive firewall where the default is to block all ports for traffic from INPUT, except those explicitly allowed.
Then, at the end of the firewall rules I can _auto-add_ all the unhandled IPs to such a set for blocking. The blocking happens at top by testing whether the IP is in that set.
But another feature of this solution is that not only
the firewall can (auto-) add IPs to the set for blocking, but also the external handlers after ACCEPT can do it,
ie. say a mailserver. It too has to be able to add an IP to the same set for blocking. The blockign itself happens centrally in the firewall script at the next attempt of the attacker.
Lately I've extended this to make it a 2-stage: if blocked IP
continues sending more than x packets while in timeout of y minutes, then add this attacker to the second set that has a much higher timeout of z minutes. One additional practical benefit of this approach is that
now one sees the hardcore attackers grouped (they are those in set2).
The correct managing of these two sets requires the said atomicity by testing of BOTH sets before adding the IP to the first set...
If you have got a better solution for this use-case,
so let me/us know please. As said I'm new even to ipset
but which I find very effective & useful in practice.
As said I'm still continuing using iptables with ipset, just evaluating whether it would make sense to switch to nftables/nft, though the learning-curve seems not that small, IMO.
Pablo Neira Ayuso
maintainerQuoted 2 lines · U.Mutlu
Pablo Neira Ayuso wrote on 10/18/23 12:00:
Quoted 1 line · U.Mutlu
[...]
Quoted 11 lines
Quoted 7 lines
Quoted 2 lines
I just don't understand why the author cannot simply add a real 'test' function to the nft tool...
I just don't understand your usecase :-), why do you need this atomic check on two different sets?
Could you explain your ruleset in more detail?
It's maybe complicated: I've a restrictive firewall where the default is to block all ports for traffic from INPUT, except those explicitly allowed.
INPUT is late if you know what ports you want to allow, use netdev/ingress instead.
Quoted 3 lines
Then, at the end of the firewall rules I can _auto-add_ all the unhandled IPs to such a set for blocking. The blocking happens at top by testing whether the IP is in that set.
This should be easy to handle with a dynamic set, which allows for packet path to add elements to set.
Quoted 15 lines
But another feature of this solution is that not only
the firewall can (auto-) add IPs to the set for blocking, but also the external handlers after ACCEPT can do it,
ie. say a mailserver. It too has to be able to add an IP to the same set for blocking. The blockign itself happens centrally in the firewall script at the next attempt of the attacker.Lately I've extended this to make it a 2-stage: if blocked IP continues sending more than x packets while in timeout of y minutes, then add this attacker to the second set that has a much higher timeout of z minutes.
One additional practical benefit of this approach is that now one sees the hardcore attackers grouped (they are those in set2).
Quoted 6 lines
The correct managing of these two sets requires the said atomicity by testing of BOTH sets before adding the IP to the first set...
If you have got a better solution for this use-case, so let me/us know please. As said I'm new even to ipset but which I find very effective & useful in practice.
You should look at nftables concatenations, you do not have to split this information accross two sets in nftables. For adding entries from packet path, have a look at dynamic sets.
Two sets also means two lookups from packet path.
Quoted 6 lines
If you have got a better solution for this use-case,
so let me/us know please. As said I'm new even to ipset
but which I find very effective & useful in practice.
As said I'm still continuing using iptables with ipset, just evaluating whether it would make sense to switch to nftables/nft, though the learning-curve seems not that small, IMO.
You pick your poison. I keep listening to this argument, I think things got a lot better. There is still room to improve documentation, that I can take. I don't think it makes sense to start a firewall with iptables/ipset in 2023.
Quoted 22 lines · On Wed, 18 Oct 2023 11:54:30 +0200 "U.Mutlu" <[email protected]> wrote:
Pablo Neira Ayuso wrote on 10/18/23 11:36:
Quoted 12 lines · On Tue, Oct 17, 2023 at 08:00:57PM -0400, [email protected] wrote:
Quoted 11 lines
Quoted 7 lines · On Wed, 18 Oct 2023 00:36:37 +0200 "U.Mutlu" <[email protected]> wrote:
...
Actualy I need to do this monster: :-)IP="1.2.3.4"
! nft "get element inet mytable myset { $IP }" >/dev/null2>&1 && \
! nft "get element inet mytable myset2 { $IP }" >/dev/null2>&1 && \
nft "add element inet mytable myset { $IP }"Quoted 8 lines
Quoted 1 line
Try using '||', akin to:
Please, use 'nft create' for this, no need for an explicit test and then add from command line.
The idiom above is an antipattern, because it is not atomic, the 'create' command provides a way to first test if the element exists (if so it fails) then add it.
Quoted 5 lines
Pablo, unfortunately your solution with 'create' cannot be used in my above said special use-case of testing first in BOTH sets...
I just don't understand why the author cannot simply add a real 'test' function to the nft tool...
One a feature has been added, it usually has to be maintained forever so it is to be expected that the use case has to be strongly justified. In my opinion, the principal shortcomings of "get element" are twofold.
Firstly, there is no way to distinguish between nft(8) failing because it did not find the specified element or for some other, wholly unrelated, reason. In both cases, the exit status is likely to be 1. That makes it a poor interface. One solution could be for nft to at least promise to exit >=2 in the case of syntax errors, syscall failures etc.
Secondly, the use of "get element" entails spewing a diagnostic message to STDERR in the case that the element isn't found. The user is thus presented with the unenviable choice of silencing STDERR. This is a bad thing because, in doing so, *all* errors and dignostics will be silenced.
Now, as concerns the matter of producing monstrosities, there is always the option to write the code in a more elegant fashion. One way would be to compose a function.
in_all_sets() {
local ip=$1 set
shift
for set; do
nft -t "get element $set { $ip }" >/dev/null 2>&1 || return
done
}
if ! in_all_sets "$ip" "inet mytable myset" "inet mytable myset"; then
nft "add element inet mytable myset { $ip }"
fi
Of course, this does not address the aformentioned shortcomings of the interface but the legibility and maintainability of the code is improved.
Another would be to apply the redirections to a compound command.
{
# Neither of the following will be seen
echo "stdout"
echo "stderr" >&2
} >/dev/null 2>&1Though Pablo already mentioned it, the overall approach amounts to a TOCTOU race. The prospect of being able to atomically check for the existence of an element in multiple sets is a curious one. I, too, would be interested in understanding the underlying use case.
Signature · 1 line
Kerin Millar
Pablo Neira Ayuso wrote on 10/18/23 13:49:
Quoted 10 lines · U.Mutlu
[...]
Quoted 8 lines
Lately I've extended this to make it a 2-stage: if blocked IP continues sending more than x packets while in timeout of y minutes, then add this attacker to the second set that has a much higher timeout of z minutes.
One additional practical benefit of this approach is that now one sees the hardcore attackers grouped (they are those in set2).
Quoted 8 lines
Quoted 3 lines
The correct managing of these two sets requires the said atomicity by testing of BOTH sets before adding the IP to the first set...
You should look at nftables concatenations, you do not have to split this information accross two sets in nftables. For adding entries from packet path, have a look at dynamic sets.
Two sets also means two lookups from packet path.
But as said above, I need a seperate 2nd set anyway, to be able to see the hardcore attackers. For example for auto-generating and filing an Abuse Report to the abuse-address (WHOIS) of the owning ISP of that attacker/hacker IP.
Your other suggestions make sense, indeed, but ATM
are too advanced for me; I would need some time to
learn these advanced concepts possible in current nftables.
In the meantime iptables with ipset shall suffice for my non-HA needs. :-)
Thx.
Phil Sutter
reviewerOn Wed, Oct 18, 2023 at 01:07:07PM +0200, U.Mutlu wrote:
[...]
Quoted 9 lines
Lately I've extended this to make it a 2-stage: if blocked IP continues sending more than x packets while in timeout of y minutes, then add this attacker to the second set that has a much higher timeout of z minutes.
One additional practical benefit of this approach is that
now one sees the hardcore attackers grouped (they are those in set2).The correct managing of these two sets requires the said atomicity by testing of BOTH sets before adding the IP to the first set...
I may lack context, but IMO you may simplify your scripts/algorithm:
If a daemon (or fail2ban) identifies an IP to block, it must not have been in either of the sets (or otherwise the firewall had dropped the packet already). So they may just unconditionally add to the first set.
Also, you may get away with a single set only:
| table t {
| set ban {
| type ipv4_addr
| timeout 2s
| }
| }You may add elements like so:
| add element t ban { 1.2.3.4 timeout 5m }
I.e., override the default 2s timeout and set a larger one for individual elements. This may happen from packet path, as well:
| add rule t input ip saddr @ban update @ban { ip saddr timeout 5m }
Cheers, Phil