DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org A7327606A32D
Precedence: list
MIME-Version: 1.0
References: <20261106072412.7909-1-kanchanapsridhar2026@gmail.com> <au5l3KEjG_mri-br@google.com>
In-Reply-To: <au5l3KEjG_mri-br@google.com>
From: "Kanchana P. Sridhar" <kanchanapsridhar2026@gmail.com>
Date: Fri, 6 Nov 2026 17:28:01 -0800
Message-ID: <CACpmpoc3R8gwkMouwyFLw9PEqyDN1i6OcmPxsvZ1d0QOPSS_fQ@mail.gmail.com>
Subject: Re: [PATCH v3 0/8] Batched zswap_store() framework with compression batching.
To: Yosry Ahmed <yosry@kernel.org>
Cc: hannes@cmpxchg.org, nphamcs@gmail.com, chengming.zhou@linux.dev, 
	akpm@linux-foundation.org, linux-mm@kvack.org, herbert@gondor.apana.org.au, 
	davem@davemloft.net, linux-crypto@vger.kernel.org, 
	linux-kernel@vger.kernel.org, rientjes@google.com, willy@infradead.org, 
	senozhatsky@chromium.org, 
	"Kanchana P. Sridhar" <kanchanapsridhar2026@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Nov 6, 2026 at 3:59=E2=80=AFPM Yosry Ahmed <yosry@kernel.org> wrote=
:
>
> On Thu, Nov 05, 2026 at 11:24:04PM -0800, Kanchana P. Sridhar wrote:
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >    usemem PMD THP BPFTRACE
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >            Baseline           zswap           zswap          Change    =
Change
> >         mm-unstable           store        compress            with    =
  with
> >          10-25-2026        batching        batching        compress  co=
mpress
> >                         (patch 6-7)       (patch 8)        batching  ba=
tching
> >                                                                (ns)    =
   %
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >   zswap_store_order0_folio (ns):
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >   p50        39,631          37,538          37,765          -1,866    =
-4.71%
> >   p95        53,088          49,690          49,349          -3,739    =
-7.04%
> >   p99        65,065          63,529          57,404          -7,661   -=
11.77%
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >   zswap_store_higher_order_folio (ns):
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >   p50    21,460,366      21,201,411      20,383,233      -1,077,133    =
-5.02%
> >   p95    25,975,341      27,558,253      25,045,350        -929,991    =
-3.58%
> >   p99    29,538,493      61,926,648      27,184,835      -2,353,658    =
-7.97%
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >   zswap_single_compress_per_page (ns):
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >   p50        37,545          39,320          37,218            -327    =
-0.87%
> >   p95        51,549          66,503          51,481             -68    =
-0.13%
> >   p99        89,866         134,854          87,091          -2,775    =
-3.09%
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >   zswap_batch_compress_per_page (ns):
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> >   p50                                        40,493           2,948    =
 7.85%
> >   p95                                        56,944           5,395    =
10.47%
> >   p99                                        76,242         -13,624   -=
15.16%
> >  =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
>
> As with previous versions, I cannot reproduce these results. I tried
> with a QEMU VM on a Cascade Lake host. I saw marginal improvements with
> patch 6 and no significant gain with patches 7-8.
>
> I noticed that you omitted high-level metrics (e.g. throughput), I
> didn't see significant changes in throughput, and I also ran a kernel
> build and saw a small regression in some cases (probably with order-0
> folios) and no significant improvement otherwise. Even if latency for
> some functions does improve slightly, it doesn't matter if it doesn't
> show up on higher-level metrics.
>
> It could be the difference in setups, or the overhead of bpftrace hooks
> for zswap_compress() might affect measured zswap_store() latencies, as
> you will have fewer zswap_compress() calls with patch 8.
>
> As advised previously, I would trim down patch 6 to the bare minimum
> (without all the scaffolding and prep work for patches 7-8), and send it
> separately (with patch 5), and we can check if it's worth adding, I am
> still concerned it might be adding overhead for order-0 to be honest.
>
> I am questioning the benefit of batching here without HW batching, so
> far the results are more concerning than encouraging. I already spent
> too much time trying to reproduce the performance gains. Also, sending
> new versions without resolving discussions on previous ones is not
> really productive.

Thanks for taking the time to test and provide feedback, Yosry.

I omitted high-level metrics like throughput since you raised a
concern about variation. Hence, I decided to go with the "ultimate
truth" which is bpftrace latency measurements.

For performance validation, I only have access to a 16G Chromebook
with a Crostini VM to which 13G is allocated. I am setting aside 3G
for swap and 7G for QEMU. Within the QEMU VM, I am selecting 2 E-cores
to run the 2 usemem processes to minimize variation due to frequency
fluctuations. This setup has its own limitations such as being unable
to fix the frequency.

Despite all this, I have been able to consistently replicate the
bpftrace latency improvements with patch 8. My results show regression
with patch 6-7, and improvement only with patch 8. As noted in the
cover-letter, patch 8 adds significant latency improvement for order-0
folios. This is with ZSTD. FWIW, these are bpftrace latency
improvements in a significantly memory constrained environment and
this ought to be a useful data point. I have been posting general
vmstat "system health" metrics to prove that nothing unexpected is
going on.

My data shows that order-0 zswap_store() tail latency improves by 11%
over mm-unstable with patch 8, although p50 is roughly similar between
patches 6-7 and 8. PMD store latency is significantly better at p50
and p99 with patch 8.

Likewise, p99 latency of zswap_compress() alone improves by 3% with
patch 8, while it degrades with patch 6-7. p50 is almost the same with
patch 8, although patch 6-7 regresses. The page-normalized
batch_compress() p99 latency is better by 15%. All comparisons are wrt
mm-unstable.

Hence, I believe even in a memory-constrained setup such as mine,
batching significantly (7-15%) improves p99 in all cases, with 4-5%
gains in p50: for order-0 and PMD folios.

In addition to the latency benefits shown through my validation, I
would like to also emphasize the strategic benefit of moving zswap to
use batching, as noted under "Conclusions" in my cover letter. IOW,
batching within large folios to improve workload metrics can only do
so much (in proportion to the # of large folios reclaimed and
contribution of zswap_store() to the overall allocation-reclaim path).
What this patch series attempts to provide is a zswap batching
framework upon which additional improvements can be built, such as
reclaiming multiple folios as a batch. I believe that compress
batching is essential to support these next steps, which improve
workload throughput/latency metrics even more than simply batching
within folios. This is with ZSTD.

Due to the environmental constraints, I haven't been able to reliably
use kernel compilation for validation.

I have been doing my best to resolve previous discussions by
incorporating the calculate_high_delay() hack that you suggested to
reduce variation. The data posted in v3 comprehends this suggestion.

I would like to suggest/request the following, as next steps:

1) Can we reconcile our methodologies for usemem bpftrace
measurements: # of vCPU, usemem command line, reading Silesia, etc.
Also swap space and QEMU memory allocation, using linear lhists() in
bpftrace, etc. If you can share these details, it would be very
helpful. Thanks again for your time!

2) Does the kernel community offer servers for contributors to test
their optimizations at scale?

3) Can I please request the help of other zswap maintainers/reviewers
to validate this series with ZSTD on a server setup with usemem
running on 30+ cores, under high over-commit ratios; through bpftrace
and/or throughput metrics.

Thanks,
Kanchana