zram: avoid preemption with CPU-based compression backends
Subsystems (3)
RFC PATCH 1/1 zram: avoid preemption with CPU-based compression backends
Since commit 2efa9e9eb4db ("zram: permit preemption with active compression stream"), a major Android regression has been reported.
The reason is that compression/decompression is now sleepable and preemptible. This means a stream may be migrated to another CPU or be preempted while holding the stream mutex. As a result, high priority UI threads may get stuck waiting for the mutex during swap-in.
The worst case is when a stream is migrated from a big core to a little core. On the little core, the stream can be repeatedly preempted by other threads, while a high priority task running on a big core is blocked waiting for the mutex to perform decompression and swap-in.
We add an async flag (currently false for almost all backends) to indicate whether a backend is asynchronous. For synchronous backends, we use preempt_disable() in the !PREEMPT_RT case. A zram_zs_malloc() wrapper is provided to support a two-stage zs_malloc() path, allowing allocation to transition from a non-sleepable context to a sleepable context.
--- drivers/block/zram/zcomp.c | 14 ++++-- drivers/block/zram/zcomp.h | 3 +- drivers/block/zram/zram_drv.c | 87 +++++++++++++++++++---------------- 3 files changed, 61 insertions(+), 43 deletions(-)
drivers/block/zram/zcomp.c +11 -3
@@ -128,14 +128,20 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp)* so then unlock and re-try on the current CPU. */ mutex_lock (&zstrm->lock); +if (!comp->ops->async && !IS_ENABLED (CONFIG_PREEMPT_RT )) +preempt_disable ();if (likely (zstrm->buffer))return zstrm;mutex_unlock (&zstrm->lock); } } -void zcomp_stream_put (struct zcomp_strm *zstrm) +void zcomp_stream_put (struct zcomp *comp) { +struct zcomp_strm *zstrm =raw_cpu_ptr (comp->stream); + +if (!comp->ops->async && !IS_ENABLED (CONFIG_PREEMPT_RT )) +preempt_enable ();mutex_unlock (&zstrm->lock); }
@@ -150,7 +156,8 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, };int ret; -might_sleep (); +if (comp->ops->async) +might_sleep (); ret = comp->ops->compress (comp->params, &zstrm->ctx, &req);if (!ret) *dst_len = req.dst_len;
@@ -167,7 +174,8 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, .dst_len = PAGE_SIZE, }; -might_sleep (); +if (comp->ops->async) +might_sleep ();return comp->ops->decompress (comp->params, &zstrm->ctx, &req); }
drivers/block/zram/zcomp.h +2 -1
@@ -67,6 +67,7 @@ struct zcomp_ops {void (*release_params)(struct zcomp_params *params);const char *name; +bool async; };/* dynamic per-device compression frontend */
@@ -86,7 +87,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params);void zcomp_destroy (struct zcomp *comp);struct zcomp_strm *zcomp_stream_get (struct zcomp *comp); -void zcomp_stream_put (struct zcomp_strm *zstrm); +void zcomp_stream_put (struct zcomp *comp);int zcomp_compress (struct zcomp *comp,struct zcomp_strm *zstrm,const void *src,unsigned int *dst_len);
drivers/block/zram/zram_drv.c +48 -39
186 lines not on this page · raw message
Signature · 1 line
2.39.3 (Apple Git-146)
Sergey Senozhatsky
maintainerQuoted 7 lines · Barry Song (Xiaomi)
Since commit
2efa9e9eb4db("zram: permit preemption with active compression stream"), a major Android regression has been reported.The reason is that compression/decompression is now sleepable and preemptible. This means a stream may be migrated to another CPU or be preempted while holding the stream mutex. As a result, high priority UI threads may get stuck waiting for the mutex during swap-in.
[..]
Quoted 3 lines
We add an async flag (currently false for almost all backends) to indicate whether a backend is asynchronous. For synchronous backends, we use
preempt_disable()in the !PREEMPT_RT case. A
I wonder what does that report say. Is that what I think it is (we discussed something RT related privately recently)?
Quoted 9 lines · Sergey Senozhatsky
Quoted 7 lines · Barry Song (Xiaomi)
Since commit
2efa9e9eb4db("zram: permit preemption with active compression stream"), a major Android regression has been reported.The reason is that compression/decompression is now sleepable and preemptible. This means a stream may be migrated to another CPU or be preempted while holding the stream mutex. As a result, high priority UI threads may get stuck waiting for the mutex during swap-in.
Quoted 7 lines
[..]
Quoted 3 lines
We add an async flag (currently false for almost all backends) to indicate whether a backend is asynchronous. For synchronous backends, we use
preempt_disable()in the !PREEMPT_RT case. AI wonder what does that report say. Is that what I think it is (we discussed something RT related privately recently)?
This report shows that the zram mutex has become the top lock contributing to UI frame drops, even surpassing mmap_lock, which we are also addressing in multiple threads. :-)
Locally, we have reverted the patch, but for upstream we still need to find a proper solution.
Sergey Senozhatsky
maintainerCc-ing Suleiman
Quoted 13 lines · Barry Song
Quoted 13 lines · Barry Song (Xiaomi)
Quoted 7 lines
Since commit
2efa9e9eb4db("zram: permit preemption with active compression stream"), a major Android regression has been reported.The reason is that compression/decompression is now sleepable and preemptible. This means a stream may be migrated to another CPU or be preempted while holding the stream mutex. As a result, high priority UI threads may get stuck waiting for the mutex during swap-in.
[..]
Quoted 3 lines
We add an async flag (currently false for almost all backends) to indicate whether a backend is asynchronous. For synchronous backends, we use
preempt_disable()in the !PREEMPT_RT case. A
Quoted 6 lines
Quoted 2 lines
I wonder what does that report say. Is that what I think it is (we discussed something RT related privately recently)?
This report shows that the zram mutex has become the top lock contributing to UI frame drops, even surpassing
mmap_lock, which we are also addressing in multiple threads. :-)
Any chance you can share more details? Are there perhaps RT tasks in the mix, priority inversion, starvations and so on? Can proxy execution address any of those (if it has relevance to the report you are looking at)?
Quoted 20 lines · Sergey Senozhatsky
Cc-ing Suleiman
Quoted 16 lines · Barry Song
Quoted 15 lines · Barry Song (Xiaomi)
Quoted 7 lines
Since commit
2efa9e9eb4db("zram: permit preemption with active compression stream"), a major Android regression has been reported.The reason is that compression/decompression is now sleepable and preemptible. This means a stream may be migrated to another CPU or be preempted while holding the stream mutex. As a result, high priority UI threads may get stuck waiting for the mutex during swap-in.
[..]
Quoted 3 lines
We add an async flag (currently false for almost all backends) to indicate whether a backend is asynchronous. For synchronous backends, we use
preempt_disable()in the !PREEMPT_RT case. AI wonder what does that report say. Is that what I think it is (we discussed something RT related privately recently)?
Quoted 8 lines
Quoted 3 lines
This report shows that the zram mutex has become the top lock contributing to UI frame drops, even surpassing
mmap_lock, which we are also addressing in multiple threads. :-)Any chance you can share more details? Are there perhaps RT tasks in the mix, priority inversion, starvations and so on? Can proxy execution address any of those (if it has relevance to the report you are looking at)?
Hi Sergey,
talked with our engineers reporting the issue. i believe it is all about priority inversion.
proxy execution wont resolve it as we have a sleepable zs-malloc within the mutex.
i believe i need v2 to release the mutex before doing the 2nd stagezs_malloc with
direct reclaim.
proxy execution could only defer the preemption for the compression and decompression, it wont address the potential long sleep of zs-malloc. we should not do sleepable zs-malloc
within the mutex.
i will send v2 with more details.
Best Regards
Barry
Sergey Senozhatsky
maintainerHi Barry,
Quoted 11 lines · Barry Song
Quoted 8 lines
Quoted 3 lines
This report shows that the zram mutex has become the top lock contributing to UI frame drops, even surpassing
mmap_lock, which we are also addressing in multiple threads. :-)Any chance you can share more details? Are there perhaps RT tasks in the mix, priority inversion, starvations and so on? Can proxy execution address any of those (if it has relevance to the report you are looking at)?
Hi Sergey,
Quoted 7 lines
talked with our engineers reporting the issue. i believe it is all about priority inversion.
proxy execution wont resolve it as we have a sleepable zs-malloc within the mutex.
i believe i need v2 to release the mutex before doing the 2nd stagezs_mallocwith
direct reclaim.
Well, we cannot just drop the stream mutex and do sleepable zsmalloc allocation, because this will invalidate compression buffer. So we then will need to do re-compression. Something that I was really happy to drop [1].
Is there any we can do apart from making zram and zsmalloc atomic again? It's hard to believe that this priority inversion hits only zram and no other locks in the system.
I really really really don't want to return back to atomic zram/zsmalloc.
Is the report you are talking about some test or is it a real world scenario?
Quoted 23 lines · Sergey Senozhatsky
Hi Barry,
Quoted 11 lines · Barry Song
Quoted 8 lines
Quoted 3 lines
This report shows that the zram mutex has become the top lock contributing to UI frame drops, even surpassing
mmap_lock, which we are also addressing in multiple threads. :-)Any chance you can share more details? Are there perhaps RT tasks in the mix, priority inversion, starvations and so on? Can proxy execution address any of those (if it has relevance to the report you are looking at)?
Hi Sergey,
Quoted 7 lines
talked with our engineers reporting the issue. i believe it is all about priority inversion.
proxy execution wont resolve it as we have a sleepable zs-malloc within the mutex.
i believe i need v2 to release the mutex before doing the 2nd stagezs_mallocwith
direct reclaim.
Quoted 4 lines
Well, we cannot just drop the stream mutex and do sleepable zsmalloc allocation, because this will invalidate compression buffer. So we then will need to do re-compression. Something that I was really happy to drop [1].
We used to do that by an temp GFP_ATOMIC buffer and memcpy:
https://lore.kernel.org/all/[email protected]/
As long as we copy zstrm->buffer to a temporary buffer, we are free to go anywhere afterwards.
BTW, I wonder if compression and decompression could use separate mutexes. That way, a sleepable zs_malloc() in the compression path would not block decompression, which is the more latency-sensitive operation.
Quoted 4 lines
Is there any we can do apart from making zram and zsmalloc atomic again? It's hard to believe that this priority inversion hits only zram and no other locks in the system.
It affects binder, mmap, and other paths. Some of them have already been addressed in various ways. One common approach is to move slow operations out of critical sections.
Quoted 3 lines
I really really really don't want to return back to atomic zram/zsmalloc.
Let's not go back to atomic again. :-)
From what I can see, mm/zswap.c uses a non-sleepable zs_malloc(), which likely helps avoid this problem. If that allocation fails, zswap can still fall back to writing the page back to disk. For zram, however, a failure of the non-sleepable zs_malloc() would mean the swapout() operation itself fails.
Quoted 3 lines
Is the report you are talking about some test or is it a real world scenario?
It happens in real-world scenarios as well as during testing. For example, if you flash a Pixel 6 with an image containing those non-atomic commits, the zram mutex becomes a bottleneck. Hongru reported it as the top source of UI frame drops on Pixel 6, as shown in the attached zram-mutex.png.
On our OEM devices, it is also a serious problem.
Quoted 2 lines
Thanks
Barry
Sergey Senozhatsky
maintainerQuoted 13 lines · Barry Song
Quoted 12 lines
Quoted 7 lines
talked with our engineers reporting the issue. i believe it is all about priority inversion.
proxy execution wont resolve it as we have a sleepable zs-malloc within the mutex.
i believe i need v2 to release the mutex before doing the 2nd stagezs_mallocwith
direct reclaim.Well, we cannot just drop the stream mutex and do sleepable zsmalloc allocation, because this will invalidate compression buffer. So we then will need to do re-compression. Something that I was really happy to drop [1].
[..]
Quoted 4 lines
BTW, I wonder if compression and decompression could use separate mutexes. That way, a sleepable
zs_malloc()in the compression path would not block decompression, which is the more latency-sensitive operation.
This sounds interesting. I think all of the S/W backends that we use have stateless decompression, so we probably can just split per-CPU stream mutex for R and W paths w/o the need for any additional scratch buffers. Wanna give it a try?
Back to preemption:
Is there maybe a common hot preemption point where stream mutex owners get scheduled out? E.g. inside zs_malloc()?
Sergey Senozhatsky
maintainerQuoted 5 lines · Sergey Senozhatsky
On (26/08/05 15:50), Barry Song wrote:
This sounds interesting. I think all of the S/W backends that we use have stateless decompression
OK, that doesn't seem to be true, but just for initial testing on lzo/lz4 we can simply split the lock. zstd/deflate need context for decompression, this doesn't look stateless.
Quoted 31 lines · Barry Song
Quoted 23 lines · Sergey Senozhatsky
Hi Barry,
Quoted 11 lines · Barry Song
Quoted 8 lines
Quoted 3 lines
This report shows that the zram mutex has become the top lock contributing to UI frame drops, even surpassing
mmap_lock, which we are also addressing in multiple threads. :-)Any chance you can share more details? Are there perhaps RT tasks in the mix, priority inversion, starvations and so on? Can proxy execution address any of those (if it has relevance to the report you are looking at)?
Hi Sergey,
Quoted 7 lines
talked with our engineers reporting the issue. i believe it is all about priority inversion.
proxy execution wont resolve it as we have a sleepable zs-malloc within the mutex.
i believe i need v2 to release the mutex before doing the 2nd stagezs_mallocwith
direct reclaim.Quoted 4 lines
Well, we cannot just drop the stream mutex and do sleepable zsmalloc allocation, because this will invalidate compression buffer. So we then will need to do re-compression. Something that I was really happy to drop [1].
Quoted 6 lines
We used to do that by an temp
GFP_ATOMICbuffer and memcpy:
https://lore.kernel.org/all/[email protected]/As long as we copy
zstrm->bufferto a temporary buffer, we are free to go anywhere afterwards.
Hi Sergey,
Just as a proof of concept, I changed one path and it seems to work. We release the mutex before calling zs_malloc(), which may enter direct reclaim, and we no longer need the mutex afterwards. also, we can avoid re-compression:
From d1a4cbe63fc2f7336c23bc268b1dffe15b0e7444 Mon Sep 17 00:00:00 2001
From: "Barry Song (Xiaomi)" <[email protected]>
Date: Wed, 5 Aug 2026 17:53:49 +0800
Subject: [PATCH] zram: avoid doing zs_malloc() with direct reclaim within
mutex
--- drivers/block/zram/zram_drv.c | 43 ++++++++++++++++++++++++++++------- 1 file changed, 35 insertions(+), 8 deletions(-)
drivers/block/zram/zram_drv.c +35 -8
@@ -2218,6 +2218,35 @@ static int write_same_filled_page(struct zram *zram, unsigned long fill,return 0 ; } +/* +* try non-sleepable allocation for !async backend, then try +* sleepable allocation; for async backend, we always begin +* from sleepable allocation +*/ +static unsigned long zram_zs_malloc (struct zram *zram,struct zcomp_strm *zstrm, +size_t comp_len,const int nid,void **bounce) +{ +unsigned long handle; + + handle =zs_malloc (zram->mem_pool, comp_len, + __GFP_KSWAPD_RECLAIM | __GFP_NOWARN | + __GFP_HIGHMEM | __GFP_MOVABLE, nid); +if (!IS_ERR_VALUE (handle)) +return handle; + + *bounce =kmalloc (comp_len,GFP_ATOMIC ); +if (!*bounce) +return (unsigned long )ERR_PTR (-ENOMEM ); +memcpy (*bounce, zstrm->buffer, comp_len); + +/* Don't hold mutex to do a sleepable allocation */ +zcomp_stream_put (zstrm); + handle =zs_malloc (zram->mem_pool, comp_len, +GFP_NOIO | __GFP_NOWARN | + __GFP_HIGHMEM | __GFP_MOVABLE, nid); +return handle; +} +static int write_incompressible_page (struct zram *zram,struct page *page,u32 index) {
@@ -2264,7 +2293,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index)int ret =0 ;unsigned long handle;unsigned int comp_len; -void *mem; +void *mem, *bounce =NULL ;struct zcomp_strm *zstrm;unsigned long element;bool same_filled;
@@ -2292,22 +2321,20 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index)return write_incompressible_page (zram, page, index); } - handle =zs_malloc (zram->mem_pool, comp_len, -GFP_NOIO | __GFP_NOWARN | - __GFP_HIGHMEM | __GFP_MOVABLE,page_to_nid (page)); + handle =zram_zs_malloc (zram, zstrm, comp_len,page_to_nid (page), &bounce);if (IS_ERR_VALUE (handle)) { -zcomp_stream_put (zstrm); + bounce ?kfree (bounce) :zcomp_stream_put (zstrm);return PTR_ERR ((void *)handle); }if (!zram_can_store_page (zram)) { -zcomp_stream_put (zstrm); + bounce ?kfree (bounce) :zcomp_stream_put (zstrm);zs_free (zram->mem_pool, handle);return -ENOMEM ; } -zs_obj_write (zram->mem_pool, handle, zstrm->buffer, comp_len); -zcomp_stream_put (zstrm); +zs_obj_write (zram->mem_pool, handle, bounce ? : zstrm->buffer, comp_len); + bounce ?kfree (bounce) :zcomp_stream_put (zstrm);slot_lock (zram, index);slot_free (zram, index);
Signature · 1 line
2.39.3 (Apple Git-146)
Sergey Senozhatsky
maintainerHi Barry,
Quoted 4 lines · Barry Song
BTW, I wonder if compression and decompression could use separate mutexes. That way, a sleepable
zs_malloc()in the compression path would not block decompression, which is the more latency-sensitive operation.
quick and dirty patch. Just curious if this improves anything on your side.
We also maybe can have more that num_online_cpus() stream, if we switch to idle streams list instead [1]
[1] https://lore.kernel.org/lkml/[email protected]/
----
drivers/block/zram/zcomp.c +62 -29
@@ -112,21 +112,28 @@ ssize_t zcomp_available_show(const char *comp, char *buf, ssize_t at)return at; } -struct zcomp_strm *zcomp_stream_get (struct zcomp *comp) +struct zcomp_strm *zcomp_stream_get_write (struct zcomp *comp) {for (;;) { -struct zcomp_strm *zstrm =raw_cpu_ptr (comp->stream); - -/* -* Inspired by zswap -* -* stream is returned with ->mutex locked which prevents -* cpu_dead() from releasing this stream under us, however -* there is still a race window between raw_cpu_ptr() and -* mutex_lock(), during which we could have been migrated -* from a CPU that has already destroyed its stream. If -* so then unlock and re-try on the current CPU. -*/ +struct zcomp_strm *zstrm =raw_cpu_ptr (comp->stream_write); + +mutex_lock (&zstrm->lock); +if (likely (zstrm->buffer)) +return zstrm; +mutex_unlock (&zstrm->lock); + } +} + +void zcomp_stream_put_write (struct zcomp_strm *zstrm) +{ +mutex_unlock (&zstrm->lock); +} + +struct zcomp_strm *zcomp_stream_get_read (struct zcomp *comp) +{ +for (;;) { +struct zcomp_strm *zstrm =raw_cpu_ptr (comp->stream_read); +mutex_lock (&zstrm->lock);if (likely (zstrm->buffer))return zstrm;
@@ -134,7 +141,7 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) } } -void zcomp_stream_put (struct zcomp_strm *zstrm) +void zcomp_stream_put_read (struct zcomp_strm *zstrm) {mutex_unlock (&zstrm->lock); }
@@ -174,23 +181,39 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,int zcomp_cpu_up_prepare (unsigned int cpu,struct hlist_node *node) { struct zcomp *comp = hlist_entry(node, struct zcomp, node); -struct zcomp_strm *zstrm =per_cpu_ptr (comp->stream, cpu); +struct zcomp_strm *zstrm_w =per_cpu_ptr (comp->stream_write, cpu); +struct zcomp_strm *zstrm_r =per_cpu_ptr (comp->stream_read, cpu);int ret; - ret =zcomp_strm_init (comp, zstrm); -if (ret) -pr_err ("Can't allocate a compression stream\n" ); -return ret; + ret =zcomp_strm_init (comp, zstrm_w); +if (ret) { +pr_err ("Can't allocate a compression write stream\n" ); +return ret; + } + + ret =zcomp_strm_init (comp, zstrm_r); +if (ret) { +pr_err ("Can't allocate a compression read stream\n" ); +zcomp_strm_free (comp, zstrm_w); +return ret; + } + +return 0 ; }int zcomp_cpu_dead (unsigned int cpu,struct hlist_node *node) { struct zcomp *comp = hlist_entry(node, struct zcomp, node); -struct zcomp_strm *zstrm =per_cpu_ptr (comp->stream, cpu); +struct zcomp_strm *zstrm_w =per_cpu_ptr (comp->stream_write, cpu); +struct zcomp_strm *zstrm_r =per_cpu_ptr (comp->stream_read, cpu); -mutex_lock (&zstrm->lock); -zcomp_strm_free (comp, zstrm); -mutex_unlock (&zstrm->lock); +mutex_lock (&zstrm_w->lock); +zcomp_strm_free (comp, zstrm_w); +mutex_unlock (&zstrm_w->lock); + +mutex_lock (&zstrm_r->lock); +zcomp_strm_free (comp, zstrm_r); +mutex_unlock (&zstrm_r->lock);return 0 ; }
@@ -198,17 +221,25 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) {int ret, cpu; - comp->stream = alloc_percpu(struct zcomp_strm); -if (!comp->stream) + comp->stream_write = alloc_percpu(struct zcomp_strm); +if (!comp->stream_write)return -ENOMEM ; + comp->stream_read = alloc_percpu(struct zcomp_strm); +if (!comp->stream_read) { +free_percpu (comp->stream_write); +return -ENOMEM ; + } + comp->params = params; ret = comp->ops->setup_params (comp->params);if (ret) goto cleanup; -for_each_possible_cpu (cpu) -mutex_init (&per_cpu_ptr (comp->stream, cpu)->lock); +for_each_possible_cpu (cpu) { +mutex_init (&per_cpu_ptr (comp->stream_write, cpu)->lock); +mutex_init (&per_cpu_ptr (comp->stream_read, cpu)->lock); + } ret =cpuhp_state_add_instance (CPUHP_ZCOMP_PREPARE , &comp->node);if (ret <0 )
@@ -218,7 +249,8 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) cleanup: comp->ops->release_params (comp->params); -free_percpu (comp->stream); +free_percpu (comp->stream_read); +free_percpu (comp->stream_write);return ret; }
@@ -226,7 +258,8 @@ void zcomp_destroy(struct zcomp *comp) {cpuhp_state_remove_instance (CPUHP_ZCOMP_PREPARE , &comp->node); comp->ops->release_params (comp->params); -free_percpu (comp->stream); +free_percpu (comp->stream_read); +free_percpu (comp->stream_write);kfree (comp); }
drivers/block/zram/zcomp.h +7 -3
@@ -71,7 +71,8 @@ struct zcomp_ops {/* dynamic per-device compression frontend */ struct zcomp { -struct zcomp_strm __percpu *stream; +struct zcomp_strm __percpu *stream_write; +struct zcomp_strm __percpu *stream_read;const struct zcomp_ops *ops;struct zcomp_params *params;struct hlist_node node;
@@ -85,8 +86,11 @@ const char *zcomp_lookup_backend_name(const char *comp);struct zcomp *zcomp_create (const char *alg,struct zcomp_params *params);void zcomp_destroy (struct zcomp *comp); -struct zcomp_strm *zcomp_stream_get (struct zcomp *comp); -void zcomp_stream_put (struct zcomp_strm *zstrm); +struct zcomp_strm *zcomp_stream_get_write (struct zcomp *comp); +void zcomp_stream_put_write (struct zcomp_strm *zstrm); + +struct zcomp_strm *zcomp_stream_get_read (struct zcomp *comp); +void zcomp_stream_put_read (struct zcomp_strm *zstrm);int zcomp_compress (struct zcomp *comp,struct zcomp_strm *zstrm,const void *src,unsigned int *dst_len);
drivers/block/zram/zram_drv.c +17 -17
@@ -1358,14 +1358,14 @@ static int decompress_bdev_page(struct zram *zram, struct page *page, u32 index) size =get_slot_size (zram, index); prio =get_slot_comp_priority (zram, index); - zstrm =zcomp_stream_get (zram->comps[prio]); + zstrm =zcomp_stream_get_read (zram->comps[prio]); src =kmap_local_page (page); ret =zcomp_decompress (zram->comps[prio], zstrm, src, size, zstrm->local_copy);if (!ret)copy_page (src, zstrm->local_copy);kunmap_local (src); -zcomp_stream_put (zstrm); +zcomp_stream_put_read (zstrm);slot_unlock (zram, index);return ret;
@@ -2101,14 +2101,14 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) size =get_slot_size (zram, index); prio =get_slot_comp_priority (zram, index); - zstrm =zcomp_stream_get (zram->comps[prio]); + zstrm =zcomp_stream_get_read (zram->comps[prio]); src =zs_obj_read_begin (zram->mem_pool, handle, size, zstrm->local_copy); dst =kmap_local_page (page); ret =zcomp_decompress (zram->comps[prio], zstrm, src, size, dst);kunmap_local (dst);zs_obj_read_end (zram->mem_pool, handle, size, src); -zcomp_stream_put (zstrm); +zcomp_stream_put_read (zstrm);return ret; }
@@ -2129,12 +2129,12 @@ static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index)* case if object spans two physical pages. No decompression * takes place here, as we read raw compressed data. */ - zstrm =zcomp_stream_get (zram->comps[ZRAM_PRIMARY_COMP ]); + zstrm =zcomp_stream_get_read (zram->comps[ZRAM_PRIMARY_COMP ]); src =zs_obj_read_begin (zram->mem_pool, handle, size, zstrm->local_copy);memcpy_to_page (page,0 , src, size);zs_obj_read_end (zram->mem_pool, handle, size, src); -zcomp_stream_put (zstrm); +zcomp_stream_put_read (zstrm);memzero_page (page, size,PAGE_SIZE - size);
@@ -2285,20 +2285,20 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index)if (same_filled)return write_same_filled_page (zram, element, index); - zstrm =zcomp_stream_get (zram->comps[ZRAM_PRIMARY_COMP ]); + zstrm =zcomp_stream_get_write (zram->comps[ZRAM_PRIMARY_COMP ]); mem =kmap_local_page (page); ret =zcomp_compress (zram->comps[ZRAM_PRIMARY_COMP ], zstrm, mem, &comp_len);kunmap_local (mem);if (unlikely (ret)) { -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);pr_err ("Compression failed! err=%d\n" , ret);return ret; }if (comp_len >= huge_class_size) { -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);return write_incompressible_page (zram, page, index); }
@@ -2306,18 +2306,18 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index)GFP_NOIO | __GFP_NOWARN | __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page));if (IS_ERR_VALUE (handle)) { -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);return PTR_ERR ((void *)handle); }if (!zram_can_store_page (zram)) { -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);zs_free (zram->mem_pool, handle);return -ENOMEM ; }zs_obj_write (zram->mem_pool, handle, zstrm->buffer, comp_len); -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);slot_lock (zram, index);slot_free (zram, index);
@@ -2457,7 +2457,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page,*/ clear_slot_flag (zram, index,ZRAM_IDLE ); - zstrm =zcomp_stream_get (zram->comps[prio]); + zstrm =zcomp_stream_get_write (zram->comps[prio]); src =kmap_local_page (page); ret =zcomp_compress (zram->comps[prio], zstrm, src, &comp_len_new);kunmap_local (src);
@@ -2472,7 +2472,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, *num_recomp_pages -=1 ;if (ret) { -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);return ret; }
@@ -2481,7 +2481,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page,if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);/* * Secondary algorithms failed to re-compress the page
@@ -2510,12 +2510,12 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page));if (IS_ERR_VALUE (handle_new)) { -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);return PTR_ERR ((void *)handle_new); }zs_obj_write (zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); -zcomp_stream_put (zstrm); +zcomp_stream_put_write (zstrm);slot_free (zram, index);set_slot_handle (zram, index, handle_new);
Quoted 12 lines · Sergey Senozhatsky
Hi Barry,
Quoted 4 lines · Barry Song
BTW, I wonder if compression and decompression could use separate mutexes. That way, a sleepable
zs_malloc()in the compression path would not block decompression, which is the more latency-sensitive operation.quick and dirty patch. Just curious if this improves anything on your side.
Thanks very much for your quick patch, Sergey.
We are going to run three experiments:
1. Use the approach I just sent, which avoids calling zs_malloc() with direct reclaim while holding the mutex, and extend it to the other zs_malloc() call sites:
https://lore.kernel.org/all/[email protected]/
2. Experiment 1 plus preemption disabled.
3. Your patch, which separates the compression and decompression mutexes.
I'd like to understand which change has the biggest impact on performance.
Let me gather some data, and I'll get back to you.
Quoted 54 lines
We also maybe can have more that
num_online_cpus()stream, if we switch to idle streams list instead [1][1] https://lore.kernel.org/lkml/[email protected]/
----
drivers/block/zram/zcomp.c +20
-13@@ -112,21 +112,28 @@ ssize_t zcomp_available_show(const char *comp, char *buf, ssize_t at)return at; } -struct zcomp_strm *zcomp_stream_get (struct zcomp *comp) +struct zcomp_strm *zcomp_stream_get_write (struct zcomp *comp) {for (;;) { -struct zcomp_strm *zstrm =raw_cpu_ptr (comp->stream); - -/* -* Inspired by zswap -* -* stream is returned with ->mutex locked which prevents -* cpu_dead() from releasing this stream under us, however -* there is still a race window between raw_cpu_ptr() and -* mutex_lock(), during which we could have been migrated -* from a CPU that has already destroyed its stream. If -* so then unlock and re-try on the current CPU. -*/ +struct zcomp_strm *zstrm =raw_cpu_ptr (comp->stream_write); + +mutex_lock (&zstrm->lock); +if (likely (zstrm->buffer)) +return zstrm; +mutex_unlock (&zstrm->lock); + } +} + +void zcomp_stream_put_write (struct zcomp_strm *zstrm) +{ +mutex_unlock (&zstrm->lock); +} + +struct zcomp_strm *zcomp_stream_get_read (struct zcomp *comp) +{ +for (;;) { +struct zcomp_strm *zstrm =raw_cpu_ptr (comp->stream_read); +mutex_lock (&zstrm->lock);if (likely (zstrm->buffer))return zstrm;
drivers/block/zram/zcomp.c struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) · line 143
Quoted 3 lines
drivers/block/zram/zcomp.c +0
-0@@ -134,7 +141,7 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) } }
[...]
Best Regards
Barry
Sergey Senozhatsky
maintainerQuoted 12 lines · Barry Song
Thanks very much for your quick patch, Sergey.
We are going to run three experiments:
1. Use the approach I just sent, which avoids calling
zs_malloc()with direct reclaim while holding the mutex, and extend it to the otherzs_malloc()call sites:https://lore.kernel.org/all/[email protected]/
2. Experiment 1 plus preemption disabled.
Quoted 7 lines
3. Your patch, which separates the compression and decompression mutexes.
I'd like to understand which change has the biggest impact on performance.
Let me gather some data, and I'll get back to you.
Thanks a lot! Take your time!
Quoted 8 lines · Sergey Senozhatsky
Hi Barry,
Quoted 4 lines · Barry Song
BTW, I wonder if compression and decompression could use separate mutexes. That way, a sleepable
zs_malloc()in the compression path would not block decompression, which is the more latency-sensitive operation.
Quoted 7 lines
quick and dirty patch. Just curious if this improves anything on your side.
We also maybe can have more that
num_online_cpus()stream, if we switch to idle streams list instead [1]
After reading this thread, I got to say that this looks quite nice compared to the other two ;)
Sebastian
Quoted 2 lines · Sergey Senozhatsky
quick and dirty patch. Just curious if this improves anything on your side.
Hi Sergey, Barry,
I ran an A/B of the three approaches on a real Android device (MemTotal ~7.7GB, zram ~8GB). Same workload for all three: cycle 20 memory-heavy apps through fg/bg to force swap-out/in, 8 rounds (~550s). The stream mutex is instrumented with a trylock-based counter; the measurement code is identical across all three builds, so the only difference is the patch.
baseline preempt-disable read/write split contended count 8,765 150 6,557 wait_ns total 52.28 s 1.29 s 65.86 s wait_ns max 95.6 ms 46.2 ms 130.7 ms swap-out volume ~2.86 GB ~3.04 GB ~2.87 GB
For the split build, contention by stream:
read (decompress): 6,349 (96.8%)
write (compress): 208 ( 3.2%)
Summary:
- preempt-disable: contention -98% (8765 -> 150), total wait -97.5% (52.3s -> 1.29s), worst case halved.
- read/write split: barely helps (-25%), total wait and worst case actually got worse.
The split doesn't help because 96.8% of the contention is on the read stream alone, which is concurrent swap-in decompressions queuing on the same per-CPU read stream while the holder gets preempted/migrated (big -> little core), not readers vs writers. Splitting by direction doesn't address that, and it doubles per-CPU stream memory.
So in our workload the root cause is the holder being preemptible or migratable, which preempt-disable addresses directly.
Happy to share scripts or re-run any variation.
Thanks,
Bo
Sergey Senozhatsky
maintainerHi,
Quoted 4 lines · Bo Zhang
Quoted 2 lines · Sergey Senozhatsky
quick and dirty patch. Just curious if this improves anything on your side.
Quoted 8 lines
Hi Sergey, Barry,
I ran an A/B of the three approaches on a real Android device (MemTotal ~7.7GB, zram ~8GB). Same workload for all three: cycle 20 memory-heavy apps through fg/bg to force swap-out/in, 8 rounds (~550s). The stream mutex is instrumented with a trylock-based counter; the measurement code is identical across all three builds, so the only difference is the patch.
Yeah, that patch was too quick and too dirty. We have significantly reworked it in [1], specifically patches 9 and 10.
Quoted 9 lines · Barry Song (Xiaomi)
Since commit
2efa9e9eb4db("zram: permit preemption with active compression stream"), a major Android regression has been reported.We add an async flag (currently false for almost all backends) to indicate whether a backend is asynchronous. For synchronous backends, we use
preempt_disable()in the !PREEMPT_RT case. Azram_zs_malloc()wrapper is provided to support a two-stagezs_malloc()path, allowing allocation to transition from a non-sleepable context to a sleepable context.
Hi Barry,
Thanks for working on this. The priority inversion issue on Android is real and we've hit it too.
However, I think there's a bug in the zcomp_stream_put() change.
The new zcomp_stream_put() uses raw_cpu_ptr() to find the stream:
void zcomp_stream_put(struct zcomp *comp)
{
struct zcomp_strm *zstrm = raw_cpu_ptr(comp->stream);
if (!comp->ops->async && !IS_ENABLED(CONFIG_PREEMPT_RT))
preempt_enable();
mutex_unlock(&zstrm->lock);
}It relies on the thread still being on the same CPU where zcomp_stream_get() locked the stream. In the normal path it works fine, because preempt is disabled the entire time.
But zram_zs_malloc() breaks this assumption in the fallback path:
static unsigned long zram_zs_malloc(...)
{
if (!async && !IS_ENABLED(CONFIG_PREEMPT_RT)) {
handle = zs_malloc(..., __GFP_KSWAPD_RECLAIM | ...);
if (!IS_ERR_VALUE(handle))
return handle;
preempt_enable(); // preempt is now enabled
} handle = zs_malloc(..., GFP_NOIO | ...); // may sleep, may migrate
if (!async && !IS_ENABLED(CONFIG_PREEMPT_RT))
preempt_disable(); // Preempt disabled, but it may work on another CPU
return handle;}
If the first zs_malloc() fails, preempt is enabled and the second zs_malloc(GFP_NOIO) can sleep. During this window the task can migrate to another CPU. After preempt_disable(), we continue on the new CPU.
When zcomp_stream_put() is later called, raw_cpu_ptr() returns the new CPU's stream, which is not the one that was originally locked. Which may cause a mutex_unlock() on the wrong CPU.
The original code didn't have this problem because zcomp_stream_put() took the zstrm pointer directly:
void zcomp_stream_put(struct zcomp_strm *zstrm)
{
mutex_unlock(&zstrm->lock);
}The caller always passed the saved pointer from zcomp_stream_get(), so regardless of CPU migration, the correct mutex was always unlocked.
I think the fix is to keep passing the zstrm pointer:
void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *zstrm)
{
if (!comp->ops->async && !IS_ENABLED(CONFIG_PREEMPT_RT))
preempt_enable();
mutex_unlock(&zstrm->lock);
}All callers already have zstrm available. Then It works as expected.
Bo
Quoted 71 lines · Bo Zhang
Quoted 9 lines · Barry Song (Xiaomi)
Since commit
2efa9e9eb4db("zram: permit preemption with active compression stream"), a major Android regression has been reported.We add an async flag (currently false for almost all backends) to indicate whether a backend is asynchronous. For synchronous backends, we use
preempt_disable()in the !PREEMPT_RT case. Azram_zs_malloc()wrapper is provided to support a two-stagezs_malloc()path, allowing allocation to transition from a non-sleepable context to a sleepable context.Hi Barry,
Thanks for working on this. The priority inversion issue on Android is real and we've hit it too.
However, I think there's a bug in the
zcomp_stream_put()change.The new
zcomp_stream_put()usesraw_cpu_ptr()to find the stream:void zcomp_stream_put(struct zcomp *comp) { struct zcomp_strm *zstrm = raw_cpu_ptr(comp->stream); if (!comp->ops->async && !IS_ENABLED(CONFIG_PREEMPT_RT)) preempt_enable(); mutex_unlock(&zstrm->lock); }It relies on the thread still being on the same CPU where
zcomp_stream_get()locked the stream. In the normal path it works fine, because preempt is disabled the entire time.But
zram_zs_malloc()breaks this assumption in the fallback path:static unsigned long zram_zs_malloc(...) { if (!async && !IS_ENABLED(CONFIG_PREEMPT_RT)) { handle = zs_malloc(..., __GFP_KSWAPD_RECLAIM | ...); if (!IS_ERR_VALUE(handle)) return handle; preempt_enable(); // preempt is now enabled }handle = zs_malloc(..., GFP_NOIO | ...); // may sleep, may migrate if (!async && !IS_ENABLED(CONFIG_PREEMPT_RT)) preempt_disable(); // Preempt disabled, but it may work on another CPU return handle;}
If the first
zs_malloc()fails, preempt is enabled and the secondzs_malloc(GFP_NOIO)can sleep. During this window the task can migrate to another CPU. Afterpreempt_disable(), we continue on the new CPU.When
zcomp_stream_put()is later called,raw_cpu_ptr()returns the new CPU's stream, which is not the one that was originally locked. Which may cause amutex_unlock()on the wrong CPU.The original code didn't have this problem because
zcomp_stream_put()took the zstrm pointer directly:void zcomp_stream_put(struct zcomp_strm *zstrm) { mutex_unlock(&zstrm->lock); }The caller always passed the saved pointer from
zcomp_stream_get(), so regardless of CPU migration, the correct mutex was always unlocked.I think the fix is to keep passing the zstrm pointer:
Quoted 8 lines
void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *zstrm) { if (!comp->ops->async && !IS_ENABLED(CONFIG_PREEMPT_RT)) preempt_enable(); mutex_unlock(&zstrm->lock); }All callers already have zstrm available. Then It works as expected.
Good catch. You're absolutely right. We need the following (will fix in v2):
drivers/block/zram/zcomp.c +1 -3
@@ -136,10 +136,8 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) } } -void zcomp_stream_put (struct zcomp *comp) +void zcomp_stream_put (struct zcomp_strm *zstrm,struct zcomp *comp) { -struct zcomp_strm *zstrm =raw_cpu_ptr (comp->stream); -if (!comp->ops->async && !IS_ENABLED (CONFIG_PREEMPT_RT ))preempt_enable ();mutex_unlock (&zstrm->lock);
drivers/block/zram/zcomp.h +0 -0
@@ -87,7 +87,7 @@ struct zcomp *zcomp_create(const char *alg, struct
zcomp_params *params);
void zcomp_destroy(struct zcomp *comp);
struct zcomp_strm *zcomp_stream_get(struct zcomp *comp);
-void zcomp_stream_put(struct zcomp *comp);
+void zcomp_stream_put(struct zcomp_strm *zstrm, struct zcomp *comp);
int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
const void *src, unsigned int *dst_len);
drivers/block/zram/zram_drv.c +0 -0
@@ -1364,7 +1364,7 @@ static int decompress_bdev_page(struct zram
*zram, struct page *page, u32 index)
if (!ret)
copy_page(src, zstrm->local_copy);
kunmap_local(src);
- zcomp_stream_put(zram->comps[prio]);
+ zcomp_stream_put(zstrm, zram->comps[prio]);
slot_unlock(zram, index);
return ret;
@@ -2098,7 +2098,7 @@ static int read_compressed_page(struct zram
*zram, struct page *page, u32 index)
ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst);
kunmap_local(dst);
zs_obj_read_end(zram->mem_pool, handle, size, src);- zcomp_stream_put(zram->comps[prio]);
+ zcomp_stream_put(zstrm, zram->comps[prio]);
return ret;
}
@@ -2124,7 +2124,7 @@ static int read_from_zspool_raw(struct zram
*zram, struct page *page, u32 index)
zstrm->local_copy);
memcpy_to_page(page, 0, src, size);
zs_obj_read_end(zram->mem_pool, handle, size, src);- zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]);
+ zcomp_stream_put(zstrm, zram->comps[ZRAM_PRIMARY_COMP]);
memzero_page(page, size, PAGE_SIZE - size);
@@ -2304,30 +2304,30 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index)
kunmap_local(mem);
if (unlikely(ret)) {
- zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]);
+ zcomp_stream_put(zstrm, zram->comps[ZRAM_PRIMARY_COMP]);
pr_err("Compression failed! err=%d\n", ret);
return ret;
}
if (comp_len >= huge_class_size) {
- zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]);
+ zcomp_stream_put(zstrm, zram->comps[ZRAM_PRIMARY_COMP]);
return write_incompressible_page(zram, page, index, async);
}
handle = zram_zs_malloc(zram, comp_len, async, page_to_nid(page));
if (IS_ERR_VALUE(handle)) {
- zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]);
+ zcomp_stream_put(zstrm, zram->comps[ZRAM_PRIMARY_COMP]);
return PTR_ERR((void *)handle);
}
if (!zram_can_store_page(zram)) {
- zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]);
+ zcomp_stream_put(zstrm, zram->comps[ZRAM_PRIMARY_COMP]);
zs_free(zram->mem_pool, handle);
return -ENOMEM;
}
zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len);- zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]);
+ zcomp_stream_put(zstrm, zram->comps[ZRAM_PRIMARY_COMP]);
slot_lock(zram, index);
slot_free(zram, index);
@@ -2484,7 +2484,7 @@ static int recompress_slot(struct zram *zram,
u32 index, struct page *page,
*num_recomp_pages -= 1;
if (ret) {
- zcomp_stream_put(zram->comps[prio]);
+ zcomp_stream_put(zstrm, zram->comps[prio]);
return ret;
}@@ -2493,7 +2493,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page,
if (class_index_new >= class_index_old ||
(threshold && comp_len_new >= threshold)) {- zcomp_stream_put(zram->comps[prio]);
+ zcomp_stream_put(zstrm, zram->comps[prio]);
/*
* Secondary algorithms failed to re-compress the page
@@ -2509,12 +2509,12 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page,
handle_new = zram_zs_malloc(zram, comp_len_new, async,
page_to_nid(page));
if (IS_ERR_VALUE(handle_new)) {
- zcomp_stream_put(zram->comps[prio]);
+ zcomp_stream_put(zstrm, zram->comps[prio]);
return PTR_ERR((void *)handle_new);
}
zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new);- zcomp_stream_put(zram->comps[prio]);
+ zcomp_stream_put(zstrm, zram->comps[prio]);
slot_free(zram, index);
set_slot_handle(zram, index, handle_new);Sergey Senozhatsky
maintainerdrivers/block/zram/zcomp.c struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) · line 132
Quoted 6 lines · Barry Song (Xiaomi)
drivers/block/zram/zcomp.c +2
-0@@ -128,14 +128,20 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp)* so then unlock and re-try on the current CPU. */ mutex_lock (&zstrm->lock); +if (!comp->ops->async && !IS_ENABLED (CONFIG_PREEMPT_RT )) +preempt_disable ();
We take slot_lock(), which is preemptible, under stream lock on some paths.
Quoted 5 lines
drivers/block/zram/zcomp.c +0
-0if (likely (zstrm->buffer))return zstrm;mutex_unlock (&zstrm->lock); } }
Quoted 11 lines · Sergey Senozhatsky
drivers/block/zram/zcomp.c struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) · line 132
Quoted 6 lines · Barry Song (Xiaomi)
drivers/block/zram/zcomp.c +2
-0@@ -128,14 +128,20 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp)* so then unlock and re-try on the current CPU. */ mutex_lock (&zstrm->lock); +if (!comp->ops->async && !IS_ENABLED (CONFIG_PREEMPT_RT )) +preempt_disable ();We take
slot_lock(), which is preemptible, under stream lock on some paths.
Hi Sergey,
As promised in [1], those patches are still under testing on real Android phones. However, I'm quite curious which path takes slot_lock while holding the zstream lock. My initial testing shows that this path doesn't seem to be exercised at all. Also, after inspecting the code, I couldn't find any path that does this.
[1] https://lore.kernel.org/all/CAGsJ_4wd60ok7BF9XGRiO6dvddkaYTziPV-Bxq+3mA+-c7MvQA@mail.gmail.com/
Best Regards
Barry
Sergey Senozhatsky
maintainerHello,
Quoted 11 lines · Barry Song
Quoted 10 lines · Barry Song (Xiaomi)
drivers/block/zram/zcomp.c struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) · line 132
Quoted 6 lines
drivers/block/zram/zcomp.c +2
-0@@ -128,14 +128,20 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp)* so then unlock and re-try on the current CPU. */ mutex_lock (&zstrm->lock); +if (!comp->ops->async && !IS_ENABLED (CONFIG_PREEMPT_RT )) +preempt_disable ();We take
slot_lock(), which is preemptible, under stream lock on some paths.
Quoted 5 lines
Hi Sergey,
As promised in [1], those patches are still under testing on real Android phones. However, I'm quite curious which path takes
slot_lockwhile holding the zstream lock
Oh, I think you are right. We do the opposite: stream lock under slot lock, and we also do the stream lock -> zsmalloc lock/unlock -> stream unlock -> slot lock/unlock chain. But it doesn't look like we ever do stream lock -> slot lock, you are right. Sorry for the confusion.