erofs-utils: do not check ->idata_size for compressed files in erofs_prepare_inode_buffer()
PATCH 1/1 erofs-utils: do not check ->idata_size for compressed files in erofs_prepare_inode_buffer()
From: Yue Hu <[email protected]>
erofs_write_compressed_file() will always set inode->idata_size = 0 if succeed, that means no tail-end data for compressed files. So, no need to call erofs_prepare_tail_block() which is used to handle tail-end data for that case. Just skip it.
Also, correct 'a inode' -> 'an inode'.
--- lib/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
lib/inode.c +2 -2
@@ -111,7 +111,7 @@ struct erofs_dentry *erofs_d_alloc(struct erofs_inode *parent,return d; } -/* allocate main data for a inode */ +/* allocate main data for an inode */ static int __allocate_inode_bh_data (struct erofs_inode *inode,unsigned long nblocks) {
@@ -572,11 +572,11 @@ static int erofs_prepare_inode_buffer(struct erofs_inode *inode)int ret; inode->datalayout =EROFS_INODE_FLAT_PLAIN ; -noinline:/* expend an extra block for tail-end data */ ret =erofs_prepare_tail_block (inode);if (ret)return ret; +noinline: bh =erofs_balloc (INODE , inodesize,0 ,0 );if (IS_ERR (bh))return PTR_ERR (bh);
Signature · 1 line
1.9.1
Hi Yue,
Quoted 6 lines · Yue Hu
From: Yue Hu <[email protected]>
erofs_write_compressed_file()will always setinode->idata_size= 0 if succeed, that means no tail-end data for compressed files. So, no need to callerofs_prepare_tail_block()which is used to handle tail-end data for that case. Just skip it.
Thanks for the patch, due to somewhat long time so I don't quite remember the exact logic here now...
Yet from the description before, it's not strictly correct since my original intention would be to support tail-packing inline compressed data which is similar to uncompressed case to decrease tail extent I/O and save more space.
BTW, if you have some interest, would you like to implement it? :)
Thanks,
Gao Xiang
Quoted 36 lines
Also, correct 'a inode' -> 'an inode'.
Signed-off-by Yue Hu <[email protected]>--- lib/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/inode.c +2
-2@@ -111,7 +111,7 @@ struct erofs_dentry *erofs_d_alloc(struct erofs_inode *parent,return d; } -/* allocate main data for a inode */ +/* allocate main data for an inode */ static int __allocate_inode_bh_data (struct erofs_inode *inode,unsigned long nblocks) {@@ -572,11 +572,11 @@ static int erofs_prepare_inode_buffer(struct erofs_inode *inode)int ret; inode->datalayout =EROFS_INODE_FLAT_PLAIN ; -noinline:/* expend an extra block for tail-end data */ ret =erofs_prepare_tail_block (inode);if (ret)return ret; +noinline: bh =erofs_balloc (INODE , inodesize,0 ,0 );if (IS_ERR (bh))return PTR_ERR (bh);Signature · 1 line
1.9.1
Quoted 10 lines · On Thu, 17 Jun 2021 17:04:29 +0800 Gao Xiang <[email protected]>…
Hi Yue,
Quoted 6 lines · Yue Hu
From: Yue Hu <[email protected]>
erofs_write_compressed_file()will always setinode->idata_size= 0 if succeed, that means no tail-end data for compressed files. So, no need to callerofs_prepare_tail_block()which is used to handle tail-end data for that case. Just skip it.
Quoted 7 lines
Thanks for the patch, due to somewhat long time so I don't quite remember the exact logic here now...
Yet from the description before, it's not strictly correct since my original intention would be to support tail-packing inline compressed data which is similar to uncompressed case to decrease tail extent I/O and save more space.
nice.
Quoted 2 lines
BTW, if you have some interest, would you like to implement it? :)
I don't know if i can finish it. But i would like to have a try :)
Thanks.
Quoted 40 lines
Thanks,
Gao XiangQuoted 36 lines
Also, correct 'a inode' -> 'an inode'.
Signed-off-by Yue Hu <[email protected]>--- lib/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/inode.c +2
-2@@ -111,7 +111,7 @@ struct erofs_dentry *erofs_d_alloc(struct erofs_inode *parent,return d; } -/* allocate main data for a inode */ +/* allocate main data for an inode */ static int __allocate_inode_bh_data (struct erofs_inode *inode,unsigned long nblocks) {@@ -572,11 +572,11 @@ static int erofs_prepare_inode_buffer(struct erofs_inode *inode)int ret; inode->datalayout =EROFS_INODE_FLAT_PLAIN ; -noinline:/* expend an extra block for tail-end data */ ret =erofs_prepare_tail_block (inode);if (ret)return ret; +noinline: bh =erofs_balloc (INODE , inodesize,0 ,0 );if (IS_ERR (bh))return PTR_ERR (bh);Signature · 1 line
1.9.1
Quoted 21 lines · Yue Hu
Quoted 10 lines · On Thu, 17 Jun 2021 17:04:29 +0800 Gao Xiang <[email protected]>…
Hi Yue,
Quoted 6 lines · Yue Hu
From: Yue Hu <[email protected]>
erofs_write_compressed_file()will always setinode->idata_size= 0 if succeed, that means no tail-end data for compressed files. So, no need to callerofs_prepare_tail_block()which is used to handle tail-end data for that case. Just skip it.Quoted 7 lines
Thanks for the patch, due to somewhat long time so I don't quite remember the exact logic here now...
Yet from the description before, it's not strictly correct since my original intention would be to support tail-packing inline compressed data which is similar to uncompressed case to decrease tail extent I/O and save more space.
Quoted 6 lines
nice.
Quoted 2 lines
BTW, if you have some interest, would you like to implement it? :)
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
Yeah, if you have extra time and interest, more ideas / thoughts / discussions are always welcomed ;)
Thanks,
Gao Xiang
Quoted 43 lines
Thanks.
Quoted 40 lines
Thanks,
Gao XiangQuoted 36 lines
Also, correct 'a inode' -> 'an inode'.
Signed-off-by Yue Hu <[email protected]>--- lib/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/inode.c +2
-2@@ -111,7 +111,7 @@ struct erofs_dentry *erofs_d_alloc(struct erofs_inode *parent,return d; } -/* allocate main data for a inode */ +/* allocate main data for an inode */ static int __allocate_inode_bh_data (struct erofs_inode *inode,unsigned long nblocks) {@@ -572,11 +572,11 @@ static int erofs_prepare_inode_buffer(struct erofs_inode *inode)int ret; inode->datalayout =EROFS_INODE_FLAT_PLAIN ; -noinline:/* expend an extra block for tail-end data */ ret =erofs_prepare_tail_block (inode);if (ret)return ret; +noinline: bh =erofs_balloc (INODE , inodesize,0 ,0 );if (IS_ERR (bh))return PTR_ERR (bh);Signature · 1 line
1.9.1
Quoted 25 lines · Gao Xiang
Quoted 24 lines · Yue Hu
Quoted 10 lines · On Thu, 17 Jun 2021 17:04:29 +0800 Gao Xiang <[email protected]>…
Hi Yue,
Quoted 6 lines · Yue Hu
From: Yue Hu <[email protected]>
erofs_write_compressed_file()will always setinode->idata_size= 0 if succeed, that means no tail-end data for compressed files. So, no need to callerofs_prepare_tail_block()which is used to handle tail-end data for that case. Just skip it.Quoted 7 lines
Thanks for the patch, due to somewhat long time so I don't quite remember the exact logic here now...
Yet from the description before, it's not strictly correct since my original intention would be to support tail-packing inline compressed data which is similar to uncompressed case to decrease tail extent I/O and save more space.
nice.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
BTW, if you have some interest, would you like to implement it? :)
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
I mean try to implement non-compact (legacy) compress index format cases first.
Quoted 51 lines
Yeah, if you have extra time and interest, more ideas / thoughts / discussions are always welcomed ;)
Thanks,
Gao XiangQuoted 43 lines
Thanks.
Quoted 40 lines
Thanks,
Gao XiangQuoted 36 lines
Also, correct 'a inode' -> 'an inode'.
Signed-off-by Yue Hu <[email protected]>--- lib/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/inode.c +2
-2@@ -111,7 +111,7 @@ struct erofs_dentry *erofs_d_alloc(struct erofs_inode *parent,return d; } -/* allocate main data for a inode */ +/* allocate main data for an inode */ static int __allocate_inode_bh_data (struct erofs_inode *inode,unsigned long nblocks) {@@ -572,11 +572,11 @@ static int erofs_prepare_inode_buffer(struct erofs_inode *inode)int ret; inode->datalayout =EROFS_INODE_FLAT_PLAIN ; -noinline:/* expend an extra block for tail-end data */ ret =erofs_prepare_tail_block (inode);if (ret)return ret; +noinline: bh =erofs_balloc (INODE , inodesize,0 ,0 );if (IS_ERR (bh))return PTR_ERR (bh);Signature · 1 line
1.9.1
Quoted 28 lines · On Thu, 17 Jun 2021 17:30:26 +0800 Gao Xiang <[email protected]>…
Quoted 27 lines · Gao Xiang
Quoted 26 lines · Yue Hu
Quoted 10 lines · On Thu, 17 Jun 2021 17:04:29 +0800 Gao Xiang <[email protected]>…
Hi Yue,
Quoted 6 lines · Yue Hu
From: Yue Hu <[email protected]>
erofs_write_compressed_file()will always setinode->idata_size= 0 if succeed, that means no tail-end data for compressed files. So, no need to callerofs_prepare_tail_block()which is used to handle tail-end data for that case. Just skip it.Quoted 7 lines
Thanks for the patch, due to somewhat long time so I don't quite remember the exact logic here now...
Yet from the description before, it's not strictly correct since my original intention would be to support tail-packing inline compressed data which is similar to uncompressed case to decrease tail extent I/O and save more space.
nice.
Quoted 2 lines
BTW, if you have some interest, would you like to implement it? :)
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
I mean try to implement non-compact (legacy) compress index format cases first.
Ok, let me try to implement it.
Thanks.
Quoted 52 lines
Quoted 52 lines
Yeah, if you have extra time and interest, more ideas / thoughts / discussions are always welcomed ;)
Thanks,
Gao XiangQuoted 43 lines
Thanks.
Quoted 40 lines
Thanks,
Gao XiangQuoted 36 lines
Also, correct 'a inode' -> 'an inode'.
Signed-off-by Yue Hu <[email protected]>--- lib/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/inode.c +2
-2@@ -111,7 +111,7 @@ struct erofs_dentry *erofs_d_alloc(struct erofs_inode *parent,return d; } -/* allocate main data for a inode */ +/* allocate main data for an inode */ static int __allocate_inode_bh_data (struct erofs_inode *inode,unsigned long nblocks) {@@ -572,11 +572,11 @@ static int erofs_prepare_inode_buffer(struct erofs_inode *inode)int ret; inode->datalayout =EROFS_INODE_FLAT_PLAIN ; -noinline:/* expend an extra block for tail-end data */ ret =erofs_prepare_tail_block (inode);if (ret)return ret; +noinline: bh =erofs_balloc (INODE , inodesize,0 ,0 );if (IS_ERR (bh))return PTR_ERR (bh);Signature · 1 line
1.9.1
Quoted 31 lines · On Thu, 17 Jun 2021 18:14:17 +0800 Yue Hu <[email protected]> wrote:
Quoted 28 lines · On Thu, 17 Jun 2021 17:30:26 +0800 Gao Xiang <[email protected]>…
Quoted 27 lines · Gao Xiang
Quoted 26 lines · Yue Hu
Quoted 10 lines · On Thu, 17 Jun 2021 17:04:29 +0800 Gao Xiang <[email protected]>…
Hi Yue,
Quoted 6 lines · Yue Hu
From: Yue Hu <[email protected]>
erofs_write_compressed_file()will always setinode->idata_size= 0 if succeed, that means no tail-end data for compressed files. So, no need to callerofs_prepare_tail_block()which is used to handle tail-end data for that case. Just skip it.Quoted 7 lines
Thanks for the patch, due to somewhat long time so I don't quite remember the exact logic here now...
Yet from the description before, it's not strictly correct since my original intention would be to support tail-packing inline compressed data which is similar to uncompressed case to decrease tail extent I/O and save more space.
nice.
Quoted 2 lines
BTW, if you have some interest, would you like to implement it? :)
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
I mean try to implement non-compact (legacy) compress index format cases first.
I'm trying to do it under 4.19 code (since i have no 5.x environment temporarily).
Now, i think mkfs should be done. But, kernel side seems not working fine(no crash, no decompression warning/bug). Only some files are working, others not. I'm sure i can catch the inline data correctly via file dump. And I'm trying debug the issue. Maybe i need more time to read/understand more decompression code related.
BTW, now i understand no need to go z_erofs_vle_work_xxx for inline part(cur-end) , just go next_part after mapping as below, am i right?
Thanks.
Quoted 57 lines
Ok, let me try to implement it.
Thanks.
Quoted 52 lines
Quoted 52 lines
Yeah, if you have extra time and interest, more ideas / thoughts / discussions are always welcomed ;)
Thanks,
Gao XiangQuoted 43 lines
Thanks.
Quoted 40 lines
Thanks,
Gao XiangQuoted 36 lines
Also, correct 'a inode' -> 'an inode'.
Signed-off-by Yue Hu <[email protected]>--- lib/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)lib/inode.c +2
-2@@ -111,7 +111,7 @@ struct erofs_dentry *erofs_d_alloc(struct erofs_inode *parent,return d; } -/* allocate main data for a inode */ +/* allocate main data for an inode */ static int __allocate_inode_bh_data (struct erofs_inode *inode,unsigned long nblocks) {@@ -572,11 +572,11 @@ static int erofs_prepare_inode_buffer(struct erofs_inode *inode)int ret; inode->datalayout =EROFS_INODE_FLAT_PLAIN ; -noinline:/* expend an extra block for tail-end data */ ret =erofs_prepare_tail_block (inode);if (ret)return ret; +noinline: bh =erofs_balloc (INODE , inodesize,0 ,0 );if (IS_ERR (bh))return PTR_ERR (bh);Signature · 1 line
1.9.1
Hi Yue,
Quoted 45 lines · Yue Hu
Quoted 31 lines · On Thu, 17 Jun 2021 18:14:17 +0800 Yue Hu <[email protected]> wrote:
Quoted 28 lines · On Thu, 17 Jun 2021 17:30:26 +0800 Gao Xiang <[email protected]>…
Quoted 27 lines · Gao Xiang
Quoted 26 lines · Yue Hu
Quoted 10 lines · On Thu, 17 Jun 2021 17:04:29 +0800 Gao Xiang <[email protected]>…
Hi Yue,
Quoted 6 lines · Yue Hu
From: Yue Hu <[email protected]>
erofs_write_compressed_file()will always setinode->idata_size= 0 if succeed, that means no tail-end data for compressed files. So, no need to callerofs_prepare_tail_block()which is used to handle tail-end data for that case. Just skip it.Quoted 7 lines
Thanks for the patch, due to somewhat long time so I don't quite remember the exact logic here now...
Yet from the description before, it's not strictly correct since my original intention would be to support tail-packing inline compressed data which is similar to uncompressed case to decrease tail extent I/O and save more space.
nice.
Quoted 2 lines
BTW, if you have some interest, would you like to implement it? :)
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
I mean try to implement non-compact (legacy) compress index format cases first.
I'm trying to do it under 4.19 code (since i have no 5.x environment temporarily).
Quoted 8 lines
Now, i think mkfs should be done. But, kernel side seems not working
fine(no crash, no decompression warning/bug). Only some files are working, others not. I'm sure i can catch the inline data correctly via file dump. And I'm trying debug the issue. Maybe i need more time to read/understand more decompression code related.BTW, now i understand no need to go
z_erofs_vle_work_xxxfor inline part(cur-end) , just gonext_partafter mapping as below, am i right?
You are right. For the common cases (except for fiemap or cases to get the exact decompressed length), we only need to calculate the start of the compression extent, so it's transversal in the reverse order.
But really... Again, I don't suggest using 4.19 staging code for real production or further development. The uncompressed part is considered as stable, but compression side may not (also it was disabled by default). Please also see,
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/erofs/Kconfig?h=v4.19#n86
" config EROFS_FS_ZIP
bool "EROFS Data Compresssion Support"
depends on EROFS_FS
help
Currently we support VLE Compression only.
Play at your own risk.
If you don't want to use compression feature, say N. "
Our original first real production codebase was between 5.2~5.3. Therefore, I suggest using >= 5.4 LTS codebase for production. You could also find some backport codebase on github, e.g.:
https://github.com/nolange/erofs_kernel_4_19
, which backports 5.6 erofs codebase to 4.19.
As for tail-packing inline extent feature, how about focusing on on-disk design and mkfs/erofsfuse implementation first as PoC?
I'm afraid that if you only focus on 4.19 codebase, the format of compact indexes will be ignored, but "compact indexes" is the default option for erofs now since it has less metadata overhead than non-compact indexes, so both the sequential / random read are better.
Thanks,
Gao Xiang
Quoted 6 lines
Thanks.
Quoted 4 lines
Ok, let me try to implement it.
Thanks.
Quoted 81 lines · On Tue, 31 Aug 2021 19:15:50 +0800 Gao Xiang <[email protected]>…
Hi Yue,
Quoted 45 lines · Yue Hu
Quoted 31 lines · On Thu, 17 Jun 2021 18:14:17 +0800 Yue Hu <[email protected]> wrote:
Quoted 28 lines · On Thu, 17 Jun 2021 17:30:26 +0800 Gao Xiang <[email protected]>…
Quoted 27 lines · Gao Xiang
Quoted 26 lines · Yue Hu
Quoted 10 lines · On Thu, 17 Jun 2021 17:04:29 +0800 Gao Xiang <[email protected]>…
Hi Yue,
Quoted 6 lines · Yue Hu
From: Yue Hu <[email protected]>
erofs_write_compressed_file()will always setinode->idata_size= 0 if succeed, that means no tail-end data for compressed files. So, no need to callerofs_prepare_tail_block()which is used to handle tail-end data for that case. Just skip it.Quoted 7 lines
Thanks for the patch, due to somewhat long time so I don't quite remember the exact logic here now...
Yet from the description before, it's not strictly correct since my original intention would be to support tail-packing inline compressed data which is similar to uncompressed case to decrease tail extent I/O and save more space.
nice.
Quoted 2 lines
BTW, if you have some interest, would you like to implement it? :)
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
I mean try to implement non-compact (legacy) compress index format cases first.
I'm trying to do it under 4.19 code (since i have no 5.x environment temporarily).
Quoted 8 lines
Now, i think mkfs should be done. But, kernel side seems not working
fine(no crash, no decompression warning/bug). Only some files are working, others not. I'm sure i can catch the inline data correctly via file dump. And I'm trying debug the issue. Maybe i need more time to read/understand more decompression code related.BTW, now i understand no need to go
z_erofs_vle_work_xxxfor inline part(cur-end) , just gonext_partafter mapping as below, am i right?You are right. For the common cases (except for fiemap or cases to get the exact decompressed length), we only need to calculate the start of the compression extent, so it's transversal in the reverse order.
But really... Again, I don't suggest using 4.19 staging code for real production or further development. The uncompressed part is considered as stable, but compression side may not (also it was disabled by default). Please also see,
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/erofs/Kconfig?h=v4.19#n86" config
EROFS_FS_ZIP
bool "EROFS Data Compresssion Support"
depends onEROFS_FS
help
Currently we support VLE Compression only.
Play at your own risk.If you don't want to use compression feature, say N. "
Our original first real production codebase was between 5.2~5.3. Therefore, I suggest using >= 5.4 LTS codebase for production. You could also find some backport codebase on github, e.g.:
https://github.com/nolange/erofs_kernel_4_19
, which backports 5.6 erofs codebase to 4.19.
Quoted 7 lines
As for tail-packing inline extent feature, how about focusing on on-disk design and mkfs/erofsfuse implementation first as PoC?
I'm afraid that if you only focus on 4.19 codebase, the format of compact indexes will be ignored, but "compact indexes" is the default option for erofs now since it has less metadata overhead than non-compact indexes, so both the sequential / random read are better.
OK, let me develop it under 5.4. I need taking time to find it:)
Thanks.
Quoted 10 lines
Thanks,
Gao XiangQuoted 6 lines
Thanks.
Quoted 4 lines
Ok, let me try to implement it.
Thanks.
Quoted 2 lines · Yue Hu
On Tue, 31 Aug 2021 19:15:50 +0800
Gao Xiang <[email protected]> wrote:
...
Quoted 51 lines
Quoted 51 lines
Quoted 15 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
BTW, if you have some interest, would you like to implement it? :)
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
I mean try to implement non-compact (legacy) compress index format cases first.
I'm trying to do it under 4.19 code (since i have no 5.x environment temporarily).
Quoted 8 lines
Now, i think mkfs should be done. But, kernel side seems not working
fine(no crash, no decompression warning/bug). Only some files are working, others not. I'm sure i can catch the inline data correctly via file dump. And I'm trying debug the issue. Maybe i need more time to read/understand more decompression code related.BTW, now i understand no need to go
z_erofs_vle_work_xxxfor inline part(cur-end) , just gonext_partafter mapping as below, am i right?You are right. For the common cases (except for fiemap or cases to get the exact decompressed length), we only need to calculate the start of the compression extent, so it's transversal in the reverse order.
But really... Again, I don't suggest using 4.19 staging code for real production or further development. The uncompressed part is considered as stable, but compression side may not (also it was disabled by default). Please also see,
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/erofs/Kconfig?h=v4.19#n86" config
EROFS_FS_ZIP
bool "EROFS Data Compresssion Support"
depends onEROFS_FS
help
Currently we support VLE Compression only.
Play at your own risk.If you don't want to use compression feature, say N. "
Our original first real production codebase was between 5.2~5.3. Therefore, I suggest using >= 5.4 LTS codebase for production. You could also find some backport codebase on github, e.g.:
https://github.com/nolange/erofs_kernel_4_19
, which backports 5.6 erofs codebase to 4.19.As for tail-packing inline extent feature, how about focusing on on-disk design and mkfs/erofsfuse implementation first as PoC?
Quoted 6 lines
Quoted 4 lines
I'm afraid that if you only focus on 4.19 codebase, the format of compact indexes will be ignored, but "compact indexes" is the default option for erofs now since it has less metadata overhead than non-compact indexes, so both the sequential / random read are better.
OK, let me develop it under 5.4. I need taking time to find it:)
As the first step of kernel development, I think using x86 qemu should be better since it's easier to debug than on the embedded device.
For this feature, I'm very glad to discuss some on-disk format first. Since it's not trivial for compact indexes since it's impossible to mark tailing-packing extent with some special blkaddr like non-compact indexes.
My rough thought about this is "to add some new feature flag to "struct z_erofs_map_header" and trigger z_erofs_map_blocks(i_size - 1); at a proper time to get all information about the last tail-packing compression extent", and when submitting io, we erofs_get_meta_page() instead and fill the compressed pages.
But anyway, I still think focusing on mkfs.erofs and erofsfuse is a good start for this.
Thanks,
Gao Xiang
Quoted 13 lines
Thanks.
Quoted 10 lines
Thanks,
Gao XiangQuoted 6 lines
Thanks.
Quoted 4 lines
Ok, let me try to implement it.
Thanks.
Quoted 62 lines · On Tue, 31 Aug 2021 20:20:11 +0800 Gao Xiang <[email protected]>…
Quoted 2 lines · Yue Hu
On Tue, 31 Aug 2021 19:15:50 +0800
Gao Xiang <[email protected]> wrote:...
Quoted 56 lines
Quoted 55 lines
Quoted 15 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
BTW, if you have some interest, would you like to implement it? :)
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
I mean try to implement non-compact (legacy) compress index format cases first.
I'm trying to do it under 4.19 code (since i have no 5.x environment temporarily).
Quoted 8 lines
Now, i think mkfs should be done. But, kernel side seems not working
fine(no crash, no decompression warning/bug). Only some files are working, others not. I'm sure i can catch the inline data correctly via file dump. And I'm trying debug the issue. Maybe i need more time to read/understand more decompression code related.BTW, now i understand no need to go
z_erofs_vle_work_xxxfor inline part(cur-end) , just gonext_partafter mapping as below, am i right?You are right. For the common cases (except for fiemap or cases to get the exact decompressed length), we only need to calculate the start of the compression extent, so it's transversal in the reverse order.
But really... Again, I don't suggest using 4.19 staging code for real production or further development. The uncompressed part is considered as stable, but compression side may not (also it was disabled by default). Please also see,
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/erofs/Kconfig?h=v4.19#n86" config
EROFS_FS_ZIP
bool "EROFS Data Compresssion Support"
depends onEROFS_FS
help
Currently we support VLE Compression only.
Play at your own risk.If you don't want to use compression feature, say N. "
Our original first real production codebase was between 5.2~5.3. Therefore, I suggest using >= 5.4 LTS codebase for production. You could also find some backport codebase on github, e.g.:
https://github.com/nolange/erofs_kernel_4_19
, which backports 5.6 erofs codebase to 4.19.As for tail-packing inline extent feature, how about focusing on on-disk design and mkfs/erofsfuse implementation first as PoC?
I'm afraid that if you only focus on 4.19 codebase, the format of compact indexes will be ignored, but "compact indexes" is the default option for erofs now since it has less metadata overhead than non-compact indexes, so both the sequential / random read are better.
Quoted 4 lines
Quoted 1 line
OK, let me develop it under 5.4. I need taking time to find it:)
As the first step of kernel development, I think using x86 qemu should be better since it's easier to debug than on the embedded device.
Agree.
Quoted 5 lines
For this feature, I'm very glad to discuss some on-disk format first. Since it's not trivial for compact indexes since it's impossible to mark tailing-packing extent with some special blkaddr like non-compact indexes.
Yes, blkaddr should be an issue for inline case. I can feel that faintly.
Quoted 6 lines
My rough thought about this is "to add some new feature flag to "struct
z_erofs_map_header" and triggerz_erofs_map_blocks(i_size - 1); at a proper time to get all information about the last tail-packing compression extent", and when submitting io, weerofs_get_meta_page()instead and fill the compressed pages.
Firstly, I need to add code about inline part to verify my understanding. I think i did it almost about what i want to know including z_erofs_map_blocks() since i can catch the inline data which is key point for me although kernel side does not work fine totally.
Then i can re-factor/re-write it based on that. Yes, i will switch it on >=5.4 to continue developing later.
I also think we need a new flag for inline case. I'm just not focus on the flag due to my working step above.
Now, i think i can check it about where to add the new flag more proper. Let me check it also for your thought mentioned above.
Quoted 3 lines
But anyway, I still think focusing on mkfs.erofs and erofsfuse is a good start for this.
Yes, we should check the compression firstly.
One more question:
There's a piece of code (as below) to handle small output size(< PAGE_SIZE) which looks like for inline part in z_erofs_decompress_generic()? If so, we also need to go vle decompression flow for inline data just like other data case?
```code
if (rq->outputsize <= PAGE_SIZE * 7 / 8) {
dst = erofs_get_pcpubuf(0);
if (IS_ERR(dst))
return PTR_ERR(dst);
rq->inplace_io = false;
ret = alg->decompress(rq, dst);
if (!ret)
copy_from_pcpubuf(rq->out, dst, rq->pageofs_out,
rq->outputsize);
erofs_put_pcpubuf(dst);
return ret;
}
```Thanks.
Quoted 17 lines
Thanks,
Gao XiangQuoted 13 lines
Thanks.
Quoted 10 lines
Thanks,
Gao XiangQuoted 6 lines
Thanks.
Quoted 4 lines
Ok, let me try to implement it.
Thanks.
Quoted 73 lines · Yue Hu
Quoted 62 lines · On Tue, 31 Aug 2021 20:20:11 +0800 Gao Xiang <[email protected]>…
Quoted 2 lines · Yue Hu
On Tue, 31 Aug 2021 19:15:50 +0800
Gao Xiang <[email protected]> wrote:...
Quoted 56 lines
Quoted 55 lines
Quoted 15 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
BTW, if you have some interest, would you like to implement it? :)
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I don't know if i can finish it. But i would like to have a try :)
My rough thought is to try to inline the last tail compresseed extent after the on-disk compressed extents, maybe we could let it work for non-compact (legacy) compress index format cases...
I mean try to implement non-compact (legacy) compress index format cases first.
I'm trying to do it under 4.19 code (since i have no 5.x environment temporarily).
Quoted 8 lines
Now, i think mkfs should be done. But, kernel side seems not working
fine(no crash, no decompression warning/bug). Only some files are working, others not. I'm sure i can catch the inline data correctly via file dump. And I'm trying debug the issue. Maybe i need more time to read/understand more decompression code related.BTW, now i understand no need to go
z_erofs_vle_work_xxxfor inline part(cur-end) , just gonext_partafter mapping as below, am i right?You are right. For the common cases (except for fiemap or cases to get the exact decompressed length), we only need to calculate the start of the compression extent, so it's transversal in the reverse order.
But really... Again, I don't suggest using 4.19 staging code for real production or further development. The uncompressed part is considered as stable, but compression side may not (also it was disabled by default). Please also see,
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/staging/erofs/Kconfig?h=v4.19#n86" config
EROFS_FS_ZIP
bool "EROFS Data Compresssion Support"
depends onEROFS_FS
help
Currently we support VLE Compression only.
Play at your own risk.If you don't want to use compression feature, say N. "
Our original first real production codebase was between 5.2~5.3. Therefore, I suggest using >= 5.4 LTS codebase for production. You could also find some backport codebase on github, e.g.:
https://github.com/nolange/erofs_kernel_4_19
, which backports 5.6 erofs codebase to 4.19.As for tail-packing inline extent feature, how about focusing on on-disk design and mkfs/erofsfuse implementation first as PoC?
I'm afraid that if you only focus on 4.19 codebase, the format of compact indexes will be ignored, but "compact indexes" is the default option for erofs now since it has less metadata overhead than non-compact indexes, so both the sequential / random read are better.
Quoted 4 lines
Quoted 1 line
OK, let me develop it under 5.4. I need taking time to find it:)
As the first step of kernel development, I think using x86 qemu should be better since it's easier to debug than on the embedded device.
Agree.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
For this feature, I'm very glad to discuss some on-disk format first. Since it's not trivial for compact indexes since it's impossible to mark tailing-packing extent with some special blkaddr like non-compact indexes.
Yes, blkaddr should be an issue for inline case. I can feel that faintly.
Yeah, my point is that "don't mark tail-packing extent since it's somewhat unfriendly to compact indexes."
Quoted 8 lines
Quoted 7 lines
My rough thought about this is "to add some new feature flag to "struct
z_erofs_map_header" and triggerz_erofs_map_blocks(i_size - 1); at a proper time to get all information about the last tail-packing compression extent", and when submitting io, weerofs_get_meta_page()instead and fill the compressed pages.
Quoted 4 lines
Firstly, I need to add code about inline part to verify my understanding. I think i did it almost about what i want to know including
z_erofs_map_blocks()since i can catch the inline data which is key point for me although kernel side does not work fine totally.
Ok, for z_erofs_map_blocks part, erofs-utils and erofs are similiar, I think maybe debugging erofs-utils makes the life easier...
Quoted 6 lines
Then i can re-factor/re-write it based on that. Yes, i will switch it on >=5.4 to continue developing later.
I also think we need a new flag for inline case. I'm just not focus on the flag due to my working step above.
Yeah, my suggestion is adding a new flag in "struct z_erofs_map_header". And reading the last extent when initializing inode or first read.
Quoted 16 lines
Now, i think i can check it about where to add the new flag more proper. Let me check it also for your thought mentioned above.
Quoted 3 lines
But anyway, I still think focusing on mkfs.erofs and erofsfuse is a good start for this.
Yes, we should check the compression firstly.
One more question:
There's a piece of code (as below) to handle small output
size(< PAGE_SIZE)which looks like for inline part inz_erofs_decompress_generic()? If so, we also need to go vle decompression flow for inline data just like other data case?
Quoted 16 lines
```code
if (rq->outputsize <= PAGE_SIZE * 7 / 8) {
dst = erofs_get_pcpubuf(0);
if (IS_ERR(dst))
return PTR_ERR(dst);
rq->inplace_io = false;
ret = alg->decompress(rq, dst);
if (!ret)
copy_from_pcpubuf(rq->out, dst, rq->pageofs_out,
rq->outputsize);
erofs_put_pcpubuf(dst);
return ret;
}
```Hmmmm.... Nope, this is mainly used to avoid vmap or copy if total output size is smaller than PAGE_SIZE for inplace I/O.
Considering the following cases, if we read compressed data into a file page and inplace decompression is _not_ feasible due to lack of safe inplace margin so we must copy out compressed data in advance.
Generally, for the case above we copy compressed data to some per-CPU buffer and then do vmap for all needed file pages. But if decompressed length is less than one page, we could decompress into per-CPU buffer and copy decompressed data into file page instead, which is faster than the generic way and is measurable.
Thanks,
Gao Xiang
Quoted 21 lines
Thanks.
Quoted 17 lines
Thanks,
Gao XiangQuoted 13 lines
Thanks.
Quoted 10 lines
Thanks,
Gao XiangQuoted 6 lines
Thanks.
Quoted 4 lines
Ok, let me try to implement it.
Thanks.