compressdev: add LZ4 support
cover compressdev: add LZ4 support
Add support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already exists in both app doc and arguments but isn't implemented inside the application.
LZ4: https://github.com/lz4/lz4
RFC: https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
Depends-on: patch-26344 ("compressdev: fix end of comp PMD list macro conflict")
Michael Baum (4):
compressdev: add LZ4 algorithm support app/test-compress-perf: allow test single compress operation app/test-compress-perf: add algo option app/test-compress-perf: add LZ4 support
app/test-compress-perf/comp_perf_options.h | 9 +- .../comp_perf_options_parse.c | 83 +++++++++++- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 82 +++++++----- .../comp_perf_test_throughput.c | 75 +++++++---- .../comp_perf_test_verify.c | 75 +++++++---- app/test-compress-perf/main.c | 63 +++++++-- doc/guides/compressdevs/features/default.ini | 4 + doc/guides/rel_notes/release_23_03.rst | 16 +++ doc/guides/tools/comp_perf.rst | 5 + lib/compressdev/rte_comp.c | 6 + lib/compressdev/rte_comp.h | 51 ++++++- 12 files changed, 451 insertions(+), 144 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/4 compressdev: add LZ4 algorithm support
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.
--- doc/guides/compressdevs/features/default.ini | 4 ++ doc/guides/rel_notes/release_23_03.rst | 7 +++ lib/compressdev/rte_comp.c | 6 +++ lib/compressdev/rte_comp.h | 51 +++++++++++++++++++- 4 files changed, 66 insertions(+), 2 deletions(-)
doc/guides/compressdevs/features/default.ini +4 -0
@@ -20,8 +20,12 @@ OOP SGL In LB Out = OOP LB In SGL Out = Deflate = LZS = +LZ4 = Adler32 = Crc32 = Adler32&Crc32 = +xxHash32 = Fixed = Dynamic = +LZ4 Block Checksum = +LZ4 Block Independence =
doc/guides/rel_notes/release_23_03.rst +7 -0
@@ -55,6 +55,13 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added LZ4 algorithm in Compressdev Library.** + + Added new compression algorithm, including: + + * Added support for ``RTE_COMP_ALGO_LZ4``. + * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + Removed Items -------------
lib/compressdev/rte_comp.c +6 -0
@@ -39,6 +39,12 @@ rte_comp_get_feature_name(uint64_t flag)return "HUFFMAN_FIXED" ;case RTE_COMP_FF_HUFFMAN_DYNAMIC :return "HUFFMAN_DYNAMIC" ; +case RTE_COMP_FF_XXHASH32_CHECKSUM : +return "XXHASH32_CHECKSUM" ; +case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE : +return "LZ4_BLOCK_INDEPENDENCE" ; +case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM : +return "LZ4_BLOCK_WITH_CHECKSUM" ;default :return NULL ; }
lib/compressdev/rte_comp.h +49 -2
95 lines not on this page · raw message
Signature · 1 line
2.25.1
RE: [EXT] [PATCH 1/4] compressdev: add LZ4 algorithm support
Quoted 16 lines
lib/compressdev/rte_comp.h +16
-0+/** +* Block checksum flag. +* If this flag is set, each data block will be followed by a 4-bytes checksum, +* calculated by using the xxHash-32 algorithm on the raw (compressed) data +* block. The intention is to detect data corruption (storage or transmission +* errors) immediately, before decoding. Block checksum usage is optional. +*/ +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4) + +/** +* Block Independence flag. +* If this flag is set to 1, blocks are independent. +* If this flag is set to 0, each block depends on previous ones (up to LZ4 +* window size, which is 64 KB). In such case, it is necessary to decode all +* blocks in sequence. +* Block dependency improves compression ratio, especially for small blocks.
Quoted 5 lines
On
+ * the other hand, it makes random access or multi-threaded decoding impossible.
+ */
+#defineRTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE(1 << 5)
Why did you start with 4th and 5th bit of the flags? Why not first two bits?
++ more people for review.
RE: [EXT] [PATCH 1/4] compressdev: add LZ4 algorithm support
Quoted 16 lines · Akhil Goyal
Quoted 16 lines
lib/compressdev/rte_comp.h +15
-0+/** +* Block checksum flag. +* If this flag is set, each data block will be followed by a 4-bytes checksum, +* calculated by using the xxHash-32 algorithm on the raw (compressed) data +* block. The intention is to detect data corruption (storage or transmission +* errors) immediately, before decoding. Block checksum usage is optional. +*/ +#define RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM (1 << 4) + +/** +* Block Independence flag. +* If this flag is set to 1, blocks are independent. +* If this flag is set to 0, each block depends on previous ones (up to LZ4 +* window size, which is 64 KB). In such case, it is necessary to decode all +* blocks in sequence.
Quoted 8 lines
Quoted 6 lines
+ * Block dependency improves compression ratio, especially for small blocks. On
+ * the other hand, it makes random access or multi-threaded decoding impossible.
+ */
+#defineRTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE(1 << 5)Why did you start with 4th and 5th bit of the flags? Why not first two bits?
I didn't choose the values by myself, I took them from LZ4 standard:
https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor
Quoted 2 lines
++ more people for review.
RE: [EXT] [PATCH 1/4] compressdev: add LZ4 algorithm support
Quoted 38 lines
-----Original Message-----
From: Michael Baum <[email protected]>
Sent: Tuesday, January 31, 2023 2:07 AM
To: Akhil Goyal <[email protected]>; [email protected]; Mahipal Challa <[email protected]>; Fan Zhang <[email protected]>; Ashish Gupta <[email protected]>
Cc: Matan Azrad <[email protected]>; Fiona Trahe <[email protected]>; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>
Subject: RE: [EXT] [PATCH 1/4] compressdev: add LZ4 algorithm supportQuoted 5 lines · Akhil Goyal
Quoted 5 lines
+/**
+ * Block checksum flag.
+ * If this flag is set, each data block will be followed by a 4-bytes checksum,
+ * calculated by using the xxHash-32 algorithm on the raw (compressed)data
Quoted 4 lines
Quoted 4 lines
+ * block. The intention is to detect data corruption (storage or transmission
+ * errors) immediately, before decoding. Block checksum usage is optional.
+ */
+#defineRTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM(1 << 4)Quoted 8 lines
lib/compressdev/rte_comp.h enum rte_comp_op_type {
Quoted 8 lines
lib/compressdev/rte_comp.h +8
-0+ +/** +* Block Independence flag. +* If this flag is set to 1, blocks are independent. +* If this flag is set to 0, each block depends on previous ones (up to LZ4 +* window size, which is 64 KB). In such case, it is necessary to decode all +* blocks in sequence. +* Block dependency improves compression ratio, especially for small blocks.
Quoted 6 lines
Quoted 5 lines
On
+ * the other hand, it makes random access or multi-threaded decoding impossible.
+ */
+#defineRTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE(1 << 5)
Quoted 4 lines
Quoted 1 line
Why did you start with 4th and 5th bit of the flags? Why not first two bits?
I didn't choose the values by myself, I took them from LZ4 standard:
https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor
Ok, Are we defining and making the frame descriptor inside library? Also, why all the bits not added?
I believe these are just flags which PMD would use to make the frame descriptors. If so, we should start with 0th bit and let the PMD handle bit shifting.
PATCH 2/4 app/test-compress-perf: allow test single compress operation
In reply to Michael BaumPart of the application options is to test only compress and only decompress but actually the application ignores this user option and tries to test always both compress and decompress.
Allow testing only compress and only decompress.
--- app/test-compress-perf/comp_perf_options.h | 6 +- .../comp_perf_options_parse.c | 4 +- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 71 ++++++---- .../comp_perf_test_throughput.c | 64 +++++---- .../comp_perf_test_verify.c | 64 +++++---- app/test-compress-perf/main.c | 8 ++ doc/guides/rel_notes/release_23_03.rst | 4 + 8 files changed, 230 insertions(+), 117 deletions(-)
app/test-compress-perf/comp_perf_options.h +3 -3
@@ -32,9 +32,9 @@ enum cperf_test_type { };enum comp_operation { -COMPRESS_ONLY , -DECOMPRESS_ONLY , -COMPRESS_DECOMPRESS +COMPRESS = (1 <<0 ), +DECOMPRESS = (1 <<1 ), +COMPRESS_DECOMPRESS = (COMPRESS |DECOMPRESS ), };struct range_list {
app/test-compress-perf/comp_perf_options_parse.c +2 -2
@@ -446,11 +446,11 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)struct name_id_map optype_namemap[] = { {"comp" , -COMPRESS_ONLY +COMPRESS }, {"decomp" , -DECOMPRESS_ONLY +DECOMPRESS }, {"comp_and_decomp" ,
app/test-compress-perf/comp_perf_test_common.c +91 -35
222 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_cyclecount.c +44 -27
89 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_throughput.c +38 -26
79 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_verify.c +40 -24
105 lines not on this page · raw message
app/test-compress-perf/main.c +8 -0
@@ -254,6 +254,14 @@ comp_perf_dump_input_data(struct comp_test_data *test_data) goto end; } +if (!(test_data->test_op &COMPRESS ) && + test_data->input_data_sz > + (size_t ) test_data->seg_sz * (size_t ) test_data->max_sgl_segs) { +RTE_LOG (ERR ,USER1 , +"Size of input must be less than total segments\n" ); + goto end; + } + test_data->input_data =rte_zmalloc_socket (NULL , test_data->input_data_sz,0 ,rte_socket_id ());
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -62,6 +62,10 @@ New Features * Added support for ``RTE_COMP_ALGO_LZ4``. * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. +* **Allowed test single compress operation in test-compress-perf.** + + Enable the application options for testing only compress and only decompress. + Removed Items -------------
Signature · 1 line
2.25.1
PATCH 3/4 app/test-compress-perf: add algo option
In reply to Michael BaumAdd a command line option algo to select the compress algorithm supported by the compress API: null (DMA), deflate and lzs.
Default for deflate.
--- app/test-compress-perf/comp_perf_options.h | 2 + .../comp_perf_options_parse.c | 40 ++++++++++++++++++- .../comp_perf_test_cyclecount.c | 4 +- .../comp_perf_test_throughput.c | 4 +- .../comp_perf_test_verify.c | 4 +- app/test-compress-perf/main.c | 38 +++++++++++------- doc/guides/rel_notes/release_23_03.rst | 4 ++ doc/guides/tools/comp_perf.rst | 2 + 8 files changed, 76 insertions(+), 22 deletions(-)
app/test-compress-perf/comp_perf_options.h +2 -0
@@ -63,6 +63,8 @@ struct comp_test_data {enum rte_comp_huffman huffman_enc;enum comp_operation test_op; +enum rte_comp_algorithm test_algo; +int window_sz;struct range_list level_lst;uint8_t level;
app/test-compress-perf/comp_perf_options_parse.c +38 -2
@@ -25,6 +25,7 @@#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")#define CPERF_NUM_ITER ("num-iter")#define CPERF_OPTYPE ("operation") +#define CPERF_ALGO ("algo")#define CPERF_HUFFMAN_ENC ("huffman-enc")#define CPERF_LEVEL ("compress-level")#define CPERF_WINDOW_SIZE ("window-sz")
@@ -56,6 +57,8 @@ usage(char *progname)" compressed/decompressed (default: 10000)\n" " --operation [comp/decomp/comp_and_decomp]: perform test on\n" " compression, decompression or both operations\n" +" --algo [null/deflate/lzs]: perform test on algorithm\n" +" null(DMA), deflate or lzs (default: deflate)\n" " --huffman-enc [fixed/dynamic/default]: Huffman encoding\n" " (default: dynamic)\n" " --compress-level N: compression level, which could be a single value, list or range\n"
@@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)return 0 ; } +static int +parse_algo (struct comp_test_data *test_data,const char *arg) +{ +struct name_id_map algo_namemap[] = { + { +"null" , +RTE_COMP_ALGO_NULL + }, + { +"deflate" , +RTE_COMP_ALGO_DEFLATE + }, + { +"lzs" , +RTE_COMP_ALGO_LZS + } + }; + +int id =get_str_key_id_mapping (algo_namemap, +RTE_DIM (algo_namemap), arg); +if (id <0 ) { +RTE_LOG (ERR ,USER1 ,"Invalid algorithm specified\n" ); +return -1 ; + } + + test_data->test_algo = (enum rte_comp_algorithm )id; + +return 0 ; +} +static int parse_huffman_enc (struct comp_test_data *test_data,const char *arg) {
@@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)int id =get_str_key_id_mapping (huffman_namemap,RTE_DIM (huffman_namemap), arg);if (id <0 ) { -RTE_LOG (ERR ,USER1 ,"Invalid Huffmane encoding specified\n" ); +RTE_LOG (ERR ,USER1 ,"Invalid Huffman encoding specified\n" );return -1 ; }
@@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)/* * Try parsing the argument as a range, if it fails, -* arse it as a list +* parse it as a list */ if (parse_range(arg, &test_data->level_lst.min, &test_data->level_lst.max,
@@ -572,6 +605,7 @@ static struct option lgopts[] = { { CPERF_MAX_SGL_SEGS, required_argument, 0, 0}, { CPERF_NUM_ITER, required_argument, 0, 0 }, { CPERF_OPTYPE, required_argument, 0, 0 }, + { CPERF_ALGO, required_argument, 0, 0 }, { CPERF_HUFFMAN_ENC, required_argument, 0, 0 }, { CPERF_LEVEL, required_argument, 0, 0 }, { CPERF_WINDOW_SIZE, required_argument, 0, 0 },
@@ -594,6 +628,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data) { CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs }, { CPERF_NUM_ITER, parse_num_iter }, { CPERF_OPTYPE, parse_op_type }, + { CPERF_ALGO, parse_algo }, { CPERF_HUFFMAN_ENC, parse_huffman_enc }, { CPERF_LEVEL, parse_level }, { CPERF_WINDOW_SIZE, parse_window_sz },
@@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data *test_data) test_data->num_iter =10000 ; test_data->huffman_enc =RTE_COMP_HUFFMAN_DYNAMIC ; test_data->test_op =COMPRESS_DECOMPRESS ; + test_data->test_algo =RTE_COMP_ALGO_DEFLATE ; test_data->window_sz =-1 ; test_data->level_lst.min =RTE_COMP_LEVEL_MIN ; test_data->level_lst.max =RTE_COMP_LEVEL_MAX ;
app/test-compress-perf/comp_perf_test_cyclecount.c +2 -2
@@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/comp_perf_test_throughput.c +2 -2
@@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/comp_perf_test_verify.c +2 -2
@@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/main.c +24 -14
@@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id) {const struct rte_compressdev_capabilities *cap; - cap =rte_compressdev_capability_get (cdev_id, -RTE_COMP_ALGO_DEFLATE ); + cap =rte_compressdev_capability_get (cdev_id, test_data->test_algo);if (cap ==NULL ) {RTE_LOG (ERR ,USER1 , -"Compress device does not support DEFLATE\n" ); +"Compress device does not support %u algorithm\n" , + test_data->test_algo);return -1 ; }uint64_t comp_flags = cap->comp_feature_flags; -/* Huffman encoding */ -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_FIXED && - (comp_flags &RTE_COMP_FF_HUFFMAN_FIXED ) ==0 ) { -RTE_LOG (ERR ,USER1 , -"Compress device does not supported Fixed Huffman\n" ); -return -1 ; - } +/* Algorithm type */ +switch (test_data->test_algo) { +case RTE_COMP_ALGO_DEFLATE : +/* Huffman encoding */ +if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_FIXED && + (comp_flags &RTE_COMP_FF_HUFFMAN_FIXED ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not supported Fixed Huffman\n" ); +return -1 ; + } -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_DYNAMIC && - (comp_flags &RTE_COMP_FF_HUFFMAN_DYNAMIC ) ==0 ) { -RTE_LOG (ERR ,USER1 , -"Compress device does not supported Dynamic Huffman\n" ); +if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_DYNAMIC && + (comp_flags &RTE_COMP_FF_HUFFMAN_DYNAMIC ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not supported Dynamic Huffman\n" ); +return -1 ; + } +break ; +case RTE_COMP_ALGO_LZS : +case RTE_COMP_ALGO_NULL : +break ; +default :return -1 ; }
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -66,6 +66,10 @@ New Features Enable the application options for testing only compress and only decompress. +* **Added algo option in test-compress-perf.** + + Added support for testing other algorithms except for DEFLAT. + Removed Items -------------
doc/guides/tools/comp_perf.rst +2 -0
@@ -84,6 +84,8 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations + ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate) + ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic) ``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
Signature · 1 line
2.25.1
PATCH 4/4 app/test-compress-perf: add LZ4 support
In reply to Michael BaumAdd support for LZ4 algorithm and add a new parameter for it.
--- app/test-compress-perf/comp_perf_options.h | 1 + .../comp_perf_options_parse.c | 43 ++++++++++++++++++- .../comp_perf_test_cyclecount.c | 7 ++- .../comp_perf_test_throughput.c | 7 ++- .../comp_perf_test_verify.c | 7 ++- app/test-compress-perf/main.c | 17 ++++++++ doc/guides/rel_notes/release_23_03.rst | 1 + doc/guides/tools/comp_perf.rst | 5 ++- 8 files changed, 82 insertions(+), 6 deletions(-)
app/test-compress-perf/comp_perf_options.h +1 -0
@@ -61,6 +61,7 @@ struct comp_test_data {uint16_t max_sgl_segs;uint32_t total_segs; +uint8_t lz4_flags;enum rte_comp_huffman huffman_enc;enum comp_operation test_op;enum rte_comp_algorithm test_algo;
app/test-compress-perf/comp_perf_options_parse.c +41 -2
93 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_cyclecount.c +6 -1
@@ -194,13 +194,16 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; input_bufs = mem->decomp_bufs; output_bufs = mem->comp_bufs; out_seg_sz = test_data->out_seg_sz;
@@ -214,6 +217,8 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; input_bufs = mem->comp_bufs; output_bufs = mem->decomp_bufs; out_seg_sz = test_data->seg_sz;
app/test-compress-perf/comp_perf_test_throughput.c +6 -1
@@ -85,13 +85,16 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; input_bufs = mem->decomp_bufs; output_bufs = mem->comp_bufs; out_seg_sz = test_data->out_seg_sz;
@@ -105,6 +108,8 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; input_bufs = mem->comp_bufs; output_bufs = mem->decomp_bufs; out_seg_sz = test_data->seg_sz;
app/test-compress-perf/comp_perf_test_verify.c +6 -1
@@ -88,13 +88,16 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; output_data_ptr = ctx->mem.compressed_data; output_data_sz = &ctx->comp_data_sz; input_bufs = mem->decomp_bufs;
@@ -110,6 +113,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; output_data_ptr = ctx->mem.decompressed_data; output_data_sz = &ctx->decomp_data_sz; input_bufs = mem->comp_bufs;
app/test-compress-perf/main.c +17 -0
@@ -86,6 +86,23 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)return -1 ; }break ; +case RTE_COMP_ALGO_LZ4 : +/* LZ4 flags */ +if ((test_data->lz4_flags &RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM ) && + (comp_flags &RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not support LZ4 block with checksum\n" ); +return -1 ; + } + +if ((test_data->lz4_flags & +RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE ) && + (comp_flags &RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not support LZ4 independent blocks\n" ); +return -1 ; + } +break ;case RTE_COMP_ALGO_LZS :case RTE_COMP_ALGO_NULL :break ;
doc/guides/rel_notes/release_23_03.rst +1 -0
@@ -61,6 +61,7 @@ New Features * Added support for ``RTE_COMP_ALGO_LZ4``. * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + * Added support for ``lz4`` in test-compress-perf algo options. * **Allowed test single compress operation in test-compress-perf.**
doc/guides/tools/comp_perf.rst +4 -1
@@ -84,10 +84,13 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations - ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate) + ``--algo [null/deflate/lzs/lz4]`` : perform test on algorithm null(DMA), Deflate, lzs or lz4 (default: Deflate) ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic) + ``--lz4-flags N``: flags to for LZ4 parameters, + see `LZ4 Frame Descriptor <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor>`_ (default: no flags) + ``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9) ``--window-sz N``: base two log value of compression window size (default: max supported by PMD)
Signature · 1 line
2.25.1
v2 cover compressdev: add LZ4 support
In reply to Michael BaumAdd support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already exists in both app doc and arguments but isn't implemented inside the application.
LZ4: https://github.com/lz4/lz4
RFC:
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.
Depends-on: series-26738 ("compressdev: fix end of list enums conflict")
Michael Baum (4):
compressdev: add LZ4 algorithm support app/test-compress-perf: allow test single compress operation app/test-compress-perf: add algo option app/test-compress-perf: add LZ4 support
app/test-compress-perf/comp_perf_options.h | 9 +- .../comp_perf_options_parse.c | 83 +++++++++++- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 82 +++++++----- .../comp_perf_test_throughput.c | 75 +++++++---- .../comp_perf_test_verify.c | 75 +++++++---- app/test-compress-perf/main.c | 63 +++++++-- doc/guides/compressdevs/features/default.ini | 7 + doc/guides/rel_notes/release_23_03.rst | 16 +++ doc/guides/tools/comp_perf.rst | 5 + lib/compressdev/rte_comp.c | 12 ++ lib/compressdev/rte_comp.h | 79 ++++++++++- 12 files changed, 488 insertions(+), 144 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/4 compressdev: add LZ4 algorithm support
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.
--- doc/guides/compressdevs/features/default.ini | 7 ++ doc/guides/rel_notes/release_23_03.rst | 7 ++ lib/compressdev/rte_comp.c | 12 +++ lib/compressdev/rte_comp.h | 79 +++++++++++++++++++- 4 files changed, 103 insertions(+), 2 deletions(-)
doc/guides/compressdevs/features/default.ini +7 -0
@@ -20,8 +20,15 @@ OOP SGL In LB Out = OOP LB In SGL Out = Deflate = LZS = +LZ4 = Adler32 = Crc32 = Adler32&Crc32 = +xxHash32 = Fixed = Dynamic = +LZ4 Dictionary ID = +LZ4 Content Checksum = +LZ4 Content Size = +LZ4 Block Checksum = +LZ4 Block Independence =
doc/guides/rel_notes/release_23_03.rst +7 -0
@@ -78,6 +78,13 @@ New Features ``rte_event_dev_config::nb_single_link_event_port_queues`` parameter required for eth_rx, eth_tx, crypto and timer eventdev adapters. +* **Added LZ4 algorithm in Compressdev Library.** + + Added new compression algorithm, including: + + * Added support for ``RTE_COMP_ALGO_LZ4``. + * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + Removed Items -------------
lib/compressdev/rte_comp.c +12 -0
@@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag)return "HUFFMAN_FIXED" ;case RTE_COMP_FF_HUFFMAN_DYNAMIC :return "HUFFMAN_DYNAMIC" ; +case RTE_COMP_FF_XXHASH32_CHECKSUM : +return "XXHASH32_CHECKSUM" ; +case RTE_COMP_FF_LZ4_DICT_ID : +return "LZ4_DICT_ID" ; +case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM : +return "LZ4_CONTENT_WITH_CHECKSUM" ; +case RTE_COMP_FF_LZ4_CONTENT_SIZE : +return "LZ4_CONTENT_SIZE" ; +case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE : +return "LZ4_BLOCK_INDEPENDENCE" ; +case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM : +return "LZ4_BLOCK_WITH_CHECKSUM" ;default :return NULL ; }
lib/compressdev/rte_comp.h +77 -2
123 lines not on this page · raw message
Signature · 1 line
2.25.1
RE: [EXT] [PATCH v2 1/4] compressdev: add LZ4 algorithm support
Quoted 7 lines
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.Signed-off-by Michael Baum <[email protected]>---
Please review the series, or else it will be merged in RC1.
PATCH v2 2/4 app/test-compress-perf: allow test single compress operation
In reply to Michael BaumPart of the application options is to test only compress and only decompress but actually the application ignores this user option and tries to test always both compress and decompress.
Allow testing only compress and only decompress.
--- app/test-compress-perf/comp_perf_options.h | 6 +- .../comp_perf_options_parse.c | 4 +- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 71 ++++++---- .../comp_perf_test_throughput.c | 64 +++++---- .../comp_perf_test_verify.c | 64 +++++---- app/test-compress-perf/main.c | 8 ++ doc/guides/rel_notes/release_23_03.rst | 4 + 8 files changed, 230 insertions(+), 117 deletions(-)
app/test-compress-perf/comp_perf_options.h +3 -3
@@ -32,9 +32,9 @@ enum cperf_test_type { };enum comp_operation { -COMPRESS_ONLY , -DECOMPRESS_ONLY , -COMPRESS_DECOMPRESS +COMPRESS = (1 <<0 ), +DECOMPRESS = (1 <<1 ), +COMPRESS_DECOMPRESS = (COMPRESS |DECOMPRESS ), };struct range_list {
app/test-compress-perf/comp_perf_options_parse.c +2 -2
@@ -446,11 +446,11 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)struct name_id_map optype_namemap[] = { {"comp" , -COMPRESS_ONLY +COMPRESS }, {"decomp" , -DECOMPRESS_ONLY +DECOMPRESS }, {"comp_and_decomp" ,
app/test-compress-perf/comp_perf_test_common.c +91 -35
222 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_cyclecount.c +44 -27
89 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_throughput.c +38 -26
79 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_verify.c +40 -24
105 lines not on this page · raw message
app/test-compress-perf/main.c +8 -0
@@ -254,6 +254,14 @@ comp_perf_dump_input_data(struct comp_test_data *test_data) goto end; } +if (!(test_data->test_op &COMPRESS ) && + test_data->input_data_sz > + (size_t ) test_data->seg_sz * (size_t ) test_data->max_sgl_segs) { +RTE_LOG (ERR ,USER1 , +"Size of input must be less than total segments\n" ); + goto end; + } + test_data->input_data =rte_zmalloc_socket (NULL , test_data->input_data_sz,0 ,rte_socket_id ());
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -85,6 +85,10 @@ New Features * Added support for ``RTE_COMP_ALGO_LZ4``. * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. +* **Allowed test single compress operation in test-compress-perf.** + + Enable the application options for testing only compress and only decompress. + Removed Items -------------
Signature · 1 line
2.25.1
PATCH v2 3/4 app/test-compress-perf: add algo option
In reply to Michael BaumAdd a command line option algo to select the compress algorithm supported by the compress API: null (DMA), deflate and lzs.
Default for deflate.
--- app/test-compress-perf/comp_perf_options.h | 2 + .../comp_perf_options_parse.c | 40 ++++++++++++++++++- .../comp_perf_test_cyclecount.c | 4 +- .../comp_perf_test_throughput.c | 4 +- .../comp_perf_test_verify.c | 4 +- app/test-compress-perf/main.c | 38 +++++++++++------- doc/guides/rel_notes/release_23_03.rst | 4 ++ doc/guides/tools/comp_perf.rst | 2 + 8 files changed, 76 insertions(+), 22 deletions(-)
app/test-compress-perf/comp_perf_options.h +2 -0
@@ -63,6 +63,8 @@ struct comp_test_data {enum rte_comp_huffman huffman_enc;enum comp_operation test_op; +enum rte_comp_algorithm test_algo; +int window_sz;struct range_list level_lst;uint8_t level;
app/test-compress-perf/comp_perf_options_parse.c +38 -2
@@ -25,6 +25,7 @@#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")#define CPERF_NUM_ITER ("num-iter")#define CPERF_OPTYPE ("operation") +#define CPERF_ALGO ("algo")#define CPERF_HUFFMAN_ENC ("huffman-enc")#define CPERF_LEVEL ("compress-level")#define CPERF_WINDOW_SIZE ("window-sz")
@@ -56,6 +57,8 @@ usage(char *progname)" compressed/decompressed (default: 10000)\n" " --operation [comp/decomp/comp_and_decomp]: perform test on\n" " compression, decompression or both operations\n" +" --algo [null/deflate/lzs]: perform test on algorithm\n" +" null(DMA), deflate or lzs (default: deflate)\n" " --huffman-enc [fixed/dynamic/default]: Huffman encoding\n" " (default: dynamic)\n" " --compress-level N: compression level, which could be a single value, list or range\n"
@@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)return 0 ; } +static int +parse_algo (struct comp_test_data *test_data,const char *arg) +{ +struct name_id_map algo_namemap[] = { + { +"null" , +RTE_COMP_ALGO_NULL + }, + { +"deflate" , +RTE_COMP_ALGO_DEFLATE + }, + { +"lzs" , +RTE_COMP_ALGO_LZS + } + }; + +int id =get_str_key_id_mapping (algo_namemap, +RTE_DIM (algo_namemap), arg); +if (id <0 ) { +RTE_LOG (ERR ,USER1 ,"Invalid algorithm specified\n" ); +return -1 ; + } + + test_data->test_algo = (enum rte_comp_algorithm )id; + +return 0 ; +} +static int parse_huffman_enc (struct comp_test_data *test_data,const char *arg) {
@@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)int id =get_str_key_id_mapping (huffman_namemap,RTE_DIM (huffman_namemap), arg);if (id <0 ) { -RTE_LOG (ERR ,USER1 ,"Invalid Huffmane encoding specified\n" ); +RTE_LOG (ERR ,USER1 ,"Invalid Huffman encoding specified\n" );return -1 ; }
@@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)/* * Try parsing the argument as a range, if it fails, -* arse it as a list +* parse it as a list */ if (parse_range(arg, &test_data->level_lst.min, &test_data->level_lst.max,
@@ -572,6 +605,7 @@ static struct option lgopts[] = { { CPERF_MAX_SGL_SEGS, required_argument, 0, 0}, { CPERF_NUM_ITER, required_argument, 0, 0 }, { CPERF_OPTYPE, required_argument, 0, 0 }, + { CPERF_ALGO, required_argument, 0, 0 }, { CPERF_HUFFMAN_ENC, required_argument, 0, 0 }, { CPERF_LEVEL, required_argument, 0, 0 }, { CPERF_WINDOW_SIZE, required_argument, 0, 0 },
@@ -594,6 +628,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data) { CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs }, { CPERF_NUM_ITER, parse_num_iter }, { CPERF_OPTYPE, parse_op_type }, + { CPERF_ALGO, parse_algo }, { CPERF_HUFFMAN_ENC, parse_huffman_enc }, { CPERF_LEVEL, parse_level }, { CPERF_WINDOW_SIZE, parse_window_sz },
@@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data *test_data) test_data->num_iter =10000 ; test_data->huffman_enc =RTE_COMP_HUFFMAN_DYNAMIC ; test_data->test_op =COMPRESS_DECOMPRESS ; + test_data->test_algo =RTE_COMP_ALGO_DEFLATE ; test_data->window_sz =-1 ; test_data->level_lst.min =RTE_COMP_LEVEL_MIN ; test_data->level_lst.max =RTE_COMP_LEVEL_MAX ;
app/test-compress-perf/comp_perf_test_cyclecount.c +2 -2
@@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/comp_perf_test_throughput.c +2 -2
@@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/comp_perf_test_verify.c +2 -2
@@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/main.c +24 -14
@@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id) {const struct rte_compressdev_capabilities *cap; - cap =rte_compressdev_capability_get (cdev_id, -RTE_COMP_ALGO_DEFLATE ); + cap =rte_compressdev_capability_get (cdev_id, test_data->test_algo);if (cap ==NULL ) {RTE_LOG (ERR ,USER1 , -"Compress device does not support DEFLATE\n" ); +"Compress device does not support %u algorithm\n" , + test_data->test_algo);return -1 ; }uint64_t comp_flags = cap->comp_feature_flags; -/* Huffman encoding */ -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_FIXED && - (comp_flags &RTE_COMP_FF_HUFFMAN_FIXED ) ==0 ) { -RTE_LOG (ERR ,USER1 , -"Compress device does not supported Fixed Huffman\n" ); -return -1 ; - } +/* Algorithm type */ +switch (test_data->test_algo) { +case RTE_COMP_ALGO_DEFLATE : +/* Huffman encoding */ +if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_FIXED && + (comp_flags &RTE_COMP_FF_HUFFMAN_FIXED ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not supported Fixed Huffman\n" ); +return -1 ; + } -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_DYNAMIC && - (comp_flags &RTE_COMP_FF_HUFFMAN_DYNAMIC ) ==0 ) { -RTE_LOG (ERR ,USER1 , -"Compress device does not supported Dynamic Huffman\n" ); +if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_DYNAMIC && + (comp_flags &RTE_COMP_FF_HUFFMAN_DYNAMIC ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not supported Dynamic Huffman\n" ); +return -1 ; + } +break ; +case RTE_COMP_ALGO_LZS : +case RTE_COMP_ALGO_NULL : +break ; +default :return -1 ; }
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -89,6 +89,10 @@ New Features Enable the application options for testing only compress and only decompress. +* **Added algo option in test-compress-perf.** + + Added support for testing other algorithms except for DEFLAT. + Removed Items -------------
doc/guides/tools/comp_perf.rst +2 -0
@@ -84,6 +84,8 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations + ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate) + ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic) ``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
Signature · 1 line
2.25.1
PATCH v2 4/4 app/test-compress-perf: add LZ4 support
In reply to Michael BaumAdd support for LZ4 algorithm and add a new parameter for it.
--- app/test-compress-perf/comp_perf_options.h | 1 + .../comp_perf_options_parse.c | 43 ++++++++++++++++++- .../comp_perf_test_cyclecount.c | 7 ++- .../comp_perf_test_throughput.c | 7 ++- .../comp_perf_test_verify.c | 7 ++- app/test-compress-perf/main.c | 17 ++++++++ doc/guides/rel_notes/release_23_03.rst | 1 + doc/guides/tools/comp_perf.rst | 5 ++- 8 files changed, 82 insertions(+), 6 deletions(-)
app/test-compress-perf/comp_perf_options.h +1 -0
@@ -61,6 +61,7 @@ struct comp_test_data {uint16_t max_sgl_segs;uint32_t total_segs; +uint8_t lz4_flags;enum rte_comp_huffman huffman_enc;enum comp_operation test_op;enum rte_comp_algorithm test_algo;
app/test-compress-perf/comp_perf_options_parse.c +41 -2
93 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_cyclecount.c +6 -1
@@ -194,13 +194,16 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; input_bufs = mem->decomp_bufs; output_bufs = mem->comp_bufs; out_seg_sz = test_data->out_seg_sz;
@@ -214,6 +217,8 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; input_bufs = mem->comp_bufs; output_bufs = mem->decomp_bufs; out_seg_sz = test_data->seg_sz;
app/test-compress-perf/comp_perf_test_throughput.c +6 -1
@@ -85,13 +85,16 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; input_bufs = mem->decomp_bufs; output_bufs = mem->comp_bufs; out_seg_sz = test_data->out_seg_sz;
@@ -105,6 +108,8 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; input_bufs = mem->comp_bufs; output_bufs = mem->decomp_bufs; out_seg_sz = test_data->seg_sz;
app/test-compress-perf/comp_perf_test_verify.c +6 -1
@@ -88,13 +88,16 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; output_data_ptr = ctx->mem.compressed_data; output_data_sz = &ctx->comp_data_sz; input_bufs = mem->decomp_bufs;
@@ -110,6 +113,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; output_data_ptr = ctx->mem.decompressed_data; output_data_sz = &ctx->decomp_data_sz; input_bufs = mem->comp_bufs;
app/test-compress-perf/main.c +17 -0
@@ -86,6 +86,23 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)return -1 ; }break ; +case RTE_COMP_ALGO_LZ4 : +/* LZ4 flags */ +if ((test_data->lz4_flags &RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM ) && + (comp_flags &RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not support LZ4 block with checksum\n" ); +return -1 ; + } + +if ((test_data->lz4_flags & +RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE ) && + (comp_flags &RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not support LZ4 independent blocks\n" ); +return -1 ; + } +break ;case RTE_COMP_ALGO_LZS :case RTE_COMP_ALGO_NULL :break ;
doc/guides/rel_notes/release_23_03.rst +1 -0
@@ -84,6 +84,7 @@ New Features * Added support for ``RTE_COMP_ALGO_LZ4``. * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + * Added support for ``lz4`` in test-compress-perf algo options. * **Allowed test single compress operation in test-compress-perf.**
doc/guides/tools/comp_perf.rst +4 -1
@@ -84,10 +84,13 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations - ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate) + ``--algo [null/deflate/lzs/lz4]`` : perform test on algorithm null(DMA), Deflate, lzs or lz4 (default: Deflate) ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic) + ``--lz4-flags N``: flags to for LZ4 parameters, + see `LZ4 Frame Descriptor <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor>`_ (default: no flags) + ``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9) ``--window-sz N``: base two log value of compression window size (default: max supported by PMD)
Signature · 1 line
2.25.1
v3 cover compressdev: add LZ4 support
In reply to Michael BaumAdd support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already exists in both app doc and arguments but isn't implemented inside the application.
LZ4: https://github.com/lz4/lz4
RFC:
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.
v3:
- Rebase.
Michael Baum (4):
compressdev: add LZ4 algorithm support app/test-compress-perf: allow test single compress operation app/test-compress-perf: add algo option app/test-compress-perf: add LZ4 support
app/test-compress-perf/comp_perf_options.h | 9 +- .../comp_perf_options_parse.c | 83 +++++++++++- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 82 +++++++----- .../comp_perf_test_throughput.c | 75 +++++++---- .../comp_perf_test_verify.c | 75 +++++++---- app/test-compress-perf/main.c | 63 +++++++-- doc/guides/compressdevs/features/default.ini | 7 + doc/guides/rel_notes/release_23_03.rst | 16 +++ doc/guides/tools/comp_perf.rst | 5 + lib/compressdev/rte_comp.c | 12 ++ lib/compressdev/rte_comp.h | 79 ++++++++++- 12 files changed, 488 insertions(+), 144 deletions(-)
Signature · 1 line
2.25.1
PATCH v3 1/4 compressdev: add LZ4 algorithm support
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.
--- doc/guides/compressdevs/features/default.ini | 7 ++ doc/guides/rel_notes/release_23_03.rst | 7 ++ lib/compressdev/rte_comp.c | 12 +++ lib/compressdev/rte_comp.h | 79 +++++++++++++++++++- 4 files changed, 103 insertions(+), 2 deletions(-)
doc/guides/compressdevs/features/default.ini +7 -0
@@ -20,8 +20,15 @@ OOP SGL In LB Out = OOP LB In SGL Out = Deflate = LZS = +LZ4 = Adler32 = Crc32 = Adler32&Crc32 = +xxHash32 = Fixed = Dynamic = +LZ4 Dictionary ID = +LZ4 Content Checksum = +LZ4 Content Size = +LZ4 Block Checksum = +LZ4 Block Independence =
doc/guides/rel_notes/release_23_03.rst +7 -0
@@ -105,6 +105,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added LZ4 algorithm in Compressdev Library.** + + Added new compression algorithm, including: + + * Added support for ``RTE_COMP_ALGO_LZ4``. + * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + Removed Items -------------
lib/compressdev/rte_comp.c +12 -0
@@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag)return "HUFFMAN_FIXED" ;case RTE_COMP_FF_HUFFMAN_DYNAMIC :return "HUFFMAN_DYNAMIC" ; +case RTE_COMP_FF_XXHASH32_CHECKSUM : +return "XXHASH32_CHECKSUM" ; +case RTE_COMP_FF_LZ4_DICT_ID : +return "LZ4_DICT_ID" ; +case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM : +return "LZ4_CONTENT_WITH_CHECKSUM" ; +case RTE_COMP_FF_LZ4_CONTENT_SIZE : +return "LZ4_CONTENT_SIZE" ; +case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE : +return "LZ4_BLOCK_INDEPENDENCE" ; +case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM : +return "LZ4_BLOCK_WITH_CHECKSUM" ;default :return NULL ; }
lib/compressdev/rte_comp.h +77 -2
123 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v3 2/4 app/test-compress-perf: allow test single compress operation
In reply to Michael BaumPart of the application options is to test only compress and only decompress but actually the application ignores this user option and tries to test always both compress and decompress.
Allow testing only compress and only decompress.
--- app/test-compress-perf/comp_perf_options.h | 6 +- .../comp_perf_options_parse.c | 4 +- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 71 ++++++---- .../comp_perf_test_throughput.c | 64 +++++---- .../comp_perf_test_verify.c | 64 +++++---- app/test-compress-perf/main.c | 8 ++ doc/guides/rel_notes/release_23_03.rst | 4 + 8 files changed, 230 insertions(+), 117 deletions(-)
app/test-compress-perf/comp_perf_options.h +3 -3
@@ -32,9 +32,9 @@ enum cperf_test_type { };enum comp_operation { -COMPRESS_ONLY , -DECOMPRESS_ONLY , -COMPRESS_DECOMPRESS +COMPRESS = (1 <<0 ), +DECOMPRESS = (1 <<1 ), +COMPRESS_DECOMPRESS = (COMPRESS |DECOMPRESS ), };struct range_list {
app/test-compress-perf/comp_perf_options_parse.c +2 -2
@@ -446,11 +446,11 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)struct name_id_map optype_namemap[] = { {"comp" , -COMPRESS_ONLY +COMPRESS }, {"decomp" , -DECOMPRESS_ONLY +DECOMPRESS }, {"comp_and_decomp" ,
app/test-compress-perf/comp_perf_test_common.c +91 -35
222 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_cyclecount.c +44 -27
89 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_throughput.c +38 -26
79 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_verify.c +40 -24
105 lines not on this page · raw message
app/test-compress-perf/main.c +8 -0
@@ -254,6 +254,14 @@ comp_perf_dump_input_data(struct comp_test_data *test_data) goto end; } +if (!(test_data->test_op &COMPRESS ) && + test_data->input_data_sz > + (size_t ) test_data->seg_sz * (size_t ) test_data->max_sgl_segs) { +RTE_LOG (ERR ,USER1 , +"Size of input must be less than total segments\n" ); + goto end; + } + test_data->input_data =rte_zmalloc_socket (NULL , test_data->input_data_sz,0 ,rte_socket_id ());
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -112,6 +112,10 @@ New Features * Added support for ``RTE_COMP_ALGO_LZ4``. * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. +* **Allowed test single compress operation in test-compress-perf.** + + Enable the application options for testing only compress and only decompress. + Removed Items -------------
Signature · 1 line
2.25.1
RE: [EXT] [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation
Quoted 4 lines
Part of the application options is to test only compress and only decompress but actually the application ignores this user option and tries to test always both compress and decompress.
From the description it seems to be a bug in previous implementation. Can we back port it? And update the patch title accordingly? Also, release note update is not required for this fix.
Quoted 5 lines
Allow testing only compress and only decompress.
Signed-off-by Matan Azrad <[email protected]>Signed-off-by Michael Baum <[email protected]>
RE: [EXT] [PATCH v3 2/4] app/test-compress-perf: allow test single compress operation
Quoted 3 lines · Akhil Goyal
From the description it seems to be a bug in previous implementation. Can we back port it? And update the patch title accordingly?
I'll update and send a new version.
Quoted 1 line
Also, release note update is not required for this fix.
I'll remove it, thank you.
Quoted 6 lines
Quoted 6 lines
Allow testing only compress and only decompress.
Signed-off-by Matan Azrad <[email protected]>Signed-off-by Michael Baum <[email protected]>
PATCH v3 3/4 app/test-compress-perf: add algo option
In reply to Michael BaumAdd a command line option algo to select the compress algorithm supported by the compress API: null (DMA), deflate and lzs.
Default for deflate.
--- app/test-compress-perf/comp_perf_options.h | 2 + .../comp_perf_options_parse.c | 40 ++++++++++++++++++- .../comp_perf_test_cyclecount.c | 4 +- .../comp_perf_test_throughput.c | 4 +- .../comp_perf_test_verify.c | 4 +- app/test-compress-perf/main.c | 38 +++++++++++------- doc/guides/rel_notes/release_23_03.rst | 4 ++ doc/guides/tools/comp_perf.rst | 2 + 8 files changed, 76 insertions(+), 22 deletions(-)
app/test-compress-perf/comp_perf_options.h +2 -0
@@ -63,6 +63,8 @@ struct comp_test_data {enum rte_comp_huffman huffman_enc;enum comp_operation test_op; +enum rte_comp_algorithm test_algo; +int window_sz;struct range_list level_lst;uint8_t level;
app/test-compress-perf/comp_perf_options_parse.c +38 -2
@@ -25,6 +25,7 @@#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")#define CPERF_NUM_ITER ("num-iter")#define CPERF_OPTYPE ("operation") +#define CPERF_ALGO ("algo")#define CPERF_HUFFMAN_ENC ("huffman-enc")#define CPERF_LEVEL ("compress-level")#define CPERF_WINDOW_SIZE ("window-sz")
@@ -56,6 +57,8 @@ usage(char *progname)" compressed/decompressed (default: 10000)\n" " --operation [comp/decomp/comp_and_decomp]: perform test on\n" " compression, decompression or both operations\n" +" --algo [null/deflate/lzs]: perform test on algorithm\n" +" null(DMA), deflate or lzs (default: deflate)\n" " --huffman-enc [fixed/dynamic/default]: Huffman encoding\n" " (default: dynamic)\n" " --compress-level N: compression level, which could be a single value, list or range\n"
@@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)return 0 ; } +static int +parse_algo (struct comp_test_data *test_data,const char *arg) +{ +struct name_id_map algo_namemap[] = { + { +"null" , +RTE_COMP_ALGO_NULL + }, + { +"deflate" , +RTE_COMP_ALGO_DEFLATE + }, + { +"lzs" , +RTE_COMP_ALGO_LZS + } + }; + +int id =get_str_key_id_mapping (algo_namemap, +RTE_DIM (algo_namemap), arg); +if (id <0 ) { +RTE_LOG (ERR ,USER1 ,"Invalid algorithm specified\n" ); +return -1 ; + } + + test_data->test_algo = (enum rte_comp_algorithm )id; + +return 0 ; +} +static int parse_huffman_enc (struct comp_test_data *test_data,const char *arg) {
@@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)int id =get_str_key_id_mapping (huffman_namemap,RTE_DIM (huffman_namemap), arg);if (id <0 ) { -RTE_LOG (ERR ,USER1 ,"Invalid Huffmane encoding specified\n" ); +RTE_LOG (ERR ,USER1 ,"Invalid Huffman encoding specified\n" );return -1 ; }
@@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)/* * Try parsing the argument as a range, if it fails, -* arse it as a list +* parse it as a list */ if (parse_range(arg, &test_data->level_lst.min, &test_data->level_lst.max,
@@ -572,6 +605,7 @@ static struct option lgopts[] = { { CPERF_MAX_SGL_SEGS, required_argument, 0, 0}, { CPERF_NUM_ITER, required_argument, 0, 0 }, { CPERF_OPTYPE, required_argument, 0, 0 }, + { CPERF_ALGO, required_argument, 0, 0 }, { CPERF_HUFFMAN_ENC, required_argument, 0, 0 }, { CPERF_LEVEL, required_argument, 0, 0 }, { CPERF_WINDOW_SIZE, required_argument, 0, 0 },
@@ -594,6 +628,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data) { CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs }, { CPERF_NUM_ITER, parse_num_iter }, { CPERF_OPTYPE, parse_op_type }, + { CPERF_ALGO, parse_algo }, { CPERF_HUFFMAN_ENC, parse_huffman_enc }, { CPERF_LEVEL, parse_level }, { CPERF_WINDOW_SIZE, parse_window_sz },
@@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data *test_data) test_data->num_iter =10000 ; test_data->huffman_enc =RTE_COMP_HUFFMAN_DYNAMIC ; test_data->test_op =COMPRESS_DECOMPRESS ; + test_data->test_algo =RTE_COMP_ALGO_DEFLATE ; test_data->window_sz =-1 ; test_data->level_lst.min =RTE_COMP_LEVEL_MIN ; test_data->level_lst.max =RTE_COMP_LEVEL_MAX ;
app/test-compress-perf/comp_perf_test_cyclecount.c +2 -2
@@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/comp_perf_test_throughput.c +2 -2
@@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/comp_perf_test_verify.c +2 -2
@@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz,
@@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE
app/test-compress-perf/main.c +24 -14
@@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id) {const struct rte_compressdev_capabilities *cap; - cap =rte_compressdev_capability_get (cdev_id, -RTE_COMP_ALGO_DEFLATE ); + cap =rte_compressdev_capability_get (cdev_id, test_data->test_algo);if (cap ==NULL ) {RTE_LOG (ERR ,USER1 , -"Compress device does not support DEFLATE\n" ); +"Compress device does not support %u algorithm\n" , + test_data->test_algo);return -1 ; }uint64_t comp_flags = cap->comp_feature_flags; -/* Huffman encoding */ -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_FIXED && - (comp_flags &RTE_COMP_FF_HUFFMAN_FIXED ) ==0 ) { -RTE_LOG (ERR ,USER1 , -"Compress device does not supported Fixed Huffman\n" ); -return -1 ; - } +/* Algorithm type */ +switch (test_data->test_algo) { +case RTE_COMP_ALGO_DEFLATE : +/* Huffman encoding */ +if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_FIXED && + (comp_flags &RTE_COMP_FF_HUFFMAN_FIXED ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not supported Fixed Huffman\n" ); +return -1 ; + } -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_DYNAMIC && - (comp_flags &RTE_COMP_FF_HUFFMAN_DYNAMIC ) ==0 ) { -RTE_LOG (ERR ,USER1 , -"Compress device does not supported Dynamic Huffman\n" ); +if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_DYNAMIC && + (comp_flags &RTE_COMP_FF_HUFFMAN_DYNAMIC ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not supported Dynamic Huffman\n" ); +return -1 ; + } +break ; +case RTE_COMP_ALGO_LZS : +case RTE_COMP_ALGO_NULL : +break ; +default :return -1 ; }
doc/guides/rel_notes/release_23_03.rst +4 -0
@@ -116,6 +116,10 @@ New Features Enable the application options for testing only compress and only decompress. +* **Added algo option in test-compress-perf.** + + Added support for testing other algorithms except for DEFLAT. + Removed Items -------------
doc/guides/tools/comp_perf.rst +2 -0
@@ -84,6 +84,8 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations + ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate) + ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic) ``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9)
Signature · 1 line
2.25.1
RE: [EXT] [PATCH v3 3/4] app/test-compress-perf: add algo option
Quoted 51 lines
Add a command line option
algoto select the compress algorithm supported by the compress API: null (DMA), deflate and lzs.Default for deflate.
Signed-off-by Matan Azrad <[email protected]>Signed-off-by Michael Baum <[email protected]>--- app/test-compress-perf/comp_perf_options.h | 2 + .../comp_perf_options_parse.c | 40 ++++++++++++++++++- .../comp_perf_test_cyclecount.c | 4 +- .../comp_perf_test_throughput.c | 4 +- .../comp_perf_test_verify.c | 4 +- app/test-compress-perf/main.c | 38 +++++++++++------- doc/guides/rel_notes/release_23_03.rst | 4 ++ doc/guides/tools/comp_perf.rst | 2 + 8 files changed, 76 insertions(+), 22 deletions(-)app/test-compress- renamed from app/test-compress-perf/comp_perf_options.h +0
-0
perf/comp_perf_options.h
indexd00b299247..5e5227a700100644app/test-compress-perf/comp_perf_options.h +2
-0@@ -63,6 +63,8 @@ struct comp_test_data {enum rte_comp_huffman huffman_enc;enum comp_operation test_op; +enum rte_comp_algorithm test_algo; +int window_sz;struct range_list level_lst;uint8_t level;app/test- renamed from app/test-compress-perf/comp_perf_options_parse.c +0
-0
compress-perf/comp_perf_options_parse.c
index7a992bf43e..97ddff87ef100644app/test-compress-perf/comp_perf_options_parse.c +1
-0@@ -25,6 +25,7 @@#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")#define CPERF_NUM_ITER ("num-iter")#define CPERF_OPTYPE ("operation") +#define CPERF_ALGO ("algo")#define CPERF_HUFFMAN_ENC ("huffman-enc")#define CPERF_LEVEL ("compress-level")#define CPERF_WINDOW_SIZE ("window-sz")@@ -56,6 +57,8 @@ usage(char *progname)" compressed/decompressed (default: 10000)\n" " --operation [comp/decomp/comp_and_decomp]: performtest on\n"
" compression, decompression or both
operations\n"
app/test-compress-perf/comp_perf_options_parse.c usage(char *progname) · line 61
Quoted 2 lines
app/test-compress-perf/comp_perf_options_parse.c +2
-0+" --algo [null/deflate/lzs]: perform test on algorithm\n" +" null(DMA), deflate or lzs (default: deflate)\n"
Since LZ4 is also supported now, is it not good to add that also in this patch?
Quoted 43 lines
app/test-compress-perf/comp_perf_options_parse.c +0
-0" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n" " (default: dynamic)\n" " --compress-level N: compression level, which could be a singlevalue, list or range\n"
app/test-compress-perf/comp_perf_options_parse.c +30
-0@@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data, const char *arg) return 0; } +static int +parse_algo (struct comp_test_data *test_data,const char *arg) +{ +struct name_id_map algo_namemap[] = { + { +"null" , +RTE_COMP_ALGO_NULL + }, + { +"deflate" , +RTE_COMP_ALGO_DEFLATE + }, + { +"lzs" , +RTE_COMP_ALGO_LZS + } + }; + +int id =get_str_key_id_mapping (algo_namemap, +RTE_DIM (algo_namemap), arg); +if (id <0 ) { +RTE_LOG (ERR ,USER1 ,"Invalid algorithm specified\n" ); +return -1 ; + } + + test_data->test_algo = (enum rte_comp_algorithm )id; + +return 0 ; +} +static int parse_huffman_enc (struct comp_test_data *test_data,const char *arg){
app/test-compress-perf/comp_perf_options_parse.c parse_huffman_enc(struct comp_test_data *test_data, · line 529
Quoted 7 lines
app/test-compress-perf/comp_perf_options_parse.c +1
-1@@ -491,7 +524,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg) int id = get_str_key_id_mapping(huffman_namemap,RTE_DIM (huffman_namemap), arg);if (id <0 ) { -RTE_LOG (ERR ,USER1 ,"Invalid Huffmane encoding specified\n"); +RTE_LOG (ERR ,USER1 ,"Invalid Huffman encoding specified\n" );
This change is not related to this patch. Should be a separate patch and should be backported.
app/test-compress-perf/comp_perf_options_parse.c parse_level(struct comp_test_data *test_data, const · line 544
Quoted 10 lines
app/test-compress-perf/comp_perf_options_parse.c +0
-0return -1 ;app/test-compress-perf/comp_perf_options_parse.c +1
-1}@@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)/* * Try parsing the argument as a range, if it fails, -* arse it as a list +* parse it as a list
Same here.
Quoted 177 lines
app/test-compress-perf/comp_perf_options_parse.c +0
-0*/ if (parse_range(arg, &test_data->level_lst.min,app/test-compress-perf/comp_perf_options_parse.c +2
-0&test_data->level_lst.max,@@ -572,6 +605,7 @@ static struct option lgopts[] = { { CPERF_MAX_SGL_SEGS, required_argument, 0, 0}, { CPERF_NUM_ITER, required_argument, 0, 0 }, { CPERF_OPTYPE, required_argument, 0, 0 }, + { CPERF_ALGO, required_argument, 0, 0 }, { CPERF_HUFFMAN_ENC, required_argument, 0, 0 }, { CPERF_LEVEL, required_argument, 0, 0 }, { CPERF_WINDOW_SIZE, required_argument, 0, 0 },@@ -594,6 +628,7 @@ comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data) { CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs }, { CPERF_NUM_ITER, parse_num_iter }, { CPERF_OPTYPE, parse_op_type }, + { CPERF_ALGO, parse_algo }, { CPERF_HUFFMAN_ENC, parse_huffman_enc }, { CPERF_LEVEL, parse_level },app/test-compress-perf/comp_perf_options_parse.c +1
-0{ CPERF_WINDOW_SIZE, parse_window_sz },@@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data *test_data) test_data->num_iter =10000 ; test_data->huffman_enc =RTE_COMP_HUFFMAN_DYNAMIC ; test_data->test_op =COMPRESS_DECOMPRESS ; + test_data->test_algo =RTE_COMP_ALGO_DEFLATE ; test_data->window_sz =-1 ; test_data->level_lst.min =RTE_COMP_LEVEL_MIN ;app/test-compress-perf/comp_perf_options_parse.c +0
-0test_data->level_lst.max =RTE_COMP_LEVEL_MAX ;app/test- renamed from app/test-compress-perf/comp_perf_test_cyclecount.c +0
-0
compress-perf/comp_perf_test_cyclecount.c
indexce6c4d7605..81c3d30038100644app/test-compress-perf/comp_perf_test_cyclecount.c +1
-1@@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level,app/test-compress-perf/comp_perf_test_cyclecount.c +1
-1.window_size = test_data->window_sz,@@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz,app/test-compress-perf/comp_perf_test_cyclecount.c +0
-0.hash_algo = RTE_COMP_HASH_ALGO_NONEapp/test- renamed from app/test-compress-perf/comp_perf_test_throughput.c +0
-0
compress-perf/comp_perf_test_throughput.c
indexc9f8237626..2545ee9925100644app/test-compress-perf/comp_perf_test_throughput.c +1
-1@@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level,app/test-compress-perf/comp_perf_test_throughput.c +1
-1.window_size = test_data->window_sz,@@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz,app/test-compress-perf/comp_perf_test_throughput.c +0
-0.hash_algo = RTE_COMP_HASH_ALGO_NONEapp/test- renamed from app/test-compress-perf/comp_perf_test_verify.c +0
-0
compress-perf/comp_perf_test_verify.c
index7d6b6abecd..88f4f41851100644app/test-compress-perf/comp_perf_test_verify.c +1
-1@@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level,app/test-compress-perf/comp_perf_test_verify.c +1
-1.window_size = test_data->window_sz,@@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz,app/test-compress-perf/comp_perf_test_verify.c +0
-0.hash_algo = RTE_COMP_HASH_ALGO_NONEapp/test-compress-perf/main.c +8
-10@@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id) {const struct rte_compressdev_capabilities *cap; - cap =rte_compressdev_capability_get (cdev_id, -RTE_COMP_ALGO_DEFLATE ); + cap =rte_compressdev_capability_get (cdev_id, test_data->test_algo);if (cap ==NULL ) {RTE_LOG (ERR ,USER1 , -"Compress device does not support DEFLATE\n" ); +"Compress device does not support %u algorithm\n" , + test_data->test_algo);return -1 ; }uint64_t comp_flags = cap->comp_feature_flags; -/* Huffman encoding */ -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_FIXED && - (comp_flags &RTE_COMP_FF_HUFFMAN_FIXED ) ==0 ) { -RTE_LOG (ERR ,USER1 , -"Compress device does not supported Fixed Huffman\n"); -return -1 ; - } +/* Algorithm type */ +switch (test_data->test_algo) { +case RTE_COMP_ALGO_DEFLATE : +/* Huffman encoding */ + if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED&&
app/test-compress-perf/main.c +3
-0+ (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) { + RTE_LOG(ERR, USER1, + "Compress device does not supported FixedHuffman\n");
app/test-compress-perf/main.c +2
-2+return -1 ; + } -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_DYNAMIC && - (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC)== 0) {
app/test-compress-perf/main.c +0
-2- RTE_LOG(ERR, USER1, - "Compress device does not supported DynamicHuffman\n");
app/test-compress-perf/main.c +1
-0+ if (test_data->huffman_enc ==
RTE_COMP_HUFFMAN_DYNAMIC&&app/test-compress-perf/main.c +3
-0+ (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) { + RTE_LOG(ERR, USER1, + "Compress device does not supported DynamicHuffman\n");
app/test-compress-perf/main.c +7
-0+return -1 ; + } +break ; +case RTE_COMP_ALGO_LZS : +case RTE_COMP_ALGO_NULL : +break ; +default :return -1 ; }doc/guides/rel_notes/release_23_03.rst +0
-0
doc/guides/rel_notes/release_23_03.rst New Features · line 122
Quoted 7 lines
doc/guides/rel_notes/release_23_03.rst +4
-0@@ -116,6 +116,10 @@ New Features Enable the application options for testing only compress and only decompress. +* **Added algo option in test-compress-perf.** + + Added support for testing other algorithms except for DEFLAT. +
This patch is only adding a new option to the test app. I believe release notes are not needed.
Quoted 22 lines
doc/guides/rel_notes/release_23_03.rst +0
-0Removed Items -------------doc/guides/tools/comp_perf.rst +0
-0@@ -84,6 +84,8 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test oncompression, decompression or both operations
+ ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate)
+
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default:
dynamic)``--compress-level N``: compression level, which could be a single value, list or
range (default: range between 1 and 9)
--
2.25.1
RE: [EXT] [PATCH v3 3/4] app/test-compress-perf: add algo option
Quoted 48 lines · Akhil Goyal
Quoted 48 lines
Add a command line option
algoto select the compress algorithm supported by the compress API: null (DMA), deflate and lzs.Default for deflate.
Signed-off-by Matan Azrad <[email protected]>Signed-off-by Michael Baum <[email protected]>--- app/test-compress-perf/comp_perf_options.h | 2 + .../comp_perf_options_parse.c | 40 ++++++++++++++++++- .../comp_perf_test_cyclecount.c | 4 +- .../comp_perf_test_throughput.c | 4 +- .../comp_perf_test_verify.c | 4 +- app/test-compress-perf/main.c | 38 +++++++++++------- doc/guides/rel_notes/release_23_03.rst | 4 ++ doc/guides/tools/comp_perf.rst | 2 + 8 files changed, 76 insertions(+), 22 deletions(-)(quoted diff) +0
-0b/app/test-compress-
perf/comp_perf_options.hindexd00b299247..5e5227a700100644app/test-compress-perf/comp_perf_options.h +2
-0@@ -63,6 +63,8 @@ struct comp_test_data {enum rte_comp_huffman huffman_enc;enum comp_operation test_op; +enum rte_comp_algorithm test_algo; +int window_sz;struct range_list level_lst;uint8_t level;(quoted diff) +0
-0b/app/test-
compress-perf/comp_perf_options_parse.cindex7a992bf43e..97ddff87ef100644app/test-compress-perf/comp_perf_options_parse.c +1
-0@@ -25,6 +25,7 @@#define CPERF_MAX_SGL_SEGS ("max-num-sgl-segs")#define CPERF_NUM_ITER ("num-iter")#define CPERF_OPTYPE ("operation") +#define CPERF_ALGO ("algo")#define CPERF_HUFFMAN_ENC ("huffman-enc")#define CPERF_LEVEL ("compress-level")#define CPERF_WINDOW_SIZE ("window-sz")@@ -56,6 +57,8 @@ usage(char *progname)" compressed/decompressed (default: 10000)\n"
Quoted 8 lines
app/test-compress-perf/comp_perf_options_parse.c usage(char *progname) · line 61
Quoted 6 lines
app/test-compress-perf/comp_perf_options_parse.c +0
-0" --operation [comp/decomp/comp_and_decomp]: performtest on\n"
" compression, decompression or both
operations\n"app/test-compress-perf/comp_perf_options_parse.c +2
-0+" --algo [null/deflate/lzs]: perform test on algorithm\n" +" null(DMA), deflate or lzs (default: deflate)\n" Since LZ4 is also supported now, is it not good to add that also in this patch?
I agree, I'll merge them in the next version.
Quoted 45 lines
Quoted 45 lines
" --huffman-enc [fixed/dynamic/default]: Huffman encoding\n"
" (default: dynamic)\n"
" --compress-level N: compression level, which could be
a single value, list or range\n"app/test-compress-perf/comp_perf_options_parse.c +29
-0@@ -470,6 +473,36 @@ parse_op_type(struct comp_test_data *test_data, const char *arg) return 0; } +static int +parse_algo (struct comp_test_data *test_data,const char *arg) { +struct name_id_map algo_namemap[] = { + { +"null" , +RTE_COMP_ALGO_NULL + }, + { +"deflate" , +RTE_COMP_ALGO_DEFLATE + }, + { +"lzs" , +RTE_COMP_ALGO_LZS + } + }; + +int id =get_str_key_id_mapping (algo_namemap, +RTE_DIM (algo_namemap), arg); +if (id <0 ) { +RTE_LOG (ERR ,USER1 ,"Invalid algorithm specified\n" ); +return -1 ; + } + + test_data->test_algo = (enum rte_comp_algorithm )id; + +return 0 ; +} +static int parse_huffman_enc (struct comp_test_data *test_data,const char *arg){ @@ -491,7 +524,7 @@
parse_huffman_enc(struct comp_test_data *test_data, const char *arg)
int id = get_str_key_id_mapping(huffman_namemap,
RTE_DIM(huffman_namemap), arg);
Quoted 8 lines
Quoted 5 lines
if (id < 0) {- RTE_LOG(ERR, USER1, "Invalid Huffmane encoding
specified\n");
+ RTE_LOG(ERR, USER1, "Invalid Huffman encoding
+ specified\n");This change is not related to this patch. Should be a separate patch and should be backported.
I'll separate it and the next to different commit.
Quoted 197 lines
app/test-compress-perf/comp_perf_options_parse.c parse_level(struct comp_test_data *test_data, · line 544
Quoted 11 lines
return -1; }app/test-compress-perf/comp_perf_options_parse.c +1
-1@@ -507,7 +540,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)/* * Try parsing the argument as a range, if it fails, -* arse it as a list +* parse it as a list Same here.
Quoted 179 lines
app/test-compress-perf/comp_perf_options_parse.c +0
-0*/ if (parse_range(arg, &test_data->level_lst.min,&test_data->level_lst.max, @@ -572,6 +605,7 @@static struct option lgopts[] = {
{CPERF_MAX_SGL_SEGS,required_argument, 0, 0},
{CPERF_NUM_ITER,required_argument, 0, 0 },
{CPERF_OPTYPE,required_argument, 0, 0 },
+ {CPERF_ALGO,required_argument, 0, 0 },
{CPERF_HUFFMAN_ENC,required_argument, 0, 0 },
{CPERF_LEVEL,required_argument, 0, 0 },
{CPERF_WINDOW_SIZE,required_argument, 0, 0 }, @@ -594,6 +628,7
@@comp_perf_opts_parse_long(int opt_idx, struct comp_test_data *test_data){ CPERF_MAX_SGL_SEGS, parse_max_num_sgl_segs }, { CPERF_NUM_ITER, parse_num_iter }, { CPERF_OPTYPE, parse_op_type },app/test-compress-perf/comp_perf_options_parse.c +2
-0+ { CPERF_ALGO, parse_algo }, { CPERF_HUFFMAN_ENC, parse_huffman_enc }, { CPERF_LEVEL, parse_level }, { CPERF_WINDOW_SIZE, parse_window_sz },@@ -649,6 +684,7 @@ comp_perf_options_default(struct comp_test_data *test_data) test_data->num_iter =10000 ; test_data->huffman_enc =RTE_COMP_HUFFMAN_DYNAMIC ; test_data->test_op =COMPRESS_DECOMPRESS ; + test_data->test_algo =RTE_COMP_ALGO_DEFLATE ; test_data->window_sz =-1 ; test_data->level_lst.min =RTE_COMP_LEVEL_MIN ;test_data->level_lst.max = RTE_COMP_LEVEL_MAX; diff --git
a/app/test-compress-perf/comp_perf_test_cyclecount.cb/app/test-compress-perf/comp_perf_test_cyclecount.c
indexce6c4d7605..81c3d30038100644app/test-compress-perf/comp_perf_test_cyclecount.c +1
-1@@ -193,7 +193,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level,.window_size = test_data->window_sz, @@ -208,7 +208,7 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform) { .type = RTE_COMP_DECOMPRESS, .decompress = {app/test-compress-perf/comp_perf_test_cyclecount.c +1
-1- .algo = RTE_COMP_ALGO_DEFLATE, + .algo = test_data->test_algo, .chksum = RTE_COMP_CHECKSUM_NONE, .window_size = test_data->window_sz, .hash_algo = RTE_COMP_HASH_ALGO_NONE(quoted diff) +0
-0b/app/test-
compress-perf/comp_perf_test_throughput.cindexc9f8237626..2545ee9925100644app/test-compress-perf/comp_perf_test_throughput.c +1
-1@@ -84,7 +84,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level,.window_size = test_data->window_sz, @@ -99,7 +99,7 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform) { .type = RTE_COMP_DECOMPRESS, .decompress = {app/test-compress-perf/comp_perf_test_throughput.c +1
-1- .algo = RTE_COMP_ALGO_DEFLATE, + .algo = test_data->test_algo, .chksum = RTE_COMP_CHECKSUM_NONE, .window_size = test_data->window_sz, .hash_algo = RTE_COMP_HASH_ALGO_NONE(quoted diff) +0
-0b/app/test-
compress-perf/comp_perf_test_verify.c
index7d6b6abecd..88f4f41851100644app/test-compress-perf/comp_perf_test_verify.c +1
-1@@ -87,7 +87,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .deflate.huffman = test_data->huffman_enc, .level = test_data->level,.window_size = test_data->window_sz, @@ -104,7 +104,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform) { .type = RTE_COMP_DECOMPRESS, .decompress = {app/test-compress-perf/comp_perf_test_verify.c +1
-1- .algo = RTE_COMP_ALGO_DEFLATE, + .algo = test_data->test_algo, .chksum = RTE_COMP_CHECKSUM_NONE, .window_size = test_data->window_sz, .hash_algo = RTE_COMP_HASH_ALGO_NONE(quoted diff) +0
-0
b/app/test-compress-perf/main.cindexbbb4c7917b..d049527ba1100644app/test-compress-perf/main.c +9
-10@@ -57,29 +57,39 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id) {const struct rte_compressdev_capabilities *cap; - cap =rte_compressdev_capability_get (cdev_id, -RTE_COMP_ALGO_DEFLATE ); + cap =rte_compressdev_capability_get (cdev_id, + test_data->test_algo);if (cap ==NULL ) {RTE_LOG (ERR ,USER1 , -"Compress device does not support DEFLATE\n" ); +"Compress device does not support %u algorithm\n" , + test_data->test_algo);return -1 ; }uint64_t comp_flags = cap->comp_feature_flags; -/* Huffman encoding */ -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_FIXED && - (comp_flags &RTE_COMP_FF_HUFFMAN_FIXED ) ==0 ) { -RTE_LOG (ERR ,USER1 , -"Compress device does not supported Fixed Huffman\n"); -return -1 ; - } +/* Algorithm type */ +switch (test_data->test_algo) { +case RTE_COMP_ALGO_DEFLATE : +/* Huffman encoding */ + if (test_data->huffman_enc == RTE_COMP_HUFFMAN_FIXED&&
app/test-compress-perf/main.c +4
-0+ (comp_flags & RTE_COMP_FF_HUFFMAN_FIXED) == 0) { + RTE_LOG(ERR, USER1, + "Compress device does not supported + FixedHuffman\n");
app/test-compress-perf/main.c +2
-2+return -1 ; + } -if (test_data->huffman_enc ==RTE_COMP_HUFFMAN_DYNAMIC && - (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC)== 0) {
app/test-compress-perf/main.c +0
-2- RTE_LOG(ERR, USER1, - "Compress device does not supported DynamicHuffman\n");
+ if (test_data->huffman_enc==RTE_COMP_HUFFMAN_DYNAMIC&&app/test-compress-perf/main.c +4
-0+ (comp_flags & RTE_COMP_FF_HUFFMAN_DYNAMIC) == 0) { + RTE_LOG(ERR, USER1, + "Compress device does not supported + DynamicHuffman\n");
app/test-compress-perf/main.c +7
-0+return -1 ; + } +break ; +case RTE_COMP_ALGO_LZS : +case RTE_COMP_ALGO_NULL : +break ; +default :return -1 ; }doc/guides/rel_notes/release_23_03.rst +0
-0doc/guides/rel_notes/release_23_03.rst New Features · line 117
Quoted 2 lines
doc/guides/rel_notes/release_23_03.rst +0
-0@@ -116,6 +116,10 @@ New Features Enable the application options for testing only compress and onlydecompress.
Quoted 1 line
Quoted 7 lines
doc/guides/rel_notes/release_23_03.rst New Features · line 122
Quoted 4 lines
doc/guides/rel_notes/release_23_03.rst +4
-0+* **Added algo option in test-compress-perf.** + + Added support for testing other algorithms except for DEFLAT. +This patch is only adding a new option to the test app. I believe release notes are not needed.
Thank you, I'm removing it.
Quoted 25 lines
Quoted 25 lines
Removed Items
-------------(quoted diff) +0
-0
b/doc/guides/tools/comp_perf.rstindex9d2f4dbe4a..cf9035cc23100644doc/guides/tools/comp_perf.rst +0
-0@@ -84,6 +84,8 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test oncompression, decompression or both operations
+ ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA),
+ Deflate or lzs
(default: Deflate)
+
``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default:
dynamic)``--compress-level N``: compression level, which could be a single
value, list or range (default: range between 1 and 9)
--
2.25.1
PATCH v3 4/4 app/test-compress-perf: add LZ4 support
In reply to Michael BaumAdd support for LZ4 algorithm and add a new parameter for it.
--- app/test-compress-perf/comp_perf_options.h | 1 + .../comp_perf_options_parse.c | 43 ++++++++++++++++++- .../comp_perf_test_cyclecount.c | 7 ++- .../comp_perf_test_throughput.c | 7 ++- .../comp_perf_test_verify.c | 7 ++- app/test-compress-perf/main.c | 17 ++++++++ doc/guides/rel_notes/release_23_03.rst | 1 + doc/guides/tools/comp_perf.rst | 5 ++- 8 files changed, 82 insertions(+), 6 deletions(-)
app/test-compress-perf/comp_perf_options.h +1 -0
@@ -61,6 +61,7 @@ struct comp_test_data {uint16_t max_sgl_segs;uint32_t total_segs; +uint8_t lz4_flags;enum rte_comp_huffman huffman_enc;enum comp_operation test_op;enum rte_comp_algorithm test_algo;
app/test-compress-perf/comp_perf_options_parse.c +41 -2
93 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_cyclecount.c +6 -1
@@ -194,13 +194,16 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; input_bufs = mem->decomp_bufs; output_bufs = mem->comp_bufs; out_seg_sz = test_data->out_seg_sz;
@@ -214,6 +217,8 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; input_bufs = mem->comp_bufs; output_bufs = mem->decomp_bufs; out_seg_sz = test_data->seg_sz;
app/test-compress-perf/comp_perf_test_throughput.c +6 -1
@@ -85,13 +85,16 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; input_bufs = mem->decomp_bufs; output_bufs = mem->comp_bufs; out_seg_sz = test_data->out_seg_sz;
@@ -105,6 +108,8 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; input_bufs = mem->comp_bufs; output_bufs = mem->decomp_bufs; out_seg_sz = test_data->seg_sz;
app/test-compress-perf/comp_perf_test_verify.c +6 -1
@@ -88,13 +88,16 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) .type = RTE_COMP_COMPRESS, .compress = { .algo = test_data->test_algo, - .deflate.huffman = test_data->huffman_enc, .level = test_data->level, .window_size = test_data->window_sz, .chksum = RTE_COMP_CHECKSUM_NONE, .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; output_data_ptr = ctx->mem.compressed_data; output_data_sz = &ctx->comp_data_sz; input_bufs = mem->decomp_bufs;
@@ -110,6 +113,8 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) .hash_algo = RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; output_data_ptr = ctx->mem.decompressed_data; output_data_sz = &ctx->decomp_data_sz; input_bufs = mem->comp_bufs;
app/test-compress-perf/main.c +17 -0
@@ -86,6 +86,23 @@ comp_perf_check_capabilities(struct comp_test_data *test_data, uint8_t cdev_id)return -1 ; }break ; +case RTE_COMP_ALGO_LZ4 : +/* LZ4 flags */ +if ((test_data->lz4_flags &RTE_COMP_LZ4_FLAG_BLOCK_CHECKSUM ) && + (comp_flags &RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not support LZ4 block with checksum\n" ); +return -1 ; + } + +if ((test_data->lz4_flags & +RTE_COMP_LZ4_FLAG_BLOCK_INDEPENDENCE ) && + (comp_flags &RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE ) ==0 ) { +RTE_LOG (ERR ,USER1 , +"Compress device does not support LZ4 independent blocks\n" ); +return -1 ; + } +break ;case RTE_COMP_ALGO_LZS :case RTE_COMP_ALGO_NULL :break ;
doc/guides/rel_notes/release_23_03.rst +1 -0
@@ -111,6 +111,7 @@ New Features * Added support for ``RTE_COMP_ALGO_LZ4``. * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + * Added support for ``lz4`` in test-compress-perf algo options. * **Allowed test single compress operation in test-compress-perf.**
doc/guides/tools/comp_perf.rst +4 -1
@@ -84,10 +84,13 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations - ``--algo [null/deflate/lzs]`` : perform test on algorithm null(DMA), Deflate or lzs (default: Deflate) + ``--algo [null/deflate/lzs/lz4]`` : perform test on algorithm null(DMA), Deflate, lzs or lz4 (default: Deflate) ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic) + ``--lz4-flags N``: flags to for LZ4 parameters, + see `LZ4 Frame Descriptor <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor>`_ (default: no flags) + ``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9) ``--window-sz N``: base two log value of compression window size (default: max supported by PMD)
Signature · 1 line
2.25.1
Hi,
The patchset only have library change and added performance tests.
We shall have at least one PMD to support the added algorithms and unit test updated for the PMD/algorithms.
Regards,
Fan
Quoted 37 lines · Michael Baum
Add support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already exists in both app doc and arguments but isn't implemented inside the application.
LZ4: https://github.com/lz4/lz4
RFC:
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.v3:
- Rebase.Michael Baum (4):
compressdev: add LZ4 algorithm support app/test-compress-perf: allow test single compress operation app/test-compress-perf: add algo option app/test-compress-perf: add LZ4 supportapp/test-compress-perf/comp_perf_options.h | 9 +- .../comp_perf_options_parse.c | 83 +++++++++++- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 82 +++++++----- .../comp_perf_test_throughput.c | 75 +++++++---- .../comp_perf_test_verify.c | 75 +++++++---- app/test-compress-perf/main.c | 63 +++++++-- doc/guides/compressdevs/features/default.ini | 7 + doc/guides/rel_notes/release_23_03.rst | 16 +++ doc/guides/tools/comp_perf.rst | 5 + lib/compressdev/rte_comp.c | 12 ++ lib/compressdev/rte_comp.h | 79 ++++++++++- 12 files changed, 488 insertions(+), 144 deletions(-)
Quoted 6 lines · Fan
Hi,
The patchset only have library change and added performance tests.
We shall have at least one PMD to support the added algorithms and unit test updated for the PMD/algorithms.
There is another patchset which adds it in MLX5 compress PMD:
series-26766 ("compress/mlx5: add LZ4 support") https://patchwork.dpdk.org/project/dpdk/cover/[email protected]/
Quoted 44 lines
Regards,
Fan
Quoted 38 lines · Michael Baum
Add support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already exists in both app doc and arguments but isn't implemented inside the application.
LZ4: https://github.com/lz4/lz4
RFC:
https://patchwork.dpdk.org/project/dpdk/patch/20220410182622.8828-1-rz
[email protected]/v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.v3:
- Rebase.Michael Baum (4):
compressdev: add LZ4 algorithm support app/test-compress-perf: allow test single compress operation app/test-compress-perf: add algo option app/test-compress-perf: add LZ4 supportapp/test-compress-perf/comp_perf_options.h | 9 +- .../comp_perf_options_parse.c | 83 +++++++++++- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 82 +++++++----- .../comp_perf_test_throughput.c | 75 +++++++---- .../comp_perf_test_verify.c | 75 +++++++---- app/test-compress-perf/main.c | 63 +++++++-- doc/guides/compressdevs/features/default.ini | 7 + doc/guides/rel_notes/release_23_03.rst | 16 +++ doc/guides/tools/comp_perf.rst | 5 + lib/compressdev/rte_comp.c | 12 ++ lib/compressdev/rte_comp.h | 79 ++++++++++- 12 files changed, 488 insertions(+), 144 deletions(-)
v4 cover compressdev: add LZ4 support
In reply to Michael BaumAdd support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already exists in both app doc and arguments but isn't implemented inside the application.
LZ4: https://github.com/lz4/lz4
RFC:
https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/
v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.
v3:
- Rebase.
v4:
- Change "allow test single" commit to fix and add "Cc: [email protected]".
- Take all typo fixes into separate commit.
- Squash "add LZ4 algo" commit into "add algo option" commit.
- Remove unnecessary release notes updates.
Michael Baum (4):
compressdev: add LZ4 algorithm support app/test-compress-perf: fix some typos app/test-compress-perf: fix missing test single compress op app/test-compress-perf: add algo option
app/test-compress-perf/comp_perf_options.h | 9 +- .../comp_perf_options_parse.c | 83 +++++++++++- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 82 +++++++----- .../comp_perf_test_throughput.c | 75 +++++++---- .../comp_perf_test_verify.c | 75 +++++++---- app/test-compress-perf/main.c | 63 +++++++-- doc/guides/compressdevs/features/default.ini | 7 + doc/guides/rel_notes/release_23_03.rst | 7 + doc/guides/tools/comp_perf.rst | 5 + lib/compressdev/rte_comp.c | 12 ++ lib/compressdev/rte_comp.h | 79 ++++++++++- 12 files changed, 479 insertions(+), 144 deletions(-)
Signature · 1 line
2.25.1
PATCH v4 1/4 compressdev: add LZ4 algorithm support
Add support for LZ4 algorithm:
- Add Lz4 param structure to XFORM structures.
- Add capabilities flags for LZ4 params.
- Add xxHash-32 checksum and capabilities flag.
--- doc/guides/compressdevs/features/default.ini | 7 ++ doc/guides/rel_notes/release_23_03.rst | 7 ++ lib/compressdev/rte_comp.c | 12 +++ lib/compressdev/rte_comp.h | 79 +++++++++++++++++++- 4 files changed, 103 insertions(+), 2 deletions(-)
doc/guides/compressdevs/features/default.ini +7 -0
@@ -20,8 +20,15 @@ OOP SGL In LB Out = OOP LB In SGL Out = Deflate = LZS = +LZ4 = Adler32 = Crc32 = Adler32&Crc32 = +xxHash32 = Fixed = Dynamic = +LZ4 Dictionary ID = +LZ4 Content Checksum = +LZ4 Content Size = +LZ4 Block Checksum = +LZ4 Block Independence =
doc/guides/rel_notes/release_23_03.rst +7 -0
@@ -105,6 +105,13 @@ New Features * Added support to capture packets at each graph node with packet metadata and node name. +* **Added LZ4 algorithm in Compressdev Library.** + + Added new compression algorithm, including: + + * Added support for ``RTE_COMP_ALGO_LZ4``. + * Added support for ``RTE_COMP_CHECKSUM_XXHASH32``. + Removed Items -------------
lib/compressdev/rte_comp.c +12 -0
@@ -39,6 +39,18 @@ rte_comp_get_feature_name(uint64_t flag)return "HUFFMAN_FIXED" ;case RTE_COMP_FF_HUFFMAN_DYNAMIC :return "HUFFMAN_DYNAMIC" ; +case RTE_COMP_FF_XXHASH32_CHECKSUM : +return "XXHASH32_CHECKSUM" ; +case RTE_COMP_FF_LZ4_DICT_ID : +return "LZ4_DICT_ID" ; +case RTE_COMP_FF_LZ4_CONTENT_WITH_CHECKSUM : +return "LZ4_CONTENT_WITH_CHECKSUM" ; +case RTE_COMP_FF_LZ4_CONTENT_SIZE : +return "LZ4_CONTENT_SIZE" ; +case RTE_COMP_FF_LZ4_BLOCK_INDEPENDENCE : +return "LZ4_BLOCK_INDEPENDENCE" ; +case RTE_COMP_FF_LZ4_BLOCK_WITH_CHECKSUM : +return "LZ4_BLOCK_WITH_CHECKSUM" ;default :return NULL ; }
lib/compressdev/rte_comp.h +77 -2
123 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 2/4 app/test-compress-perf: fix some typos
In reply to Michael BaumFix some typos in comments and prints in this app.
--- app/test-compress-perf/comp_perf_options_parse.c | 4 ++-- app/test-compress-perf/comp_perf_test_cyclecount.c | 2 +- app/test-compress-perf/comp_perf_test_verify.c | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
app/test-compress-perf/comp_perf_options_parse.c +2 -2
@@ -491,7 +491,7 @@ parse_huffman_enc(struct comp_test_data *test_data, const char *arg)int id =get_str_key_id_mapping (huffman_namemap,RTE_DIM (huffman_namemap), arg);if (id <0 ) { -RTE_LOG (ERR ,USER1 ,"Invalid Huffmane encoding specified\n" ); +RTE_LOG (ERR ,USER1 ,"Invalid Huffman encoding specified\n" );return -1 ; }
@@ -507,7 +507,7 @@ parse_level(struct comp_test_data *test_data, const char *arg)/* * Try parsing the argument as a range, if it fails, -* arse it as a list +* parse it as a list */ if (parse_range(arg, &test_data->level_lst.min, &test_data->level_lst.max,
app/test-compress-perf/comp_perf_test_cyclecount.c +1 -1
@@ -563,7 +563,7 @@ cperf_cyclecount_test_runner(void *test_ctx)" [D-e] - decompression enqueue\n" " [D-d] - decompression dequeue\n" " - Cycles section: number of cycles per 'op' for the following operations:\n" -" setup/op - memory allocation, op configuration and memory dealocation\n" +" setup/op - memory allocation, op configuration and memory deallocation\n" " [C-e] - compression enqueue\n" " [C-d] - compression dequeue\n" " [D-e] - decompression enqueue\n"
app/test-compress-perf/comp_perf_test_verify.c +2 -2
@@ -226,7 +226,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) op->status == RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {RTE_LOG (ERR ,USER1 , -"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n" ); +"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n" ); res =-1 ; goto end; } else if (op->status !=
@@ -311,7 +311,7 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) op->status == RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE) {RTE_LOG (ERR ,USER1 , -"Out of space error occurred due to uncompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n" ); +"Out of space error occurred due to incompressible input data expanding to larger than destination buffer. Increase the EXPANSE_RATIO constant to use this data.\n" ); res =-1 ; goto end; } else if (op->status !=
Signature · 1 line
2.25.1
PATCH v4 3/4 app/test-compress-perf: fix missing test single compress op
In reply to Michael BaumPart of the application options is to test only compress and only decompress but actually the application ignores this user option and tries to test always both compress and decompress.
Allow testing only compress and only decompress.
--- app/test-compress-perf/comp_perf_options.h | 6 +- .../comp_perf_options_parse.c | 4 +- .../comp_perf_test_common.c | 126 +++++++++++++----- .../comp_perf_test_cyclecount.c | 69 ++++++---- .../comp_perf_test_throughput.c | 64 +++++---- .../comp_perf_test_verify.c | 60 ++++++--- app/test-compress-perf/main.c | 8 ++ 7 files changed, 223 insertions(+), 114 deletions(-)
app/test-compress-perf/comp_perf_options.h +3 -3
@@ -32,9 +32,9 @@ enum cperf_test_type { };enum comp_operation { -COMPRESS_ONLY , -DECOMPRESS_ONLY , -COMPRESS_DECOMPRESS +COMPRESS = (1 <<0 ), +DECOMPRESS = (1 <<1 ), +COMPRESS_DECOMPRESS = (COMPRESS |DECOMPRESS ), };struct range_list {
app/test-compress-perf/comp_perf_options_parse.c +2 -2
@@ -446,11 +446,11 @@ parse_op_type(struct comp_test_data *test_data, const char *arg)struct name_id_map optype_namemap[] = { {"comp" , -COMPRESS_ONLY +COMPRESS }, {"decomp" , -DECOMPRESS_ONLY +DECOMPRESS }, {"comp_and_decomp" ,
app/test-compress-perf/comp_perf_test_common.c +91 -35
222 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_cyclecount.c +43 -26
81 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_throughput.c +38 -26
79 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_verify.c +38 -22
89 lines not on this page · raw message
app/test-compress-perf/main.c +8 -0
@@ -254,6 +254,14 @@ comp_perf_dump_input_data(struct comp_test_data *test_data) goto end; } +if (!(test_data->test_op &COMPRESS ) && + test_data->input_data_sz > + (size_t ) test_data->seg_sz * (size_t ) test_data->max_sgl_segs) { +RTE_LOG (ERR ,USER1 , +"Size of input must be less than total segments\n" ); + goto end; + } + test_data->input_data =rte_zmalloc_socket (NULL , test_data->input_data_sz,0 ,rte_socket_id ());
Signature · 1 line
2.25.1
PATCH v4 4/4 app/test-compress-perf: add algo option
In reply to Michael BaumAdd a command line option algo to select the compress algorithm supported by the compress API: null (DMA), deflate, lz4 and lzs.
Default for deflate.
--- app/test-compress-perf/comp_perf_options.h | 3 + .../comp_perf_options_parse.c | 75 +++++++++++++++++++ .../comp_perf_test_cyclecount.c | 11 ++- .../comp_perf_test_throughput.c | 11 ++- .../comp_perf_test_verify.c | 11 ++- app/test-compress-perf/main.c | 55 ++++++++++---- doc/guides/tools/comp_perf.rst | 5 ++ 7 files changed, 148 insertions(+), 23 deletions(-)
app/test-compress-perf/comp_perf_options.h +3 -0
@@ -61,8 +61,11 @@ struct comp_test_data {uint16_t max_sgl_segs;uint32_t total_segs; +uint8_t lz4_flags;enum rte_comp_huffman huffman_enc;enum comp_operation test_op; +enum rte_comp_algorithm test_algo; +int window_sz;struct range_list level_lst;uint8_t level;
app/test-compress-perf/comp_perf_options_parse.c +75 -0
130 lines not on this page · raw message
app/test-compress-perf/comp_perf_test_cyclecount.c +8 -3
@@ -193,14 +193,17 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , - .deflate.huffman = test_data->huffman_enc, + .algo = test_data->test_algo, .level = test_data->level, .window_size = test_data->window_sz, .chksum =RTE_COMP_CHECKSUM_NONE , .hash_algo =RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; input_bufs = mem->decomp_bufs; output_bufs = mem->comp_bufs; out_seg_sz = test_data->out_seg_sz;
@@ -208,12 +211,14 @@ main_loop(struct cperf_cyclecount_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; input_bufs = mem->comp_bufs; output_bufs = mem->decomp_bufs; out_seg_sz = test_data->seg_sz;
app/test-compress-perf/comp_perf_test_throughput.c +8 -3
@@ -84,14 +84,17 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , - .deflate.huffman = test_data->huffman_enc, + .algo = test_data->test_algo, .level = test_data->level, .window_size = test_data->window_sz, .chksum =RTE_COMP_CHECKSUM_NONE , .hash_algo =RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; input_bufs = mem->decomp_bufs; output_bufs = mem->comp_bufs; out_seg_sz = test_data->out_seg_sz;
@@ -99,12 +102,14 @@ main_loop(struct cperf_benchmark_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; input_bufs = mem->comp_bufs; output_bufs = mem->decomp_bufs; out_seg_sz = test_data->seg_sz;
app/test-compress-perf/comp_perf_test_verify.c +8 -3
@@ -87,14 +87,17 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_COMPRESS , .compress = { - .algo =RTE_COMP_ALGO_DEFLATE , - .deflate.huffman = test_data->huffman_enc, + .algo = test_data->test_algo, .level = test_data->level, .window_size = test_data->window_sz, .chksum =RTE_COMP_CHECKSUM_NONE , .hash_algo =RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_DEFLATE ) + xform.compress.deflate.huffman = test_data->huffman_enc; +else if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.compress.lz4.flags = test_data->lz4_flags; output_data_ptr = ctx->mem.compressed_data; output_data_sz = &ctx->comp_data_sz; input_bufs = mem->decomp_bufs;
@@ -104,12 +107,14 @@ main_loop(struct cperf_verify_ctx *ctx, enum rte_comp_xform_type type) xform = (struct rte_comp_xform ) { .type =RTE_COMP_DECOMPRESS , .decompress = { - .algo =RTE_COMP_ALGO_DEFLATE , + .algo = test_data->test_algo, .chksum =RTE_COMP_CHECKSUM_NONE , .window_size = test_data->window_sz, .hash_algo =RTE_COMP_HASH_ALGO_NONE } }; +if (test_data->test_algo ==RTE_COMP_ALGO_LZ4 ) + xform.decompress.lz4.flags = test_data->lz4_flags; output_data_ptr = ctx->mem.decompressed_data; output_data_sz = &ctx->decomp_data_sz; input_bufs = mem->comp_bufs;
app/test-compress-perf/main.c +41 -14
70 lines not on this page · raw message
doc/guides/tools/comp_perf.rst +5 -0
@@ -84,8 +84,13 @@ Application Options ``--operation [comp/decomp/comp_and_decomp]``: perform test on compression, decompression or both operations + ``--algo [null/deflate/lzs/lz4]`` : perform test on algorithm null(DMA), Deflate, lzs or lz4 (default: Deflate) + ``--huffman-enc [fixed/dynamic/default]``: Huffman encoding (default: dynamic) + ``--lz4-flags N``: flags to for LZ4 parameters, + see `LZ4 Frame Descriptor <https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#frame-descriptor>`_ (default: no flags) + ``--compress-level N``: compression level, which could be a single value, list or range (default: range between 1 and 9) ``--window-sz N``: base two log value of compression window size (default: max supported by PMD)
Signature · 1 line
2.25.1
Akhil Goyal
AppliedQuoted 26 lines
Add support for LZ4 compression algorithm for both API and app.
In addition, enable app to test de/comp only. This option already exists in both app doc and arguments but isn't implemented inside the application.
LZ4: https://urldefense.proofpoint.com/v2/url?u=https-
3A__github.com_lz4_lz4&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2 wl_PRwpZ9TWey3eu68gBzn7DkPwuqhd6WNyo&m=zuvFwq- wCfo4E5qcsXuVW4Tg-xq4cnKWlMdx8BrlcwM9- mjeKpQgAejfzaUzSwyP&s=89JL5ZLAeh2gshZ3Gqohsr3JZomgK9pjaCnLRHpnG0w &e=
RFC:
https://urldefense.proofpoint.com/v2/url?u=https-
3A__patchwork.dpdk.org_project_dpdk_patch_20220410182622.8828-2D1- 2Drzidane- 40nvidia.com_&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=DnL7Si2wl_PRwpZ 9TWey3eu68gBzn7DkPwuqhd6WNyo&m=zuvFwq-wCfo4E5qcsXuVW4Tg- xq4cnKWlMdx8BrlcwM9- mjeKpQgAejfzaUzSwyP&s=HhD8pa6ivCYKy89RMuLN2OhehLXxbluIyVWxF0QdjXE &e=v2:
- Add macros for all LZ4 flags in the standard.
- Rebase.
Quoted 8 lines
v3:
- Rebase.v4:
- Change "allow test single" commit to fix and add "Cc: [email protected]".
- Take all typo fixes into separate commit.
- Squash "add LZ4 algo" commit into "add algo option" commit.
- Remove unnecessary release notes updates.
Applied to dpdk-next-crypto
Thanks