kdevops: add milvus with minio support
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest2 patches 5 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/2 | ai: add Milvus vector database benchmarking support.gitignore README.md defconfigs/ +70 more | +9999 | 0 replies | |
| 2/2 | ai: add multi-filesystem testing support for Milvus benchmarks.github/ Makefile defconfigs/ +36 more | +4356 | 4 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [PATCH 0/2] kdevops: add milvus with minio support
This adds the ability to test milvus on minio with different filesystem configuration targets. There's a basic configuration you can run which will just support one default filesystem, which will be used where you place your docker image and also where we place the minio instance. Then there is multifs support where just as with fstests support on kdevops you can select a slew of different filesystem targets to try to test.
Recommendation is to stick to 40 iterations at 1,000,000 tests unless you have more than 100 GiB per guest to spare. If you have space to spare then you know how to ballpark it.
On High Capacity SSDs, the world is our oyster.
You can see a demo of results here:
https://github.com/mcgrof/demo-milvus-kdevops-results
These are just demos. On guests. Nothing really useful. I should point out this has AB testing automated as well so we can leverage this to test for instance ... parallel writeback in an automated way ;)
If you want to test this you can also use this branch on kdevops:
https://github.com/linux-kdevops/kdevops/tree/mcgrof/20250827-milvus
I am in hopes someone will just prompt an AI for bare metal support while I sleep. It should be... easy. Just create the partitions already, use one host and ask the prompt to not mkfs for you. So don't use multi-fs support at first. Just use the option to create the storage partition where you place docker. In fact you can copy and paste this prompt the the AI, and I think it will know what to do. You just skip some steps as the filesystems can be created and mounted for you. You just need the host file created by you manually for the target node. That and infer user and group id support (WORKFLOW_INFER_USER_AND_GROUP).
Luis Chamberlain (2):
ai: add Milvus vector database benchmarking support
ai: add multi-filesystem testing support for Milvus benchmarks
.github/workflows/docker-tests.yml | 6 + .gitignore | 3 +- Makefile | 2 +- README.md | 18 + defconfigs/ai-milvus-docker | 113 ++ defconfigs/ai-milvus-docker-ci | 51 + defconfigs/ai-milvus-multifs | 67 + defconfigs/ai-milvus-multifs-distro | 109 ++ defconfigs/ai-milvus-multifs-extended | 108 ++ docs/ai/README.md | 108 ++ docs/ai/vector-databases/README.md | 75 + docs/ai/vector-databases/milvus.md | 264 +++ kconfigs/workflows/Kconfig | 27 + playbooks/ai.yml | 11 + playbooks/ai_benchmark.yml | 8 + playbooks/ai_destroy.yml | 24 + playbooks/ai_install.yml | 14 + playbooks/ai_multifs.yml | 24 + playbooks/ai_results.yml | 6 + playbooks/ai_setup.yml | 6 + playbooks/ai_tests.yml | 31 + playbooks/ai_uninstall.yml | 6 + .../debian13-ai-btrfs-default-dev.yml | 8 + .../host_vars/debian13-ai-btrfs-default.yml | 8 + .../debian13-ai-ext4-16k-bigalloc-dev.yml | 8 + .../debian13-ai-ext4-16k-bigalloc.yml | 8 + .../host_vars/debian13-ai-ext4-4k-dev.yml | 8 + playbooks/host_vars/debian13-ai-ext4-4k.yml | 8 + .../host_vars/debian13-ai-xfs-16k-4ks-dev.yml | 10 + .../host_vars/debian13-ai-xfs-16k-4ks.yml | 10 + .../host_vars/debian13-ai-xfs-32k-4ks-dev.yml | 10 + .../host_vars/debian13-ai-xfs-32k-4ks.yml | 10 + .../host_vars/debian13-ai-xfs-4k-4ks-dev.yml | 10 + .../host_vars/debian13-ai-xfs-64k-4ks-dev.yml | 10 + .../host_vars/debian13-ai-xfs-64k-4ks.yml | 10 + .../files/analyze_results.py | 1701 +++++++++++++++++ .../files/generate_better_graphs.py | 550 ++++++ .../files/generate_graphs.py | 362 ++++ .../files/generate_html_report.py | 610 ++++++ .../roles/ai_collect_results/tasks/main.yml | 202 ++ .../templates/analysis_config.json.j2 | 6 + playbooks/roles/ai_destroy/tasks/main.yml | 63 + .../roles/ai_docker_storage/tasks/main.yml | 123 ++ playbooks/roles/ai_install/tasks/main.yml | 90 + .../roles/ai_milvus_storage/tasks/main.yml | 161 ++ .../tasks/generate_comparison.yml | 279 +++ playbooks/roles/ai_multifs_run/tasks/main.yml | 23 + .../tasks/run_single_filesystem.yml | 104 + .../templates/milvus_config.json.j2 | 42 + .../roles/ai_multifs_setup/defaults/main.yml | 49 + .../roles/ai_multifs_setup/tasks/main.yml | 70 + playbooks/roles/ai_results/tasks/main.yml | 22 + .../files/milvus_benchmark.py | 556 ++++++ .../roles/ai_run_benchmarks/tasks/main.yml | 181 ++ .../templates/benchmark_config.json.j2 | 24 + playbooks/roles/ai_setup/tasks/main.yml | 115 ++ playbooks/roles/ai_uninstall/tasks/main.yml | 62 + playbooks/roles/gen_hosts/tasks/main.yml | 33 + .../roles/gen_hosts/templates/fstests.j2 | 2 + playbooks/roles/gen_hosts/templates/gitr.j2 | 2 + playbooks/roles/gen_hosts/templates/hosts.j2 | 99 + .../roles/gen_hosts/templates/nfstest.j2 | 2 + playbooks/roles/gen_hosts/templates/pynfs.j2 | 2 + playbooks/roles/gen_nodes/tasks/main.yml | 124 ++ .../roles/guestfs/tasks/bringup/main.yml | 15 + playbooks/roles/milvus/README.md | 181 ++ playbooks/roles/milvus/defaults/main.yml | 74 + .../roles/milvus/files/milvus_benchmark.py | 348 ++++ playbooks/roles/milvus/files/milvus_utils.py | 134 ++ playbooks/roles/milvus/meta/main.yml | 30 + playbooks/roles/milvus/tasks/benchmark.yml | 61 + .../roles/milvus/tasks/benchmark_setup.yml | 58 + .../roles/milvus/tasks/install_docker.yml | 97 + playbooks/roles/milvus/tasks/main.yml | 52 + playbooks/roles/milvus/tasks/setup.yml | 107 ++ .../milvus/templates/benchmark_config.json.j2 | 25 + .../templates/docker-compose.override.yml.j2 | 24 + .../milvus/templates/docker-compose.yml.j2 | 64 + .../roles/milvus/templates/milvus.yaml.j2 | 30 + .../milvus/templates/test_connection.py.j2 | 25 + scripts/guestfs.Makefile | 2 +- workflows/Makefile | 4 + workflows/ai/Kconfig | 177 ++ workflows/ai/Kconfig.docker | 172 ++ workflows/ai/Kconfig.docker-storage | 201 ++ workflows/ai/Kconfig.fs | 118 ++ workflows/ai/Kconfig.multifs | 184 ++ workflows/ai/Kconfig.native | 184 ++ workflows/ai/Makefile | 160 ++ workflows/ai/scripts/analysis_config.json | 6 + workflows/ai/scripts/analyze_results.py | 1701 +++++++++++++++++ workflows/ai/scripts/generate_graphs.py | 362 ++++ workflows/ai/scripts/generate_html_report.py | 610 ++++++ 93 files changed, 12061 insertions(+), 3 deletions(-) create mode 100644 defconfigs/ai-milvus-docker create mode 100644 defconfigs/ai-milvus-docker-ci create mode 100644 defconfigs/ai-milvus-multifs create mode 100644 defconfigs/ai-milvus-multifs-distro create mode 100644 defconfigs/ai-milvus-multifs-extended create mode 100644 docs/ai/README.md create mode 100644 docs/ai/vector-databases/README.md create mode 100644 docs/ai/vector-databases/milvus.md create mode 100644 playbooks/ai.yml create mode 100644 playbooks/ai_benchmark.yml create mode 100644 playbooks/ai_destroy.yml create mode 100644 playbooks/ai_install.yml create mode 100644 playbooks/ai_multifs.yml create mode 100644 playbooks/ai_results.yml create mode 100644 playbooks/ai_setup.yml create mode 100644 playbooks/ai_tests.yml create mode 100644 playbooks/ai_uninstall.yml create mode 100644 playbooks/host_vars/debian13-ai-btrfs-default-dev.yml create mode 100644 playbooks/host_vars/debian13-ai-btrfs-default.yml create mode 100644 playbooks/host_vars/debian13-ai-ext4-16k-bigalloc-dev.yml create mode 100644 playbooks/host_vars/debian13-ai-ext4-16k-bigalloc.yml create mode 100644 playbooks/host_vars/debian13-ai-ext4-4k-dev.yml create mode 100644 playbooks/host_vars/debian13-ai-ext4-4k.yml create mode 100644 playbooks/host_vars/debian13-ai-xfs-16k-4ks-dev.yml create mode 100644 playbooks/host_vars/debian13-ai-xfs-16k-4ks.yml create mode 100644 playbooks/host_vars/debian13-ai-xfs-32k-4ks-dev.yml create mode 100644 playbooks/host_vars/debian13-ai-xfs-32k-4ks.yml create mode 100644 playbooks/host_vars/debian13-ai-xfs-4k-4ks-dev.yml create mode 100644 playbooks/host_vars/debian13-ai-xfs-64k-4ks-dev.yml create mode 100644 playbooks/host_vars/debian13-ai-xfs-64k-4ks.yml create mode 100755 playbooks/roles/ai_collect_results/files/analyze_results.py create mode 100755 playbooks/roles/ai_collect_results/files/generate_better_graphs.py create mode 100755 playbooks/roles/ai_collect_results/files/generate_graphs.py create mode 100755 playbooks/roles/ai_collect_results/files/generate_html_report.py create mode 100644 playbooks/roles/ai_collect_results/tasks/main.yml create mode 100644 playbooks/roles/ai_collect_results/templates/analysis_config.json.j2 create mode 100644 playbooks/roles/ai_destroy/tasks/main.yml create mode 100644 playbooks/roles/ai_docker_storage/tasks/main.yml create mode 100644 playbooks/roles/ai_install/tasks/main.yml create mode 100644 playbooks/roles/ai_milvus_storage/tasks/main.yml create mode 100644 playbooks/roles/ai_multifs_run/tasks/generate_comparison.yml create mode 100644 playbooks/roles/ai_multifs_run/tasks/main.yml create mode 100644 playbooks/roles/ai_multifs_run/tasks/run_single_filesystem.yml create mode 100644 playbooks/roles/ai_multifs_run/templates/milvus_config.json.j2 create mode 100644 playbooks/roles/ai_multifs_setup/defaults/main.yml create mode 100644 playbooks/roles/ai_multifs_setup/tasks/main.yml create mode 100644 playbooks/roles/ai_results/tasks/main.yml create mode 100644 playbooks/roles/ai_run_benchmarks/files/milvus_benchmark.py create mode 100644 playbooks/roles/ai_run_benchmarks/tasks/main.yml create mode 100644 playbooks/roles/ai_run_benchmarks/templates/benchmark_config.json.j2 create mode 100644 playbooks/roles/ai_setup/tasks/main.yml create mode 100644 playbooks/roles/ai_uninstall/tasks/main.yml create mode 100644 playbooks/roles/milvus/README.md create mode 100644 playbooks/roles/milvus/defaults/main.yml create mode 100644 playbooks/roles/milvus/files/milvus_benchmark.py create mode 100644 playbooks/roles/milvus/files/milvus_utils.py create mode 100644 playbooks/roles/milvus/meta/main.yml create mode 100644 playbooks/roles/milvus/tasks/benchmark.yml create mode 100644 playbooks/roles/milvus/tasks/benchmark_setup.yml create mode 100644 playbooks/roles/milvus/tasks/install_docker.yml create mode 100644 playbooks/roles/milvus/tasks/main.yml create mode 100644 playbooks/roles/milvus/tasks/setup.yml create mode 100644 playbooks/roles/milvus/templates/benchmark_config.json.j2 create mode 100644 playbooks/roles/milvus/templates/docker-compose.override.yml.j2 create mode 100644 playbooks/roles/milvus/templates/docker-compose.yml.j2 create mode 100644 playbooks/roles/milvus/templates/milvus.yaml.j2 create mode 100644 playbooks/roles/milvus/templates/test_connection.py.j2 create mode 100644 workflows/ai/Kconfig create mode 100644 workflows/ai/Kconfig.docker create mode 100644 workflows/ai/Kconfig.docker-storage create mode 100644 workflows/ai/Kconfig.fs create mode 100644 workflows/ai/Kconfig.multifs create mode 100644 workflows/ai/Kconfig.native create mode 100644 workflows/ai/Makefile create mode 100644 workflows/ai/scripts/analysis_config.json create mode 100755 workflows/ai/scripts/analyze_results.py create mode 100755 workflows/ai/scripts/generate_graphs.py create mode 100755 workflows/ai/scripts/generate_html_report.py
Signature · 1 line
2.50.1