Skip to content

kdevops: add milvus with minio support

Luis Chamberlain kdevops 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest2 patches 5 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2ai: add Milvus vector database benchmarking support.gitignore README.md defconfigs/ai-milvus-docker +70 more+9999 -10 replies
2/2ai: add multi-filesystem testing support for Milvus benchmarks.github/workflows/docker-tests.yml Makefile defconfigs/ai-milvus-multifs +36 more+4356 -22964 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/2] kdevops: add milvus with minio support

This adds the ability to test milvus on minio with different filesystem configuration targets. There's a basic configuration you can run which will just support one default filesystem, which will be used where you place your docker image and also where we place the minio instance. Then there is multifs support where just as with fstests support on kdevops you can select a slew of different filesystem targets to try to test.

Recommendation is to stick to 40 iterations at 1,000,000 tests unless you have more than 100 GiB per guest to spare. If you have space to spare then you know how to ballpark it.

On High Capacity SSDs, the world is our oyster.

You can see a demo of results here:

https://github.com/mcgrof/demo-milvus-kdevops-results

These are just demos. On guests. Nothing really useful. I should point out this has AB testing automated as well so we can leverage this to test for instance ... parallel writeback in an automated way ;)

If you want to test this you can also use this branch on kdevops:

https://github.com/linux-kdevops/kdevops/tree/mcgrof/20250827-milvus

I am in hopes someone will just prompt an AI for bare metal support while I sleep. It should be... easy. Just create the partitions already, use one host and ask the prompt to not mkfs for you. So don't use multi-fs support at first. Just use the option to create the storage partition where you place docker. In fact you can copy and paste this prompt the the AI, and I think it will know what to do. You just skip some steps as the filesystems can be created and mounted for you. You just need the host file created by you manually for the target node. That and infer user and group id support (WORKFLOW_INFER_USER_AND_GROUP).

Luis Chamberlain (2):
ai: add Milvus vector database benchmarking support
ai: add multi-filesystem testing support for Milvus benchmarks

 .github/workflows/docker-tests.yml            |    6 +
 .gitignore                                    |    3 +-
 Makefile                                      |    2 +-
 README.md                                     |   18 +
 defconfigs/ai-milvus-docker                   |  113 ++
 defconfigs/ai-milvus-docker-ci                |   51 +
 defconfigs/ai-milvus-multifs                  |   67 +
 defconfigs/ai-milvus-multifs-distro           |  109 ++
 defconfigs/ai-milvus-multifs-extended         |  108 ++
 docs/ai/README.md                             |  108 ++
 docs/ai/vector-databases/README.md            |   75 +
 docs/ai/vector-databases/milvus.md            |  264 +++
 kconfigs/workflows/Kconfig                    |   27 +
 playbooks/ai.yml                              |   11 +
 playbooks/ai_benchmark.yml                    |    8 +
 playbooks/ai_destroy.yml                      |   24 +
 playbooks/ai_install.yml                      |   14 +
 playbooks/ai_multifs.yml                      |   24 +
 playbooks/ai_results.yml                      |    6 +
 playbooks/ai_setup.yml                        |    6 +
 playbooks/ai_tests.yml                        |   31 +
 playbooks/ai_uninstall.yml                    |    6 +
 .../debian13-ai-btrfs-default-dev.yml         |    8 +
 .../host_vars/debian13-ai-btrfs-default.yml   |    8 +
 .../debian13-ai-ext4-16k-bigalloc-dev.yml     |    8 +
 .../debian13-ai-ext4-16k-bigalloc.yml         |    8 +
 .../host_vars/debian13-ai-ext4-4k-dev.yml     |    8 +
 playbooks/host_vars/debian13-ai-ext4-4k.yml   |    8 +
 .../host_vars/debian13-ai-xfs-16k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-16k-4ks.yml     |   10 +
 .../host_vars/debian13-ai-xfs-32k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-32k-4ks.yml     |   10 +
 .../host_vars/debian13-ai-xfs-4k-4ks-dev.yml  |   10 +
 .../host_vars/debian13-ai-xfs-64k-4ks-dev.yml |   10 +
 .../host_vars/debian13-ai-xfs-64k-4ks.yml     |   10 +
 .../files/analyze_results.py                  | 1701 +++++++++++++++++
 .../files/generate_better_graphs.py           |  550 ++++++
 .../files/generate_graphs.py                  |  362 ++++
 .../files/generate_html_report.py             |  610 ++++++
 .../roles/ai_collect_results/tasks/main.yml   |  202 ++
 .../templates/analysis_config.json.j2         |    6 +
 playbooks/roles/ai_destroy/tasks/main.yml     |   63 +
 .../roles/ai_docker_storage/tasks/main.yml    |  123 ++
 playbooks/roles/ai_install/tasks/main.yml     |   90 +
 .../roles/ai_milvus_storage/tasks/main.yml    |  161 ++
 .../tasks/generate_comparison.yml             |  279 +++
 playbooks/roles/ai_multifs_run/tasks/main.yml |   23 +
 .../tasks/run_single_filesystem.yml           |  104 +
 .../templates/milvus_config.json.j2           |   42 +
 .../roles/ai_multifs_setup/defaults/main.yml  |   49 +
 .../roles/ai_multifs_setup/tasks/main.yml     |   70 +
 playbooks/roles/ai_results/tasks/main.yml     |   22 +
 .../files/milvus_benchmark.py                 |  556 ++++++
 .../roles/ai_run_benchmarks/tasks/main.yml    |  181 ++
 .../templates/benchmark_config.json.j2        |   24 +
 playbooks/roles/ai_setup/tasks/main.yml       |  115 ++
 playbooks/roles/ai_uninstall/tasks/main.yml   |   62 +
 playbooks/roles/gen_hosts/tasks/main.yml      |   33 +
 .../roles/gen_hosts/templates/fstests.j2      |    2 +
 playbooks/roles/gen_hosts/templates/gitr.j2   |    2 +
 playbooks/roles/gen_hosts/templates/hosts.j2  |   99 +
 .../roles/gen_hosts/templates/nfstest.j2      |    2 +
 playbooks/roles/gen_hosts/templates/pynfs.j2  |    2 +
 playbooks/roles/gen_nodes/tasks/main.yml      |  124 ++
 .../roles/guestfs/tasks/bringup/main.yml      |   15 +
 playbooks/roles/milvus/README.md              |  181 ++
 playbooks/roles/milvus/defaults/main.yml      |   74 +
 .../roles/milvus/files/milvus_benchmark.py    |  348 ++++
 playbooks/roles/milvus/files/milvus_utils.py  |  134 ++
 playbooks/roles/milvus/meta/main.yml          |   30 +
 playbooks/roles/milvus/tasks/benchmark.yml    |   61 +
 .../roles/milvus/tasks/benchmark_setup.yml    |   58 +
 .../roles/milvus/tasks/install_docker.yml     |   97 +
 playbooks/roles/milvus/tasks/main.yml         |   52 +
 playbooks/roles/milvus/tasks/setup.yml        |  107 ++
 .../milvus/templates/benchmark_config.json.j2 |   25 +
 .../templates/docker-compose.override.yml.j2  |   24 +
 .../milvus/templates/docker-compose.yml.j2    |   64 +
 .../roles/milvus/templates/milvus.yaml.j2     |   30 +
 .../milvus/templates/test_connection.py.j2    |   25 +
 scripts/guestfs.Makefile                      |    2 +-
 workflows/Makefile                            |    4 +
 workflows/ai/Kconfig                          |  177 ++
 workflows/ai/Kconfig.docker                   |  172 ++
 workflows/ai/Kconfig.docker-storage           |  201 ++
 workflows/ai/Kconfig.fs                       |  118 ++
 workflows/ai/Kconfig.multifs                  |  184 ++
 workflows/ai/Kconfig.native                   |  184 ++
 workflows/ai/Makefile                         |  160 ++
 workflows/ai/scripts/analysis_config.json     |    6 +
 workflows/ai/scripts/analyze_results.py       | 1701 +++++++++++++++++
 workflows/ai/scripts/generate_graphs.py       |  362 ++++
 workflows/ai/scripts/generate_html_report.py  |  610 ++++++
 93 files changed, 12061 insertions(+), 3 deletions(-)
 create mode 100644 defconfigs/ai-milvus-docker
 create mode 100644 defconfigs/ai-milvus-docker-ci
 create mode 100644 defconfigs/ai-milvus-multifs
 create mode 100644 defconfigs/ai-milvus-multifs-distro
 create mode 100644 defconfigs/ai-milvus-multifs-extended
 create mode 100644 docs/ai/README.md
 create mode 100644 docs/ai/vector-databases/README.md
 create mode 100644 docs/ai/vector-databases/milvus.md
 create mode 100644 playbooks/ai.yml
 create mode 100644 playbooks/ai_benchmark.yml
 create mode 100644 playbooks/ai_destroy.yml
 create mode 100644 playbooks/ai_install.yml
 create mode 100644 playbooks/ai_multifs.yml
 create mode 100644 playbooks/ai_results.yml
 create mode 100644 playbooks/ai_setup.yml
 create mode 100644 playbooks/ai_tests.yml
 create mode 100644 playbooks/ai_uninstall.yml
 create mode 100644 playbooks/host_vars/debian13-ai-btrfs-default-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-btrfs-default.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-16k-bigalloc-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-16k-bigalloc.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-4k-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-ext4-4k.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-16k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-16k-4ks.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-32k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-32k-4ks.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-4k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-64k-4ks-dev.yml
 create mode 100644 playbooks/host_vars/debian13-ai-xfs-64k-4ks.yml
 create mode 100755 playbooks/roles/ai_collect_results/files/analyze_results.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_better_graphs.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_graphs.py
 create mode 100755 playbooks/roles/ai_collect_results/files/generate_html_report.py
 create mode 100644 playbooks/roles/ai_collect_results/tasks/main.yml
 create mode 100644 playbooks/roles/ai_collect_results/templates/analysis_config.json.j2
 create mode 100644 playbooks/roles/ai_destroy/tasks/main.yml
 create mode 100644 playbooks/roles/ai_docker_storage/tasks/main.yml
 create mode 100644 playbooks/roles/ai_install/tasks/main.yml
 create mode 100644 playbooks/roles/ai_milvus_storage/tasks/main.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/generate_comparison.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/main.yml
 create mode 100644 playbooks/roles/ai_multifs_run/tasks/run_single_filesystem.yml
 create mode 100644 playbooks/roles/ai_multifs_run/templates/milvus_config.json.j2
 create mode 100644 playbooks/roles/ai_multifs_setup/defaults/main.yml
 create mode 100644 playbooks/roles/ai_multifs_setup/tasks/main.yml
 create mode 100644 playbooks/roles/ai_results/tasks/main.yml
 create mode 100644 playbooks/roles/ai_run_benchmarks/files/milvus_benchmark.py
 create mode 100644 playbooks/roles/ai_run_benchmarks/tasks/main.yml
 create mode 100644 playbooks/roles/ai_run_benchmarks/templates/benchmark_config.json.j2
 create mode 100644 playbooks/roles/ai_setup/tasks/main.yml
 create mode 100644 playbooks/roles/ai_uninstall/tasks/main.yml
 create mode 100644 playbooks/roles/milvus/README.md
 create mode 100644 playbooks/roles/milvus/defaults/main.yml
 create mode 100644 playbooks/roles/milvus/files/milvus_benchmark.py
 create mode 100644 playbooks/roles/milvus/files/milvus_utils.py
 create mode 100644 playbooks/roles/milvus/meta/main.yml
 create mode 100644 playbooks/roles/milvus/tasks/benchmark.yml
 create mode 100644 playbooks/roles/milvus/tasks/benchmark_setup.yml
 create mode 100644 playbooks/roles/milvus/tasks/install_docker.yml
 create mode 100644 playbooks/roles/milvus/tasks/main.yml
 create mode 100644 playbooks/roles/milvus/tasks/setup.yml
 create mode 100644 playbooks/roles/milvus/templates/benchmark_config.json.j2
 create mode 100644 playbooks/roles/milvus/templates/docker-compose.override.yml.j2
 create mode 100644 playbooks/roles/milvus/templates/docker-compose.yml.j2
 create mode 100644 playbooks/roles/milvus/templates/milvus.yaml.j2
 create mode 100644 playbooks/roles/milvus/templates/test_connection.py.j2
 create mode 100644 workflows/ai/Kconfig
 create mode 100644 workflows/ai/Kconfig.docker
 create mode 100644 workflows/ai/Kconfig.docker-storage
 create mode 100644 workflows/ai/Kconfig.fs
 create mode 100644 workflows/ai/Kconfig.multifs
 create mode 100644 workflows/ai/Kconfig.native
 create mode 100644 workflows/ai/Makefile
 create mode 100644 workflows/ai/scripts/analysis_config.json
 create mode 100755 workflows/ai/scripts/analyze_results.py
 create mode 100755 workflows/ai/scripts/generate_graphs.py
 create mode 100755 workflows/ai/scripts/generate_html_report.py
Signature · 1 line
2.50.1