Skip to contentSkip to conversation

scripts: Add clean-hashserver-database script

PATCH Superseded Alexander Marques 3 participants 5 messages Mar 12, 2025 – Mar 14, 2025

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 scripts: Add clean-hashserver-database script

From: Alexandre Marques <[email protected]>

Auxiliary script to clean the hashserver database based on the files available in the sstate directory.

It makes used of the new "hashclient gc-mark-stream" command to mark all sstate relevant hashes as "alive" and removes everything else from the database.

Usage example:

```
./scripts/clean-hashserver-database \
    --sstate-dir ~/build/sstate-cache \
    --hashclient ./bitbake/bin/bitabke-hashclient \
    --hashserver-address "ws://localhost:8688/ws" \
    --mark "alive" \
    --clean-db
```
---
 scripts/clean-hashserver-database | 73 +++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
 create mode 100755 scripts/clean-hashserver-database
scripts/clean-hashserver-database new file +73 -0
new file mode 100755
index 0000000000..6eb006758e
@@ -0,0 +1,73 @@
+#!/bin/bash
+set -euo pipefail
+
+SSTATE_DIR=""
+BB_HASHCLIENT=""
+BB_HASHSERVER=""
+
+ALIVE_DB_MARK="alive"
+CLEAN_DB="false"
+
+function help() {
+    cat <<HELP_TEXT
+Usage: $0 --sstate-dir path --hashclient path --hashserver-address address [--mark value] [--clean-db]
+
+Auxiliary script remove unused or no longer relevant entries from the hashequivalence database, based
+on the files available on the sstate directory.
+
+    -h | --help)               Show this help message and exit
+    -s | --sstate-dir)         Path to the sstate dir
+    -c | --hashclient)         Path to bitbake-hashclient
+    -a | --hashserver-adress)  bitbake-hashserver address
+    -m | --mark)               Marker string to mark database entries
+    --clean-db)                Remove all unmarked and unused entries from the database
+HELP_TEXT
+}
+
+function argument_parser() {
+    while [ $# -gt 0 ]; do
+        case "$1" in
+            -h | --help) help; exit 0 ;;
+            -s | --sstate-dir) SSTATE_DIR="$2"; shift ;;
+            -c | --hashclient) BB_HASHCLIENT="$2"; shift ;;
+            -a | --hashserver-address) BB_HASHSERVER="$2"; shift ;;
+            -m | --mark) ALIVE_DB_MARK="$2"; shift ;;
+            --clean-db) CLEAN_DB="true";;
+            *)
+                echo "Argument '$1' is not supported" >&2
+                help >&2
+                exit 1
+                ;;
+        esac
+        shift
+    done
+
+    function validate_mandatory_argument() {
+        local var_value="$1"
+        local error_message="$2"
+
+        if [ -z "$var_value" ]; then
+            echo "$error_message"
+            help >&2
+            exit 1
+        fi
+    }
+
+    validate_mandatory_argument "$SSTATE_DIR" "Please provide the path to the sstate dir."
+    validate_mandatory_argument "$BB_HASHCLIENT" "Please provide the path to bitbake-hashclient."
+    validate_mandatory_argument "$BB_HASHSERVER" "Please provide the address of bitbake-hashserver."
+}
+
+# -- main code --
+argument_parser $@
+
+# Mark all db sstate hashes
+find "$SSTATE_DIR" -name "*.tar.zst" | \
+sed 's/.*:\([^_]*\)_.*/unihash \1/' | \
+$BB_HASHCLIENT --address "$BB_HASHSERVER" gc-mark-stream "${ALIVE_DB_MARK}"
+
+# Remove unmarked and unused entries
+if [ "$CLEAN_DB" = "true" ]; then
+    $BB_HASHCLIENT --address "$BB_HASHSERVER" gc-sweep "${ALIVE_DB_MARK}"
+    $BB_HASHCLIENT --address "$BB_HASHSERVER" clean-unused 0
+fi
Signature · 1 line
2.34.1
#1 lore raw
Quoted 94 lines · Alexandre Marques via lists.openembedded.org

From: Alexandre Marques <[email protected]>

Auxiliary script to clean the hashserver database based on the files available in the sstate directory.

It makes used of the new "hashclient gc-mark-stream" command to mark all sstate relevant hashes as "alive" and removes everything else from the database.

Usage example:

```
./scripts/clean-hashserver-database \
    --sstate-dir ~/build/sstate-cache \
    --hashclient ./bitbake/bin/bitabke-hashclient \
    --hashserver-address "ws://localhost:8688/ws" \
    --mark "alive" \
    --clean-db
```
---
 scripts/clean-hashserver-database | 73 +++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
 create mode 100755 scripts/clean-hashserver-database
scripts/clean-hashserver-database new file +62 -0
new file mode 100755
index 0000000000..6eb006758e
@@ -0,0 +1,73 @@
+#!/bin/bash
+set -euo pipefail
+
+SSTATE_DIR=""
+BB_HASHCLIENT=""
+BB_HASHSERVER=""
+
+ALIVE_DB_MARK="alive"
+CLEAN_DB="false"
+
+function help() {
+    cat <<HELP_TEXT
+Usage: $0 --sstate-dir path --hashclient path --hashserver-address address [--mark value] [--clean-db]
+
+Auxiliary script remove unused or no longer relevant entries from the hashequivalence database, based
+on the files available on the sstate directory.
+
+    -h | --help)               Show this help message and exit
+    -s | --sstate-dir)         Path to the sstate dir
+    -c | --hashclient)         Path to bitbake-hashclient
+    -a | --hashserver-adress)  bitbake-hashserver address
+    -m | --mark)               Marker string to mark database entries
+    --clean-db)                Remove all unmarked and unused entries from the database
+HELP_TEXT
+}
+
+function argument_parser() {
+    while [ $# -gt 0 ]; do
+        case "$1" in
+            -h | --help) help; exit 0 ;;
+            -s | --sstate-dir) SSTATE_DIR="$2"; shift ;;
+            -c | --hashclient) BB_HASHCLIENT="$2"; shift ;;
+            -a | --hashserver-address) BB_HASHSERVER="$2"; shift ;;
+            -m | --mark) ALIVE_DB_MARK="$2"; shift ;;
+            --clean-db) CLEAN_DB="true";;
+            *)
+                echo "Argument '$1' is not supported" >&2
+                help >&2
+                exit 1
+                ;;
+        esac
+        shift
+    done
+
+    function validate_mandatory_argument() {
+        local var_value="$1"
+        local error_message="$2"
+
+        if [ -z "$var_value" ]; then
+            echo "$error_message"
+            help >&2
+            exit 1
+        fi
+    }
+
+    validate_mandatory_argument "$SSTATE_DIR" "Please provide the path to the sstate dir."
+    validate_mandatory_argument "$BB_HASHCLIENT" "Please provide the path to bitbake-hashclient."
+    validate_mandatory_argument "$BB_HASHSERVER" "Please provide the address of bitbake-hashserver."
+}
+
+# -- main code --
+argument_parser $@

scripts/clean-hashserver-database line 72

Quoted 10 lines
scripts/clean-hashserver-database new file +10 -0
+
+# Mark all db sstate hashes
+find "$SSTATE_DIR" -name "*.tar.zst" | \
+sed 's/.*:\([^_]*\)_.*/unihash \1/' | \
+$BB_HASHCLIENT --address "$BB_HASHSERVER" gc-mark-stream "${ALIVE_DB_MARK}"
+
+# Remove unmarked and unused entries
+if [ "$CLEAN_DB" = "true" ]; then
+    $BB_HASHCLIENT --address "$BB_HASHSERVER" gc-sweep "${ALIVE_DB_MARK}"
+    $BB_HASHCLIENT --address "$BB_HASHSERVER" clean-unused 0

The reason for the time is that entries can appear to be unused if they are created while a build is in progress and you don't necessarily want to remove them. Ideally, this is longer than your longest build time. Either way, 0 is probably too aggressive and/or it should be configurable on the command line.

Quoted 13 lines
scripts/clean-hashserver-database new file +1 -1
+fi
--

2.34.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#212644): https://lists.openembedded.org/g/openembedded-core/message/212644
Mute This Topic: https://lists.openembedded.org/mt/111657664/3616693
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw
Quoted 114 lines · Joshua Watt
Quoted 109 lines · Alexandre Marques via lists.openembedded.org

From: Alexandre Marques <[email protected]>

Auxiliary script to clean the hashserver database based on the files available in the sstate directory.

It makes used of the new "hashclient gc-mark-stream" command to mark all sstate
relevant hashes as "alive" and removes everything else from the database.

Usage example:

```
./scripts/clean-hashserver-database \
--sstate-dir ~/build/sstate-cache \
--hashclient ./bitbake/bin/bitabke-hashclient \
--hashserver-address "ws://localhost:8688/ws" \
--mark "alive" \
--clean-db
```

---
scripts/clean-hashserver-database | 73 +++++++++++++++++++++++++++++++

1 file changed, 73 insertions(+)

create mode 100755 scripts/clean-hashserver-database

scripts/clean-hashserver-database new file +13 -0
new file mode 100755
index 0000000000..6eb006758e
@@ -0,0 +1,73 @@
+#!/bin/bash
+set -euo pipefail
+
+SSTATE_DIR=""
+BB_HASHCLIENT=""
+BB_HASHSERVER=""
+
+ALIVE_DB_MARK="alive"
+CLEAN_DB="false"
+
+function help() {
+ cat <<HELP_TEXT
+Usage: $0 --sstate-dir path --hashclient path --hashserver-address

address [--mark value] [--clean-db]

scripts/clean-hashserver-database new file +2 -0
+
+Auxiliary script remove unused or no longer relevant entries from the

hashequivalence database, based

scripts/clean-hashserver-database new file +41 -0
+on the files available on the sstate directory.
+
+ -h | --help) Show this help message and exit
+ -s | --sstate-dir) Path to the sstate dir
+ -c | --hashclient) Path to bitbake-hashclient
+ -a | --hashserver-adress) bitbake-hashserver address
+ -m | --mark) Marker string to mark database entries
+ --clean-db) Remove all unmarked and unused entries from the database
+HELP_TEXT
+}
+
+function argument_parser() {
+ while [ $# -gt 0 ]; do
+ case "$1" in
+ -h | --help) help; exit 0 ;;
+ -s | --sstate-dir) SSTATE_DIR="$2"; shift ;;
+ -c | --hashclient) BB_HASHCLIENT="$2"; shift ;;
+ -a | --hashserver-address) BB_HASHSERVER="$2"; shift ;;
+ -m | --mark) ALIVE_DB_MARK="$2"; shift ;;
+ --clean-db) CLEAN_DB="true";;
+ *)
+ echo "Argument '$1' is not supported" >&2
+ help >&2
+ exit 1
+ ;;
+ esac
+ shift
+ done
+
+ function validate_mandatory_argument() {
+ local var_value="$1"
+ local error_message="$2"
+
+ if [ -z "$var_value" ]; then
+ echo "$error_message"
+ help >&2
+ exit 1
+ fi
+ }
+
+ validate_mandatory_argument "$SSTATE_DIR" "Please provide the path to

the sstate dir."
+ validate_mandatory_argument "$BB_HASHCLIENT" "Please provide the path to bitbake-hashclient."
+ validate_mandatory_argument "$BB_HASHSERVER" "Please provide the address of bitbake-hashserver."

scripts/clean-hashserver-database new file +9 -0
+}
+
+# -- main code --
+argument_parser $@
+
+# Mark all db sstate hashes
+find "$SSTATE_DIR" -name "*.tar.zst" | \
+sed 's/.*:\([^_]*\)_.*/unihash \1/' | \
+$BB_HASHCLIENT --address "$BB_HASHSERVER" gc-mark-stream

"${ALIVE_DB_MARK}"

scripts/clean-hashserver-database new file +3 -0
+
+# Remove unmarked and unused entries
+if [ "$CLEAN_DB" = "true" ]; then
Quoted 8 lines
Quoted 2 lines

+ $BB_HASHCLIENT --address "$BB_HASHSERVER" gc-sweep "${ALIVE_DB_MARK}"
+ $BB_HASHCLIENT --address "$BB_HASHSERVER" clean-unused 0

The reason for the time is that entries can appear to be unused if they are created while a build is in progress and you don't necessarily want to remove them. Ideally, this is longer than your longest build time. Either way, 0 is probably too aggressive and/or it should be configurable on the command line.

Makes sense. Thanks for the review.
Pushed a new version in the meanwhile.

Quoted 11 lines
Quoted 9 lines

+fi
--
2.34.1

#3 lore raw
Quoted 128 lines · Alexandre Marques via lists.openembedded.org
Quoted 117 lines · Joshua Watt

On Wed, Mar 12, 2025 at 5:24 AM Alexandre Marques via lists.openembedded.org <[email protected]>

Quoted 113 lines · wrote:

From: Alexandre Marques <[email protected]>

Auxiliary script to clean the hashserver database based on the files
available in the sstate directory.

It makes used of the new "hashclient gc-mark-stream" command to mark all sstate
relevant hashes as "alive" and removes everything else from the database.

Usage example:

```
./scripts/clean-hashserver-database \
--sstate-dir ~/build/sstate-cache \
--hashclient ./bitbake/bin/bitabke-hashclient \
--hashserver-address "ws://localhost:8688/ws" \
--mark "alive" \
--clean-db
```

---
scripts/clean-hashserver-database | 73
+++++++++++++++++++++++++++++++

1 file changed, 73 insertions(+)

create mode 100755 scripts/clean-hashserver-database

scripts/clean- renamed from scripts/clean-hashserver-database +0 -0

hashserver-database
new file mode 100755
index 0000000000..6eb006758e

scripts/clean-hashserver-database new file +13 -0
@@ -0,0 +1,73 @@
+#!/bin/bash
+set -euo pipefail
+
+SSTATE_DIR=""
+BB_HASHCLIENT=""
+BB_HASHSERVER=""
+
+ALIVE_DB_MARK="alive"
+CLEAN_DB="false"
+
+function help() {
+ cat <<HELP_TEXT
+Usage: $0 --sstate-dir path --hashclient path --hashserver-

address address [--mark value] [--clean-db]

scripts/clean-hashserver-database new file +2 -0
+
+Auxiliary script remove unused or no longer relevant entries

from the hashequivalence database, based

scripts/clean-hashserver-database new file +8 -0
+on the files available on the sstate directory.
+
+ -h | --help) Show this help message and exit
+ -s | --sstate-dir) Path to the sstate dir
+ -c | --hashclient) Path to bitbake-hashclient
+ -a | --hashserver-adress) bitbake-hashserver address
+ -m | --mark) Marker string to mark database entries
+ --clean-db) Remove all unmarked and unused entries from the

database

scripts/clean-hashserver-database new file +33 -0
+HELP_TEXT
+}
+
+function argument_parser() {
+ while [ $# -gt 0 ]; do
+ case "$1" in
+ -h | --help) help; exit 0 ;;
+ -s | --sstate-dir) SSTATE_DIR="$2"; shift ;;
+ -c | --hashclient) BB_HASHCLIENT="$2"; shift ;;
+ -a | --hashserver-address) BB_HASHSERVER="$2"; shift ;;
+ -m | --mark) ALIVE_DB_MARK="$2"; shift ;;
+ --clean-db) CLEAN_DB="true";;
+ *)
+ echo "Argument '$1' is not supported" >&2
+ help >&2
+ exit 1
+ ;;
+ esac
+ shift
+ done
+
+ function validate_mandatory_argument() {
+ local var_value="$1"
+ local error_message="$2"
+
+ if [ -z "$var_value" ]; then
+ echo "$error_message"
+ help >&2
+ exit 1
+ fi
+ }
+
+ validate_mandatory_argument "$SSTATE_DIR" "Please provide the

path to the sstate dir."
+ validate_mandatory_argument "$BB_HASHCLIENT" "Please provide the path to bitbake-hashclient."
+ validate_mandatory_argument "$BB_HASHSERVER" "Please provide the address of bitbake-hashserver."

scripts/clean-hashserver-database new file +9 -0
+}
+
+# -- main code --
+argument_parser $@
+
+# Mark all db sstate hashes
+find "$SSTATE_DIR" -name "*.tar.zst" | \
+sed 's/.*:\([^_]*\)_.*/unihash \1/' | \
+$BB_HASHCLIENT --address "$BB_HASHSERVER" gc-mark-stream

"${ALIVE_DB_MARK}"

scripts/clean-hashserver-database new file +4 -0
+
+# Remove unmarked and unused entries
+if [ "$CLEAN_DB" = "true" ]; then
+ $BB_HASHCLIENT --address "$BB_HASHSERVER" gc-sweep
Quoted 8 lines
Quoted 2 lines

"${ALIVE_DB_MARK}"
+ $BB_HASHCLIENT --address "$BB_HASHSERVER" clean-unused 0

The reason for the time is that entries can appear to be unused if they are created while a build is in progress and you don't necessarily want to remove them. Ideally, this is longer than your longest build time. Either way, 0 is probably too aggressive and/or it
should be configurable on the command line.

Quoted 3 lines

Makes sense. Thanks for the review.
Pushed a new version in the meanwhile.

Was the new version meant to make the 0 configurable?

Cheers,

Richard

#4 lore raw
Quoted 136 lines · Richard Purdie
Quoted 132 lines · Alexandre Marques via lists.openembedded.org
Quoted 127 lines · Joshua Watt

On Wed, Mar 12, 2025 at 5:24 AM Alexandre Marques via lists.openembedded.org <[email protected]>

Quoted 113 lines · wrote:

From: Alexandre Marques <[email protected]>

Auxiliary script to clean the hashserver database based on the files
available in the sstate directory.

It makes used of the new "hashclient gc-mark-stream" command to mark all sstate
relevant hashes as "alive" and removes everything else from the database.

Usage example:

```
./scripts/clean-hashserver-database \
--sstate-dir ~/build/sstate-cache \
--hashclient ./bitbake/bin/bitabke-hashclient \
--hashserver-address "ws://localhost:8688/ws" \
--mark "alive" \
--clean-db
```

---
scripts/clean-hashserver-database | 73
+++++++++++++++++++++++++++++++

1 file changed, 73 insertions(+)

create mode 100755 scripts/clean-hashserver-database

scripts/clean- renamed from scripts/clean-hashserver-database +0 -0

hashserver-database
new file mode 100755
index 0000000000..6eb006758e

scripts/clean-hashserver-database new file +13 -0
@@ -0,0 +1,73 @@
+#!/bin/bash
+set -euo pipefail
+
+SSTATE_DIR=""
+BB_HASHCLIENT=""
+BB_HASHSERVER=""
+
+ALIVE_DB_MARK="alive"
+CLEAN_DB="false"
+
+function help() {
+ cat <<HELP_TEXT
+Usage: $0 --sstate-dir path --hashclient path --hashserver-

address address [--mark value] [--clean-db]

scripts/clean-hashserver-database new file +2 -0
+
+Auxiliary script remove unused or no longer relevant entries

from the hashequivalence database, based

scripts/clean-hashserver-database new file +8 -0
+on the files available on the sstate directory.
+
+ -h | --help) Show this help message and exit
+ -s | --sstate-dir) Path to the sstate dir
+ -c | --hashclient) Path to bitbake-hashclient
+ -a | --hashserver-adress) bitbake-hashserver address
+ -m | --mark) Marker string to mark database entries
+ --clean-db) Remove all unmarked and unused entries from the

database

scripts/clean-hashserver-database new file +33 -0
+HELP_TEXT
+}
+
+function argument_parser() {
+ while [ $# -gt 0 ]; do
+ case "$1" in
+ -h | --help) help; exit 0 ;;
+ -s | --sstate-dir) SSTATE_DIR="$2"; shift ;;
+ -c | --hashclient) BB_HASHCLIENT="$2"; shift ;;
+ -a | --hashserver-address) BB_HASHSERVER="$2"; shift ;;
+ -m | --mark) ALIVE_DB_MARK="$2"; shift ;;
+ --clean-db) CLEAN_DB="true";;
+ *)
+ echo "Argument '$1' is not supported" >&2
+ help >&2
+ exit 1
+ ;;
+ esac
+ shift
+ done
+
+ function validate_mandatory_argument() {
+ local var_value="$1"
+ local error_message="$2"
+
+ if [ -z "$var_value" ]; then
+ echo "$error_message"
+ help >&2
+ exit 1
+ fi
+ }
+
+ validate_mandatory_argument "$SSTATE_DIR" "Please provide the

path to the sstate dir."
+ validate_mandatory_argument "$BB_HASHCLIENT" "Please provide the path to bitbake-hashclient."
+ validate_mandatory_argument "$BB_HASHSERVER" "Please provide the address of bitbake-hashserver."

scripts/clean-hashserver-database new file +9 -0
+}
+
+# -- main code --
+argument_parser $@
+
+# Mark all db sstate hashes
+find "$SSTATE_DIR" -name "*.tar.zst" | \
+sed 's/.*:\([^_]*\)_.*/unihash \1/' | \
+$BB_HASHCLIENT --address "$BB_HASHSERVER" gc-mark-stream

"${ALIVE_DB_MARK}"

scripts/clean-hashserver-database new file +4 -0
+
+# Remove unmarked and unused entries
+if [ "$CLEAN_DB" = "true" ]; then
+ $BB_HASHCLIENT --address "$BB_HASHSERVER" gc-sweep
Quoted 2 lines

"${ALIVE_DB_MARK}"
+ $BB_HASHCLIENT --address "$BB_HASHSERVER" clean-unused 0

The reason for the time is that entries can appear to be unused if they are created while a build is in progress and you don't necessarily want to remove them. Ideally, this is longer than your longest build time. Either way, 0 is probably too aggressive and/or it
should be configurable on the command line.

Quoted 8 lines
Quoted 2 lines

Makes sense. Thanks for the review.
Pushed a new version in the meanwhile.

Was the new version meant to make the 0 configurable?

Cheers,

Richard

It was... but seems I amended the commit without adding the changes.. :,)

Sent a new version in the meanwhile, thanks for the patience and for pointing it out.