Skip to contentSkip to conversation

introduce QArray

PATCH Superseded Christian Schoenebeck 2 participants 9 messages Aug 21, 2021 – Aug 22, 2021

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover introduce QArray

Patches 1 and 2 introduce include/qemu/qarray.h which implements a deep auto free mechanism for arrays. Unlike GArray it does not require special macros, function calls or member dereferencing to access the individual array elements. So existing C-style array code can be retained with only very little changes.

In this initial version QArray only supports the concept of unique pointers, i.e. it does not support reference counting. The array (and all dynamically allocated memory of individual array elements) is auto freed once execution leaves the scope of the reference variable (unique pointer) associated with the array.

Patches 3..5 are provided (e.g. as example) for 9p being the first user of this new QArray API. These particular patches 3..5 are rebased on my current 9p queue: https://github.com/cschoenebeck/qemu/commits/9p.next
which are basically just the following two queued patches:

https://github.com/cschoenebeck/qemu/commit/7772715d43908235940f5b7dec68d0458b1ccdf4
https://github.com/cschoenebeck/qemu/commit/838b55e392ea7d52e714fdba1db777f658aee2cc

Christian Schoenebeck (5):
qemu/qarray.h: introduce QArray
qemu/qarray.h: weak scalar type check in QARRAY_CREATE() 9pfs: make V9fsString usable via QArray API 9pfs: make V9fsPath usable via QArray API
9pfs: use QArray in v9fs_walk()

 fsdev/9p-marshal.c    |   2 +
 fsdev/9p-marshal.h    |   3 +
 fsdev/file-op-9p.h    |   2 +
 hw/9pfs/9p.c          |  19 ++----
 include/qemu/qarray.h | 154 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 168 insertions(+), 12 deletions(-)
 create mode 100644 include/qemu/qarray.h
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/5 qemu/qarray.h: introduce QArray

Implements deep auto free of arrays while retaining common C-style squared bracket access.

---
 include/qemu/qarray.h | 148 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100644 include/qemu/qarray.h
include/qemu/qarray.h new file +148 -0
new file mode 100644
index 0000000000..230a556e81
@@ -0,0 +1,148 @@
+/*
+ * QArray - deep auto free C-array
+ *
+ * Copyright (c) 2021 Crudebyte
+ *
+ * Authors:
+ *   Christian Schoenebeck <[email protected]>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+#ifndef QEMU_QARRAY_H
+#define QEMU_QARRAY_H
+
+/**
+ * QArray provides a mechanism to access arrays in common C-style (e.g. by
+ * square bracket [] operator) in conjunction with reference variables that
+ * perform deep auto free of the array when leaving the scope of the auto
+ * reference variable. That means not only is the array itself automatically
+ * freed, but also memory dynamically allocated by the individual array
+ * elements.
+ *
+ * Example:
+ *
+ * Consider the following user struct @c Foo which shall be used as scalar
+ * (element) type of an array:
+ * @code
+ * typedef struct Foo {
+ *     int i;
+ *     char *s;
+ * } Foo;
+ * @endcode
+ * and assume it has the following function to free memory allocated by @c Foo
+ * instances:
+ * @code
+ * void free_foo(Foo *foo) {
+ *     free(foo->s);
+ * }
+ * @endcode
+ * Add the following to a shared header file:
+ * @code
+ * DECLARE_QARRAY_TYPE(Foo);
+ * @endcode
+ * and the following to a C unit file:
+ * @code
+ * DEFINE_QARRAY_TYPE(Foo, free_foo);
+ * @endcode
+ * Finally the array may then be used like this:
+ * @code
+ * void doSomething(int n) {
+ *     QArrayRef(Foo) foos = NULL;
+ *     QARRAY_CREATE(Foo, foos, n);
+ *     for (size_t i = 0; i < n; ++i) {
+ *         foos[i].i = i;
+ *         foos[i].s = calloc(4096, 1);
+ *         snprintf(foos[i].s, 4096, "foo %d", i);
+ *     }
+ * }
+ * @endcode
+ */
+
+/**
+ * Declares an array for the passed @a scalar_type.
+ *
+ * This is typically used from a shared header file.
+ *
+ * @param scalar_type - type of the individual array elements
+ */
+#define DECLARE_QARRAY_TYPE(scalar_type) \
+    typedef struct QArray##scalar_type { \
+        size_t len; \
+        scalar_type first[]; \
+    } QArray##scalar_type; \
+    \
+    void qarray_create_##scalar_type(scalar_type **auto_var, size_t len); \
+    void qarray_auto_free_##scalar_type(scalar_type **auto_var); \
+
+/**
+ * Defines an array for the passed @a scalar_type and appropriate
+ * @a scalar_cleanup_func.
+ *
+ * @param scalar_type - type of the individual array elements
+ * @param scalar_cleanup_func - appropriate function to free memory dynamically
+ *                              allocated by individual array elements before
+ */
+#define DEFINE_QARRAY_TYPE(scalar_type, scalar_cleanup_func) \
+    void qarray_create_##scalar_type(scalar_type **auto_var, size_t len) \
+    { \
+        qarray_auto_free_##scalar_type(auto_var); \
+        QArray##scalar_type *arr = g_malloc0(sizeof(QArray##scalar_type) + \
+            len * sizeof(scalar_type)); \
+        arr->len = len; \
+        *auto_var = &arr->first[0]; \
+    } \
+    \
+    void qarray_auto_free_##scalar_type(scalar_type **auto_var) \
+    { \
+        scalar_type *first = (*auto_var); \
+        if (!first) { \
+            return; \
+        } \
+        QArray##scalar_type *arr = (QArray##scalar_type *) ( \
+            ((char *)first) - offsetof(QArray##scalar_type, first) \
+        ); \
+        for (size_t i = 0; i < arr->len; ++i) { \
+            scalar_cleanup_func(&arr->first[i]); \
+        } \
+        g_free(arr); \
+    } \
+
+/**
+ * Used to declare a reference variable (unique pointer) for an array. After
+ * leaving the scope of the reference variable, the associated array is
+ * automatically freed.
+ *
+ * @param scalar_type - type of the individual array elements
+ */
+#define QArrayRef(scalar_type) \
+    __attribute((__cleanup__(qarray_auto_free_##scalar_type))) scalar_type*
+
+/**
+ * Allocates a new array of passed @a scalar_type with @a len number of array
+ * elements and assigns the created array to the reference variable
+ * @a auto_var.
+ *
+ * @param scalar_type - type of the individual array elements
+ * @param auto_var - destination reference variable
+ * @param len - amount of array elements to be allocated immediately
+ */
+#define QARRAY_CREATE(scalar_type, auto_var, len) \
+    qarray_create_##scalar_type((&auto_var), len)
+
+#endif /* QEMU_QARRAY_H */
Signature · 1 line
2.20.1
#3 lore raw

PATCH 2/5 qemu/qarray.h: weak scalar type check in QARRAY_CREATE()

In reply to Christian Schoenebeck

Unfortunately something like

  _Static_assert(typeof(a) == typeof(b), "type mismatch");

is currently not suported by C. So for the time being at least check that the size of the scalar types match at compile time.

---
 include/qemu/qarray.h | 6 ++++++
 1 file changed, 6 insertions(+)
include/qemu/qarray.h +6 -0
index 230a556e81..2cb8656c5d 100644
@@ -27,6 +27,8 @@
 #ifndef QEMU_QARRAY_H
 #define QEMU_QARRAY_H
 
+#include "qemu/compiler.h"
+
 /**
  * QArray provides a mechanism to access arrays in common C-style (e.g. by
  * square bracket [] operator) in conjunction with reference variables that
@@ -143,6 +145,10 @@
  * @param len - amount of array elements to be allocated immediately
  */
 #define QARRAY_CREATE(scalar_type, auto_var, len) \
+    QEMU_BUILD_BUG_MSG( \
+        sizeof(scalar_type) != sizeof(*auto_var), \
+        "QArray scalar type mismatch" \
+    ); \
     qarray_create_##scalar_type((&auto_var), len)
 
 #endif /* QEMU_QARRAY_H */
Signature · 1 line
2.20.1
#6 lore raw

PATCH 3/5 9pfs: make V9fsString usable via QArray API

In reply to Christian Schoenebeck
---
 fsdev/9p-marshal.c | 2 ++
 fsdev/9p-marshal.h | 3 +++
 2 files changed, 5 insertions(+)
fsdev/9p-marshal.c +2 -0
index a01bba6908..fbfc2a62cd 100644
@@ -18,6 +18,8 @@
 
 #include "9p-marshal.h"
 
+DEFINE_QARRAY_TYPE(V9fsString, v9fs_string_free);
+
 void v9fs_string_free(V9fsString *str)
 {
     g_free(str->data);
fsdev/9p-marshal.h +3 -0
index ceaf2f521e..7229e4e617 100644
@@ -1,10 +1,13 @@
 #ifndef QEMU_9P_MARSHAL_H
 #define QEMU_9P_MARSHAL_H
 
+#include "qemu/qarray.h"
+
 typedef struct V9fsString {
     uint16_t size;
     char *data;
 } V9fsString;
+DECLARE_QARRAY_TYPE(V9fsString);
 
 typedef struct V9fsQID {
     uint8_t type;
Signature · 1 line
2.20.1
#7 lore raw

PATCH 4/5 9pfs: make V9fsPath usable via QArray API

In reply to Christian Schoenebeck
---
 fsdev/file-op-9p.h | 2 ++
 hw/9pfs/9p.c       | 2 ++
 2 files changed, 4 insertions(+)
fsdev/file-op-9p.h +2 -0
index 42f677cf38..7630f0e538 100644
@@ -18,6 +18,7 @@
 #include <utime.h>
 #include <sys/vfs.h>
 #include "qemu-fsdev-throttle.h"
+#include "qemu/qarray.h"
 
 #define SM_LOCAL_MODE_BITS    0600
 #define SM_LOCAL_DIR_MODE_BITS    0700
@@ -105,6 +106,7 @@ struct V9fsPath {
     uint16_t size;
     char *data;
 };
+DECLARE_QARRAY_TYPE(V9fsPath);
 
 typedef union V9fsFidOpenState V9fsFidOpenState;
 
hw/9pfs/9p.c +2 -0
index c857b31321..b59572fa79 100644
@@ -50,6 +50,8 @@ enum {
     Oappend = 0x80,
 };
 
+DEFINE_QARRAY_TYPE(V9fsPath, v9fs_path_free);
+
 static ssize_t pdu_marshal(V9fsPDU *pdu, size_t offset, const char *fmt, ...)
 {
     ssize_t ret;
Signature · 1 line
2.20.1
#8 lore raw

PATCH 5/5 9pfs: use QArray in v9fs_walk()

In reply to Christian Schoenebeck
---
 hw/9pfs/9p.c | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)
hw/9pfs/9p.c +5 -12
index b59572fa79..91062ee4d6 100644
@@ -1707,13 +1707,14 @@ static void coroutine_fn v9fs_walk(void *opaque)
     int name_idx;
     g_autofree V9fsQID *qids = NULL;
     int i, err = 0;
-    V9fsPath dpath, path, *pathes = NULL;
+    V9fsPath dpath, path;
+    QArrayRef(V9fsPath) pathes = NULL;
     uint16_t nwnames;
     struct stat stbuf, fidst;
     g_autofree struct stat *stbufs = NULL;
     size_t offset = 7;
     int32_t fid, newfid;
-    V9fsString *wnames = NULL;
+    QArrayRef(V9fsString) wnames = NULL;
     V9fsFidState *fidp;
     V9fsFidState *newfidp = NULL;
     V9fsPDU *pdu = opaque;
@@ -1734,10 +1735,10 @@ static void coroutine_fn v9fs_walk(void *opaque)
         goto out_nofid;
     }
     if (nwnames) {
-        wnames = g_new0(V9fsString, nwnames);
+        QARRAY_CREATE(V9fsString, wnames, nwnames);
         qids   = g_new0(V9fsQID, nwnames);
         stbufs = g_new0(struct stat, nwnames);
-        pathes = g_new0(V9fsPath, nwnames);
+        QARRAY_CREATE(V9fsPath, pathes, nwnames);
         for (i = 0; i < nwnames; i++) {
             err = pdu_unmarshal(pdu, offset, "s", &wnames[i]);
             if (err < 0) {
@@ -1869,14 +1870,6 @@ out:
     v9fs_path_free(&path);
 out_nofid:
     pdu_complete(pdu, err);
-    if (nwnames && nwnames <= P9_MAXWELEM) {
-        for (name_idx = 0; name_idx < nwnames; name_idx++) {
-            v9fs_string_free(&wnames[name_idx]);
-            v9fs_path_free(&pathes[name_idx]);
-        }
-        g_free(wnames);
-        g_free(pathes);
-    }
 }
 
 static int32_t coroutine_fn get_iounit(V9fsPDU *pdu, V9fsPath *path)
Signature · 1 line
2.20.1
#4 lore raw

Re: [PATCH 2/5] qemu/qarray.h: weak scalar type check in QARRAY_CREATE()

In reply to Christian Schoenebeck
Quoted 6 lines · Christian Schoenebeck

Unfortunately something like

   _Static_assert(typeof(a) == typeof(b), "type mismatch");

is currently not suported by C. So for the time being at least check that the size of the scalar types match at compile time.

Did you try _Static_assert(__builtin_types_compatible_p(X, Y), "type mismatch");

r~

Quoted 31 lines
---
  include/qemu/qarray.h | 6 ++++++
  1 file changed, 6 insertions(+)
include/qemu/qarray.h +6 -0
index 230a556e81..2cb8656c5d 100644
@@ -27,6 +27,8 @@
  #ifndef QEMU_QARRAY_H
  #define QEMU_QARRAY_H
 
+#include "qemu/compiler.h"
+
  /**
   * QArray provides a mechanism to access arrays in common C-style (e.g. by
   * square bracket [] operator) in conjunction with reference variables that
@@ -143,6 +145,10 @@
   * @param len - amount of array elements to be allocated immediately
   */
  #define QARRAY_CREATE(scalar_type, auto_var, len) \
+    QEMU_BUILD_BUG_MSG( \
+        sizeof(scalar_type) != sizeof(*auto_var), \
+        "QArray scalar type mismatch" \
+    ); \
      qarray_create_##scalar_type((&auto_var), len)
 
  #endif /* QEMU_QARRAY_H */
#5 lore raw

Re: [PATCH 2/5] qemu/qarray.h: weak scalar type check in QARRAY_CREATE()

Quoted 5 lines · On Sonntag, 22. August 2021 06:11:58 CEST Richard Henderson wrote:
Quoted 4 lines · Christian Schoenebeck

Unfortunately something like

   _Static_assert(typeof(a) == typeof(b), "type mismatch");
Quoted 8 lines
Quoted 2 lines

is currently not suported by C. So for the time being at least check that the size of the scalar types match at compile time.

Did you try _Static_assert(__builtin_types_compatible_p(X, Y), "type mismatch");

r~

Ah, you are right. I was trying it, but now as you pointed me at it again, I realized I was just missing something. The specific use case here is like:

struct Foo {
...
} Foo;

Foo *var;

_Static_assert(__builtin_types_compatible_p(Foo, typeof(*var)),

               "type mismatch");

So I was missing the typeof() keyword to deduce the scalar type from the passed variable.

I'll send a v2.

Thanks!

Best regards,
Christian Schoenebeck

#2 lore raw

Re: [PATCH 1/5] qemu/qarray.h: introduce QArray

In reply to Christian Schoenebeck
Quoted 5 lines · On Samstag, 21. August 2021 22:18:18 CEST Christian Schoenebeck wrote:

Implements deep auto free of arrays while retaining common C-style squared bracket access.

---

As I'm going to send a v2 anyway, I'll also just do some minor API comment changes in this patch, specifically ...

Quoted 86 lines
 include/qemu/qarray.h | 148 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100644 include/qemu/qarray.h
include/qemu/qarray.h new file +9 -0
new file mode 100644
index 0000000000..230a556e81
@@ -0,0 +1,148 @@
+/*
+ * QArray - deep auto free C-array
+ *
+ * Copyright (c) 2021 Crudebyte
+ *
+ * Authors:
+ *   Christian Schoenebeck <[email protected]>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a

copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + * THE SOFTWARE.
+ */

include/qemu/qarray.h new file +8 -0
+#ifndef QEMU_QARRAY_H
+#define QEMU_QARRAY_H
+
+/**
+ * QArray provides a mechanism to access arrays in common C-style (e.g. by
+ * square bracket [] operator) in conjunction with reference variables that
+ * perform deep auto free of the array when leaving the scope of the auto
+ * reference variable. That means not only is the array itself

automatically + * freed, but also memory dynamically allocated by the individual array + * elements.
+ *
+ * Example:
+ *
+ * Consider the following user struct @c Foo which shall be used as scalar
+ * (element) type of an array:
+ * @code
+ * typedef struct Foo {
+ * int i;
+ * char *s;
+ * } Foo;
+ * @endcode
+ * and assume it has the following function to free memory allocated by @c Foo + * instances:
+ * @code
+ * void free_foo(Foo *foo) {
+ * free(foo->s);
+ * }
+ * @endcode
+ * Add the following to a shared header file:
+ * @code
+ * DECLARE_QARRAY_TYPE(Foo);
+ * @endcode
+ * and the following to a C unit file:
+ * @code
+ * DEFINE_QARRAY_TYPE(Foo, free_foo);
+ * @endcode
+ * Finally the array may then be used like this:
+ * @code
+ * void doSomething(int n) {

+ *     QArrayRef(Foo) foos = NULL;
+ *     QARRAY_CREATE(Foo, foos, n);
+ *     for (size_t i = 0; i < n; ++i) {
+ *         foos[i].i = i;
+ *         foos[i].s = calloc(4096, 1);
+ *         snprintf(foos[i].s, 4096, "foo %d", i);

+ * }
+ * }
+ * @endcode
+ */

include/qemu/qarray.h line 78

Quoted 3 lines
include/qemu/qarray.h new file +3 -0
+
+/**
+ * Declares an array for the passed @a scalar_type.

"Declares an array *type* ...".

To not confuse it with declaring an array instance/variable.

Quoted 6 lines

+ *
+ * This is typically used from a shared header file.
+ *
+ * @param scalar_type - type of the individual array elements
+ */

include/qemu/qarray.h new file +1 -0
+#define DECLARE_QARRAY_TYPE(scalar_type) \

include/qemu/qarray.h line 95

Quoted 10 lines
include/qemu/qarray.h new file +10 -0
+    typedef struct QArray##scalar_type { \
+        size_t len; \
+        scalar_type first[]; \
+    } QArray##scalar_type; \
+    \
+    void qarray_create_##scalar_type(scalar_type **auto_var, size_t len); \
+    void qarray_auto_free_##scalar_type(scalar_type **auto_var); \ +
+/**
+ * Defines an array for the passed @a scalar_type and appropriate
+ * @a scalar_cleanup_func.

Likewise "Defines an array *type* ...".

Plus missing:

"This is typically used from a C unit file."

Quoted 49 lines

+ *
+ * @param scalar_type - type of the individual array elements
+ * @param scalar_cleanup_func - appropriate function to free memory
dynamically + * allocated by individual array
elements before + */

include/qemu/qarray.h new file +5 -0
+#define DEFINE_QARRAY_TYPE(scalar_type, scalar_cleanup_func) \
+    void qarray_create_##scalar_type(scalar_type **auto_var, size_t len) \
+    { \
+        qarray_auto_free_##scalar_type(auto_var); \
+        QArray##scalar_type *arr = g_malloc0(sizeof(QArray##scalar_type) +

\ + len * sizeof(scalar_type)); \

include/qemu/qarray.h new file +21 -0
+        arr->len = len; \
+        *auto_var = &arr->first[0]; \
+    } \
+    \
+    void qarray_auto_free_##scalar_type(scalar_type **auto_var) \
+    { \
+        scalar_type *first = (*auto_var); \
+        if (!first) { \
+            return; \
+        } \
+        QArray##scalar_type *arr = (QArray##scalar_type *) ( \
+            ((char *)first) - offsetof(QArray##scalar_type, first) \
+        ); \
+        for (size_t i = 0; i < arr->len; ++i) { \
+            scalar_cleanup_func(&arr->first[i]); \
+        } \
+        g_free(arr); \
+    } \
+
+/**
+ * Used to declare a reference variable (unique pointer) for an array.

After + * leaving the scope of the reference variable, the associated array is + * automatically freed.
+ *
+ * @param scalar_type - type of the individual array elements
+ */

include/qemu/qarray.h new file +5 -0
+#define QArrayRef(scalar_type) \
+    __attribute((__cleanup__(qarray_auto_free_##scalar_type))) scalar_type*
+
+/**
+ * Allocates a new array of passed @a scalar_type with @a len number of

array + * elements and assigns the created array to the reference variable
+ * @a auto_var.
+ *
+ * @param scalar_type - type of the individual array elements
+ * @param auto_var - destination reference variable
+ * @param len - amount of array elements to be allocated immediately
+ */

include/qemu/qarray.h line 148

Quoted 4 lines
include/qemu/qarray.h new file +4 -0
+#define QARRAY_CREATE(scalar_type, auto_var, len) \
+    qarray_create_##scalar_type((&auto_var), len)
+
+#endif /* QEMU_QARRAY_H */

Best regards,
Christian Schoenebeck