Skip to contentSkip to conversation

kunit: introduce KUnit, the Linux kernel unit testing framework

PATCH Superseded Brendan Higgins 4 participants 57 messages Jul 12, 2019 – Aug 12, 2019

Subsystems (11)
Documentation VFS Kbuild Kselftest +7 more subsystems

v9 of 18 Superseded Latest is v18 →

lore
Share & tools
lore
#0 lore raw

v9 cover kunit: introduce KUnit, the Linux kernel unit testing framework

## TL;DR

This new patch set only contains a very minor change to address a sparse warning in the PROC SYSCTL KUnit test. Otherwise this patchset is identical to the previous.

As I mentioned in the previous patchset, all patches now have acks and reviews.

## Background

This patch set proposes KUnit, a lightweight unit testing and mocking framework for the Linux kernel.

Unlike Autotest and kselftest, KUnit is a true unit testing framework; it does not require installing the kernel on a test machine or in a VM (however, KUnit still allows you to run tests on test machines or in VMs if you want[1]) and does not require tests to be written in userspace running on a host kernel. Additionally, KUnit is fast: From invocation to completion KUnit can run several dozen tests in about a second. Currently, the entire KUnit test suite for KUnit runs in under a second from the initial invocation (build time excluded).

KUnit is heavily inspired by JUnit, Python's unittest.mock, and Googletest/Googlemock for C++. KUnit provides facilities for defining unit test cases, grouping related test cases into test suites, providing common infrastructure for running tests, mocking, spying, and much more.

### What's so special about unit testing?

A unit test is supposed to test a single unit of code in isolation, hence the name. There should be no dependencies outside the control of the test; this means no external dependencies, which makes tests orders of magnitudes faster. Likewise, since there are no external dependencies, there are no hoops to jump through to run the tests. Additionally, this makes unit tests deterministic: a failing unit test always indicates a problem. Finally, because unit tests necessarily have finer granularity, they are able to test all code paths easily solving the classic problem of difficulty in exercising error handling code.

### Is KUnit trying to replace other testing frameworks for the kernel?

No. Most existing tests for the Linux kernel are end-to-end tests, which have their place. A well tested system has lots of unit tests, a reasonable number of integration tests, and some end-to-end tests. KUnit is just trying to address the unit test space which is currently not being addressed.

### More information on KUnit

There is a bunch of documentation near the end of this patch set that describes how to use KUnit and best practices for writing unit tests. For convenience I am hosting the compiled docs here[2].

Additionally for convenience, I have applied these patches to a branch[3]. The repo may be cloned with:
git clone https://kunit.googlesource.com/linux
This patchset is on the kunit/rfc/v5.2/v9 branch.

## Changes Since Last Version

Like I said in the TL;DR, there is only one minor change since the previous revision. That change only affects patch 17/18; it addresses a sparse warning in the PROC SYSCTL unit test.

Thanks to Masahiro for applying previous patches to a branch in his kbuild tree and running sparse and other static analysis tools against my patches.

[1] https://google.github.io/kunit-docs/third_party/kernel/docs/usage.html#kunit-on-non-uml-architectures
[2] https://google.github.io/kunit-docs/third_party/kernel/docs/
[3] https://kunit.googlesource.com/linux/+/kunit/rfc/v5.2/v9

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#1 lore raw

PATCH v9 1/18 kunit: test: add KUnit test runner core

Add core facilities for defining unit tests; this provides a common way to define test cases, functions that execute code which is under test and determine whether the code under test behaves as expected; this also provides a way to group together related test cases in test suites (here we call them test_modules).

Just define test cases and how to execute them for now; setting expectations on code will be defined later.

---
 include/kunit/test.h | 179 ++++++++++++++++++++++++++++++++++++++++
 kunit/Kconfig        |  17 ++++
 kunit/Makefile       |   1 +
 kunit/test.c         | 189 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 386 insertions(+)
 create mode 100644 include/kunit/test.h
 create mode 100644 kunit/Kconfig
 create mode 100644 kunit/Makefile
 create mode 100644 kunit/test.c
include/kunit/test.h new file +179 -0
new file mode 100644
index 0000000000000..e0b34acb9ee4e
@@ -0,0 +1,179 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Base unit test (KUnit) API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_TEST_H
+#define _KUNIT_TEST_H
+
+#include <linux/types.h>
+
+struct kunit;
+
+/**
+ * struct kunit_case - represents an individual test case.
+ * @run_case: the function representing the actual test case.
+ * @name: the name of the test case.
+ *
+ * A test case is a function with the signature, ``void (*)(struct kunit *)``
+ * that makes expectations (see KUNIT_EXPECT_TRUE()) about code under test. Each
+ * test case is associated with a &struct kunit_suite and will be run after the
+ * suite's init function and followed by the suite's exit function.
+ *
+ * A test case should be static and should only be created with the KUNIT_CASE()
+ * macro; additionally, every array of test cases should be terminated with an
+ * empty test case.
+ *
+ * Example:
+ *
+ * .. code-block:: c
+ *
+ *	void add_test_basic(struct kunit *test)
+ *	{
+ *		KUNIT_EXPECT_EQ(test, 1, add(1, 0));
+ *		KUNIT_EXPECT_EQ(test, 2, add(1, 1));
+ *		KUNIT_EXPECT_EQ(test, 0, add(-1, 1));
+ *		KUNIT_EXPECT_EQ(test, INT_MAX, add(0, INT_MAX));
+ *		KUNIT_EXPECT_EQ(test, -1, add(INT_MAX, INT_MIN));
+ *	}
+ *
+ *	static struct kunit_case example_test_cases[] = {
+ *		KUNIT_CASE(add_test_basic),
+ *		{}
+ *	};
+ *
+ */
+struct kunit_case {
+	void (*run_case)(struct kunit *test);
+	const char *name;
+
+	/* private: internal use only. */
+	bool success;
+};
+
+/**
+ * KUNIT_CASE - A helper for creating a &struct kunit_case
+ * @test_name: a reference to a test case function.
+ *
+ * Takes a symbol for a function representing a test case and creates a
+ * &struct kunit_case object from it. See the documentation for
+ * &struct kunit_case for an example on how to use it.
+ */
+#define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
+
+/**
+ * struct kunit_suite - describes a related collection of &struct kunit_case s.
+ * @name: the name of the test. Purely informational.
+ * @init: called before every test case.
+ * @exit: called after every test case.
+ * @test_cases: a null terminated array of test cases.
+ *
+ * A kunit_suite is a collection of related &struct kunit_case s, such that
+ * @init is called before every test case and @exit is called after every test
+ * case, similar to the notion of a *test fixture* or a *test class* in other
+ * unit testing frameworks like JUnit or Googletest.
+ *
+ * Every &struct kunit_case must be associated with a kunit_suite for KUnit to
+ * run it.
+ */
+struct kunit_suite {
+	const char name[256];
+	int (*init)(struct kunit *test);
+	void (*exit)(struct kunit *test);
+	struct kunit_case *test_cases;
+};
+
+/**
+ * struct kunit - represents a running instance of a test.
+ * @priv: for user to store arbitrary data. Commonly used to pass data created
+ * in the init function (see &struct kunit_suite).
+ *
+ * Used to store information about the current context under which the test is
+ * running. Most of this data is private and should only be accessed indirectly
+ * via public functions; the one exception is @priv which can be used by the
+ * test writer to store arbitrary data.
+ */
+struct kunit {
+	void *priv;
+
+	/* private: internal use only. */
+	const char *name; /* Read only after initialization! */
+	/*
+	 * success starts as true, and may only be set to false during a test
+	 * case; thus, it is safe to update this across multiple threads using
+	 * WRITE_ONCE; however, as a consequence, it may only be read after the
+	 * test case finishes once all threads associated with the test case
+	 * have terminated.
+	 */
+	bool success; /* Read only after test_case finishes! */
+};
+
+void kunit_init_test(struct kunit *test, const char *name);
+
+int kunit_run_tests(struct kunit_suite *suite);
+
+/**
+ * kunit_test_suite() - used to register a &struct kunit_suite with KUnit.
+ * @suite: a statically allocated &struct kunit_suite.
+ *
+ * Registers @suite with the test framework. See &struct kunit_suite for more
+ * information.
+ *
+ * NOTE: Currently KUnit tests are all run as late_initcalls; this means that
+ * they cannot test anything where tests must run at a different init phase. One
+ * significant restriction resulting from this is that KUnit cannot reliably
+ * test anything that is initialize in the late_init phase; another is that
+ * KUnit is useless to test things that need to be run in an earlier init phase.
+ */
+#define kunit_test_suite(suite)						       \
+		/*
+		 * TODO([email protected]): Don't run all KUnit tests as
+		 * late_initcalls.  I have some future work planned to dispatch
+		 * all KUnit tests from the same place, and at the very least to
+		 * do so after everything else is definitely initialized.
+		 */							       \
+		static int kunit_suite_init##suite(void)		       \
+		{							       \
+			return kunit_run_tests(&suite);			       \
+		}							       \
+		late_initcall(kunit_suite_init##suite)
+
+void __printf(3, 4) kunit_printk(const char *level,
+				 const struct kunit *test,
+				 const char *fmt, ...);
+
+/**
+ * kunit_info() - Prints an INFO level message associated with the current test.
+ * @test: The test context object.
+ * @fmt: A printk() style format string.
+ *
+ * Prints an info level message associated with the test suite being run. Takes
+ * a variable number of format parameters just like printk().
+ */
+#define kunit_info(test, fmt, ...) \
+		kunit_printk(KERN_INFO, test, fmt, ##__VA_ARGS__)
+
+/**
+ * kunit_warn() - Prints a WARN level message associated with the current test.
+ * @test: The test context object.
+ * @fmt: A printk() style format string.
+ *
+ * Prints a warning level message.
+ */
+#define kunit_warn(test, fmt, ...) \
+		kunit_printk(KERN_WARNING, test, fmt, ##__VA_ARGS__)
+
+/**
+ * kunit_err() - Prints an ERROR level message associated with the current test.
+ * @test: The test context object.
+ * @fmt: A printk() style format string.
+ *
+ * Prints an error level message.
+ */
+#define kunit_err(test, fmt, ...) \
+		kunit_printk(KERN_ERR, test, fmt, ##__VA_ARGS__)
+
+#endif /* _KUNIT_TEST_H */
kunit/Kconfig new file +17 -0
new file mode 100644
index 0000000000000..330ae83527c23
@@ -0,0 +1,17 @@
+#
+# KUnit base configuration
+#
+
+menu "KUnit support"
+
+config KUNIT
+	bool "Enable support for unit tests (KUnit)"
+	help
+	  Enables support for kernel unit tests (KUnit), a lightweight unit
+	  testing and mocking framework for the Linux kernel. These tests are
+	  able to be run locally on a developer's workstation without a VM or
+	  special hardware when using UML. Can also be used on most other
+	  architectures. For more information, please see
+	  Documentation/dev-tools/kunit/.
+
+endmenu
kunit/Makefile new file +1 -0
new file mode 100644
index 0000000000000..5efdc4dea2c08
@@ -0,0 +1 @@
+obj-$(CONFIG_KUNIT) +=			test.o
kunit/test.c new file +189 -0

189 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
Reviewed-by on v9 1/18
#2 lore raw

Re: [PATCH v9 01/18] kunit: test: add KUnit test runner core

Quoting Brendan Higgins (2019-07-12 01:17:27)

Quoted 6 lines

Add core facilities for defining unit tests; this provides a common way to define test cases, functions that execute code which is under test and determine whether the code under test behaves as expected; this also provides a way to group together related test cases in test suites (here we call them test_modules).

Quoted 7 lines

Just define test cases and how to execute them for now; setting expectations on code will be defined later.

Minor nits below.

Quoted 22 lines
kunit/test.c new file +16 -0
new file mode 100644
index 0000000000000..571e4c65deb5c
@@ -0,0 +1,189 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Base unit test (KUnit) API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#include <linux/kernel.h>
+#include <kunit/test.h>
+
+static void kunit_set_failure(struct kunit *test)
+{
+       WRITE_ONCE(test->success, false);
+}
+

[...]

Quoted 10 lines
kunit/test.c new file +10 -0
+
+void kunit_init_test(struct kunit *test, const char *name)
+{
+       test->name = name;
+       test->success = true;
+}
+
+/*
+ * Performs all logic to run a test case.
+ */

kunit/test.c line 141

Quoted 10 lines
kunit/test.c new file +10 -0
+static void kunit_run_case(struct kunit_suite *suite,
+                          struct kunit_case *test_case)
+{
+       struct kunit test;
+       int ret = 0;
+
+       kunit_init_test(&test, test_case->name);
+
+       if (suite->init) {
+               ret = suite->init(&test);

Can you push the ret definition into this if scope? That way we can avoid default initialize to 0 for it.

kunit/test.c line 144

Quoted 3 lines
kunit/test.c new file +3 -0
+               if (ret) {
+                       kunit_err(&test, "failed to initialize: %d\n", ret);
+                       kunit_set_failure(&test);

Do we need to 'test_case->success = test.success' here too? Or is the test failure extracted somewhere else?

Quoted 10 lines
kunit/test.c new file +10 -0
+                       return;
+               }
+       }
+
+       test_case->run_case(&test);
+
+       if (suite->exit)
+               suite->exit(&test);
+
+       test_case->success = test.success;
#3 lore raw

Re: [PATCH v9 01/18] kunit: test: add KUnit test runner core

Quoted 58 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-12 01:17:27)

Quoted 6 lines

Add core facilities for defining unit tests; this provides a common way to define test cases, functions that execute code which is under test and determine whether the code under test behaves as expected; this also provides a way to group together related test cases in test suites (here we call them test_modules).

Quoted 7 lines

Just define test cases and how to execute them for now; setting expectations on code will be defined later.

Minor nits below.

Quoted 22 lines
kunit/test.c new file +16 -0
new file mode 100644
index 0000000000000..571e4c65deb5c
@@ -0,0 +1,189 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Base unit test (KUnit) API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#include <linux/kernel.h>
+#include <kunit/test.h>
+
+static void kunit_set_failure(struct kunit *test)
+{
+       WRITE_ONCE(test->success, false);
+}
+

[...]

Quoted 15 lines
kunit/test.c new file +15 -0
+
+void kunit_init_test(struct kunit *test, const char *name)
+{
+       test->name = name;
+       test->success = true;
+}
+
+/*
+ * Performs all logic to run a test case.
+ */
+static void kunit_run_case(struct kunit_suite *suite,
+                          struct kunit_case *test_case)
+{
+       struct kunit test;
+       int ret = 0;
Quoted 8 lines

kunit/test.c line 141

Quoted 5 lines
kunit/test.c new file +5 -0
+
+       kunit_init_test(&test, test_case->name);
+
+       if (suite->init) {
+               ret = suite->init(&test);

Can you push the ret definition into this if scope? That way we can avoid default initialize to 0 for it.

Sure! I would actually prefer that from a cosmetic standpoint. I just thought that mixing declarations and code was against the style guide.

Quoted 6 lines

kunit/test.c line 144

Quoted 3 lines
kunit/test.c new file +3 -0
+               if (ret) {
+                       kunit_err(&test, "failed to initialize: %d\n", ret);
+                       kunit_set_failure(&test);

Do we need to 'test_case->success = test.success' here too? Or is the test failure extracted somewhere else?

Er, yes. That's kind of embarrassing. Good catch.

Quoted 10 lines

kunit/test.c line 154

Quoted 10 lines
kunit/test.c new file +10 -0
+                       return;
+               }
+       }
+
+       test_case->run_case(&test);
+
+       if (suite->exit)
+               suite->exit(&test);
+
+       test_case->success = test.success;

Thanks!

#4 lore raw

PATCH v9 2/18 kunit: test: add test resource management API

In reply to Brendan Higgins

Create a common API for test managed resources like memory and test objects. A lot of times a test will want to set up infrastructure to be used in test cases; this could be anything from just wanting to allocate some memory to setting up a driver stack; this defines facilities for creating "test resources" which are managed by the test infrastructure and are automatically cleaned up at the conclusion of the test.

---
 include/kunit/test.h | 116 +++++++++++++++++++++++++++++++++++++++++++
 kunit/test.c         |  94 +++++++++++++++++++++++++++++++++++
 2 files changed, 210 insertions(+)
include/kunit/test.h +116 -0
index e0b34acb9ee4e..bdf41d31c343c 100644
@@ -10,6 +10,70 @@
 #define _KUNIT_TEST_H
 
 #include <linux/types.h>
+#include <linux/slab.h>
+
+struct kunit_resource;
+
+typedef int (*kunit_resource_init_t)(struct kunit_resource *, void *);
+typedef void (*kunit_resource_free_t)(struct kunit_resource *);
+
+/**
+ * struct kunit_resource - represents a *test managed resource*
+ * @allocation: for the user to store arbitrary data.
+ * @free: a user supplied function to free the resource. Populated by
+ * kunit_alloc_resource().
+ *
+ * Represents a *test managed resource*, a resource which will automatically be
+ * cleaned up at the end of a test case.
+ *
+ * Example:
+ *
+ * .. code-block:: c
+ *
+ *	struct kunit_kmalloc_params {
+ *		size_t size;
+ *		gfp_t gfp;
+ *	};
+ *
+ *	static int kunit_kmalloc_init(struct kunit_resource *res, void *context)
+ *	{
+ *		struct kunit_kmalloc_params *params = context;
+ *		res->allocation = kmalloc(params->size, params->gfp);
+ *
+ *		if (!res->allocation)
+ *			return -ENOMEM;
+ *
+ *		return 0;
+ *	}
+ *
+ *	static void kunit_kmalloc_free(struct kunit_resource *res)
+ *	{
+ *		kfree(res->allocation);
+ *	}
+ *
+ *	void *kunit_kmalloc(struct kunit *test, size_t size, gfp_t gfp)
+ *	{
+ *		struct kunit_kmalloc_params params;
+ *		struct kunit_resource *res;
+ *
+ *		params.size = size;
+ *		params.gfp = gfp;
+ *
+ *		res = kunit_alloc_resource(test, kunit_kmalloc_init,
+ *			kunit_kmalloc_free, &params);
+ *		if (res)
+ *			return res->allocation;
+ *
+ *		return NULL;
+ *	}
+ */
+struct kunit_resource {
+	void *allocation;
+	kunit_resource_free_t free;
+
+	/* private: internal use only. */
+	struct list_head node;
+};
 
 struct kunit;
 
@@ -109,6 +173,13 @@ struct kunit {
 	 * have terminated.
 	 */
 	bool success; /* Read only after test_case finishes! */
+	struct mutex lock; /* Gaurds all mutable test state. */
+	/*
+	 * Because resources is a list that may be updated multiple times (with
+	 * new resources) from any thread associated with a test case, we must
+	 * protect it with some type of lock.
+	 */
+	struct list_head resources; /* Protected by lock. */
 };
 
 void kunit_init_test(struct kunit *test, const char *name);
@@ -141,6 +212,51 @@ int kunit_run_tests(struct kunit_suite *suite);
 		}							       \
 		late_initcall(kunit_suite_init##suite)
 
+/**
+ * kunit_alloc_resource() - Allocates a *test managed resource*.
+ * @test: The test context object.
+ * @init: a user supplied function to initialize the resource.
+ * @free: a user supplied function to free the resource.
+ * @context: for the user to pass in arbitrary data to the init function.
+ *
+ * Allocates a *test managed resource*, a resource which will automatically be
+ * cleaned up at the end of a test case. See &struct kunit_resource for an
+ * example.
+ */
+struct kunit_resource *kunit_alloc_resource(struct kunit *test,
+					    kunit_resource_init_t init,
+					    kunit_resource_free_t free,
+					    void *context);
+
+void kunit_free_resource(struct kunit *test, struct kunit_resource *res);
+
+/**
+ * kunit_kmalloc() - Like kmalloc() except the allocation is *test managed*.
+ * @test: The test context object.
+ * @size: The size in bytes of the desired memory.
+ * @gfp: flags passed to underlying kmalloc().
+ *
+ * Just like `kmalloc(...)`, except the allocation is managed by the test case
+ * and is automatically cleaned up after the test case concludes. See &struct
+ * kunit_resource for more information.
+ */
+void *kunit_kmalloc(struct kunit *test, size_t size, gfp_t gfp);
+
+/**
+ * kunit_kzalloc() - Just like kunit_kmalloc(), but zeroes the allocation.
+ * @test: The test context object.
+ * @size: The size in bytes of the desired memory.
+ * @gfp: flags passed to underlying kmalloc().
+ *
+ * See kzalloc() and kunit_kmalloc() for more information.
+ */
+static inline void *kunit_kzalloc(struct kunit *test, size_t size, gfp_t gfp)
+{
+	return kunit_kmalloc(test, size, gfp | __GFP_ZERO);
+}
+
+void kunit_cleanup(struct kunit *test);
+
 void __printf(3, 4) kunit_printk(const char *level,
 				 const struct kunit *test,
 				 const char *fmt, ...);
kunit/test.c +94 -0

112 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#5 lore raw

Re: [PATCH v9 02/18] kunit: test: add test resource management API

Quoting Brendan Higgins (2019-07-12 01:17:28)

Quoted 7 lines
kunit/test.c +0 -0
index 571e4c65deb5c..f165c9d8e10b0 100644
@@ -171,6 +175,96 @@ int kunit_run_tests(struct kunit_suite *suite)
        return 0;
 }

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 186

Quoted 10 lines
kunit/test.c +9 -0
 
+struct kunit_resource *kunit_alloc_resource(struct kunit *test,
+                                           kunit_resource_init_t init,
+                                           kunit_resource_free_t free,
+                                           void *context)
+{
+       struct kunit_resource *res;
+       int ret;
+
+       res = kzalloc(sizeof(*res), GFP_KERNEL);

This uses GFP_KERNEL.

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 195

Quoted 9 lines
kunit/test.c +9 -0
+       if (!res)
+               return NULL;
+
+       ret = init(res, context);
+       if (ret)
+               return NULL;
+
+       res->free = free;
+       mutex_lock(&test->lock);

And this can sleep.

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 202

Quoted 7 lines
kunit/test.c +7 -0
+       list_add_tail(&res->node, &test->resources);
+       mutex_unlock(&test->lock);
+
+       return res;
+}
+
+void kunit_free_resource(struct kunit *test, struct kunit_resource *res)

Should probably add a note that we assume the test lock is held here, or even add a lockdep_assert_held(&test->lock) into the function to document that and assert it at the same time.

Quoted 26 lines
kunit/test.c +26 -0
+{
+       res->free(res);
+       list_del(&res->node);
+       kfree(res);
+}
+
+struct kunit_kmalloc_params {
+       size_t size;
+       gfp_t gfp;
+};
+
+static int kunit_kmalloc_init(struct kunit_resource *res, void *context)
+{
+       struct kunit_kmalloc_params *params = context;
+
+       res->allocation = kmalloc(params->size, params->gfp);
+       if (!res->allocation)
+               return -ENOMEM;
+
+       return 0;
+}
+
+static void kunit_kmalloc_free(struct kunit_resource *res)
+{
+       kfree(res->allocation);
+}

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 238

Quoted 10 lines
kunit/test.c +10 -0
+
+void *kunit_kmalloc(struct kunit *test, size_t size, gfp_t gfp)
+{
+       struct kunit_kmalloc_params params;
+       struct kunit_resource *res;
+
+       params.size = size;
+       params.gfp = gfp;
+
+       res = kunit_alloc_resource(test,

This calls that sleeping function above...

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 241

Quoted 3 lines
kunit/test.c +3 -0
+                                  kunit_kmalloc_init,
+                                  kunit_kmalloc_free,
+                                  &params);

but this passes a GFP flags parameter through to the kunit_kmalloc_init() function. How is this going to work if some code uses GFP_ATOMIC, but then we try to allocate and sleep in kunit_alloc_resource() with GFP_KERNEL?

One solution would be to piggyback on all the existing devres allocation logic we already have and make each struct kunit a device that we pass into the devres functions. A far simpler solution would be to just copy/paste what devres does and use a spinlock and an allocation function that takes GFP flags.

Quoted 6 lines
kunit/test.c +6 -0
+
+       if (res)
+               return res->allocation;
+
+       return NULL;
+}
#6 lore raw

Re: [PATCH v9 02/18] kunit: test: add test resource management API

Quoted 34 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-12 01:17:28)

Quoted 7 lines
kunit/test.c +0 -0
index 571e4c65deb5c..f165c9d8e10b0 100644
@@ -171,6 +175,96 @@ int kunit_run_tests(struct kunit_suite *suite)
        return 0;
 }

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 186

Quoted 10 lines
kunit/test.c +9 -0
 
+struct kunit_resource *kunit_alloc_resource(struct kunit *test,
+                                           kunit_resource_init_t init,
+                                           kunit_resource_free_t free,
+                                           void *context)
+{
+       struct kunit_resource *res;
+       int ret;
+
+       res = kzalloc(sizeof(*res), GFP_KERNEL);

This uses GFP_KERNEL.

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 195

Quoted 9 lines
kunit/test.c +9 -0
+       if (!res)
+               return NULL;
+
+       ret = init(res, context);
+       if (ret)
+               return NULL;
+
+       res->free = free;
+       mutex_lock(&test->lock);

And this can sleep.

Quoted 11 lines

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 202

Quoted 7 lines
kunit/test.c +7 -0
+       list_add_tail(&res->node, &test->resources);
+       mutex_unlock(&test->lock);
+
+       return res;
+}
+
+void kunit_free_resource(struct kunit *test, struct kunit_resource *res)

Should probably add a note that we assume the test lock is held here, or even add a lockdep_assert_held(&test->lock) into the function to document that and assert it at the same time.

Seems reasonable.

Quoted 39 lines
Quoted 26 lines
kunit/test.c +26 -0
+{
+       res->free(res);
+       list_del(&res->node);
+       kfree(res);
+}
+
+struct kunit_kmalloc_params {
+       size_t size;
+       gfp_t gfp;
+};
+
+static int kunit_kmalloc_init(struct kunit_resource *res, void *context)
+{
+       struct kunit_kmalloc_params *params = context;
+
+       res->allocation = kmalloc(params->size, params->gfp);
+       if (!res->allocation)
+               return -ENOMEM;
+
+       return 0;
+}
+
+static void kunit_kmalloc_free(struct kunit_resource *res)
+{
+       kfree(res->allocation);
+}

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 238

Quoted 10 lines
kunit/test.c +10 -0
+
+void *kunit_kmalloc(struct kunit *test, size_t size, gfp_t gfp)
+{
+       struct kunit_kmalloc_params params;
+       struct kunit_resource *res;
+
+       params.size = size;
+       params.gfp = gfp;
+
+       res = kunit_alloc_resource(test,

This calls that sleeping function above...

Quoted 8 lines

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 241

Quoted 3 lines
kunit/test.c +3 -0
+                                  kunit_kmalloc_init,
+                                  kunit_kmalloc_free,
+                                  &params);

but this passes a GFP flags parameter through to the kunit_kmalloc_init() function. How is this going to work if some code uses GFP_ATOMIC, but then we try to allocate and sleep in kunit_alloc_resource() with GFP_KERNEL?

Yeah, that's an inconsistency. I need to fix that.

Quoted 5 lines

One solution would be to piggyback on all the existing devres allocation logic we already have and make each struct kunit a device that we pass into the devres functions. A far simpler solution would be to just copy/paste what devres does and use a spinlock and an allocation function that takes GFP flags.

Yeah, that's what I did originally, but I thought from the discussion on patch 01 that you thought a spinlock was overkill for struct kunit. I take it you only meant in that initial patch?

Quoted 6 lines

kunit/test.c int kunit_run_tests(struct kunit_suite *suite) · line 247

Quoted 6 lines
kunit/test.c +6 -0
+
+       if (res)
+               return res->allocation;
+
+       return NULL;
+}

Cheers

#7 lore raw

Re: [PATCH v9 02/18] kunit: test: add test resource management API

Quoting Brendan Higgins (2019-07-15 13:30:22)

Quoted 12 lines · Stephen Boyd
Quoted 10 lines

Quoting Brendan Higgins (2019-07-12 01:17:28)

kunit/test.c

Quoted 2 lines
kunit/test.c +0 -0
index 571e4c65deb5c..f165c9d8e10b0 100644

One solution would be to piggyback on all the existing devres allocation logic we already have and make each struct kunit a device that we pass into the devres functions. A far simpler solution would be to just copy/paste what devres does and use a spinlock and an allocation function that takes GFP flags.

Quoted 3 lines

Yeah, that's what I did originally, but I thought from the discussion on patch 01 that you thought a spinlock was overkill for struct kunit. I take it you only meant in that initial patch?

Correct. I was only talking about the success bit in there.

#8 lore raw

PATCH v9 3/18 kunit: test: add string_stream a std::stream like string builder

In reply to Brendan Higgins

A number of test features need to do pretty complicated string printing where it may not be possible to rely on a single preallocated string with parameters.

So provide a library for constructing the string as you go similar to C++'s std::string.

---
 include/kunit/string-stream.h |  49 ++++++++++++
 kunit/Makefile                |   3 +-
 kunit/string-stream.c         | 147 ++++++++++++++++++++++++++++++++++
 3 files changed, 198 insertions(+), 1 deletion(-)
 create mode 100644 include/kunit/string-stream.h
 create mode 100644 kunit/string-stream.c
include/kunit/string-stream.h new file +49 -0
new file mode 100644
index 0000000000000..0552a05781afe
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string builder used in KUnit for building messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_STRING_STREAM_H
+#define _KUNIT_STRING_STREAM_H
+
+#include <linux/types.h>
+#include <linux/spinlock.h>
+#include <linux/kref.h>
+#include <stdarg.h>
+
+struct string_stream_fragment {
+	struct list_head node;
+	char *fragment;
+};
+
+struct string_stream {
+	size_t length;
+	struct list_head fragments;
+	/* length and fragments are protected by this lock */
+	spinlock_t lock;
+};
+
+struct kunit;
+
+struct string_stream *alloc_string_stream(struct kunit *test);
+
+void string_stream_get(struct string_stream *stream);
+
+int string_stream_put(struct string_stream *stream);
+
+int string_stream_add(struct string_stream *stream, const char *fmt, ...);
+
+int string_stream_vadd(struct string_stream *stream,
+		       const char *fmt,
+		       va_list args);
+
+char *string_stream_get_string(struct string_stream *stream);
+
+void string_stream_clear(struct string_stream *stream);
+
+bool string_stream_is_empty(struct string_stream *stream);
+
+#endif /* _KUNIT_STRING_STREAM_H */
kunit/Makefile +2 -1
index 5efdc4dea2c08..275b565a0e81f 100644
@@ -1 +1,2 @@
-obj-$(CONFIG_KUNIT) +=			test.o
+obj-$(CONFIG_KUNIT) +=			test.o \
+					string-stream.o
kunit/string-stream.c new file +147 -0

147 lines not on this page · raw message

Signature · 6 lines
2.22.0.410.gd8fdbe21b5-goog

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#9 lore raw

Re: [PATCH v9 03/18] kunit: test: add string_stream a std::stream like string builder

Quoting Brendan Higgins (2019-07-12 01:17:29)

Quoted 10 lines
include/kunit/string-stream.h new file +4 -0
new file mode 100644
index 0000000000000..0552a05781afe
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string builder used in KUnit for building messages.
+ *

include/kunit/string-stream.h line 14

Quoted 10 lines
include/kunit/string-stream.h new file +10 -0
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_STRING_STREAM_H
+#define _KUNIT_STRING_STREAM_H
+
+#include <linux/types.h>
+#include <linux/spinlock.h>
+#include <linux/kref.h>

What is this include for? I'd expect to see linux/list.h instead.

Quoted 41 lines
include/kunit/string-stream.h new file +14 -0
+#include <stdarg.h>
+
+struct string_stream_fragment {
+       struct list_head node;
+       char *fragment;
+};
+
+struct string_stream {
+       size_t length;
+       struct list_head fragments;
+       /* length and fragments are protected by this lock */
+       spinlock_t lock;
+};
+
kunit/string-stream.c new file +21 -0
new file mode 100644
index 0000000000000..0463a92dad74b
@@ -0,0 +1,147 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ stream style string builder used in KUnit for building messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <kunit/string-stream.h>
+#include <kunit/test.h>
+
+int string_stream_vadd(struct string_stream *stream,
+                      const char *fmt,
+                      va_list args)
+{
+       struct string_stream_fragment *frag_container;
+       int len;
+       va_list args_for_counting;
+       unsigned long flags;

kunit/string-stream.c line 31

Quoted 10 lines
kunit/string-stream.c new file +10 -0
+
+       /* Make a copy because `vsnprintf` could change it */
+       va_copy(args_for_counting, args);
+
+       /* Need space for null byte. */
+       len = vsnprintf(NULL, 0, fmt, args_for_counting) + 1;
+
+       va_end(args_for_counting);
+
+       frag_container = kmalloc(sizeof(*frag_container), GFP_KERNEL);

This is confusing in that it allocates with GFP_KERNEL but then grabs a spinlock to add and remove from the fragment list. Is it ever going to be called from a place where it can't sleep? If so, the GFP_KERNEL needs to be changed. Otherwise, maybe a mutex would work better to protect access to the fragment list.

I also wonder if it would be better to just have a big slop buffer of a 4K page or something so that we almost never have to allocate anything with a string_stream and we can just rely on a reader consuming data while writers are writing. That might work out better, but I don't quite understand the use case for the string stream.

Quoted 18 lines
kunit/string-stream.c new file +18 -0
+       if (!frag_container)
+               return -ENOMEM;
+
+       frag_container->fragment = kmalloc(len, GFP_KERNEL);
+       if (!frag_container->fragment) {
+               kfree(frag_container);
+               return -ENOMEM;
+       }
+
+       len = vsnprintf(frag_container->fragment, len, fmt, args);
+       spin_lock_irqsave(&stream->lock, flags);
+       stream->length += len;
+       list_add_tail(&frag_container->node, &stream->fragments);
+       spin_unlock_irqrestore(&stream->lock, flags);
+
+       return 0;
+}
+

[...]

kunit/string-stream.c line 104

Quoted 7 lines
kunit/string-stream.c new file +7 -0
+
+bool string_stream_is_empty(struct string_stream *stream)
+{
+       bool is_empty;
+       unsigned long flags;
+
+       spin_lock_irqsave(&stream->lock, flags);

I'm not sure what benefit grabbing the lock is having here. If the list isn't empty after this is called then the race isn't resolved by grabbing and releasing the lock. The function is returning stale data in that case.

Quoted 32 lines
kunit/string-stream.c new file +32 -0
+       is_empty = list_empty(&stream->fragments);
+       spin_unlock_irqrestore(&stream->lock, flags);
+
+       return is_empty;
+}
+
+static int string_stream_init(struct kunit_resource *res, void *context)
+{
+       struct string_stream *stream;
+
+       stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+       if (!stream)
+               return -ENOMEM;
+
+       res->allocation = stream;
+       INIT_LIST_HEAD(&stream->fragments);
+       spin_lock_init(&stream->lock);
+
+       return 0;
+}
+
+static void string_stream_free(struct kunit_resource *res)
+{
+       struct string_stream *stream = res->allocation;
+
+       string_stream_clear(stream);
+       kfree(stream);
+}
+
+struct string_stream *alloc_string_stream(struct kunit *test)
+{
+       struct kunit_resource *res;

kunit/string-stream.c line 146

Quoted 10 lines
kunit/string-stream.c new file +10 -0
+
+       res = kunit_alloc_resource(test,
+                                  string_stream_init,
+                                  string_stream_free,
+                                  NULL);
+
+       if (!res)
+               return NULL;
+
+       return res->allocation;

Maybe kunit_alloc_resource() should just return res->allocation, or NULL, so that these functions can be simplified to 'return kunit_alloc_resource()'? Does the caller ever care to do anything with struct kunit_resource anyway?

#10 lore raw

Re: [PATCH v9 03/18] kunit: test: add string_stream a std::stream like string builder

Quoted 16 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-12 01:17:29)

Quoted 14 lines
include/kunit/string-stream.h new file +8 -0
new file mode 100644
index 0000000000000..0552a05781afe
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string builder used in KUnit for building messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
Quoted 8 lines

include/kunit/string-stream.h line 14

Quoted 6 lines
include/kunit/string-stream.h new file +6 -0
+#ifndef _KUNIT_STRING_STREAM_H
+#define _KUNIT_STRING_STREAM_H
+
+#include <linux/types.h>
+#include <linux/spinlock.h>
+#include <linux/kref.h>

What is this include for? I'd expect to see linux/list.h instead.

Sorry about that. I used to reference count this before I made it a kunit managed resource.

Quoted 49 lines
Quoted 49 lines
include/kunit/string-stream.h new file +14 -0
+#include <stdarg.h>
+
+struct string_stream_fragment {
+       struct list_head node;
+       char *fragment;
+};
+
+struct string_stream {
+       size_t length;
+       struct list_head fragments;
+       /* length and fragments are protected by this lock */
+       spinlock_t lock;
+};
+
kunit/string-stream.c new file +29 -0
new file mode 100644
index 0000000000000..0463a92dad74b
@@ -0,0 +1,147 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ stream style string builder used in KUnit for building messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <kunit/string-stream.h>
+#include <kunit/test.h>
+
+int string_stream_vadd(struct string_stream *stream,
+                      const char *fmt,
+                      va_list args)
+{
+       struct string_stream_fragment *frag_container;
+       int len;
+       va_list args_for_counting;
+       unsigned long flags;
+
+       /* Make a copy because `vsnprintf` could change it */
+       va_copy(args_for_counting, args);
+
+       /* Need space for null byte. */
+       len = vsnprintf(NULL, 0, fmt, args_for_counting) + 1;
+
+       va_end(args_for_counting);
Quoted 8 lines

kunit/string-stream.c line 31

Quoted 2 lines
kunit/string-stream.c new file +2 -0
+
+       frag_container = kmalloc(sizeof(*frag_container), GFP_KERNEL);

This is confusing in that it allocates with GFP_KERNEL but then grabs a spinlock to add and remove from the fragment list. Is it ever going to be called from a place where it can't sleep? If so, the GFP_KERNEL needs to be changed. Otherwise, maybe a mutex would work better to protect access to the fragment list.

Right, using a mutex here would be fine. Sorry, I meant to filter for my usage of them after you asked me to remove them in 01, but evidently I forgot to do so. Sorry, will fix.

Quoted 5 lines

I also wonder if it would be better to just have a big slop buffer of a 4K page or something so that we almost never have to allocate anything with a string_stream and we can just rely on a reader consuming data while writers are writing. That might work out better, but I don't quite understand the use case for the string stream.

That makes sense, but might that also waste memory since we will almost never need that much memory?

Quoted 19 lines
Quoted 18 lines
kunit/string-stream.c new file +18 -0
+       if (!frag_container)
+               return -ENOMEM;
+
+       frag_container->fragment = kmalloc(len, GFP_KERNEL);
+       if (!frag_container->fragment) {
+               kfree(frag_container);
+               return -ENOMEM;
+       }
+
+       len = vsnprintf(frag_container->fragment, len, fmt, args);
+       spin_lock_irqsave(&stream->lock, flags);
+       stream->length += len;
+       list_add_tail(&frag_container->node, &stream->fragments);
+       spin_unlock_irqrestore(&stream->lock, flags);
+
+       return 0;
+}
+

[...]

Quoted 12 lines

kunit/string-stream.c line 104

Quoted 7 lines
kunit/string-stream.c new file +7 -0
+
+bool string_stream_is_empty(struct string_stream *stream)
+{
+       bool is_empty;
+       unsigned long flags;
+
+       spin_lock_irqsave(&stream->lock, flags);

I'm not sure what benefit grabbing the lock is having here. If the list isn't empty after this is called then the race isn't resolved by grabbing and releasing the lock. The function is returning stale data in that case.

Good point, I didn't realize list_empty was protected by READ_ONCE. Will fix.

Quoted 39 lines
Quoted 39 lines
kunit/string-stream.c new file +39 -0
+       is_empty = list_empty(&stream->fragments);
+       spin_unlock_irqrestore(&stream->lock, flags);
+
+       return is_empty;
+}
+
+static int string_stream_init(struct kunit_resource *res, void *context)
+{
+       struct string_stream *stream;
+
+       stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+       if (!stream)
+               return -ENOMEM;
+
+       res->allocation = stream;
+       INIT_LIST_HEAD(&stream->fragments);
+       spin_lock_init(&stream->lock);
+
+       return 0;
+}
+
+static void string_stream_free(struct kunit_resource *res)
+{
+       struct string_stream *stream = res->allocation;
+
+       string_stream_clear(stream);
+       kfree(stream);
+}
+
+struct string_stream *alloc_string_stream(struct kunit *test)
+{
+       struct kunit_resource *res;
+
+       res = kunit_alloc_resource(test,
+                                  string_stream_init,
+                                  string_stream_free,
+                                  NULL);
+
+       if (!res)
Quoted 8 lines

kunit/string-stream.c line 146

Quoted 3 lines
kunit/string-stream.c new file +3 -0
+               return NULL;
+
+       return res->allocation;

Maybe kunit_alloc_resource() should just return res->allocation, or NULL, so that these functions can be simplified to 'return kunit_alloc_resource()'? Does the caller ever care to do anything with struct kunit_resource anyway?

Another good point. I think originally I thought it might, but now with the mandatory init function, the user has to provide a function where they can do the init work. They might as well do it there. Will fix.

#11 lore raw

Re: [PATCH v9 03/18] kunit: test: add string_stream a std::stream like string builder

Quoting Brendan Higgins (2019-07-15 14:11:50)

Quoted 10 lines · Stephen Boyd
Quoted 6 lines

I also wonder if it would be better to just have a big slop buffer of a 4K page or something so that we almost never have to allocate anything with a string_stream and we can just rely on a reader consuming data while writers are writing. That might work out better, but I don't quite understand the use case for the string stream.

That makes sense, but might that also waste memory since we will almost never need that much memory?

Why do we care? These are unit tests. Having allocations in here makes things more complicated, whereas it would be simpler to have a pointer and a spinlock operating on a chunk of memory that gets flushed out periodically.

#12 lore raw

Re: [PATCH v9 03/18] kunit: test: add string_stream a std::stream like string builder

Quoted 10 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-15 14:11:50)

Quoted 8 lines · Stephen Boyd
Quoted 6 lines

I also wonder if it would be better to just have a big slop buffer of a 4K page or something so that we almost never have to allocate anything with a string_stream and we can just rely on a reader consuming data while writers are writing. That might work out better, but I don't quite understand the use case for the string stream.

Quoted 4 lines
Quoted 2 lines

That makes sense, but might that also waste memory since we will almost never need that much memory?

Why do we care? These are unit tests.

Agreed.

Quoted 4 lines

Having allocations in here makes
things more complicated, whereas it would be simpler to have a pointer and a spinlock operating on a chunk of memory that gets flushed out periodically.

I am not so sure. I have to have the logic to allocate memory in some case no matter what (what if I need more memory that my preallocated chuck?). I think it is simpler to always request an allocation than to only sometimes request an allocation.

#13 lore raw

Re: [PATCH v9 03/18] kunit: test: add string_stream a std::stream like string builder

Quoted 24 lines · Brendan Higgins
Quoted 10 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-15 14:11:50)

Quoted 8 lines · Stephen Boyd
Quoted 6 lines

I also wonder if it would be better to just have a big slop buffer of a 4K page or something so that we almost never have to allocate anything with a string_stream and we can just rely on a reader consuming data while writers are writing. That might work out better, but I don't quite understand the use case for the string stream.

Quoted 4 lines
Quoted 2 lines

That makes sense, but might that also waste memory since we will almost never need that much memory?

Why do we care? These are unit tests.

Agreed.

Quoted 4 lines

Having allocations in here makes
things more complicated, whereas it would be simpler to have a pointer and a spinlock operating on a chunk of memory that gets flushed out periodically.

Quoted 4 lines

I am not so sure. I have to have the logic to allocate memory in some case no matter what (what if I need more memory that my preallocated chuck?). I think it is simpler to always request an allocation than to only sometimes request an allocation.

Another even simpler alternative might be to just allocate memory using kunit_kmalloc as we need it and just let the kunit_resource code handle cleaning it all up when the test case finishes.

What do you think?

#14 lore raw

Re: [PATCH v9 03/18] kunit: test: add string_stream a std::stream like string builder

Quoting Brendan Higgins (2019-07-15 15:43:20)

Quoted 26 lines
Quoted 24 lines · Brendan Higgins
Quoted 10 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-15 14:11:50)

Quoted 8 lines · Stephen Boyd
Quoted 6 lines

I also wonder if it would be better to just have a big slop buffer of a 4K page or something so that we almost never have to allocate anything with a string_stream and we can just rely on a reader consuming data while writers are writing. That might work out better, but I don't quite understand the use case for the string stream.

Quoted 4 lines
Quoted 2 lines

That makes sense, but might that also waste memory since we will almost never need that much memory?

Why do we care? These are unit tests.

Agreed.

Quoted 4 lines

Having allocations in here makes
things more complicated, whereas it would be simpler to have a pointer and a spinlock operating on a chunk of memory that gets flushed out periodically.

Quoted 8 lines
Quoted 4 lines

I am not so sure. I have to have the logic to allocate memory in some case no matter what (what if I need more memory that my preallocated chuck?). I think it is simpler to always request an allocation than to only sometimes request an allocation.

Another even simpler alternative might be to just allocate memory using kunit_kmalloc as we need it and just let the kunit_resource code handle cleaning it all up when the test case finishes.

Sure, sounds like a nice way to avoid duplicating similar logic to maintain a list of things to free later.

Quoted 2 lines

What do you think?

If you go the allocation route then you'll need to have the flags to know what context you're in to allocate appropriately. Does that mean all the string operations will now take GFP flags?

#15 lore raw

Re: [PATCH v9 03/18] kunit: test: add string_stream a std::stream like string builder

Quoted 33 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-15 15:43:20)

Quoted 31 lines
Quoted 28 lines · Brendan Higgins
Quoted 10 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-15 14:11:50)

Quoted 8 lines · Stephen Boyd
Quoted 6 lines

I also wonder if it would be better to just have a big slop buffer of a 4K page or something so that we almost never have to allocate anything with a string_stream and we can just rely on a reader consuming data while writers are writing. That might work out better, but I don't quite understand the use case for the string stream.

Quoted 4 lines
Quoted 2 lines

That makes sense, but might that also waste memory since we will almost never need that much memory?

Why do we care? These are unit tests.

Agreed.

Quoted 4 lines

Having allocations in here makes
things more complicated, whereas it would be simpler to have a pointer and a spinlock operating on a chunk of memory that gets flushed out periodically.

I am not so sure. I have to have the logic to allocate memory in some case no matter what (what if I need more memory that my preallocated chuck?). I think it is simpler to always request an allocation than to only sometimes request an allocation.

Quoted 6 lines
Quoted 3 lines

Another even simpler alternative might be to just allocate memory using kunit_kmalloc as we need it and just let the kunit_resource code handle cleaning it all up when the test case finishes.

Sure, sounds like a nice way to avoid duplicating similar logic to maintain a list of things to free later.

I think I will go that route for now.

Quoted 6 lines
Quoted 2 lines

What do you think?

If you go the allocation route then you'll need to have the flags to know what context you're in to allocate appropriately. Does that mean all the string operations will now take GFP flags?

We could set the GFP flags in the constructor, store them in a field, and then just reuse them.

Thanks!

#16 lore raw

PATCH v9 4/18 kunit: test: add kunit_stream a std::stream like logger

In reply to Brendan Higgins

A lot of the expectation and assertion infrastructure prints out fairly complicated test failure messages, so add a C++ style log library for for logging test results.

---
 include/kunit/kunit-stream.h |  81 +++++++++++++++++++++++
 include/kunit/test.h         |   3 +
 kunit/Makefile               |   3 +-
 kunit/kunit-stream.c         | 123 +++++++++++++++++++++++++++++++++++
 kunit/test.c                 |   6 ++
 5 files changed, 215 insertions(+), 1 deletion(-)
 create mode 100644 include/kunit/kunit-stream.h
 create mode 100644 kunit/kunit-stream.c
include/kunit/kunit-stream.h new file +81 -0
new file mode 100644
index 0000000000000..a7b53eabf6be4
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_KUNIT_STREAM_H
+#define _KUNIT_KUNIT_STREAM_H
+
+#include <linux/types.h>
+#include <kunit/string-stream.h>
+
+struct kunit;
+
+/**
+ * struct kunit_stream - a std::stream style string builder.
+ *
+ * A std::stream style string builder. Allows messages to be built up and
+ * printed all at once.
+ */
+struct kunit_stream {
+	/* private: internal use only. */
+	struct kunit *test;
+	const char *level;
+	struct string_stream *internal_stream;
+};
+
+/**
+ * alloc_kunit_stream() - constructs a new &struct kunit_stream.
+ * @test: The test context object.
+ * @level: The log level at which to print out the message.
+ *
+ * Constructs a new test managed &struct kunit_stream.
+ */
+struct kunit_stream *alloc_kunit_stream(struct kunit *test, const char *level);
+
+/**
+ * kunit_stream_add(): adds the formatted input to the internal buffer.
+ * @kstream: the stream being operated on.
+ * @fmt: printf style format string to append to stream.
+ *
+ * Appends the formatted string, @fmt, to the internal buffer.
+ */
+void __printf(2, 3) kunit_stream_add(struct kunit_stream *kstream,
+				     const char *fmt, ...);
+
+/**
+ * kunit_stream_append(): appends the contents of @other to @kstream.
+ * @kstream: the stream to which @other is appended.
+ * @other: the stream whose contents are appended to @kstream.
+ *
+ * Appends the contents of @other to @kstream.
+ */
+void kunit_stream_append(struct kunit_stream *kstream,
+			 struct kunit_stream *other);
+
+/**
+ * kunit_stream_commit(): prints out the internal buffer to the user.
+ * @kstream: the stream being operated on.
+ *
+ * Outputs the contents of the internal buffer as a kunit_printk formatted
+ * output. KUNIT_STREAM ONLY OUTPUTS ITS BUFFER TO THE USER IF COMMIT IS
+ * CALLED!!! The reason for this is that it allows us to construct a message
+ * before we know whether we want to print it out; this can be extremely handy
+ * if there is information you might need for a failure message that is easiest
+ * to collect in the steps leading up to the actual check.
+ */
+void kunit_stream_commit(struct kunit_stream *kstream);
+
+/**
+ * kunit_stream_clear(): clears the internal buffer.
+ * @kstream: the stream being operated on.
+ *
+ * Clears the contents of the internal buffer.
+ */
+void kunit_stream_clear(struct kunit_stream *kstream);
+
+#endif /* _KUNIT_KUNIT_STREAM_H */
include/kunit/test.h +3 -0
index bdf41d31c343c..bc7dbdcf8abab 100644
@@ -11,6 +11,7 @@
 
 #include <linux/types.h>
 #include <linux/slab.h>
+#include <kunit/kunit-stream.h>
 
 struct kunit_resource;
 
@@ -184,6 +185,8 @@ struct kunit {
 
 void kunit_init_test(struct kunit *test, const char *name);
 
+void kunit_fail(struct kunit *test, struct kunit_stream *stream);
+
 int kunit_run_tests(struct kunit_suite *suite);
 
 /**
kunit/Makefile +2 -1
index 275b565a0e81f..6ddc622ee6b1c 100644
@@ -1,2 +1,3 @@
 obj-$(CONFIG_KUNIT) +=			test.o \
-					string-stream.o
+					string-stream.o \
+					kunit-stream.o
kunit/kunit-stream.c new file +123 -0

123 lines not on this page · raw message

kunit/test.c +6 -0
index f165c9d8e10b0..29edf34a89a37 100644
@@ -120,6 +120,12 @@ static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case,
 			      test_case->name);
 }
 
+void kunit_fail(struct kunit *test, struct kunit_stream *stream)
+{
+	kunit_set_failure(test);
+	kunit_stream_commit(stream);
+}
+
 void kunit_init_test(struct kunit *test, const char *name)
 {
 	mutex_init(&test->lock);
Signature · 6 lines
2.22.0.410.gd8fdbe21b5-goog

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#17 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoting Brendan Higgins (2019-07-12 01:17:30)

Quoted 23 lines
include/kunit/kunit-stream.h new file +17 -0
new file mode 100644
index 0000000000000..a7b53eabf6be4
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_KUNIT_STREAM_H
+#define _KUNIT_KUNIT_STREAM_H
+
+#include <linux/types.h>
+#include <kunit/string-stream.h>
+
+struct kunit;
+

include/kunit/kunit-stream.h line 27

Quoted 10 lines
include/kunit/kunit-stream.h new file +10 -0
+/**
+ * struct kunit_stream - a std::stream style string builder.
+ *
+ * A std::stream style string builder. Allows messages to be built up and
+ * printed all at once.
+ */
+struct kunit_stream {
+       /* private: internal use only. */
+       struct kunit *test;
+       const char *level;

Is the level changed? See my comment below, but I wonder if this whole struct can go away and the wrappers can just operate on 'struct string_stream' instead.

Quoted 42 lines
include/kunit/kunit-stream.h new file +2 -0
+       struct string_stream *internal_stream;
+};
kunit/kunit-stream.c new file +34 -0
new file mode 100644
index 0000000000000..8bea1f22eafb5
@@ -0,0 +1,123 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#include <kunit/test.h>
+#include <kunit/kunit-stream.h>
+#include <kunit/string-stream.h>
+
+void kunit_stream_add(struct kunit_stream *kstream, const char *fmt, ...)
+{
+       va_list args;
+       struct string_stream *stream = kstream->internal_stream;
+
+       va_start(args, fmt);
+
+       if (string_stream_vadd(stream, fmt, args) < 0)
+               kunit_err(kstream->test,
+                         "Failed to allocate fragment: %s\n",
+                         fmt);
+
+       va_end(args);
+}
+
+void kunit_stream_append(struct kunit_stream *kstream,
+                               struct kunit_stream *other)
+{
+       struct string_stream *other_stream = other->internal_stream;
+       const char *other_content;
+

kunit/kunit-stream.c line 44

Quoted 10 lines
kunit/kunit-stream.c new file +10 -0
+       other_content = string_stream_get_string(other_stream);
+
+       if (!other_content) {
+               kunit_err(kstream->test,
+                         "Failed to get string from second argument for appending\n");
+               return;
+       }
+
+       kunit_stream_add(kstream, other_content);
+}

Why can't this function be implemented in the string_stream API? Seems valid to want to append one stream to another and that isn't kunit_stream specific.

Quoted 18 lines
kunit/kunit-stream.c new file +18 -0
+
+void kunit_stream_clear(struct kunit_stream *kstream)
+{
+       string_stream_clear(kstream->internal_stream);
+}
+
+void kunit_stream_commit(struct kunit_stream *kstream)
+{
+       struct string_stream *stream = kstream->internal_stream;
+       struct string_stream_fragment *fragment;
+       struct kunit *test = kstream->test;
+       char *buf;
+
+       buf = string_stream_get_string(stream);
+       if (!buf) {
+               kunit_err(test,
+                         "Could not allocate buffer, dumping stream:\n");
+               list_for_each_entry(fragment, &stream->fragments, node) {

kunit/kunit-stream.c line 72

Quoted 10 lines
kunit/kunit-stream.c new file +10 -0
+                       kunit_err(test, fragment->fragment);
+               }
+               kunit_err(test, "\n");
+               goto cleanup;
+       }
+
+       kunit_printk(kstream->level, test, buf);
+       kfree(buf);
+
+cleanup:

Drop the goto and use an 'else' please.

Quoted 23 lines
kunit/kunit-stream.c new file +23 -0
+       kunit_stream_clear(kstream);
+}
+
+static int kunit_stream_init(struct kunit_resource *res, void *context)
+{
+       struct kunit *test = context;
+       struct kunit_stream *stream;
+
+       stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+       if (!stream)
+               return -ENOMEM;
+
+       res->allocation = stream;
+       stream->test = test;
+       stream->internal_stream = alloc_string_stream(test);
+
+       if (!stream->internal_stream)
+               return -ENOMEM;
+
+       return 0;
+}
+
+static void kunit_stream_free(struct kunit_resource *res)

kunit/kunit-stream.c line 105

Quoted 10 lines
kunit/kunit-stream.c new file +10 -0
+{
+       struct kunit_stream *stream = res->allocation;
+
+       if (!string_stream_is_empty(stream->internal_stream)) {
+               kunit_err(stream->test,
+                         "End of test case reached with uncommitted stream entries\n");
+               kunit_stream_commit(stream);
+       }
+}
+

Nitpick: Drop this extra newline.

kunit/test.c static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case, · line 123

Quoted 9 lines
kunit/test.c +1 -0
index f165c9d8e10b0..29edf34a89a37 100644
@@ -120,6 +120,12 @@ static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case,
                              test_case->name);
 }
 
+void kunit_fail(struct kunit *test, struct kunit_stream *stream)

Why doesn't 'struct kunit' have a 'struct kunit_stream' inside of it? It seems that the two are highly related, to the point that it might just make sense to have

	struct kunit {
		struct kunit_stream stream;
		...
	};

kunit/test.c static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case, · line 126

Quoted 3 lines
kunit/test.c +3 -0
+{
+       kunit_set_failure(test);
+       kunit_stream_commit(stream);

And then this function can just take a test and the stream can be associated with the test directly. Use container_of() to get to the test when the only pointer in hand is for the stream too.

Quoted 5 lines
kunit/test.c +2 -0
+}
+
 void kunit_init_test(struct kunit *test, const char *name)
 {
        mutex_init(&test->lock);
#18 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 31 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-12 01:17:30)

Quoted 29 lines
include/kunit/kunit-stream.h new file +23 -0
new file mode 100644
index 0000000000000..a7b53eabf6be4
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_KUNIT_STREAM_H
+#define _KUNIT_KUNIT_STREAM_H
+
+#include <linux/types.h>
+#include <kunit/string-stream.h>
+
+struct kunit;
+
+/**
+ * struct kunit_stream - a std::stream style string builder.
+ *
+ * A std::stream style string builder. Allows messages to be built up and
+ * printed all at once.
+ */
Quoted 8 lines

include/kunit/kunit-stream.h line 27

Quoted 4 lines
include/kunit/kunit-stream.h new file +4 -0
+struct kunit_stream {
+       /* private: internal use only. */
+       struct kunit *test;
+       const char *level;

Is the level changed? See my comment below, but I wonder if this whole struct can go away and the wrappers can just operate on 'struct string_stream' instead.

I was inclined to agree with you when I first read your comment, but then I thought about the case that someone wants to add in a debug message (of which I currently have none). I think under most circumstances a user of kunit_stream would likely want to pick a default verbosity that maybe I should provide, but may still want different verbosity levels.

The main reason I want to keep the types separate, string_stream vs. kunit_stream, is that they are intended to be used differently. string_stream is just a generic string builder. If you are using that, you are expecting to see someone building the string at some point and then doing something interesting with it. kunit_stream really tells you specifically that KUnit is putting together a message to communicate something to a user of KUnit. It is really used in a very specific way, and I wouldn't want to generalize its usage beyond how it is currently used. I think in order to preserve the author's intention it adds clarity to keep the types separate regardless of how similar they might be in reality.

Quoted 48 lines
Quoted 48 lines
include/kunit/kunit-stream.h new file +2 -0
+       struct string_stream *internal_stream;
+};
kunit/kunit-stream.c new file +40 -0
new file mode 100644
index 0000000000000..8bea1f22eafb5
@@ -0,0 +1,123 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#include <kunit/test.h>
+#include <kunit/kunit-stream.h>
+#include <kunit/string-stream.h>
+
+void kunit_stream_add(struct kunit_stream *kstream, const char *fmt, ...)
+{
+       va_list args;
+       struct string_stream *stream = kstream->internal_stream;
+
+       va_start(args, fmt);
+
+       if (string_stream_vadd(stream, fmt, args) < 0)
+               kunit_err(kstream->test,
+                         "Failed to allocate fragment: %s\n",
+                         fmt);
+
+       va_end(args);
+}
+
+void kunit_stream_append(struct kunit_stream *kstream,
+                               struct kunit_stream *other)
+{
+       struct string_stream *other_stream = other->internal_stream;
+       const char *other_content;
+
+       other_content = string_stream_get_string(other_stream);
+
+       if (!other_content) {
+               kunit_err(kstream->test,
+                         "Failed to get string from second argument for appending\n");
+               return;
Quoted 8 lines

kunit/kunit-stream.c line 44

Quoted 4 lines
kunit/kunit-stream.c new file +4 -0
+       }
+
+       kunit_stream_add(kstream, other_content);
+}

Why can't this function be implemented in the string_stream API? Seems valid to want to append one stream to another and that isn't kunit_stream specific.

Fair point. Will do.

Quoted 22 lines
Quoted 22 lines
kunit/kunit-stream.c new file +22 -0
+
+void kunit_stream_clear(struct kunit_stream *kstream)
+{
+       string_stream_clear(kstream->internal_stream);
+}
+
+void kunit_stream_commit(struct kunit_stream *kstream)
+{
+       struct string_stream *stream = kstream->internal_stream;
+       struct string_stream_fragment *fragment;
+       struct kunit *test = kstream->test;
+       char *buf;
+
+       buf = string_stream_get_string(stream);
+       if (!buf) {
+               kunit_err(test,
+                         "Could not allocate buffer, dumping stream:\n");
+               list_for_each_entry(fragment, &stream->fragments, node) {
+                       kunit_err(test, fragment->fragment);
+               }
+               kunit_err(test, "\n");
+               goto cleanup;
Quoted 8 lines

kunit/kunit-stream.c line 72

Quoted 6 lines
kunit/kunit-stream.c new file +6 -0
+       }
+
+       kunit_printk(kstream->level, test, buf);
+       kfree(buf);
+
+cleanup:

Drop the goto and use an 'else' please.

Will do.

Quoted 27 lines
Quoted 27 lines
kunit/kunit-stream.c new file +27 -0
+       kunit_stream_clear(kstream);
+}
+
+static int kunit_stream_init(struct kunit_resource *res, void *context)
+{
+       struct kunit *test = context;
+       struct kunit_stream *stream;
+
+       stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+       if (!stream)
+               return -ENOMEM;
+
+       res->allocation = stream;
+       stream->test = test;
+       stream->internal_stream = alloc_string_stream(test);
+
+       if (!stream->internal_stream)
+               return -ENOMEM;
+
+       return 0;
+}
+
+static void kunit_stream_free(struct kunit_resource *res)
+{
+       struct kunit_stream *stream = res->allocation;
+
+       if (!string_stream_is_empty(stream->internal_stream)) {
Quoted 8 lines

kunit/kunit-stream.c line 105

Quoted 6 lines
kunit/kunit-stream.c new file +6 -0
+               kunit_err(stream->test,
+                         "End of test case reached with uncommitted stream entries\n");
+               kunit_stream_commit(stream);
+       }
+}
+

Nitpick: Drop this extra newline.

Oops, nice catch.

Quoted 8 lines
Quoted 8 lines
kunit/test.c +0 -0
index f165c9d8e10b0..29edf34a89a37 100644
@@ -120,6 +120,12 @@ static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case,
                              test_case->name);
 }
 
Quoted 5 lines

kunit/test.c static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case, · line 123

Quoted 1 line
kunit/test.c +1 -0
+void kunit_fail(struct kunit *test, struct kunit_stream *stream)

Why doesn't 'struct kunit' have a 'struct kunit_stream' inside of it? It seems that the two are highly related, to the point that it might just make sense to have

A struct kunit_stream is usually associated with a message that is being built up over time like maybe an expectation; it is meant to capture the idea that we might want to send some information out to the user pertaining to some thing 'X', but we aren't sure that we actually want to send it until 'X' is complete, but do to the nature of 'X' it is easier to start constructing the message before 'X' is complete.

Consider a complicated expectation, there might be multiple conditions that satisfy it and multiple conditions which could make it fail. As we start exploring the input to the expectation we gain information that we might want to share back with the user if the expectation were to fail and we might get that information before we are actually sure that the expectation does indeed fail.

When we first step into the expectation we immediately know the function name, file name, and line number where we are called and would want to put that information into any message we would send to the user about this expectation. Next, we might want to check a property of the input, it may or may not be enough information on its own for the expectation to fail, but we want to share the result of the property check with the user regardless, BUT only if the expectation as a whole fails.

Hence, we can have multiple struct kunit_streams associated with a struct kunit active at any given time.

Quoted 5 lines
        struct kunit {
                struct kunit_stream stream;
                ...
        };
Quoted 7 lines

kunit/test.c static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case, · line 126

Quoted 3 lines
kunit/test.c +3 -0
+{
+       kunit_set_failure(test);
+       kunit_stream_commit(stream);

And then this function can just take a test and the stream can be associated with the test directly. Use container_of() to get to the test when the only pointer in hand is for the stream too.

Unfortunately that wouldn't work. See my above explanation.

Quoted 5 lines

kunit/test.c static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case, · line 131

Quoted 5 lines
kunit/test.c +2 -0
+}
+
 void kunit_init_test(struct kunit *test, const char *name)
 {
        mutex_init(&test->lock);

Thanks!

#19 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 189 lines · Brendan Higgins
Quoted 31 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-12 01:17:30)

Quoted 29 lines
include/kunit/kunit-stream.h new file +23 -0
new file mode 100644
index 0000000000000..a7b53eabf6be4
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_KUNIT_STREAM_H
+#define _KUNIT_KUNIT_STREAM_H
+
+#include <linux/types.h>
+#include <kunit/string-stream.h>
+
+struct kunit;
+
+/**
+ * struct kunit_stream - a std::stream style string builder.
+ *
+ * A std::stream style string builder. Allows messages to be built up and
+ * printed all at once.
+ */
Quoted 8 lines

include/kunit/kunit-stream.h line 27

Quoted 4 lines
include/kunit/kunit-stream.h new file +4 -0
+struct kunit_stream {
+       /* private: internal use only. */
+       struct kunit *test;
+       const char *level;

Is the level changed? See my comment below, but I wonder if this whole struct can go away and the wrappers can just operate on 'struct string_stream' instead.

I was inclined to agree with you when I first read your comment, but then I thought about the case that someone wants to add in a debug message (of which I currently have none). I think under most circumstances a user of kunit_stream would likely want to pick a default verbosity that maybe I should provide, but may still want different verbosity levels.

The main reason I want to keep the types separate, string_stream vs. kunit_stream, is that they are intended to be used differently. string_stream is just a generic string builder. If you are using that, you are expecting to see someone building the string at some point and then doing something interesting with it. kunit_stream really tells you specifically that KUnit is putting together a message to communicate something to a user of KUnit. It is really used in a very specific way, and I wouldn't want to generalize its usage beyond how it is currently used. I think in order to preserve the author's intention it adds clarity to keep the types separate regardless of how similar they might be in reality.

Quoted 48 lines
Quoted 48 lines
include/kunit/kunit-stream.h new file +2 -0
+       struct string_stream *internal_stream;
+};
kunit/kunit-stream.c new file +40 -0
new file mode 100644
index 0000000000000..8bea1f22eafb5
@@ -0,0 +1,123 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * C++ stream style string formatter and printer used in KUnit for outputting
+ * KUnit messages.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#include <kunit/test.h>
+#include <kunit/kunit-stream.h>
+#include <kunit/string-stream.h>
+
+void kunit_stream_add(struct kunit_stream *kstream, const char *fmt, ...)
+{
+       va_list args;
+       struct string_stream *stream = kstream->internal_stream;
+
+       va_start(args, fmt);
+
+       if (string_stream_vadd(stream, fmt, args) < 0)
+               kunit_err(kstream->test,
+                         "Failed to allocate fragment: %s\n",
+                         fmt);
+
+       va_end(args);
+}
+
+void kunit_stream_append(struct kunit_stream *kstream,
+                               struct kunit_stream *other)
+{
+       struct string_stream *other_stream = other->internal_stream;
+       const char *other_content;
+
+       other_content = string_stream_get_string(other_stream);
+
+       if (!other_content) {
+               kunit_err(kstream->test,
+                         "Failed to get string from second argument for appending\n");
+               return;
Quoted 8 lines

kunit/kunit-stream.c line 44

Quoted 4 lines
kunit/kunit-stream.c new file +4 -0
+       }
+
+       kunit_stream_add(kstream, other_content);
+}

Why can't this function be implemented in the string_stream API? Seems valid to want to append one stream to another and that isn't kunit_stream specific.

Fair point. Will do.

Quoted 22 lines
Quoted 22 lines
kunit/kunit-stream.c new file +22 -0
+
+void kunit_stream_clear(struct kunit_stream *kstream)
+{
+       string_stream_clear(kstream->internal_stream);
+}
+
+void kunit_stream_commit(struct kunit_stream *kstream)
+{
+       struct string_stream *stream = kstream->internal_stream;
+       struct string_stream_fragment *fragment;
+       struct kunit *test = kstream->test;
+       char *buf;
+
+       buf = string_stream_get_string(stream);
+       if (!buf) {
+               kunit_err(test,
+                         "Could not allocate buffer, dumping stream:\n");
+               list_for_each_entry(fragment, &stream->fragments, node) {
+                       kunit_err(test, fragment->fragment);
+               }
+               kunit_err(test, "\n");
+               goto cleanup;
Quoted 8 lines

kunit/kunit-stream.c line 72

Quoted 6 lines
kunit/kunit-stream.c new file +6 -0
+       }
+
+       kunit_printk(kstream->level, test, buf);
+       kfree(buf);
+
+cleanup:

Drop the goto and use an 'else' please.

Will do.

Quoted 27 lines
Quoted 27 lines
kunit/kunit-stream.c new file +27 -0
+       kunit_stream_clear(kstream);
+}
+
+static int kunit_stream_init(struct kunit_resource *res, void *context)
+{
+       struct kunit *test = context;
+       struct kunit_stream *stream;
+
+       stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+       if (!stream)
+               return -ENOMEM;
+
+       res->allocation = stream;
+       stream->test = test;
+       stream->internal_stream = alloc_string_stream(test);
+
+       if (!stream->internal_stream)
+               return -ENOMEM;
+
+       return 0;
+}
+
+static void kunit_stream_free(struct kunit_resource *res)
+{
+       struct kunit_stream *stream = res->allocation;
+
+       if (!string_stream_is_empty(stream->internal_stream)) {
Quoted 8 lines

kunit/kunit-stream.c line 105

Quoted 6 lines
kunit/kunit-stream.c new file +6 -0
+               kunit_err(stream->test,
+                         "End of test case reached with uncommitted stream entries\n");
+               kunit_stream_commit(stream);
+       }
+}
+

Nitpick: Drop this extra newline.

Quoted 1 line

Oops, nice catch.

Not super important, but I don't want you to think that I am ignoring you. I think you must have unintentionally deleted the last function in this file, or maybe you are referring to something that I am just not seeing, but I don't see the extra newline here.

Quoted 63 lines
Quoted 8 lines
Quoted 8 lines
kunit/test.c +0 -0
index f165c9d8e10b0..29edf34a89a37 100644
@@ -120,6 +120,12 @@ static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case,
                              test_case->name);
 }
 
Quoted 5 lines

kunit/test.c static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case, · line 123

Quoted 1 line
kunit/test.c +1 -0
+void kunit_fail(struct kunit *test, struct kunit_stream *stream)

Why doesn't 'struct kunit' have a 'struct kunit_stream' inside of it? It seems that the two are highly related, to the point that it might just make sense to have

A struct kunit_stream is usually associated with a message that is being built up over time like maybe an expectation; it is meant to capture the idea that we might want to send some information out to the user pertaining to some thing 'X', but we aren't sure that we actually want to send it until 'X' is complete, but do to the nature of 'X' it is easier to start constructing the message before 'X' is complete.

Consider a complicated expectation, there might be multiple conditions that satisfy it and multiple conditions which could make it fail. As we start exploring the input to the expectation we gain information that we might want to share back with the user if the expectation were to fail and we might get that information before we are actually sure that the expectation does indeed fail.

When we first step into the expectation we immediately know the function name, file name, and line number where we are called and would want to put that information into any message we would send to the user about this expectation. Next, we might want to check a property of the input, it may or may not be enough information on its own for the expectation to fail, but we want to share the result of the property check with the user regardless, BUT only if the expectation as a whole fails.

Hence, we can have multiple struct kunit_streams associated with a struct kunit active at any given time.

Quoted 5 lines
        struct kunit {
                struct kunit_stream stream;
                ...
        };
Quoted 7 lines

kunit/test.c static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case, · line 126

Quoted 3 lines
kunit/test.c +3 -0
+{
+       kunit_set_failure(test);
+       kunit_stream_commit(stream);

And then this function can just take a test and the stream can be associated with the test directly. Use container_of() to get to the test when the only pointer in hand is for the stream too.

Unfortunately that wouldn't work. See my above explanation.

Quoted 5 lines

kunit/test.c static void kunit_print_test_case_ok_not_ok(struct kunit_case *test_case, · line 131

Quoted 5 lines
kunit/test.c +2 -0
+}
+
 void kunit_init_test(struct kunit *test, const char *name)
 {
        mutex_init(&test->lock);

Thanks!

#20 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoting Brendan Higgins (2019-07-16 01:37:34)

Quoted 33 lines
Quoted 31 lines · Brendan Higgins
Quoted 13 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-12 01:17:30)

Quoted 11 lines
include/kunit/kunit-stream.h new file +6 -0
new file mode 100644
index 0000000000000..a7b53eabf6be4
+/**
+ * struct kunit_stream - a std::stream style string builder.
+ *
+ * A std::stream style string builder. Allows messages to be built up and
+ * printed all at once.
+ */
Quoted 8 lines

include/kunit/kunit-stream.h line 27

Quoted 4 lines
include/kunit/kunit-stream.h new file +4 -0
+struct kunit_stream {
+       /* private: internal use only. */
+       struct kunit *test;
+       const char *level;

Is the level changed? See my comment below, but I wonder if this whole struct can go away and the wrappers can just operate on 'struct string_stream' instead.

I was inclined to agree with you when I first read your comment, but then I thought about the case that someone wants to add in a debug message (of which I currently have none). I think under most circumstances a user of kunit_stream would likely want to pick a default verbosity that maybe I should provide, but may still want different verbosity levels.

Quoted 11 lines
Quoted 11 lines

The main reason I want to keep the types separate, string_stream vs. kunit_stream, is that they are intended to be used differently. string_stream is just a generic string builder. If you are using that, you are expecting to see someone building the string at some point and then doing something interesting with it. kunit_stream really tells you specifically that KUnit is putting together a message to communicate something to a user of KUnit. It is really used in a very specific way, and I wouldn't want to generalize its usage beyond how it is currently used. I think in order to preserve the author's intention it adds clarity to keep the types separate regardless of how similar they might be in reality.

You may want to add some of these reasons to the commit text.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

include/kunit/kunit-stream.h

Quoted 8 lines
include/kunit/kunit-stream.h new file +8 -0
+
+       if (!string_stream_is_empty(stream->internal_stream)) {
+               kunit_err(stream->test,
+                         "End of test case reached with uncommitted stream entries\n");
+               kunit_stream_commit(stream);
+       }
+}
+

Nitpick: Drop this extra newline.

Quoted 6 lines
Quoted 1 line

Oops, nice catch.

Not super important, but I don't want you to think that I am ignoring you. I think you must have unintentionally deleted the last function in this file, or maybe you are referring to something that I am just not seeing, but I don't see the extra newline here.

No worries. Sorry for the noise.

Quoted 7 lines
Quoted 7 lines

property of the input, it may or may not be enough information on its own for the expectation to fail, but we want to share the result of the property check with the user regardless, BUT only if the expectation as a whole fails.

Hence, we can have multiple struct kunit_streams associated with a struct kunit active at any given time.

Makes sense. I wasn't sure if there were more than one stream associated with a test. Sounds like there are many to one so it can't just be a member of the test. This could be documented somewhere so this question doesn't come up again.

#21 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 47 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-16 01:37:34)

Quoted 33 lines
Quoted 31 lines · Brendan Higgins
Quoted 13 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-12 01:17:30)

Quoted 11 lines
include/kunit/kunit-stream.h new file +6 -0
new file mode 100644
index 0000000000000..a7b53eabf6be4
+/**
+ * struct kunit_stream - a std::stream style string builder.
+ *
+ * A std::stream style string builder. Allows messages to be built up and
+ * printed all at once.
+ */
Quoted 8 lines

include/kunit/kunit-stream.h line 27

Quoted 4 lines
include/kunit/kunit-stream.h new file +4 -0
+struct kunit_stream {
+       /* private: internal use only. */
+       struct kunit *test;
+       const char *level;

Is the level changed? See my comment below, but I wonder if this whole struct can go away and the wrappers can just operate on 'struct string_stream' instead.

I was inclined to agree with you when I first read your comment, but then I thought about the case that someone wants to add in a debug message (of which I currently have none). I think under most circumstances a user of kunit_stream would likely want to pick a default verbosity that maybe I should provide, but may still want different verbosity levels.

Quoted 11 lines
Quoted 11 lines

The main reason I want to keep the types separate, string_stream vs. kunit_stream, is that they are intended to be used differently. string_stream is just a generic string builder. If you are using that, you are expecting to see someone building the string at some point and then doing something interesting with it. kunit_stream really tells you specifically that KUnit is putting together a message to communicate something to a user of KUnit. It is really used in a very specific way, and I wouldn't want to generalize its usage beyond how it is currently used. I think in order to preserve the author's intention it adds clarity to keep the types separate regardless of how similar they might be in reality.

Quoted 1 line

You may want to add some of these reasons to the commit text.

Will do.

Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

include/kunit/kunit-stream.h

Quoted 8 lines
include/kunit/kunit-stream.h new file +8 -0
+
+       if (!string_stream_is_empty(stream->internal_stream)) {
+               kunit_err(stream->test,
+                         "End of test case reached with uncommitted stream entries\n");
+               kunit_stream_commit(stream);
+       }
+}
+

Nitpick: Drop this extra newline.

Quoted 6 lines
Quoted 1 line

Oops, nice catch.

Not super important, but I don't want you to think that I am ignoring you. I think you must have unintentionally deleted the last function in this file, or maybe you are referring to something that I am just not seeing, but I don't see the extra newline here.

No worries. Sorry for the noise.

Quoted 5 lines
Quoted 5 lines

property of the input, it may or may not be enough information on its own for the expectation to fail, but we want to share the result of the property check with the user regardless, BUT only if the expectation as a whole fails.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Hence, we can have multiple struct kunit_streams associated with a struct kunit active at any given time.

Makes sense. I wasn't sure if there were more than one stream associated with a test. Sounds like there are many to one so it can't just be a member of the test. This could be documented somewhere so this question doesn't come up again.

Sounds good. Will do.

Thanks!
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#22 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

In reply to Brendan Higgins

Quoting Brendan Higgins (2019-07-16 01:37:34)

Quoted 27 lines
Quoted 25 lines · Brendan Higgins

A struct kunit_stream is usually associated with a message that is being built up over time like maybe an expectation; it is meant to capture the idea that we might want to send some information out to the user pertaining to some thing 'X', but we aren't sure that we actually want to send it until 'X' is complete, but do to the nature of 'X' it is easier to start constructing the message before 'X' is complete.

Consider a complicated expectation, there might be multiple conditions that satisfy it and multiple conditions which could make it fail. As we start exploring the input to the expectation we gain information that we might want to share back with the user if the expectation were to fail and we might get that information before we are actually sure that the expectation does indeed fail.

When we first step into the expectation we immediately know the function name, file name, and line number where we are called and would want to put that information into any message we would send to the user about this expectation. Next, we might want to check a property of the input, it may or may not be enough information on its own for the expectation to fail, but we want to share the result of the property check with the user regardless, BUT only if the expectation as a whole fails.

Quoted 2 lines
Quoted 2 lines

Hence, we can have multiple struct kunit_streams associated with a struct kunit active at any given time.

I'm coming back to this now after reading the rest of the patches that deal with assertions and expectations. It looks like the string stream is there to hold a few different pieces of information:

- Line Number
- File Name
- Function Name

The above items could be stored in a structure on the stack that then gets printed and formatted when the expectation or assertion fails. That would make the whole string stream structure and code unnecessary.

The only hypothetical case where this can't be done is a complicated assertion or expectation that does more than one check and can't be written as a function that dumps out what went wrong. Is this a real problem? Maybe such an assertion should just open code that logic so we don't have to build up a string for all the other simple cases.

It seems far simpler to get rid of the string stream API and just have a struct for this.

	struct kunit_fail_msg {
		const char *line;
		const char *file;
		const char *func;
		const char *msg;
	};

Then you can have the assertion macros create this on the stack (with another macro?).

	#define DEFINE_KUNIT_FAIL_MSG(name, _msg) \
		struct kunit_fail_msg name = { \
			.line =  __LINE__, \
			.file = __FILE__, \
			.func = __func__, \
			.msg = _msg, \
		}

Note: I don't know if the __LINE__ above will use the macro location, so this probably needs another wrapper to put the right line number there.

I don't want to derail this whole series on this topic, but it seems like a bunch of code is there to construct this same set of information over and over again into a buffer a little bit at a time and then throw it away when nothing fails just because we may want to support the case where we have some unstructured data to inform the user about.

Why not build in the structured part into the framework (i.e. the struct above) so that it's always there and then add the string building part later when we have to?

#23 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 36 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-16 01:37:34)

Quoted 27 lines
Quoted 25 lines · Brendan Higgins

A struct kunit_stream is usually associated with a message that is being built up over time like maybe an expectation; it is meant to capture the idea that we might want to send some information out to the user pertaining to some thing 'X', but we aren't sure that we actually want to send it until 'X' is complete, but do to the nature of 'X' it is easier to start constructing the message before 'X' is complete.

Consider a complicated expectation, there might be multiple conditions that satisfy it and multiple conditions which could make it fail. As we start exploring the input to the expectation we gain information that we might want to share back with the user if the expectation were to fail and we might get that information before we are actually sure that the expectation does indeed fail.

When we first step into the expectation we immediately know the function name, file name, and line number where we are called and would want to put that information into any message we would send to the user about this expectation. Next, we might want to check a property of the input, it may or may not be enough information on its own for the expectation to fail, but we want to share the result of the property check with the user regardless, BUT only if the expectation as a whole fails.

Quoted 2 lines
Quoted 2 lines

Hence, we can have multiple struct kunit_streams associated with a struct kunit active at any given time.

I'm coming back to this now after reading the rest of the patches that deal with assertions and expectations. It looks like the string stream is there to hold a few different pieces of information:

Quoted 7 lines

- Line Number
- File Name
- Function Name

The above items could be stored in a structure on the stack that then gets printed and formatted when the expectation or assertion fails. That would make the whole string stream structure and code unnecessary.

Most of the expectations and assertions in this patchset are fairly simple, and what you are describing would probably work. However, I have some expectations I plan on adding in later patchsets that make much more complicated checks.

Quoted 5 lines

The only hypothetical case where this can't be done is a complicated assertion or expectation that does more than one check and can't be written as a function that dumps out what went wrong. Is this a real problem? Maybe such an assertion should just open code that logic so we don't have to build up a string for all the other simple cases.

I have some expectations in follow up patchsets for which I created a set of composable matchers for matching structures and function calls that by their nature cannot be written as a single function. The matcher thing is a bit speculative, I know, but for any kind of function call matching, you need to store a record of functions you are expecting to have called and then each one needs to have a set of expectations defined by the user; I don't think there is a way to do that that doesn't involve having multiple separate functions each having some information useful to constructing the message.

I know the code in question isn't in this patchset; the function matching code was in one of the earlier versions of the RFC, but I dropped it to make this patchset smaller and more manageable. So I get it if you would like me to drop it and add it back in when I try to get the function and structure matching stuff in, but I would really prefer to keep it as is if you don't care too much.

Quoted 21 lines

It seems far simpler to get rid of the string stream API and just have a struct for this.

        struct kunit_fail_msg {
                const char *line;
                const char *file;
                const char *func;
                const char *msg;
        };

Then you can have the assertion macros create this on the stack (with another macro?).

        #define DEFINE_KUNIT_FAIL_MSG(name, _msg) \
                struct kunit_fail_msg name = { \
                        .line =  __LINE__, \
                        .file = __FILE__, \
                        .func = __func__, \
                        .msg = _msg, \
                }
Quoted 2 lines

Note: I don't know if the __LINE__ above will use the macro location, so this probably needs another wrapper to put the right line number there.

No, that should work. It picks up where the macro ends up being finally evaluated.

Quoted 5 lines

I don't want to derail this whole series on this topic, but it seems like a bunch of code is there to construct this same set of information over and over again into a buffer a little bit at a time and then throw it away when nothing fails just because we may want to support the case where we have some unstructured data to inform the user about.

Yeah, that's fair. I think there are a number of improvements to be made with how the expectations are defined other than that, but I was hoping I could do that after this patchset is merged. I just figured with the kinds of things I would like to do, it would lead to a whole new round of discussion.

In either case, I think I would still like to use the struct kunit_stream as part of the interface to share the failure message with the test case runner code in test.c, at least eventually, so that I only have to have one way to receive data from expectations, but I think I can do that and still do what you suggest by just constructing the kunit_stream at the end of expectations where it is feasible.

All in all I agree with what you are saying, but I would rather do it as a follow up possibly once we have some more code on the table. I could just see this opening up a whole new can of worms where we debate about exactly how expectations and assertions work for another several months, only to rip it all out shortly there after. I know that's how these things go, but that's my preference.

I can do what you suggest if you feel strongly about it, but I would prefer to hold off until later. It's your call.

Quoted 3 lines

Why not build in the structured part into the framework (i.e. the struct above) so that it's always there and then add the string building part later when we have to?

See above comments.

#24 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoting Brendan Higgins (2019-07-16 11:52:01)

Quoted 19 lines · Stephen Boyd
Quoted 7 lines

The only hypothetical case where this can't be done is a complicated assertion or expectation that does more than one check and can't be written as a function that dumps out what went wrong. Is this a real problem? Maybe such an assertion should just open code that logic so we don't have to build up a string for all the other simple cases.

I have some expectations in follow up patchsets for which I created a set of composable matchers for matching structures and function calls that by their nature cannot be written as a single function. The matcher thing is a bit speculative, I know, but for any kind of function call matching, you need to store a record of functions you are expecting to have called and then each one needs to have a set of expectations defined by the user; I don't think there is a way to do that that doesn't involve having multiple separate functions each having some information useful to constructing the message.

Quoted 6 lines

I know the code in question isn't in this patchset; the function matching code was in one of the earlier versions of the RFC, but I dropped it to make this patchset smaller and more manageable. So I get it if you would like me to drop it and add it back in when I try to get the function and structure matching stuff in, but I would really prefer to keep it as is if you don't care too much.

Do you have a link to those earlier patches?

Quoted 48 lines
Quoted 22 lines

It seems far simpler to get rid of the string stream API and just have a struct for this.

        struct kunit_fail_msg {
                const char *line;
                const char *file;
                const char *func;
                const char *msg;
        };

Then you can have the assertion macros create this on the stack (with another macro?).

        #define DEFINE_KUNIT_FAIL_MSG(name, _msg) \
                struct kunit_fail_msg name = { \
                        .line =  __LINE__, \
                        .file = __FILE__, \
                        .func = __func__, \
                        .msg = _msg, \
                }
Quoted 5 lines

I don't want to derail this whole series on this topic, but it seems like a bunch of code is there to construct this same set of information over and over again into a buffer a little bit at a time and then throw it away when nothing fails just because we may want to support the case where we have some unstructured data to inform the user about.

Yeah, that's fair. I think there are a number of improvements to be made with how the expectations are defined other than that, but I was hoping I could do that after this patchset is merged. I just figured with the kinds of things I would like to do, it would lead to a whole new round of discussion.

In either case, I think I would still like to use the struct kunit_stream as part of the interface to share the failure message with the test case runner code in test.c, at least eventually, so that I only have to have one way to receive data from expectations, but I think I can do that and still do what you suggest by just constructing the kunit_stream at the end of expectations where it is feasible.

All in all I agree with what you are saying, but I would rather do it as a follow up possibly once we have some more code on the table. I could just see this opening up a whole new can of worms where we debate about exactly how expectations and assertions work for another several months, only to rip it all out shortly there after. I know that's how these things go, but that's my preference.

Quoted 3 lines

I can do what you suggest if you feel strongly about it, but I would prefer to hold off until later. It's your call.

The crux of my complaint is that the string stream API is too loosely defined to be usable. It allows tests to build up a string of unstructured information, but with certain calling constraints so we have to tread carefully. If there was more structure to the data that's being recorded then the test case runner could operate on the data without having to do string/stream operations, allocations, etc. This would make the assertion logic much more concrete and specific to kunit, instead of this small kunit wrapper that's been placed on top of string stream.

TL;DR: If we can get rid of the string stream API I'd view that as an improvement because building arbitrary strings in the kernel is complex, error prone and has calling context concerns.

Is the intention that other code besides unit tests will use this string stream API to build up strings? Any targets in mind? This would be a good way to get the API merged upstream given that its 2019 and we haven't had such an API in the kernel so far.

An "object oriented" (strong quotes!) approach where kunit_fail_msg is the innermost struct in some assertion specific structure might work nicely and allow the test runner to call a generic 'format' function to print out the message based on the type of assertion/expectation it is. It probably would mean less code size too because the strings that are common will be in the common printing function instead of created twice, in the macros/code and then copied to the heap for the string stream.

	struct kunit_assert {
		const char *line;
		const char *file;
		const char *func;
		void (*format)(struct kunit_assert *assert);
	};

	struct kunit_comparison_assert {
		enum operator operator;
		const char *left;
		const char *right;
		struct kunit_assert assert;
	};

	struct kunit_bool_assert {
		const char *truth;
		const char *statement;
		struct kunit_assert assert;
	};

	void kunit_format_comparison(struct kunit_assert *assert)
	{
		struct kunit_comparison_assert *comp = container_of(assert, ...)

		kunit_printk(...)
	}

Maybe other people have opinions here on if you should do it now or later. Future coding is not a great argument because it's hard to predict the future. On the other hand, this patchset is in good shape to merge and I'd like to use it to write unit tests for code I maintain so I don't want to see this stall out. Sorry if I'm opening the can of worms you're talking about.

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#25 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 21 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-16 11:52:01)

Quoted 19 lines · Stephen Boyd
Quoted 7 lines

The only hypothetical case where this can't be done is a complicated assertion or expectation that does more than one check and can't be written as a function that dumps out what went wrong. Is this a real problem? Maybe such an assertion should just open code that logic so we don't have to build up a string for all the other simple cases.

I have some expectations in follow up patchsets for which I created a set of composable matchers for matching structures and function calls that by their nature cannot be written as a single function. The matcher thing is a bit speculative, I know, but for any kind of function call matching, you need to store a record of functions you are expecting to have called and then each one needs to have a set of expectations defined by the user; I don't think there is a way to do that that doesn't involve having multiple separate functions each having some information useful to constructing the message.

Quoted 8 lines
Quoted 6 lines

I know the code in question isn't in this patchset; the function matching code was in one of the earlier versions of the RFC, but I dropped it to make this patchset smaller and more manageable. So I get it if you would like me to drop it and add it back in when I try to get the function and structure matching stuff in, but I would really prefer to keep it as is if you don't care too much.

Do you have a link to those earlier patches?

This is the first patchset:

https://www.mail-archive.com/[email protected]/msg1788057.html

In particular you can see the code for matching functions here:

https://www.mail-archive.com/[email protected]/msg1788073.html

And parameter matching code here:

https://www.mail-archive.com/[email protected]/msg1788072.html

https://www.mail-archive.com/[email protected]/msg1788086.html

My apologies in advance, but the code at this early stage had not adopted the kunit_* prefix and was still using the test_* and mock_* prefix. (Hence, struct kunit_stream was known as struct test_stream).

Quoted 51 lines
Quoted 47 lines
Quoted 21 lines

It seems far simpler to get rid of the string stream API and just have a struct for this.

        struct kunit_fail_msg {
                const char *line;
                const char *file;
                const char *func;
                const char *msg;
        };

Then you can have the assertion macros create this on the stack (with another macro?).

        #define DEFINE_KUNIT_FAIL_MSG(name, _msg) \
                struct kunit_fail_msg name = { \
                        .line =  __LINE__, \
                        .file = __FILE__, \
                        .func = __func__, \
                        .msg = _msg, \
                }
Quoted 5 lines

I don't want to derail this whole series on this topic, but it seems like a bunch of code is there to construct this same set of information over and over again into a buffer a little bit at a time and then throw it away when nothing fails just because we may want to support the case where we have some unstructured data to inform the user about.

Yeah, that's fair. I think there are a number of improvements to be made with how the expectations are defined other than that, but I was hoping I could do that after this patchset is merged. I just figured with the kinds of things I would like to do, it would lead to a whole new round of discussion.

In either case, I think I would still like to use the struct kunit_stream as part of the interface to share the failure message with the test case runner code in test.c, at least eventually, so that I only have to have one way to receive data from expectations, but I think I can do that and still do what you suggest by just constructing the kunit_stream at the end of expectations where it is feasible.

All in all I agree with what you are saying, but I would rather do it as a follow up possibly once we have some more code on the table. I could just see this opening up a whole new can of worms where we debate about exactly how expectations and assertions work for another several months, only to rip it all out shortly there after. I know that's how these things go, but that's my preference.

Quoted 3 lines

I can do what you suggest if you feel strongly about it, but I would prefer to hold off until later. It's your call.

Quoted 9 lines

The crux of my complaint is that the string stream API is too loosely defined to be usable. It allows tests to build up a string of unstructured information, but with certain calling constraints so we have to tread carefully. If there was more structure to the data that's being recorded then the test case runner could operate on the data without having to do string/stream operations, allocations, etc. This would make the assertion logic much more concrete and specific to kunit, instead of this small kunit wrapper that's been placed on top of string stream.

Yeah, I can see the point of wanting something that provides more structure than the raw struct kunit_stream interface. In fact, it is something I had already started working on, when I had determined it would be a large effort to capture all the variations. I was further put off from the idea when I had been asked to convert the KUnit intermediate format from what I was using to TAP, because, as it is, the current data printed out by KUnit doesn't contain all the data I would like to put in it in a way that best takes advantage of the TAP specification. One problematic area in particular: TAP already provides a way to present a lot of the data I would like to export, but it involves JSON serialization which was an idea that some of the other reviewers understandably weren't too keen on. TAP also wants to report data some time after it is available, which is generally not a good idea for test debug information; you want to make it available as soon as you can or you risk crashing with the data still inside.

Hence, I decided we could probably spend a good long while debating how I present the information. So the idea of having a loose definition seemed attractive to me in its own right since it would likely conform to whatever we ended up deciding in the long run. Also, all the better that it was what I already had and no one seemed to mind too much.

The only constant I expect is that struct kunit will likely need to take an abstract object with a commit method, or a format method or whatever so it could control when data was going to be printed out to the user. We will probably also use a string builder in there somewhere.

Quoted 3 lines

TL;DR: If we can get rid of the string stream API I'd view that as an improvement because building arbitrary strings in the kernel is complex, error prone and has calling context concerns.

True. No argument there.

Quoted 4 lines

Is the intention that other code besides unit tests will use this string stream API to build up strings? Any targets in mind? This would be a good way to get the API merged upstream given that its 2019 and we haven't had such an API in the kernel so far.

Someone, (was it you?) asked about code sharing with a string builder thingy that was used for creating structured human readable files, but that seemed like a pretty massive undertaking.

Aside from that, no. I would kind of prefered that nobody used it for anything else because I the issues you described.

Nevertheless, I think the debate over the usefulness of the string_stream and kunit_stream are separate topics. Even if we made kunit_stream more structured, I am pretty sure I would want to use string_stream or some variation for constructing the message.

Quoted 26 lines

An "object oriented" (strong quotes!) approach where kunit_fail_msg is the innermost struct in some assertion specific structure might work nicely and allow the test runner to call a generic 'format' function to print out the message based on the type of assertion/expectation it is. It probably would mean less code size too because the strings that are common will be in the common printing function instead of created twice, in the macros/code and then copied to the heap for the string stream.

        struct kunit_assert {
                const char *line;
                const char *file;
                const char *func;
                void (*format)(struct kunit_assert *assert);
        };

        struct kunit_comparison_assert {
                enum operator operator;
                const char *left;
                const char *right;
                struct kunit_assert assert;
        };

        struct kunit_bool_assert {
                const char *truth;
                const char *statement;
                struct kunit_assert assert;
Quoted 8 lines
        };

        void kunit_format_comparison(struct kunit_assert *assert)
        {
                struct kunit_comparison_assert *comp = container_of(assert, ...)

                kunit_printk(...)
        }

I started working on something similarish, but by the time I ended up coming up with a parent object whose definition was loose enough to satisfy all the properties required by the child classes it ended up basically being the same as what I have now just with a more complex hierarchy of message manipulation logic.

On the other hand, I didn't have the idea of doing the parent object quite the way you did and that would clean up a lot of the duplicated first line logic.

I would like to give it a try, but I am afraid I am going to get sucked down a really deep rabbit hole.

Quoted 3 lines

Maybe other people have opinions here on if you should do it now or later. Future coding is not a great argument because it's hard to predict the future. On the other hand, this patchset is in good shape to

Yeah, that's kind of why I am afraid to go down this road when I have something that works now and I know works with the mocking stuff I want to do.

I would like to try your suggestion, but I want to try to make it work with my mocking patches before I commit to it because otherwise I am just going to have to back it out in a follow up patchset.

Quoted 3 lines

merge and I'd like to use it to write unit tests for code I maintain so I don't want to see this stall out. Sorry if I'm opening the can of worms you're talking about.

Don't be sorry. I agree with you that the kunit_stream stuff is not very pretty.

Shuah, have we missed the merge window for 5.3?

I saw you only sent one PR out so far for this release, and there wasn't much in it; I imagine you are going to send at least one more?

I figure, if we still got time to try out your suggestion, Stephen, no harm in trying.

Also if we missed it, then I have another couple months to play around with it.

What do you think?

#26 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 4 lines · Brendan Higgins
Quoted 3 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-16 11:52:01)

Quoted 1 line · Stephen Boyd

[...]

Quoted 19 lines
Quoted 1 line

Do you have a link to those earlier patches?

This is the first patchset:

https://www.mail-archive.com/[email protected]/msg1788057.html

In particular you can see the code for matching functions here:

https://www.mail-archive.com/[email protected]/msg1788073.html

And parameter matching code here:

https://www.mail-archive.com/[email protected]/msg1788072.html

https://www.mail-archive.com/[email protected]/msg1788086.html

My apologies in advance, but the code at this early stage had not adopted the kunit_* prefix and was still using the test_* and mock_* prefix. (Hence, struct kunit_stream was known as struct test_stream).

[...]

Quoted 54 lines
Quoted 9 lines

The crux of my complaint is that the string stream API is too loosely defined to be usable. It allows tests to build up a string of unstructured information, but with certain calling constraints so we have to tread carefully. If there was more structure to the data that's being recorded then the test case runner could operate on the data without having to do string/stream operations, allocations, etc. This would make the assertion logic much more concrete and specific to kunit, instead of this small kunit wrapper that's been placed on top of string stream.

Yeah, I can see the point of wanting something that provides more structure than the raw struct kunit_stream interface. In fact, it is something I had already started working on, when I had determined it would be a large effort to capture all the variations. I was further put off from the idea when I had been asked to convert the KUnit intermediate format from what I was using to TAP, because, as it is, the current data printed out by KUnit doesn't contain all the data I would like to put in it in a way that best takes advantage of the TAP specification. One problematic area in particular: TAP already provides a way to present a lot of the data I would like to export, but it involves JSON serialization which was an idea that some of the other reviewers understandably weren't too keen on. TAP also wants to report data some time after it is available, which is generally not a good idea for test debug information; you want to make it available as soon as you can or you risk crashing with the data still inside.

Hence, I decided we could probably spend a good long while debating how I present the information. So the idea of having a loose definition seemed attractive to me in its own right since it would likely conform to whatever we ended up deciding in the long run. Also, all the better that it was what I already had and no one seemed to mind too much.

The only constant I expect is that struct kunit will likely need to take an abstract object with a commit method, or a format method or whatever so it could control when data was going to be printed out to the user. We will probably also use a string builder in there somewhere.

Quoted 3 lines

TL;DR: If we can get rid of the string stream API I'd view that as an improvement because building arbitrary strings in the kernel is complex, error prone and has calling context concerns.

True. No argument there.

Quoted 4 lines

Is the intention that other code besides unit tests will use this string stream API to build up strings? Any targets in mind? This would be a good way to get the API merged upstream given that its 2019 and we haven't had such an API in the kernel so far.

Someone, (was it you?) asked about code sharing with a string builder thingy that was used for creating structured human readable files, but that seemed like a pretty massive undertaking.

Quoted 42 lines

Aside from that, no. I would kind of prefered that nobody used it for anything else because I the issues you described.

Nevertheless, I think the debate over the usefulness of the string_stream and kunit_stream are separate topics. Even if we made kunit_stream more structured, I am pretty sure I would want to use string_stream or some variation for constructing the message.

Quoted 34 lines

An "object oriented" (strong quotes!) approach where kunit_fail_msg is the innermost struct in some assertion specific structure might work nicely and allow the test runner to call a generic 'format' function to print out the message based on the type of assertion/expectation it is. It probably would mean less code size too because the strings that are common will be in the common printing function instead of created twice, in the macros/code and then copied to the heap for the string stream.

        struct kunit_assert {
                const char *line;
                const char *file;
                const char *func;
                void (*format)(struct kunit_assert *assert);
        };

        struct kunit_comparison_assert {
                enum operator operator;
                const char *left;
                const char *right;
                struct kunit_assert assert;
        };

        struct kunit_bool_assert {
                const char *truth;
                const char *statement;
                struct kunit_assert assert;
        };

        void kunit_format_comparison(struct kunit_assert *assert)
        {
                struct kunit_comparison_assert *comp = container_of(assert, ...)

                kunit_printk(...)
        }

I started poking around with your suggestion while we are waiting. A couple early observations:

1) It is actually easier to do than I previously thought and will probably help with getting more of the planned TAP output stuff working.

That being said, this is still a pretty substantial undertaking and will likely take *at least* a week to implement and properly review. Assuming everything goes extremely well (no unexpected issues on my end, very responsive reviewers, etc).

2) It *will* eliminate the need for kunit_stream.

3) ...but, it *will not* eliminate the need for string_stream.

Based on my early observations, I do think it is worth doing, but I don't think it is worth trying to make it in this patchset (unless I have already missed the window, or it is going to be open for a while):
I do think it will make things much cleaner, but I don't think it will achieve your desired goal of getting rid of an unstructured {kunit|string}_stream style interface; it just adds a layer on top of it that makes it harder to misuse.

I attached a patch of what I have so far at the end of this email so you can see what I am talking about. And of course, if you agree with my assessment, so we can start working on it as a future patch.

A couple things in regard to the patch I attached:

1) I wrote it pretty quickly so there are almost definitely mistakes. You should consider it RFC. I did verify it compiles though.

2) Also, I did use kunit_stream in writing it: all occurences should be pretty easy to replace with string_stream; nevertheless, the reason for this is just to make it easier to play with the current APIs. I wanted to have something working before I went through a big tedious refactoring. So sorry if it causes any confusion.

3) I also based the patch on all the KUnit patches I have queued up (includes things like mocking and such) since I want to see how this serialization thing will work with mocks and matchers and things like that.

Quoted 36 lines

I started working on something similarish, but by the time I ended up coming up with a parent object whose definition was loose enough to satisfy all the properties required by the child classes it ended up basically being the same as what I have now just with a more complex hierarchy of message manipulation logic.

On the other hand, I didn't have the idea of doing the parent object quite the way you did and that would clean up a lot of the duplicated first line logic.

I would like to give it a try, but I am afraid I am going to get sucked down a really deep rabbit hole.

Quoted 3 lines

Maybe other people have opinions here on if you should do it now or later. Future coding is not a great argument because it's hard to predict the future. On the other hand, this patchset is in good shape to

Yeah, that's kind of why I am afraid to go down this road when I have something that works now and I know works with the mocking stuff I want to do.

I would like to try your suggestion, but I want to try to make it work with my mocking patches before I commit to it because otherwise I am just going to have to back it out in a follow up patchset.

Quoted 3 lines

merge and I'd like to use it to write unit tests for code I maintain so I don't want to see this stall out. Sorry if I'm opening the can of worms you're talking about.

Don't be sorry. I agree with you that the kunit_stream stuff is not very pretty.

Shuah, have we missed the merge window for 5.3?

I saw you only sent one PR out so far for this release, and there wasn't much in it; I imagine you are going to send at least one more?

Quoted 6 lines

I figure, if we still got time to try out your suggestion, Stephen, no harm in trying.

Also if we missed it, then I have another couple months to play around with it.

What do you think?

I attached the patch mentioned above below. Let me know what you think!

Cheers!

Quoted 1 line

From 53d475d3d56afcf92b452c6d347dbedfa1a17d34 Mon Sep 17 00:00:00 2001

From: Brendan Higgins <[email protected]>
Date: Thu, 18 Jul 2019 16:08:52 -0700
Subject: [PATCH v1] DO NOT MERGE: started playing around with the
serialization api

---
 include/kunit/assert.h | 130 ++++++++++++++++++++++++++++++
 include/kunit/mock.h   |   4 +
 kunit/Makefile         |   3 +-
 kunit/assert.c         | 179 +++++++++++++++++++++++++++++++++++++++++
 kunit/mock.c           |   6 +-
 5 files changed, 318 insertions(+), 4 deletions(-)
 create mode 100644 include/kunit/assert.h
 create mode 100644 kunit/assert.c
include/kunit/assert.h new file +130 -0
new file mode 100644
index 0000000000000..e054fdff4642f
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_ASSERT_H
+#define _KUNIT_ASSERT_H
+
+#include <kunit/test.h>
+#include <kunit/mock.h>
+
+enum kunit_assert_type {
+	KUNIT_ASSERTION,
+	KUNIT_EXPECTATION,
+};
+
+struct kunit_assert {
+	enum kunit_assert_type type;
+	const char *line;
+	const char *file;
+	struct va_format message;
+	void (*format)(struct kunit_assert *assert,
+		       struct kunit_stream *stream);
+};
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+			      struct kunit_stream *stream);
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+			    struct kunit_stream *stream);
+
+struct kunit_unary_assert {
+	struct kunit_assert assert;
+	const char *condition;
+	bool expected_true;
+};
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+			       struct kunit_stream *stream);
+
+struct kunit_ptr_not_err_assert {
+	struct kunit_assert assert;
+	const char *text;
+	void *value;
+};
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+				     struct kunit_stream *stream);
+
+struct kunit_binary_assert {
+	struct kunit_assert assert;
+	const char *operation;
+	const char *left_text;
+	long long left_value;
+	const char *right_text;
+	long long right_value;
+};
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+				struct kunit_stream *stream);
+
+struct kunit_binary_ptr_assert {
+	struct kunit_assert assert;
+	const char *operation;
+	const char *left_text;
+	void *left_value;
+	const char *right_text;
+	void *right_value;
+};
+
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+				    struct kunit_stream *stream);
+
+struct kunit_binary_str_assert {
+	struct kunit_assert assert;
+	const char *operation;
+	const char *left_text;
+	const char *left_value;
+	const char *right_text;
+	const char *right_value;
+};
+
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+				    struct kunit_stream *stream);
+
+struct kunit_mock_assert {
+	struct kunit_assert assert;
+};
+
+struct kunit_mock_no_expectations {
+	struct kunit_mock_assert assert;
+};
+
+struct kunit_mock_declaration {
+	const char *function_name;
+	const char **type_names;
+	const void **params;
+	int len;
+};
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,
+				   struct kunit_stream *stream);
+
+struct kunit_matcher_result {
+	struct kunit_assert assert;
+};
+
+struct kunit_mock_failed_match {
+	struct list_head node;
+	const char *expectation_text;
+	struct kunit_matcher_result *matcher_list;
+	size_t matcher_list_len;
+};
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+				    struct kunit_stream *stream);
+
+struct kunit_mock_no_match {
+	struct kunit_mock_assert assert;
+	struct kunit_mock_declaration declaration;
+	struct list_head failed_match_list;
+};
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+				struct kunit_stream *stream);
+
+#endif /*  _KUNIT_ASSERT_H */
include/kunit/mock.h +4 -0
index 001b96af62f1e..52c9e427c831b 100644
@@ -144,6 +144,10 @@ void mock_register_formatter(struct mock_param_formatter *formatter);
 
 void mock_unregister_formatter(struct mock_param_formatter *formatter);
 
+void mock_format_param(struct kunit_stream *stream,
+		       const char *type_name,
+		       const void *param);
+
 struct mock *mock_get_global_mock(void);
 
 #define MOCK(name) name##_mock
kunit/Makefile +2 -1
index bbf43fcfb93a9..149d856a30f04 100644
@@ -3,7 +3,8 @@ obj-$(CONFIG_KUNIT) +=			test.o \
 					common-mocks.o \
 					string-stream.o \
 					kunit-stream.o \
-					try-catch.o
+					try-catch.o \
+					assert.o
 
 obj-$(CONFIG_KUNIT_TEST) +=		test-test.o \
 					test-mock.o \
kunit/assert.c new file +179 -0

179 lines not on this page · raw message

kunit/mock.c +3 -3
index ccb0abe111402..ab441a58a918c 100644
@@ -269,9 +269,9 @@ struct mock_param_formatter *mock_find_formatter(const char *type_name)
 	return NULL;
 }
 
-static void mock_format_param(struct kunit_stream *stream,
-			      const char *type_name,
-			      const void *param)
+void mock_format_param(struct kunit_stream *stream,
+		       const char *type_name,
+		       const void *param)
 {
 	struct mock_param_formatter *formatter;
 
Signature · 1 line
2.22.0.657.g960e92d24f-goog
#27 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 162 lines · Brendan Higgins
Quoted 4 lines · Brendan Higgins
Quoted 3 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-16 11:52:01)

Quoted 1 line · Stephen Boyd

[...]

Quoted 19 lines
Quoted 1 line

Do you have a link to those earlier patches?

This is the first patchset:

https://www.mail-archive.com/[email protected]/msg1788057.html

In particular you can see the code for matching functions here:

https://www.mail-archive.com/[email protected]/msg1788073.html

And parameter matching code here:

https://www.mail-archive.com/[email protected]/msg1788072.html

https://www.mail-archive.com/[email protected]/msg1788086.html

My apologies in advance, but the code at this early stage had not adopted the kunit_* prefix and was still using the test_* and mock_* prefix. (Hence, struct kunit_stream was known as struct test_stream).

[...]

Quoted 54 lines
Quoted 9 lines

The crux of my complaint is that the string stream API is too loosely defined to be usable. It allows tests to build up a string of unstructured information, but with certain calling constraints so we have to tread carefully. If there was more structure to the data that's being recorded then the test case runner could operate on the data without having to do string/stream operations, allocations, etc. This would make the assertion logic much more concrete and specific to kunit, instead of this small kunit wrapper that's been placed on top of string stream.

Yeah, I can see the point of wanting something that provides more structure than the raw struct kunit_stream interface. In fact, it is something I had already started working on, when I had determined it would be a large effort to capture all the variations. I was further put off from the idea when I had been asked to convert the KUnit intermediate format from what I was using to TAP, because, as it is, the current data printed out by KUnit doesn't contain all the data I would like to put in it in a way that best takes advantage of the TAP specification. One problematic area in particular: TAP already provides a way to present a lot of the data I would like to export, but it involves JSON serialization which was an idea that some of the other reviewers understandably weren't too keen on. TAP also wants to report data some time after it is available, which is generally not a good idea for test debug information; you want to make it available as soon as you can or you risk crashing with the data still inside.

Hence, I decided we could probably spend a good long while debating how I present the information. So the idea of having a loose definition seemed attractive to me in its own right since it would likely conform to whatever we ended up deciding in the long run. Also, all the better that it was what I already had and no one seemed to mind too much.

The only constant I expect is that struct kunit will likely need to take an abstract object with a commit method, or a format method or whatever so it could control when data was going to be printed out to the user. We will probably also use a string builder in there somewhere.

Quoted 3 lines

TL;DR: If we can get rid of the string stream API I'd view that as an improvement because building arbitrary strings in the kernel is complex, error prone and has calling context concerns.

True. No argument there.

Quoted 4 lines

Is the intention that other code besides unit tests will use this string stream API to build up strings? Any targets in mind? This would be a good way to get the API merged upstream given that its 2019 and we haven't had such an API in the kernel so far.

Someone, (was it you?) asked about code sharing with a string builder thingy that was used for creating structured human readable files, but that seemed like a pretty massive undertaking.

Quoted 42 lines

Aside from that, no. I would kind of prefered that nobody used it for anything else because I the issues you described.

Nevertheless, I think the debate over the usefulness of the string_stream and kunit_stream are separate topics. Even if we made kunit_stream more structured, I am pretty sure I would want to use string_stream or some variation for constructing the message.

Quoted 34 lines

An "object oriented" (strong quotes!) approach where kunit_fail_msg is the innermost struct in some assertion specific structure might work nicely and allow the test runner to call a generic 'format' function to print out the message based on the type of assertion/expectation it is. It probably would mean less code size too because the strings that are common will be in the common printing function instead of created twice, in the macros/code and then copied to the heap for the string stream.

        struct kunit_assert {
                const char *line;
                const char *file;
                const char *func;
                void (*format)(struct kunit_assert *assert);
        };

        struct kunit_comparison_assert {
                enum operator operator;
                const char *left;
                const char *right;
                struct kunit_assert assert;
        };

        struct kunit_bool_assert {
                const char *truth;
                const char *statement;
                struct kunit_assert assert;
        };

        void kunit_format_comparison(struct kunit_assert *assert)
        {
                struct kunit_comparison_assert *comp = container_of(assert, ...)

                kunit_printk(...)
        }

I started poking around with your suggestion while we are waiting. A couple early observations:

1) It is actually easier to do than I previously thought and will probably help with getting more of the planned TAP output stuff working.

That being said, this is still a pretty substantial undertaking and will likely take *at least* a week to implement and properly review. Assuming everything goes extremely well (no unexpected issues on my end, very responsive reviewers, etc).

2) It *will* eliminate the need for kunit_stream.

3) ...but, it *will not* eliminate the need for string_stream.

Based on my early observations, I do think it is worth doing, but I don't think it is worth trying to make it in this patchset (unless I have already missed the window, or it is going to be open for a while):
I do think it will make things much cleaner, but I don't think it will achieve your desired goal of getting rid of an unstructured {kunit|string}_stream style interface; it just adds a layer on top of it that makes it harder to misuse.

I attached a patch of what I have so far at the end of this email so you can see what I am talking about. And of course, if you agree with my assessment, so we can start working on it as a future patch.

A couple things in regard to the patch I attached:

1) I wrote it pretty quickly so there are almost definitely mistakes. You should consider it RFC. I did verify it compiles though.

2) Also, I did use kunit_stream in writing it: all occurences should be pretty easy to replace with string_stream; nevertheless, the reason for this is just to make it easier to play with the current APIs. I wanted to have something working before I went through a big tedious refactoring. So sorry if it causes any confusion.

Quoted 47 lines

3) I also based the patch on all the KUnit patches I have queued up (includes things like mocking and such) since I want to see how this serialization thing will work with mocks and matchers and things like that.

Quoted 42 lines

I started working on something similarish, but by the time I ended up coming up with a parent object whose definition was loose enough to satisfy all the properties required by the child classes it ended up basically being the same as what I have now just with a more complex hierarchy of message manipulation logic.

On the other hand, I didn't have the idea of doing the parent object quite the way you did and that would clean up a lot of the duplicated first line logic.

I would like to give it a try, but I am afraid I am going to get sucked down a really deep rabbit hole.

Quoted 3 lines

Maybe other people have opinions here on if you should do it now or later. Future coding is not a great argument because it's hard to predict the future. On the other hand, this patchset is in good shape to

Yeah, that's kind of why I am afraid to go down this road when I have something that works now and I know works with the mocking stuff I want to do.

I would like to try your suggestion, but I want to try to make it work with my mocking patches before I commit to it because otherwise I am just going to have to back it out in a follow up patchset.

Quoted 3 lines

merge and I'd like to use it to write unit tests for code I maintain so I don't want to see this stall out. Sorry if I'm opening the can of worms you're talking about.

Don't be sorry. I agree with you that the kunit_stream stuff is not very pretty.

Shuah, have we missed the merge window for 5.3?

I saw you only sent one PR out so far for this release, and there wasn't much in it; I imagine you are going to send at least one more?

I figure, if we still got time to try out your suggestion, Stephen, no harm in trying.

Also if we missed it, then I have another couple months to play around with it.

What do you think?

I talked to Shuah off thread, she would like us to resolve this discussion before accepting the patchset.

She also said that this is probably going to have to wait until v5.4.

Nevertheless, Stephen, would you mind taking a look at the patch I posted below? I would like to get your thoughts on the sum of all the changes I am going to have to make before I try to integrate them into the existing patches.

Sorry for being lazy, but I suspect you won't like the first pass of how I am doing it, and I think it will probably be easier for you to give early feedback on it as its own change anyway.

Quoted 270 lines

I attached the patch mentioned above below. Let me know what you think!

Cheers!

From 53d475d3d56afcf92b452c6d347dbedfa1a17d34 Mon Sep 17 00:00:00 2001
From: Brendan Higgins <[email protected]>
Date: Thu, 18 Jul 2019 16:08:52 -0700
Subject: [PATCH v1] DO NOT MERGE: started playing around with the
serialization api

---
 include/kunit/assert.h | 130 ++++++++++++++++++++++++++++++
 include/kunit/mock.h   |   4 +
 kunit/Makefile         |   3 +-
 kunit/assert.c         | 179 +++++++++++++++++++++++++++++++++++++++++
 kunit/mock.c           |   6 +-
 5 files changed, 318 insertions(+), 4 deletions(-)
 create mode 100644 include/kunit/assert.h
 create mode 100644 kunit/assert.c
include/kunit/assert.h new file +130 -0
new file mode 100644
index 0000000000000..e054fdff4642f
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_ASSERT_H
+#define _KUNIT_ASSERT_H
+
+#include <kunit/test.h>
+#include <kunit/mock.h>
+
+enum kunit_assert_type {
+       KUNIT_ASSERTION,
+       KUNIT_EXPECTATION,
+};
+
+struct kunit_assert {
+       enum kunit_assert_type type;
+       const char *line;
+       const char *file;
+       struct va_format message;
+       void (*format)(struct kunit_assert *assert,
+                      struct kunit_stream *stream);
+};
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+                             struct kunit_stream *stream);
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+                           struct kunit_stream *stream);
+
+struct kunit_unary_assert {
+       struct kunit_assert assert;
+       const char *condition;
+       bool expected_true;
+};
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+                              struct kunit_stream *stream);
+
+struct kunit_ptr_not_err_assert {
+       struct kunit_assert assert;
+       const char *text;
+       void *value;
+};
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+                                    struct kunit_stream *stream);
+
+struct kunit_binary_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       long long left_value;
+       const char *right_text;
+       long long right_value;
+};
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+                               struct kunit_stream *stream);
+
+struct kunit_binary_ptr_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       void *left_value;
+       const char *right_text;
+       void *right_value;
+};
+
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+                                   struct kunit_stream *stream);
+
+struct kunit_binary_str_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       const char *left_value;
+       const char *right_text;
+       const char *right_value;
+};
+
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+                                   struct kunit_stream *stream);
+
+struct kunit_mock_assert {
+       struct kunit_assert assert;
+};
+
+struct kunit_mock_no_expectations {
+       struct kunit_mock_assert assert;
+};
+
+struct kunit_mock_declaration {
+       const char *function_name;
+       const char **type_names;
+       const void **params;
+       int len;
+};
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,
+                                  struct kunit_stream *stream);
+
+struct kunit_matcher_result {
+       struct kunit_assert assert;
+};
+
+struct kunit_mock_failed_match {
+       struct list_head node;
+       const char *expectation_text;
+       struct kunit_matcher_result *matcher_list;
+       size_t matcher_list_len;
+};
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+                                   struct kunit_stream *stream);
+
+struct kunit_mock_no_match {
+       struct kunit_mock_assert assert;
+       struct kunit_mock_declaration declaration;
+       struct list_head failed_match_list;
+};
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+                               struct kunit_stream *stream);
+
+#endif /*  _KUNIT_ASSERT_H */
include/kunit/mock.h +4 -0
index 001b96af62f1e..52c9e427c831b 100644
@@ -144,6 +144,10 @@ void mock_register_formatter(struct mock_param_formatter *formatter);
 
 void mock_unregister_formatter(struct mock_param_formatter *formatter);
 
+void mock_format_param(struct kunit_stream *stream,
+                      const char *type_name,
+                      const void *param);
+
 struct mock *mock_get_global_mock(void);
 
 #define MOCK(name) name##_mock
kunit/Makefile +2 -1
index bbf43fcfb93a9..149d856a30f04 100644
@@ -3,7 +3,8 @@ obj-$(CONFIG_KUNIT) +=                  test.o \
                                        common-mocks.o \
                                        string-stream.o \
                                        kunit-stream.o \
-                                       try-catch.o
+                                       try-catch.o \
+                                       assert.o
 
 obj-$(CONFIG_KUNIT_TEST) +=            test-test.o \
                                        test-mock.o \
kunit/assert.c new file +79 -0
new file mode 100644
index 0000000000000..75bb6922a994e
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+#include <kunit/assert.h>
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+                             struct kunit_stream *stream)
+{
+       const char *expect_or_assert;
+
+       if (assert->type == KUNIT_EXPECTATION)
+               expect_or_assert = "EXPECTATION";
+       else
+               expect_or_assert = "ASSERTION";
+
+       kunit_stream_add(stream, "%s FAILED at %s:%s\n",
+                        expect_or_assert, assert->file, assert->line);
+}
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+                           struct kunit_stream *stream)
+{
+       if (assert->message.fmt)
+               kunit_stream_add(stream, "\n%pV", &assert->message);
+}
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+                              struct kunit_stream *stream)
+{
+       struct kunit_unary_assert *unary_assert = container_of(
+                       assert, struct kunit_unary_assert, assert);
+
+       kunit_base_assert_format(assert, stream);
+       if (unary_assert->expected_true)
+               kunit_stream_add(stream,
+                                "\tExpected %s to be true, but is false\n",
+                                unary_assert->condition);
+       else
+               kunit_stream_add(stream,
+                                "\tExpected %s to be false, but is true\n",
+                                unary_assert->condition);
+       kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+                                    struct kunit_stream *stream)
+{
+       struct kunit_ptr_not_err_assert *ptr_assert = container_of(
+                       assert, struct kunit_ptr_not_err_assert, assert);
+
+       kunit_base_assert_format(assert, stream);
+       if (!ptr_assert->value) {
+               kunit_stream_add(stream,
+                                "\tExpected %s is not null, but is\n",
+                                ptr_assert->text);
+       } else if (IS_ERR(ptr_assert->value)) {
+               kunit_stream_add(stream,
+                                "\tExpected %s is not error, but is: %ld\n",
+                                ptr_assert->text,
+                                PTR_ERR(ptr_assert->value));
+       }
+       kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+                               struct kunit_stream *stream)
+{
+       struct kunit_binary_assert *binary_assert = container_of(
+                       assert, struct kunit_binary_assert, assert);
+
+       kunit_base_assert_format(assert, stream);
+       kunit_stream_add(stream,
+                        "\tExpected %s %s %s, but\n",
+                        binary_assert->left_text,
+                        binary_assert->operation,

kunit/assert.c line 89

Quoted 10 lines
kunit/assert.c new file +10 -0
+                        binary_assert->right_text);
+       kunit_stream_add(stream, "\t\t%s == %lld\n",
+                        binary_assert->left_text,
+                        binary_assert->left_value);
+       kunit_stream_add(stream, "\t\t%s == %lld",
+                        binary_assert->right_text,
+                        binary_assert->right_value);
+       kunit_assert_print_msg(assert, stream);
+}
+

I could probably reduce some of the code duplication here by using a variable type struct for left_value and right_value, but that would actually increase the usage of {kunit|string}_stream; it is probably the right thing to do, but I wanted to get your thoughts on it first.

Quoted 108 lines
kunit/assert.c new file +90 -0
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+                                   struct kunit_stream *stream)
+{
+       struct kunit_binary_ptr_assert *binary_assert = container_of(
+                       assert, struct kunit_binary_ptr_assert, assert);
+
+       kunit_base_assert_format(assert, stream);
+       kunit_stream_add(stream,
+                        "\tExpected %s %s %s, but\n",
+                        binary_assert->left_text,
+                        binary_assert->operation,
+                        binary_assert->right_text);
+       kunit_stream_add(stream, "\t\t%s == %pK\n",
+                        binary_assert->left_text,
+                        binary_assert->left_value);
+       kunit_stream_add(stream, "\t\t%s == %pK",
+                        binary_assert->right_text,
+                        binary_assert->right_value);
+       kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+                                   struct kunit_stream *stream)
+{
+       struct kunit_binary_str_assert *binary_assert = container_of(
+                       assert, struct kunit_binary_str_assert, assert);
+
+       kunit_base_assert_format(assert, stream);
+       kunit_stream_add(stream,
+                        "\tExpected %s %s %s, but\n",
+                        binary_assert->left_text,
+                        binary_assert->operation,
+                        binary_assert->right_text);
+       kunit_stream_add(stream, "\t\t%s == %s\n",
+                        binary_assert->left_text,
+                        binary_assert->left_value);
+       kunit_stream_add(stream, "\t\t%s == %s",
+                        binary_assert->right_text,
+                        binary_assert->right_value);
+       kunit_assert_print_msg(assert, stream);
+}
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,
+                                  struct kunit_stream *stream)
+{
+       int i;
+
+       kunit_stream_add(stream, "%s(", declaration->function_name);
+       for (i = 0; i < declaration->len; i++) {
+               mock_format_param(stream,
+                                 declaration->type_names[i],
+                                 declaration->params[i]);
+               if (i < declaration->len - 1)
+                       kunit_stream_add(stream, ", ");
+       }
+       kunit_stream_add(stream, ")\n");
+}
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+                                   struct kunit_stream *stream)
+{
+       struct kunit_matcher_result *result;
+       size_t i;
+
+       kunit_stream_add(stream,
+                        "Tried expectation: %s, but\n",
+                        match->expectation_text);
+       for (i = 0; i < match->matcher_list_len; i++) {
+               result = &match->matcher_list[i];
+               kunit_stream_add(stream, "\t");
+               result->assert.format(&result->assert, stream);
+               kunit_stream_add(stream, "\n");
+       }
+}
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+                               struct kunit_stream *stream)
+{
+       struct kunit_mock_assert *mock_assert = container_of(
+                       assert, struct kunit_mock_assert, assert);
+       struct kunit_mock_no_match *no_match = container_of(
+                       mock_assert, struct kunit_mock_no_match, assert);
+       struct kunit_mock_failed_match *expectation;
+
+       kunit_base_assert_format(assert, stream);
+       kunit_mock_declaration_format(&no_match->declaration, stream);
+
+       list_for_each_entry(expectation, &no_match->failed_match_list, node)
+               kunit_mock_failed_match_format(expectation, stream);
+}
kunit/mock.c +3 -4
index ccb0abe111402..ab441a58a918c 100644
@@ -269,9 +269,9 @@ struct mock_param_formatter *mock_find_formatter(const char *type_name)
        return NULL;
 }
 
-static void mock_format_param(struct kunit_stream *stream,
-                             const char *type_name,
-                             const void *param)
+void mock_format_param(struct kunit_stream *stream,
+                      const char *type_name,
+                      const void *param)
 {
        struct mock_param_formatter *formatter;
 
--
Quoted 2 lines

2.22.0.657.g960e92d24f-goog

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#28 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

In reply to Brendan Higgins

Quoting Brendan Higgins (2019-07-18 17:08:34)

Quoted 13 lines · Brendan Higgins

I started poking around with your suggestion while we are waiting. A couple early observations:

1) It is actually easier to do than I previously thought and will probably help with getting more of the planned TAP output stuff working.

That being said, this is still a pretty substantial undertaking and will likely take *at least* a week to implement and properly review. Assuming everything goes extremely well (no unexpected issues on my end, very responsive reviewers, etc).

Quoted 7 lines

2) It *will* eliminate the need for kunit_stream.

3) ...but, it *will not* eliminate the need for string_stream.

Based on my early observations, I do think it is worth doing, but I don't think it is worth trying to make it in this patchset (unless I have already missed the window, or it is going to be open for a while):

The merge window is over. Typically code needs to be settled a few weeks before it opens (i.e. around -rc4 or -rc5) for most maintainers to pick up patches for the next merge window.

Quoted 4 lines

I do think it will make things much cleaner, but I don't think it will achieve your desired goal of getting rid of an unstructured {kunit|string}_stream style interface; it just adds a layer on top of it that makes it harder to misuse.

Ok.

Quoted 16 lines

I attached a patch of what I have so far at the end of this email so you can see what I am talking about. And of course, if you agree with my assessment, so we can start working on it as a future patch.

A couple things in regard to the patch I attached:

1) I wrote it pretty quickly so there are almost definitely mistakes. You should consider it RFC. I did verify it compiles though.

2) Also, I did use kunit_stream in writing it: all occurences should be pretty easy to replace with string_stream; nevertheless, the reason for this is just to make it easier to play with the current APIs. I wanted to have something working before I went through a big tedious refactoring. So sorry if it causes any confusion.

Quoted 4 lines

3) I also based the patch on all the KUnit patches I have queued up (includes things like mocking and such) since I want to see how this serialization thing will work with mocks and matchers and things like that.

Great!

Quoted 39 lines

From 53d475d3d56afcf92b452c6d347dbedfa1a17d34 Mon Sep 17 00:00:00 2001
From: Brendan Higgins <[email protected]>
Date: Thu, 18 Jul 2019 16:08:52 -0700
Subject: [PATCH v1] DO NOT MERGE: started playing around with the
serialization api

---
 include/kunit/assert.h | 130 ++++++++++++++++++++++++++++++
 include/kunit/mock.h   |   4 +
 kunit/Makefile         |   3 +-
 kunit/assert.c         | 179 +++++++++++++++++++++++++++++++++++++++++
 kunit/mock.c           |   6 +-
 5 files changed, 318 insertions(+), 4 deletions(-)
 create mode 100644 include/kunit/assert.h
 create mode 100644 kunit/assert.c
include/kunit/assert.h new file +16 -0
new file mode 100644
index 0000000000000..e054fdff4642f
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_ASSERT_H
+#define _KUNIT_ASSERT_H
+
+#include <kunit/test.h>
+#include <kunit/mock.h>
+
+enum kunit_assert_type {
+       KUNIT_ASSERTION,

include/kunit/assert.h line 26

Quoted 10 lines
include/kunit/assert.h new file +10 -0
+       KUNIT_EXPECTATION,
+};
+
+struct kunit_assert {
+       enum kunit_assert_type type;
+       const char *line;
+       const char *file;
+       struct va_format message;
+       void (*format)(struct kunit_assert *assert,
+                      struct kunit_stream *stream);

Would passing in the test help too?

Quoted 59 lines
include/kunit/assert.h new file +59 -0
+};
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+                             struct kunit_stream *stream);
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+                           struct kunit_stream *stream);
+
+struct kunit_unary_assert {
+       struct kunit_assert assert;
+       const char *condition;
+       bool expected_true;
+};
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+                              struct kunit_stream *stream);
+
+struct kunit_ptr_not_err_assert {
+       struct kunit_assert assert;
+       const char *text;
+       void *value;
+};
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+                                    struct kunit_stream *stream);
+
+struct kunit_binary_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       long long left_value;
+       const char *right_text;
+       long long right_value;
+};
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+                               struct kunit_stream *stream);
+
+struct kunit_binary_ptr_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       void *left_value;
+       const char *right_text;
+       void *right_value;
+};
+
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+                                   struct kunit_stream *stream);
+
+struct kunit_binary_str_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       const char *left_value;
+       const char *right_text;
+       const char *right_value;
+};
+

include/kunit/assert.h line 95

Quoted 10 lines
include/kunit/assert.h new file +10 -0
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+                                   struct kunit_stream *stream);
+
+struct kunit_mock_assert {
+       struct kunit_assert assert;
+};
+
+struct kunit_mock_no_expectations {
+       struct kunit_mock_assert assert;
+};

What's the purpose of making a wrapper struct with no other members? Just to make a different struct for some sort of type checking? I guess it's OK but I don't think it will be very useful in practice.

Quoted 9 lines
include/kunit/assert.h new file +9 -0
+
+struct kunit_mock_declaration {
+       const char *function_name;
+       const char **type_names;
+       const void **params;
+       int len;
+};
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,

include/kunit/assert.h line 114

Quoted 10 lines
include/kunit/assert.h new file +10 -0
+                                  struct kunit_stream *stream);
+
+struct kunit_matcher_result {
+       struct kunit_assert assert;
+};
+
+struct kunit_mock_failed_match {
+       struct list_head node;
+       const char *expectation_text;
+       struct kunit_matcher_result *matcher_list;

Minor nitpick: this code could use some const sprinkling.

Quoted 30 lines
include/kunit/assert.h new file +16 -0
+       size_t matcher_list_len;
+};
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+                                   struct kunit_stream *stream);
+
+struct kunit_mock_no_match {
+       struct kunit_mock_assert assert;
+       struct kunit_mock_declaration declaration;
+       struct list_head failed_match_list;
+};
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+                               struct kunit_stream *stream);
+
+#endif /*  _KUNIT_ASSERT_H */
kunit/assert.c new file +8 -0
new file mode 100644
index 0000000000000..75bb6922a994e
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+#include <kunit/assert.h>

kunit/assert.c line 18

Quoted 10 lines
kunit/assert.c new file +10 -0
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+                             struct kunit_stream *stream)
+{
+       const char *expect_or_assert;
+
+       if (assert->type == KUNIT_EXPECTATION)
+               expect_or_assert = "EXPECTATION";
+       else
+               expect_or_assert = "ASSERTION";

Make this is a switch statement so we can have the compiler complain if an enum is missing.

Quoted 12 lines
kunit/assert.c new file +12 -0
+
+       kunit_stream_add(stream, "%s FAILED at %s:%s\n",
+                        expect_or_assert, assert->file, assert->line);
+}
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+                           struct kunit_stream *stream)
+{
+       if (assert->message.fmt)
+               kunit_stream_add(stream, "\n%pV", &assert->message);
+}
+

[...]

Quoted 6 lines
kunit/assert.c new file +6 -0
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+                                   struct kunit_stream *stream)
+{
+       struct kunit_matcher_result *result;
+       size_t i;

kunit/assert.c line 162

Quoted 10 lines
kunit/assert.c new file +10 -0
+
+       kunit_stream_add(stream,
+                        "Tried expectation: %s, but\n",
+                        match->expectation_text);
+       for (i = 0; i < match->matcher_list_len; i++) {
+               result = &match->matcher_list[i];
+               kunit_stream_add(stream, "\t");
+               result->assert.format(&result->assert, stream);
+               kunit_stream_add(stream, "\n");
+       }

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there. I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

Maybe it's this part that you wrote up above?

Quoted 4 lines
Quoted 4 lines

Nevertheless, I think the debate over the usefulness of the string_stream and kunit_stream are separate topics. Even if we made kunit_stream more structured, I am pretty sure I would want to use string_stream or some variation for constructing the message.

Why do we need string_stream to construct the message? Can't we just print it as we process it?

#29 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 19 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-18 17:08:34)

Quoted 17 lines · Brendan Higgins

I started poking around with your suggestion while we are waiting. A couple early observations:

1) It is actually easier to do than I previously thought and will probably help with getting more of the planned TAP output stuff working.

That being said, this is still a pretty substantial undertaking and will likely take *at least* a week to implement and properly review. Assuming everything goes extremely well (no unexpected issues on my end, very responsive reviewers, etc).

2) It *will* eliminate the need for kunit_stream.

3) ...but, it *will not* eliminate the need for string_stream.

Quoted 7 lines
Quoted 3 lines

Based on my early observations, I do think it is worth doing, but I don't think it is worth trying to make it in this patchset (unless I have already missed the window, or it is going to be open for a while):

The merge window is over. Typically code needs to be settled a few weeks before it opens (i.e. around -rc4 or -rc5) for most maintainers to pick up patches for the next merge window.

Yeah, it closed on Sunday, right?

I thought we might be able to squeak in since it was *mostly* settled, and Shuah sent me an email two weeks ago which I interpreted to mean she was still willing to take it.

In any case, it doesn't matter now.

Quoted 73 lines
Quoted 4 lines

I do think it will make things much cleaner, but I don't think it will achieve your desired goal of getting rid of an unstructured {kunit|string}_stream style interface; it just adds a layer on top of it that makes it harder to misuse.

Ok.

Quoted 16 lines

I attached a patch of what I have so far at the end of this email so you can see what I am talking about. And of course, if you agree with my assessment, so we can start working on it as a future patch.

A couple things in regard to the patch I attached:

1) I wrote it pretty quickly so there are almost definitely mistakes. You should consider it RFC. I did verify it compiles though.

2) Also, I did use kunit_stream in writing it: all occurences should be pretty easy to replace with string_stream; nevertheless, the reason for this is just to make it easier to play with the current APIs. I wanted to have something working before I went through a big tedious refactoring. So sorry if it causes any confusion.

Quoted 4 lines

3) I also based the patch on all the KUnit patches I have queued up (includes things like mocking and such) since I want to see how this serialization thing will work with mocks and matchers and things like that.

Great!

Quoted 43 lines

From 53d475d3d56afcf92b452c6d347dbedfa1a17d34 Mon Sep 17 00:00:00 2001
From: Brendan Higgins <[email protected]>
Date: Thu, 18 Jul 2019 16:08:52 -0700
Subject: [PATCH v1] DO NOT MERGE: started playing around with the
serialization api

---
 include/kunit/assert.h | 130 ++++++++++++++++++++++++++++++
 include/kunit/mock.h   |   4 +
 kunit/Makefile         |   3 +-
 kunit/assert.c         | 179 +++++++++++++++++++++++++++++++++++++++++
 kunit/mock.c           |   6 +-
 5 files changed, 318 insertions(+), 4 deletions(-)
 create mode 100644 include/kunit/assert.h
 create mode 100644 kunit/assert.c
include/kunit/assert.h new file +20 -0
new file mode 100644
index 0000000000000..e054fdff4642f
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+
+#ifndef _KUNIT_ASSERT_H
+#define _KUNIT_ASSERT_H
+
+#include <kunit/test.h>
+#include <kunit/mock.h>
+
+enum kunit_assert_type {
+       KUNIT_ASSERTION,
+       KUNIT_EXPECTATION,
+};
+
+struct kunit_assert {
Quoted 8 lines

include/kunit/assert.h line 26

Quoted 6 lines
include/kunit/assert.h new file +6 -0
+       enum kunit_assert_type type;
+       const char *line;
+       const char *file;
+       struct va_format message;
+       void (*format)(struct kunit_assert *assert,
+                      struct kunit_stream *stream);

Would passing in the test help too?

Yeah, it would probably be good to put one in struct kunit_assert.

Quoted 65 lines
Quoted 65 lines
include/kunit/assert.h new file +65 -0
+};
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+                             struct kunit_stream *stream);
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+                           struct kunit_stream *stream);
+
+struct kunit_unary_assert {
+       struct kunit_assert assert;
+       const char *condition;
+       bool expected_true;
+};
+
+void kunit_unary_assert_format(struct kunit_assert *assert,
+                              struct kunit_stream *stream);
+
+struct kunit_ptr_not_err_assert {
+       struct kunit_assert assert;
+       const char *text;
+       void *value;
+};
+
+void kunit_ptr_not_err_assert_format(struct kunit_assert *assert,
+                                    struct kunit_stream *stream);
+
+struct kunit_binary_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       long long left_value;
+       const char *right_text;
+       long long right_value;
+};
+
+void kunit_binary_assert_format(struct kunit_assert *assert,
+                               struct kunit_stream *stream);
+
+struct kunit_binary_ptr_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       void *left_value;
+       const char *right_text;
+       void *right_value;
+};
+
+void kunit_binary_ptr_assert_format(struct kunit_assert *assert,
+                                   struct kunit_stream *stream);
+
+struct kunit_binary_str_assert {
+       struct kunit_assert assert;
+       const char *operation;
+       const char *left_text;
+       const char *left_value;
+       const char *right_text;
+       const char *right_value;
+};
+
+void kunit_binary_str_assert_format(struct kunit_assert *assert,
+                                   struct kunit_stream *stream);
+
+struct kunit_mock_assert {
+       struct kunit_assert assert;
+};
Quoted 8 lines

include/kunit/assert.h line 95

Quoted 4 lines
include/kunit/assert.h new file +4 -0
+
+struct kunit_mock_no_expectations {
+       struct kunit_mock_assert assert;
+};

What's the purpose of making a wrapper struct with no other members? Just to make a different struct for some sort of type checking? I guess it's OK but I don't think it will be very useful in practice.

Yeah, just for typing purposes. I don't mind integrating this into the current patchset and then deciding if we want it or not.

Quoted 13 lines
Quoted 13 lines
include/kunit/assert.h new file +13 -0
+
+struct kunit_mock_declaration {
+       const char *function_name;
+       const char **type_names;
+       const void **params;
+       int len;
+};
+
+void kunit_mock_declaration_format(struct kunit_mock_declaration *declaration,
+                                  struct kunit_stream *stream);
+
+struct kunit_matcher_result {
+       struct kunit_assert assert;
Quoted 8 lines

include/kunit/assert.h line 114

Quoted 6 lines
include/kunit/assert.h new file +6 -0
+};
+
+struct kunit_mock_failed_match {
+       struct list_head node;
+       const char *expectation_text;
+       struct kunit_matcher_result *matcher_list;

Minor nitpick: this code could use some const sprinkling.

Will do.

Quoted 35 lines
Quoted 35 lines
include/kunit/assert.h new file +16 -0
+       size_t matcher_list_len;
+};
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+                                   struct kunit_stream *stream);
+
+struct kunit_mock_no_match {
+       struct kunit_mock_assert assert;
+       struct kunit_mock_declaration declaration;
+       struct list_head failed_match_list;
+};
+
+void kunit_mock_no_match_format(struct kunit_assert *assert,
+                               struct kunit_stream *stream);
+
+#endif /*  _KUNIT_ASSERT_H */
kunit/assert.c new file +13 -0
new file mode 100644
index 0000000000000..75bb6922a994e
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Assertion and expectation serialization API.
+ *
+ * Copyright (C) 2019, Google LLC.
+ * Author: Brendan Higgins <[email protected]>
+ */
+#include <kunit/assert.h>
+
+void kunit_base_assert_format(struct kunit_assert *assert,
+                             struct kunit_stream *stream)
+{
+       const char *expect_or_assert;
Quoted 8 lines

kunit/assert.c line 18

Quoted 5 lines
kunit/assert.c new file +5 -0
+
+       if (assert->type == KUNIT_EXPECTATION)
+               expect_or_assert = "EXPECTATION";
+       else
+               expect_or_assert = "ASSERTION";

Make this is a switch statement so we can have the compiler complain if an enum is missing.

Nice call! I didn't know the compiler warned about that. Will fix.

Quoted 26 lines
Quoted 12 lines
kunit/assert.c new file +12 -0
+
+       kunit_stream_add(stream, "%s FAILED at %s:%s\n",
+                        expect_or_assert, assert->file, assert->line);
+}
+
+void kunit_assert_print_msg(struct kunit_assert *assert,
+                           struct kunit_stream *stream)
+{
+       if (assert->message.fmt)
+               kunit_stream_add(stream, "\n%pV", &assert->message);
+}
+

[...]

Quoted 13 lines
kunit/assert.c new file +13 -0
+
+void kunit_mock_failed_match_format(struct kunit_mock_failed_match *match,
+                                   struct kunit_stream *stream)
+{
+       struct kunit_matcher_result *result;
+       size_t i;
+
+       kunit_stream_add(stream,
+                        "Tried expectation: %s, but\n",
+                        match->expectation_text);
+       for (i = 0; i < match->matcher_list_len; i++) {
+               result = &match->matcher_list[i];
+               kunit_stream_add(stream, "\t");
Quoted 8 lines

kunit/assert.c line 162

Quoted 3 lines
kunit/assert.c new file +3 -0
+               result->assert.format(&result->assert, stream);
+               kunit_stream_add(stream, "\n");
+       }

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Quoted 6 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 9 lines

Maybe it's this part that you wrote up above?

Quoted 4 lines
Quoted 4 lines

Nevertheless, I think the debate over the usefulness of the string_stream and kunit_stream are separate topics. Even if we made kunit_stream more structured, I am pretty sure I would want to use string_stream or some variation for constructing the message.

Why do we need string_stream to construct the message? Can't we just print it as we process it?

See preceding comment.

#30 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 9 lines · Stephen Boyd
Quoted 6 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Quoted 8 lines
Quoted 7 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

Quoted 4 lines

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

Can printk be called once with whatever is in the struct? Otherwise if this is about making printk into a structured log then maybe printk isn't the proper solution anyway. Maybe a dev interface should be used instead that can handle starting and stopping tests (via ioctl) in addition to reading test results, records, etc. with read() and a clearing of the records. Then the seqfile API works naturally. All of this is a bit premature, but it looks like you're going down the path of making something akin to ftrace that stores binary formatted assertion/expectation records in a lockless ring buffer that then formats those records when the user asks for them.

I can imagine someone wanting to write unit tests that check conditions from a simulated hardirq context via irq works (a driver mock framework?), so this doesn't seem far off.

#31 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 5 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 3 lines · Stephen Boyd
Quoted 2 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Yeah. Luis suggested just not supporting the IRQ context until later. See my later comment.

Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 6 lines

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

It says KERN_CONT is not SMP safe, and it isn't supposed to contain newlines, so it doesn't sound like it does any buffering for you. I looked at it a while ago and those comments agreed with my understanding of the code, but I could be wrong.

Quoted 1 line

Can printk be called once with whatever is in the struct?

Unfortunately, no. That is part of what I was trying to illustrate with this patch. Most of the messages are deterministic, but hardcoding all the possible message types would lead to a massive number of hard coded strings. However, even this would break down for the mocking formatters. All the different ways a function can be called are just too complex to encode into a finite set of hard coded fmt strings.

Quoted 6 lines

Otherwise if
this is about making printk into a structured log then maybe printk isn't the proper solution anyway. Maybe a dev interface should be used instead that can handle starting and stopping tests (via ioctl) in addition to reading test results, records, etc. with read() and a clearing of the records. Then the seqfile API works naturally. All of

Ehhh...I wouldn't mind providing such an interface, but I would really like to be able to provide the results without having to depend on a userland doing something to get test results. That has always been a pretty important goal for me.

Quoted 4 lines

this is a bit premature, but it looks like you're going down the path of making something akin to ftrace that stores binary formatted assertion/expectation records in a lockless ring buffer that then formats those records when the user asks for them.

Like you said, I think it is a bit premature to go that far.

In anycase, I don't see a way to get rid of string_stream, without significantly sacrificing usability.

Quoted 3 lines

I can imagine someone wanting to write unit tests that check conditions from a simulated hardirq context via irq works (a driver mock framework?), so this doesn't seem far off.

Yep, I actually presented the first pieces of that in the RFC v1 that I linked to you earlier in this discussion. I have a more fleshed out example here:

https://kunit.googlesource.com/linux/+/e10484ad2f9fc7926412ec84739fe105981b4771/drivers/i2c/busses/i2c-aspeed-test.c

I actually already have some people at Google playing around with it. So yeah, not far off at all! However, in these cases we are not actually running in the IRQ context (despite the fact that we are testing IRQ code) because we provide a fake IRQ chip, or some other fake mechanism that triggers the IRQ. Still, I could see someone wanting to do it in a non-fake-IRQ context.

Luis' suggestion was just to hold off on the IRQ safe stuff at the outset, since that is going to require a lot more effort to review. I know that's kind of the future coding argument again, but maybe the answer will be to just restrict what features an IRQ user has access to (maybe just really simple expectations, for example). I mean, we will probably have to restrict what they are allowed to use anyway.

Luis, do you have any ideas?

Cheers

#32 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

In reply to Stephen Boyd
Quoted 26 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 9 lines · Stephen Boyd
Quoted 6 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Quoted 8 lines
Quoted 7 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

Quoted 4 lines

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 6 lines

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

KERN_CONT does not guarantee that the message will get printed together. The pieces get interleaved with messages printed in parallel.

Note that KERN_CONT was originally really meant to be used only during boot. It was later used more widely and ended in the best effort category.

There were several attempts to make it more reliable. But it was always either too complicated or error prone or both.

You need to use your own buffering if you rely want perfect output. The question is if it is really worth the complexity. Also note that any buffering reduces the chance that the messages will reach the console.

BTW: There is a work in progress on a lockless printk ring buffer. It will make printk() more secure regarding deadlocks. But it might make transparent handling of continuous lines even more tricky.

I guess that local buffering, before calling printk(), will be even more important then. Well, it might really force us to create an API for it.

Quoted 10 lines

Can printk be called once with whatever is in the struct? Otherwise if this is about making printk into a structured log then maybe printk isn't the proper solution anyway. Maybe a dev interface should be used instead that can handle starting and stopping tests (via ioctl) in addition to reading test results, records, etc. with read() and a clearing of the records. Then the seqfile API works naturally. All of this is a bit premature, but it looks like you're going down the path of making something akin to ftrace that stores binary formatted assertion/expectation records in a lockless ring buffer that then formats those records when the user asks for them.

IMHO, ftrace postpones the text formatting primary because it does not not want to slow down the traced code more than necessary. It is yet another layer and there should be some strong reason for it.

Quoted 3 lines

I can imagine someone wanting to write unit tests that check conditions from a simulated hardirq context via irq works (a driver mock framework?), so this doesn't seem far off.

Note that stroring the messages into the printk log is basically safe in any context. It uses temporary per-CPU buffers for recursive messages and in NMI. The only problem is panic() when some CPU gets stuck with the lock taken. This will get solved by the lockless ringbuffer. Also the temporary buffers will not be necessary any longer.

Much bigger problems are with consoles. There are many of them. It means a lot of code and more locks involved, including scheduler locks. Note that console lock is a semaphore.

Best Regards,
Petr

#33 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 42 lines · Petr Mladek
Quoted 26 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 9 lines · Stephen Boyd
Quoted 6 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Quoted 8 lines
Quoted 7 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

Quoted 4 lines

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 6 lines

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

KERN_CONT does not guarantee that the message will get printed together. The pieces get interleaved with messages printed in parallel.

Note that KERN_CONT was originally really meant to be used only during boot. It was later used more widely and ended in the best effort category.

Quoted 7 lines

There were several attempts to make it more reliable. But it was always either too complicated or error prone or both.

You need to use your own buffering if you rely want perfect output. The question is if it is really worth the complexity. Also note that any buffering reduces the chance that the messages will reach the console.

Seems like that settles it then. Thanks!

Quoted 7 lines

BTW: There is a work in progress on a lockless printk ring buffer. It will make printk() more secure regarding deadlocks. But it might make transparent handling of continuous lines even more tricky.

I guess that local buffering, before calling printk(), will be even more important then. Well, it might really force us to create an API for it.

Cool! Can you CC me on that discussion?

Quoted 11 lines
Quoted 10 lines

Can printk be called once with whatever is in the struct? Otherwise if this is about making printk into a structured log then maybe printk isn't the proper solution anyway. Maybe a dev interface should be used instead that can handle starting and stopping tests (via ioctl) in addition to reading test results, records, etc. with read() and a clearing of the records. Then the seqfile API works naturally. All of this is a bit premature, but it looks like you're going down the path of making something akin to ftrace that stores binary formatted assertion/expectation records in a lockless ring buffer that then formats those records when the user asks for them.

Quoted 3 lines

IMHO, ftrace postpones the text formatting primary because it does not not want to slow down the traced code more than necessary. It is yet another layer and there should be some strong reason for it.

Noted. Yeah, I would prefer avoiding printing out the info at a separate time.

Quoted 9 lines
Quoted 3 lines

I can imagine someone wanting to write unit tests that check conditions from a simulated hardirq context via irq works (a driver mock framework?), so this doesn't seem far off.

Note that stroring the messages into the printk log is basically safe in any context. It uses temporary per-CPU buffers for recursive messages and in NMI. The only problem is panic() when some CPU gets stuck with the lock taken. This will get solved by the lockless ringbuffer. Also the temporary buffers will not be necessary any longer.

Sure, I think Stephen's concern is all the supporting code that is involved. Not printk specifically. It just means a lot more of KUnit has to be IRQ safe.

Quoted 3 lines

Much bigger problems are with consoles. There are many of them. It means a lot of code and more locks involved, including scheduler locks. Note that console lock is a semaphore.

That shouldn't affect us though, right? As long as we continue to use the printk interface?

#34 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 57 lines · Brendan Higgins
Quoted 42 lines · Petr Mladek
Quoted 26 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 9 lines · Stephen Boyd
Quoted 6 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Quoted 8 lines
Quoted 7 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

Quoted 4 lines

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 6 lines

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

KERN_CONT does not guarantee that the message will get printed together. The pieces get interleaved with messages printed in parallel.

Note that KERN_CONT was originally really meant to be used only during boot. It was later used more widely and ended in the best effort category.

Quoted 7 lines

There were several attempts to make it more reliable. But it was always either too complicated or error prone or both.

You need to use your own buffering if you rely want perfect output. The question is if it is really worth the complexity. Also note that any buffering reduces the chance that the messages will reach the console.

Seems like that settles it then. Thanks!

Quoted 4 lines

BTW: There is a work in progress on a lockless printk ring buffer. It will make printk() more secure regarding deadlocks. But it might make transparent handling of continuous lines even more tricky.

Quoted 5 lines
Quoted 3 lines

I guess that local buffering, before calling printk(), will be even more important then. Well, it might really force us to create an API for it.

Cool! Can you CC me on that discussion?

Adding John Oggness into CC.

John, please CC Brendan Higgins on the patchsets eventually switching printk() into the lockless buffer. The test framework is going to do its own buffering to keep the related messages together.

The lockless ringbuffer might make handling of related (partial) lines worse or better. It might justify KUnit's extra buffering or it might allow to get rid of it.

Quoted 9 lines
Quoted 5 lines

Note that stroring the messages into the printk log is basically safe in any context. It uses temporary per-CPU buffers for recursive messages and in NMI. The only problem is panic() when some CPU gets stuck with the lock taken. This will get solved by the lockless ringbuffer. Also the temporary buffers will not be necessary any longer.

Sure, I think Stephen's concern is all the supporting code that is involved. Not printk specifically. It just means a lot more of KUnit has to be IRQ safe.

I see.

BTW: I wonder if KUnit could reuse the existing seq_buf implementation for buffering messages.

I am sorry if it has already been proposed and rejected for some reason. I might have missed it. Feel free to just point me to same older mail.

Quoted 6 lines
Quoted 3 lines

Much bigger problems are with consoles. There are many of them. It means a lot of code and more locks involved, including scheduler locks. Note that console lock is a semaphore.

That shouldn't affect us though, right? As long as we continue to use the printk interface?

I guess that it should not affect KUnit.

The only problem might be if the testing framework calls printk() inside scheduler or console code. And only when the tested code uses the same locks that will be used by the called printk().

To be honest I do not fully understand KUnit design. I am not completely sure how the tested code is isolated from the running system. Namely, I do not know if the tested code shares the same locks with the system running the test.

Best Regards,
Petr

#35 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 67 lines · Petr Mladek
Quoted 57 lines · Brendan Higgins
Quoted 42 lines · Petr Mladek
Quoted 26 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 9 lines · Stephen Boyd
Quoted 6 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Quoted 8 lines
Quoted 7 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

Quoted 4 lines

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 6 lines

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

KERN_CONT does not guarantee that the message will get printed together. The pieces get interleaved with messages printed in parallel.

Note that KERN_CONT was originally really meant to be used only during boot. It was later used more widely and ended in the best effort category.

Quoted 7 lines

There were several attempts to make it more reliable. But it was always either too complicated or error prone or both.

You need to use your own buffering if you rely want perfect output. The question is if it is really worth the complexity. Also note that any buffering reduces the chance that the messages will reach the console.

Seems like that settles it then. Thanks!

Quoted 4 lines

BTW: There is a work in progress on a lockless printk ring buffer. It will make printk() more secure regarding deadlocks. But it might make transparent handling of continuous lines even more tricky.

Quoted 5 lines
Quoted 3 lines

I guess that local buffering, before calling printk(), will be even more important then. Well, it might really force us to create an API for it.

Cool! Can you CC me on that discussion?

Adding John Oggness into CC.

Quoted 7 lines

John, please CC Brendan Higgins on the patchsets eventually switching printk() into the lockless buffer. The test framework is going to do its own buffering to keep the related messages together.

The lockless ringbuffer might make handling of related (partial) lines worse or better. It might justify KUnit's extra buffering or it might allow to get rid of it.

Thanks for CC'ing me on the printk ringbuffer thread. It looks like it actually probably won't affect my needs for KUnit logging. The biggest reason I need some sort of buffering system is to be able to compose messages piece meal into a single message that will be printed out to the user as a single message with no messages from other printk callers printed out in the middle of mine.

The prb does look interesting; however, it appears that to get the semantics that I need, I would have to put my entire message in a single data block and would consequently need to know the size of my message a priori, which is problematic. Consequently, it seems as though I will probably need to compose my entire message using my own buffering system.

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

Note that stroring the messages into the printk log is basically safe in any context. It uses temporary per-CPU buffers for recursive messages and in NMI. The only problem is panic() when some CPU gets stuck with the lock taken. This will get solved by the lockless ringbuffer. Also the temporary buffers will not be necessary any longer.

Sure, I think Stephen's concern is all the supporting code that is involved. Not printk specifically. It just means a lot more of KUnit has to be IRQ safe.

Quoted 8 lines

I see.

BTW: I wonder if KUnit could reuse the existing seq_buf implementation for buffering messages.

I am sorry if it has already been proposed and rejected for some reason. I might have missed it. Feel free to just point me to same older mail.

Yeah, we discussed it briefly here:

https://lkml.org/lkml/2019/5/17/497

Looks like I forgot to include my reasoning in the commit text, sorry about that.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Much bigger problems are with consoles. There are many of them. It means a lot of code and more locks involved, including scheduler locks. Note that console lock is a semaphore.

Quoted 8 lines
Quoted 2 lines

That shouldn't affect us though, right? As long as we continue to use the printk interface?

I guess that it should not affect KUnit.

The only problem might be if the testing framework calls printk() inside scheduler or console code. And only when the tested code uses the same locks that will be used by the called printk().

Yeah, well printk will not be our only problem in those instances.

Quoted 4 lines

To be honest I do not fully understand KUnit design. I am not completely sure how the tested code is isolated from the running system. Namely, I do not know if the tested code shares the same locks with the system running the test.

No worries, I don't expect printk to be the hang up in those cases. It sounds like KUnit has a long way to evolve before printk is going to be a limitation.

Thanks!

#36 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 132 lines · Brendan Higgins
Quoted 67 lines · Petr Mladek
Quoted 57 lines · Brendan Higgins
Quoted 42 lines · Petr Mladek
Quoted 26 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 9 lines · Stephen Boyd
Quoted 6 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Quoted 8 lines
Quoted 7 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

Quoted 4 lines

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 6 lines

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

KERN_CONT does not guarantee that the message will get printed together. The pieces get interleaved with messages printed in parallel.

Note that KERN_CONT was originally really meant to be used only during boot. It was later used more widely and ended in the best effort category.

Quoted 7 lines

There were several attempts to make it more reliable. But it was always either too complicated or error prone or both.

You need to use your own buffering if you rely want perfect output. The question is if it is really worth the complexity. Also note that any buffering reduces the chance that the messages will reach the console.

Seems like that settles it then. Thanks!

Quoted 4 lines

BTW: There is a work in progress on a lockless printk ring buffer. It will make printk() more secure regarding deadlocks. But it might make transparent handling of continuous lines even more tricky.

Quoted 5 lines
Quoted 3 lines

I guess that local buffering, before calling printk(), will be even more important then. Well, it might really force us to create an API for it.

Cool! Can you CC me on that discussion?

Adding John Oggness into CC.

Quoted 7 lines

John, please CC Brendan Higgins on the patchsets eventually switching printk() into the lockless buffer. The test framework is going to do its own buffering to keep the related messages together.

The lockless ringbuffer might make handling of related (partial) lines worse or better. It might justify KUnit's extra buffering or it might allow to get rid of it.

Thanks for CC'ing me on the printk ringbuffer thread. It looks like it actually probably won't affect my needs for KUnit logging. The biggest reason I need some sort of buffering system is to be able to compose messages piece meal into a single message that will be printed out to the user as a single message with no messages from other printk callers printed out in the middle of mine.

The prb does look interesting; however, it appears that to get the semantics that I need, I would have to put my entire message in a single data block and would consequently need to know the size of my message a priori, which is problematic. Consequently, it seems as though I will probably need to compose my entire message using my own buffering system.

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

Note that stroring the messages into the printk log is basically safe in any context. It uses temporary per-CPU buffers for recursive messages and in NMI. The only problem is panic() when some CPU gets stuck with the lock taken. This will get solved by the lockless ringbuffer. Also the temporary buffers will not be necessary any longer.

Sure, I think Stephen's concern is all the supporting code that is involved. Not printk specifically. It just means a lot more of KUnit has to be IRQ safe.

Quoted 8 lines

I see.

BTW: I wonder if KUnit could reuse the existing seq_buf implementation for buffering messages.

I am sorry if it has already been proposed and rejected for some reason. I might have missed it. Feel free to just point me to same older mail.

Yeah, we discussed it briefly here:

https://lkml.org/lkml/2019/5/17/497

Looks like I forgot to include my reasoning in the commit text, sorry about that.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Much bigger problems are with consoles. There are many of them. It means a lot of code and more locks involved, including scheduler locks. Note that console lock is a semaphore.

Quoted 8 lines
Quoted 2 lines

That shouldn't affect us though, right? As long as we continue to use the printk interface?

I guess that it should not affect KUnit.

The only problem might be if the testing framework calls printk() inside scheduler or console code. And only when the tested code uses the same locks that will be used by the called printk().

Yeah, well printk will not be our only problem in those instances.

Quoted 8 lines
Quoted 4 lines

To be honest I do not fully understand KUnit design. I am not completely sure how the tested code is isolated from the running system. Namely, I do not know if the tested code shares the same locks with the system running the test.

No worries, I don't expect printk to be the hang up in those cases. It sounds like KUnit has a long way to evolve before printk is going to be a limitation.

So Stephen, what do you think?

Do you want me to go forward with the new kunit_assert API wrapping the string_stream as I have it now? Would you prefer to punt this to a later patch? Or would you prefer something else?

Cheers

#37 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoting Brendan Higgins (2019-08-01 11:59:57)

Quoted 14 lines
Quoted 11 lines · Brendan Higgins
Quoted 4 lines · Petr Mladek

To be honest I do not fully understand KUnit design. I am not completely sure how the tested code is isolated from the running system. Namely, I do not know if the tested code shares the same locks with the system running the test.

No worries, I don't expect printk to be the hang up in those cases. It sounds like KUnit has a long way to evolve before printk is going to be a limitation.

Quoted 6 lines

So Stephen, what do you think?

Do you want me to go forward with the new kunit_assert API wrapping the string_stream as I have it now? Would you prefer to punt this to a later patch? Or would you prefer something else?

I like the struct based approach. If anything, it can be adjusted to make the code throw some records into a spinlock later on and delay the formatting of the assertion if need be. Can you resend with that approach? I don't think I'll have any more comments after that.

#38 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 18 lines · Stephen Boyd

Quoting Brendan Higgins (2019-08-01 11:59:57)

Quoted 16 lines
Quoted 11 lines · Brendan Higgins
Quoted 4 lines · Petr Mladek

To be honest I do not fully understand KUnit design. I am not completely sure how the tested code is isolated from the running system. Namely, I do not know if the tested code shares the same locks with the system running the test.

No worries, I don't expect printk to be the hang up in those cases. It sounds like KUnit has a long way to evolve before printk is going to be a limitation.

So Stephen, what do you think?

Quoted 8 lines
Quoted 4 lines

Do you want me to go forward with the new kunit_assert API wrapping the string_stream as I have it now? Would you prefer to punt this to a later patch? Or would you prefer something else?

I like the struct based approach. If anything, it can be adjusted to make the code throw some records into a spinlock later on and delay the formatting of the assertion if need be.

That's a fair point.

Quoted 2 lines

Can you resend with that
approach? I don't think I'll have any more comments after that.

Cool, will do.

Thanks!

#39 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 25 lines · Brendan Higgins
Quoted 23 lines · Stephen Boyd

Quoting Brendan Higgins (2019-08-01 11:59:57)

Quoted 20 lines
Quoted 11 lines · Brendan Higgins
Quoted 4 lines · Petr Mladek

To be honest I do not fully understand KUnit design. I am not completely sure how the tested code is isolated from the running system. Namely, I do not know if the tested code shares the same locks with the system running the test.

No worries, I don't expect printk to be the hang up in those cases. It sounds like KUnit has a long way to evolve before printk is going to be a limitation.

So Stephen, what do you think?

Do you want me to go forward with the new kunit_assert API wrapping the string_stream as I have it now? Would you prefer to punt this to a later patch? Or would you prefer something else?

Quoted 8 lines
Quoted 3 lines

I like the struct based approach. If anything, it can be adjusted to make the code throw some records into a spinlock later on and delay the formatting of the assertion if need be.

That's a fair point.

Quoted 2 lines

Can you resend with that
approach? I don't think I'll have any more comments after that.

I sent a new revision, v12, that incorporates the kunit_assert stuff.

Let me know what you think!

#40 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

In reply to Brendan Higgins
Quoted 73 lines · Brendan Higgins
Quoted 64 lines · Petr Mladek
Quoted 55 lines · Brendan Higgins
Quoted 40 lines · Petr Mladek
Quoted 23 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 7 lines · Stephen Boyd
Quoted 4 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Quoted 11 lines
Quoted 6 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 7 lines

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

KERN_CONT does not guarantee that the message will get printed together. The pieces get interleaved with messages printed in parallel.

Note that KERN_CONT was originally really meant to be used only during boot. It was later used more widely and ended in the best effort category.

Quoted 7 lines

There were several attempts to make it more reliable. But it was always either too complicated or error prone or both.

You need to use your own buffering if you rely want perfect output. The question is if it is really worth the complexity. Also note that any buffering reduces the chance that the messages will reach the console.

Seems like that settles it then. Thanks!

Quoted 4 lines

BTW: There is a work in progress on a lockless printk ring buffer. It will make printk() more secure regarding deadlocks. But it might make transparent handling of continuous lines even more tricky.

Quoted 5 lines
Quoted 3 lines

I guess that local buffering, before calling printk(), will be even more important then. Well, it might really force us to create an API for it.

Cool! Can you CC me on that discussion?

Adding John Oggness into CC.

Quoted 7 lines

John, please CC Brendan Higgins on the patchsets eventually switching printk() into the lockless buffer. The test framework is going to do its own buffering to keep the related messages together.

The lockless ringbuffer might make handling of related (partial) lines worse or better. It might justify KUnit's extra buffering or it might allow to get rid of it.

Quoted 6 lines

Thanks for CC'ing me on the printk ringbuffer thread. It looks like it actually probably won't affect my needs for KUnit logging. The biggest reason I need some sort of buffering system is to be able to compose messages piece meal into a single message that will be printed out to the user as a single message with no messages from other printk callers printed out in the middle of mine.

printk has this same requirement for its CONT messages. You can read about how I propose to implement that here[0], using a separate prb ringbuffer for buffered storage until all the pieces are available.

It is not my goal that multiple subsystems start making use of the prb ringbuffer. However, its features can be attractive if you don't want to worry about multiple writers/readers or context (including NMI). Before writing "yet another ringbuffer" [1] it might be worth the effort to at least see if one of the existing implementations can work (or be extended to work) for you.

John Ogness

[0] https://lkml.kernel.org/r/[email protected]
[1] https://lwn.net/Articles/789603/

Quoted 59 lines

The prb does look interesting; however, it appears that to get the semantics that I need, I would have to put my entire message in a single data block and would consequently need to know the size of my message a priori, which is problematic. Consequently, it seems as though I will probably need to compose my entire message using my own buffering system.

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

Note that stroring the messages into the printk log is basically safe in any context. It uses temporary per-CPU buffers for recursive messages and in NMI. The only problem is panic() when some CPU gets stuck with the lock taken. This will get solved by the lockless ringbuffer. Also the temporary buffers will not be necessary any longer.

Sure, I think Stephen's concern is all the supporting code that is involved. Not printk specifically. It just means a lot more of KUnit has to be IRQ safe.

Quoted 8 lines

I see.

BTW: I wonder if KUnit could reuse the existing seq_buf implementation for buffering messages.

I am sorry if it has already been proposed and rejected for some reason. I might have missed it. Feel free to just point me to same older mail.

Yeah, we discussed it briefly here:

https://lkml.org/lkml/2019/5/17/497

Looks like I forgot to include my reasoning in the commit text, sorry about that.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Much bigger problems are with consoles. There are many of them. It means a lot of code and more locks involved, including scheduler locks. Note that console lock is a semaphore.

Quoted 8 lines
Quoted 2 lines

That shouldn't affect us though, right? As long as we continue to use the printk interface?

I guess that it should not affect KUnit.

The only problem might be if the testing framework calls printk() inside scheduler or console code. And only when the tested code uses the same locks that will be used by the called printk().

Yeah, well printk will not be our only problem in those instances.

Quoted 4 lines

To be honest I do not fully understand KUnit design. I am not completely sure how the tested code is isolated from the running system. Namely, I do not know if the tested code shares the same locks with the system running the test.

No worries, I don't expect printk to be the hang up in those cases. It sounds like KUnit has a long way to evolve before printk is going to be a limitation.

Thanks!

#41 lore raw

Re: [PATCH v9 04/18] kunit: test: add kunit_stream a std::stream like logger

Quoted 81 lines · John Ogness
Quoted 73 lines · Brendan Higgins
Quoted 64 lines · Petr Mladek
Quoted 55 lines · Brendan Higgins
Quoted 40 lines · Petr Mladek
Quoted 23 lines · Stephen Boyd

Quoting Brendan Higgins (2019-07-22 15:30:49)

Quoted 7 lines · Stephen Boyd
Quoted 4 lines

What's the calling context of the assertions and expectations? I still don't like the fact that string stream needs to allocate buffers and throw them into a list somewhere because the calling context matters there.

The calling context is the same as before, which is anywhere.

Ok. That's concerning then.

Quoted 11 lines
Quoted 6 lines

I'd prefer we just wrote directly to the console/log via printk instead. That way things are simple because we use the existing buffering path of printk, but maybe there's some benefit to the string stream that I don't see? Right now it looks like it builds a string and then dumps it to printk so I'm sort of lost what the benefit is over just writing directly with printk.

It's just buffering it so the whole string gets printed uninterrupted. If we were to print out piecemeal to printk, couldn't we have another call to printk come in causing it to garble the KUnit message we are in the middle of printing?

Quoted 7 lines

Yes, printing piecemeal by calling printk many times could lead to interleaving of messages if something else comes in such as an interrupt printing something. Printk has some support to hold "records" but I'm not sure how that would work here because KERN_CONT talks about only being used early on in boot code. I haven't looked at printk in detail though so maybe I'm all wrong and KERN_CONT just works?

KERN_CONT does not guarantee that the message will get printed together. The pieces get interleaved with messages printed in parallel.

Note that KERN_CONT was originally really meant to be used only during boot. It was later used more widely and ended in the best effort category.

Quoted 7 lines

There were several attempts to make it more reliable. But it was always either too complicated or error prone or both.

You need to use your own buffering if you rely want perfect output. The question is if it is really worth the complexity. Also note that any buffering reduces the chance that the messages will reach the console.

Seems like that settles it then. Thanks!

Quoted 4 lines

BTW: There is a work in progress on a lockless printk ring buffer. It will make printk() more secure regarding deadlocks. But it might make transparent handling of continuous lines even more tricky.

Quoted 5 lines
Quoted 3 lines

I guess that local buffering, before calling printk(), will be even more important then. Well, it might really force us to create an API for it.

Cool! Can you CC me on that discussion?

Adding John Oggness into CC.

Quoted 7 lines

John, please CC Brendan Higgins on the patchsets eventually switching printk() into the lockless buffer. The test framework is going to do its own buffering to keep the related messages together.

The lockless ringbuffer might make handling of related (partial) lines worse or better. It might justify KUnit's extra buffering or it might allow to get rid of it.

Quoted 6 lines

Thanks for CC'ing me on the printk ringbuffer thread. It looks like it actually probably won't affect my needs for KUnit logging. The biggest reason I need some sort of buffering system is to be able to compose messages piece meal into a single message that will be printed out to the user as a single message with no messages from other printk callers printed out in the middle of mine.

Quoted 7 lines

printk has this same requirement for its CONT messages. You can read about how I propose to implement that here[0], using a separate prb ringbuffer for buffered storage until all the pieces are available.

It is not my goal that multiple subsystems start making use of the prb ringbuffer. However, its features can be attractive if you don't want to worry about multiple writers/readers or context (including NMI). Before

That sounds like it might be useful down the road, but not to replace the string_stream.

Quoted 3 lines

writing "yet another ringbuffer" [1] it might be worth the effort to at least see if one of the existing implementations can work (or be extended to work) for you.

In regards to the conversation here about string_stream/kunit_stream, I think Petr already answered that question. As I said previously:

Quoted 6 lines

[I]t appears that to get the
semantics that I need, I would have to put my entire message in a single data block and would consequently need to know the size of my message a priori, which is problematic. Consequently, it seems as though I will probably need to compose my entire message using my own buffering system.

I could potentially use my own set of prbs for that buffering; however, I think this use case is probably closer to seq_buf than your prb. Really, I just want some kind of string builder, not a message queue.

The place where I think your prb is relevant here is once I have composed the message and I just want to print it, having a way to print it without worrying about context is nice, but I think that is a separate discussion from the main topic here which was just figuring out the right way to compose that message in the first place.

Cheers

Quoted 64 lines

John Ogness

[0] https://lkml.kernel.org/r/[email protected]
[1] https://lwn.net/Articles/789603/

Quoted 59 lines

The prb does look interesting; however, it appears that to get the semantics that I need, I would have to put my entire message in a single data block and would consequently need to know the size of my message a priori, which is problematic. Consequently, it seems as though I will probably need to compose my entire message using my own buffering system.

Quoted 11 lines
Quoted 10 lines
Quoted 6 lines

Note that stroring the messages into the printk log is basically safe in any context. It uses temporary per-CPU buffers for recursive messages and in NMI. The only problem is panic() when some CPU gets stuck with the lock taken. This will get solved by the lockless ringbuffer. Also the temporary buffers will not be necessary any longer.

Sure, I think Stephen's concern is all the supporting code that is involved. Not printk specifically. It just means a lot more of KUnit has to be IRQ safe.

Quoted 8 lines

I see.

BTW: I wonder if KUnit could reuse the existing seq_buf implementation for buffering messages.

I am sorry if it has already been proposed and rejected for some reason. I might have missed it. Feel free to just point me to same older mail.

Yeah, we discussed it briefly here:

https://lkml.org/lkml/2019/5/17/497

Looks like I forgot to include my reasoning in the commit text, sorry about that.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Much bigger problems are with consoles. There are many of them. It means a lot of code and more locks involved, including scheduler locks. Note that console lock is a semaphore.

Quoted 8 lines
Quoted 2 lines

That shouldn't affect us though, right? As long as we continue to use the printk interface?

I guess that it should not affect KUnit.

The only problem might be if the testing framework calls printk() inside scheduler or console code. And only when the tested code uses the same locks that will be used by the called printk().

Yeah, well printk will not be our only problem in those instances.

Quoted 4 lines

To be honest I do not fully understand KUnit design. I am not completely sure how the tested code is isolated from the running system. Namely, I do not know if the tested code shares the same locks with the system running the test.

No worries, I don't expect printk to be the hang up in those cases. It sounds like KUnit has a long way to evolve before printk is going to be a limitation.

Thanks!

#42 lore raw

PATCH v9 5/18 kunit: test: add the concept of expectations

In reply to Brendan Higgins

Add support for expectations, which allow properties to be specified and then verified in tests.

---
 include/kunit/test.h | 525 +++++++++++++++++++++++++++++++++++++++++++
 kunit/test.c         |  66 ++++++
 2 files changed, 591 insertions(+)
include/kunit/test.h +525 -0
index bc7dbdcf8abab..df18765e6ea25 100644
@@ -9,6 +9,7 @@
 #ifndef _KUNIT_TEST_H
 #define _KUNIT_TEST_H
 
+#include <linux/kernel.h>
 #include <linux/types.h>
 #include <linux/slab.h>
 #include <kunit/kunit-stream.h>
@@ -295,4 +296,528 @@ void __printf(3, 4) kunit_printk(const char *level,
 #define kunit_err(test, fmt, ...) \
 		kunit_printk(KERN_ERR, test, fmt, ##__VA_ARGS__)
 
+/*
+ * Generates a compile-time warning in case of comparing incompatible types.
+ */
+#define __kunit_typecheck(lhs, rhs) \
+	((void) __typecheck(lhs, rhs))
+
+static inline struct kunit_stream *kunit_expect_start(struct kunit *test,
+						      const char *file,
+						      const char *line)
+{
+	struct kunit_stream *stream = alloc_kunit_stream(test, KERN_ERR);
+
+	kunit_stream_add(stream, "EXPECTATION FAILED at %s:%s\n\t", file, line);
+
+	return stream;
+}
+
+static inline void kunit_expect_end(struct kunit *test,
+				    bool success,
+				    struct kunit_stream *stream)
+{
+	if (!success)
+		kunit_fail(test, stream);
+	else
+		kunit_stream_clear(stream);
+}
+
+#define KUNIT_EXPECT_START(test) \
+		kunit_expect_start(test, __FILE__, __stringify(__LINE__))
+
+#define KUNIT_EXPECT_END(test, success, stream) \
+		kunit_expect_end(test, success, stream)
+
+#define KUNIT_EXPECT_MSG(test, success, message, fmt, ...) do {		       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+									       \
+	kunit_stream_add(__stream, message);				       \
+	kunit_stream_add(__stream, fmt, ##__VA_ARGS__);			       \
+	KUNIT_EXPECT_END(test, success, __stream);			       \
+} while (0)
+
+#define KUNIT_EXPECT(test, success, message) do {			       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+									       \
+	kunit_stream_add(__stream, message);				       \
+	KUNIT_EXPECT_END(test, success, __stream);			       \
+} while (0)
+
+/**
+ * KUNIT_SUCCEED() - A no-op expectation. Only exists for code clarity.
+ * @test: The test context object.
+ *
+ * The opposite of KUNIT_FAIL(), it is an expectation that cannot fail. In other
+ * words, it does nothing and only exists for code clarity. See
+ * KUNIT_EXPECT_TRUE() for more information.
+ */
+#define KUNIT_SUCCEED(test) do {} while (0)
+
+/**
+ * KUNIT_FAIL() - Always causes a test to fail when evaluated.
+ * @test: The test context object.
+ * @fmt: an informational message to be printed when the assertion is made.
+ * @...: string format arguments.
+ *
+ * The opposite of KUNIT_SUCCEED(), it is an expectation that always fails. In
+ * other words, it always results in a failed expectation, and consequently
+ * always causes the test case to fail when evaluated. See KUNIT_EXPECT_TRUE()
+ * for more information.
+ */
+#define KUNIT_FAIL(test, fmt, ...) do {					       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+									       \
+	kunit_stream_add(__stream, fmt, ##__VA_ARGS__);			       \
+	KUNIT_EXPECT_END(test, false, __stream);			       \
+} while (0)
+
+/**
+ * KUNIT_EXPECT_TRUE() - Causes a test failure when the expression is not true.
+ * @test: The test context object.
+ * @condition: an arbitrary boolean expression. The test fails when this does
+ * not evaluate to true.
+ *
+ * This and expectations of the form `KUNIT_EXPECT_*` will cause the test case
+ * to fail when the specified condition is not met; however, it will not prevent
+ * the test case from continuing to run; this is otherwise known as an
+ * *expectation failure*.
+ */
+#define KUNIT_EXPECT_TRUE(test, condition)				       \
+		KUNIT_EXPECT(test, (condition),				       \
+		       "Expected " #condition " is true, but is false\n")
+
+#define KUNIT_EXPECT_TRUE_MSG(test, condition, fmt, ...)		       \
+		KUNIT_EXPECT_MSG(test, (condition),			       \
+				"Expected " #condition " is true, but is false\n",\
+				fmt, ##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_FALSE() - Makes a test failure when the expression is not false.
+ * @test: The test context object.
+ * @condition: an arbitrary boolean expression. The test fails when this does
+ * not evaluate to false.
+ *
+ * Sets an expectation that @condition evaluates to false. See
+ * KUNIT_EXPECT_TRUE() for more information.
+ */
+#define KUNIT_EXPECT_FALSE(test, condition)				       \
+		KUNIT_EXPECT(test, !(condition),			       \
+		       "Expected " #condition " is false, but is true\n")
+
+#define KUNIT_EXPECT_FALSE_MSG(test, condition, fmt, ...)		       \
+		KUNIT_EXPECT_MSG(test, !(condition),			       \
+				"Expected " #condition " is false, but is true\n",\
+				fmt, ##__VA_ARGS__)
+
+void kunit_expect_binary_msg(struct kunit *test,
+			     long long left, const char *left_name,
+			     long long right, const char *right_name,
+			     bool compare_result,
+			     const char *compare_name,
+			     const char *file,
+			     const char *line,
+			     const char *fmt, ...);
+
+static inline void kunit_expect_binary(struct kunit *test,
+				       long long left, const char *left_name,
+				       long long right, const char *right_name,
+				       bool compare_result,
+				       const char *compare_name,
+				       const char *file,
+				       const char *line)
+{
+	kunit_expect_binary_msg(test,
+				left, left_name,
+				right, right_name,
+				compare_result,
+				compare_name,
+				file,
+				line,
+				NULL);
+}
+
+void kunit_expect_ptr_binary_msg(struct kunit *test,
+				 void *left, const char *left_name,
+				 void *right, const char *right_name,
+				 bool compare_result,
+				 const char *compare_name,
+				 const char *file,
+				 const char *line,
+				 const char *fmt, ...);
+
+static inline void kunit_expect_ptr_binary(struct kunit *test,
+					   void *left, const char *left_name,
+					   void *right, const char *right_name,
+					   bool compare_result,
+					   const char *compare_name,
+					   const char *file,
+					   const char *line)
+{
+	kunit_expect_ptr_binary_msg(test,
+				    left, left_name,
+				    right, right_name,
+				    compare_result,
+				    compare_name,
+				    file,
+				    line,
+				    NULL);
+}
+
+/*
+ * A factory macro for defining the expectations for the basic comparisons
+ * defined for the built in types.
+ *
+ * Unfortunately, there is no common type that all types can be promoted to for
+ * which all the binary operators behave the same way as for the actual types
+ * (for example, there is no type that long long and unsigned long long can
+ * both be cast to where the comparison result is preserved for all values). So
+ * the best we can do is do the comparison in the original types and then coerce
+ * everything to long long for printing; this way, the comparison behaves
+ * correctly and the printed out value usually makes sense without
+ * interpretation, but can always be interpretted to figure out the actual
+ * value.
+ */
+#define KUNIT_EXPECT_BINARY(test, left, condition, right) do {		       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+	__kunit_typecheck(__left, __right);				       \
+	kunit_expect_binary(test,					       \
+			    (long long) __left, #left,			       \
+			    (long long) __right, #right,		       \
+			    __left condition __right, #condition,	       \
+			    __FILE__, __stringify(__LINE__));		       \
+} while (0)
+
+#define KUNIT_EXPECT_BINARY_MSG(test, left, condition, right, fmt, ...) do {   \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+	__kunit_typecheck(__left, __right);				       \
+	kunit_expect_binary_msg(test,					       \
+				(long long) __left, #left,		       \
+				(long long) __right, #right,		       \
+				__left condition __right, #condition,	       \
+				__FILE__, __stringify(__LINE__),	       \
+				fmt, ##__VA_ARGS__);			       \
+} while (0)
+
+/*
+ * Just like KUNIT_EXPECT_BINARY, but for comparing pointer types.
+ */
+#define KUNIT_EXPECT_PTR_BINARY(test, left, condition, right) do {	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+	__kunit_typecheck(__left, __right);				       \
+	kunit_expect_ptr_binary(test,					       \
+			    (void *) __left, #left,			       \
+			    (void *) __right, #right,			       \
+			    __left condition __right, #condition,	       \
+			    __FILE__, __stringify(__LINE__));		       \
+} while (0)
+
+#define KUNIT_EXPECT_PTR_BINARY_MSG(test, left, condition, right, fmt, ...)    \
+do {									       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+	__kunit_typecheck(__left, __right);				       \
+	kunit_expect_ptr_binary_msg(test,				       \
+				    (void *) __left, #left,		       \
+				    (void *) __right, #right,		       \
+				    __left condition __right, #condition,      \
+				    __FILE__, __stringify(__LINE__),	       \
+				    fmt, ##__VA_ARGS__);		       \
+} while (0)
+
+/**
+ * KUNIT_EXPECT_EQ() - Sets an expectation that @left and @right are equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an expectation that the values that @left and @right evaluate to are
+ * equal. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, (@left) == (@right)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_EQ(test, left, right) \
+		KUNIT_EXPECT_BINARY(test, left, ==, right)
+
+#define KUNIT_EXPECT_EQ_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_EXPECT_BINARY_MSG(test,				       \
+					left,				       \
+					==,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_PTR_EQ() - Expects that pointers @left and @right are equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a pointer.
+ * @right: an arbitrary expression that evaluates to a pointer.
+ *
+ * Sets an expectation that the values that @left and @right evaluate to are
+ * equal. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, (@left) == (@right)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_PTR_EQ(test, left, right) \
+		KUNIT_EXPECT_PTR_BINARY(test, left, ==, right)
+
+#define KUNIT_EXPECT_PTR_EQ_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_EXPECT_PTR_BINARY_MSG(test,			       \
+					    left,			       \
+					    ==,				       \
+					    right,			       \
+					    fmt,			       \
+					    ##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_NE() - An expectation that @left and @right are not equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an expectation that the values that @left and @right evaluate to are not
+ * equal. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, (@left) != (@right)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_NE(test, left, right) \
+		KUNIT_EXPECT_BINARY(test, left, !=, right)
+
+#define KUNIT_EXPECT_NE_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_EXPECT_BINARY_MSG(test,				       \
+					left,				       \
+					!=,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_PTR_NE() - Expects that pointers @left and @right are not equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a pointer.
+ * @right: an arbitrary expression that evaluates to a pointer.
+ *
+ * Sets an expectation that the values that @left and @right evaluate to are not
+ * equal. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, (@left) != (@right)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_PTR_NE(test, left, right) \
+		KUNIT_EXPECT_PTR_BINARY(test, left, !=, right)
+
+#define KUNIT_EXPECT_PTR_NE_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_EXPECT_PTR_BINARY_MSG(test,			       \
+					    left,			       \
+					    !=,				       \
+					    right,			       \
+					    fmt,			       \
+					    ##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_LT() - An expectation that @left is less than @right.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an expectation that the value that @left evaluates to is less than the
+ * value that @right evaluates to. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, (@left) < (@right)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_LT(test, left, right) \
+		KUNIT_EXPECT_BINARY(test, left, <, right)
+
+#define KUNIT_EXPECT_LT_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_EXPECT_BINARY_MSG(test,				       \
+					left,				       \
+					<,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_LE() - Expects that @left is less than or equal to @right.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an expectation that the value that @left evaluates to is less than or
+ * equal to the value that @right evaluates to. Semantically this is equivalent
+ * to KUNIT_EXPECT_TRUE(@test, (@left) <= (@right)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_LE(test, left, right) \
+		KUNIT_EXPECT_BINARY(test, left, <=, right)
+
+#define KUNIT_EXPECT_LE_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_EXPECT_BINARY_MSG(test,				       \
+					left,				       \
+					<=,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_GT() - An expectation that @left is greater than @right.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an expectation that the value that @left evaluates to is greater than
+ * the value that @right evaluates to. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, (@left) > (@right)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_GT(test, left, right) \
+		KUNIT_EXPECT_BINARY(test, left, >, right)
+
+#define KUNIT_EXPECT_GT_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_EXPECT_BINARY_MSG(test,				       \
+					left,				       \
+					>,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_GE() - Expects that @left is greater than or equal to @right.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an expectation that the value that @left evaluates to is greater than
+ * the value that @right evaluates to. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, (@left) >= (@right)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_GE(test, left, right) \
+		KUNIT_EXPECT_BINARY(test, left, >=, right)
+
+#define KUNIT_EXPECT_GE_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_EXPECT_BINARY_MSG(test,				       \
+					left,				       \
+					>=,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_EXPECT_STREQ() - Expects that strings @left and @right are equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a null terminated string.
+ * @right: an arbitrary expression that evaluates to a null terminated string.
+ *
+ * Sets an expectation that the values that @left and @right evaluate to are
+ * equal. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, !strcmp((@left), (@right))). See KUNIT_EXPECT_TRUE()
+ * for more information.
+ */
+#define KUNIT_EXPECT_STREQ(test, left, right) do {			       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+									       \
+	kunit_stream_add(__stream, "Expected " #left " == " #right ", but\n"); \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #left, __left);	       \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #right, __right);	       \
+									       \
+	KUNIT_EXPECT_END(test, !strcmp(left, right), __stream);		       \
+} while (0)
+
+#define KUNIT_EXPECT_STREQ_MSG(test, left, right, fmt, ...) do {	       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+									       \
+	kunit_stream_add(__stream, "Expected " #left " == " #right ", but\n"); \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #left, __left);	       \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #right, __right);	       \
+	kunit_stream_add(__stream, fmt, ##__VA_ARGS__);			       \
+									       \
+	KUNIT_EXPECT_END(test, !strcmp(left, right), __stream);		       \
+} while (0)
+
+/**
+ * KUNIT_EXPECT_STRNEQ() - Expects that strings @left and @right are not equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a null terminated string.
+ * @right: an arbitrary expression that evaluates to a null terminated string.
+ *
+ * Sets an expectation that the values that @left and @right evaluate to are
+ * not equal. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, strcmp((@left), (@right))). See KUNIT_EXPECT_TRUE()
+ * for more information.
+ */
+#define KUNIT_EXPECT_STRNEQ(test, left, right) do {			       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+									       \
+	kunit_stream_add(__stream, "Expected " #left " != " #right ", but\n"); \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #left, __left);	       \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #right, __right);	       \
+									       \
+	KUNIT_EXPECT_END(test, strcmp(left, right), __stream);		       \
+} while (0)
+
+#define KUNIT_EXPECT_STRNEQ_MSG(test, left, right, fmt, ...) do {	       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+									       \
+	kunit_stream_add(__stream, "Expected " #left " != " #right ", but\n"); \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #left, __left);	       \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #right, __right);	       \
+	kunit_stream_add(__stream, fmt, ##__VA_ARGS__);			       \
+									       \
+	KUNIT_EXPECT_END(test, strcmp(left, right), __stream);		       \
+} while (0)
+
+/**
+ * KUNIT_EXPECT_NOT_ERR_OR_NULL() - Expects that @ptr is not null and not err.
+ * @test: The test context object.
+ * @ptr: an arbitrary pointer.
+ *
+ * Sets an expectation that the value that @ptr evaluates to is not null and not
+ * an errno stored in a pointer. This is semantically equivalent to
+ * KUNIT_EXPECT_TRUE(@test, !IS_ERR_OR_NULL(@ptr)). See KUNIT_EXPECT_TRUE() for
+ * more information.
+ */
+#define KUNIT_EXPECT_NOT_ERR_OR_NULL(test, ptr) do {			       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+	typeof(ptr) __ptr = (ptr);					       \
+									       \
+	if (!__ptr)							       \
+		kunit_stream_add(__stream,				       \
+			      "Expected " #ptr " is not null, but is\n");      \
+	if (IS_ERR(__ptr))						       \
+		kunit_stream_add(__stream,				       \
+			      "Expected " #ptr " is not error, but is: %ld",   \
+			      PTR_ERR(__ptr));				       \
+									       \
+	KUNIT_EXPECT_END(test, !IS_ERR_OR_NULL(__ptr), __stream);	       \
+} while (0)
+
+#define KUNIT_EXPECT_NOT_ERR_OR_NULL_MSG(test, ptr, fmt, ...) do {	       \
+	struct kunit_stream *__stream = KUNIT_EXPECT_START(test);	       \
+	typeof(ptr) __ptr = (ptr);					       \
+									       \
+	if (!__ptr) {							       \
+		kunit_stream_add(__stream,				       \
+			      "Expected " #ptr " is not null, but is\n");      \
+		kunit_stream_add(__stream, fmt, ##__VA_ARGS__);		       \
+	}								       \
+	if (IS_ERR(__ptr)) {						       \
+		kunit_stream_add(__stream,				       \
+			      "Expected " #ptr " is not error, but is: %ld",   \
+			      PTR_ERR(__ptr));				       \
+									       \
+		kunit_stream_add(__stream, fmt, ##__VA_ARGS__);		       \
+	}								       \
+	KUNIT_EXPECT_END(test, !IS_ERR_OR_NULL(__ptr), __stream);	       \
+} while (0)
+
 #endif /* _KUNIT_TEST_H */
kunit/test.c +66 -0

69 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#43 lore raw

PATCH v9 6/18 kbuild: enable building KUnit

In reply to Brendan Higgins

KUnit is a new unit testing framework for the kernel and when used is built into the kernel as a part of it. Add KUnit to the root Kconfig and Makefile to allow it to be actually built.

---
 Kconfig  | 2 ++
 Makefile | 2 ++
 2 files changed, 4 insertions(+)
Kconfig +2 -0
index 48a80beab6853..10428501edb78 100644
@@ -30,3 +30,5 @@ source "crypto/Kconfig"
 source "lib/Kconfig"
 
 source "lib/Kconfig.debug"
+
+source "kunit/Kconfig"
Makefile +2 -0
index 3e4868a6498b2..0ce1a8a2b6fec 100644
@@ -993,6 +993,8 @@ PHONY += prepare0
 ifeq ($(KBUILD_EXTMOD),)
 core-y		+= kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/
 
+core-$(CONFIG_KUNIT) += kunit/
+
 vmlinux-dirs	:= $(patsubst %/,%,$(filter %/, $(init-y) $(init-m) \
 		     $(core-y) $(core-m) $(drivers-y) $(drivers-m) \
 		     $(net-y) $(net-m) $(libs-y) $(libs-m) $(virt-y)))
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
Reviewed-by on v9 6/18
#44 lore raw

Re: [PATCH v9 06/18] kbuild: enable building KUnit

Quoting Brendan Higgins (2019-07-12 01:17:32)

Quoted 10 lines

KUnit is a new unit testing framework for the kernel and when used is built into the kernel as a part of it. Add KUnit to the root Kconfig and Makefile to allow it to be actually built.

---

#45 lore raw

PATCH v9 7/18 kunit: test: add initial tests

In reply to Brendan Higgins

Add a test for string stream along with a simpler example.

---
 kunit/Kconfig              | 21 +++++++++
 kunit/Makefile             |  4 ++
 kunit/example-test.c       | 88 ++++++++++++++++++++++++++++++++++++++
 kunit/string-stream-test.c | 75 ++++++++++++++++++++++++++++++++
 4 files changed, 188 insertions(+)
 create mode 100644 kunit/example-test.c
 create mode 100644 kunit/string-stream-test.c
kunit/Kconfig +21 -0
index 330ae83527c23..8541ef95b65ad 100644
@@ -14,4 +14,25 @@ config KUNIT
 	  architectures. For more information, please see
 	  Documentation/dev-tools/kunit/.
 
+config KUNIT_TEST
+	bool "KUnit test for KUnit"
+	depends on KUNIT
+	help
+	  Enables the unit tests for the KUnit test framework. These tests test
+	  the KUnit test framework itself; the tests are both written using
+	  KUnit and test KUnit. This option should only be enabled for testing
+	  purposes by developers interested in testing that KUnit works as
+	  expected.
+
+config KUNIT_EXAMPLE_TEST
+	bool "Example test for KUnit"
+	depends on KUNIT
+	help
+	  Enables an example unit test that illustrates some of the basic
+	  features of KUnit. This test only exists to help new users understand
+	  what KUnit is and how it is used. Please refer to the example test
+	  itself, kunit/example-test.c, for more information. This option is
+	  intended for curious hackers who would like to understand how to use
+	  KUnit for kernel development.
+
 endmenu
kunit/Makefile +4 -0
index 6ddc622ee6b1c..60a9ea6cb4697 100644
@@ -1,3 +1,7 @@
 obj-$(CONFIG_KUNIT) +=			test.o \
 					string-stream.o \
 					kunit-stream.o
+
+obj-$(CONFIG_KUNIT_TEST) +=		string-stream-test.o
+
+obj-$(CONFIG_KUNIT_EXAMPLE_TEST) +=	example-test.o
kunit/example-test.c new file +88 -0

88 lines not on this page · raw message

kunit/string-stream-test.c new file +75 -0

75 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#46 lore raw

PATCH v9 8/18 objtool: add kunit_try_catch_throw to the noreturn list

In reply to Brendan Higgins

Fix the following warning seen on GCC 7.3:
kunit/test-test.o: warning: objtool: kunit_test_unsuccessful_try() falls through to next function kunit_test_catch()

kunit_try_catch_throw is a function added in the following patch in this series; it allows KUnit, a unit testing framework for the kernel, to bail out of a broken test. As a consequence, it is a new __noreturn function that objtool thinks is broken (as seen above). So fix this warning by adding kunit_try_catch_throw to objtool's noreturn list.

---
 tools/objtool/check.c | 1 +
 1 file changed, 1 insertion(+)
tools/objtool/check.c +1 -0
index 172f991957269..98db5fe85c797 100644
@@ -134,6 +134,7 @@ static int __dead_end_function(struct objtool_file *file, struct symbol *func,
 		"usercopy_abort",
 		"machine_real_restart",
 		"rewind_stack_do_exit",
+		"kunit_try_catch_throw",
 	};
 
 	if (func->bind == STB_WEAK)
Signature · 6 lines
2.22.0.410.gd8fdbe21b5-goog

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#47 lore raw

PATCH v9 9/18 kunit: test: add support for test abort

In reply to Brendan Higgins

Add support for aborting/bailing out of test cases, which is needed for implementing assertions.

An assertion is like an expectation, but bails out of the test case early if the assertion is not met. The idea with assertions is that you use them to state all the preconditions for your test. Logically speaking, these are the premises of the test case, so if a premise isn't true, there is no point in continuing the test case because there are no conclusions that can be drawn without the premises. Whereas, the expectation is the thing you are trying to prove.

---
 include/kunit/test.h      |  16 ++++
 include/kunit/try-catch.h |  69 +++++++++++++++
 kunit/Makefile            |   3 +-
 kunit/test.c              | 176 +++++++++++++++++++++++++++++++++++---
 kunit/try-catch.c         |  95 ++++++++++++++++++++
 5 files changed, 344 insertions(+), 15 deletions(-)
 create mode 100644 include/kunit/try-catch.h
 create mode 100644 kunit/try-catch.c
include/kunit/test.h +16 -0
index df18765e6ea25..01440f3569847 100644
@@ -13,6 +13,7 @@
 #include <linux/types.h>
 #include <linux/slab.h>
 #include <kunit/kunit-stream.h>
+#include <kunit/try-catch.h>
 
 struct kunit_resource;
 
@@ -167,6 +168,7 @@ struct kunit {
 
 	/* private: internal use only. */
 	const char *name; /* Read only after initialization! */
+	struct kunit_try_catch try_catch;
 	/*
 	 * success starts as true, and may only be set to false during a test
 	 * case; thus, it is safe to update this across multiple threads using
@@ -176,6 +178,11 @@ struct kunit {
 	 */
 	bool success; /* Read only after test_case finishes! */
 	struct mutex lock; /* Gaurds all mutable test state. */
+	/*
+	 * death_test may be both set and unset from multiple threads in a test
+	 * case.
+	 */
+	bool death_test; /* Protected by lock. */
 	/*
 	 * Because resources is a list that may be updated multiple times (with
 	 * new resources) from any thread associated with a test case, we must
@@ -184,10 +191,19 @@ struct kunit {
 	struct list_head resources; /* Protected by lock. */
 };
 
+static inline void kunit_set_death_test(struct kunit *test, bool death_test)
+{
+	mutex_lock(&test->lock);
+	test->death_test = death_test;
+	mutex_unlock(&test->lock);
+}
+
 void kunit_init_test(struct kunit *test, const char *name);
 
 void kunit_fail(struct kunit *test, struct kunit_stream *stream);
 
+void kunit_abort(struct kunit *test);
+
 int kunit_run_tests(struct kunit_suite *suite);
 
 /**
include/kunit/try-catch.h new file +69 -0

69 lines not on this page · raw message

kunit/Makefile +2 -1
index 60a9ea6cb4697..1f7680cfa11ad 100644
@@ -1,6 +1,7 @@
 obj-$(CONFIG_KUNIT) +=			test.o \
 					string-stream.o \
-					kunit-stream.o
+					kunit-stream.o \
+					try-catch.o
 
 obj-$(CONFIG_KUNIT_TEST) +=		string-stream-test.o
 
kunit/test.c +162 -14

222 lines not on this page · raw message

kunit/try-catch.c new file +95 -0

95 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#48 lore raw

PATCH v9 10/18 kunit: test: add tests for kunit test abort

In reply to Brendan Higgins

Add KUnit tests for the KUnit test abort mechanism (see preceding commit). Add tests both for general try catch mechanism as well as non-architecture specific mechanism.

---
 kunit/Makefile    |   3 +-
 kunit/test-test.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 103 insertions(+), 1 deletion(-)
 create mode 100644 kunit/test-test.c
kunit/Makefile +2 -1
index 1f7680cfa11ad..533355867abd2 100644
@@ -3,6 +3,7 @@ obj-$(CONFIG_KUNIT) +=			test.o \
 					kunit-stream.o \
 					try-catch.o
 
-obj-$(CONFIG_KUNIT_TEST) +=		string-stream-test.o
+obj-$(CONFIG_KUNIT_TEST) +=		test-test.o \
+					string-stream-test.o
 
 obj-$(CONFIG_KUNIT_EXAMPLE_TEST) +=	example-test.o
kunit/test-test.c new file +101 -0

101 lines not on this page · raw message

Signature · 6 lines
2.22.0.410.gd8fdbe21b5-goog

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#49 lore raw

PATCH v9 11/18 kunit: test: add the concept of assertions

In reply to Brendan Higgins

Add support for assertions which are like expectations except the test terminates if the assertion is not satisfied.

The idea with assertions is that you use them to state all the preconditions for your test. Logically speaking, these are the premises of the test case, so if a premise isn't true, there is no point in continuing the test case because there are no conclusions that can be drawn without the premises. Whereas, the expectation is the thing you are trying to prove. It is not used universally in x-unit style test frameworks, but I really like it as a convention. You could still express the idea of a premise using the above idiom, but I think KUNIT_ASSERT_* states the intended idea perfectly.

---
 include/kunit/test.h       | 499 ++++++++++++++++++++++++++++++++++++-
 kunit/string-stream-test.c |  12 +-
 kunit/test-test.c          |   2 +
 kunit/test.c               |  66 +++++
 4 files changed, 570 insertions(+), 9 deletions(-)
include/kunit/test.h +496 -3
index 01440f3569847..e97273eb52f55 100644
@@ -86,9 +86,10 @@ struct kunit;
  * @name: the name of the test case.
  *
  * A test case is a function with the signature, ``void (*)(struct kunit *)``
- * that makes expectations (see KUNIT_EXPECT_TRUE()) about code under test. Each
- * test case is associated with a &struct kunit_suite and will be run after the
- * suite's init function and followed by the suite's exit function.
+ * that makes expectations and assertions (see KUNIT_EXPECT_TRUE() and
+ * KUNIT_ASSERT_TRUE()) about code under test. Each test case is associated with
+ * a &struct kunit_suite and will be run after the suite's init function and
+ * followed by the suite's exit function.
  *
  * A test case should be static and should only be created with the KUNIT_CASE()
  * macro; additionally, every array of test cases should be terminated with an
@@ -836,4 +837,496 @@ do {									       \
 	KUNIT_EXPECT_END(test, !IS_ERR_OR_NULL(__ptr), __stream);	       \
 } while (0)
 
+static inline struct kunit_stream *kunit_assert_start(struct kunit *test,
+						      const char *file,
+						      const char *line)
+{
+	struct kunit_stream *stream = alloc_kunit_stream(test, KERN_ERR);
+
+	kunit_stream_add(stream, "ASSERTION FAILED at %s:%s\n\t", file, line);
+
+	return stream;
+}
+
+static inline void kunit_assert_end(struct kunit *test,
+				    bool success,
+				    struct kunit_stream *stream)
+{
+	if (!success) {
+		kunit_fail(test, stream);
+		kunit_abort(test);
+	} else {
+		kunit_stream_clear(stream);
+	}
+}
+
+#define KUNIT_ASSERT_START(test) \
+		kunit_assert_start(test, __FILE__, __stringify(__LINE__))
+
+#define KUNIT_ASSERT_END(test, success, stream) \
+		kunit_assert_end(test, success, stream)
+
+#define KUNIT_ASSERT(test, success, message) do {			       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+									       \
+	kunit_stream_add(__stream, message);				       \
+	KUNIT_ASSERT_END(test, success, __stream);			       \
+} while (0)
+
+#define KUNIT_ASSERT_MSG(test, success, message, fmt, ...) do {		       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+									       \
+	kunit_stream_add(__stream, message);				       \
+	kunit_stream_add(__stream, fmt, ##__VA_ARGS__);			       \
+	KUNIT_ASSERT_END(test, success, __stream);			       \
+} while (0)
+
+#define KUNIT_ASSERT_FAILURE(test, fmt, ...) do {			       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+									       \
+	kunit_stream_add(__stream, fmt, ##__VA_ARGS__);			       \
+	KUNIT_ASSERT_END(test, false, __stream);			       \
+} while (0)
+
+/**
+ * KUNIT_ASSERT_TRUE() - Sets an assertion that @condition is true.
+ * @test: The test context object.
+ * @condition: an arbitrary boolean expression. The test fails and aborts when
+ * this does not evaluate to true.
+ *
+ * This and assertions of the form `KUNIT_ASSERT_*` will cause the test case to
+ * fail *and immediately abort* when the specified condition is not met. Unlike
+ * an expectation failure, it will prevent the test case from continuing to run;
+ * this is otherwise known as an *assertion failure*.
+ */
+#define KUNIT_ASSERT_TRUE(test, condition)				       \
+		KUNIT_ASSERT(test, (condition),				       \
+			     "Asserted " #condition " is true, but is false\n")
+
+#define KUNIT_ASSERT_TRUE_MSG(test, condition, fmt, ...)		       \
+		KUNIT_ASSERT_MSG(test, (condition),			       \
+				 "Asserted " #condition " is true, but is false\n",\
+				 fmt, ##__VA_ARGS__)
+
+/**
+ * KUNIT_ASSERT_FALSE() - Sets an assertion that @condition is false.
+ * @test: The test context object.
+ * @condition: an arbitrary boolean expression.
+ *
+ * Sets an assertion that the value that @condition evaluates to is false. This
+ * is the same as KUNIT_EXPECT_FALSE(), except it causes an assertion failure
+ * (see KUNIT_ASSERT_TRUE()) when the assertion is not met.
+ */
+#define KUNIT_ASSERT_FALSE(test, condition)				       \
+		KUNIT_ASSERT(test, !(condition),			       \
+			     "Asserted " #condition " is false, but is true\n")
+
+#define KUNIT_ASSERT_FALSE_MSG(test, condition, fmt, ...)		       \
+		KUNIT_ASSERT_MSG(test, !(condition),			       \
+				 "Asserted " #condition " is false, but is true\n",\
+				 fmt, ##__VA_ARGS__)
+
+void kunit_assert_binary_msg(struct kunit *test,
+			     long long left, const char *left_name,
+			     long long right, const char *right_name,
+			     bool compare_result,
+			     const char *compare_name,
+			     const char *file,
+			     const char *line,
+			     const char *fmt, ...);
+
+static inline void kunit_assert_binary(struct kunit *test,
+				       long long left, const char *left_name,
+				       long long right, const char *right_name,
+				       bool compare_result,
+				       const char *compare_name,
+				       const char *file,
+				       const char *line)
+{
+	kunit_assert_binary_msg(test,
+				left, left_name,
+				right, right_name,
+				compare_result,
+				compare_name,
+				file,
+				line,
+				NULL);
+}
+
+void kunit_assert_ptr_binary_msg(struct kunit *test,
+				 void *left, const char *left_name,
+				 void *right, const char *right_name,
+				 bool compare_result,
+				 const char *compare_name,
+				 const char *file,
+				 const char *line,
+				 const char *fmt, ...);
+
+static inline void kunit_assert_ptr_binary(struct kunit *test,
+					   void *left, const char *left_name,
+					   void *right, const char *right_name,
+					   bool compare_result,
+					   const char *compare_name,
+					   const char *file,
+					   const char *line)
+{
+	kunit_assert_ptr_binary_msg(test,
+				    left, left_name,
+				    right, right_name,
+				    compare_result,
+				    compare_name,
+				    file,
+				    line,
+				    NULL);
+}
+
+/*
+ * A factory macro for defining the expectations for the basic comparisons
+ * defined for the built in types.
+ *
+ * Unfortunately, there is no common type that all types can be promoted to for
+ * which all the binary operators behave the same way as for the actual types
+ * (for example, there is no type that long long and unsigned long long can
+ * both be cast to where the comparison result is preserved for all values). So
+ * the best we can do is do the comparison in the original types and then coerce
+ * everything to long long for printing; this way, the comparison behaves
+ * correctly and the printed out value usually makes sense without
+ * interpretation, but can always be interpretted to figure out the actual
+ * value.
+ */
+#define KUNIT_ASSERT_BINARY(test, left, condition, right) do {		       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+	__kunit_typecheck(__left, __right);				       \
+	kunit_assert_binary(test,					       \
+			    (long long) __left, #left,			       \
+			    (long long) __right, #right,		       \
+			    __left condition __right, #condition,	       \
+			    __FILE__, __stringify(__LINE__));		       \
+} while (0)
+
+#define KUNIT_ASSERT_BINARY_MSG(test, left, condition, right, fmt, ...) do {   \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+	__kunit_typecheck(__left, __right);				       \
+	kunit_assert_binary_msg(test,					       \
+				(long long) __left, #left,		       \
+				(long long) __right, #right,		       \
+				__left condition __right, #condition,	       \
+				__FILE__, __stringify(__LINE__),	       \
+				fmt, ##__VA_ARGS__);			       \
+} while (0)
+
+/*
+ * Just like KUNIT_EXPECT_BINARY, but for comparing pointer types.
+ */
+#define KUNIT_ASSERT_PTR_BINARY(test, left, condition, right) do {	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+	__kunit_typecheck(__left, __right);				       \
+	kunit_assert_ptr_binary(test,					       \
+				(void *) __left, #left,			       \
+				(void *) __right, #right,		       \
+				__left condition __right, #condition,	       \
+				__FILE__, __stringify(__LINE__));	       \
+} while (0)
+
+#define KUNIT_ASSERT_PTR_BINARY_MSG(test, left, condition, right, fmt, ...)    \
+do {									       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+	__kunit_typecheck(__left, __right);				       \
+	kunit_assert_ptr_binary_msg(test,				       \
+				    (void *) __left, #left,		       \
+				    (void *) __right, #right,		       \
+				    __left condition __right, #condition,      \
+				    __FILE__, __stringify(__LINE__),	       \
+				    fmt, ##__VA_ARGS__);		       \
+} while (0)
+
+/**
+ * KUNIT_ASSERT_EQ() - Sets an assertion that @left and @right are equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an assertion that the values that @left and @right evaluate to are
+ * equal. This is the same as KUNIT_EXPECT_EQ(), except it causes an assertion
+ * failure (see KUNIT_ASSERT_TRUE()) when the assertion is not met.
+ */
+#define KUNIT_ASSERT_EQ(test, left, right) \
+		KUNIT_ASSERT_BINARY(test, left, ==, right)
+
+#define KUNIT_ASSERT_EQ_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_ASSERT_BINARY_MSG(test,				       \
+					left,				       \
+					==,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_ASSERT_PTR_EQ() - Asserts that pointers @left and @right are equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a pointer.
+ * @right: an arbitrary expression that evaluates to a pointer.
+ *
+ * Sets an assertion that the values that @left and @right evaluate to are
+ * equal. This is the same as KUNIT_EXPECT_EQ(), except it causes an assertion
+ * failure (see KUNIT_ASSERT_TRUE()) when the assertion is not met.
+ */
+#define KUNIT_ASSERT_PTR_EQ(test, left, right) \
+		KUNIT_ASSERT_PTR_BINARY(test, left, ==, right)
+
+#define KUNIT_ASSERT_PTR_EQ_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_ASSERT_PTR_BINARY_MSG(test,			       \
+					    left,			       \
+					    ==,				       \
+					    right,			       \
+					    fmt,			       \
+					    ##__VA_ARGS__)
+
+/**
+ * KUNIT_ASSERT_NE() - An assertion that @left and @right are not equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an assertion that the values that @left and @right evaluate to are not
+ * equal. This is the same as KUNIT_EXPECT_NE(), except it causes an assertion
+ * failure (see KUNIT_ASSERT_TRUE()) when the assertion is not met.
+ */
+#define KUNIT_ASSERT_NE(test, left, right) \
+		KUNIT_ASSERT_BINARY(test, left, !=, right)
+
+#define KUNIT_ASSERT_NE_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_ASSERT_BINARY_MSG(test,				       \
+					left,				       \
+					!=,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_ASSERT_PTR_NE() - Asserts that pointers @left and @right are not equal.
+ * KUNIT_ASSERT_PTR_EQ() - Asserts that pointers @left and @right are equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a pointer.
+ * @right: an arbitrary expression that evaluates to a pointer.
+ *
+ * Sets an assertion that the values that @left and @right evaluate to are not
+ * equal. This is the same as KUNIT_EXPECT_NE(), except it causes an assertion
+ * failure (see KUNIT_ASSERT_TRUE()) when the assertion is not met.
+ */
+#define KUNIT_ASSERT_PTR_NE(test, left, right) \
+		KUNIT_ASSERT_PTR_BINARY(test, left, !=, right)
+
+#define KUNIT_ASSERT_PTR_NE_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_ASSERT_PTR_BINARY_MSG(test,			       \
+					    left,			       \
+					    !=,				       \
+					    right,			       \
+					    fmt,			       \
+					    ##__VA_ARGS__)
+/**
+ * KUNIT_ASSERT_LT() - An assertion that @left is less than @right.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an assertion that the value that @left evaluates to is less than the
+ * value that @right evaluates to. This is the same as KUNIT_EXPECT_LT(), except
+ * it causes an assertion failure (see KUNIT_ASSERT_TRUE()) when the assertion
+ * is not met.
+ */
+#define KUNIT_ASSERT_LT(test, left, right) \
+		KUNIT_ASSERT_BINARY(test, left, <, right)
+
+#define KUNIT_ASSERT_LT_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_ASSERT_BINARY_MSG(test,				       \
+					left,				       \
+					<,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+/**
+ * KUNIT_ASSERT_LE() - An assertion that @left is less than or equal to @right.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an assertion that the value that @left evaluates to is less than or
+ * equal to the value that @right evaluates to. This is the same as
+ * KUNIT_EXPECT_LE(), except it causes an assertion failure (see
+ * KUNIT_ASSERT_TRUE()) when the assertion is not met.
+ */
+#define KUNIT_ASSERT_LE(test, left, right) \
+		KUNIT_ASSERT_BINARY(test, left, <=, right)
+
+#define KUNIT_ASSERT_LE_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_ASSERT_BINARY_MSG(test,				       \
+					left,				       \
+					<=,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+/**
+ * KUNIT_ASSERT_GT() - An assertion that @left is greater than @right.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an assertion that the value that @left evaluates to is greater than the
+ * value that @right evaluates to. This is the same as KUNIT_EXPECT_GT(), except
+ * it causes an assertion failure (see KUNIT_ASSERT_TRUE()) when the assertion
+ * is not met.
+ */
+#define KUNIT_ASSERT_GT(test, left, right) \
+		KUNIT_ASSERT_BINARY(test, left, >, right)
+
+#define KUNIT_ASSERT_GT_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_ASSERT_BINARY_MSG(test,				       \
+					left,				       \
+					>,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_ASSERT_GE() - Assertion that @left is greater than or equal to @right.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a primitive C type.
+ * @right: an arbitrary expression that evaluates to a primitive C type.
+ *
+ * Sets an assertion that the value that @left evaluates to is greater than the
+ * value that @right evaluates to. This is the same as KUNIT_EXPECT_GE(), except
+ * it causes an assertion failure (see KUNIT_ASSERT_TRUE()) when the assertion
+ * is not met.
+ */
+#define KUNIT_ASSERT_GE(test, left, right) \
+		KUNIT_ASSERT_BINARY(test, left, >=, right)
+
+#define KUNIT_ASSERT_GE_MSG(test, left, right, fmt, ...)		       \
+		KUNIT_ASSERT_BINARY_MSG(test,				       \
+					left,				       \
+					>=,				       \
+					right,				       \
+					fmt,				       \
+					##__VA_ARGS__)
+
+/**
+ * KUNIT_ASSERT_STREQ() - An assertion that strings @left and @right are equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a null terminated string.
+ * @right: an arbitrary expression that evaluates to a null terminated string.
+ *
+ * Sets an assertion that the values that @left and @right evaluate to are
+ * equal. This is the same as KUNIT_EXPECT_STREQ(), except it causes an
+ * assertion failure (see KUNIT_ASSERT_TRUE()) when the assertion is not met.
+ */
+#define KUNIT_ASSERT_STREQ(test, left, right) do {			       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+									       \
+	kunit_stream_add(__stream, "Asserted " #left " == " #right ", but\n"); \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #left, __left);	       \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #right, __right);	       \
+									       \
+	KUNIT_ASSERT_END(test, !strcmp(left, right), __stream);		       \
+} while (0)
+
+#define KUNIT_ASSERT_STREQ_MSG(test, left, right, fmt, ...) do {	       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+									       \
+	kunit_stream_add(__stream, "Asserted " #left " == " #right ", but\n"); \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #left, __left);	       \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #right, __right);	       \
+	kunit_stream_add(__stream, fmt, ##__VA_ARGS__);			       \
+									       \
+	KUNIT_ASSERT_END(test, !strcmp(left, right), __stream);		       \
+} while (0)
+
+/**
+ * KUNIT_ASSERT_STRNEQ() - Expects that strings @left and @right are not equal.
+ * @test: The test context object.
+ * @left: an arbitrary expression that evaluates to a null terminated string.
+ * @right: an arbitrary expression that evaluates to a null terminated string.
+ *
+ * Sets an expectation that the values that @left and @right evaluate to are
+ * not equal. This is semantically equivalent to
+ * KUNIT_ASSERT_TRUE(@test, strcmp((@left), (@right))). See KUNIT_ASSERT_TRUE()
+ * for more information.
+ */
+#define KUNIT_ASSERT_STRNEQ(test, left, right) do {			       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+									       \
+	kunit_stream_add(__stream, "Asserted " #left " == " #right ", but\n"); \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #left, __left);	       \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #right, __right);	       \
+									       \
+	KUNIT_ASSERT_END(test, strcmp(left, right), __stream);		       \
+} while (0)
+
+#define KUNIT_ASSERT_STRNEQ_MSG(test, left, right, fmt, ...) do {	       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+	typeof(left) __left = (left);					       \
+	typeof(right) __right = (right);				       \
+									       \
+	kunit_stream_add(__stream, "Asserted " #left " == " #right ", but\n"); \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #left, __left);	       \
+	kunit_stream_add(__stream, "\t\t%s == %s\n", #right, __right);	       \
+	kunit_stream_add(__stream, fmt, ##__VA_ARGS__);			       \
+									       \
+	KUNIT_ASSERT_END(test, strcmp(left, right), __stream);		       \
+} while (0)
+
+/**
+ * KUNIT_ASSERT_NOT_ERR_OR_NULL() - Assertion that @ptr is not null and not err.
+ * @test: The test context object.
+ * @ptr: an arbitrary pointer.
+ *
+ * Sets an assertion that the value that @ptr evaluates to is not null and not
+ * an errno stored in a pointer. This is the same as
+ * KUNIT_EXPECT_NOT_ERR_OR_NULL(), except it causes an assertion failure (see
+ * KUNIT_ASSERT_TRUE()) when the assertion is not met.
+ */
+#define KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr) do {			       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+	typeof(ptr) __ptr = (ptr);					       \
+									       \
+	if (!__ptr)							       \
+		kunit_stream_add(__stream,				       \
+				 "Asserted " #ptr " is not null, but is\n");  \
+	if (IS_ERR(__ptr))						       \
+		kunit_stream_add(__stream,				       \
+				 "Asserted " #ptr " is not error, but is: %ld\n",\
+				 PTR_ERR(__ptr));			       \
+									       \
+	KUNIT_ASSERT_END(test, !IS_ERR_OR_NULL(__ptr), __stream);	       \
+} while (0)
+
+#define KUNIT_ASSERT_NOT_ERR_OR_NULL_MSG(test, ptr, fmt, ...) do {	       \
+	struct kunit_stream *__stream = KUNIT_ASSERT_START(test);	       \
+	typeof(ptr) __ptr = (ptr);					       \
+									       \
+	if (!__ptr) {							       \
+		kunit_stream_add(__stream,				       \
+				 "Asserted " #ptr " is not null, but is\n");  \
+		kunit_stream_add(__stream, fmt, ##__VA_ARGS__);		       \
+	}								       \
+	if (IS_ERR(__ptr)) {						       \
+		kunit_stream_add(__stream,				       \
+				 "Asserted " #ptr " is not error, but is: %ld\n",\
+				 PTR_ERR(__ptr));			       \
+									       \
+		kunit_stream_add(__stream, fmt, ##__VA_ARGS__);		       \
+	}								       \
+	KUNIT_ASSERT_END(test, !IS_ERR_OR_NULL(__ptr), __stream);	       \
+} while (0)
+
 #endif /* _KUNIT_TEST_H */
kunit/string-stream-test.c +6 -6
index b5641b078b8f6..5f27d576d2daf 100644
@@ -34,7 +34,7 @@ static void string_stream_test_get_string(struct kunit *test)
 	string_stream_add(stream, " %s", "bar");
 
 	output = string_stream_get_string(stream);
-	KUNIT_EXPECT_STREQ(test, output, "Foo bar");
+	KUNIT_ASSERT_STREQ(test, output, "Foo bar");
 	kfree(output);
 }
 
@@ -48,16 +48,16 @@ static void string_stream_test_add_and_clear(struct kunit *test)
 		string_stream_add(stream, "A");
 
 	output = string_stream_get_string(stream);
-	KUNIT_EXPECT_STREQ(test, output, "AAAAAAAAAA");
-	KUNIT_EXPECT_EQ(test, stream->length, (size_t)10);
-	KUNIT_EXPECT_FALSE(test, string_stream_is_empty(stream));
+	KUNIT_ASSERT_STREQ(test, output, "AAAAAAAAAA");
+	KUNIT_ASSERT_EQ(test, stream->length, (size_t)10);
+	KUNIT_ASSERT_FALSE(test, string_stream_is_empty(stream));
 	kfree(output);
 
 	string_stream_clear(stream);
 
 	output = string_stream_get_string(stream);
-	KUNIT_EXPECT_STREQ(test, output, "");
-	KUNIT_EXPECT_TRUE(test, string_stream_is_empty(stream));
+	KUNIT_ASSERT_STREQ(test, output, "");
+	KUNIT_ASSERT_TRUE(test, string_stream_is_empty(stream));
 }
 
 static struct kunit_case string_stream_test_cases[] = {
kunit/test-test.c +2 -0
index 88f4cdf03db2a..058f3fb37458a 100644
@@ -78,11 +78,13 @@ static int kunit_try_catch_test_init(struct kunit *test)
 	struct kunit_try_catch_test_context *ctx;
 
 	ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx);
 	test->priv = ctx;
 
 	ctx->try_catch = kunit_kmalloc(test,
 				       sizeof(*ctx->try_catch),
 				       GFP_KERNEL);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->try_catch);
 
 	return 0;
 }
kunit/test.c +66 -0

69 lines not on this page · raw message

Signature · 6 lines
2.22.0.410.gd8fdbe21b5-goog

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#50 lore raw

PATCH v9 12/18 kunit: test: add tests for KUnit managed resources

In reply to Brendan Higgins

From: Avinash Kondareddy <[email protected]>

Add unit tests for KUnit managed resources. KUnit managed resources (struct kunit_resource) are resources that are automatically cleaned up at the end of a KUnit test, similar to the concept of devm_* managed resources.

---
 kunit/test-test.c | 219 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 219 insertions(+)
kunit/test-test.c +219 -0
index 058f3fb37458a..b044659fe868b 100644
@@ -101,3 +101,222 @@ static struct kunit_suite kunit_try_catch_test_suite = {
 	.test_cases = kunit_try_catch_test_cases,
 };
 kunit_test_suite(kunit_try_catch_test_suite);
+
+/*
+ * Context for testing test managed resources
+ * is_resource_initialized is used to test arbitrary resources
+ */
+struct kunit_test_resource_context {
+	struct kunit test;
+	bool is_resource_initialized;
+	int allocate_order[2];
+	int free_order[2];
+};
+
+static int fake_resource_init(struct kunit_resource *res, void *context)
+{
+	struct kunit_test_resource_context *ctx = context;
+
+	res->allocation = &ctx->is_resource_initialized;
+	ctx->is_resource_initialized = true;
+	return 0;
+}
+
+static void fake_resource_free(struct kunit_resource *res)
+{
+	bool *is_resource_initialized = res->allocation;
+
+	*is_resource_initialized = false;
+}
+
+static void kunit_resource_test_init_resources(struct kunit *test)
+{
+	struct kunit_test_resource_context *ctx = test->priv;
+
+	kunit_init_test(&ctx->test, "testing_test_init_test");
+
+	KUNIT_EXPECT_TRUE(test, list_empty(&ctx->test.resources));
+}
+
+static void kunit_resource_test_alloc_resource(struct kunit *test)
+{
+	struct kunit_test_resource_context *ctx = test->priv;
+	struct kunit_resource *res;
+	kunit_resource_free_t free = fake_resource_free;
+
+	res = kunit_alloc_resource(&ctx->test,
+				   fake_resource_init,
+				   fake_resource_free,
+				   ctx);
+
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, res);
+	KUNIT_EXPECT_PTR_EQ(test,
+			    &ctx->is_resource_initialized,
+			    (bool *) res->allocation);
+	KUNIT_EXPECT_TRUE(test, list_is_last(&res->node, &ctx->test.resources));
+	KUNIT_EXPECT_PTR_EQ(test, free, res->free);
+}
+
+static void kunit_resource_test_free_resource(struct kunit *test)
+{
+	struct kunit_test_resource_context *ctx = test->priv;
+	struct kunit_resource *res = kunit_alloc_resource(&ctx->test,
+							  fake_resource_init,
+							  fake_resource_free,
+							  ctx);
+
+	kunit_free_resource(&ctx->test, res);
+
+	KUNIT_EXPECT_FALSE(test, ctx->is_resource_initialized);
+	KUNIT_EXPECT_TRUE(test, list_empty(&ctx->test.resources));
+}
+
+static void kunit_resource_test_cleanup_resources(struct kunit *test)
+{
+	int i;
+	struct kunit_test_resource_context *ctx = test->priv;
+	struct kunit_resource *resources[5];
+
+	for (i = 0; i < ARRAY_SIZE(resources); i++) {
+		resources[i] = kunit_alloc_resource(&ctx->test,
+						    fake_resource_init,
+						    fake_resource_free,
+						    ctx);
+	}
+
+	kunit_cleanup(&ctx->test);
+
+	KUNIT_EXPECT_TRUE(test, list_empty(&ctx->test.resources));
+}
+
+static void kunit_resource_test_mark_order(int order_array[],
+					   size_t order_size,
+					   int key)
+{
+	int i;
+
+	for (i = 0; i < order_size && order_array[i]; i++)
+		;
+
+	order_array[i] = key;
+}
+
+#define KUNIT_RESOURCE_TEST_MARK_ORDER(ctx, order_field, key)		       \
+		kunit_resource_test_mark_order(ctx->order_field,	       \
+					       ARRAY_SIZE(ctx->order_field),   \
+					       key)
+
+static int fake_resource_2_init(struct kunit_resource *res, void *context)
+{
+	struct kunit_test_resource_context *ctx = context;
+
+	KUNIT_RESOURCE_TEST_MARK_ORDER(ctx, allocate_order, 2);
+
+	res->allocation = ctx;
+
+	return 0;
+}
+
+static void fake_resource_2_free(struct kunit_resource *res)
+{
+	struct kunit_test_resource_context *ctx = res->allocation;
+
+	KUNIT_RESOURCE_TEST_MARK_ORDER(ctx, free_order, 2);
+}
+
+static int fake_resource_1_init(struct kunit_resource *res, void *context)
+{
+	struct kunit_test_resource_context *ctx = context;
+
+	kunit_alloc_resource(&ctx->test,
+			     fake_resource_2_init,
+			     fake_resource_2_free,
+			     ctx);
+
+	KUNIT_RESOURCE_TEST_MARK_ORDER(ctx, allocate_order, 1);
+
+	res->allocation = ctx;
+
+	return 0;
+}
+
+static void fake_resource_1_free(struct kunit_resource *res)
+{
+	struct kunit_test_resource_context *ctx = res->allocation;
+
+	KUNIT_RESOURCE_TEST_MARK_ORDER(ctx, free_order, 1);
+}
+
+/*
+ * TODO([email protected]): replace the arrays that keep track of the
+ * order of allocation and freeing with strict mocks using the IN_SEQUENCE macro
+ * to assert allocation and freeing order when the feature becomes available.
+ */
+static void kunit_resource_test_proper_free_ordering(struct kunit *test)
+{
+	struct kunit_test_resource_context *ctx = test->priv;
+
+	/* fake_resource_1 allocates a fake_resource_2 in its init. */
+	kunit_alloc_resource(&ctx->test,
+			     fake_resource_1_init,
+			     fake_resource_1_free,
+			     ctx);
+
+	/*
+	 * Since fake_resource_2_init calls KUNIT_RESOURCE_TEST_MARK_ORDER
+	 * before returning to fake_resource_1_init, it should be the first to
+	 * put its key in the allocate_order array.
+	 */
+	KUNIT_EXPECT_EQ(test, ctx->allocate_order[0], 2);
+	KUNIT_EXPECT_EQ(test, ctx->allocate_order[1], 1);
+
+	kunit_cleanup(&ctx->test);
+
+	/*
+	 * Because fake_resource_2 finishes allocation before fake_resource_1,
+	 * fake_resource_1 should be freed first since it could depend on
+	 * fake_resource_2.
+	 */
+	KUNIT_EXPECT_EQ(test, ctx->free_order[0], 1);
+	KUNIT_EXPECT_EQ(test, ctx->free_order[1], 2);
+}
+
+static int kunit_resource_test_init(struct kunit *test)
+{
+	struct kunit_test_resource_context *ctx =
+			kzalloc(sizeof(*ctx), GFP_KERNEL);
+
+	if (!ctx)
+		return -ENOMEM;
+
+	test->priv = ctx;
+
+	kunit_init_test(&ctx->test, "test_test_context");
+
+	return 0;
+}
+
+static void kunit_resource_test_exit(struct kunit *test)
+{
+	struct kunit_test_resource_context *ctx = test->priv;
+
+	kunit_cleanup(&ctx->test);
+	kfree(ctx);
+}
+
+static struct kunit_case kunit_resource_test_cases[] = {
+	KUNIT_CASE(kunit_resource_test_init_resources),
+	KUNIT_CASE(kunit_resource_test_alloc_resource),
+	KUNIT_CASE(kunit_resource_test_free_resource),
+	KUNIT_CASE(kunit_resource_test_cleanup_resources),
+	KUNIT_CASE(kunit_resource_test_proper_free_ordering),
+	{}
+};
+
+static struct kunit_suite kunit_resource_test_suite = {
+	.name = "kunit-resource-test",
+	.init = kunit_resource_test_init,
+	.exit = kunit_resource_test_exit,
+	.test_cases = kunit_resource_test_cases,
+};
+kunit_test_suite(kunit_resource_test_suite);
Signature · 6 lines
2.22.0.410.gd8fdbe21b5-goog

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#51 lore raw

PATCH v9 13/18 kunit: tool: add Python wrappers for running KUnit tests

In reply to Brendan Higgins

From: Felix Guo <[email protected]>

The ultimate goal is to create minimal isolated test binaries; in the meantime we are using UML to provide the infrastructure to run tests, so define an abstract way to configure and run tests that allow us to change the context in which tests are built without affecting the user. This also makes pretty and dynamic error reporting, and a lot of other nice features easier.

kunit_config.py:
- parse .config and Kconfig files.

kunit_kernel.py: provides helper functions to:
- configure the kernel using kunitconfig.
- build the kernel with the appropriate configuration.
- provide function to invoke the kernel and stream the output back.

kunit_parser.py: parses raw logs returned out by kunit_kernel and displays them in a user friendly way.

test_data/*: samples of test data for testing kunit.py, kunit_config.py, etc.

---
 tools/testing/kunit/.gitignore                |   3 +
 tools/testing/kunit/kunit.py                  | 116 +++++++
 tools/testing/kunit/kunit_config.py           |  66 ++++
 tools/testing/kunit/kunit_kernel.py           | 148 +++++++++
 tools/testing/kunit/kunit_parser.py           | 290 ++++++++++++++++++
 tools/testing/kunit/kunit_tool_test.py        | 206 +++++++++++++
 .../test_is_test_passed-all_passed.log        |  32 ++
 .../test_data/test_is_test_passed-crash.log   |  69 +++++
 .../test_data/test_is_test_passed-failure.log |  36 +++
 .../test_is_test_passed-no_tests_run.log      |  75 +++++
 .../test_output_isolated_correctly.log        | 106 +++++++
 .../test_data/test_read_from_file.kconfig     |  17 +
 12 files changed, 1164 insertions(+)
 create mode 100644 tools/testing/kunit/.gitignore
 create mode 100755 tools/testing/kunit/kunit.py
 create mode 100644 tools/testing/kunit/kunit_config.py
 create mode 100644 tools/testing/kunit/kunit_kernel.py
 create mode 100644 tools/testing/kunit/kunit_parser.py
 create mode 100755 tools/testing/kunit/kunit_tool_test.py
 create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
 create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-crash.log
 create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-failure.log
 create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-no_tests_run.log
 create mode 100644 tools/testing/kunit/test_data/test_output_isolated_correctly.log
 create mode 100644 tools/testing/kunit/test_data/test_read_from_file.kconfig
tools/testing/kunit/.gitignore new file +3 -0
new file mode 100644
index 0000000000000..c791ff59a37a9
@@ -0,0 +1,3 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
\ No newline at end of file
tools/testing/kunit/kunit.py new file +116 -0

116 lines not on this page · raw message

tools/testing/kunit/kunit_config.py new file +66 -0

66 lines not on this page · raw message

tools/testing/kunit/kunit_kernel.py new file +148 -0

148 lines not on this page · raw message

tools/testing/kunit/kunit_parser.py new file +290 -0

290 lines not on this page · raw message

tools/testing/kunit/kunit_tool_test.py new file +206 -0

206 lines not on this page · raw message

tools/testing/kunit/test_data/test_is_test_passed-all_passed.log new file +32 -0
new file mode 100644
index 0000000000000..62ebc0288355c
@@ -0,0 +1,32 @@
+TAP version 14
+	# Subtest: sysctl_test
+	1..8
+	# sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed
+	ok 1 - sysctl_test_dointvec_null_tbl_data
+	# sysctl_test_dointvec_table_maxlen_unset: sysctl_test_dointvec_table_maxlen_unset passed
+	ok 2 - sysctl_test_dointvec_table_maxlen_unset
+	# sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed
+	ok 3 - sysctl_test_dointvec_table_len_is_zero
+	# sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed
+	ok 4 - sysctl_test_dointvec_table_read_but_position_set
+	# sysctl_test_dointvec_happy_single_positive: sysctl_test_dointvec_happy_single_positive passed
+	ok 5 - sysctl_test_dointvec_happy_single_positive
+	# sysctl_test_dointvec_happy_single_negative: sysctl_test_dointvec_happy_single_negative passed
+	ok 6 - sysctl_test_dointvec_happy_single_negative
+	# sysctl_test_dointvec_single_less_int_min: sysctl_test_dointvec_single_less_int_min passed
+	ok 7 - sysctl_test_dointvec_single_less_int_min
+	# sysctl_test_dointvec_single_greater_int_max: sysctl_test_dointvec_single_greater_int_max passed
+	ok 8 - sysctl_test_dointvec_single_greater_int_max
+kunit sysctl_test: all tests passed
+ok 1 - sysctl_test
+	# Subtest: example
+	1..2
+init_suite
+	# example_simple_test: initializing
+	# example_simple_test: example_simple_test passed
+	ok 1 - example_simple_test
+	# example_mock_test: initializing
+	# example_mock_test: example_mock_test passed
+	ok 2 - example_mock_test
+kunit example: all tests passed
+ok 2 - example
tools/testing/kunit/test_data/test_is_test_passed-crash.log new file +69 -0

69 lines not on this page · raw message

tools/testing/kunit/test_data/test_is_test_passed-failure.log new file +36 -0
new file mode 100644
index 0000000000000..2ed870b6de72f
@@ -0,0 +1,36 @@
+TAP version 14
+	# Subtest: sysctl_test
+	1..8
+	# sysctl_test_dointvec_null_tbl_data: sysctl_test_dointvec_null_tbl_data passed
+	ok 1 - sysctl_test_dointvec_null_tbl_data
+	# sysctl_test_dointvec_table_maxlen_unset: sysctl_test_dointvec_table_maxlen_unset passed
+	ok 2 - sysctl_test_dointvec_table_maxlen_unset
+	# sysctl_test_dointvec_table_len_is_zero: sysctl_test_dointvec_table_len_is_zero passed
+	ok 3 - sysctl_test_dointvec_table_len_is_zero
+	# sysctl_test_dointvec_table_read_but_position_set: sysctl_test_dointvec_table_read_but_position_set passed
+	ok 4 - sysctl_test_dointvec_table_read_but_position_set
+	# sysctl_test_dointvec_happy_single_positive: sysctl_test_dointvec_happy_single_positive passed
+	ok 5 - sysctl_test_dointvec_happy_single_positive
+	# sysctl_test_dointvec_happy_single_negative: sysctl_test_dointvec_happy_single_negative passed
+	ok 6 - sysctl_test_dointvec_happy_single_negative
+	# sysctl_test_dointvec_single_less_int_min: sysctl_test_dointvec_single_less_int_min passed
+	ok 7 - sysctl_test_dointvec_single_less_int_min
+	# sysctl_test_dointvec_single_greater_int_max: sysctl_test_dointvec_single_greater_int_max passed
+	ok 8 - sysctl_test_dointvec_single_greater_int_max
+kunit sysctl_test: all tests passed
+ok 1 - sysctl_test
+	# Subtest: example
+	1..2
+init_suite
+	# example_simple_test: initializing
+	# example_simple_test: EXPECTATION FAILED at kunit/example-test.c:30
+	Expected 1 + 1 == 3, but
+		1 + 1 == 2
+		3 == 3
+	# example_simple_test: example_simple_test failed
+	not ok 1 - example_simple_test
+	# example_mock_test: initializing
+	# example_mock_test: example_mock_test passed
+	ok 2 - example_mock_test
+kunit example: one or more tests failed
+not ok 2 - example
tools/testing/kunit/test_data/test_is_test_passed-no_tests_run.log new file +75 -0

75 lines not on this page · raw message

tools/testing/kunit/test_data/test_output_isolated_correctly.log new file +106 -0

106 lines not on this page · raw message

tools/testing/kunit/test_data/test_read_from_file.kconfig new file +17 -0
new file mode 100644
index 0000000000000..d2a4928ac773b
@@ -0,0 +1,17 @@
+#
+# Automatically generated file; DO NOT EDIT.
+# User Mode Linux/x86 4.12.0-rc3 Kernel Configuration
+#
+CONFIG_UML=y
+CONFIG_MMU=y
+
+#
+# UML-specific options
+#
+
+#
+# Host processor type and features
+#
+# CONFIG_MK8 is not set
+CONFIG_TEST=y
+CONFIG_EXAMPLE_TEST=y
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#52 lore raw

PATCH v9 14/18 kunit: defconfig: add defconfigs for building KUnit tests

In reply to Brendan Higgins

Add defconfig for UML and a fragment that can be used to configure other architectures for building KUnit tests. Add option to kunit_tool to use a defconfig to create the kunitconfig.

---
 arch/um/configs/kunit_defconfig              |  8 ++++++++
 tools/testing/kunit/configs/all_tests.config |  8 ++++++++
 tools/testing/kunit/kunit.py                 | 18 ++++++++++++++++--
 tools/testing/kunit/kunit_kernel.py          |  3 ++-
 4 files changed, 34 insertions(+), 3 deletions(-)
 create mode 100644 arch/um/configs/kunit_defconfig
 create mode 100644 tools/testing/kunit/configs/all_tests.config
arch/um/configs/kunit_defconfig new file +8 -0
new file mode 100644
index 0000000000000..bfe49689038f1
@@ -0,0 +1,8 @@
+CONFIG_OF=y
+CONFIG_OF_UNITTEST=y
+CONFIG_OF_OVERLAY=y
+CONFIG_I2C=y
+CONFIG_I2C_MUX=y
+CONFIG_KUNIT=y
+CONFIG_KUNIT_TEST=y
+CONFIG_KUNIT_EXAMPLE_TEST=y
tools/testing/kunit/configs/all_tests.config new file +8 -0
new file mode 100644
index 0000000000000..bfe49689038f1
@@ -0,0 +1,8 @@
+CONFIG_OF=y
+CONFIG_OF_UNITTEST=y
+CONFIG_OF_OVERLAY=y
+CONFIG_I2C=y
+CONFIG_I2C_MUX=y
+CONFIG_KUNIT=y
+CONFIG_KUNIT_TEST=y
+CONFIG_KUNIT_EXAMPLE_TEST=y
tools/testing/kunit/kunit.py +16 -2
index da11bd62a4b82..3e51dc4febfdc 100755
@@ -11,6 +11,7 @@ import argparse
 import sys
 import os
 import time
+import shutil
 
 from collections import namedtuple
 from enum import Enum, auto
@@ -21,7 +22,7 @@ import kunit_parser
 
 KunitResult = namedtuple('KunitResult', ['status','result'])
 
-KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs', 'build_dir'])
+KunitRequest = namedtuple('KunitRequest', ['raw_output','timeout', 'jobs', 'build_dir', 'defconfig'])
 
 class KunitStatus(Enum):
 	SUCCESS = auto()
@@ -29,8 +30,16 @@ class KunitStatus(Enum):
 	BUILD_FAILURE = auto()
 	TEST_FAILURE = auto()
 
+def create_default_kunitconfig():
+	if not os.path.exists(kunit_kernel.KUNITCONFIG_PATH):
+		shutil.copyfile('arch/um/configs/kunit_defconfig',
+				kunit_kernel.KUNITCONFIG_PATH)
+
 def run_tests(linux: kunit_kernel.LinuxSourceTree,
 	      request: KunitRequest) -> KunitResult:
+	if request.defconfig:
+		create_default_kunitconfig()
+
 	config_start = time.time()
 	success = linux.build_reconfig(request.build_dir)
 	config_end = time.time()
@@ -99,13 +108,18 @@ def main(argv, linux):
 				'directory.',
 				type=str, default=None, metavar='build_dir')
 
+	run_parser.add_argument('--defconfig',
+				help='Uses a default kunitconfig.',
+				action='store_true')
+
 	cli_args = parser.parse_args(argv)
 
 	if cli_args.subcommand == 'run':
 		request = KunitRequest(cli_args.raw_output,
 				       cli_args.timeout,
 				       cli_args.jobs,
-				       cli_args.build_dir)
+				       cli_args.build_dir,
+				       cli_args.defconfig)
 		result = run_tests(linux, request)
 		if result.status != KunitStatus.SUCCESS:
 			sys.exit(1)
tools/testing/kunit/kunit_kernel.py +2 -1
index 07c0abf2f47df..bf38768353313 100644
@@ -14,6 +14,7 @@ import os
 import kunit_config
 
 KCONFIG_PATH = '.config'
+KUNITCONFIG_PATH = 'kunitconfig'
 
 class ConfigError(Exception):
 	"""Represents an error trying to configure the Linux kernel."""
@@ -81,7 +82,7 @@ class LinuxSourceTree(object):
 
 	def __init__(self):
 		self._kconfig = kunit_config.Kconfig()
-		self._kconfig.read_from_file('kunitconfig')
+		self._kconfig.read_from_file(KUNITCONFIG_PATH)
 		self._ops = LinuxSourceTreeOperations()
 
 	def clean(self):
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#53 lore raw

PATCH v9 15/18 Documentation: kunit: add documentation for KUnit

In reply to Brendan Higgins

Add documentation for KUnit, the Linux kernel unit testing framework.
- Add intro and usage guide for KUnit
- Add API reference

---
 Documentation/dev-tools/index.rst           |   1 +
 Documentation/dev-tools/kunit/api/index.rst |  16 +
 Documentation/dev-tools/kunit/api/test.rst  |  14 +
 Documentation/dev-tools/kunit/faq.rst       |  62 +++
 Documentation/dev-tools/kunit/index.rst     |  79 +++
 Documentation/dev-tools/kunit/start.rst     | 180 ++++++
 Documentation/dev-tools/kunit/usage.rst     | 575 ++++++++++++++++++++
 7 files changed, 927 insertions(+)
 create mode 100644 Documentation/dev-tools/kunit/api/index.rst
 create mode 100644 Documentation/dev-tools/kunit/api/test.rst
 create mode 100644 Documentation/dev-tools/kunit/faq.rst
 create mode 100644 Documentation/dev-tools/kunit/index.rst
 create mode 100644 Documentation/dev-tools/kunit/start.rst
 create mode 100644 Documentation/dev-tools/kunit/usage.rst
Documentation/dev-tools/index.rst +1 -0
index b0522a4dd1073..09dee10d25928 100644
@@ -24,6 +24,7 @@ whole; patches welcome!
    gdb-kernel-debugging
    kgdb
    kselftest
+   kunit/index
 
 
 .. only::  subproject and html
Documentation/dev-tools/kunit/api/index.rst new file +16 -0
new file mode 100644
index 0000000000000..9b9bffe5d41a0
@@ -0,0 +1,16 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=============
+API Reference
+=============
+.. toctree::
+
+	test
+
+This section documents the KUnit kernel testing API. It is divided into the
+following sections:
+
+================================= ==============================================
+:doc:`test`                       documents all of the standard testing API
+                                  excluding mocking or mocking related features.
+================================= ==============================================
Documentation/dev-tools/kunit/api/test.rst new file +14 -0
new file mode 100644
index 0000000000000..d0ce19b1e1185
@@ -0,0 +1,14 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+========
+Test API
+========
+
+This file documents all of the standard testing API excluding mocking or mocking
+related features.
+
+.. kernel-doc:: include/kunit/test.h
+   :internal:
+
+.. kernel-doc:: include/kunit/kunit-stream.h
+   :internal:
Documentation/dev-tools/kunit/faq.rst new file +62 -0

62 lines not on this page · raw message

Documentation/dev-tools/kunit/index.rst new file +79 -0

79 lines not on this page · raw message

Documentation/dev-tools/kunit/start.rst new file +180 -0

180 lines not on this page · raw message

Documentation/dev-tools/kunit/usage.rst new file +575 -0

575 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#54 lore raw

PATCH v9 16/18 MAINTAINERS: add entry for KUnit the unit testing framework

In reply to Brendan Higgins

Add myself as maintainer of KUnit, the Linux kernel's unit testing framework.

---
 MAINTAINERS | 11 +++++++++++
 1 file changed, 11 insertions(+)
MAINTAINERS +11 -0
index 677ef41cb012c..48d04d180a988 100644
@@ -8599,6 +8599,17 @@ S:	Maintained
 F:	tools/testing/selftests/
 F:	Documentation/dev-tools/kselftest*
 
+KERNEL UNIT TESTING FRAMEWORK (KUnit)
+M:	Brendan Higgins <[email protected]>
+L:	[email protected]
+L:	[email protected]
+W:	https://google.github.io/kunit-docs/third_party/kernel/docs/
+S:	Maintained
+F:	Documentation/dev-tools/kunit/
+F:	include/kunit/
+F:	kunit/
+F:	tools/testing/kunit/
+
 KERNEL USERMODE HELPER
 M:	Luis Chamberlain <[email protected]>
 L:	[email protected]
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#55 lore raw

PATCH v9 17/18 kernel/sysctl-test: Add null pointer test for sysctl.c:proc_dointvec()

In reply to Brendan Higgins

From: Iurii Zaikin <[email protected]>

KUnit tests for initialized data behavior of proc_dointvec that is explicitly checked in the code. Includes basic parsing tests including int min/max overflow.

---
 kernel/Makefile      |   2 +
 kernel/sysctl-test.c | 392 +++++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug    |  11 ++
 3 files changed, 405 insertions(+)
 create mode 100644 kernel/sysctl-test.c
kernel/Makefile +2 -0
index a8d923b5481ba..50fd511cd0ee0 100644
@@ -114,6 +114,8 @@ obj-$(CONFIG_HAS_IOMEM) += iomem.o
 obj-$(CONFIG_ZONE_DEVICE) += memremap.o
 obj-$(CONFIG_RSEQ) += rseq.o
 
+obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o
+
 obj-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak.o
 KASAN_SANITIZE_stackleak.o := n
 KCOV_INSTRUMENT_stackleak.o := n
kernel/sysctl-test.c new file +392 -0

392 lines not on this page · raw message

lib/Kconfig.debug +11 -0
index cbdfae3798965..6f8007800a76f 100644
@@ -1939,6 +1939,17 @@ config TEST_SYSCTL
 
 	  If unsure, say N.
 
+config SYSCTL_KUNIT_TEST
+	bool "KUnit test for sysctl"
+	depends on KUNIT
+	help
+	  This builds the proc sysctl unit test, which runs on boot.
+	  Tests the API contract and implementation correctness of sysctl.
+	  For more information on KUnit and unit tests in general please refer
+	  to the KUnit documentation in Documentation/dev-tools/kunit/.
+
+	  If unsure, say N.
+
 config TEST_UDELAY
 	tristate "udelay test driver"
 	help
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#56 lore raw

PATCH v9 18/18 MAINTAINERS: add proc sysctl KUnit test to PROC SYSCTL section

In reply to Brendan Higgins

Add entry for the new proc sysctl KUnit test to the PROC SYSCTL section, and add Iurii as a maintainer.

---
 MAINTAINERS | 2 ++
 1 file changed, 2 insertions(+)
MAINTAINERS +2 -0
index 48d04d180a988..f8204c75114da 100644
@@ -12721,12 +12721,14 @@ F:	Documentation/filesystems/proc.txt
 PROC SYSCTL
 M:	Luis Chamberlain <[email protected]>
 M:	Kees Cook <[email protected]>
+M:	Iurii Zaikin <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Maintained
 F:	fs/proc/proc_sysctl.c
 F:	include/linux/sysctl.h
 F:	kernel/sysctl.c
+F:	kernel/sysctl-test.c
 F:	tools/testing/selftests/sysctl/
 
 PS3 NETWORK SUPPORT
Signature · 6 lines
2.22.0.410.gd8fdbe21b5-goog

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel