support/testing: remove hardcoded sleep from python-django test
PATCH 1/2 support/testing: remove hardcoded sleep from python-django test
Instead of waiting for a hardcoded time of 30s we check periodically every second if the server is already up. If it isn't up after the full timeout (which is the same as before) expired the test fails.
We need to redirect all output of the background started task to /dev/null now as it otherwise confuses the emulator.run() exit code parsing logic (as it gets out of order messages from the emulator).
--- .../tests/package/test_python_django.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
support/testing/tests/package/test_python_django.py +12 -6
@@ -1,3 +1,5 @@ +import time + from tests.package.test_python import TestPythonPackageBase
@@ -16,13 +18,17 @@ class TestPythonDjango(TestPythonPackageBase): self.assertIn("Operations to perform:", output[0]) self.assertEqual(exit_code, 0) - cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.py runserver 0.0.0.0:1234 & " - # give some time to setup the server - cmd += "sleep {}".format(str(30 * self.emulator.timeout_multiplier)) + cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.py runserver 0.0.0.0:1234 > /dev/null 2>&1 & " self.assertRunOk(cmd, timeout=timeout) - - cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234" - self.assertRunOk(cmd) + # give some time to setup the server + for attempt in range(30 * self.emulator.timeout_multiplier): + time.sleep(1) + cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234" + _, exit_code = self.emulator.run(cmd) + if exit_code == 0: + break + else: + self.assertTrue(False, "Timeout while waiting for django server") class TestPythonPy3Django(TestPythonDjango):
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/2 package/python-whitenoise: new package
The test is using the django integration of whitenoise as it's the most common setup and allows to model the test case after the django one as well.
The setup we need to do is a bit more complicated though and follows the whitenoise getting started documentation [1].
We then request a .css file from the django admin app that is enabled by default in template project. Due to running django's development server with --nostatic we ensure that static file handling is taken over by whitenoise.
[1] https://whitenoise.readthedocs.io/en/stable/django.html
--- package/Config.in | 1 + package/python-whitenoise/Config.in | 7 ++++ .../python-whitenoise/python-whitenoise.hash | 5 +++ .../python-whitenoise/python-whitenoise.mk | 14 +++++++ .../tests/package/test_python_django.py | 1 + .../tests/package/test_python_whitenoise.py | 41 +++++++++++++++++++ 6 files changed, 69 insertions(+) create mode 100644 package/python-whitenoise/Config.in create mode 100644 package/python-whitenoise/python-whitenoise.hash create mode 100644 package/python-whitenoise/python-whitenoise.mk create mode 100644 support/testing/tests/package/test_python_whitenoise.py
package/Config.in +1 -0
@@ -1417,6 +1417,7 @@ menu "External python modules" source "package/python-websocket-client/Config.in" source "package/python-websockets/Config.in" source "package/python-werkzeug/Config.in" + source "package/python-whitenoise/Config.in" source "package/python-whoosh/Config.in" source "package/python-wrapt/Config.in" source "package/python-ws4py/Config.in"
package/python-whitenoise/Config.in new file +7 -0
@@ -0,0 +1,7 @@ +config BR2_PACKAGE_PYTHON_WHITENOISE + bool "python-whitenoise" + help + Radically simplified static file serving for WSGI + applications. + + https://github.com/evansd/whitenoise
package/python-whitenoise/python-whitenoise.hash new file +5 -0
@@ -0,0 +1,5 @@ +# md5, sha256 from https://pypi.org/pypi/whitenoise/json +md5 4926cee7317ac12533549c08043ee322 whitenoise-6.6.0.tar.gz +sha256 8998f7370973447fac1e8ef6e8ded2c5209a7b1f67c1012866dbcd09681c3251 whitenoise-6.6.0.tar.gz +# Locally computed sha256 checksums +sha256 ebfd469b4fb6b5adada547747e1e8da725ecf20595d54aced043275d4f4a3600 LICENSE
package/python-whitenoise/python-whitenoise.mk new file +14 -0
@@ -0,0 +1,14 @@ +################################################################################ +# +# python-whitenoise +# +################################################################################ + +PYTHON_WHITENOISE_VERSION = 6.6.0 +PYTHON_WHITENOISE_SOURCE = whitenoise-$(PYTHON_WHITENOISE_VERSION).tar.gz +PYTHON_WHITENOISE_SITE = https://files.pythonhosted.org/packages/16/e3/adddb43cf8eb924e18eca677d4e40d47348566224b724cb8d1eaf6a48d1b +PYTHON_WHITENOISE_SETUP_TYPE = setuptools +PYTHON_WHITENOISE_LICENSE = MIT +PYTHON_WHITENOISE_LICENSE_FILES = LICENSE + +$(eval $(python-package))
support/testing/tests/package/test_python_django.py +1 -0
@@ -36,6 +36,7 @@ class TestPythonPy3Django(TestPythonDjango): config = TestPythonDjango.config + \ """ BR2_PACKAGE_PYTHON3=y + BR2_PACKAGE_PYTHON3_PY_PYC=y BR2_PACKAGE_PYTHON_DJANGO=y BR2_PACKAGE_PYTHON3_SQLITE=y """
support/testing/tests/package/test_python_whitenoise.py new file +41 -0
41 lines not on this page · raw message
Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/python-whitenoise: new package
Hello Marcus,
support/testing/tests/package/test_python_django.py class TestPythonPy3Django(TestPythonDjango): · line 42
Quoted 12 lines · On Fri, 16 Feb 2024 14:18:14 +0100 Marcus Hoffmann via buildroot…
support/testing/tests/package/test_python_django.py +1
-0@@ -36,6 +36,7 @@ class TestPythonPy3Django(TestPythonDjango): config = TestPythonDjango.config + \ """ BR2_PACKAGE_PYTHON3=y + BR2_PACKAGE_PYTHON3_PY_PYC=y BR2_PACKAGE_PYTHON_DJANGO=y BR2_PACKAGE_PYTHON3_SQLITE=y """
I'm confused about this change. How is this related to whitenoise?
I see in reply to PATCH 1/2, you mentioned:
"""
The django test (and the whitenoise test introduced in the next patch) actually both currently fail because of a problem with django 5.0 and .pyc only installations. [1]
This now has "release blocker" priority at django and a proposed patch (that I tested locally but hasn't been officially submitted to django yet) so hopefully this will be fixed with the next django point release in a couple of weeks.
"""
But now PATCH 1/2 is merged, but not PATCH 2/2. Could you clarify this aspect? Is master broken because that snippet of PATCH 2/2 should have been in PATCH 1/2, or?
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/python-whitenoise: new package
Hi Thomas,
Quoted 5 lines · Thomas Petazzoni
Hello Marcus,
On Fri, 16 Feb 2024 14:18:14 +0100
Marcus Hoffmann via buildroot <[email protected]> wrote:
Quoted 14 lines
support/testing/tests/package/test_python_django.py class TestPythonPy3Django(TestPythonDjango): · line 42
Quoted 12 lines
support/testing/tests/package/test_python_django.py +1
-0@@ -36,6 +36,7 @@ class TestPythonPy3Django(TestPythonDjango): config = TestPythonDjango.config + \ """ BR2_PACKAGE_PYTHON3=y + BR2_PACKAGE_PYTHON3_PY_PYC=y BR2_PACKAGE_PYTHON_DJANGO=y BR2_PACKAGE_PYTHON3_SQLITE=y """I'm confused about this change. How is this related to whitenoise?
Not at all related and can just be dropped. This was a leftover from some Django debugging I did around the time I sent this patch.
Django (and this the django and whitenoise tests) were broken in February with out default .pyc only install method.
This has been fixed upstream in django and django has been updated to a fixed version in buildroot since then.
Quoted 14 lines
I see in reply to PATCH 1/2, you mentioned:
"""
The django test (and the whitenoise test introduced in the next patch) actually both currently fail because of a problem with django 5.0 and .pyc only installations. [1]This now has "release blocker" priority at django and a proposed patch (that I tested locally but hasn't been officially submitted to django yet) so hopefully this will be fixed with the next django point release in a couple of weeks.
"""
Quoted 5 lines
But now PATCH 1/2 is merged, but not PATCH 2/2. Could you clarify this aspect? Is master broken because that snippet of PATCH 2/2 should have been in PATCH 1/2, or?
Thomas
Marcus
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/python-whitenoise: new package
In reply to Marcus Hoffmann via buildrootHello Marcus,
Sorry for the huge delay in getting back to you.
Quoted 7 lines · On Fri, 16 Feb 2024 14:18:14 +0100 Marcus Hoffmann via buildroot…
The test is using the django integration of whitenoise as it's the most common setup and allows to model the test case after the django one as well.
The setup we need to do is a bit more complicated though and follows the whitenoise getting started documentation [1].
Quoted 8 lines
We then request a .css file from the django admin app that is enabled by default in template project. Due to running django's development server with --nostatic we ensure that static file handling is taken over by whitenoise.
[1] https://whitenoise.readthedocs.io/en/stable/django.html
Signed-off-by Marcus Hoffmann <[email protected]>
This causes some flake8 warnings, which I'm not sure how to fix. See below.
support/testing/tests/package/test_python_django.py class TestPythonPy3Django(TestPythonDjango): · line 42
Quoted 12 lines
support/testing/tests/package/test_python_django.py +1
-0@@ -36,6 +36,7 @@ class TestPythonPy3Django(TestPythonDjango): config = TestPythonDjango.config + \ """ BR2_PACKAGE_PYTHON3=y + BR2_PACKAGE_PYTHON3_PY_PYC=y BR2_PACKAGE_PYTHON_DJANGO=y BR2_PACKAGE_PYTHON3_SQLITE=y """
When resending, please drop this chunk.
Quoted 23 lines
support/testing/tests/package/test_python_whitenoise.py new file +17
-0@@ -0,0 +1,41 @@ +import time + +from tests.package.test_python import TestPythonPackageBase + + +class TestPythonPy3Whitenoise(TestPythonPackageBase): + __test__ = True + config = TestPythonPackageBase.config + \ + """ + BR2_PACKAGE_PYTHON3=y + BR2_PACKAGE_PYTHON_DJANGO=y + BR2_PACKAGE_PYTHON_WHITENOISE=y + BR2_PACKAGE_PYTHON3_SQLITE=y + """ + + def test_run(self): + self.login()
support/testing/tests/package/test_python_whitenoise.py line 27
Quoted 10 lines
support/testing/tests/package/test_python_whitenoise.py new file +10
-0+ timeout = 35 + + cmd = "cd /opt && /usr/bin/django-admin startproject testsite" + self.assertRunOk(cmd, timeout=timeout) + # STATIC_ROOT needs to be set for 'collectstatic' to work. + self.emulator.run("echo 'STATIC_ROOT = BASE_DIR / \"staticfiles\"' >> /opt/testsite/testsite/settings.py") + cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.py collectstatic" + self.assertRunOk(cmd, timeout=timeout) + # whitenoise docs say it needs to be added directly after SecurityMiddleware, so we do this here with sed. + cmd = """sed -i -e /django.middleware.security.SecurityMiddleware/a\ \\"whitenoise.middleware.WhiteNoiseMiddleware\\", /opt/testsite/testsite/settings.py"""
flake8 doesn't like this:
support/testing/tests/package/test_python_whitenoise.py:27:76: W605 invalid escape sequence '\ ' support/testing/tests/package/test_python_whitenoise.py:27:133: E501 line too long (164 > 132 characters)
I tried to use the "r" prefix for the string, which gets rid of the first warning, but then it doesn't do the right thing on the target.
Quoted 4 lines
support/testing/tests/package/test_python_whitenoise.py new file +4
-0+ self.assertRunOk(cmd, timeout=timeout) + # --nostatic ensures the builtin django server doesn't serve the static files, + # so we can test that whitenoise serves them + cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.py runserver --nostatic 0.0.0.0:1234 > /dev/null 2>&1 & "
support/testing/tests/package/test_python_whitenoise.py line 41
Quoted 10 lines
support/testing/tests/package/test_python_whitenoise.py new file +10
-0+ self.assertRunOk(cmd, timeout=timeout) + # give some time to setup the server + for attempt in range(30 * self.emulator.timeout_multiplier): + time.sleep(1) + cmd = "wget http://127.0.0.1:1234/static/admin/css/base.css" + _, exit_code = self.emulator.run(cmd) + if exit_code == 0: + break + else: + self.assertTrue(False, "Timeout while waiting for django server")
Could you replace self.assertTrue(False, "Timeout while waiting for django server") with the logic that used in test_python_django.py, ie:
for attempt in range(30 * self.emulator.timeout_multiplier):
time.sleep(1)
cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234"
_, exit_code = self.emulator.run(cmd)
if exit_code == 0:
break
self.assertEqual(exit_code, 0, "Timeout while waiting for django server")
Thanks a lot!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 2/2] package/python-whitenoise: new package
In reply to Marcus Hoffmann via buildrootQuoted 7 lines · On Fri, 16 Feb 2024 14:18:14 +0100 Marcus Hoffmann via buildroot…
package/Config.in | 1 + package/python-whitenoise/Config.in | 7 ++++ .../python-whitenoise/python-whitenoise.hash | 5 +++ .../python-whitenoise/python-whitenoise.mk | 14 +++++++ .../tests/package/test_python_django.py | 1 + .../tests/package/test_python_whitenoise.py | 41 +++++++++++++++++++ 6 files changed, 69 insertions(+)
Also: entry in the DEVELOPERS file is missing, for the package and the test. Thanks!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 37 lines · Marcus Hoffmann via buildroot
Instead of waiting for a hardcoded time of 30s we check periodically every second if the server is already up. If it isn't up after the full timeout (which is the same as before) expired the test fails.
We need to redirect all output of the background started task to
/dev/nullnow as it otherwise confuses theemulator.run()exit code parsing logic (as it gets out of order messages from the emulator).Signed-off-by Marcus Hoffmann <[email protected]>--- .../tests/package/test_python_django.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)support/testing/tests/package/test_python_django.py +5
-6@@ -1,3 +1,5 @@ +import time + from tests.package.test_python import TestPythonPackageBase@@ -16,13 +18,17 @@ class TestPythonDjango(TestPythonPackageBase): self.assertIn("Operations to perform:", output[0]) self.assertEqual(exit_code, 0) - cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.py runserver 0.0.0.0:1234 & " - # give some time to setup the server - cmd += "sleep {}".format(str(30 * self.emulator.timeout_multiplier)) + cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.py runserver 0.0.0.0:1234 > /dev/null 2>&1 & " self.assertRunOk(cmd, timeout=timeout) - - cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234" - self.assertRunOk(cmd) + # give some time to setup the server + for attempt in range(30 * self.emulator.timeout_multiplier):
support/testing/tests/package/test_python_django.py class TestPythonDjango(TestPythonPackageBase): · line 34
Quoted 10 lines
support/testing/tests/package/test_python_django.py +7
-0+ time.sleep(1) + cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234" + _, exit_code = self.emulator.run(cmd) + if exit_code == 0: + break + else: + self.assertTrue(False, "Timeout while waiting for django server") class TestPythonPy3Django(TestPythonDjango):
The django test (and the whitenoise test introduced in the next patch) actually both currently fail because of a problem with django 5.0 and .pyc only installations. [1]
This now has "release blocker" priority at django and a proposed patch (that I tested locally but hasn't been officially submitted to django yet) so hopefully this will be fixed with the next django point release in a couple of weeks.
[1] https://code.djangoproject.com/ticket/35187
Marcus
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 48 lines · Marcus Hoffmann via buildroot
Quoted 36 lines · Marcus Hoffmann via buildroot
Instead of waiting for a hardcoded time of 30s we check periodically every second if the server is already up. If it isn't up after the full timeout (which is the same as before) expired the test fails.
We need to redirect all output of the background started task to
/dev/nullnow as it otherwise confuses theemulator.run()exit code parsing logic (as it gets out of order messages from the emulator).Signed-off-by Marcus Hoffmann <[email protected]>---
.../tests/package/test_python_django.py | 18 ++++++++++++------1 file changed, 12 insertions(+), 6 deletions(-)support/testing/tests/package/test_python_django.py +2
-0@@ -1,3 +1,5 @@ +import time +from tests.package.test_python import TestPythonPackageBase
support/testing/tests/package/test_python_django.py +0
-1@@ -16,13 +18,17 @@ class TestPythonDjango(TestPythonPackageBase): self.assertIn("Operations to perform:", output[0]) self.assertEqual(exit_code, 0) - cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.pyrunserver 0.0.0.0:1234 & "
- # give some time to setup the server
- cmd += "sleep {}".format(str(30 * self.emulator.timeout_multiplier)) + cmd = "cd/opt/testsite&& " + self.interpreter + " ./manage.py runserver 0.0.0.0:1234 >/dev/null2>&1 & "
self.assertRunOk(cmd, timeout=timeout)
-
- cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234"
-self.assertRunOk(cmd)
+ # give some time to setup the server
+ for attempt inrange(30 * self.emulator.timeout_multiplier):Quoted 8 lines
+
time.sleep(1)
+ cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234" + _,exit_code=self.emulator.run(cmd)
+ ifexit_code== 0:
+ break
+ else:
+self.assertTrue(False, "Timeout while waiting for django server")class TestPythonPy3Django(TestPythonDjango):The django test (and the whitenoise test introduced in the next patch) actually both currently fail because of a problem with django 5.0 and .pyc only installations. [1]
Quoted 1 line
This now has "release blocker" priority at django and a proposed patch (that I tested locally but hasn't been officially submitted to django yet) so hopefully this will be fixed with the next django point release in a couple of weeks.
Ugh, and we're cutting 2024.02-rc1 right about now... I guess you can submit
that patch directly to Buildroot then? Or is the point release going to be a minor one that we can still apply to the master branch in the rc period?
Regards,
Arnout
Quoted 8 lines
[1] https://code.djangoproject.com/ticket/35187
Marcus
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Yann E. MORIN
AppliedMarcus, All,
On 2024-02-16 14:18 +0100, Marcus Hoffmann via buildroot spake thusly:
Quoted 34 lines
Instead of waiting for a hardcoded time of 30s we check periodically every second if the server is already up. If it isn't up after the full timeout (which is the same as before) expired the test fails.
We need to redirect all output of the background started task to
/dev/nullnow as it otherwise confuses theemulator.run()exit code parsing logic (as it gets out of order messages from the emulator).Signed-off-by Marcus Hoffmann <[email protected]>--- .../tests/package/test_python_django.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)support/testing/tests/package/test_python_django.py +3
-5@@ -1,3 +1,5 @@ +import time + from tests.package.test_python import TestPythonPackageBase@@ -16,13 +18,17 @@ class TestPythonDjango(TestPythonPackageBase): self.assertIn("Operations to perform:", output[0]) self.assertEqual(exit_code, 0) - cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.py runserver 0.0.0.0:1234 & " - # give some time to setup the server - cmd += "sleep {}".format(str(30 * self.emulator.timeout_multiplier)) + cmd = "cd /opt/testsite && " + self.interpreter + " ./manage.py runserver 0.0.0.0:1234 > /dev/null 2>&1 & " self.assertRunOk(cmd, timeout=timeout) - - cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234"
support/testing/tests/package/test_python_django.py class TestPythonDjango(TestPythonPackageBase): · line 31
Quoted 10 lines
support/testing/tests/package/test_python_django.py +9
-1- self.assertRunOk(cmd) + # give some time to setup the server + for attempt in range(30 * self.emulator.timeout_multiplier): + time.sleep(1) + cmd = "netstat -ltn 2>/dev/null | grep 0.0.0.0:1234" + _, exit_code = self.emulator.run(cmd) + if exit_code == 0: + break + else: + self.assertTrue(False, "Timeout while waiting for django server")
I was not very happy that we test success against a constant that we know is false; this does not look great..
Instead, I've slightly simplified the test: I dropped the else clause of the for-loop, and added an asserEqual that the exit_code is indeed 0 after the loop.
Applied to master, thanks.
Regards,
Yann E. MORIN.
Quoted 9 lines
support/testing/tests/package/test_python_django.py +0
-0class TestPythonPy3Django(TestPythonDjango):Signature · 6 lines
2.34.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot