support/testing: fix hardening tests
PATCH 1/1 support/testing: fix hardening tests
Since "2467822c85 package/checksec: bump to version 2.1.0" the hardening tests fail because upstream slightly changed the way the script is called.
According to README.md: "- All options now require --$option=$value instead of --$option $value"
Instead of just replacing '--output json' with '--output=json' take into account that upstream also changed the usage example to show --format instead of --output. Both options do exactly the same, but following the usage example seems to be more future-proof.
Upstream also improved the json output. Now when a file is passed as parameter, the json has the file name as the main key, instead of the string "file". Adjust the test cases accordingly.
Fixes:
tests.core.test_hardening.TestFortifyConserv
tests.core.test_hardening.TestFortifyNone
tests.core.test_hardening.TestRelro
tests.core.test_hardening.TestRelroPartial
tests.core.test_hardening.TestSspNone
tests.core.test_hardening.TestSspStrong
---
These failures were not caught by [1] yet.
[1] https://gitlab.com/buildroot.org/buildroot/pipelines
Before this patch:
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/74576170/failures
After this patch:
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592923
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592926
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592927
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592935
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592938
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592941
--- support/testing/tests/core/test_hardening.py | 25 +++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-)
support/testing/tests/core/test_hardening.py +16 -9
@@ -26,7 +26,8 @@ class TestHardeningBase(infra.basetest.BRTest): def checksec_run(self, target_file): filepath = os.path.join(self.builddir, "target", target_file) - cmd = ["host/bin/checksec", "--output", "json", "--file", filepath] + cmd = ["host/bin/checksec", "--format=json", + "--file={}".format(filepath)] # Checksec is being used for elf file analysis only. There are no # assumptions of target/run-time checks as part of this testing. ret = subprocess.check_output(cmd,
@@ -45,8 +46,9 @@ class TestRelro(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["relro"], "full") - self.assertEqual(out["file"]["pie"], "yes") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["relro"], "full") + self.assertEqual(out[filepath]["pie"], "yes") class TestRelroPartial(TestHardeningBase):
@@ -58,8 +60,9 @@ class TestRelroPartial(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["relro"], "partial") - self.assertEqual(out["file"]["pie"], "no") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["relro"], "partial") + self.assertEqual(out[filepath]["pie"], "no") class TestSspNone(TestHardeningBase):
@@ -71,7 +74,8 @@ class TestSspNone(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["canary"], "no") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["canary"], "no") class TestSspStrong(TestHardeningBase):
@@ -83,7 +87,8 @@ class TestSspStrong(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["canary"], "yes") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["canary"], "yes") class TestFortifyNone(TestHardeningBase):
@@ -95,7 +100,8 @@ class TestFortifyNone(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["fortified"], "0") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["fortified"], "0") class TestFortifyConserv(TestHardeningBase):
@@ -107,4 +113,5 @@ class TestFortifyConserv(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertNotEqual(out["file"]["fortified"], "0") + filepath = os.path.join(self.builddir, "target", f) + self.assertNotEqual(out[filepath]["fortified"], "0")
Signature · 1 line
2.17.1
PATCH 1/1 support/testing: fix hardening tests
Ricardo,
Quoted 35 lines · Ricardo Martincoski
Since "
2467822c85package/checksec: bump to version 2.1.0" the hardening tests fail because upstream slightly changed the way the script is called.
According to README.md: "- All options now require--$option=$valueinstead of--$option $value"Instead of just replacing '--output json' with '--output=json' take into account that upstream also changed the usage example to show --format instead of --output. Both options do exactly the same, but following the usage example seems to be more future-proof.
Upstream also improved the json output. Now when a file is passed as parameter, the json has the file name as the main key, instead of the string "file". Adjust the test cases accordingly.
Fixes:
tests.core.test_hardening.TestFortifyConserv
tests.core.test_hardening.TestFortifyNone
tests.core.test_hardening.TestRelro
tests.core.test_hardening.TestRelroPartial
tests.core.test_hardening.TestSspNone
tests.core.test_hardening.TestSspStrongSigned-off-by Ricardo Martincoski <[email protected]>Cc Fabrice Fontaine <[email protected]>Cc Matt Weber <[email protected]>Cc Thomas Petazzoni <[email protected]>---
These failures were not caught by [1] yet.
[1] https://gitlab.com/buildroot.org/buildroot/pipelinesBefore this patch:
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/74576170/failures
Quoted 7 lines
After this patch:
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592923
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592926
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592927
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592935
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592938
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/265592941
Quoted 82 lines
--- support/testing/tests/core/test_hardening.py | 25 +++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-)support/testing/tests/core/test_hardening.py +16
-10@@ -26,7 +26,8 @@ class TestHardeningBase(infra.basetest.BRTest): def checksec_run(self, target_file): filepath = os.path.join(self.builddir, "target", target_file) - cmd = ["host/bin/checksec", "--output", "json", "--file", filepath] + cmd = ["host/bin/checksec", "--format=json", + "--file={}".format(filepath)] # Checksec is being used for elf file analysis only. There are no # assumptions of target/run-time checks as part of this testing. ret = subprocess.check_output(cmd,@@ -45,8 +46,9 @@ class TestRelro(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["relro"], "full") - self.assertEqual(out["file"]["pie"], "yes") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["relro"], "full") + self.assertEqual(out[filepath]["pie"], "yes") class TestRelroPartial(TestHardeningBase):@@ -58,8 +60,9 @@ class TestRelroPartial(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["relro"], "partial") - self.assertEqual(out["file"]["pie"], "no") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["relro"], "partial") + self.assertEqual(out[filepath]["pie"], "no") class TestSspNone(TestHardeningBase):@@ -71,7 +74,8 @@ class TestSspNone(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["canary"], "no") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["canary"], "no") class TestSspStrong(TestHardeningBase):@@ -83,7 +87,8 @@ class TestSspStrong(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["canary"], "yes") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["canary"], "yes") class TestFortifyNone(TestHardeningBase):@@ -95,7 +100,8 @@ class TestFortifyNone(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertEqual(out["file"]["fortified"], "0") + filepath = os.path.join(self.builddir, "target", f) + self.assertEqual(out[filepath]["fortified"], "0") class TestFortifyConserv(TestHardeningBase):@@ -107,4 +113,5 @@ class TestFortifyConserv(TestHardeningBase): def test_run(self): for f in self.checksec_files: out = self.checksec_run(f) - self.assertNotEqual(out["file"]["fortified"], "0") + filepath = os.path.join(self.builddir, "target", f) + self.assertNotEqual(out[filepath]["fortified"], "0") --2.17.1
Signature · 23 lines
Matthew Weber | Associate Director Software Engineer | Commercial Avionics COLLINS AEROSPACE 400 Collins Road NE, Cedar Rapids, Iowa 52498, USA Tel: +1 319 295 7349 | FAX: +1 319 263 6099 matthew.weber at collins.com | collinsaerospace.com CONFIDENTIALITY WARNING: This message may contain proprietary and/or privileged information of Collins Aerospace and its affiliated companies. If you are not the intended recipient, please 1) Do not disclose, copy, distribute or use this message or its contents. 2) Advise the sender by return email. 3) Delete all copies (including all attachments) from your computer. Your cooperation is greatly appreciated. Any export restricted material should be shared using my matthew.weber at corp.rockwellcollins.com address.
Thomas Petazzoni
AppliedPATCH 1/1 support/testing: fix hardening tests
In reply to Ricardo MartincoskiQuoted 23 lines · On Sun, 4 Aug 2019 22:20:50 -0300 Ricardo Martincoski…
Since "
2467822c85package/checksec: bump to version 2.1.0" the hardening tests fail because upstream slightly changed the way the script is called.
According to README.md: "- All options now require--$option=$valueinstead of--$option $value"Instead of just replacing '--output json' with '--output=json' take into account that upstream also changed the usage example to show --format instead of --output. Both options do exactly the same, but following the usage example seems to be more future-proof.
Upstream also improved the json output. Now when a file is passed as parameter, the json has the file name as the main key, instead of the string "file". Adjust the test cases accordingly.
Fixes:
tests.core.test_hardening.TestFortifyConserv
tests.core.test_hardening.TestFortifyNone
tests.core.test_hardening.TestRelro
tests.core.test_hardening.TestRelroPartial
tests.core.test_hardening.TestSspNone
tests.core.test_hardening.TestSspStrong
Quoted 7 lines
Signed-off-by Ricardo Martincoski <[email protected]>Cc Fabrice Fontaine <[email protected]>Cc Matt Weber <[email protected]>Cc Thomas Petazzoni <[email protected]>---
These failures were not caught by [1] yet.
[1] https://gitlab.com/buildroot.org/buildroot/pipelines
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com