test: py: Disable sleep test for qemu targets

Qemu for arm32/arm64 has a problem with time setup.

That's why sleep test is failing. Add boardidentity marker to remove specific boards from running that test.

Signed-off-by Michal Simek <[email protected]>

---

If you want to add this checking in one patch and then put it to one test then it is fine for me.

---
  test/py/conftest.py         | 28 ++++++++++++++++++++++++++++
  test/py/pytest.ini          |  1 +
  test/py/tests/test_sleep.py |  1 +
  3 files changed, 30 insertions(+)

test/py/conftest.py +28 -0

index 6e66a48c15fd..1812ff340e6a 100644
@@ -436,6 +436,33 @@ def setup_boardspec(item):
      if required_boards and ubconfig.board_type not in required_boards:
          pytest.skip('board "%s" not supported' % ubconfig.board_type)
 
+def setup_boardidentity(item):
+    """Process any 'boardidentity' marker for a test.
+
+    Such a marker lists the set of board identity that a test does/doesn't
+    support. If tests are being executed on an unsupported board, the test is
+    marked to be skipped.
+
+    Args:
+        item: The pytest test item.
+
+    Returns:
+        Nothing.
+    """
+    mark = item.get_marker('boardidentity')
+    if not mark:
+        return
+    required_boards = []
+    for board in mark.args:
+        if board.startswith('!'):
+            if ubconfig.board_identity == board[1:]:
+                pytest.skip('board identity not supported')
+                return
+        else:
+            required_boards.append(board)
+    if required_boards and ubconfig.board_identity not in required_boards:
+        pytest.skip('board identity not supported')
+
  def setup_buildconfigspec(item):
      """Process any 'buildconfigspec' marker for a test.
 

@@ -503,6 +530,7 @@ def pytest_runtest_setup(item):
 
      start_test_section(item)
      setup_boardspec(item)
+    setup_boardidentity(item)
      setup_buildconfigspec(item)
      setup_requiredtool(item)
 

test/py/pytest.ini +1 -0

index 67e514f42058..9d64671814de 100644
@@ -8,4 +8,5 @@
  [pytest]
  markers =
      boardspec: U-Boot: Describes the set of boards a test can/can't run on.
+    boardidentity: U-Boot: Describes the board identity a test can/can't run on.
      buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.

test/py/tests/test_sleep.py +0 -0

index 64e057132622..02a8a85b0e22 100644

test/py/tests/test_sleep.py +0 -0

  def test_sleep(u_boot_console):
      """Test the sleep command, and validate that it sleeps for approximately
      the correct amount of time."""

Quoted 1 line · Michal Simek

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Quoted 3 lines

That's why sleep test is failing. Add boardidentity marker to remove specific boards from running that test.

Isn't this what 'boardspec' is used for?

test/py/pytest.ini:
boardspec: U-Boot: Describes the set of boards a test can/can't run on.

Quoted 73 lines

Signed-off-by Michal Simek <[email protected]>

---

If you want to add this checking in one patch and then put it to one test then it is fine for me.

---
  test/py/conftest.py         | 28 ++++++++++++++++++++++++++++ test/py/pytest.ini          |  1 + test/py/tests/test_sleep.py |  1 +

  3 files changed, 30 insertions(+)

test/py/conftest.py +0 -0

index 6e66a48c15fd..1812ff340e6a 100644
@@ -436,6 +436,33 @@ def setup_boardspec(item):

      if required_boards and ubconfig.board_type not in required_boards:
          pytest.skip('board "%s" not supported' % ubconfig.board_type) +def setup_boardidentity(item):
+    """Process any 'boardidentity' marker for a test.
+
+    Such a marker lists the set of board identity that a test does/doesn't
+    support. If tests are being executed on an unsupported board, the test is
+    marked to be skipped.

test/py/conftest.py +19 -0

+
+    Args:
+        item: The pytest test item.
+
+    Returns:
+        Nothing.
+    """
+    mark = item.get_marker('boardidentity')
+    if not mark:
+        return
+    required_boards = []
+    for board in mark.args:
+        if board.startswith('!'):
+            if ubconfig.board_identity == board[1:]:
+                pytest.skip('board identity not supported')
+                return
+        else:
+            required_boards.append(board)
+    if required_boards and ubconfig.board_identity not in

required_boards:
+        pytest.skip('board identity not supported') +
  def setup_buildconfigspec(item):
      """Process any 'buildconfigspec' marker for a test. @@ -503,6 +530,7 @@ def pytest_runtest_setup(item):
        start_test_section(item) setup_boardspec(item)
+    setup_boardidentity(item) setup_buildconfigspec(item) setup_requiredtool(item)
  diff --git a/test/py/pytest.ini b/test/py/pytest.ini index 67e514f42058..9d64671814de 100644

test/py/pytest.ini +0 -0

@@ -8,4 +8,5 @@

  [pytest]
  markers =
      boardspec: U-Boot: Describes the set of boards a test can/can't run on.
+    boardidentity: U-Boot: Describes the board identity a test can/can't run on.
      buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.

test/py/tests/test_sleep.py +0 -0

index 64e057132622..02a8a85b0e22 100644

Quoted 6 lines

test/py/tests/test_sleep.py +0 -0

@@ -5,6 +5,7 @@

  import pytest
  import time
  + at pytest.mark.boardidentity("!qemu")

According to your commit message you don't want to exclude qemu-x86 here.

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 25 lines · Tom Rini

Quoted 23 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Quoted 5 lines

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

ok. By locally do you mean just using -k not sleep?

Quoted 32 lines · Michal Simek

Quoted 30 lines · Tom Rini

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Quoted 33 lines · Tom Rini

Quoted 32 lines · Michal Simek

Quoted 30 lines · Tom Rini

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 34 lines · Michal Simek

Quoted 33 lines · Tom Rini

Quoted 32 lines · Michal Simek

Quoted 30 lines · Tom Rini

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

Quoted 55 lines · Tom Rini

Quoted 34 lines · Michal Simek

Quoted 33 lines · Tom Rini

Quoted 32 lines · Michal Simek

Quoted 30 lines · Tom Rini

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

Quoted 67 lines · Stephen Warren

Quoted 55 lines · Tom Rini

Quoted 34 lines · Michal Simek

Quoted 33 lines · Tom Rini

Quoted 32 lines · Michal Simek

Quoted 30 lines · Tom Rini

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

Quoted 6 lines

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?

Quoted 75 lines · Tom Rini

Quoted 67 lines · Stephen Warren

Quoted 55 lines · Tom Rini

Quoted 34 lines · Michal Simek

Quoted 33 lines · Tom Rini

Quoted 32 lines · Michal Simek

Quoted 30 lines · Tom Rini

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

Quoted 6 lines

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

Quoted 3 lines

I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?

For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5 seconds (including my reaction time), so ~13.2 does sound like it's probably a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with real-time? I'd hope not for something simple like this, assuming you're using a recent CPU, but maybe.

Quoted 80 lines · Stephen Warren

Quoted 75 lines · Tom Rini

Quoted 67 lines · Stephen Warren

Quoted 55 lines · Tom Rini

Quoted 34 lines · Michal Simek

Quoted 33 lines · Tom Rini

Quoted 32 lines · Michal Simek

Quoted 30 lines · Tom Rini

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

Quoted 6 lines

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

Quoted 3 lines

I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?

Quoted 6 lines

For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5 seconds (including my reaction time), so ~13.2 does sound like it's probably a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with real-time? I'd hope not for something simple like this, assuming you're using a recent CPU, but maybe.

Yeah, I can do x86, ARM and PowerPC but it fails on MIPS. And my build box isn't super new but an 8core/16thread E5-2670 should be good enough :)

Hey Daniel, do you have test.py running on real MIPS hardware? Thanks!

Quoted 80 lines · Stephen Warren

Quoted 75 lines · Tom Rini

Quoted 67 lines · Stephen Warren

Quoted 55 lines · Tom Rini

Quoted 34 lines · Michal Simek

Quoted 33 lines · Tom Rini

Quoted 32 lines · Michal Simek

Quoted 30 lines · Tom Rini

Quoted 18 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 7 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still slightly too strict about which delays it accepts, or is sleep outright broken?

Quoted 5 lines

You can use command-line option -k to avoid some tests. For example "-k not sleep". That way, we don't have to hard-code the dependency into the test source. Depending on the root cause (issue in U-Boot, or issue in just your local version of qemu, or something that will never work) this might be better?

Even with the most recent relaxing of the sleep test requirements, I can still (depending on overall system load) have 'sleep' take too long, on QEMU. I think it might have been half a second slow, but I don't have the log handy anymore. Both locally and in travis we -k not sleep all of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test. OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

Quoted 6 lines

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

Quoted 3 lines

I've pushed this up to 10 seconds and 0.5s worth of overrun and on qemu-mips here I see a 13.2s sleep. That's pretty close to 1/3rd fast and to me a wrong-clocking value, yes?

Quoted 6 lines

For me the qemu-x86 build of mid-Nov commit of U-Boot running under the same qemu version that U-Boot's Travis CI builds use, "sleep 10" takes about 10.5 seconds (including my reaction time), so ~13.2 does sound like it's probably a bug. Or maybe qemu just isn't fast enough in its emulation to keep up with real-time? I'd hope not for something simple like this, assuming you're using a recent CPU, but maybe.

Yeah, I can do x86, ARM and PowerPC but it fails on MIPS. And my build box isn't super new but an 8core/16thread E5-2670 should be good enough :)

Quoted 66 lines · Tom Rini

Quoted 45 lines · Michal Simek

Quoted 44 lines · Tom Rini

Quoted 43 lines · Michal Simek

Quoted 41 lines · Tom Rini

Quoted 21 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 8 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our
qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still
slightly too strict about which delays it accepts, or is sleep outright
broken?

Quoted 9 lines

You can use command-line option -k to avoid some tests. For example "-k not
sleep". That way, we don't have to hard-code the dependency into the test
source. Depending on the root cause (issue in U-Boot, or issue in just your
local version of qemu, or something that will never work) this might be
better?

Even with the most recent relaxing of the sleep test requirements, I can
still (depending on overall system load) have 'sleep' take too long, on
QEMU.  I think it might have been half a second slow, but I don't have
the log handy anymore.  Both locally and in travis we -k not sleep all
of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test.  OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

Quoted 79 lines · Stephen Warren

Quoted 66 lines · Tom Rini

Quoted 45 lines · Michal Simek

Quoted 44 lines · Tom Rini

Quoted 43 lines · Michal Simek

Quoted 41 lines · Tom Rini

Quoted 21 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 8 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our
qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still
slightly too strict about which delays it accepts, or is sleep outright
broken?

Quoted 9 lines

You can use command-line option -k to avoid some tests. For example "-k not
sleep". That way, we don't have to hard-code the dependency into the test
source. Depending on the root cause (issue in U-Boot, or issue in just your
local version of qemu, or something that will never work) this might be
better?

Even with the most recent relaxing of the sleep test requirements, I can
still (depending on overall system load) have 'sleep' take too long, on
QEMU.  I think it might have been half a second slow, but I don't have
the log handy anymore.  Both locally and in travis we -k not sleep all
of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test.  OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

Quoted 6 lines

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).

But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration.
Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.

Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True

Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True
to skip dhcp test for whatever reason.

Then for travis-ci we can just put these lines to py/travis-ci/.

What do you think?

Quoted 106 lines · Michal Simek

Quoted 79 lines · Stephen Warren

Quoted 66 lines · Tom Rini

Quoted 45 lines · Michal Simek

Quoted 44 lines · Tom Rini

Quoted 43 lines · Michal Simek

Quoted 41 lines · Tom Rini

Quoted 21 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 8 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our
qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still
slightly too strict about which delays it accepts, or is sleep outright
broken?

Quoted 9 lines

You can use command-line option -k to avoid some tests. For example "-k not
sleep". That way, we don't have to hard-code the dependency into the test
source. Depending on the root cause (issue in U-Boot, or issue in just your
local version of qemu, or something that will never work) this might be
better?

Even with the most recent relaxing of the sleep test requirements, I can
still (depending on overall system load) have 'sleep' take too long, on
QEMU.  I think it might have been half a second slow, but I don't have
the log handy anymore.  Both locally and in travis we -k not sleep all
of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test.  OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

Quoted 6 lines

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).

But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration.
Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.

Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True

Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True
to skip dhcp test for whatever reason.

Quoted 3 lines

Then for travis-ci we can just put these lines to py/travis-ci/.

What do you think?

Ah, good idea! We have a few cases like this already, so how about env__sleep_accurate, default it to True and let the board files set it to false, and have test_sleep check for and use that?

Quoted 109 lines · Tom Rini

Quoted 108 lines · Michal Simek

Quoted 79 lines · Stephen Warren

Quoted 66 lines · Tom Rini

Quoted 45 lines · Michal Simek

Quoted 44 lines · Tom Rini

Quoted 43 lines · Michal Simek

Quoted 41 lines · Tom Rini

Quoted 21 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 8 lines

Quoted 4 lines · Michal Simek

Quoted 1 line

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our
qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still
slightly too strict about which delays it accepts, or is sleep outright
broken?

Quoted 9 lines

You can use command-line option -k to avoid some tests. For example "-k not
sleep". That way, we don't have to hard-code the dependency into the test
source. Depending on the root cause (issue in U-Boot, or issue in just your
local version of qemu, or something that will never work) this might be
better?

Even with the most recent relaxing of the sleep test requirements, I can
still (depending on overall system load) have 'sleep' take too long, on
QEMU.  I think it might have been half a second slow, but I don't have
the log handy anymore.  Both locally and in travis we -k not sleep all
of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test.  OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

Quoted 6 lines

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).

But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration.
Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.

Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True

Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True
to skip dhcp test for whatever reason.

Then for travis-ci we can just put these lines to py/travis-ci/.

Quoted 5 lines

Quoted 1 line

What do you think?

Ah, good idea! We have a few cases like this already, so how about env__sleep_accurate, default it to True and let the board files set it to false, and have test_sleep check for and use that?

ok. this is about that variable for sleep not about generic skipping testcases which are failing.

Quoted 116 lines · Michal Simek

Quoted 114 lines · Tom Rini

Quoted 101 lines · Michal Simek

Quoted 79 lines · Stephen Warren

Quoted 66 lines · Tom Rini

Quoted 45 lines · Michal Simek

Quoted 44 lines · Tom Rini

Quoted 43 lines · Michal Simek

Quoted 41 lines · Tom Rini

Quoted 21 lines · Stephen Warren

Quoted 5 lines · Michal Simek

Hi,

Quoted 2 lines · Heinrich Schuchardt

Quoted 8 lines

Quoted 4 lines · Michal Simek

Qemu for arm32/arm64 has a problem with time setup.

Wouldn't it be preferable to fix the root cause?

Definitely that would be the best and IIRC I have tried to convince our
qemu guy to do that but they have never done that.

What is the exact failure condition? Is it simply that the test is still
slightly too strict about which delays it accepts, or is sleep outright
broken?

Quoted 9 lines

You can use command-line option -k to avoid some tests. For example "-k not
sleep". That way, we don't have to hard-code the dependency into the test
source. Depending on the root cause (issue in U-Boot, or issue in just your
local version of qemu, or something that will never work) this might be
better?

Even with the most recent relaxing of the sleep test requirements, I can
still (depending on overall system load) have 'sleep' take too long, on
QEMU.  I think it might have been half a second slow, but I don't have
the log handy anymore.  Both locally and in travis we -k not sleep all
of the qemu instances.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ok. By locally do you mean just using -k not sleep?

Yes, I have that in my CI scripts and similar.

Wouldn't be easier to keep this in uboot-test-hooks repo with other target setting?

Or do as you did did and mark the tests as not allowed for qemu, yes.

Quoted 10 lines

What we are trying to do is that our testing group will run these tests for me that's why it is just easier for me to change local uboot-test-hooks repo instead of communicate with them what -k not XXX parameters to add to certain scripts.

It means in loop they will just run all tests on qemu, local targets and in boardfarm. It is probably not big deal to tell them to add -k not sleep for all qemu runs but I know that for some i2c testing qemu doesn't emulate these devices that's why these tests fails. And the amount of tests which we shouldn't run on qemu will probably grow.

Quoted 4 lines

Well, I'm still open to possibly tweaking the allowed variance in the sleep test.  OTOH, if we just say "no QEMU" here, we can then go back to "sleep should be pretty darn accurate on HW" for the test too, and perhaps that's best.

The fundamental problem of "over-sleeping" due to host system load/.. exists with all boards. There's nothing specific to qemu here except that running U-Boot on qemu on the host rather than on separate HW might more easily trigger the "high load on the host" condition; I see the issue now and then and manually retry that test, although that is a bit annoying.

Quoted 6 lines

The original test was mostly intended to make sure that e U-Boot clock didn't run at a significantly different rate to the host, since I had seen that issue during development of some board support or as a regression sometime. Perhaps the definition of "significantly different" should be more like "1/2 rate or twice rate or more" rather than "off by a small fraction of a second". That might avoid so many false positives.

We had this issue with silicon v1 and having accurate sleep measuring is definitely good thing to have (Probably make sense to enable margin setup via variable anyway).

But still I would extend this to more wider discussion how to disable just one particular test case which is verified that it is broken on certain target/target configuration.
Using -k not XXX option is possible but as I said before it is not ideal to keep track of these problematic tests in two locations and share this between two teams.

Better would be to add to u_boot_boardenv...py file line like this skip_test_sleep = True

Quoted 8 lines

Which would be parsed and test won't run for specific board/configuration. The same logic can be generic that user can add for example skip_test_net_dhcp = True
to skip dhcp test for whatever reason.

Then for travis-ci we can just put these lines to py/travis-ci/.

What do you think?

Ah, good idea! We have a few cases like this already, so how about env__sleep_accurate, default it to True and let the board files set it to false, and have test_sleep check for and use that?

Quoted 2 lines

ok. this is about that variable for sleep not about generic skipping testcases which are failing.

Right. I'm not sure we need to get too complicated on "skip these tests" logic. But I think we have enough technical information now to say it's reasonable to skip sleep in some cases.