Backend troubles - temporarily refusing to accept test results
A message that is not in the archive (sent off-list or not yet received): [email protected]
This has now been done, and the backend load average is much improved. If anyone continues to see an unresponsive part of the frontend
(https://kernelci.org) please let me know.
Quoted 23 lines · Matt Hart
Hi all,
The kernelci API has been struggling since last week, to the point that the celery service grinds to a halt and stops processing some requests. This causes the website to hang, the API to start timing out, and general failures in the service.
For reasons we (Guillaume and I) have not yet been able to fully understand, the importing of test results takes much longer than other tasks and causes the highest load. We are now running more test jobs than before and the data structures will likely need tuning in the future.
As the service failures are becoming more and more frequent, in an attempt to keep the core service up and running, we have decided to make a quick patch to production to temporarily stop processing test results, until such time as the problem is better understood.
Matt
A message that is not in the archive (sent off-list or not yet received): [email protected]
We've also had to disable the test views on the website, for the same reason, so all views of https://kernelci.org/test will get a 404.
Quoted 32 lines · Matt Hart
This has now been done, and the backend load average is much improved. If anyone continues to see an unresponsive part of the frontend
(https://kernelci.org) please let me know.Quoted 23 lines · Matt Hart
Hi all,
The kernelci API has been struggling since last week, to the point that the celery service grinds to a halt and stops processing some requests. This causes the website to hang, the API to start timing out, and general failures in the service.
For reasons we (Guillaume and I) have not yet been able to fully understand, the importing of test results takes much longer than other tasks and causes the highest load. We are now running more test jobs than before and the data structures will likely need tuning in the future.
As the service failures are becoming more and more frequent, in an attempt to keep the core service up and running, we have decided to make a quick patch to production to temporarily stop processing test results, until such time as the problem is better understood.
Matt
Quoted 3 lines · Matt Hart
We've also had to disable the test views on the website, for the same reason, so all views of https://kernelci.org/test will get a 404.
This was all re-enabled last week after addressing some performance problems in kernelci-backend.
Guillaume
Quoted 28 lines · Matt Hart
Quoted 26 lines
This has now been done, and the backend load average is much improved. If anyone continues to see an unresponsive part of the frontend
(https://kernelci.org) please let me know.Quoted 20 lines · Matt Hart
Hi all,
The kernelci API has been struggling since last week, to the point that the celery service grinds to a halt and stops processing some requests. This causes the website to hang, the API to start timing out, and general failures in the service.
For reasons we (Guillaume and I) have not yet been able to fully understand, the importing of test results takes much longer than other tasks and causes the highest load. We are now running more test jobs than before and the data structures will likely need tuning in the future.
As the service failures are becoming more and more frequent, in an attempt to keep the core service up and running, we have decided to make a quick patch to production to temporarily stop processing test results, until such time as the problem is better understood.
Matt