Skip to contentSkip to conversation

Backend troubles - temporarily refusing to accept test results

kernelci Matt Hart 2 participants 3 messages Nov 27, 2018 – Dec 14, 2018 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Matt Hart

author
#1 lore raw

This has now been done, and the backend load average is much improved. If anyone continues to see an unresponsive part of the frontend
(https://kernelci.org) please let me know.

Quoted 23 lines · Matt Hart

Hi all,

The kernelci API has been struggling since last week, to the point that the celery service grinds to a halt and stops processing some requests. This causes the website to hang, the API to start timing out, and general failures in the service.

For reasons we (Guillaume and I) have not yet been able to fully understand, the importing of test results takes much longer than other tasks and causes the highest load. We are now running more test jobs than before and the data structures will likely need tuning in the future.

As the service failures are becoming more and more frequent, in an attempt to keep the core service up and running, we have decided to make a quick patch to production to temporarily stop processing test results, until such time as the problem is better understood.

Matt

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#3 lore raw

We've also had to disable the test views on the website, for the same reason, so all views of https://kernelci.org/test will get a 404.

Quoted 32 lines · Matt Hart

This has now been done, and the backend load average is much improved. If anyone continues to see an unresponsive part of the frontend
(https://kernelci.org) please let me know.

Quoted 23 lines · Matt Hart

Hi all,

The kernelci API has been struggling since last week, to the point that the celery service grinds to a halt and stops processing some requests. This causes the website to hang, the API to start timing out, and general failures in the service.

For reasons we (Guillaume and I) have not yet been able to fully understand, the importing of test results takes much longer than other tasks and causes the highest load. We are now running more test jobs than before and the data structures will likely need tuning in the future.

As the service failures are becoming more and more frequent, in an attempt to keep the core service up and running, we have decided to make a quick patch to production to temporarily stop processing test results, until such time as the problem is better understood.

Matt

#4 lore raw
Quoted 3 lines · Matt Hart

We've also had to disable the test views on the website, for the same reason, so all views of https://kernelci.org/test will get a 404.

This was all re-enabled last week after addressing some performance problems in kernelci-backend.

Guillaume

Quoted 28 lines · Matt Hart
Quoted 26 lines

This has now been done, and the backend load average is much improved. If anyone continues to see an unresponsive part of the frontend
(https://kernelci.org) please let me know.

Quoted 20 lines · Matt Hart

Hi all,

The kernelci API has been struggling since last week, to the point that the celery service grinds to a halt and stops processing some requests. This causes the website to hang, the API to start timing out, and general failures in the service.

For reasons we (Guillaume and I) have not yet been able to fully understand, the importing of test results takes much longer than other tasks and causes the highest load. We are now running more test jobs than before and the data structures will likely need tuning in the future.

As the service failures are becoming more and more frequent, in an attempt to keep the core service up and running, we have decided to make a quick patch to production to temporarily stop processing test results, until such time as the problem is better understood.

Matt