Should we handle TPM_RC_RETRY internally?
James Bottomley
maintainerI was investigating an apparent bug in the trusted keys implementation where periodically the key operation barfs and returns an error to userspace. It turns out this error is because the TPM returns TPM_RC_RETRY to an operation.
The TPM spec is a bit unclear why the TPM would return TPM_RC_RETRY, but it is clear that it may happen on a lot of operations. I checked with the microsoft reference implementation:
https://github.com/Microsoft/ms-tpm-20-ref/
Which implies it's only set if the lockout check is invoked by the command and the previous TPM shutdown wasn't orderly. It does seem to me that I've only seen it involving objects with DA implications, which explains why it's seen in trusted keys.
If I read the UEFI TPM API, it does automatic retries. This is the note:
The firmware SHALL not return TPM2_RC_RETRY prior to the completion of the call to ExitBootServices().
Implementer's Note: the implementation of this function should check the return value in the TPM response and, if it is TPM2_RC_RETRY, resend the command. The implementation may abort if a sufficient number of retries has been done.
I really think if UEFI does it, we should do it too (and it will fix my trusted key bug).
What does everyone else think? If it's agreed, I'll code up the patch.
James
[adding Philip Tricca to the cc list]
Hello James,
Quoted 27 lines · James Bottomley
I was investigating an apparent bug in the trusted keys implementation where periodically the key operation barfs and returns an error to userspace. It turns out this error is because the TPM returns
TPM_RC_RETRYto an operation.The TPM spec is a bit unclear why the TPM would return
TPM_RC_RETRY, but it is clear that it may happen on a lot of operations. I checked with the microsoft reference implementation:https://github.com/Microsoft/ms-tpm-20-ref/
Which implies it's only set if the lockout check is invoked by the command and the previous TPM shutdown wasn't orderly. It does seem to me that I've only seen it involving objects with DA implications, which explains why it's seen in trusted keys.
If I read the UEFI TPM API, it does automatic retries. This is the note:
The firmware SHALL not return
TPM2_RC_RETRYprior to the completion of the call toExitBootServices().Implementer's Note: the implementation of this function should check the return value in the TPM response and, if it is
TPM2_RC_RETRY, resend the command. The implementation may abort if a sufficient number of retries has been done.
Quoted 5 lines
I really think if UEFI does it, we should do it too (and it will fix my trusted key bug).
What does everyone else think? If it's agreed, I'll code up the patch.
That's a very good question. I don't know the answer but maybe Philip does since he said that would ask the TCG TSS WG members for their thoughts on this [0].
We had the same issue in the tpm2-software project, and at the end Philip added a macro that retries sending commands on TPM_RC_RETRY responses [1].
If the kernel handles this, then such a macro won't be needed anymore for user-space programs and will simplify things. So I'm in favour to handle at the driver level but I'm not sure if is the correct thing to do or not.
Quoted 2 lines
James
[0]: https://github.com/tpm2-software/tpm2-tools/issues/469#issuecomment-340501128
[1]: https://github.com/tpm2-software/tpm2-tools/pull/585/commits/9b0f3e59bab0810cf3721bf6765bd083f23f811f
Best regards,
Signature · 3 lines
Javier Martinez Canillas Software Engineer - Desktop Hardware Enablement Red Hat
Quoted 42 lines · Javier Martinez Canillas
[adding Philip Tricca to the cc list]
Hello James,
Quoted 27 lines · James Bottomley
I was investigating an apparent bug in the trusted keys implementation where periodically the key operation barfs and returns an error to userspace. It turns out this error is because the TPM returns
TPM_RC_RETRYto an operation.The TPM spec is a bit unclear why the TPM would return
TPM_RC_RETRY, but it is clear that it may happen on a lot of operations. I checked with the microsoft reference implementation:https://github.com/Microsoft/ms-tpm-20-ref/
Which implies it's only set if the lockout check is invoked by the command and the previous TPM shutdown wasn't orderly. It does seem to me that I've only seen it involving objects with DA implications, which explains why it's seen in trusted keys.
If I read the UEFI TPM API, it does automatic retries. This is the note:
The firmware SHALL not return
TPM2_RC_RETRYprior to the completion of the call toExitBootServices().Implementer's Note: the implementation of this function should check the return value in the TPM response and, if it is
TPM2_RC_RETRY, resend the command. The implementation may abort if a sufficient number of retries has been done.Quoted 5 lines
I really think if UEFI does it, we should do it too (and it will fix my trusted key bug).
What does everyone else think? If it's agreed, I'll code up the patch.
That's a very good question. I don't know the answer but maybe Philip does since he said that would ask the TCG TSS WG members for their thoughts on this [0].
Quoted 6 lines
We had the same issue in the tpm2-software project, and at the end Philip added a macro that retries sending commands on
TPM_RC_RETRYresponses [1].If the kernel handles this, then such a macro won't be needed anymore for user-space programs and will simplify things. So I'm in favour to handle at the driver level but I'm not sure if is the correct thing to do or not.
Can we get input from TCG how to fix this properly and request to update their specs accordingly?
/Jarkko
Quoted 38 lines · Javier Martinez Canillas
[adding Philip Tricca to the cc list]
Hello James,
Quoted 27 lines · James Bottomley
I was investigating an apparent bug in the trusted keys implementation where periodically the key operation barfs and returns an error to userspace. It turns out this error is because the TPM returns
TPM_RC_RETRYto an operation.The TPM spec is a bit unclear why the TPM would return
TPM_RC_RETRY, but it is clear that it may happen on a lot of operations. I checked with the microsoft reference implementation:https://github.com/Microsoft/ms-tpm-20-ref/
Which implies it's only set if the lockout check is invoked by the command and the previous TPM shutdown wasn't orderly. It does seem to me that I've only seen it involving objects with DA implications, which explains why it's seen in trusted keys.
If I read the UEFI TPM API, it does automatic retries. This is the note:
The firmware SHALL not return
TPM2_RC_RETRYprior to the completion of the call toExitBootServices().Implementer's Note: the implementation of this function should check the return value in the TPM response and, if it is
TPM2_RC_RETRY, resend the command. The implementation may abort if a sufficient number of retries has been done.Quoted 5 lines
I really think if UEFI does it, we should do it too (and it will fix my trusted key bug).
What does everyone else think? If it's agreed, I'll code up the patch.
Quoted 6 lines
That's a very good question. I don't know the answer but maybe Philip does since he said that would ask the TCG TSS WG members for their thoughts on this [0].
We had the same issue in the tpm2-software project, and at the end Philip added a macro that retries sending commands on
TPM_RC_RETRYresponses [1].
We've been using this macro without indicent for some time in applications using the tss2-sys library. If memory serves I think the new tss2-esys library handles this internally so from the perspective of the user space I'd written this issue off as solved.
Quoted 3 lines
If the kernel handles this, then such a macro won't be needed anymore for user-space programs and will simplify things. So I'm in favour to handle at the driver level but I'm not sure if is the correct thing to do or not.
I can't say for sure but I'd imagine the implications of implementing a retry loop in the kernel are different than user space. We retry indefinitely but I would think having some bounded number of retries in the kernel would be prudent.
Philip
Quoted 4 lines · Philip Tricca
I can't say for sure but I'd imagine the implications of implementing a retry loop in the kernel are different than user space. We retry indefinitely but I would think having some bounded number of retries in the kernel would be prudent.
Yes, even if we would support handlig RC_RETRY we could not make promise to the user space that this error code will never fall through.
/Jarkko
Quoted 27 lines · James Bottomley
I was investigating an apparent bug in the trusted keys implementation where periodically the key operation barfs and returns an error to userspace. It turns out this error is because the TPM returns
TPM_RC_RETRYto an operation.The TPM spec is a bit unclear why the TPM would return
TPM_RC_RETRY, but it is clear that it may happen on a lot of operations. I checked with the microsoft reference implementation:https://github.com/Microsoft/ms-tpm-20-ref/
Which implies it's only set if the lockout check is invoked by the command and the previous TPM shutdown wasn't orderly. It does seem to me that I've only seen it involving objects with DA implications, which explains why it's seen in trusted keys.
If I read the UEFI TPM API, it does automatic retries. This is the note:
The firmware SHALL not return
TPM2_RC_RETRYprior to the completion of the call toExitBootServices().Implementer's Note: the implementation of this function should check the return value in the TPM response and, if it is
TPM2_RC_RETRY, resend the command. The implementation may abort if a sufficient number of retries has been done.
Quoted 6 lines
I really think if UEFI does it, we should do it too (and it will fix my trusted key bug).
What does everyone else think? If it's agreed, I'll code up the patch.
James
I think I agree but what worries me is that this error code is almost not documented at all in TCG specifications.
/Jarkko
James Bottomley
maintainerQuoted 36 lines · Jarkko Sakkinen
Quoted 35 lines · James Bottomley
I was investigating an apparent bug in the trusted keys implementation where periodically the key operation barfs and returns an error to userspace. It turns out this error is because the TPM returns
TPM_RC_RETRYto an operation.The TPM spec is a bit unclear why the TPM would return
TPM_RC_RETRY, but it is clear that it may happen on a lot of operations. I checked with the microsoft reference implementation:https://github.com/Microsoft/ms-tpm-20-ref/
Which implies it's only set if the lockout check is invoked by the command and the previous TPM shutdown wasn't orderly. It does seem to me that I've only seen it involving objects with DA implications, which explains why it's seen in trusted keys.
If I read the UEFI TPM API, it does automatic retries. This is the note:
The firmware SHALL not return
TPM2_RC_RETRYprior to the
completion
of the call toExitBootServices().Implementer's Note: the implementation of this function should
check
the return value in the TPM response and, if it isTPM2_RC_RETRY,
resend the command. The implementation may abort if a
sufficient
number of retries has been done.I really think if UEFI does it, we should do it too (and it will fix my trusted key bug).
Quoted 7 lines
Quoted 4 lines
What does everyone else think? If it's agreed, I'll code up the patch.
James
I think I agree but what worries me is that this error code is almost not documented at all in TCG specifications.
Yes, it's cryptic, it just says "the TPM was not able to start the command", which is why I checked the reference implementation. I think the UEFI ref is the clearest because it provides implementation guidance (basically to retry the command until timeout) which I see no reason we can't follow in the kernel.
The alternative is that we handle it in the TSS and I have to patch trusted keys. However none of the core TPM routines that use tpm_transmit_cmd() have any retry logic, so if we find a TPM that returns it to a kernel sent command we'll get random unexplained failures (based on the ref implementation I don't think this is likely, but by the spec it's possible, especially for a manufacturer who isn't using the reference), so we may end up having to add retry logic to every place in the kernel where we use tpm_transmit_cmd() ... then we'll wish we'd done it generically inside that routine.
Based on this, I think the UEFI spec gives us the safety of knowing that internal handling is correct and the advantages of doing so now are that we don't get a potentially exploding problem later, so it seems like a net win. I'll code up a patch.
James
Quoted 42 lines · James Bottomley
Quoted 41 lines · Jarkko Sakkinen
Quoted 39 lines · James Bottomley
I was investigating an apparent bug in the trusted keys implementation where periodically the key operation barfs and returns an error to userspace. It turns out this error is because the TPM returns
TPM_RC_RETRYto an operation.The TPM spec is a bit unclear why the TPM would return
TPM_RC_RETRY, but it is clear that it may happen on a lot of operations. I checked with the microsoft reference implementation:https://github.com/Microsoft/ms-tpm-20-ref/
Which implies it's only set if the lockout check is invoked by the command and the previous TPM shutdown wasn't orderly. It does seem to me that I've only seen it involving objects with DA implications, which explains why it's seen in trusted keys.
If I read the UEFI TPM API, it does automatic retries. This is the note:
The firmware SHALL not return
TPM2_RC_RETRYprior to the
completion
of the call toExitBootServices().Implementer's Note: the implementation of this function should
check
the return value in the TPM response and, if it isTPM2_RC_RETRY,
resend the command. The implementation may abort if a
sufficient
number of retries has been done.I really think if UEFI does it, we should do it too (and it will fix my trusted key bug).
What does everyone else think? If it's agreed, I'll code up the patch.
James
Quoted 8 lines
Quoted 2 lines
I think I agree but what worries me is that this error code is almost not documented at all in TCG specifications.
Yes, it's cryptic, it just says "the TPM was not able to start the command", which is why I checked the reference implementation. I think the UEFI ref is the clearest because it provides implementation guidance (basically to retry the command until timeout) which I see no reason we can't follow in the kernel.
I agree with your reasoning.
/Jarkko
A message that is not in the archive (sent off-list or not yet received): CAP7wa8Kq_++HasQY6bZ9idJ_TDOyXhnRAmyUY6t1B+HZWKF8ig@mail.gmail.com
Quoted 11 lines · Andrey Pronin
If the TSS starts retrying commands itself, then it's more than just handling
TPM_RC_RETRY.
There is also even more crypticTPM_RC_YIELDED, which allows (suggests?) retrying, andTPM_RC_NV_RATE, which covers one of the few legitimate and easily explainable reasons why a command can fail now but succeed later. And what aboutTPM_RC_MEMORY,TPM_RC_NV_UNAVAILABLEand such? Though quite exotic, they are all warnings, and all indicate a possibly temporary lack of access/resources. Unlike TPM_RC_*_HANDLES,CONTEXT_GAPetc, they don't require RM to do anything with objects and handles to recover, and may succeed if simply retried after a delay. Shall these retries be also handled intpm_transmit_cmd()or left up to
Quoted 8 lines
the caller to take care of? And how are they different from
TPM_RC_RETRYcase, if the latter?
If you are interested in the actual in-kernel use of the commands that can face such codes, look no further thantpm2_load_context(). The referenceA implementation may return TPM_RC_NV* fromATPM2_ContextLoadsince it callsANvIsAvailable().
--
Andrey
Hey, thanks for this feedback. It changed my point of view.
tpm_transmit_cmd() should be able to handle exactly the subset that in-kernel clients of the TPM need, not more or less.
I do no want TPM_RC_RETRY logic at this point to tpm_transmit() path. We can consider that later (maybe). It is at least too late to consider it in 4.17 timeline.
/Jarkko
James Bottomley
maintainerYour original email is missing from the list because it has a html part, which vger takes as spam. You have to send text/plain only to get email on the list
Quoted 75 lines · Andrey Pronin
Quoted 73 lines · James Bottomley
Quoted 40 lines · Jarkko Sakkinen
Quoted 38 lines · James Bottomley
I was investigating an apparent bug in the trusted keys implementation where periodically the key operation barfs and returns an error to userspace. It turns out this error is because the TPM returns
TPM_RC_RETRYto an operation.The TPM spec is a bit unclear why the TPM would return
TPM_RC_RETRY, but it is clear that it may happen on a lot of operations. I checked with the microsoft reference implementation:https://github.com/Microsoft/ms-tpm-20-ref/
Which implies it's only set if the lockout check is invoked by the command and the previous TPM shutdown wasn't orderly. It does seem to me that I've only seen it involving objects with DA implications, which explains why it's seen in trusted keys.
If I read the UEFI TPM API, it does automatic retries. This is the note:
The firmware SHALL not return
TPM2_RC_RETRYprior to the
completion
of the call toExitBootServices().Implementer's Note: the implementation of this function
should
check
the return value in the TPM response and, if it isTPM2_RC_RETRY,
resend the command. The implementation may abort if a
sufficient
number of retries has been done.I really think if UEFI does it, we should do it too (and it will fix my trusted key bug).
Quoted 7 lines
Quoted 4 lines
What does everyone else think? If it's agreed, I'll code up the patch.
James
I think I agree but what worries me is that this error code is almost not documented at all in TCG specifications.
Yes, it's cryptic, it just says "the TPM was not able to start the command", which is why I checked the reference implementation. I think the UEFI ref is the clearest because it provides implementation guidance (basically to retry the command until timeout) which I see no reason we can't follow in the kernel.
The alternative is that we handle it in the TSS and I have to patch trusted keys. However none of the core TPM routines that use
tpm_transmit_cmd()have any retry logic, so if we find a TPM that returns it to a kernel sent command we'll get random unexplained failures (based on the ref implementation I don't think this is likely, but by the spec it's possible, especially for a manufacturer who isn't using the reference), so we may end up having to add retry logic to every place in the kernel where we usetpm_transmit_cmd()... then we'll wish we'd done it generically inside that routine.Based on this, I think the UEFI spec gives us the safety of knowing that internal handling is correct and the advantages of doing so now are that we don't get a potentially exploding problem later, so it seems like a net win. I'll code up a patch.
Quoted 6 lines
Quoted 2 lines
James
If the TSS starts retrying commands itself, then it's more than just handling
TPM_RC_RETRY. There is also even more crypticTPM_RC_YIELDED, which allows (suggests?) retrying,
This is only for software based TPMs, so I suppose intel PTT could do it, but, as the note says, it's not issued by the reference implementation at all. I'd say ignore it until we find an actual implementor.
Quoted 3 lines
and
TPM_RC_NV_RATE, which covers one of the few legitimate and
easily explainable reasons why a command can fail now but succeed later.
NV_RATE is only for commands that touch NV memory, so that updates can be rate limited. The docs imply it's only for NV commands, which we don't issue in the kernel.
Quoted 1 line
And what about
TPM_RC_MEMORY,
It means there are too many objects in the TPM and is not fixable by a retry only by unloading something.
Quoted 1 line
TPM_RC_NV_UNAVAILABLEand such?
That means the platform component reports the NV memory isn't present.
This is only for TPMs with external NV chips, I believe and it means
there's a NV connection or provisioning problem.
Quoted 5 lines
Though quite exotic, they are all warnings, and all indicate a possibly temporary lack of access/resources. Unlike TPM_RC_*_HANDLES,
CONTEXT_GAPetc, they don't require RM to do anything with objects and handles to recover, and may succeed if simply retried after a delay.
No ... the only one you've cited that may be retry amenable is RC_YIELDED ... the rest indicate other problems a retry won't fix and as such should be exposed to the user.
Quoted 8 lines
Shall these retries be also handled in
tpm_transmit_cmd()or left up to the caller to take care of? And how are they different fromTPM_RC_RETRYcase, if the latter?If you are interested in the actual in-kernel use of the commands that can face such codes, look no further than
tpm2_load_context(). The reference implementation may return TPM_RC_NV* fromTPM2_ContextLoadsince it callsNvIsAvailable().
You may be looking at a different emulator. In the Microsoft one it's RETURN_IF_NV_IS_NOT_AVAILABLE and it can't actually be returned by context loading or saving.
In general I'd say we do the pragmatic not the theoretical thing, so handle retry returns that have actually been seen and which have caused problems. That set is currently only RC_RETRY.
James