evtchn/Flask: pre-allocate node on send path
PATCH 1/1 evtchn/Flask: pre-allocate node on send path
xmalloc() & Co may not be called with IRQs off, or else check_lock() will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking in evtchn_send() doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call to xsm_evtchn_send() instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.
As these nodes are used merely for caching earlier decisions' results, allocate just one node in AVC code despite two potentially being needed. Things will merely be not as performant if a second allocation was wanted, just like when the pre-allocation fails.
---
TBD: An even easier fix could be to simply guard xzalloc() by a
conditional checking local_irq_is_enabled(), but for a domain sending only interdomain events this would mean AVC's node caching would never take effect on the sending path, as allocation would then always be avoided.
xen/common/event_channel.c +6 -0
@@ -723,6 +723,12 @@ int evtchn_send(struct domain *ld, unsigif ( !port_is_valid (ld, lport) )return -EINVAL ; +/* +* As the call further down needs to avoid allocations (due to running +* with IRQs off), give XSM a chance to pre-allocate if needed. +*/ +xsm_evtchn_send (XSM_HOOK , ld,NULL ); + lchn =evtchn_from_port (ld, lport);spin_lock_irqsave (&lchn->lock, flags);
xen/xsm/flask/avc.c +72 -8
96 lines not on this page · raw message
xen/xsm/flask/hooks.c +10 -0
@@ -281,6 +281,16 @@ static int flask_evtchn_send(struct doma {int rc; +/* +* When called with non-NULL chn, memory allocation may not be permitted. +* Allow AVC to preallocate nodes as necessary upon early notification. +*/ +if ( !chn ) + { +avc_prealloc (); +return 0 ; + } +switch ( chn->state ) {case ECS_INTERDOMAIN :
xen/xsm/flask/include/avc.h +2 -0
@@ -91,6 +91,8 @@ int avc_has_perm_noaudit(u32 ssid, u32 tint avc_has_perm (u32 ssid,u32 tsid,u16 tclass,u32 requested,struct avc_audit_data *auditdata); +void avc_prealloc (void ); +/* Exported to selinuxfs */ struct xen_flask_hash_stats ;int avc_get_hash_stats (struct xen_flask_hash_stats *arg);
Hi Jan,
Quoted 7 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.
This is the sort of fall-out I was expecting when we decide to turn off the interrupts for big chunk of code. I couldn't find any at the time though...
Can you remind which caller of send_guest{global, vcpu}_virq() will call them with interrupts off?
Would it be possible to consider deferring the call to a softirq taslket? If so, this would allow us to turn the interrupts again.
Cheers,
Signature · 1 line
Julien Grall
Quoted 9 lines · Julien Grall
Quoted 7 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.
Quoted 6 lines
This is the sort of fall-out I was expecting when we decide to turn off the interrupts for big chunk of code. I couldn't find any at the time though...
Can you remind which caller of send_guest{global, vcpu}_virq() will call them with interrupts off?
I don't recall which one of the two it was that I hit; we wanted both to use the lock anyway. send_guest_pirq() very clearly also gets called with IRQs off.
Quoted 2 lines
Would it be possible to consider deferring the call to a softirq taslket? If so, this would allow us to turn the interrupts again.
Of course this is in principle possible; the question is how involved this is going to get. However, on x86 oprofile's call to send_guest_vcpu_virq() can't easily be replaced - it's dangerous enough already that in involves locks in NMI context. I don't fancy seeing it use more commonly used ones.
Jan
Hi Jan,
Quoted 21 lines · Jan Beulich
Quoted 9 lines · Julien Grall
Quoted 7 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.Quoted 6 lines
This is the sort of fall-out I was expecting when we decide to turn off the interrupts for big chunk of code. I couldn't find any at the time though...
Can you remind which caller of send_guest{global, vcpu}_virq() will call them with interrupts off?
I don't recall which one of the two it was that I hit; we wanted both to use the lock anyway.
send_guest_pirq()very clearly also gets called with IRQs off.
Quoted 8 lines
Quoted 2 lines
Would it be possible to consider deferring the call to a softirq taslket? If so, this would allow us to turn the interrupts again.
Of course this is in principle possible; the question is how involved this is going to get. However, on x86 oprofile's call to
send_guest_vcpu_virq()can't easily be replaced - it's dangerous enough already that in involves locks in NMI context. I don't fancy seeing it use more commonly used ones.
Fair enough. I would still like to consider a way where we could avoid to hack xsm_* because we have the interrupts disabled.
AFAICT, we don't need send_guest_global_virq() and evtchn_send() to be mutually exclusive. Is that correct?
So how about splitting the lock in two? One would be used when the interrupts have to be disabled while the other would be used when we can keep interrupts enabled.
The two locks would have to be taken when the event channel needs to be modified.
Cheers,
Signature · 1 line
Julien Grall
Quoted 34 lines · Julien Grall
Hi Jan,
Quoted 24 lines · Jan Beulich
Quoted 9 lines · Julien Grall
Quoted 7 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.Quoted 6 lines
This is the sort of fall-out I was expecting when we decide to turn off the interrupts for big chunk of code. I couldn't find any at the time though...
Can you remind which caller of send_guest{global, vcpu}_virq() will call them with interrupts off?
I don't recall which one of the two it was that I hit; we wanted both to use the lock anyway.
send_guest_pirq()very clearly also gets called with IRQs off.Quoted 2 lines
Would it be possible to consider deferring the call to a softirq taslket? If so, this would allow us to turn the interrupts again.
Quoted 6 lines
Of course this is in principle possible; the question is how involved this is going to get.
However, on x86 oprofile's call tosend_guest_vcpu_virq()can't easily be replaced - it's dangerous enough already that in involves locks in NMI context. I don't fancy seeing it use more commonly used ones.
Quoted 2 lines
Fair enough. I would still like to consider a way where we could avoid to hack xsm_* because we have the interrupts disabled.
Well, from a conceptual pov it's at least questionable for XSM to need any memory allocations at all when merely being asked for permission. And indeed the need for it arises solely from its desire to cache the result, for the sake of subsequent lookups.
I also find it odd that there's an XSM check on the send path in the first place. This isn't just because it would seem to me that it should be decided at binding time whether sending is permitted
- I may easily be missing something in the conceptual model here. It's also because __domain_finalise_shutdown() too uses evtchn_send(), and I didn't think this one should be subject to any XSM check (just like send_guest_*() aren't).
Quoted 2 lines
AFAICT, we don't need
send_guest_global_virq()andevtchn_send()to be mutually exclusive. Is that correct?
Yes, any number of sends (even to the same port) could in principle run concurrently, I think. Or else the FIFO code would have been broken from the very point where the per-channel lock was introduced and acquiring of the per-domain one then dropped from evtchn_send() (other sending paths weren't using the per-domain one anyway already before that).
Quoted 3 lines
So how about splitting the lock in two? One would be used when the interrupts have to be disabled while the other would be used when we can keep interrupts enabled.
Now that's an interesting proposal. I thought one lock per channel was already pretty fine-grained. Now you propose making it two.
Quoted 2 lines
The two locks would have to be taken when the event channel needs to be modified.
Requiring a total of 6 locks to be acquired when fiddling with interdomain channels... Wow. Definitely more intrusive overall than the change here.
Jan
Hi Jan,
Quoted 43 lines · Jan Beulich
Quoted 34 lines · Julien Grall
Hi Jan,
Quoted 24 lines · Jan Beulich
Quoted 9 lines · Julien Grall
Quoted 7 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.Quoted 6 lines
This is the sort of fall-out I was expecting when we decide to turn off the interrupts for big chunk of code. I couldn't find any at the time though...
Can you remind which caller of send_guest{global, vcpu}_virq() will call them with interrupts off?
I don't recall which one of the two it was that I hit; we wanted both to use the lock anyway.
send_guest_pirq()very clearly also gets called with IRQs off.Quoted 2 lines
Would it be possible to consider deferring the call to a softirq taslket? If so, this would allow us to turn the interrupts again.
Quoted 6 lines
Of course this is in principle possible; the question is how involved this is going to get.
However, on x86 oprofile's call tosend_guest_vcpu_virq()can't easily be replaced - it's dangerous enough already that in involves locks in NMI context. I don't fancy seeing it use more commonly used ones.Quoted 2 lines
Fair enough. I would still like to consider a way where we could avoid to hack xsm_* because we have the interrupts disabled.
Well, from a conceptual pov it's at least questionable for XSM to need any memory allocations at all when merely being asked for permission. And indeed the need for it arises solely from its desire to cache the result, for the sake of subsequent lookups.
Quoted 7 lines
I also find it odd that there's an XSM check on the send path in the first place. This isn't just because it would seem to me that it should be decided at binding time whether sending is permitted
- I may easily be missing something in the conceptual model here. It's also because__domain_finalise_shutdown()too usesevtchn_send(), and I didn't think this one should be subject to any XSM check (just like send_guest_*() aren't).
Maybe this is the first question we need to answer?
Quoted 18 lines
Quoted 3 lines
AFAICT, we don't need
send_guest_global_virq()andevtchn_send()to be mutually exclusive. Is that correct?Yes, any number of sends (even to the same port) could in principle run concurrently, I think. Or else the FIFO code would have been broken from the very point where the per-channel lock was introduced and acquiring of the per-domain one then dropped from
evtchn_send()(other sending paths weren't using the per-domain one anyway already before that).Quoted 3 lines
So how about splitting the lock in two? One would be used when the interrupts have to be disabled while the other would be used when we can keep interrupts enabled.
Now that's an interesting proposal. I thought one lock per channel was already pretty fine-grained. Now you propose making it two.
Quoted 6 lines
Quoted 2 lines
The two locks would have to be taken when the event channel needs to be modified.
Requiring a total of 6 locks to be acquired when fiddling with interdomain channels... Wow. Definitely more intrusive overall than the change here.
Well hacks are always more self-contained but long term they are a nightmare to maintain :).
6 locks is indeed a lot, but the discussion here shows that the current locking is probably not suitable for the current uses.
I don't have a better solution so far, but I will have a think about it. Meanwhile, it would be good to figure out why xsm_evtchn_send() is needed.
Cheers,
Signature · 1 line
Julien Grall
Quoted 18 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 2 lines · Julien Grall
Fair enough. I would still like to consider a way where we could avoid to hack xsm_* because we have the interrupts disabled.
Well, from a conceptual pov it's at least questionable for XSM to need any memory allocations at all when merely being asked for permission. And indeed the need for it arises solely from its desire to cache the result, for the sake of subsequent lookups.
Quoted 7 lines
I also find it odd that there's an XSM check on the send path in the first place. This isn't just because it would seem to me that it should be decided at binding time whether sending is permitted
- I may easily be missing something in the conceptual model here. It's also because__domain_finalise_shutdown()too usesevtchn_send(), and I didn't think this one should be subject to any XSM check (just like send_guest_*() aren't).
Quoted 1 line
Maybe this is the first question we need to answer?
Indeed that was the question I asked myself before putting together the patch. Yet I have no idea who could answer it, with Daniel having gone silent for quite long a time now. Hence I didn't even put up the question, but instead tried to find a halfway reasonable solution. After all it's not just the master branch which is stuck right now. And from a backporting perspective having the "fix" touch code which hasn't had much churn over the last years may even be beneficial. Plus, as said, the minimal change of making Flask avoid xmalloc() when IRQs are off is a change that ought to be made anyway imo, in order to favor proper functioning over performance.
Jan
Hi Jan,
Quoted 19 lines · Jan Beulich
Quoted 18 lines · Julien Grall
Quoted 16 lines · Jan Beulich
Quoted 2 lines · Julien Grall
Fair enough. I would still like to consider a way where we could avoid to hack xsm_* because we have the interrupts disabled.
Well, from a conceptual pov it's at least questionable for XSM to need any memory allocations at all when merely being asked for permission. And indeed the need for it arises solely from its desire to cache the result, for the sake of subsequent lookups.
I also find it odd that there's an XSM check on the send path in the first place. This isn't just because it would seem to me that it should be decided at binding time whether sending is permitted
- I may easily be missing something in the conceptual model here. It's also because__domain_finalise_shutdown()too usesevtchn_send(), and I didn't think this one should be subject to any XSM check (just like send_guest_*() aren't).
Quoted 7 lines
Quoted 1 line
Maybe this is the first question we need to answer?
Indeed that was the question I asked myself before putting together the patch. Yet I have no idea who could answer it, with Daniel having gone silent for quite long a time now. Hence I didn't even put up the question, but instead tried to find a halfway reasonable solution.
IIRC, XSM is used by OpenXT and QubeOS. So I have CCed them to get some input.
Quoted 6 lines
After all it's not just the master branch which is stuck right now.
And from a backporting perspective having the "fix" touch code which hasn't had much churn over the last years may even be beneficial.
Right, but dropping xsm_evtchn_send() (if it is not possible) is going to be even better. So lets explore this solution first.
Quoted 3 lines
Plus, as said, the minimal change of making Flask
avoidxmalloc()when IRQs are off is a change that ought to be made anyway imo, in order to favor proper functioning over performance.
If there are other use, then yes. What I don't like in the current approach is we are hijacking xsm_event_send() for pre-allocating memory.
Cheers,
Signature · 1 line
Julien Grall
Quoted 6 lines · Julien Grall
Quoted 3 lines · Jan Beulich
Plus, as said, the minimal change of making Flask
avoidxmalloc()when IRQs are off is a change that ought to be made anyway imo, in order to favor proper functioning over performance.If there are other use, then yes. What I don't like in the current approach is we are hijacking
xsm_event_send()for pre-allocating memory.
I first had a designated function for this, but the need to wire this through ops, dummy.h, etc made me pretty quickly turn to this lower churn variant. But that other one hasn't been ruled out if the "hijacking" is deemed too bad.
Jan
Quoted 25 lines · Julien Grall
Hi Jan,
Quoted 21 lines · Jan Beulich
Quoted 19 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 2 lines · Julien Grall
Fair enough. I would still like to consider a way where we could avoid to hack xsm_* because we have the interrupts disabled.
Well, from a conceptual pov it's at least questionable for XSM to need any memory allocations at all when merely being asked for permission. And indeed the need for it arises solely from its desire to cache the result, for the sake of subsequent lookups.
Quoted 7 lines
I also find it odd that there's an XSM check on the send path in the first place. This isn't just because it would seem to me that it should be decided at binding time whether sending is permitted
- I may easily be missing something in the conceptual model here. It's also because__domain_finalise_shutdown()too usesevtchn_send(), and I didn't think this one should be subject to any XSM check (just like send_guest_*() aren't).Maybe this is the first question we need to answer?
Quoted 8 lines
Quoted 5 lines
Indeed that was the question I asked myself before putting together the patch. Yet I have no idea who could answer it, with Daniel having gone silent for quite long a time now. Hence I didn't even put up the question, but instead tried to find a halfway reasonable solution.
IIRC, XSM is used by OpenXT and QubeOS. So I have CCed them to get some input.
I think the send hook exists because send is its own distinct operation. While most common usage could be handled by just checking at bind time, the send hoor provides more flexibility. For instance, the send hook can be used to restrict signalling to only one direction. Also, a domain label can transition (change) at runtime. Dropping the send check would latch the permission at bind time which would not necessarily be valid for the security policy.
Am I correct that the assertion mentioned in the patch description would only be seen in debug builds?
Thanks,
Jason
Quoted 32 lines · Jason Andryuk
Quoted 31 lines · Julien Grall
Hi Jan,
Quoted 26 lines · Jan Beulich
Quoted 18 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 2 lines · Julien Grall
Fair enough. I would still like to consider a way where we could avoid to hack xsm_* because we have the interrupts disabled.
Well, from a conceptual pov it's at least questionable for XSM to need any memory allocations at all when merely being asked for permission. And indeed the need for it arises solely from its desire to cache the result, for the sake of subsequent lookups.
Quoted 7 lines
I also find it odd that there's an XSM check on the send path in the first place. This isn't just because it would seem to me that it should be decided at binding time whether sending is permitted
- I may easily be missing something in the conceptual model here. It's also because__domain_finalise_shutdown()too usesevtchn_send(), and I didn't think this one should be subject to any XSM check (just like send_guest_*() aren't).Quoted 1 line
Maybe this is the first question we need to answer?
Indeed that was the question I asked myself before putting together the patch. Yet I have no idea who could answer it, with Daniel having gone silent for quite long a time now. Hence I didn't even put up the question, but instead tried to find a halfway reasonable solution.
Quoted 8 lines
Quoted 2 lines
IIRC, XSM is used by OpenXT and QubeOS. So I have CCed them to get some input.
I think the send hook exists because send is its own distinct operation. While most common usage could be handled by just checking at bind time, the send hoor provides more flexibility. For instance, the send hook can be used to restrict signalling to only one direction.
I did realize this is a special case, but it could still be dealt with at binding time, via a boolean in struct evtchn.
Quoted 3 lines
Also, a domain label can transition (change) at runtime.
Dropping the send check would latch the permission at bind time which would not necessarily be valid for the security policy.
I did realize this as a possibility too, but there the immediate question is: Why for interdomain channels, but then not also for vIRQ-s, for example? In fact, unless I'm overlooking something, for this specific case there's not even any check in the binding logic, not even for global vIRQ-s. (After all there are two aspects in the permissions here: One is to be eligible to send, which ought to not matter when the sender is Xen, while the other is permission to learn / know of certain events, i.e. in particular global vIRQ-s.)
The fundamental issue here is that the sending path should be fast and hence lightweight. This means (to me) that in particular no memory allocations should occur, and (more generally) no global or domain wide locks should be taken (for rw ones: no write locks).
Quoted 2 lines
Am I correct that the assertion mentioned in the patch description would only be seen in debug builds?
Yes. Release builds would instead have deadlock potential, which may get realized at any time (or never, if you're really lucky). Catching such cases early, and in an easy to recognize manner, is - after all - what the extra checking in debug builds is for.
Jan
Quoted 50 lines · Jan Beulich
Quoted 32 lines · Jason Andryuk
Quoted 31 lines · Julien Grall
Hi Jan,
Quoted 26 lines · Jan Beulich
Quoted 18 lines · Julien Grall
Quoted 9 lines · Jan Beulich
Quoted 2 lines · Julien Grall
Fair enough. I would still like to consider a way where we could avoid to hack xsm_* because we have the interrupts disabled.
Well, from a conceptual pov it's at least questionable for XSM to need any memory allocations at all when merely being asked for permission. And indeed the need for it arises solely from its desire to cache the result, for the sake of subsequent lookups.
Quoted 7 lines
I also find it odd that there's an XSM check on the send path in the first place. This isn't just because it would seem to me that it should be decided at binding time whether sending is permitted
- I may easily be missing something in the conceptual model here. It's also because__domain_finalise_shutdown()too usesevtchn_send(), and I didn't think this one should be subject to any XSM check (just like send_guest_*() aren't).Quoted 1 line
Maybe this is the first question we need to answer?
Indeed that was the question I asked myself before putting together the patch. Yet I have no idea who could answer it, with Daniel having gone silent for quite long a time now. Hence I didn't even put up the question, but instead tried to find a halfway reasonable solution.
Quoted 8 lines
Quoted 2 lines
IIRC, XSM is used by OpenXT and QubeOS. So I have CCed them to get some input.
I think the send hook exists because send is its own distinct operation. While most common usage could be handled by just checking at bind time, the send hoor provides more flexibility. For instance, the send hook can be used to restrict signalling to only one direction.
I did realize this is a special case, but it could still be dealt with at binding time, via a boolean in struct evtchn.
Quoted 3 lines
Also, a domain label can transition (change) at runtime.
Dropping the send check would latch the permission at bind time which would not necessarily be valid for the security policy.
Quoted 9 lines
I did realize this as a possibility too, but there the immediate question is: Why for interdomain channels, but then not also for vIRQ-s, for example? In fact, unless I'm overlooking something, for this specific case there's not even any check in the binding logic, not even for global vIRQ-s. (After all there are two aspects in the permissions here: One is to be eligible to send, which ought to not matter when the sender is Xen, while the other is permission to learn / know of certain events, i.e. in particular global vIRQ-s.)
I'm not familiar with vIRQ-s, but I did a little bit of review. A vIRQ source is always Xen and its destination is a domain, correct? They don't allow a data flow between domains, so maybe that is why they weren't hooked originally? Hmmm, even for non-XSM, there is no restriction on binding the "dom0" vIRQ-s? Like you say, maybe the "dom0" vIRQ-s should be behind a permission check?
Quoted 5 lines
The fundamental issue here is that the sending path should be fast and hence lightweight. This means (to me) that in particular no memory allocations should occur, and (more generally) no global or domain wide locks should be taken (for rw ones: no write locks).
Yes, that all seems good and reasonable. With XSM/Flask you also need the AVC entry for send to be lightweight.
It wouldn't help with the domain transition case, but you could run the xsm send hooks at bind time to pre-populate the cache. That would still require avc code to bypass the memory allocation when holding a lock in case the entry isn't found. Your preallocation idea could be generalized to have avc maintain a reserve of nodes for use when it cannot allocate. When it can allocate, it would refill the reserve in addition to whatever regular allocation it would perform. But if it's only evtchn_send that needs special handling, then the complexity may not be worth adding.
Quoted 7 lines
Quoted 2 lines
Am I correct that the assertion mentioned in the patch description would only be seen in debug builds?
Yes. Release builds would instead have deadlock potential, which may get realized at any time (or never, if you're really lucky). Catching such cases early, and in an easy to recognize manner, is - after all - what the extra checking in debug builds is for.
Thanks for the confirmation. I tested the XSA patches before seeing this patch, and it "worked". Catching the issue now is definitely better than a mysterious deadlock later.
Thanks,
Jason
Quoted 16 lines · Jason Andryuk
Quoted 5 lines · Jan Beulich
Quoted 3 lines · Jason Andryuk
Also, a domain label can transition (change) at runtime.
Dropping the send check would latch the permission at bind time which would not necessarily be valid for the security policy.Quoted 9 lines
I did realize this as a possibility too, but there the immediate question is: Why for interdomain channels, but then not also for vIRQ-s, for example? In fact, unless I'm overlooking something, for this specific case there's not even any check in the binding logic, not even for global vIRQ-s. (After all there are two aspects in the permissions here: One is to be eligible to send, which ought to not matter when the sender is Xen, while the other is permission to learn / know of certain events, i.e. in particular global vIRQ-s.)
Quoted 3 lines
I'm not familiar with vIRQ-s, but I did a little bit of review. A vIRQ source is always Xen and its destination is a domain, correct? They don't allow a data flow between domains,
Yes and yes.
Quoted 1 line
so maybe that is why they weren't hooked originally?
Not so much, I assume. Looking a little more closely I find that ...
Quoted 2 lines
Hmmm, even for non-XSM, there is no restriction on binding the "dom0" vIRQ-s?
... while binding is allowed, an event would never be received unless the domain was designated as the receiver via XEN_DOMCTL_set_virq_handler.
Quoted 11 lines
Quoted 5 lines
The fundamental issue here is that the sending path should be fast and hence lightweight. This means (to me) that in particular no memory allocations should occur, and (more generally) no global or domain wide locks should be taken (for rw ones: no write locks).
Yes, that all seems good and reasonable. With XSM/Flask you also need the AVC entry for send to be lightweight.
It wouldn't help with the domain transition case, but you could run the xsm send hooks at bind time to pre-populate the cache.
Question is for how long such an entry would remain in the cache, i.e. whether pre-filling is useful at all. After all pre-filling has the downside of potentially masking real issues when testing (as opposed to running in the wild).
Quoted 8 lines
That would
still require avc code to bypass the memory allocation when holding a lock in case the entry isn't found. Your preallocation idea could be generalized to have avc maintain a reserve of nodes for use when it cannot allocate. When it can allocate, it would refill the reserve in addition to whatever regular allocation it would perform. But if it's onlyevtchn_sendthat needs special handling, then the complexity may not be worth adding.
It was this last aspect which made me not introduce a general mechanism.
Jan
Jason Andryuk
Tested-byQuoted 26 lines · Jan Beulich
Quoted 16 lines · Jason Andryuk
Quoted 5 lines · Jan Beulich
Quoted 3 lines · Jason Andryuk
Also, a domain label can transition (change) at runtime.
Dropping the send check would latch the permission at bind time which would not necessarily be valid for the security policy.Quoted 9 lines
I did realize this as a possibility too, but there the immediate question is: Why for interdomain channels, but then not also for vIRQ-s, for example? In fact, unless I'm overlooking something, for this specific case there's not even any check in the binding logic, not even for global vIRQ-s. (After all there are two aspects in the permissions here: One is to be eligible to send, which ought to not matter when the sender is Xen, while the other is permission to learn / know of certain events, i.e. in particular global vIRQ-s.)
Quoted 3 lines
I'm not familiar with vIRQ-s, but I did a little bit of review. A vIRQ source is always Xen and its destination is a domain, correct? They don't allow a data flow between domains,
Yes and yes.
Quoted 1 line
so maybe that is why they weren't hooked originally?
Quoted 8 lines
Not so much, I assume. Looking a little more closely I find that ...
Quoted 2 lines
Hmmm, even for non-XSM, there is no restriction on binding the "dom0" vIRQ-s?
... while binding is allowed, an event would never be received unless the domain was designated as the receiver via
XEN_DOMCTL_set_virq_handler.
And all of those default to the hardware_domain. This permission isn't granted in the default policy, so I was initially confused as to how it worked.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
The fundamental issue here is that the sending path should be fast and hence lightweight. This means (to me) that in particular no memory allocations should occur, and (more generally) no global or domain wide locks should be taken (for rw ones: no write locks).
Yes, that all seems good and reasonable. With XSM/Flask you also need the AVC entry for send to be lightweight.
Quoted 7 lines
Quoted 2 lines
It wouldn't help with the domain transition case, but you could run the xsm send hooks at bind time to pre-populate the cache.
Question is for how long such an entry would remain in the cache, i.e. whether pre-filling is useful at all. After all pre-filling has the downside of potentially masking real issues when testing (as opposed to running in the wild).
Yes, good point.
Quoted 11 lines
Quoted 8 lines
That would
still require avc code to bypass the memory allocation when holding a lock in case the entry isn't found. Your preallocation idea could be generalized to have avc maintain a reserve of nodes for use when it cannot allocate. When it can allocate, it would refill the reserve in addition to whatever regular allocation it would perform. But if it's onlyevtchn_sendthat needs special handling, then the complexity may not be worth adding.It was this last aspect which made me not introduce a general mechanism.
I think we go with this patch since it doesn't restrict the use of XSM and tries to populate the avc.
Thanks,
Jason
Quoted 21 lines · Jan Beulich
Quoted 9 lines · Julien Grall
Quoted 7 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.Quoted 6 lines
This is the sort of fall-out I was expecting when we decide to turn off the interrupts for big chunk of code. I couldn't find any at the time though...
Can you remind which caller of send_guest{global, vcpu}_virq() will call them with interrupts off?
I don't recall which one of the two it was that I hit; we wanted both to use the lock anyway.
send_guest_pirq()very clearly also gets called with IRQs off.
Quoted 8 lines
Quoted 2 lines
Would it be possible to consider deferring the call to a softirq taslket? If so, this would allow us to turn the interrupts again.
Of course this is in principle possible; the question is how involved this is going to get. However, on x86 oprofile's call to
send_guest_vcpu_virq()can't easily be replaced - it's dangerous enough already that in involves locks in NMI context. I don't fancy seeing it use more commonly used ones.
Is it really so hard to avoid calling send_guest_vcpu_virq() in NMI context?
Today it is called only if the NMI happened inside the guest, so the main Xen stack is unused at this time. It should be rather straight forward to mimic a stack frame on the main stack and iret to a special handler from NMI context. This handler would then call send_guest_vcpu_virq() and return to the guest.
This would basically be similar to the Linux kernel's __run_on_irqstack().
Juergen
Quoted 31 lines · Jürgen Groß
Quoted 21 lines · Jan Beulich
Quoted 9 lines · Julien Grall
Quoted 7 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.Quoted 6 lines
This is the sort of fall-out I was expecting when we decide to turn off the interrupts for big chunk of code. I couldn't find any at the time though...
Can you remind which caller of send_guest{global, vcpu}_virq() will call them with interrupts off?
I don't recall which one of the two it was that I hit; we wanted both to use the lock anyway.
send_guest_pirq()very clearly also gets called with IRQs off.Quoted 8 lines
Quoted 2 lines
Would it be possible to consider deferring the call to a softirq taslket? If so, this would allow us to turn the interrupts again.
Of course this is in principle possible; the question is how involved this is going to get. However, on x86 oprofile's call to
send_guest_vcpu_virq()can't easily be replaced - it's dangerous enough already that in involves locks in NMI context. I don't fancy seeing it use more commonly used ones.
Quoted 8 lines
Is it really so hard to avoid calling
send_guest_vcpu_virq()in NMI context?Today it is called only if the NMI happened inside the guest, so the main Xen stack is unused at this time. It should be rather straight forward to mimic a stack frame on the main stack and iret to a special handler from NMI context. This handler would then call
send_guest_vcpu_virq()and return to the guest.
Quite possible that it's not overly difficult to arrange for. But even with this out of the way I don't really view this softirq tasklet route as viable; I could be proven wrong by demonstrating that it's sufficiently straightforward.
Jan
Hi,
Quoted 8 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.
Quoted 7 lines
As these nodes are used merely for caching earlier decisions' results, allocate just one node in AVC code despite two potentially being needed. Things will merely be not as performant if a second allocation was wanted, just like when the pre-allocation fails.
Fixes c0ddc8634845 ("evtchn: convert per-channel lock to be IRQ-safe")Signed-off-by Jan Beulich <[email protected]>
As discussed on the community call with one comment below:
Quoted 10 lines
---
TBD: An even easier fix could be to simply guardxzalloc()by a
conditional checkinglocal_irq_is_enabled(), but for a domain sending only interdomain events this would mean AVC's node caching would never take effect on the sending path, as allocation would then always be avoided.
xen/common/event_channel.c int evtchn_send(struct domain *ld, unsig · line 730
Quoted 8 lines
xen/common/event_channel.c +5
-0@@ -723,6 +723,12 @@ int evtchn_send(struct domain *ld, unsigif ( !port_is_valid (ld, lport) )return -EINVAL ; +/* +* As the call further down needs to avoid allocations (due to running +* with IRQs off), give XSM a chance to pre-allocate if needed. +*/ +xsm_evtchn_send (XSM_HOOK , ld,NULL );
I would suggest to add a comment on top of the evtchn_send callback in the XSM hook. This would be helpful for any developer of a new XSM policy.
Cheers,
Signature · 1 line
Julien Grall
Quoted 17 lines · Julien Grall
Hi,
Quoted 13 lines · Jan Beulich
xmalloc()& Co may not be called with IRQs off, or elsecheck_lock()will have its assertion trigger about locks getting acquired inconsistently. Re-arranging the locking inevtchn_send()doesn't seem very reasonable, especially since the per-channel lock was introduced to avoid acquiring the per-domain event lock on the send paths. Issue a second call toxsm_evtchn_send()instead, before acquiring the lock, to give XSM / Flask a chance to pre-allocate whatever it may need.As these nodes are used merely for caching earlier decisions' results, allocate just one node in AVC code despite two potentially being needed. Things will merely be not as performant if a second allocation was wanted, just like when the pre-allocation fails.
Quoted 6 lines
Quoted 2 lines
Fixes c0ddc8634845 ("evtchn: convert per-channel lock to be IRQ-safe")Signed-off-by Jan Beulich <[email protected]>As discussed on the community call with one comment below:
Acked-by Julien Grall <[email protected]>
Thanks,
Jason