Skip to contentSkip to conversation

RT and XFS

PATCH Daniel Walker 8 participants 22 messages Jul 12, 2005 – Jul 19, 2005 1 message not in the archive XFS filesystem

lore
Share & tools
lore
#0 lore raw

Is there something so odd about the XFS locking, that it can't use the rt_lock ?

fs/xfs/linux-2.6/mrlock.h +2 -2
@@ -37,12 +37,12 @@
 enum { MR_NONE, MR_ACCESS, MR_UPDATE };
 
 typedef struct {
-	struct rw_semaphore	mr_lock;
-	int			mr_writer;
+	struct compat_rw_semaphore	mr_lock;
+	int				mr_writer;
 } mrlock_t;
 
 
 
#1 lore raw
Quoted 7 lines · Daniel Walker

Is there something so odd about the XFS locking, that it can't use the rt_lock ?

fs/xfs/linux-2.6/mrlock.h line 42

Quoted 8 lines
fs/xfs/linux-2.6/mrlock.h +2 -2
@@ -37,12 +37,12 @@
 enum { MR_NONE, MR_ACCESS, MR_UPDATE };
 
 typedef struct {
-	struct rw_semaphore	mr_lock;
-	int			mr_writer;
+	struct compat_rw_semaphore	mr_lock;
+	int				mr_writer;
 } mrlock_t;

BTW, what's the difference between rw_semaphore and compat_rw_semaphore? Or between semaphore and compat_semaphore? I ran into a similar issue (needing compat_semaphore) with the IVTV drivers. The following is a portion of my patch to get IVTV running under RT (the other portions are just compile-time semantics):

--- ivtv-0.2.0-rc3k.orig/driver/msp3400.c 2004-11-19 08:21:04.000000000 -0800
+++ ivtv-0.2.0-rc3k/driver/msp3400.c 2005-06-22 17:26:24.000000000
-0700
@@ -115,7 +115,7 @@
struct task_struct *thread;
wait_queue_head_t wq;

- struct semaphore *notify;
+ struct compat_semaphore *notify;
int active,restart,rmmod;

int watch_stereo;

--ww

#2 lore raw
In reply to Daniel Walker
Quoted 3 lines · Daniel Walker

Is there something so odd about the XFS locking, that it can't use the rt_lock ?

Not that I know of - XFS does use the downgrade_write interface, whose use isn't overly common in the rest of the kernel... maybe that has caused some confusion, dunno.

fs/xfs/linux-2.6/mrlock.h line 42

Quoted 11 lines
fs/xfs/linux-2.6/mrlock.h +2 -2
@@ -37,12 +37,12 @@
 enum { MR_NONE, MR_ACCESS, MR_UPDATE };
 
 typedef struct {
-	struct rw_semaphore	mr_lock;
-	int			mr_writer;
+	struct compat_rw_semaphore	mr_lock;
+	int				mr_writer;
 } mrlock_t;

The XFS code is also written such that it just releases a mrlock without tracking whether it had it for access/update in the end (end lock state is not necessarily how it started out, since it may have downgraded the lock at some point, or it may not have). Its a non-trivial change to track that state within XFS itself, so the above mr_writer field in XFS's mrlock wrapper tracks that state alongside the rw_semaphore. It would prefer to be getting that out of the rw_semaphore itself, alot, but there's not any mechanism for doing so (its not a particularly nice API change either, really, for the generic locking code). I guess that may have been another reason for the above change in the RT patch, I don't know all the details there.

cheers.

Signature · 1 line
Nathan
#3 lore raw
Quoted 8 lines · Nathan Scott
Quoted 3 lines · Daniel Walker

Is there something so odd about the XFS locking, that it can't use the rt_lock ?

Not that I know of - XFS does use the downgrade_write interface, whose use isn't overly common in the rest of the kernel... maybe that has caused some confusion, dunno.

Current RT doesn't implement downgrade_write() , but it's trivial to add it.

Quoted 16 lines

fs/xfs/linux-2.6/mrlock.h line 42

Quoted 11 lines
fs/xfs/linux-2.6/mrlock.h +2 -2
@@ -37,12 +37,12 @@
 enum { MR_NONE, MR_ACCESS, MR_UPDATE };
 
 typedef struct {
-	struct rw_semaphore	mr_lock;
-	int			mr_writer;
+	struct compat_rw_semaphore	mr_lock;
+	int				mr_writer;
 } mrlock_t;

The XFS code is also written such that it just releases a mrlock without tracking whether it had it for access/update in the end (end lock state is not necessarily how it started out, since it may have downgraded the lock at some point, or it may not have).

Quoted 8 lines

Its a non-trivial change to track that state within XFS itself, so the above mr_writer field in XFS's mrlock wrapper tracks that state alongside the rw_semaphore. It would prefer to be getting that out of the rw_semaphore itself, alot, but there's not any mechanism for doing so (its not a particularly nice API change either, really, for the generic locking code). I guess that may have been another reason for the above change in the RT patch, I don't know all the details there.

So it calls up_read if it has a read lock ? Or up_write if it has a write lock? I suppose it would be broken if it didn't though.

Daniel

#4 lore raw
Quoted 6 lines · Daniel Walker
Quoted 5 lines · Nathan Scott
Quoted 3 lines · Daniel Walker

Is there something so odd about the XFS locking, that it can't use the rt_lock ?

Quoted 6 lines
Quoted 3 lines

Not that I know of - XFS does use the downgrade_write interface, whose use isn't overly common in the rest of the kernel... maybe that has caused some confusion, dunno.

Current RT doesn't implement downgrade_write() , but it's trivial to add it.

Ah, thats probably it then.

Quoted 2 lines

So it calls up_read if it has a read lock ? Or up_write if it has a write lock? I suppose it would be broken if it didn't though.

Thats correct.

cheers.

Signature · 1 line
Nathan
#5 lore raw
In reply to Nathan Scott
Quoted 8 lines · Nathan Scott
Quoted 3 lines · Daniel Walker

Is there something so odd about the XFS locking, that it can't use the rt_lock ?

Not that I know of - XFS does use the downgrade_write interface, whose use isn't overly common in the rest of the kernel... maybe that has caused some confusion, dunno.

downgrade_write() wasnt the main problem - the main problem was that for PREEMPT_RT i implemented 'strict' semaphores, which are not identical to vanilla kernel semaphores. The thing that seemed to impact XFS the most is the 'acquirer thread has to release the lock' rule of strict semaphores. Both the XFS logging code and the XFS IO completion code seems to release locks in a different context from where the acquire happened. It's of course valid upstream behavior, but without these extra rules it's hard to do sane priority inheritance. (who do you boost if you dont really know who 'owns' the lock?) It might make sense to introduce some sort of sem_pass_to(new_owner) interface? For now i introduced a compat type, which lets those semaphores fall back to the vanilla implementation.

Ingo

#6 lore raw
Quoted 13 lines · Ingo Molnar

downgrade_write() wasnt the main problem - the main problem was that for PREEMPT_RT i implemented 'strict' semaphores, which are not identical to vanilla kernel semaphores. The thing that seemed to impact XFS the most is the 'acquirer thread has to release the lock' rule of strict semaphores. Both the XFS logging code and the XFS IO completion code seems to release locks in a different context from where the acquire happened. It's of course valid upstream behavior, but without these extra rules it's hard to do sane priority inheritance. (who do you boost if you dont really know who 'owns' the lock?) It might make sense to introduce some sort of sem_pass_to(new_owner) interface? For now i introduced a compat type, which lets those semaphores fall back to the vanilla implementation.

There's a lot of code like this in there .. I've seen some that down() in process contex, and up() in interrupt contex which is weird .. But those aren't major features, just little drivers. XFS is pretty major feature.

Nathan, does XFS need this property or could we convert it to synchronize the locking (with ease?)?

Daniel

#7 lore raw

Hi there,

Quoted 14 lines · Daniel Walker
Quoted 13 lines · Ingo Molnar

downgrade_write() wasnt the main problem - the main problem was that for PREEMPT_RT i implemented 'strict' semaphores, which are not identical to vanilla kernel semaphores. The thing that seemed to impact XFS the most is the 'acquirer thread has to release the lock' rule of strict semaphores. Both the XFS logging code and the XFS IO completion code seems to release locks in a different context from where the acquire happened. It's of course valid upstream behavior, but without these extra rules it's hard to do sane priority inheritance. (who do you boost if you dont really know who 'owns' the lock?) It might make sense to introduce some sort of sem_pass_to(new_owner) interface? For now i introduced a compat type, which lets those semaphores fall back to the vanilla implementation.

Hmm, I'm not aware of anywhere in XFS where we do that. From talking to some colleagues here, they're claiming that we can't be doing that since it'd trip an assert in the IRIX mrlock code.

Quoted 7 lines

There's a lot of code like this in there .. I've seen some that down() in process contex, and up() in interrupt contex which is weird .. But those aren't major features, just little drivers. XFS is pretty major feature.

Nathan, does XFS need this property or could we convert it to synchronize the locking (with ease?)?

I'm not yet sure in what situations we are doing this, so can't really say. It'd be interesting to see an implementation of the downgrade_write functionality and then a specific case where the above locking behaviour happens ... and I'd then be able to say how tricky that would be to resolve.

Steve, are you aware of situations where we unlock in a different thread to where we acquired the lock? It'd surprise me, as we're holding these things for as short a time as possible - afaict the transactions always ilock, copy delta to iclog, pin, and unlock, no? (all in the same thread). I can't see the iolock being used in this way anywhere either... you?

cheers.

Signature · 1 line
Nathan
#8 lore raw
Quoted 18 lines · Nathan Scott

Hi there,

Quoted 14 lines · Daniel Walker
Quoted 13 lines · Ingo Molnar

downgrade_write() wasnt the main problem - the main problem was that for PREEMPT_RT i implemented 'strict' semaphores, which are not identical to vanilla kernel semaphores. The thing that seemed to impact XFS the most is the 'acquirer thread has to release the lock' rule of strict semaphores. Both the XFS logging code and the XFS IO completion code seems to release locks in a different context from where the acquire happened. It's of course valid upstream behavior, but without these extra rules it's hard to do sane priority inheritance. (who do you boost if you dont really know who 'owns' the lock?) It might make sense to introduce some sort of sem_pass_to(new_owner) interface? For now i introduced a compat type, which lets those semaphores fall back to the vanilla implementation.

Quoted 3 lines

Hmm, I'm not aware of anywhere in XFS where we do that. From talking to some colleagues here, they're claiming that we can't be doing that since it'd trip an assert in the IRIX mrlock code.

Now that I've read the thread, I see it's not mrlocks that is the issue with unlocking in a different context - it's semaphores.

All the pagebuf synchronisation is done with a semaphore because it's held across the I/O and it's _most definitely_ released in a different context when doing async I/O. Just about all metadata I/O is async because once the transaction has been logged to disk we don't need to write these buffers out synchronously. Not to mention the log I/O completion unlocks the buffers in a transaction in a different context as well.

The whole point of using a semaphore in the pagebuf is because there is no tracking of who "owns" the lock so we can actually release it in a different context. Semaphores were invented for this purpose, and we use them in the way they were intended. ;)

Realistically, I seriously doubt the need for any sort of rt changes to these semaphores. They can be held for indeterminant periods of time potentially across multiple disk I/Os (e.g. when held locked in a transaction that requires more metadata to be read in from disk to make progress). Hence there is no really no point in making them RT aware because if you end up waiting on one of them you can forget about pretty much any RT guarantee that you've ever given....

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
R&D Software Engineer
SGI Australian Software Group
#9 lore raw
Quoted 11 lines · Dave Chinner

Now that I've read the thread, I see it's not mrlocks that is the issue with unlocking in a different context - it's semaphores.

All the pagebuf synchronisation is done with a semaphore because it's held across the I/O and it's _most definitely_ released in a different context when doing async I/O. Just about all metadata I/O is async because once the transaction has been logged to disk we don't need to write these buffers out synchronously. Not to mention the log I/O completion unlocks the buffers in a transaction in a different context as well.

Quoted 4 lines

The whole point of using a semaphore in the pagebuf is because there is no tracking of who "owns" the lock so we can actually release it in a different context. Semaphores were invented for this purpose, and we use them in the way they were intended. ;)

Where is the that semaphore spec, is that posix ? There is a new construct called "complete" that is good for this type of stuff too. No owner needed , just something running, and something waiting till it completes.

Quoted 7 lines

Realistically, I seriously doubt the need for any sort of rt changes to these semaphores. They can be held for indeterminant periods of time potentially across multiple disk I/Os (e.g. when held locked in a transaction that requires more metadata to be read in from disk to make progress). Hence there is no really no point in making them RT aware because if you end up waiting on one of them you can forget about pretty much any RT guarantee that you've ever given....

PI is always good, cause it allows the tracking of what is high priority , and what is not .

Daniel

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw
Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Daniel Walker
Quoted 4 lines

The whole point of using a semaphore in the pagebuf is because there is no tracking of who "owns" the lock so we can actually release it in a different context. Semaphores were invented for this purpose, and we use them in the way they were intended. ;)

Quoted 8 lines
Quoted 4 lines

Where is the that semaphore spec, is that posix ? There is a new construct called "complete" that is good for this type of stuff too. No owner needed , just something running, and something waiting till it completes.

wrt. posix, we dont really care about that for kernel-internal primitives like struct semaphore. So whether it's posix or not has no relevance.

This reminds me of Documentation/stable_api_nonsense.txt . That no one should really be dependent on a particular kernel API doing a particular thing. The kernel is play dough for the kernel hacker (as it should be), including kernel semaphores.

So we can change whatever we want, and make no excuses, as long as we fix the rest of the kernel to work with our change. That seems pretty sensible , because Linux should be an evolution.

Daniel

#12 lore raw
Quoted 8 lines · Daniel Walker

This reminds me of Documentation/stable_api_nonsense.txt . That no one should really be dependent on a particular kernel API doing a particular thing. The kernel is play dough for the kernel hacker (as it should be), including kernel semaphores.

So we can change whatever we want, and make no excuses, as long as we fix the rest of the kernel to work with our change. That seems pretty sensible , because Linux should be an evolution.

Daniel, get a fucking clue. Read some CS 101 literature on what a semaphore is defined to be. If you want PI singing dancing blinking christmas tree locking primites call them a mutex, but not a semaphore.

#13 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Daniel Walker

This reminds me of Documentation/stable_api_nonsense.txt . That no one should really be dependent on a particular kernel API doing a particular thing. The kernel is play dough for the kernel hacker (as it should be), including kernel semaphores.

Quoted 8 lines
Quoted 3 lines

So we can change whatever we want, and make no excuses, as long as we fix the rest of the kernel to work with our change. That seems pretty sensible , because Linux should be an evolution.

Daniel, get a fucking clue. Read some CS 101 literature on what a semaphore is defined to be. If you want PI singing dancing blinking christmas tree locking primites call them a mutex, but not a semaphore.

As a matter of fact I just finished what corresponds to your "CS 101" (I study CS in spare time while having a full time job coding RT stuff):
To the one lecture I attended they talked about sempahores. They tought students to use binary semphores for locking. Based on real-life experience (and the Pathfinder story), I complained and told
them they ought to teach the students to use a mutex instead. They had no clue "It is the same thing they said". Yes, a mutex can be implemented just as a binary semaphore but the semantics of it is different. In RT the difference is very important and even without-RT it is a good idea to maintain the difference for readability and deadlock detection. If you later on want to optimize the semaphore for what it is used for it is also good to have maintained that information. It is a bit like discarding the type information from you programs. You want to keep the type information even though the compilere end up producing the same code.

The kernel developer clearly have followed the same lectures and used plain binary semaphores, sometimes calling the mutex sometimes semaphore. I believe that the semaphore ought to be removed. Either use a mutex or a completion. Far the most code is using a sempahore as either signalling
- i.e. as a completion - or critical sections - i.e. as a mutex. If code mixes the usage it is must likely very hard to read....

Unfortunately, one of the goals of the preempt-rt branch is to avoid altering too much code. Therefore the type semaphore can't be removed there. Therefore the name still lingers ... :-(

Esben

#14 lore raw
Quoted 3 lines · Esben Nielsen

Unfortunately, one of the goals of the preempt-rt branch is to avoid altering too much code. Therefore the type semaphore can't be removed there. Therefore the name still lingers ... :-(

This is where you failed. You assumed that that person making the comment, Christopher, in the first place didn't have his head up his ass in the first place and was open to your end of the discussion.

bill

#15 lore raw
Quoted 8 lines · Bill Huey
Quoted 3 lines · Esben Nielsen

Unfortunately, one of the goals of the preempt-rt branch is to avoid altering too much code. Therefore the type semaphore can't be removed there. Therefore the name still lingers ... :-(

This is where you failed. You assumed that that person making the comment, Christopher, in the first place didn't have his head up his ass in the first place and was open to your end of the discussion.

please take me off the Cc: list for such kind of replies. Christoph is very much entitled to his opinion, which i happen to mostly share in this case: we should not be bothering upstream with requirements unique to PREEMPT_RT. PREEMPT_RT restricts struct semaphore to be a mutex, and that doesnt make it a classic semaphore anymore. We had no other choice but it's still somewhat unclean in that regard.

(I do disagree with Christoph on another point: i do think we eventually want to change the standard semaphore type in a similar fashion upstream as well - but that probably has to come with a s/struct semaphore/struct mutex/ change as well.)

Ingo

#16 lore raw
Quoted 4 lines · Ingo Molnar

(I do disagree with Christoph on another point: i do think we eventually want to change the standard semaphore type in a similar fashion upstream as well - but that probably has to come with a s/struct semaphore/struct mutex/ change as well.)

Actually having a mutex_t in mainline would be a good idea even without preempt rt, to document better what kind of locking we expect.

#17 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Ingo Molnar

(I do disagree with Christoph on another point: i do think we eventually want to change the standard semaphore type in a similar fashion upstream as well - but that probably has to come with a s/struct semaphore/struct mutex/ change as well.)

Actually having a mutex_t in mainline would be a good idea even without preempt rt, to document better what kind of locking we expect.

cool! I'll cook up a patch for that. Right now these are the numbers:
there are 526 uses of struct semaphore in 2.6.12. In the -RT tree i had to change 23 of them to be compat_semaphore - i.e. 23 uses were definitely non-mutex.

(We sure have missed some cases - but it would be fair to say that the expected number of cases is less than 50, and that we've mapped the most common ones already. That makes it a 90%/10% splitup: more than 90% of all struct semaphore use is pure mutex.)

Of the remaining <10% cases, the majority is of the type of completions, and there are a handful of (<10) cases of 'counted semaphore' uses:
semaphores with a count larger than 1. (e.g. ACPI uses it to count resources, some audio code too - but it's very rare) Btw., that's the only 'true' (in terms of CS) semaphore use.

Ingo

#18 lore raw
In reply to Daniel Walker
Quoted 26 lines · Daniel Walker
Quoted 7 lines · Ingo Molnar
Quoted 5 lines · Daniel Walker
Quoted 4 lines

The whole point of using a semaphore in the pagebuf is because there is no tracking of who "owns" the lock so we can actually release it in a different context. Semaphores were invented for this purpose, and we use them in the way they were intended. ;)

Quoted 8 lines
Quoted 4 lines

Where is the that semaphore spec, is that posix ? There is a new construct called "complete" that is good for this type of stuff too. No owner needed , just something running, and something waiting till it completes.

wrt. posix, we dont really care about that for kernel-internal primitives like struct semaphore. So whether it's posix or not has no relevance.

This reminds me of Documentation/stable_api_nonsense.txt . That no one should really be dependent on a particular kernel API doing a particular thing. The kernel is play dough for the kernel hacker (as it should be), including kernel semaphores.

So we can change whatever we want, and make no excuses, as long as we fix the rest of the kernel to work with our change. That seems pretty sensible , because Linux should be an evolution.

Quoted 7 lines

Daniel

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

---end quoted text---

#19 lore raw
In reply to Daniel Walker
Quoted 2 lines · Daniel Walker

PI is always good, cause it allows the tracking of what is high priority , and what is not .

that's just plain wrong. PI might be good if one cares about priorities and worst-case latencies, but most of the time the kernel is plain good enough and we dont care. PI can also be pretty expensive. So in no way, shape or form can PI be "always good".

Ingo

#20 lore raw
Quoted 9 lines · Ingo Molnar
Quoted 2 lines · Daniel Walker

PI is always good, cause it allows the tracking of what is high priority , and what is not .

that's just plain wrong. PI might be good if one cares about priorities and worst-case latencies, but most of the time the kernel is plain good enough and we dont care. PI can also be pretty expensive. So in no way, shape or form can PI be "always good".

I don't agree with that. But of course I'm always speaking from a real time perspective . PI is expensive , but it won't always be. However, no one is forcing PI on anyone, even if I think it's good ..

Daniel

#21 lore raw
Quoted 6 lines · Daniel Walker
Quoted 5 lines · Ingo Molnar
Quoted 2 lines · Daniel Walker

PI is always good, cause it allows the tracking of what is high priority , and what is not .

Quoted 9 lines
Quoted 4 lines

that's just plain wrong. PI might be good if one cares about priorities and worst-case latencies, but most of the time the kernel is plain good enough and we dont care. PI can also be pretty expensive. So in no way, shape or form can PI be "always good".

I don't agree with that. But of course I'm always speaking from a real time perspective . PI is expensive , but it won't always be. However, no one is forcing PI on anyone, even if I think it's good ..

Is PI needed? If you use a mutex to protect a critical area you are destroying the strict meaning of priorities if the mutex doesn't have PI:
Priority inversion can effectively make the high priority task low priority in that situation and postpone it's execution indefinitely. For RT applications that is clearly unacceptable.

One can argue that for non-RT tasks priorities aren't supposed to be that rigid as for RT tasks, anyway. Therefore it doesn't matter so much. But as I read the comments in sched.c a nice -20 task have to preempt any nice 0 task no matter how much a cpu-hog it is. If it happens to share a critical section with a nice +19 task, priority inversion will occationally destroy that property. If we disregard the costs of PI, PI is thus a good thing.

But how expensive is PI? Ofcourse there is an overhead in doing the calculations. Ingo's implementation can be optimized quite a bit once things are settled but it will always be many times more expensive than a raw spin-lock. But is it much more expensive than a plain binary semaphore?

If the is no congestion on a mutex the PI code will not be called at all. On UP, the only occation where congestion can occur is when a low priority task is preempted by a higher priority task while it has the mutex. So let us look at the expensive part where the high priority task tries to grab the mutex:

With PI: The owner have to be boosted, an immediate task switch have to take place, the owner runs to the unlock operation and it set down in priority, whereafter there is a task-switch again to the highpriority task.

Without PI: The owner waits and there is a task switch to some thread which might not be the owner but often is. When the owner eventually unlocks the mutex it will be follow by a task-switch - because congestion can only occur when the task trying to get the task preempts and thus have higher priority than the owner.

The number of task switches are thus the same with and without PI!

And then there is the cache issue: When other tasks gets scheduled in the priority inversion case the data being protected can be flushed from the cache while they are running. With PI the CPU continues to work with the same data - and most often in the same code module. I.e. there is a higher chance that the instruction and data cache contains the right data.

Thus in the end it all depends on how cheaply the PI calculations can be made.

Esben

Quoted 8 lines

Daniel

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#22 lore raw
In reply to Daniel Walker
Quoted 3 lines · Daniel Walker

I don't agree with that. But of course I'm always speaking from a real time perspective . PI is expensive , but it won't always be. However, no one is forcing PI on anyone, even if I think it's good ..

It depends on what kind of PI under specific circumstances. In the general kernel, it's really to be avoided at all costs since it's masking a general contention problem at those places. In a formally provable worst case system using priority ceiling emulation and stuff, PI really valuable. How a system like the Linux kernel fits into that is a totally different story. General purpose kernels using general purpose facilities don't.

That's how I see it.

bill