Skip to contentSkip to conversation

quota: dqio_mutex design

DISCUSSION Andrew Perepechko 4 participants 22 messages Feb 2, 2017 – Aug 14, 2017 1 message not in the archive VFS

lore
Share & tools
lore
#0 lore raw

Hello!

We have a heavy metadata related workload (ext4, quota journalling) and profiling shows that there's significant dqio_mutex contention.

Quoted 1 line

From the quota code, it looks like every time dqio_mutex is taken

it protects access to only one quota file.

Is it possible to split dqio_mutex for each of MAXQUOTAS so that e.g. 2 parallel dquot_commit()'s can be running for user and group quota update? Am I missing any dqio_mutex function that requires dqio_mutex to be monolithic?

Thank you,
Andrew

reviewer
#1 lore raw

Hello!

Quoted 10 lines · Andrew Perepechko

We have a heavy metadata related workload (ext4, quota journalling) and profiling shows that there's significant dqio_mutex contention.

From the quota code, it looks like every time dqio_mutex is taken it protects access to only one quota file.

Is it possible to split dqio_mutex for each of MAXQUOTAS so that e.g. 2 parallel dquot_commit()'s can be running for user and group quota update? Am I missing any dqio_mutex function that requires dqio_mutex to be monolithic?

So we can certainly make dqio_mutex less heavy. Making it per-quota-type would OK but I suspect it will not bring a big benefit. What would likely be more noticeable is if we avoided dqio_mutex for updates of quota information - that should not be that hard to do since we update that in-place and so don't really need the serialization for anything substantial. However we will need some restructuring of the code to make such locking scheme possible in a clean way...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#2 lore raw

Hello!

Jan, do you think it makes sense, as an improvement until the code restructuring, to exit immediately from ext4_mark_dquot_dirty() if dquot_mark_dquot_dirty() returns 1?

It seems that in this case we are guaranteed that some thread is somewhere in the middle of mark_dquot_dirty() and clear_dquot_dirty(), so it will update the quota file buffer with the latest dquot data.

That would improve a single user/group scenario like:
thread 1) processing dquot_commit()
thread 2) dirtied dquot and is waiting for dqio_mutex thread 3, 4, 5 ...) dirtied dquot and are waiting for dqio_mutex

If we exit immediately on dquot dirtying, threads 3, 4, 5, ... can let thread 2 update the buffer data and themselves may not block on the mutex.

Thank you,
Andrew

Quoted 23 lines

Hello!

Quoted 10 lines · Andrew Perepechko

We have a heavy metadata related workload (ext4, quota journalling) and profiling shows that there's significant dqio_mutex contention.

From the quota code, it looks like every time dqio_mutex is taken it protects access to only one quota file.

Is it possible to split dqio_mutex for each of MAXQUOTAS so that e.g. 2 parallel dquot_commit()'s can be running for user and group quota update? Am I missing any dqio_mutex function that requires dqio_mutex to be monolithic?

So we can certainly make dqio_mutex less heavy. Making it per-quota-type would OK but I suspect it will not bring a big benefit. What would likely be more noticeable is if we avoided dqio_mutex for updates of quota information - that should not be that hard to do since we update that in-place and so don't really need the serialization for anything substantial. However we will need some restructuring of the code to make such locking scheme possible in a clean way...

Honza

reviewer
#3 lore raw

Hi,

Quoted 9 lines · Andrew Perepechko

Jan, do you think it makes sense, as an improvement until the code restructuring, to exit immediately from ext4_mark_dquot_dirty() if dquot_mark_dquot_dirty() returns 1?

It seems that in this case we are guaranteed that some thread is somewhere in the middle of mark_dquot_dirty() and clear_dquot_dirty(), so it will update the quota file buffer with the latest dquot data.

Well, it would mostly work, except if process A dirties dquot outside of transaction (e.g. dquot_set_dqblk()), it could happen that other updates of dquot inside a running transaction will end up relying on update of dquot buffer by process A and that may end only in the next transaction thus breaking the journalling guarantees.

Honza

Quoted 25 lines
Quoted 23 lines

Hello!

Quoted 10 lines · Andrew Perepechko

We have a heavy metadata related workload (ext4, quota journalling) and profiling shows that there's significant dqio_mutex contention.

From the quota code, it looks like every time dqio_mutex is taken it protects access to only one quota file.

Is it possible to split dqio_mutex for each of MAXQUOTAS so that e.g. 2 parallel dquot_commit()'s can be running for user and group quota update? Am I missing any dqio_mutex function that requires dqio_mutex to be monolithic?

So we can certainly make dqio_mutex less heavy. Making it per-quota-type would OK but I suspect it will not bring a big benefit. What would likely be more noticeable is if we avoided dqio_mutex for updates of quota information - that should not be that hard to do since we update that in-place and so don't really need the serialization for anything substantial. However we will need some restructuring of the code to make such locking scheme possible in a clean way...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
reviewer
#4 lore raw
In reply to Jan Kara
Quoted 14 lines · Jan Kara

Hello!

Quoted 10 lines · Andrew Perepechko

We have a heavy metadata related workload (ext4, quota journalling) and profiling shows that there's significant dqio_mutex contention.

From the quota code, it looks like every time dqio_mutex is taken it protects access to only one quota file.

Is it possible to split dqio_mutex for each of MAXQUOTAS so that e.g. 2 parallel dquot_commit()'s can be running for user and group quota update? Am I missing any dqio_mutex function that requires dqio_mutex to be monolithic?

Quoted 7 lines

So we can certainly make dqio_mutex less heavy. Making it per-quota-type would OK but I suspect it will not bring a big benefit. What would likely be more noticeable is if we avoided dqio_mutex for updates of quota information - that should not be that hard to do since we update that in-place and so don't really need the serialization for anything substantial. However we will need some restructuring of the code to make such locking scheme possible in a clean way...

So I'm experimenting with some patches. However I have trouble creating a workload where quota updates would show significant overhead. Can you share which workload is problematic for you? Thanks!

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
reviewer
#6 lore raw
In reply to a message not in the archive

Hi Andrew,

Quoted 3 lines · Andrew Perepechko

The original workload was 50 threads sequentially creating files, each

thread in its own directory, over a fast RAID array.

OK, I can reproduce this. Actually I can reproduce on normal SATA drive. Originally I've tried on ramdisk to simulate really fast drive but there dq_list_lock and dq_data_lock contention is much more visible and the contention on dqio_mutex is minimal (two orders of magnitude smaller). On SATA drive we spend ~45% of runtime contending on dqio_mutex when creating empty files.

The problem is that if it is single user that is creating all these files, it is not clear how we could do much better - all processes contend to update the same location on disk with quota information for that user and they have to be synchronized somehow. If there are more users, we could do better by splitting dqio_mutex on per-dquot basis (I have some preliminary patches for that).

One idea I have how we could make things faster is that instead of having dquot dirty flag, we would have a sequence counter. So currently dquot modification looks like:

update counters in dquot
dquot_mark_dquot_dirty(dquot);
dquot_commit(dquot)
mutex_lock(dqio_mutex);
if (!clear_dquot_dirty(dquot))

    nothing to do -> bail

->commit_dqblk(dquot)
mutex_unlock(dqio_mutex);

When several processes race updating the same dquot, they very often all end up updating dquot on disk even though another process has already written dquot for them while they were waiting for dqio_sem - in my test above the ratio of commit_dqblk / dquot_commit calls was 59%. What we could do is that dquot_mark_dquot_dirty() would return "current sequence of dquot", dquot_commit() would then get sequence that is required to be written and if that is already written (we would also store in dquot latest written sequence), it would bail out doing nothing. This should cut down dqio_mutex hold times and thus wait times but I need to experiment and measure that...

Honza

Quoted 60 lines
Quoted 55 lines · Jan Kara
Quoted 35 lines

Hello!

Quoted 12 lines · Andrew Perepechko

We have a heavy metadata related workload (ext4, quota journalling)

and profiling shows that there's significant dqio_mutex contention.

From the quota code, it looks like every time dqio_mutex is taken

it protects access to only one quota file.

Quoted 7 lines

Is it possible to split dqio_mutex for each of MAXQUOTAS so that

e.g. 2 parallel dquot_commit()'s can be running for user and group

quota update? Am I missing any dqio_mutex function that requires

dqio_mutex to be monolithic?

So we can certainly make dqio_mutex less heavy. Making it per-quota-type

would OK but I suspect it will not bring a big benefit. What would likely

be more noticeable is if we avoided dqio_mutex for updates of quota

Quoted 7 lines

information - that should not be that hard to do since we update that

in-place and so don't really need the serialization for anything

substantial. However we will need some restructuring of the code to make

such locking scheme possible in a clean way...

So I'm experimenting with some patches. However I have trouble creating

a workload where quota updates would show significant overhead. Can you

share which workload is problematic for you? Thanks!

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
reviewer
#7 lore raw
Quoted 5 lines · Jan Kara

Hi Andrew,

Quoted 2 lines · Andrew Perepechko

The original workload was 50 threads sequentially creating files, each

Quoted 8 lines
Quoted 1 line

thread in its own directory, over a fast RAID array.

OK, I can reproduce this. Actually I can reproduce on normal SATA drive. Originally I've tried on ramdisk to simulate really fast drive but there dq_list_lock and dq_data_lock contention is much more visible and the contention on dqio_mutex is minimal (two orders of magnitude smaller). On SATA drive we spend ~45% of runtime contending on dqio_mutex when creating empty files.

So this was just me misinterpretting lockstat data (forgot to divide the wait time by number of processes) - then the result would be that each process waits only ~1% of its runtime for dqio_mutex.

Anyway, my patches show ~10% improvement in runtime when 50 different processes create empty files for 50 different users. As expected there's not measurable benefit when all processes create files for the same user.

Quoted 20 lines

The problem is that if it is single user that is creating all these files, it is not clear how we could do much better - all processes contend to update the same location on disk with quota information for that user and they have to be synchronized somehow. If there are more users, we could do better by splitting dqio_mutex on per-dquot basis (I have some preliminary patches for that).

One idea I have how we could make things faster is that instead of having dquot dirty flag, we would have a sequence counter. So currently dquot modification looks like:

update counters in dquot
dquot_mark_dquot_dirty(dquot);
dquot_commit(dquot)
mutex_lock(dqio_mutex);
if (!clear_dquot_dirty(dquot))

    nothing to do -> bail

->commit_dqblk(dquot)
mutex_unlock(dqio_mutex);

Quoted 10 lines

When several processes race updating the same dquot, they very often all end up updating dquot on disk even though another process has already written dquot for them while they were waiting for dqio_sem - in my test above the ratio of commit_dqblk / dquot_commit calls was 59%. What we could do is that dquot_mark_dquot_dirty() would return "current sequence of dquot", dquot_commit() would then get sequence that is required to be written and if that is already written (we would also store in dquot latest written sequence), it would bail out doing nothing. This should cut down dqio_mutex hold times and thus wait times but I need to experiment and measure that...

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#8 lore raw
Quoted 59 lines · Jan Kara
Quoted 5 lines

Hi Andrew,

Quoted 2 lines · Andrew Perepechko

The original workload was 50 threads sequentially creating files, each

Quoted 8 lines
Quoted 1 line

thread in its own directory, over a fast RAID array.

OK, I can reproduce this. Actually I can reproduce on normal SATA drive. Originally I've tried on ramdisk to simulate really fast drive but there dq_list_lock and dq_data_lock contention is much more visible and the contention on dqio_mutex is minimal (two orders of magnitude smaller). On SATA drive we spend ~45% of runtime contending on dqio_mutex when creating empty files.

So this was just me misinterpretting lockstat data (forgot to divide the wait time by number of processes) - then the result would be that each process waits only ~1% of its runtime for dqio_mutex.

Anyway, my patches show ~10% improvement in runtime when 50 different processes create empty files for 50 different users. As expected there's not measurable benefit when all processes create files for the same user.

Quoted 23 lines

The problem is that if it is single user that is creating all these files, it is not clear how we could do much better - all processes contend to update the same location on disk with quota information for that user and they have to be synchronized somehow. If there are more users, we could do better by splitting dqio_mutex on per-dquot basis (I have some preliminary patches for that).

One idea I have how we could make things faster is that instead of having dquot dirty flag, we would have a sequence counter. So currently dquot modification looks like:

update counters in dquot
dquot_mark_dquot_dirty(dquot);
dquot_commit(dquot)

  mutex_lock(dqio_mutex);
  if (!clear_dquot_dirty(dquot))
    nothing to do -> bail
  ->commit_dqblk(dquot)
  mutex_unlock(dqio_mutex);
Quoted 12 lines

When several processes race updating the same dquot, they very often all end up updating dquot on disk even though another process has already written dquot for them while they were waiting for dqio_sem - in my test above the ratio of commit_dqblk / dquot_commit calls was 59%. What we could
do is that dquot_mark_dquot_dirty() would return "current sequence of dquot", dquot_commit() would then get sequence that is required to be written and if that is already written (we would also store in dquot latest
written sequence), it would bail out doing nothing. This should cut down dqio_mutex hold times and thus wait times but I need to experiment and measure that...

Quoted 6 lines

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

I think, you also mentioned that some mark_dquot_dirty callers, such as do_set_dqblk, may not be running with an open transaction handle, so we cannot assume this optimization is atomic. However, we don't use do_set_dqblk and seem safe wrt journalling.

Thank you,
Andrew

reviewer
#9 lore raw
Quoted 38 lines · Andrew Perepechko
Quoted 14 lines · Jan Kara
Quoted 4 lines

When several processes race updating the same dquot, they very often all end up updating dquot on disk even though another process has already written dquot for them while they were waiting for dqio_sem - in my test above the ratio of commit_dqblk / dquot_commit calls was 59%. What we

Quoted 8 lines

could
do is that dquot_mark_dquot_dirty() would return "current sequence of dquot", dquot_commit() would then get sequence that is required to be written and if that is already written (we would also store in dquot latest
written sequence), it would bail out doing nothing. This should cut down dqio_mutex hold times and thus wait times but I need to experiment and measure that...

Quoted 6 lines

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);
Quoted 8 lines

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

I think, you also mentioned that some mark_dquot_dirty callers, such as do_set_dqblk, may not be running with an open transaction handle, so we cannot assume this optimization is atomic. However, we don't use do_set_dqblk and seem safe wrt journalling.

OK, thanks for info. I'm reluctant to make ext4_mark_dquot_dirty() return before quota data is actually copied to the transaction which is what probably brings you the benefit. I'll think about it some more.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#10 lore raw
In reply to Andrew Perepechko

Hello Guys,

We DDN is investigating the same issue!

Some comments comes:

Quoted 22 lines · Andrew Perepechko
Quoted 9 lines · Jan Kara
Quoted 2 lines

Hi Andrew,

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {

Review

Quoted 2 lines
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Here is some ideas that i have thought:

1) switch dqio_mutex to a read/write lock, especially, i think most of time journal quota updates is in-place update, that means we don't need change quota tree in memory, firstly try read lock, retry with write lock if there is real tree change.

2)another is similar idea of Andrew's walkaround, but we need make correct fix, maintain dirty list for per transaction, and gurantee quota updates are flushed when commit transaction, this might be complex, i am not very familiar with JBD2 codes.

It will be really nice if we could fix this regression, as we see 20% performace regression.

Thanks,
Shilong

Quoted 7 lines
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

#11 lore raw
Quoted 5 lines

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Hello Wang!

There is no transaction change in this case because all callers of this function have open handles for the same transaction.

If you enter that DQ_MOD_B check, you are guaranteed to reference the SAME transaction as the thread that's in between of mark_dirty and clear_dirty.

Thank you,
Andrew

#12 lore raw

Hi,

Quoted 6 lines · Andrew Perepechko
Quoted 5 lines

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Quoted 9 lines

Hello Wang!

There is no transaction change in this case because all callers of this function have open handles for the same transaction.

If you enter that DQ_MOD_B check, you are guaranteed to reference the SAME transaction as the thread that's in between of mark_dirty and clear_dirty.

This change mean if this dquot is dirty we skip, this
won't work because in this way, quota update is only kept in vfs dquota memory and newer update is not wrote to journal file and not wrapped into transaction too.

This is not what journal quota means to do.

Thanks,
Shilong

Quoted 2 lines

Thank you,
Andrew

#13 lore raw
Quoted 5 lines

This change mean if this dquot is dirty we skip, this
won't work because in this way, quota update is only kept in vfs dquota memory and newer update is not wrote to journal file and not wrapped into transaction too.

That's not true.

As I explained earlier, having DQ_MOD_B set at this point means another thread is going to write dquot but hasn't yet started doing so. This thread does not care whether it updates the ondisk dquot with its own data or with fresher data which came from another thread. In-core dquot has no indication of whose data in contains.

As I also explained earlier, the update cannot happen in the context of another transaction because thread A which sees DQ_MOD_B set and thread B which is running dquot_commit() both have journal handles to the same transaction. There's only one running transaction at a time and thread B does not switch to another transaction.

Please read the code carefully.

Quoted 9 lines

This is not what journal quota means to do.

Thanks,
Shilong

Quoted 2 lines

Thank you,
Andrew

#14 lore raw

Let me put it this way:

Under file creation from different threads, ext4 will generate a series of dquot updates (incore and then ondisk, through journal):

dquot update1
dquot update2
dquot update3
...
dquot updateN

Either with my patch or without it, ondisk dquot update through journal may miss dquot update1, dquot update2, ... dquot update{N-1}.

You can easily see that from the code of dquot_commit():

int dquot_commit(struct dquot *dquot)

{
        int ret = 0;
        struct quota_info *dqopt = sb_dqopt(dquot->dq_sb);

        mutex_lock(&dqopt->dqio_mutex);
        spin_lock(&dq_list_lock);
        if (!clear_dquot_dirty(dquot)) {
                spin_unlock(&dq_list_lock);
                goto out_sem;
        }

...
}

If actual dquot_commit() wrote dquot update N, the threads commiting updates 1 through N-1 will exit immediately once they get dqio_mutex since the dquot will NOT be dirty.

My patch only avoids blocking on dqio_mutex when we know for sure that another will NECESSARILY write the needed or a FRESHER dquot ondisk.

Quoted 29 lines
Quoted 4 lines

This change mean if this dquot is dirty we skip, this
won't work because in this way, quota update is only kept in vfs dquota memory and newer update is not wrote to journal file and not wrapped into transaction too.

That's not true.

As I explained earlier, having DQ_MOD_B set at this point means another thread is going to write dquot but hasn't yet started doing so. This thread does not care whether it updates the ondisk dquot with its own data or with fresher data which came from another thread. In-core dquot has no indication of whose data in contains.

As I also explained earlier, the update cannot happen in the context of another transaction because thread A which sees DQ_MOD_B set and thread B which is running dquot_commit() both have journal handles to the same transaction. There's only one running transaction at a time and thread B does not switch to another transaction.

Please read the code carefully.

Quoted 8 lines

This is not what journal quota means to do.

Thanks,
Shilong

Quoted 2 lines

Thank you,
Andrew

reviewer
#15 lore raw
Quoted 31 lines · Andrew Perepechko

Let me put it this way:

Under file creation from different threads, ext4 will generate a series of dquot updates (incore and then ondisk, through journal):

dquot update1
dquot update2
dquot update3
...
dquot updateN

Either with my patch or without it, ondisk dquot update through journal may miss dquot update1, dquot update2, ... dquot update{N-1}.

You can easily see that from the code of dquot_commit():

int dquot_commit(struct dquot *dquot)

{
        int ret = 0;
        struct quota_info *dqopt = sb_dqopt(dquot->dq_sb);

        mutex_lock(&dqopt->dqio_mutex);
        spin_lock(&dq_list_lock);
        if (!clear_dquot_dirty(dquot)) {
                spin_unlock(&dq_list_lock);
                goto out_sem;
        }

...
}

Quoted 6 lines

If actual dquot_commit() wrote dquot update N, the threads commiting updates 1 through N-1 will exit immediately once they get dqio_mutex since the dquot will NOT be dirty.

My patch only avoids blocking on dqio_mutex when we know for sure that another will NECESSARILY write the needed or a FRESHER dquot ondisk.

Yeah, I agree with Andrew. What they did is *almost* safe for ext4. The only moment when it is not safe is when someone calls mark_dquot_dirty() outside of a scope of a transaction which happens when doing Q_SETQUOTA quotactl.

Another things which is subtle with Andrew's approach is that process modifying quota information can return and stop its handle before quota data gets copied to transaction buffer. This does not currently create any real problem since nobody is relying on that however it relies on intimate details of JBD2 transaction machinery and that could bite us in the future.

Honza

Quoted 31 lines
Quoted 29 lines
Quoted 4 lines

This change mean if this dquot is dirty we skip, this
won't work because in this way, quota update is only kept in vfs dquota memory and newer update is not wrote to journal file and not wrapped into transaction too.

That's not true.

As I explained earlier, having DQ_MOD_B set at this point means another thread is going to write dquot but hasn't yet started doing so. This thread does not care whether it updates the ondisk dquot with its own data or with fresher data which came from another thread. In-core dquot has no indication of whose data in contains.

As I also explained earlier, the update cannot happen in the context of another transaction because thread A which sees DQ_MOD_B set and thread B which is running dquot_commit() both have journal handles to the same transaction. There's only one running transaction at a time and thread B does not switch to another transaction.

Please read the code carefully.

Quoted 8 lines

This is not what journal quota means to do.

Thanks,
Shilong

Quoted 2 lines

Thank you,
Andrew

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
reviewer
#16 lore raw
In reply to Wang Shilong

Hello!

Quoted 41 lines · Wang Shilong

We DDN is investigating the same issue!

Some comments comes:

Quoted 22 lines · Andrew Perepechko
Quoted 9 lines · Jan Kara
Quoted 2 lines

Hi Andrew,

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {

Review

Quoted 2 lines
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Here is some ideas that i have thought:

1) switch dqio_mutex to a read/write lock, especially, i think most of time journal quota updates is in-place update, that means we don't need change quota tree in memory, firstly try read lock, retry with write lock if there is real tree change.

Quoted 7 lines

2)another is similar idea of Andrew's walkaround, but we need make correct fix, maintain dirty list for per transaction, and gurantee quota updates are flushed when commit transaction, this might be complex, i am not very familiar with JBD2 codes.

It will be really nice if we could fix this regression, as we see 20% performace regression.

So I have couple of patches:

1) I convert dqio_mutex do rw semaphore and use it in exclusive mode only when quota tree is going to change. We also use dq_lock to serialize writes of dquot - you cannot have two writes happening in parallel as that could result in stale data being on disk. This patch brings benefit when there are multiple users - now they don't contend on common lock. It shows advantage in my testing so I plan to merge these patches. When the contention is on a structure for single user this change however doesn't bring much (the performance change is in statistical noise in my testing).

2) I have patches to remove some contention on dq_list_lock by not using dirty list for tracking dquots in ext4 (and thus avoid dq_list_lock completely in quota modification path). This does not bring measurable benefit in my testing even on ramdisk but lockstat data for dq_list_lock looks much better after this - it seems lock contention just shifted to dq_data_lock - I'll try to address that as well and see whether I'll be able to measure some advantage.

3) I have patches to convert dquot dirty bit to sequence counter so that in commit_dqblk() we can check whether dquot state we wanted to write is already on disk. Note that this is different from Andrew's approach in that we do wait for dquot to be actually written before returning. We just don't repeat the write unnecessarily. However this didn't bring any measurable benefit in my testing so unless I'll be able to confirm it benefits some workloads I won't merge this change.

If you can experiment with your workloads, I can send you patches. I'd be keen on having some performance data from real setups...

Honza

Quoted 11 lines

Thanks,
Shilong

Quoted 7 lines
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#17 lore raw

Hello Jan,

Please send me patches, we could test and response you!

Thanks,
Shilong

Quoted 98 lines · Jan Kara

Hello!

Quoted 41 lines · Wang Shilong

We DDN is investigating the same issue!

Some comments comes:

Quoted 22 lines · Andrew Perepechko
Quoted 9 lines · Jan Kara
Quoted 2 lines

Hi Andrew,

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {

Review

Quoted 2 lines
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Here is some ideas that i have thought:

1) switch dqio_mutex to a read/write lock, especially, i think most of time journal quota updates is in-place update, that means we don't need change quota tree in memory, firstly try read lock, retry with write lock if there is real tree change.

Quoted 7 lines

2)another is similar idea of Andrew's walkaround, but we need make correct fix, maintain dirty list for per transaction, and gurantee quota updates are flushed when commit transaction, this might be complex, i am not very familiar with JBD2 codes.

It will be really nice if we could fix this regression, as we see 20% performace regression.

So I have couple of patches:

1) I convert dqio_mutex do rw semaphore and use it in exclusive mode only when quota tree is going to change. We also use dq_lock to serialize writes of dquot - you cannot have two writes happening in parallel as that could result in stale data being on disk. This patch brings benefit when there are multiple users - now they don't contend on common lock. It shows advantage in my testing so I plan to merge these patches. When the contention is on a structure for single user this change however doesn't bring much (the performance change is in statistical noise in my testing).

2) I have patches to remove some contention on dq_list_lock by not using dirty list for tracking dquots in ext4 (and thus avoid dq_list_lock completely in quota modification path). This does not bring measurable benefit in my testing even on ramdisk but lockstat data for dq_list_lock looks much better after this - it seems lock contention just shifted to dq_data_lock - I'll try to address that as well and see whether I'll be able to measure some advantage.

3) I have patches to convert dquot dirty bit to sequence counter so that in commit_dqblk() we can check whether dquot state we wanted to write is already on disk. Note that this is different from Andrew's approach in that we do wait for dquot to be actually written before returning. We just don't repeat the write unnecessarily. However this didn't bring any measurable benefit in my testing so unless I'll be able to confirm it benefits some workloads I won't merge this change.

If you can experiment with your workloads, I can send you patches. I'd be keen on having some performance data from real setups...

Honza

Quoted 11 lines

Thanks,
Shilong

Quoted 7 lines
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

--
Jan Kara <[email protected]>
SUSE Labs, CR

reviewer
#18 lore raw

Hi,

Quoted 1 line · Wang Shilong

Please send me patches, we could test and response you!

So I finally have something which isn't obviously wrong (it survives basic testing and gives me improvements for some workloads). I have pushed out the patches to:

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git quota_scaling

I'd be happy if you can share your results with my patches. I have not yet figured out a safe way to reduce the contention on dq_lock during update of on-disk structure when lot of processes bang single dquot. I have experimental patch but it didn't bring any benefit in my testing - I'll rebase it on top of other patches I have send it to you for some testing.

Honza

Quoted 99 lines · Jan Kara
Quoted 98 lines

Hello!

Quoted 41 lines · Wang Shilong

We DDN is investigating the same issue!

Some comments comes:

Quoted 22 lines · Andrew Perepechko
Quoted 9 lines · Jan Kara
Quoted 2 lines

Hi Andrew,

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {

Review

Quoted 2 lines
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Here is some ideas that i have thought:

1) switch dqio_mutex to a read/write lock, especially, i think most of time journal quota updates is in-place update, that means we don't need change quota tree in memory, firstly try read lock, retry with write lock if there is real tree change.

Quoted 7 lines

2)another is similar idea of Andrew's walkaround, but we need make correct fix, maintain dirty list for per transaction, and gurantee quota updates are flushed when commit transaction, this might be complex, i am not very familiar with JBD2 codes.

It will be really nice if we could fix this regression, as we see 20% performace regression.

So I have couple of patches:

1) I convert dqio_mutex do rw semaphore and use it in exclusive mode only when quota tree is going to change. We also use dq_lock to serialize writes of dquot - you cannot have two writes happening in parallel as that could result in stale data being on disk. This patch brings benefit when there are multiple users - now they don't contend on common lock. It shows advantage in my testing so I plan to merge these patches. When the contention is on a structure for single user this change however doesn't bring much (the performance change is in statistical noise in my testing).

2) I have patches to remove some contention on dq_list_lock by not using dirty list for tracking dquots in ext4 (and thus avoid dq_list_lock completely in quota modification path). This does not bring measurable benefit in my testing even on ramdisk but lockstat data for dq_list_lock looks much better after this - it seems lock contention just shifted to dq_data_lock - I'll try to address that as well and see whether I'll be able to measure some advantage.

3) I have patches to convert dquot dirty bit to sequence counter so that in commit_dqblk() we can check whether dquot state we wanted to write is already on disk. Note that this is different from Andrew's approach in that we do wait for dquot to be actually written before returning. We just don't repeat the write unnecessarily. However this didn't bring any measurable benefit in my testing so unless I'll be able to confirm it benefits some workloads I won't merge this change.

If you can experiment with your workloads, I can send you patches. I'd be keen on having some performance data from real setups...

Honza

Quoted 11 lines

Thanks,
Shilong

Quoted 7 lines
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

--
Jan Kara <[email protected]>
SUSE Labs, CR

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#19 lore raw

Hello Jan,

We have tested your patches, in generally, it helped in our case. Noticed,
our test case is only one user with many process create/remove file.

4.13.0-rc3 without any patches
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0	93,068 	         296,028 	            86,860 	285,131 	           85,199 	        189,653 
1	79,501 	         280,921 	            91,079 	277,349 	          186,279 	170,982 
2	79,932 	         299,750 	            90,246 	274,457 	           133,922 	191,677 
3	80,146 	         297,525 	            86,416 	272,160 	           192,354 	198,869 

4.13.0-rc3/w Jan Kara patch
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink
0 73,057 311,217 74,898 286,120 81,217 288,138 ops/per second

1	78,872             312,471 	         76,470 	          277,033 	 77,014 	        288,057 
2	79,170             291,440 	         76,174              283,525 	 73,686            283,526 
3	79,941             309,168            78,493              277,331          78,751            281,377 

4.13.0-rc3/with https://patchwork.ozlabs.org/patch/799014/
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink
0 100,319 322,746 87,480 302,579 84,569 218,969
1 728,424 299,808 312,766 293,471 219,198 199,389
2 729,410 300,930 315,590 289,664 218,283 197,871
3 727,555 298,797 316,837 289,108 213,095 213,458

4.13.0-rc3/w https://patchwork.ozlabs.org/patch/799014/ + Jan Kara patch
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0	100,312 	         324,871 	         87,076 	         267,303 	          86,258 	        288,137 
1	707,524 	         298,892           361,963           252,493            421,919 	282,492 
2	707,792            298,162           363,450           264,923            397,723 	283,675 
3	707,420            302,552 	         354,013 	         266,638 	          421,537 	281,763 

In conclusion, your patches helped a lot for our testing, noticed, please ignored test0 running for creation, the first time testing will loaded inode cache in memory, we used test1-3 to compare.

With extra patch applied, your patches improved File creation(quota+project) 2X, File unlink 1.5X.

Thanks,
Shilong

________________________________________
From: Jan Kara [[email protected]]
Sent: Wednesday, August 09, 2017 0:06
To: Wang Shilong
Cc: Jan Kara; Andrew Perepechko; Shuichi Ihara; Wang Shilong; Li Xi; Ext4 Developers List; [email protected]
Subject: Re: quota: dqio_mutex design

Hi,

Quoted 1 line · Wang Shilong

Please send me patches, we could test and response you!

So I finally have something which isn't obviously wrong (it survives basic testing and gives me improvements for some workloads). I have pushed out the patches to:

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git quota_scaling

I'd be happy if you can share your results with my patches. I have not yet figured out a safe way to reduce the contention on dq_lock during update of on-disk structure when lot of processes bang single dquot. I have experimental patch but it didn't bring any benefit in my testing - I'll rebase it on top of other patches I have send it to you for some testing.

Honza

Quoted 92 lines · Jan Kara
Quoted 91 lines

Hello!

Quoted 41 lines · Wang Shilong

We DDN is investigating the same issue!

Some comments comes:

Quoted 22 lines · Andrew Perepechko
Quoted 9 lines · Jan Kara
Quoted 2 lines

Hi Andrew,

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {

Review

Quoted 2 lines
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Here is some ideas that i have thought:

1) switch dqio_mutex to a read/write lock, especially, i think most of time journal quota updates is in-place update, that means we don't need change quota tree in memory, firstly try read lock, retry with write lock if there is real tree change.

Quoted 7 lines

2)another is similar idea of Andrew's walkaround, but we need make correct fix, maintain dirty list for per transaction, and gurantee quota updates are flushed when commit transaction, this might be complex, i am not very familiar with JBD2 codes.

It will be really nice if we could fix this regression, as we see 20% performace regression.

So I have couple of patches:

1) I convert dqio_mutex do rw semaphore and use it in exclusive mode only when quota tree is going to change. We also use dq_lock to serialize writes of dquot - you cannot have two writes happening in parallel as that could result in stale data being on disk. This patch brings benefit when there are multiple users - now they don't contend on common lock. It shows advantage in my testing so I plan to merge these patches. When the contention is on a structure for single user this change however doesn't bring much (the performance change is in statistical noise in my testing).

2) I have patches to remove some contention on dq_list_lock by not using dirty list for tracking dquots in ext4 (and thus avoid dq_list_lock completely in quota modification path). This does not bring measurable benefit in my testing even on ramdisk but lockstat data for dq_list_lock looks much better after this - it seems lock contention just shifted to dq_data_lock - I'll try to address that as well and see whether I'll be able to measure some advantage.

3) I have patches to convert dquot dirty bit to sequence counter so that in commit_dqblk() we can check whether dquot state we wanted to write is already on disk. Note that this is different from Andrew's approach in that we do wait for dquot to be actually written before returning. We just don't repeat the write unnecessarily. However this didn't bring any measurable benefit in my testing so unless I'll be able to confirm it benefits some workloads I won't merge this change.

If you can experiment with your workloads, I can send you patches. I'd be keen on having some performance data from real setups...

Honza

Quoted 7 lines

Thanks,
Shilong

Quoted 3 lines
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

--
Jan Kara <[email protected]>
SUSE Labs, CR

--
Jan Kara <[email protected]>
SUSE Labs, CR

#20 lore raw

sorry, format did not look fine, please use attachment.

Quoted 176 lines · Wang Shilong

Hello Jan,

We have tested your patches, in generally, it helped in our case. Noticed,
our test case is only one user with many process create/remove file.

4.13.0-rc3 without any patches
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       93,068           296,028                    86,860      285,131      85,199               189,653
1       79,501           280,921                    91,079      277,349                   186,279       170,982
2       79,932           299,750                    90,246      274,457                    133,922      191,677
3       80,146           297,525                    86,416      272,160                    192,354      198,869

4.13.0-rc3/w Jan Kara patch
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       73,057          311,217                  74,898                   286,120        81,217                 288,138  ops/per second
1       78,872             312,471               76,470                   277,033        77,014                 288,057
2       79,170             291,440               76,174              283,525     73,686            283,526
3       79,941             309,168            78,493              277,331          78,751            281,377

4.13.0-rc3/with https://patchwork.ozlabs.org/patch/799014/
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       100,319                 322,746                  87,480                  302,579                 84,569                 218,969
1       728,424                 299,808            312,766           293,471           219,198          199,389
2       729,410           300,930            315,590           289,664           218,283          197,871
3       727,555           298,797                316,837                 289,108           213,095          213,458

4.13.0-rc3/w https://patchwork.ozlabs.org/patch/799014/ + Jan Kara patch
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       100,312                  324,871                 87,076                  267,303                  86,258                288,137
1       707,524                  298,892           361,963           252,493            421,919         282,492
2       707,792            298,162           363,450           264,923            397,723       283,675
3       707,420            302,552               354,013                 266,638                  421,537       281,763

In conclusion, your patches helped a lot for our testing, noticed, please ignored test0 running for creation, the first time testing will loaded inode cache in memory, we used test1-3 to compare.

With extra patch applied, your patches improved File creation(quota+project) 2X, File unlink 1.5X.

Thanks,
Shilong

________________________________________
From: Jan Kara [[email protected]]
Sent: Wednesday, August 09, 2017 0:06
To: Wang Shilong
Cc: Jan Kara; Andrew Perepechko; Shuichi Ihara; Wang Shilong; Li Xi; Ext4 Developers List; [email protected]
Subject: Re: quota: dqio_mutex design

Hi,

Quoted 1 line · Wang Shilong

Please send me patches, we could test and response you!

So I finally have something which isn't obviously wrong (it survives basic testing and gives me improvements for some workloads). I have pushed out the patches to:

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git quota_scaling

I'd be happy if you can share your results with my patches. I have not yet figured out a safe way to reduce the contention on dq_lock during update of on-disk structure when lot of processes bang single dquot. I have experimental patch but it didn't bring any benefit in my testing - I'll rebase it on top of other patches I have send it to you for some testing.

Honza

Quoted 92 lines · Jan Kara
Quoted 91 lines

Hello!

Quoted 41 lines · Wang Shilong

We DDN is investigating the same issue!

Some comments comes:

Quoted 22 lines · Andrew Perepechko
Quoted 9 lines · Jan Kara
Quoted 2 lines

Hi Andrew,

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {

Review

Quoted 2 lines
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Here is some ideas that i have thought:

1) switch dqio_mutex to a read/write lock, especially, i think most of time journal quota updates is in-place update, that means we don't need change quota tree in memory, firstly try read lock, retry with write lock if there is real tree change.

Quoted 7 lines

2)another is similar idea of Andrew's walkaround, but we need make correct fix, maintain dirty list for per transaction, and gurantee quota updates are flushed when commit transaction, this might be complex, i am not very familiar with JBD2 codes.

It will be really nice if we could fix this regression, as we see 20% performace regression.

So I have couple of patches:

1) I convert dqio_mutex do rw semaphore and use it in exclusive mode only when quota tree is going to change. We also use dq_lock to serialize writes of dquot - you cannot have two writes happening in parallel as that could result in stale data being on disk. This patch brings benefit when there are multiple users - now they don't contend on common lock. It shows advantage in my testing so I plan to merge these patches. When the contention is on a structure for single user this change however doesn't bring much (the performance change is in statistical noise in my testing).

2) I have patches to remove some contention on dq_list_lock by not using dirty list for tracking dquots in ext4 (and thus avoid dq_list_lock completely in quota modification path). This does not bring measurable benefit in my testing even on ramdisk but lockstat data for dq_list_lock looks much better after this - it seems lock contention just shifted to dq_data_lock - I'll try to address that as well and see whether I'll be able to measure some advantage.

3) I have patches to convert dquot dirty bit to sequence counter so that in commit_dqblk() we can check whether dquot state we wanted to write is already on disk. Note that this is different from Andrew's approach in that we do wait for dquot to be actually written before returning. We just don't repeat the write unnecessarily. However this didn't bring any measurable benefit in my testing so unless I'll be able to confirm it benefits some workloads I won't merge this change.

If you can experiment with your workloads, I can send you patches. I'd be keen on having some performance data from real setups...

Honza

Quoted 7 lines

Thanks,
Shilong

Quoted 3 lines
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

--
Jan Kara <[email protected]>
SUSE Labs, CR

--
Jan Kara <[email protected]>
SUSE Labs, CR

#21 lore raw
In reply to Wang Shilong

Txt format attched.

BTW, Jan, it will be cool if you could point which patch help a lot for our test case, since there are a lot of patches there, we want to port some of patches to RHEL7.

Thanks,
Shilong

Quoted 170 lines · Wang Shilong

Hello Jan,

We have tested your patches, in generally, it helped in our case. Noticed,
our test case is only one user with many process create/remove file.

4.13.0-rc3 without any patches
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       93,068           296,028                    86,860      285,131                    85,199               189,653
1       79,501           280,921                    91,079      277,349                   186,279       170,982
2       79,932           299,750                    90,246      274,457                    133,922      191,677
3       80,146           297,525                    86,416      272,160                    192,354      198,869

4.13.0-rc3/w Jan Kara patch
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       73,057          311,217                  74,898                   286,120        81,217                 288,138  ops/per second
1       78,872             312,471               76,470                   277,033        77,014                 288,057
2       79,170             291,440               76,174              283,525     73,686            283,526
3       79,941             309,168            78,493              277,331          78,751            281,377

4.13.0-rc3/with https://patchwork.ozlabs.org/patch/799014/
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       100,319                 322,746                  87,480                  302,579                 84,569                 218,969
1       728,424                 299,808            312,766           293,471           219,198          199,389
2       729,410           300,930            315,590           289,664           218,283          197,871
3       727,555           298,797                316,837                 289,108           213,095          213,458

4.13.0-rc3/w https://patchwork.ozlabs.org/patch/799014/ + Jan Kara patch
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       100,312                  324,871                 87,076                  267,303                  86,258                288,137
1       707,524                  298,892           361,963           252,493            421,919         282,492
2       707,792            298,162           363,450           264,923            397,723       283,675
3       707,420            302,552               354,013                 266,638                  421,537       281,763

In conclusion, your patches helped a lot for our testing, noticed, please ignored test0 running for creation, the first time testing will loaded inode cache in memory, we used test1-3 to compare.

With extra patch applied, your patches improved File creation(quota+project) 2X, File unlink 1.5X.

Thanks,
Shilong

________________________________________
From: Jan Kara [[email protected]]
Sent: Wednesday, August 09, 2017 0:06
To: Wang Shilong
Cc: Jan Kara; Andrew Perepechko; Shuichi Ihara; Wang Shilong; Li Xi; Ext4 Developers List; [email protected]
Subject: Re: quota: dqio_mutex design

Hi,

Quoted 1 line · Wang Shilong

Please send me patches, we could test and response you!

So I finally have something which isn't obviously wrong (it survives basic testing and gives me improvements for some workloads). I have pushed out the patches to:

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git quota_scaling

I'd be happy if you can share your results with my patches. I have not yet figured out a safe way to reduce the contention on dq_lock during update of on-disk structure when lot of processes bang single dquot. I have experimental patch but it didn't bring any benefit in my testing - I'll rebase it on top of other patches I have send it to you for some testing.

Honza

Quoted 96 lines · Jan Kara
Quoted 95 lines

Hello!

Quoted 41 lines · Wang Shilong

We DDN is investigating the same issue!

Some comments comes:

Quoted 22 lines · Andrew Perepechko
Quoted 9 lines · Jan Kara
Quoted 2 lines

Hi Andrew,

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {

Review

Quoted 2 lines
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Here is some ideas that i have thought:

1) switch dqio_mutex to a read/write lock, especially, i think most of time journal quota updates is in-place update, that means we don't need change quota tree in memory, firstly try read lock, retry with write lock if there is real tree change.

Quoted 7 lines

2)another is similar idea of Andrew's walkaround, but we need make correct fix, maintain dirty list for per transaction, and gurantee quota updates are flushed when commit transaction, this might be complex, i am not very familiar with JBD2 codes.

It will be really nice if we could fix this regression, as we see 20% performace regression.

So I have couple of patches:

1) I convert dqio_mutex do rw semaphore and use it in exclusive mode only when quota tree is going to change. We also use dq_lock to serialize writes of dquot - you cannot have two writes happening in parallel as that could result in stale data being on disk. This patch brings benefit when there are multiple users - now they don't contend on common lock. It shows advantage in my testing so I plan to merge these patches. When the contention is on a structure for single user this change however doesn't bring much (the performance change is in statistical noise in my testing).

2) I have patches to remove some contention on dq_list_lock by not using dirty list for tracking dquots in ext4 (and thus avoid dq_list_lock completely in quota modification path). This does not bring measurable benefit in my testing even on ramdisk but lockstat data for dq_list_lock looks much better after this - it seems lock contention just shifted to dq_data_lock - I'll try to address that as well and see whether I'll be able to measure some advantage.

3) I have patches to convert dquot dirty bit to sequence counter so that in commit_dqblk() we can check whether dquot state we wanted to write is already on disk. Note that this is different from Andrew's approach in that we do wait for dquot to be actually written before returning. We just don't repeat the write unnecessarily. However this didn't bring any measurable benefit in my testing so unless I'll be able to confirm it benefits some workloads I won't merge this change.

If you can experiment with your workloads, I can send you patches. I'd be keen on having some performance data from real setups...

Honza

Quoted 11 lines

Thanks,
Shilong

Quoted 7 lines
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

--
Jan Kara <[email protected]>
SUSE Labs, CR

--
Jan Kara <[email protected]>
SUSE Labs, CR

4.13.0-rc3 without any patches
no Quota -O quota -O quota,project
creation unlink creation unlink creation unlink

0   93,068  296,028      86,860  285,131      85,199    189,653     ops/per second
1   79,501  280,921      91,079  277,349     186,279    170,982
2   79,932  299,750      90,246  274,457     133,922    191,677
3   80,146  297,525      86,416  272,160     192,354    198,869

Jan Kara branch (quota_scaling)
no Quota -O quota -O quota,project

  creation   unlink      creation   unlink    creation  unlink
0   73,057  311,217      74,898  286,120      81,217    288,138
1   78,872  312,471      76,470  277,033      77,014    288,057
2   79,170  291,440      76,174  283,525      73,686    283,526
3   79,941  309,168      78,493  277,331      78,751    281,377

4.13.0-rc3 with v5 patch https://patchwork.ozlabs.org/patch/799014/
no Quota -O quota -O quota,project

  creation   unlink     creation   unlink    creation   unlink
0  100,319  322,746     87,480   302,579      84,569    218,969
1  728,424  299,808     312,766  293,471     219,198    199,389
2  729,410  300,930     315,590  289,664     218,283    197,871
3  727,555  298,797     316,837  289,108     213,095    213,458

Jan Kara branch (quota_scaling) with v5 patch https://patchwork.ozlabs.org/patch/799014/
no Quota -O quota -O quota,project
creation unlink creation unlink creation unlink

0  100,312  324,871      87,076  267,303      86,258    288,137
1  707,524  298,892     361,963  252,493     421,919    282,492
2  707,792  298,162     363,450  264,923     397,723    283,675
3  707,420  302,552     354,013  266,638     421,537    281,763
reviewer
#22 lore raw

Hello,

Quoted 5 lines · Wang Shilong

Txt format attched.

BTW, Jan, it will be cool if you could point which patch help a lot for our test case, since there are a lot of patches there, we want to port some of patches to RHEL7.

Thanks for the test results! They are really interesting. Do you have any explanation why without any patches the '-O quota,project' runs for 'File Creation' are faster than runs without quota or any other runs in the test?

WRT which patches helped I don't have a good subset for you. In my testing each patch helped a bit. I expect in your setup the conversion of dqio_sem to rwsem and then to use dq_lock might not have that big impact. So you might try backporting patches from "quota: Fix possible corruption of dqi_flags" onward.

Honza

Quoted 209 lines · Wang Shilong
Quoted 170 lines

Hello Jan,

We have tested your patches, in generally, it helped in our case. Noticed,
our test case is only one user with many process create/remove file.

4.13.0-rc3 without any patches
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       93,068           296,028                    86,860      285,131                    85,199               189,653
1       79,501           280,921                    91,079      277,349                   186,279       170,982
2       79,932           299,750                    90,246      274,457                    133,922      191,677
3       80,146           297,525                    86,416      272,160                    192,354      198,869

4.13.0-rc3/w Jan Kara patch
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       73,057          311,217                  74,898                   286,120        81,217                 288,138  ops/per second
1       78,872             312,471               76,470                   277,033        77,014                 288,057
2       79,170             291,440               76,174              283,525     73,686            283,526
3       79,941             309,168            78,493              277,331          78,751            281,377

4.13.0-rc3/with https://patchwork.ozlabs.org/patch/799014/
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       100,319                 322,746                  87,480                  302,579                 84,569                 218,969
1       728,424                 299,808            312,766           293,471           219,198          199,389
2       729,410           300,930            315,590           289,664           218,283          197,871
3       727,555           298,797                316,837                 289,108           213,095          213,458

4.13.0-rc3/w https://patchwork.ozlabs.org/patch/799014/ + Jan Kara patch
no Quota -O quota' -O quota, project'
File Creation File Unlink File Creation File Unlink File Creation File Unlink

0       100,312                  324,871                 87,076                  267,303                  86,258                288,137
1       707,524                  298,892           361,963           252,493            421,919         282,492
2       707,792            298,162           363,450           264,923            397,723       283,675
3       707,420            302,552               354,013                 266,638                  421,537       281,763

In conclusion, your patches helped a lot for our testing, noticed, please ignored test0 running for creation, the first time testing will loaded inode cache in memory, we used test1-3 to compare.

With extra patch applied, your patches improved File creation(quota+project) 2X, File unlink 1.5X.

Thanks,
Shilong

________________________________________
From: Jan Kara [[email protected]]
Sent: Wednesday, August 09, 2017 0:06
To: Wang Shilong
Cc: Jan Kara; Andrew Perepechko; Shuichi Ihara; Wang Shilong; Li Xi; Ext4 Developers List; [email protected]
Subject: Re: quota: dqio_mutex design

Hi,

Quoted 1 line · Wang Shilong

Please send me patches, we could test and response you!

So I finally have something which isn't obviously wrong (it survives basic testing and gives me improvements for some workloads). I have pushed out the patches to:

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git quota_scaling

I'd be happy if you can share your results with my patches. I have not yet figured out a safe way to reduce the contention on dq_lock during update of on-disk structure when lot of processes bang single dquot. I have experimental patch but it didn't bring any benefit in my testing - I'll rebase it on top of other patches I have send it to you for some testing.

Honza

Quoted 96 lines · Jan Kara
Quoted 95 lines

Hello!

Quoted 41 lines · Wang Shilong

We DDN is investigating the same issue!

Some comments comes:

Quoted 22 lines · Andrew Perepechko
Quoted 9 lines · Jan Kara
Quoted 2 lines

Hi Andrew,

I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you?

Honza

Hi Jan!

My setup was with a single user. Unfortunately, it may take some time before I can try a patched kernel other than RHEL6 or RHEL7 with the same test, we have a lot of dependencies on these kernels.

The actual test we ran was mdtest.

By the way, we had 15+% performance improvement in creates from the change that was discussed earlier in this thread:

           EXT4_SB(dquot->dq_sb)->s_qf_names[GRPQUOTA]) {

Review

Quoted 2 lines
(quoted diff) +2 -0
+              if (test_bit(DQ_MOD_B, &dquot->dq_flags))
+                       return 0;

I don't think this is right, as far as i understand, journal quota need go together with quota space change update inside same transaction, this will break consistency if power off or RO happen.

Here is some ideas that i have thought:

1) switch dqio_mutex to a read/write lock, especially, i think most of time journal quota updates is in-place update, that means we don't need change quota tree in memory, firstly try read lock, retry with write lock if there is real tree change.

Quoted 7 lines

2)another is similar idea of Andrew's walkaround, but we need make correct fix, maintain dirty list for per transaction, and gurantee quota updates are flushed when commit transaction, this might be complex, i am not very familiar with JBD2 codes.

It will be really nice if we could fix this regression, as we see 20% performace regression.

So I have couple of patches:

1) I convert dqio_mutex do rw semaphore and use it in exclusive mode only when quota tree is going to change. We also use dq_lock to serialize writes of dquot - you cannot have two writes happening in parallel as that could result in stale data being on disk. This patch brings benefit when there are multiple users - now they don't contend on common lock. It shows advantage in my testing so I plan to merge these patches. When the contention is on a structure for single user this change however doesn't bring much (the performance change is in statistical noise in my testing).

2) I have patches to remove some contention on dq_list_lock by not using dirty list for tracking dquots in ext4 (and thus avoid dq_list_lock completely in quota modification path). This does not bring measurable benefit in my testing even on ramdisk but lockstat data for dq_list_lock looks much better after this - it seems lock contention just shifted to dq_data_lock - I'll try to address that as well and see whether I'll be able to measure some advantage.

3) I have patches to convert dquot dirty bit to sequence counter so that in commit_dqblk() we can check whether dquot state we wanted to write is already on disk. Note that this is different from Andrew's approach in that we do wait for dquot to be actually written before returning. We just don't repeat the write unnecessarily. However this didn't bring any measurable benefit in my testing so unless I'll be able to confirm it benefits some workloads I won't merge this change.

If you can experiment with your workloads, I can send you patches. I'd be keen on having some performance data from real setups...

Honza

Quoted 11 lines

Thanks,
Shilong

Quoted 7 lines
               dquot_mark_dquot_dirty(dquot);
               return ext4_write_dquot(dquot);

The idea was that if we know that some thread is somewhere between mark_dirty and clear_dirty, then we can avoid blocking on dqio_mutex, since that thread will update the ondisk dquot for us.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

--
Jan Kara <[email protected]>
SUSE Labs, CR

--
Jan Kara <[email protected]>
SUSE Labs, CR

4.13.0-rc3 without any patches
no Quota -O quota -O quota,project
creation unlink creation unlink creation unlink

0   93,068  296,028      86,860  285,131      85,199    189,653     ops/per second
1   79,501  280,921      91,079  277,349     186,279    170,982
2   79,932  299,750      90,246  274,457     133,922    191,677
3   80,146  297,525      86,416  272,160     192,354    198,869

Jan Kara branch (quota_scaling)
no Quota -O quota -O quota,project

  creation   unlink      creation   unlink    creation  unlink
0   73,057  311,217      74,898  286,120      81,217    288,138
1   78,872  312,471      76,470  277,033      77,014    288,057
2   79,170  291,440      76,174  283,525      73,686    283,526
3   79,941  309,168      78,493  277,331      78,751    281,377

4.13.0-rc3 with v5 patch https://patchwork.ozlabs.org/patch/799014/
no Quota -O quota -O quota,project

  creation   unlink     creation   unlink    creation   unlink
0  100,319  322,746     87,480   302,579      84,569    218,969
1  728,424  299,808     312,766  293,471     219,198    199,389
2  729,410  300,930     315,590  289,664     218,283    197,871
3  727,555  298,797     316,837  289,108     213,095    213,458

Jan Kara branch (quota_scaling) with v5 patch https://patchwork.ozlabs.org/patch/799014/
no Quota -O quota -O quota,project
creation unlink creation unlink creation unlink

0  100,312  324,871      87,076  267,303      86,258    288,137
1  707,524  298,892     361,963  252,493     421,919    282,492
2  707,792  298,162     363,450  264,923     397,723    283,675
3  707,420  302,552     354,013  266,638     421,537    281,763
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR