Skip to contentSkip to conversation

btrfs: add the beginning of async discard, discard workqueue

PATCH Nick Desaulniers 3 participants 7 messages Nov 22, 2019 – Nov 26, 2019 1 message not in the archive Btrfs file system

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 201911220351.HPI9gxNo%[email protected]

#1 lore raw

Hi Dennis,
Below is a 0day bot report from a build w/ Clang. Warning looks legit, can you please take a look?

Quoted 79 lines · kbuild test robot

CC: kbuild-all(a)lists.01.org
In-Reply-To: <63d3257efe1158a6fbbd7abe865cd9250b494438.1574282259.git.dennis@kernel.org>
References: <63d3257efe1158a6fbbd7abe865cd9250b494438.1574282259.git.dennis@kernel.org>
TO: Dennis Zhou <[email protected]>
CC: David Sterba <[email protected]>, Chris Mason <[email protected]>, Josef Bacik <[email protected]>, Omar Sandoval <[email protected]>
CC: kernel-team(a)fb.com, linux-btrfs(a)vger.kernel.org, Dennis Zhou <[email protected]>

Hi Dennis,

I love your patch! Perhaps something to improve:

[auto build test WARNING on kdave/for-next]
[also build test WARNING on next-20191121]
[cannot apply to v5.4-rc8]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Dennis-Zhou/btrfs-async-discard-support/20191121-230429
base:   https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-next
config: arm64-defconfig (attached as .config)
compiler: clang version 10.0.0 (git://gitmirror/llvm_project cf823ce4ad9d04c69b7c29d236f7b14c875111c2)

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=arm64

If you fix the issue, kindly add following tag

All warnings (new ones prefixed by >>):

Quoted 1 line
Quoted 1 line

fs/btrfs/free-space-cache.c:3238:6: warning: variable 'trim_state' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]

           if (!ret) {
               ^~~~
   fs/btrfs/free-space-cache.c:3251:53: note: uninitialized use occurs here
           __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
                                                              ^~~~~~~~~~
   fs/btrfs/free-space-cache.c:3238:2: note: remove the 'if' if its condition is always true
           if (!ret) {
           ^~~~~~~~~~
   fs/btrfs/free-space-cache.c:3224:2: note: variable 'trim_state' is declared here
           enum btrfs_trim_state trim_state;
           ^

1 warning generated.

vim +3238 fs/btrfs/free-space-cache.c

3210
3211 static int do_trimming(struct btrfs_block_group *block_group,

  3212                         u64 *total_trimmed, u64 start, u64 bytes,
  3213                         u64 reserved_start, u64 reserved_bytes,
  3214                         enum btrfs_trim_state reserved_trim_state,
  3215                         struct btrfs_trim_range *trim_entry)
  3216  {
  3217          struct btrfs_space_info *space_info = block_group->space_info;
  3218          struct btrfs_fs_info *fs_info = block_group->fs_info;
  3219          struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
  3220          int ret;
  3221          int update = 0;
  3222          u64 end = start + bytes;
  3223          u64 reserved_end = reserved_start + reserved_bytes;
  3224          enum btrfs_trim_state trim_state;
  3225          u64 trimmed = 0;
  3226
  3227          spin_lock(&space_info->lock);
  3228          spin_lock(&block_group->lock);
  3229          if (!block_group->ro) {
  3230                  block_group->reserved += reserved_bytes;
  3231                  space_info->bytes_reserved += reserved_bytes;
  3232                  update = 1;
  3233          }
  3234          spin_unlock(&block_group->lock);
  3235          spin_unlock(&space_info->lock);

3236
3237 ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);

Quoted 1 line

3238 if (!ret) {

Quoted 13 lines

3239 *total_trimmed += trimmed;
3240 trim_state = BTRFS_TRIM_STATE_TRIMMED;
3241 }
3242
3243 mutex_lock(&ctl->cache_writeout_mutex);
3244 if (reserved_start < start)
3245 __btrfs_add_free_space(fs_info, ctl, reserved_start,
3246 start - reserved_start,
3247 reserved_trim_state);
3248 if (start + bytes < reserved_start + reserved_bytes)
3249 __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
3250 reserved_trim_state);
3251 __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);

^ oops

Quoted 21 lines
  3252          list_del(&trim_entry->list);
  3253          mutex_unlock(&ctl->cache_writeout_mutex);
  3254
  3255          if (update) {
  3256                  spin_lock(&space_info->lock);
  3257                  spin_lock(&block_group->lock);
  3258                  if (block_group->ro)
  3259                          space_info->bytes_readonly += reserved_bytes;
  3260                  block_group->reserved -= reserved_bytes;
  3261                  space_info->bytes_reserved -= reserved_bytes;
  3262                  spin_unlock(&block_group->lock);
  3263                  spin_unlock(&space_info->lock);
  3264          }
  3265
  3266          return ret;
  3267  }
  3268

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

Signature · 2 lines
Thanks,
~Nick Desaulniers
#2 lore raw
Quoted 4 lines · Nick Desaulniers

Hi Dennis,
Below is a 0day bot report from a build w/ Clang. Warning looks legit, can you please take a look?

Ah thanks for this! Yeah that was a miss when I switched from flags -> an enum and didn't update the declaration properly. I'll be sending out a v4 as another fix for arm has some rebase conflicts.

Is there a way to enable so I get these emails directly?

Thanks,
Dennis

Quoted 123 lines · kbuild test robot
Quoted 79 lines

CC: kbuild-all(a)lists.01.org
In-Reply-To: <63d3257efe1158a6fbbd7abe865cd9250b494438.1574282259.git.dennis@kernel.org>
References: <63d3257efe1158a6fbbd7abe865cd9250b494438.1574282259.git.dennis@kernel.org>
TO: Dennis Zhou <[email protected]>
CC: David Sterba <[email protected]>, Chris Mason <[email protected]>, Josef Bacik <[email protected]>, Omar Sandoval <[email protected]>
CC: kernel-team(a)fb.com, linux-btrfs(a)vger.kernel.org, Dennis Zhou <[email protected]>

Hi Dennis,

I love your patch! Perhaps something to improve:

[auto build test WARNING on kdave/for-next]
[also build test WARNING on next-20191121]
[cannot apply to v5.4-rc8]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Dennis-Zhou/btrfs-async-discard-support/20191121-230429
base:   https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-next
config: arm64-defconfig (attached as .config)
compiler: clang version 10.0.0 (git://gitmirror/llvm_project cf823ce4ad9d04c69b7c29d236f7b14c875111c2)

reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=arm64

If you fix the issue, kindly add following tag

All warnings (new ones prefixed by >>):

Quoted 1 line
Quoted 1 line

fs/btrfs/free-space-cache.c:3238:6: warning: variable 'trim_state' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]

           if (!ret) {
               ^~~~
   fs/btrfs/free-space-cache.c:3251:53: note: uninitialized use occurs here
           __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
                                                              ^~~~~~~~~~
   fs/btrfs/free-space-cache.c:3238:2: note: remove the 'if' if its condition is always true
           if (!ret) {
           ^~~~~~~~~~
   fs/btrfs/free-space-cache.c:3224:2: note: variable 'trim_state' is declared here
           enum btrfs_trim_state trim_state;
           ^

1 warning generated.

vim +3238 fs/btrfs/free-space-cache.c

3210
3211 static int do_trimming(struct btrfs_block_group *block_group,

  3212                         u64 *total_trimmed, u64 start, u64 bytes,
  3213                         u64 reserved_start, u64 reserved_bytes,
  3214                         enum btrfs_trim_state reserved_trim_state,
  3215                         struct btrfs_trim_range *trim_entry)
  3216  {
  3217          struct btrfs_space_info *space_info = block_group->space_info;
  3218          struct btrfs_fs_info *fs_info = block_group->fs_info;
  3219          struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
  3220          int ret;
  3221          int update = 0;
  3222          u64 end = start + bytes;
  3223          u64 reserved_end = reserved_start + reserved_bytes;
  3224          enum btrfs_trim_state trim_state;
  3225          u64 trimmed = 0;
  3226
  3227          spin_lock(&space_info->lock);
  3228          spin_lock(&block_group->lock);
  3229          if (!block_group->ro) {
  3230                  block_group->reserved += reserved_bytes;
  3231                  space_info->bytes_reserved += reserved_bytes;
  3232                  update = 1;
  3233          }
  3234          spin_unlock(&block_group->lock);
  3235          spin_unlock(&space_info->lock);

3236
3237 ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);

Quoted 1 line

3238 if (!ret) {

Quoted 13 lines

3239 *total_trimmed += trimmed;
3240 trim_state = BTRFS_TRIM_STATE_TRIMMED;
3241 }
3242
3243 mutex_lock(&ctl->cache_writeout_mutex);
3244 if (reserved_start < start)
3245 __btrfs_add_free_space(fs_info, ctl, reserved_start,
3246 start - reserved_start,
3247 reserved_trim_state);
3248 if (start + bytes < reserved_start + reserved_bytes)
3249 __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
3250 reserved_trim_state);
3251 __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);

^ oops

Quoted 21 lines
  3252          list_del(&trim_entry->list);
  3253          mutex_unlock(&ctl->cache_writeout_mutex);
  3254
  3255          if (update) {
  3256                  spin_lock(&space_info->lock);
  3257                  spin_lock(&block_group->lock);
  3258                  if (block_group->ro)
  3259                          space_info->bytes_readonly += reserved_bytes;
  3260                  block_group->reserved -= reserved_bytes;
  3261                  space_info->bytes_reserved -= reserved_bytes;
  3262                  spin_unlock(&block_group->lock);
  3263                  spin_unlock(&space_info->lock);
  3264          }
  3265
  3266          return ret;
  3267  }
  3268

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

Signature · 2 lines
Thanks,
~Nick Desaulniers
#3 lore raw
Quoted 7 lines · Dennis Zhou
Quoted 4 lines · Nick Desaulniers

Hi Dennis,
Below is a 0day bot report from a build w/ Clang. Warning looks legit, can you please take a look?

Quoted 5 lines

Ah thanks for this! Yeah that was a miss when I switched from flags -> an enum and didn't update the declaration properly. I'll be sending out a v4 as another fix for arm has some rebase conflicts.

Is there a way to enable so I get these emails directly?

+ Rong, Philip

The reports have only been sent to our mailing list where we've been manually triaging them. The issue with enabling them globally was that the script to reproduce the warning still doesn't mention how to build w/ Clang.

In general the reports have been high value (I ignore most reports with -Wimplicit-function-declaration, which is the most frequent as it shows the patch was not compile tested at all).

Rong, Philip, it's been a while since we talked about this last. Is there a general timeline of when these reports will be turned on globally? Even if the directions to reproduce aren't quite right, generally there's enough info in the existing bugs where authors can rewrite their patch without even needing to rebuild with Clang (though having correct directions to reproduce would be nice, we could wait until someone asked for them explicitly).

Signature · 2 lines
Thanks,
~Nick Desaulniers
#4 lore raw
Quoted 12 lines · Nick Desaulniers
Quoted 11 lines · Dennis Zhou
Quoted 4 lines · Nick Desaulniers

Hi Dennis,
Below is a 0day bot report from a build w/ Clang. Warning looks legit, can you please take a look?

Ah thanks for this! Yeah that was a miss when I switched from flags -> an enum and didn't update the declaration properly. I'll be sending out a v4 as another fix for arm has some rebase conflicts.

Quoted 8 lines
Quoted 1 line

Is there a way to enable so I get these emails directly?

+ Rong, Philip

The reports have only been sent to our mailing list where we've been manually triaging them. The issue with enabling them globally was that the script to reproduce the warning still doesn't mention how to build w/ Clang.

Thanks Nick for continuous caring on this. One thing we initially worry is how to avoid duplicated reports to developer, like the one that can be same as gcc's finding. We haven't found a way to effectively handle this.

Quoted 4 lines

In general the reports have been high value (I ignore most reports with -Wimplicit-function-declaration, which is the most frequent as it shows the patch was not compile tested at all).

Do we mean the report with -Wimplicit-function-declaration can be duplicated to gcc, so we can ignore them to avoid duplication to developer?

Quoted 4 lines

Rong, Philip, it's been a while since we talked about this last. Is there a general timeline of when these reports will be turned on globally? Even if the directions to reproduce aren't quite right,

For the timeline, it's not decided due to the duplication concern. We tend to look into next year after other priorities are solved for this year.

Quoted 8 lines

generally there's enough info in the existing bugs where authors can rewrite their patch without even needing to rebuild with Clang (though having correct directions to reproduce would be nice, we could wait until someone asked for them explicitly).

Signature · 2 lines
Thanks,
~Nick Desaulniers
#5 lore raw
Quoted 18 lines · Philip Li
Quoted 16 lines · Nick Desaulniers
Quoted 7 lines · Dennis Zhou
Quoted 4 lines · Nick Desaulniers

Hi Dennis,
Below is a 0day bot report from a build w/ Clang. Warning looks legit, can you please take a look?

Quoted 5 lines

Ah thanks for this! Yeah that was a miss when I switched from flags -> an enum and didn't update the declaration properly. I'll be sending out a v4 as another fix for arm has some rebase conflicts.

Is there a way to enable so I get these emails directly?

+ Rong, Philip

Quoted 8 lines
Quoted 4 lines

The reports have only been sent to our mailing list where we've been manually triaging them. The issue with enabling them globally was that the script to reproduce the warning still doesn't mention how to build w/ Clang.

Thanks Nick for continuous caring on this. One thing we initially worry is how to avoid duplicated reports to developer, like the one that can be same as gcc's finding. We haven't found a way to effectively handle this.

Thanks for maintaining an invaluable tool.

How would the reports be duplicated? Does 0day bot build with GCC, then rebuild with Clang?

Regardless, does it matter? If I make a mistake, and get two build failure emails from 0day bot instead of one, does it matter? Sometimes developers may just get one, as some warnings are unique to each compiler. Maybe it runs the risk of folks ignoring the email if the volume is too much, but do authors generally ignore 0day bot emails? (I'd hope not).

Quoted 7 lines
Quoted 5 lines

In general the reports have been high value (I ignore most reports with -Wimplicit-function-declaration, which is the most frequent as it shows the patch was not compile tested at all).

Do we mean the report with -Wimplicit-function-declaration can be duplicated to gcc, so we can ignore them to avoid duplication to developer?

Many of the warnings GCC has Clang does as well. -Wimplicit-function-declaration is the most common warning I see in triage, which developers would see regardless of toolchain had they compiled first before pushing. It might be interesting to see maybe the intersection or common flags between GCC and Clang, and only email Clang reports for warnings unique to clang? I think CFLAGS can even be passed into make invocations so you could do:

$ make CC=clang KBUILD_CFLAGS=<list of flags common to GCC and Clang;
-Wno-implicit-function-declaration -Wno-...>
such that any resulting warnings were unique to Clang.  I'd expect
such a list to quickly get stale over time though.
Quoted 16 lines
Quoted 5 lines

Rong, Philip, it's been a while since we talked about this last. Is there a general timeline of when these reports will be turned on globally? Even if the directions to reproduce aren't quite right,

For the timeline, it's not decided due to the duplication concern. We tend to look into next year after other priorities are solved for this year.

Quoted 8 lines

generally there's enough info in the existing bugs where authors can rewrite their patch without even needing to rebuild with Clang (though having correct directions to reproduce would be nice, we could wait until someone asked for them explicitly).

--
Thanks,
~Nick Desaulniers

Signature · 2 lines
Thanks,
~Nick Desaulniers
#6 lore raw
Quoted 28 lines · Nick Desaulniers
Quoted 18 lines · Philip Li
Quoted 16 lines · Nick Desaulniers
Quoted 7 lines · Dennis Zhou
Quoted 4 lines · Nick Desaulniers

Hi Dennis,
Below is a 0day bot report from a build w/ Clang. Warning looks legit, can you please take a look?

Quoted 5 lines

Ah thanks for this! Yeah that was a miss when I switched from flags -> an enum and didn't update the declaration properly. I'll be sending out a v4 as another fix for arm has some rebase conflicts.

Is there a way to enable so I get these emails directly?

+ Rong, Philip

Quoted 8 lines
Quoted 4 lines

The reports have only been sent to our mailing list where we've been manually triaging them. The issue with enabling them globally was that the script to reproduce the warning still doesn't mention how to build w/ Clang.

Thanks Nick for continuous caring on this. One thing we initially worry is how to avoid duplicated reports to developer, like the one that can be same as gcc's finding. We haven't found a way to effectively handle this.

Quoted 4 lines

Thanks for maintaining an invaluable tool.

How would the reports be duplicated? Does 0day bot build with GCC, then rebuild with Clang?

no, they are built separately. For duplication, i refer to the issue can be detected by both tool, and gcc reports out already (or clang first).

Quoted 7 lines

Regardless, does it matter? If I make a mistake, and get two build failure emails from 0day bot instead of one, does it matter? Sometimes developers may just get one, as some warnings are unique to each compiler. Maybe it runs the risk of folks ignoring the email if the volume is too much, but do authors generally ignore 0day bot emails? (I'd hope not).

:-) this is a good point, and recently we are working to make the service more stable to generate reports in time.

Quoted 16 lines
Quoted 8 lines
Quoted 6 lines

In general the reports have been high value (I ignore most reports with -Wimplicit-function-declaration, which is the most frequent as it shows the patch was not compile tested at all).

Do we mean the report with -Wimplicit-function-declaration can be duplicated to gcc, so we can ignore them to avoid duplication to developer?

Many of the warnings GCC has Clang does as well. -Wimplicit-function-declaration is the most common warning I see in triage, which developers would see regardless of toolchain had they compiled first before pushing. It might be interesting to see maybe the intersection or common flags between GCC and Clang, and only email Clang reports for warnings unique to clang? I think CFLAGS can even be passed into make invocations so you could do:

Quoted 4 lines
$ make CC=clang KBUILD_CFLAGS=<list of flags common to GCC and Clang;
-Wno-implicit-function-declaration -Wno-...>
such that any resulting warnings were unique to Clang.  I'd expect
such a list to quickly get stale over time though.

thanks for the idea, we will look into this.

Quoted 23 lines
Quoted 17 lines
Quoted 6 lines

Rong, Philip, it's been a while since we talked about this last. Is there a general timeline of when these reports will be turned on globally? Even if the directions to reproduce aren't quite right,

For the timeline, it's not decided due to the duplication concern. We tend to look into next year after other priorities are solved for this year.

Quoted 8 lines

generally there's enough info in the existing bugs where authors can rewrite their patch without even needing to rebuild with Clang (though having correct directions to reproduce would be nice, we could wait until someone asked for them explicitly).

--
Thanks,
~Nick Desaulniers

Signature · 2 lines
Thanks,
~Nick Desaulniers
#7 lore raw
In reply to Nick Desaulniers
Quoted 15 lines · Nick Desaulniers
Quoted 14 lines · Philip Li
Quoted 12 lines · Nick Desaulniers
Quoted 11 lines · Dennis Zhou
Quoted 4 lines · Nick Desaulniers

Hi Dennis,
Below is a 0day bot report from a build w/ Clang. Warning looks legit, can you please take a look?

Ah thanks for this! Yeah that was a miss when I switched from flags -> an enum and didn't update the declaration properly. I'll be sending out a v4 as another fix for arm has some rebase conflicts.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line

Is there a way to enable so I get these emails directly?

+ Rong, Philip

The reports have only been sent to our mailing list where we've been manually triaging them. The issue with enabling them globally was that the script to reproduce the warning still doesn't mention how to build w/ Clang.

Hi Nick, i forgot one question. Is it still expected to use latest clang to build test? Any possibility the issue is related to clang compiler itself?

Thanks

Quoted 59 lines
Quoted 4 lines

Thanks Nick for continuous caring on this. One thing we initially worry is how to avoid duplicated reports to developer, like the one that can be same as gcc's finding. We haven't found a way to effectively handle this.

Thanks for maintaining an invaluable tool.

How would the reports be duplicated? Does 0day bot build with GCC, then rebuild with Clang?

Regardless, does it matter? If I make a mistake, and get two build failure emails from 0day bot instead of one, does it matter? Sometimes developers may just get one, as some warnings are unique to each compiler. Maybe it runs the risk of folks ignoring the email if the volume is too much, but do authors generally ignore 0day bot emails? (I'd hope not).

Quoted 7 lines
Quoted 5 lines

In general the reports have been high value (I ignore most reports with -Wimplicit-function-declaration, which is the most frequent as it shows the patch was not compile tested at all).

Do we mean the report with -Wimplicit-function-declaration can be duplicated to gcc, so we can ignore them to avoid duplication to developer?

Many of the warnings GCC has Clang does as well. -Wimplicit-function-declaration is the most common warning I see in triage, which developers would see regardless of toolchain had they compiled first before pushing. It might be interesting to see maybe the intersection or common flags between GCC and Clang, and only email Clang reports for warnings unique to clang? I think CFLAGS can even be passed into make invocations so you could do:

$ make CC=clang KBUILD_CFLAGS=<list of flags common to GCC and Clang;
-Wno-implicit-function-declaration -Wno-...>
such that any resulting warnings were unique to Clang.  I'd expect
such a list to quickly get stale over time though.
Quoted 16 lines
Quoted 5 lines

Rong, Philip, it's been a while since we talked about this last. Is there a general timeline of when these reports will be turned on globally? Even if the directions to reproduce aren't quite right,

For the timeline, it's not decided due to the duplication concern. We tend to look into next year after other priorities are solved for this year.

Quoted 8 lines

generally there's enough info in the existing bugs where authors can rewrite their patch without even needing to rebuild with Clang (though having correct directions to reproduce would be nice, we could wait until someone asked for them explicitly).

--
Thanks,
~Nick Desaulniers

Signature · 2 lines
Thanks,
~Nick Desaulniers