Skip to contentSkip to conversation

NFS: Fix missing files in `ls` command output

RFC PATCH In review Yafang Shao 4 participants 10 messages Aug 29, 2024 – Sep 3, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 NFS: Fix missing files in `ls` command output

In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:

  $ cd nfs
  $ ls
  a b c e f            <<<< 'd' is missing
  $ cd d               <<<< success

I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.

Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.

However, I am unable to reproduce this issue with a simple example; it only occurs on our production servers.

---
 fs/nfs/dir.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)
fs/nfs/dir.c +17 -5
index 07a7be27182e..1f5a99888a11 100644
@@ -310,7 +310,7 @@ static int nfs_readdir_array_can_expand(struct nfs_cache_array *array)
 
 static int nfs_readdir_folio_array_append(struct folio *folio,
 					  const struct nfs_entry *entry,
-					  u64 *cookie)
+					  u64 *cookie, u64 *prev_cookie)
 {
 	struct nfs_cache_array *array;
 	struct nfs_cache_array_entry *cache_entry;
@@ -342,6 +342,7 @@ static int nfs_readdir_folio_array_append(struct folio *folio,
 		nfs_readdir_array_set_eof(array);
 out:
 	*cookie = array->last_cookie;
+	*prev_cookie = cache_entry->cookie;
 	kunmap_local(array);
 	return ret;
 }
@@ -826,10 +827,11 @@ static int nfs_readdir_folio_filler(struct nfs_readdir_descriptor *desc,
 {
 	struct address_space *mapping = desc->file->f_mapping;
 	struct folio *new, *folio = *arrays;
+	struct nfs_cache_array *array;
 	struct xdr_stream stream;
+	u64 cookie, prev_cookie;
 	struct page *scratch;
 	struct xdr_buf buf;
-	u64 cookie;
 	int status;
 
 	scratch = alloc_page(GFP_KERNEL);
@@ -841,10 +843,20 @@ static int nfs_readdir_folio_filler(struct nfs_readdir_descriptor *desc,
 
 	do {
 		status = nfs_readdir_entry_decode(desc, entry, &stream);
-		if (status != 0)
+		if (status != 0) {
+			if (status == -EAGAIN && entry->cookie == cookie) {
+				/* Revert the bad entry */
+				array = kmap_local_folio(folio, 0);
+				array->last_cookie = prev_cookie;
+				desc->last_cookie = 0;
+				desc->dir_cookie = 0;
+				array->size--;
+				kunmap_local(array);
+			}
 			break;
+		}
 
-		status = nfs_readdir_folio_array_append(folio, entry, &cookie);
+		status = nfs_readdir_folio_array_append(folio, entry, &cookie, &prev_cookie);
 		if (status != -ENOSPC)
 			continue;
 
@@ -866,7 +878,7 @@ static int nfs_readdir_folio_filler(struct nfs_readdir_descriptor *desc,
 			folio = new;
 		}
 		desc->folio_index_max++;
-		status = nfs_readdir_folio_array_append(folio, entry, &cookie);
+		status = nfs_readdir_folio_array_append(folio, entry, &cookie, &prev_cookie);
 	} while (!status && !entry->eof);
 
 	switch (status) {
Signature · 1 line
2.30.1 (Apple Git-130)
#1 lore raw
Quoted 17 lines · Yafang Shao

In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:

  $ cd nfs
  $ ls
  a b c e f            <<<< 'd' is missing
  $ cd d               <<<< success

I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.

Quoted 5 lines

Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.

It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?

Ben

#2 lore raw
Quoted 20 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao

In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:

  $ cd nfs
  $ ls
  a b c e f            <<<< 'd' is missing
  $ cd d               <<<< success

I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.

Quoted 8 lines
Quoted 5 lines

Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.

It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?

I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?

Interestingly, when we increase the dtsize, the issue goes away. This suggests that the problem might not be with the server itself, but rather with the NFS readdir operation. The change in dtsize is as follows,

fs/nfs/dir.c +1 -1
index 56a8aee..39847e1 100644
@@ -81,7 +81,7 @@
        ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT);
        if (ctx != NULL) {
                ctx->attr_gencount = nfsi->attr_gencount;
-               ctx->dtsize = NFS_INIT_DTSIZE;
+               ctx->dtsize = 2 * NFS_INIT_DTSIZE; // 8K
                spin_lock(&dir->i_lock);
                if (list_empty(&nfsi->open_files) &&
                    (nfsi->cache_validity & NFS_INO_DATA_INVAL_DEFER))

--
Regards
Yafang

#3 lore raw
Quoted 27 lines · Yafang Shao
Quoted 26 lines · Benjamin Coddington
Quoted 22 lines · Yafang Shao

In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:

  $ cd nfs
  $ ls
  a b c e f            <<<< 'd' is missing
  $ cd d               <<<< success

I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.

Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.

Quoted 5 lines
Quoted 2 lines

It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?

I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?

I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.

Ben

#4 lore raw
Quoted 33 lines · Benjamin Coddington
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao

In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:

  $ cd nfs
  $ ls
  a b c e f            <<<< 'd' is missing
  $ cd d               <<<< success

I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.

Quoted 5 lines

Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.

It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?

Quoted 6 lines
Quoted 2 lines

I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?

I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.

Thank you for the suggestion. I have captured and analyzed the NFS RPC traffic using Wireshark. I noticed that the ls command is being split into two NFS READDIR operations. In the first READDIR request, both the dircount and maxcount parameters are set to 4008. In the subsequent READDIR request, both dircount and maxcount are set to 8192.

Interestingly, when I increase the value of ctx->dtsize to 8192, the ls command now generates only a single NFS READDIR RPC call. In this case, both the dircount and maxcount parameters are set to 8104. This issue disappears as well.

--
Regards
Yafang

#5 lore raw
In reply to Benjamin Coddington
Quoted 33 lines · Benjamin Coddington
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao

In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:

  $ cd nfs
  $ ls
  a b c e f            <<<< 'd' is missing
  $ cd d               <<<< success

I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.

Quoted 5 lines

Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.

It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?

Quoted 9 lines
Quoted 2 lines

I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?

I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.

Ben

Hello Ben,

Upon thorough examination, we have identified the root cause of the issue to lie within the NFS server, specifically its behavior of truncating file listings to match the client's READDIR RPC args->size parameter without appropriately adjusting the cookie value. After implementing a fix on the server side, the issue has been resolved. However, to enhance resilience and mitigate future server-side vulnerabilities, it may be prudent to implement client-side handling mechanisms for such issues. What do you think?

Signature · 2 lines
Regards
Yafang
#6 lore raw
Quoted 48 lines · Yafang Shao
Quoted 33 lines · Benjamin Coddington
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao

In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:

 $ cd nfs
 $ ls
 a b c e f            <<<< 'd' is missing
 $ cd d               <<<< success

I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.

Quoted 5 lines

Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.

It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?

Quoted 9 lines
Quoted 2 lines

I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?

I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.

Ben

Quoted 7 lines

Hello Ben,

Upon thorough examination, we have identified the root cause of the issue to lie within the NFS server, specifically its behavior of truncating file listings to match the client's READDIR RPC args->size parameter without appropriately adjusting the cookie value. After implementing a fix on the server side, the issue has been resolved.

Please post your server fix on this mailing list. Thanks!

Quoted 3 lines

However, to enhance resilience and mitigate future server-side vulnerabilities, it may be prudent to implement client-side handling mechanisms for such issues. What do you think?

The general policy we follow is to avoid fixing server bugs via client-side workarounds. Fix the server in that case.

--
Chuck Lever

#7 lore raw
Quoted 54 lines · Chuck Lever III
Quoted 51 lines · Yafang Shao
Quoted 33 lines · Benjamin Coddington
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao

In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:

$ cd nfs
$ ls
a b c e f            <<<< 'd' is missing
$ cd d               <<<< success

I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.

Quoted 5 lines

Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.

It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?

Quoted 9 lines
Quoted 2 lines

I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?

I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.

Ben

Hello Ben,

Quoted 7 lines
Quoted 5 lines

Upon thorough examination, we have identified the root cause of the issue to lie within the NFS server, specifically its behavior of truncating file listings to match the client's READDIR RPC args->size parameter without appropriately adjusting the cookie value. After implementing a fix on the server side, the issue has been resolved.

Please post your server fix on this mailing list. Thanks!

I was assuming your test server was Linux NFSD. If not, then please ignore me!

--
Chuck Lever

#8 lore raw
In reply to Yafang Shao
Quoted 7 lines · Yafang Shao

Hello Ben,

Upon thorough examination, we have identified the root cause of the issue to lie within the NFS server, specifically its behavior of truncating file listings to match the client's READDIR RPC args->size parameter without appropriately adjusting the cookie value. After implementing a fix on the server side, the issue has been resolved.

Nice work! Out of curiosity, what server implemenation did you fix?

Quoted 3 lines

However, to enhance resilience and mitigate future server-side vulnerabilities, it may be prudent to implement client-side handling mechanisms for such issues. What do you think?

We have in the past modified the client to be more resilient, but usually only for cases where the server can cause the client to crash and/or corrupt data. For a bug like this, the maintainers usually assert "we do not fix the client for server bugs", since doing so can paper over protocol correctness issues created by the server. I think that's what would happen here if you keep working to justify your fix -- which you're free to do, of course!

Regards,
Ben

#9 lore raw
In reply to Yafang Shao
kernel test robot noticed the following build warnings:

Hi Yafang,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on trondmy-nfs/linux-next]
[also build test WARNING on linus/master v6.11-rc6 next-20240830] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Yafang-Shao/NFS-Fix-missing-files-in-ls-command-output/20240829-171711
base:   git://git.linux-nfs.org/projects/trondmy/linux-nfs.git linux-next
patch link:    https://lore.kernel.org/r/20240829091340.2043-1-laoar.shao%40gmail.com
patch subject: [RFC PATCH] NFS: Fix missing files in `ls` command output
config: i386-randconfig-001-20240901 (https://download.01.org/0day-ci/archive/20240901/[email protected]/config)
compiler: clang version 18.1.5 (https://github.com/llvm/llvm-project 617a15a9eac96088ae5e9134248d8236e34b91b1)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240901/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> fs/nfs/dir.c:326:6: warning: variable 'cache_entry' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
     326 |         if (ret) {
         |             ^~~
   fs/nfs/dir.c:345:17: note: uninitialized use occurs here
     345 |         *prev_cookie = cache_entry->cookie;
         |                        ^~~~~~~~~~~
   fs/nfs/dir.c:326:2: note: remove the 'if' if its condition is always false
     326 |         if (ret) {
         |         ^~~~~~~~~~
     327 |                 kfree(name);
         |                 ~~~~~~~~~~~~
     328 |                 goto out;
         |                 ~~~~~~~~~
     329 |         }
         |         ~
   fs/nfs/dir.c:323:6: warning: variable 'cache_entry' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
     323 |         if (!name)
         |             ^~~~~
   fs/nfs/dir.c:345:17: note: uninitialized use occurs here
     345 |         *prev_cookie = cache_entry->cookie;
         |                        ^~~~~~~~~~~
   fs/nfs/dir.c:323:2: note: remove the 'if' if its condition is always false
     323 |         if (!name)
         |         ^~~~~~~~~~
     324 |                 goto out;
         |                 ~~~~~~~~
   fs/nfs/dir.c:316:43: note: initialize the variable 'cache_entry' to silence this warning
     316 |         struct nfs_cache_array_entry *cache_entry;
         |                                                  ^
         |                                                   = NULL

2 warnings generated.

vim +326 fs/nfs/dir.c

d1bacf9eb2fd0e Bryan Schumaker       2010-09-24  310  
ec108d3cc7663d Anna Schumaker        2023-04-04  311  static int nfs_readdir_folio_array_append(struct folio *folio,
0adf85b445c7fb Trond Myklebust       2022-02-27  312  					  const struct nfs_entry *entry,
b04660a1b4ac01 Yafang Shao           2024-08-29  313  					  u64 *cookie, u64 *prev_cookie)
d1bacf9eb2fd0e Bryan Schumaker       2010-09-24  314  {
a52a8a6adad99e Trond Myklebust       2020-11-01  315  	struct nfs_cache_array *array;
4a201d6e3f4253 Trond Myklebust       2010-10-23  316  	struct nfs_cache_array_entry *cache_entry;
a52a8a6adad99e Trond Myklebust       2020-11-01  317  	const char *name;
0adf85b445c7fb Trond Myklebust       2022-02-27  318  	int ret = -ENOMEM;
4a201d6e3f4253 Trond Myklebust       2010-10-23  319  
a52a8a6adad99e Trond Myklebust       2020-11-01  320  	name = nfs_readdir_copy_name(entry->name, entry->len);
3020093f578fb6 Trond Myklebust       2010-11-20  321  
4b71e2416ec4cb Fabio M. De Francesco 2023-05-03  322  	array = kmap_local_folio(folio, 0);
0adf85b445c7fb Trond Myklebust       2022-02-27  323  	if (!name)
0adf85b445c7fb Trond Myklebust       2022-02-27  324  		goto out;
b1e21c97437f64 Trond Myklebust       2020-11-01  325  	ret = nfs_readdir_array_can_expand(array);
a52a8a6adad99e Trond Myklebust       2020-11-01 @326  	if (ret) {
a52a8a6adad99e Trond Myklebust       2020-11-01  327  		kfree(name);
4a201d6e3f4253 Trond Myklebust       2010-10-23  328  		goto out;
a52a8a6adad99e Trond Myklebust       2020-11-01  329  	}
d1bacf9eb2fd0e Bryan Schumaker       2010-09-24  330  
b1e21c97437f64 Trond Myklebust       2020-11-01  331  	cache_entry = &array->array[array->size];
0adf85b445c7fb Trond Myklebust       2022-02-27  332  	cache_entry->cookie = array->last_cookie;
4a201d6e3f4253 Trond Myklebust       2010-10-23  333  	cache_entry->ino = entry->ino;
0b26a0bf6ff398 Trond Myklebust       2010-11-20  334  	cache_entry->d_type = entry->d_type;
a52a8a6adad99e Trond Myklebust       2020-11-01  335  	cache_entry->name_len = entry->len;
a52a8a6adad99e Trond Myklebust       2020-11-01  336  	cache_entry->name = name;
d1bacf9eb2fd0e Bryan Schumaker       2010-09-24  337  	array->last_cookie = entry->cookie;
762567b7c798af Trond Myklebust       2020-11-04  338  	if (array->last_cookie <= cache_entry->cookie)
762567b7c798af Trond Myklebust       2020-11-04  339  		array->cookies_are_ordered = 0;
8cd51a0ccd1bed Trond Myklebust       2010-11-15  340  	array->size++;
47c716cbf638a1 Trond Myklebust       2010-12-07  341  	if (entry->eof != 0)
b1e21c97437f64 Trond Myklebust       2020-11-01  342  		nfs_readdir_array_set_eof(array);
4a201d6e3f4253 Trond Myklebust       2010-10-23  343  out:
0adf85b445c7fb Trond Myklebust       2022-02-27  344  	*cookie = array->last_cookie;
b04660a1b4ac01 Yafang Shao           2024-08-29  345  	*prev_cookie = cache_entry->cookie;
4b71e2416ec4cb Fabio M. De Francesco 2023-05-03  346  	kunmap_local(array);
4a201d6e3f4253 Trond Myklebust       2010-10-23  347  	return ret;
d1bacf9eb2fd0e Bryan Schumaker       2010-09-24  348  }
d1bacf9eb2fd0e Bryan Schumaker       2010-09-24  349  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki