Files/Folders invisibles with 'ls -a' but can 'cd' to folder
Hi
As I say in my subject, I'm facing a weird problem with my btrfs partition (I already sent this message on reddit /r/btrfs/ )
It's in fact a btrfs partition in a raid5 synology system.
3 days ago, the volume 'crashed' (synology terms) ,however SMART data is ok and I don't have sector relcocation errors or CRC.... I rebooted several times , and after dozen of reboots my partition shows up , but 3 TB of 10 TB are missing, I made a scrub but it did made my missing files appears.
desperately I made a 'cd xyz' in a directory (I remember some of the folder names) and it works ; and inside this folder I can do "ls" and all files and subfolders appears .
I made a copy elsewhere of some files and these ones are not corrupted or bit roted.
I don't want to make a btrfs check --repair of course.
Is there a way to "relink" indexes/root or whatever it is called to bring back these files/folder visible and accessible with a safe command ?
I'm planning to backup all , is 'btrfs restore' will access to these "non visible" directories ?
"I saw similar case here : The Directory Who Wasn't There : btrfs (reddit.com) , but I can't find a reply that solve the problem"
cdly
Quoted 5 lines · k g
Hi
As I say in my subject, I'm facing a weird problem with my btrfs partition (I already sent this message on reddit /r/btrfs/ )
Sorry, reddit is not really the go-to place for technical discussion nor bug report.
Quoted 2 lines
It's in fact a btrfs partition in a raid5 synology system.
We don't know how heavily backported the synology kernel is, thus it's normally better to ask for help from synology.
Quoted 15 lines
3 days ago, the volume 'crashed' (synology terms) ,however SMART data is ok and I don't have sector relcocation errors or CRC.... I rebooted several times , and after dozen of reboots my partition shows up , but 3 TB of 10 TB are missing, I made a scrub but it did made my missing files appears.
desperately I made a 'cd xyz' in a directory (I remember some of the folder names) and it works ; and inside this folder I can do "ls" and all files and subfolders appears .
Quoted 6 lines
I made a copy elsewhere of some files and these ones are not corrupted or bit roted.
I don't want to make a btrfs check --repair of course.
But "btrfs check" without --repair should be the best tool to show what's going wrong.
Alternatively, "btrfs check --mode=lowmem" could provide a better human readable output.
Quoted 5 lines
Is there a way to "relink" indexes/root or whatever it is called to bring back these files/folder visible and accessible with a safe command ?
It's not that simple, from your description, it looks like the dir has some DIR_ITEM but no DIR_INDEX, thus it doesn't shows up in ls, but cd still work.
This normally indicates much bigger problem.
Thanks,
Qu
Quoted 13 lines
I'm planning to backup all , is 'btrfs restore' will access to these "non visible" directories ?
"I saw similar case here : The Directory Who Wasn't There : btrfs (reddit.com) , but I can't find a reply that solve the problem"
cdly
Hi Qu ,
Thanks for your valuable answers,
The synology system I'm using has btrfs tools v4. I compiled a v5 version because "btrfs check" version 4 returns "Couldn't open file system"
a little bit late, (I'm 1000 km away from my crashed server, I'm doing all of this remotly) ,Here is some output of btrfs check I made today (output is very long , here is some samples of the messages returned)
Opening filesystem to check...
Checking filesystem on /dev/md2
UUID: 306faa08-9e17-406b-924e-57e06e2c2763
[1/7] checking root items (0:03:47 elapsed, 4389038 items checked)
[2/7] checking extents (0:06:51 elapsed, 3092 items checked)
cache and super generation don't match, space cache will be invalidated [3/7] checking free space cache (0:00:00 elapsed) [4/7] checking fs roots (0:00:00 elapsed, 2 items checked)
found 196033421312 bytes used, error(s) found
total csum bytes: 15533120
total tree bytes: 50102272
total fs tree bytes: 17711104
total extent tree bytes: 31784960
btree space waste bytes: 10681343
file data blocks allocated: 0
referenced 0
ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent level=2 child bytenr=229931827200 child level=0
ERROR: extent[193738493952, 16384] referencer count mismatch (root: 257,
owner: 226748, offset: 0) wanted: 1, have: 0
parent transid verify failed on 229931827200 wanted 11406678 found 11406670 Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent level=2 child bytenr=229931827200 child level=0
ERROR: extent[193738510336, 16384] referencer count mismatch (root: 257,
owner: 226749, offset: 0) wanted: 1, have: 0
ERROR: extent [197244026880 16384] referencer bytenr mismatch, wanted:
197244026880, have: 229859393536
ERROR: extent [197244043264 16384] referencer bytenr mismatch, wanted:
197244043264, have: 229859393536
parent transid verify failed on 229931827200 wanted 11406678 found 11406670 Ignoring transid failure
WARNING: tree block [197151997952, 197152014336) crosses 64K page boudnary, may cause problem for 64K page system
WARNING: tree block [197152063488, 197152079872) crosses 64K page boudnary, may cause problem for 64K page system
Wrong key of child node/leaf, wanted: (197244633088, 169, 0), have:
(40699577, 108, 0)
Wrong generation of child node/leaf, wanted: 11406682, have: 11406670 parent transid verify failed on 229934661632 wanted 11406672 found 11406677 Ignoring transid failure
ERROR: block group[9778059280384 10737418240] used 0 but extent items used 10735706112
ERROR: block group[9788796698624 10737418240] used 0 but extent items used 10736922624
leaf parent key incorrect 229908168704
parent transid verify failed on 229908168704 wanted 11406668 found 11406678 Ignoring transid failure
ERROR: free space cache inode 41584067 has invalid mode: has 0100644 expect 0100600
parent transid verify failed on 229926993920 wanted 11406669 found 11406676 Ignoring transid failure
parent transid verify failed on 229958598656 wanted 11406672 found 11406678 Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=229823332352 item=5 parent level=1 child bytenr=229958598656 child level=1
ERROR: errors found in fs roots
extent buffer leak: start 229934661632 len 16384
extent buffer leak: start 229823561728 len 16384
extent buffer leak: start 229931532288 len 16384
extent buffer leak: start 229931827200 len 16384
By "luck" I have a sql dump that contain 80% of the paths of the lost files, so I can make a bash or python script to recover them (by doing a copy elsewhere or mv two times the hidden folders)
unfortunately, these path are samba paths and some of them are mangled (I did not manage, or had the time to reverse engineer the samba source code to rebuild linux paths from samba mangled path despite that I asked some help in stackoverflow...)
But before starting building these scripts , I want to have your feedback before launching any maintenance operation, or at best a procedure to relink these DIR_INDEX
all the best.
.k.
Quoted 74 lines · Qu Wenruo
Quoted 5 lines · k g
Hi
As I say in my subject, I'm facing a weird problem with my btrfs partition (I already sent this message on reddit /r/btrfs/ )
Sorry, reddit is not really the go-to place for technical discussion nor bug report.
Quoted 2 lines
It's in fact a btrfs partition in a raid5 synology system.
We don't know how heavily backported the synology kernel is, thus it's normally better to ask for help from synology.
Quoted 15 lines
3 days ago, the volume 'crashed' (synology terms) ,however SMART data is ok and I don't have sector relcocation errors or CRC.... I rebooted several times , and after dozen of reboots my partition shows up , but 3 TB of 10 TB are missing, I made a scrub but it did made my missing files appears.
desperately I made a 'cd xyz' in a directory (I remember some of the folder names) and it works ; and inside this folder I can do "ls" and all files and subfolders appears .
Quoted 6 lines
I made a copy elsewhere of some files and these ones are not corrupted or bit roted.
I don't want to make a btrfs check --repair of course.
But "btrfs check" without --repair should be the best tool to show what's going wrong.
Alternatively, "btrfs check --mode=lowmem" could provide a better human readable output.
Quoted 6 lines
Is there a way to "relink" indexes/root or whatever it is called to bring back these files/folder visible and accessible with a safe command ?
It's not that simple, from your description, it looks like the dir has some
DIR_ITEMbut noDIR_INDEX, thus it doesn't shows up in ls, but cd still work.This normally indicates much bigger problem.
Thanks,
QuQuoted 13 lines
I'm planning to backup all , is 'btrfs restore' will access to these "non visible" directories ?
"I saw similar case here : The Directory Who Wasn't There : btrfs (reddit.com) , but I can't find a reply that solve the problem"
cdly
Quoted 36 lines · k g
Hi Qu ,
Thanks for your valuable answers,
The synology system I'm using has btrfs tools v4. I compiled a v5 version because "btrfs check" version 4 returns "Couldn't open file system"
a little bit late, (I'm 1000 km away from my crashed server, I'm doing all of this remotly) ,Here is some output of btrfs check I made today (output is very long , here is some samples of the messages returned)
Opening filesystem to check...
Checking filesystem on/dev/md2
UUID: 306faa08-9e17-406b-924e-57e06e2c2763
[1/7] checking root items (0:03:47 elapsed, 4389038 items checked)
[2/7] checking extents (0:06:51 elapsed, 3092 items checked)
cache and super generation don't match, space cache will be invalidated [3/7] checking free space cache (0:00:00 elapsed) [4/7] checking fs roots (0:00:00 elapsed, 2 items checked)
found 196033421312 bytes used, error(s) found
total csum bytes: 15533120
total tree bytes: 50102272
total fs tree bytes: 17711104
total extent tree bytes: 31784960
btree space waste bytes: 10681343
file data blocks allocated: 0
referenced 0
Quoted 2 lines
ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent level=2 child bytenr=229931827200 child level=0
This is the worst case, metadata corruption between nodes and leaves.
Quoted 9 lines
ERROR: extent[193738493952, 16384] referencer count mismatch (root: 257,
owner: 226748, offset: 0) wanted: 1, have: 0
parent transid verify failed on 229931827200 wanted 11406678 found 11406670 Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=229829935104 item=4 parent level=2 child bytenr=229931827200 child level=0
ERROR: extent[193738510336, 16384] referencer count mismatch (root: 257,
owner: 226749, offset: 0) wanted: 1, have: 0
Quoted 7 lines
ERROR: extent [197244026880 16384] referencer bytenr mismatch, wanted:
197244026880, have: 229859393536
ERROR: extent [197244043264 16384] referencer bytenr mismatch, wanted:
197244043264, have: 229859393536parent transid verify failed on 229931827200 wanted 11406678 found 11406670 Ignoring transid failure
Furthermore, transid mismatch...
Quoted 23 lines
WARNING: tree block [197151997952, 197152014336) crosses 64K page boudnary, may cause problem for 64K page system
WARNING: tree block [197152063488, 197152079872) crosses 64K page boudnary, may cause problem for 64K page systemWrong key of child node/leaf, wanted: (197244633088, 169, 0), have:
(40699577, 108, 0)
Wrong generation of child node/leaf, wanted: 11406682, have: 11406670 parent transid verify failed on 229934661632 wanted 11406672 found 11406677 Ignoring transid failureERROR: block group[9778059280384 10737418240] used 0 but extent items used 10735706112
ERROR: block group[9788796698624 10737418240] used 0 but extent items used 10736922624leaf parent key incorrect 229908168704
parent transid verify failed on 229908168704 wanted 11406668 found 11406678 Ignoring transid failure
Quoted 13 lines
ERROR: free space cache inode 41584067 has invalid mode: has 0100644 expect 0100600
parent transid verify failed on 229926993920 wanted 11406669 found 11406676 Ignoring transid failure
parent transid verify failed on 229958598656 wanted 11406672 found 11406678 Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=229823332352 item=5 parent level=1 child bytenr=229958598656 child level=1
ERROR: errors found in fs roots
extent buffer leak: start 229934661632 len 16384
extent buffer leak: start 229823561728 len 16384
extent buffer leak: start 229931532288 len 16384
extent buffer leak: start 229931827200 len 16384
Overall, the metadata is mostly screwed up, thus the DIR_ITEM/DIR_INDEX mismatch happens.
Quoted 7 lines
By "luck" I have a sql dump that contain 80% of the paths of the lost files, so I can make a bash or python script to recover them (by doing a copy elsewhere or mv two times the hidden folders)
Quoted 8 lines
unfortunately, these path are samba paths and some of them are mangled (I did not manage, or had the time to reverse engineer the samba source code to rebuild linux paths from samba mangled path despite that I asked some help in stackoverflow...)
But before starting building these scripts , I want to have your feedback before launching any maintenance operation, or at best a procedure to relink these
DIR_INDEX
It's way worse than just DIR_INDEX/DIR_ITEM missing, it's some metadata corruption that mostly screw up the filesystem.
I would go btrfs-restore directly to salvage as much data as possible.
Thanks,
Qu
Quoted 83 lines
all the best.
.k.
Quoted 74 lines · Qu Wenruo
Quoted 5 lines · k g
Hi
As I say in my subject, I'm facing a weird problem with my btrfs partition (I already sent this message on reddit /r/btrfs/ )
Sorry, reddit is not really the go-to place for technical discussion nor bug report.
Quoted 2 lines
It's in fact a btrfs partition in a raid5 synology system.
We don't know how heavily backported the synology kernel is, thus it's normally better to ask for help from synology.
Quoted 15 lines
3 days ago, the volume 'crashed' (synology terms) ,however SMART data is ok and I don't have sector relcocation errors or CRC.... I rebooted several times , and after dozen of reboots my partition shows up , but 3 TB of 10 TB are missing, I made a scrub but it did made my missing files appears.
desperately I made a 'cd xyz' in a directory (I remember some of the folder names) and it works ; and inside this folder I can do "ls" and all files and subfolders appears .
Quoted 6 lines
I made a copy elsewhere of some files and these ones are not corrupted or bit roted.
I don't want to make a btrfs check --repair of course.
But "btrfs check" without --repair should be the best tool to show what's going wrong.
Alternatively, "btrfs check --mode=lowmem" could provide a better human readable output.
Quoted 6 lines
Is there a way to "relink" indexes/root or whatever it is called to bring back these files/folder visible and accessible with a safe command ?
It's not that simple, from your description, it looks like the dir has some
DIR_ITEMbut noDIR_INDEX, thus it doesn't shows up in ls, but cd still work.This normally indicates much bigger problem.
Thanks,
QuQuoted 13 lines
I'm planning to backup all , is 'btrfs restore' will access to these "non visible" directories ?
"I saw similar case here : The Directory Who Wasn't There : btrfs (reddit.com) , but I can't find a reply that solve the problem"
cdly