fs/fat/fatwrite: Local variable as buffer to store dir_slot entries
PATCH 1/2 fs/fat/fatwrite: Local variable as buffer to store dir_slot entries
fill_dir_slot use get_contents_vfatname_block as a temporary buffer for constructing a list of dir_slot entries. To save the memory and providing correct type of memory for above usage, a local buffer with accurate size declaration is introduced.
The local array size 640 is used because for long file name entry, each entry use 32 bytes, one entry can store up to 13 characters. The maximum number of entry possible is 20. So, total size is 32*20=640bytes.
--- fs/fat/fat_write.c | 3 ++- include/fat.h | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-)
fs/fat/fat_write.c +2 -1
@@ -323,7 +323,8 @@ static void flush_dir_table(fsdata *mydata, dir_entry **dentptr);static void fill_dir_slot (fsdata *mydata,dir_entry **dentptr,const char *l_name) { -dir_slot *slotptr = (dir_slot *)get_contents_vfatname_block; +__u8 temp_dir_slot_buffer[MAX_LFN_SLOT *sizeof (dir_slot)]; +dir_slot *slotptr = (dir_slot *)temp_dir_slot_buffer;__u8 counter =0 , checksum;int idx =0 , ret;char s_name[16 ];
include/fat.h +3 -0
@@ -33,6 +33,9 @@#define FAT16BUFSIZE (FATBUFSIZE/2)#define FAT32BUFSIZE (FATBUFSIZE/4) +/* Maximum number of entry for long file name according to spec */ +#define MAX_LFN_SLOT 20 +/* Filesystem identifiers */ #define FAT12_SIGN "FAT12 "
Signature · 1 line
1.7.7.4
PATCH 2/2 fs/fat/fat: Optimizes memory size with single global variable instead of 3
Single 64KB get_contents_vfatname_block global variable would be used for all FAT implementation instead of allocating additional two global variables which are get_denfromdir_block and do_fat_read_at_block. This implementation can help in saving up 128KB memory space.
--- fs/fat/fat.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-)
fs/fat/fat.c +2 -4
@@ -579,8 +579,7 @@ static __u8 mkcksum(const char name[8], const char ext[3])* Get the directory entry associated with 'filename' from the directory * starting at 'startsect' */ -__u8 get_dentfromdir_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *get_dentfromdir_block = get_contents_vfatname_block; static dir_entry *get_dentfromdir(fsdata *mydata, int startsect, char *filename, dir_entry *retdent,
@@ -811,8 +810,7 @@ exit:return ret; } -__u8 do_fat_read_at_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *do_fat_read_at_block = get_contents_vfatname_block;int do_fat_read_at (const char *filename,loff_t pos,void *buffer,loff_t maxsize,int dols,int dogetsize,loff_t *size)
Signature · 1 line
1.7.7.4
PATCH 2/2 fs/fat/fat: Optimizes memory size with single global variable instead of 3
Dear Tien Fong Chee,
Quoted 32 lines · Tien Fong Chee
Single 64KB
get_contents_vfatname_blockglobal variable would be used for all FAT implementation instead of allocating additional two global variables which areget_denfromdir_blockanddo_fat_read_at_block. This implementation can help in saving up 128KB memory space.Signed-off-by Tien Fong Chee <[email protected]>Cc Dinh Nguyen <[email protected]>Cc Dinh Nguyen <[email protected]>Cc ChinLiang <[email protected]>Cc Vagrant Cascadian <[email protected]>Cc Simon Glass <[email protected]>Cc Stephen Warren <[email protected]>Cc Beno?t Th?baudeau <[email protected]>--- fs/fat/fat.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-)fs/fat/fat.c +1
-2@@ -579,8 +579,7 @@ static __u8 mkcksum(const char name[8], const char ext[3])* Get the directory entry associated with 'filename' from the directory * starting at 'startsect' */ -__u8 get_dentfromdir_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *get_dentfromdir_block = get_contents_vfatname_block; static dir_entry *get_dentfromdir(fsdata *mydata, int startsect, char *filename, dir_entry *retdent,
fs/fat/fat.c exit: · line 816
Quoted 9 lines
fs/fat/fat.c +1
-2@@ -811,8 +810,7 @@ exit:return ret; } -__u8 do_fat_read_at_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *do_fat_read_at_block = get_contents_vfatname_block;int do_fat_read_at (const char *filename,loff_t pos,void *buffer,loff_t maxsize,int dols,int dogetsize,loff_t *size)
This probably breaks at least fat_write.c, which uses:
memcpy(get_dentfromdir_block, get_contents_vfatname_block,
Best regards,
Beno?t
PATCH 2/2 fs/fat/fat: Optimizes memory size with single global variable instead of 3
Dear Beno?t,
Quoted 55 lines · Beno?t Th?baudeau
Dear Tien Fong Chee,
Quoted 47 lines · Tien Fong Chee
Single 64KB
get_contents_vfatname_blockglobal variable would be used for
all FAT implementation instead of allocating additional two global variables
which areget_denfromdir_blockanddo_fat_read_at_block. This implementation
can help in saving up 128KB memory space.Signed-off-by Tien Fong Chee <[email protected]>Cc Dinh Nguyen <[email protected]>Cc Dinh Nguyen <[email protected]>Cc ChinLiang <[email protected]>Cc Vagrant Cascadian <[email protected]>Cc Simon Glass <[email protected]>Cc Stephen Warren <[email protected]>Cc Beno?t Th?baudeau <[email protected]>--- fs/fat/fat.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-)fs/fat/fat.c +0
-0@@ -579,8 +579,7 @@ static __u8 mkcksum(const char name[8], const char ext[3]) * Get the directory entry associated with 'filename' from thedirectory
* starting at 'startsect'
*/fs/fat/fat.c +1
-2-__u8 get_dentfromdir_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *get_dentfromdir_block = get_contents_vfatname_block;static dir_entry *get_dentfromdir (fsdata *mydata,int startsect, char *filename, dir_entry*retdent,
fs/fat/fat.c +1
-2@@ -811,8 +810,7 @@ exit:return ret; } -__u8 do_fat_read_at_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *do_fat_read_at_block = get_contents_vfatname_block;int do_fat_read_at (const char *filename,loff_t pos,void *buffer, loff_t maxsize, int dols, int dogetsize, loff_tQuoted 1 line
*size)
This probably breaks at least
fat_write.c, which uses:
memcpy(get_dentfromdir_block,get_contents_vfatname_block,
Quoted 2 lines
Best regards,
Beno?t
With this patch, this line code "memcpy(get_dentfromdir_block, get_contents_vfatname_block," is not required anymore since both
get_dentfromdir_block and get_contents_vfatname_block are sharing the
same content and memory. So, this line code can be removed or leaving in there. However, there is only one place within fill_dir_slot_buffer function where it can corrupt the the global memory, and it is fixed by replacing with local buffer. This was sent out with another patch called "[PATCH 1/2] fs/fat/fatwrite: Local variable as buffer to store dir_slot entries". Overall, applying these two patches, a lot memory space can be saved and fitting into small OCRAM, but need to be very careful when modifying the code related to global memory.
Best regards,
Tien Fong
PATCH 2/2 fs/fat/fat: Optimizes memory size with single global variable instead of 3
Dear Tien Fong,
Quoted 58 lines · Tien Fong Chee
Dear Beno?t,
Quoted 48 lines · Beno?t Th?baudeau
Dear Tien Fong Chee,
Quoted 45 lines · Tien Fong Chee
Single 64KB
get_contents_vfatname_blockglobal variable would be used for
all FAT implementation instead of allocating additional two global variables
which areget_denfromdir_blockanddo_fat_read_at_block. This implementation
can help in saving up 128KB memory space.Signed-off-by Tien Fong Chee <[email protected]>Cc Dinh Nguyen <[email protected]>Cc Dinh Nguyen <[email protected]>Cc ChinLiang <[email protected]>Cc Vagrant Cascadian <[email protected]>Cc Simon Glass <[email protected]>Cc Stephen Warren <[email protected]>Cc Beno?t Th?baudeau <[email protected]>--- fs/fat/fat.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-)fs/fat/fat.c +0
-0@@ -579,8 +579,7 @@ static __u8 mkcksum(const char name[8], const char ext[3]) * Get the directory entry associated with 'filename' from thedirectory
* starting at 'startsect'
*/fs/fat/fat.c +1
-2-__u8 get_dentfromdir_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *get_dentfromdir_block = get_contents_vfatname_block;static dir_entry *get_dentfromdir (fsdata *mydata,int startsect, char *filename, dir_entry*retdent,
fs/fat/fat.c +1
-2@@ -811,8 +810,7 @@ exit:return ret; } -__u8 do_fat_read_at_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *do_fat_read_at_block = get_contents_vfatname_block;Quoted 6 lines
Quoted 3 lines
fs/fat/fat.c +0
-0int do_fat_read_at (const char *filename,loff_t pos,void *buffer, loff_t maxsize, int dols, int dogetsize, loff_t*size)
This probably breaks at least
fat_write.c, which uses:
memcpy(get_dentfromdir_block,get_contents_vfatname_block,
Quoted 11 lines
With this patch, this line code "memcpy(get_dentfromdir_block,
get_contents_vfatname_block," is not required anymore since both
get_dentfromdir_blockandget_contents_vfatname_blockare sharing the
same content and memory. So, this line code can be removed or leaving in there. However, there is only one place withinfill_dir_slot_bufferfunction where it can corrupt the the global memory, and it is fixed by replacing with local buffer. This was sent out with another patch called "[PATCH 1/2]fs/fat/fatwrite: Local variable as buffer to storedir_slotentries". Overall, applying these two patches, a lot memory space can be saved and fitting into small OCRAM, but need to be very careful when modifying the code related to global memory.
I get the point, but I am a bit concerned because these changes make the code even more fragile and hard to maintain than it currently is. Perhaps it would be time to switch to FatFs as previously suggested. Here is its memory usage:
http://elm-chan.org/fsw/ff/en/appnote.html#memory
I've not checked in detail all the possible code paths, but for instance, if I look at get_vfatname(), it's called twice in fat.c, once with cluster and retdent pointing to somewhere in get_dentfromdir_block, and once with them pointing to somewhere in do_fat_read_at_block (through other pointer variables), while get_vfatname() may write to get_contents_vfatname_block. get_vfatname() then uses the original data pointed to by slotptr >= retdent. To make things worse, there is a memcpy (not memmove) from realdent, which may point to somewhere in get_contents_vfatname_block, to retdent. It's almost impossible for the involved buffer areas not to overlap in that case since a whole cluster is read.
I also agree with Stephen's recommendations.
Best regards,
Beno?t
PATCH 2/2 fs/fat/fat: Optimizes memory size with single global variable instead of 3
Quoted 86 lines · Beno?t Th?baudeau
Dear Tien Fong,
Quoted 73 lines · Tien Fong Chee
Dear Beno?t,
Quoted 52 lines · Beno?t Th?baudeau
Dear Tien Fong Chee,
Quoted 49 lines · Tien Fong Chee
Single 64KB
get_contents_vfatname_blockglobal variable would be
used for
all FAT implementation instead of allocating additional two global
variables
which areget_denfromdir_blockanddo_fat_read_at_block. This implementation
can help in saving up 128KB memory space.Signed-off-by Tien Fong Chee <[email protected]>Cc Dinh Nguyen <[email protected]>Cc Dinh Nguyen <[email protected]>Cc ChinLiang <[email protected]>Cc Vagrant Cascadian <[email protected]>Cc Simon Glass <[email protected]>Cc Stephen Warren <[email protected]>Cc Beno?t Th?baudeau <[email protected]>--- fs/fat/fat.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-)fs/fat/fat.c +0
-0@@ -579,8 +579,7 @@ static __u8 mkcksum(const char name[8],const
char ext[3])
* Get the directory entry associated with 'filename' from the
directory
* starting at 'startsect'
*/fs/fat/fat.c +1
-2-__u8 get_dentfromdir_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *get_dentfromdir_block = get_contents_vfatname_block; static dir_entry *get_dentfromdir(fsdata *mydata, intstartsect,
char *filename,dir_entry
*retdent,fs/fat/fat.c +1
-2@@ -811,8 +810,7 @@ exit:return ret; } -__u8 do_fat_read_at_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *do_fat_read_at_block = get_contents_vfatname_block;Quoted 7 lines
Quoted 4 lines
fs/fat/fat.c +0
-0int do_fat_read_at(const char *filename, loff_t pos, void *buffer,
loff_tmaxsize, int dols, int dogetsize,loff_t
*size)This probably breaks at least
fat_write.c, which uses:
memcpy(get_dentfromdir_block,get_contents_vfatname_block,With this patch, this line code "memcpy(get_dentfromdir_block,
get_contents_vfatname_block," is not required anymore since both
get_dentfromdir_blockandget_contents_vfatname_blockare sharing
the
same content and memory. So, this line code can be removed or leaving
in there. However, there is only one place withinfill_dir_slot_buffer
function where it can corrupt the the global memory, and it is fixed byQuoted 8 lines
replacing with local buffer. This was sent out with another patch called "[PATCH 1/2]
fs/fat/fatwrite: Local variable as buffer to storedir_slotentries". Overall, applying these two patches, a lot memory
space can be saved and fitting into small OCRAM, but need to be very
careful when modifying the code related to global memory.
Quoted 2 lines
I get the point, but I am a bit concerned because these changes make the code even more fragile and hard to maintain than it currently is.
Yeah, i agree with you, and there is trade-off in saving the memory space.
Quoted 4 lines
Perhaps it would be time to switch to FatFs as previously suggested. Here is its memory usage:
http://elm-chan.org/fsw/ff/en/appnote.html#memory
Cool. If i am not mistaken, this implementation would impact a lot of areas, especially interface level. What's about the testing? I am still voting for this simple patch changes, until we have enough resources to do the switching.
Quoted 4 lines
I've not checked in detail all the possible code paths, but for instance, if I look at
get_vfatname(), it's called twice infat.c, once with cluster and retdent pointing to somewhere inget_dentfromdir_block, and once with them pointing to somewhere in
Quoted 9 lines
do_fat_read_at_block(through other pointer variables), whileget_vfatname()may write toget_contents_vfatname_block.get_vfatname()then uses the original data pointed to by slotptr >= retdent. To make things worse, there is a memcpy (not memmove) from realdent, which may point to somewhere inget_contents_vfatname_block,
to retdent. It's almost impossible for the involved buffer areas not to overlap in that case since a whole cluster is read.
Yeah, i know your concern. For our codes design at this moment, it still safe for those involved buffer areas to get overlaped, because most of them are referring the data after overlaping. fill_dir_slot function is the only corruption area i have found so far.
Quoted 4 lines
I also agree with Stephen's recommendations.
Best regards,
Beno?t
PATCH 2/2 fs/fat/fat: Optimizes memory size with single global variable instead of 3
Quoted 91 lines · Tien Fong Chee
Quoted 86 lines · Beno?t Th?baudeau
Dear Tien Fong,
Quoted 73 lines · Tien Fong Chee
Dear Beno?t,
Quoted 52 lines · Beno?t Th?baudeau
Dear Tien Fong Chee,
Quoted 49 lines · Tien Fong Chee
Single 64KB
get_contents_vfatname_blockglobal variable would be
used for
all FAT implementation instead of allocating additional two global
variables
which areget_denfromdir_blockanddo_fat_read_at_block. This implementation
can help in saving up 128KB memory space.Signed-off-by Tien Fong Chee <[email protected]>Cc Dinh Nguyen <[email protected]>Cc Dinh Nguyen <[email protected]>Cc ChinLiang <[email protected]>Cc Vagrant Cascadian <[email protected]>Cc Simon Glass <[email protected]>Cc Stephen Warren <[email protected]>Cc Beno?t Th?baudeau <[email protected]>--- fs/fat/fat.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-)fs/fat/fat.c +0
-0@@ -579,8 +579,7 @@ static __u8 mkcksum(const char name[8],const
char ext[3])
* Get the directory entry associated with 'filename' from the
directory
* starting at 'startsect'
*/fs/fat/fat.c +1
-2-__u8 get_dentfromdir_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *get_dentfromdir_block = get_contents_vfatname_block; static dir_entry *get_dentfromdir(fsdata *mydata, intstartsect,
char *filename,dir_entry
*retdent,fs/fat/fat.c +1
-2@@ -811,8 +810,7 @@ exit:return ret; } -__u8 do_fat_read_at_block[MAX_CLUSTSIZE ] -__aligned(ARCH_DMA_MINALIGN) ; +__u8 *do_fat_read_at_block = get_contents_vfatname_block;Quoted 7 lines
Quoted 4 lines
fs/fat/fat.c +0
-0int do_fat_read_at(const char *filename, loff_t pos, void *buffer,
loff_tmaxsize, int dols, int dogetsize,loff_t
*size)This probably breaks at least
fat_write.c, which uses:
memcpy(get_dentfromdir_block,get_contents_vfatname_block,With this patch, this line code "memcpy(get_dentfromdir_block,
get_contents_vfatname_block," is not required anymore since both
get_dentfromdir_blockandget_contents_vfatname_blockare sharing
the
same content and memory. So, this line code can be removed or leaving
in there. However, there is only one place withinfill_dir_slot_buffer
function where it can corrupt the the global memory, and it is fixed byQuoted 8 lines
replacing with local buffer. This was sent out with another patch called "[PATCH 1/2]
fs/fat/fatwrite: Local variable as buffer to storedir_slotentries". Overall, applying these two patches, a lot memory
space can be saved and fitting into small OCRAM, but need to be very
careful when modifying the code related to global memory.Quoted 2 lines
I get the point, but I am a bit concerned because these changes make the code even more fragile and hard to maintain than it currently is.
Yeah, i agree with you, and there is trade-off in saving the memory space.
Quoted 8 lines
Quoted 4 lines
Perhaps it would be time to switch to FatFs as previously suggested. Here is its memory usage:
http://elm-chan.org/fsw/ff/en/appnote.html#memoryCool. If i am not mistaken, this implementation would impact a lot of areas, especially interface level. What's about the testing? I am still voting for this simple patch changes, until we have enough resources to do the switching.
I think switching to the FF library is a non-starter. It's excruciatingly slow since it always reads even contiguous files one cluster at a time. I did propose a change to the library, but the maintainer didn't seem interested in fixing the problem. If we were to switch, the Tianocore implementation might be worth looking at, now they've fixed the license of the FAT code to remove the requirement that it only be used in EFI implementations. I haven't looked the code to know whether it'd be possible/good to switch though.
PATCH 1/2 fs/fat/fatwrite: Local variable as buffer to store dir_slot entries
In reply to Tien Fong CheeQuoted 9 lines · Tien Fong Chee
fill_dir_slotuseget_contents_vfatname_blockas a temporary buffer for constructing a list ofdir_slotentries. To save the memory and providing correct type of memory for above usage, a local buffer with accurate size declaration is introduced.The local array size 640 is used because for long file name entry, each entry use 32 bytes, one entry can store up to 13 characters. The maximum number of entry possible is 20. So, total size is 32*20=640bytes.
I'm fairly sure this series is correct[1], although the FAT write code is a little hard to follow.
[1] except in the face of disk read errors or corrupted data while parsing long filename entries, but the code already looks broken in the case of disk errors, and the corrupted data case probably isn't much worse now.
include/fat.h line 41
Quoted 8 lines
include/fat.h +3
-0+/* Maximum number of entry for long file name according to spec */ +#define MAX_LFN_SLOT 20 +/* Filesystem identifiers */ #define FAT12_SIGN "FAT12 "
Why 2 blank lines there?
Also, I'd suggest simply deleting get_contents_vfatname_block and renaming all references to it to use get_dentfromdir_block instead. That way, anyone reading the code in the future won't assume the two "block" variables refer to different memory, when in fact they share storage. Related, there's little point keeping the now-redundant memcpy() call at the end of get_long_file_name():
Quoted 2 lines
memcpy(get_dentfromdir_block, get_contents_vfatname_block, mydata->clust_size * mydata->sect_size);
PATCH 1/2 fs/fat/fatwrite: Local variable as buffer to store dir_slot entries
Quoted 26 lines · Stephen Warren
Quoted 8 lines · Tien Fong Chee
fill_dir_slotuseget_contents_vfatname_blockas a temporary buffer for
constructing a list ofdir_slotentries. To save the memory and providing
correct type of memory for above usage, a local buffer with accurate size
declaration is introduced.Quoted 4 lines
The local array size 640 is used because for long file name entry, each entry use 32 bytes, one entry can store up to 13 characters. The maximum number of entry possible is 20. So, total size is 32*20=640bytes.
I'm fairly sure this series is correct[1], although the FAT write code
is a little hard to follow.[1] except in the face of disk read errors or corrupted data while parsing long filename entries, but the code already looks broken in the
case of disk errors, and the corrupted data case probably isn't much worse now.Quoted 1 line
include/fat.h +0
-0
Quoted 8 lines
include/fat.h line 41
Quoted 6 lines
include/fat.h +3
-0+/* Maximum number of entry for long file name according to spec */ +#define MAX_LFN_SLOT 20 +/* Filesystem identifiers */ #define FAT12_SIGN "FAT12 "Why 2 blank lines there?
Good catch!!
Quoted 15 lines
Also, I'd suggest simply deleting
get_contents_vfatname_blockand renaming all references to it to useget_dentfromdir_blockinstead. That
way, anyone reading the code in the future won't assume the two "block"
variables refer to different memory, when in fact they share storage. Related, there's little point keeping the now-redundantmemcpy()call at
the end ofget_long_file_name():Quoted 3 lines
memcpy(get_dentfromdir_block,
get_contents_vfatname_block,mydata->clust_size * mydata->sect_size);
Yeah, i agree with you that we should renaming all to single name to avoid any confusion. I would remove those redundant codes.
Thanks.