Btrfs: send - sparse file support for btrfs-send mechanism
PATCH 1/1 Btrfs: send - sparse file support for btrfs-send mechanism
when send/receive a sparse file, the holes of the original file will be filled with zero. The holes will be sent as ZERO streams, and it's unnecessary.
So, I improved this by skipping the hole of file while sending.
--- fs/btrfs/send.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
fs/btrfs/send.c +6 -0
@@ -3718,6 +3718,7 @@ static int send_write_or_clone(struct send_ctx *sctx,u64 pos =0 ;u64 len;u32 l; +u64 bytenr;u8 type; ei = btrfs_item_ptr(path->nodes[0], path->slots[0],
@@ -3732,6 +3733,11 @@ static int send_write_or_clone(struct send_ctx *sctx,*/ len =PAGE_CACHE_ALIGN (len); }else { + bytenr =btrfs_file_extent_disk_bytenr (path->nodes[0 ], ei); +if (bytenr ==0 ) { + ret =0 ; + goto out; + } len =btrfs_file_extent_num_bytes (path->nodes[0 ], ei); }
Signature · 1 line
1.7.7.6
PATCH 1/1 Btrfs: send - pre-allocated file support for btrfs-send mechanism
Quoted 1 line
From
4222aa3a32cf4db161fcbbb87ad29b714f1bfaf2Mon Sep 17 00:00:00 2001
From: Chen Yang <[email protected]>
Date: Thu, 29 Nov 2012 13:22:09 +0800
Subject: [PATCH] Btrfs: send - pre-allocated file support for btrfs-send
mechanism
when send or receive a file which has pre-allocated part, the pre-allocated parts of the file will be sent as ZERO streams, and it's unnecessary.
So we can improve this by skip these EMPTY parts.
---
this patch is based on
"Btrfs: send - sparse file support for btrfs-send mechanism"
--- fs/btrfs/send.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)
fs/btrfs/send.c +6 -1
@@ -3732,13 +3732,18 @@ static int send_write_or_clone(struct send_ctx *sctx,* sure to send the whole thing */ len =PAGE_CACHE_ALIGN (len); - }else { + }else if (type ==BTRFS_FILE_EXTENT_REG ) { bytenr =btrfs_file_extent_disk_bytenr (path->nodes[0 ], ei);if (bytenr ==0 ) { ret =0 ; goto out; } len =btrfs_file_extent_num_bytes (path->nodes[0 ], ei); + }else if (type ==BTRFS_FILE_EXTENT_PREALLOC ) { + ret =0 ; + goto out; + }else { +BUG (); }if (offset + len > sctx->cur_inode_size)
Signature · 1 line
1.7.7.6
Re: [PATCH] Btrfs: send - pre-allocated file support for btrfs-send mechanism
Quoted 47 lines · Chen Yang
From
4222aa3a32cf4db161fcbbb87ad29b714f1bfaf2Mon Sep 17 00:00:00 2001
From: Chen Yang <[email protected]>
Date: Thu, 29 Nov 2012 13:22:09 +0800
Subject: [PATCH] Btrfs: send - pre-allocated file support for btrfs-send
mechanismwhen send or receive a file which has pre-allocated part, the pre-allocated parts of the file will be sent as ZERO streams, and it's unnecessary.
So we can improve this by skip these EMPTY parts.
Signed-off-by Cheng Yang <[email protected]>---
this patch is based on
"Btrfs: send - sparse file support for btrfs-send mechanism"--- fs/btrfs/send.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)fs/btrfs/send.c +6
-2@@ -3732,13 +3732,18 @@ static int send_write_or_clone(struct send_ctx *sctx,* sure to send the whole thing */ len =PAGE_CACHE_ALIGN (len); - }else { + }else if (type ==BTRFS_FILE_EXTENT_REG ) { bytenr =btrfs_file_extent_disk_bytenr (path->nodes[0 ], ei);if (bytenr ==0 ) { ret =0 ; goto out; } len =btrfs_file_extent_num_bytes (path->nodes[0 ], ei); + }else if (type ==BTRFS_FILE_EXTENT_PREALLOC ) { + ret =0 ; + goto out; + }else { +BUG (); }if (offset + len > sctx->cur_inode_size) --1.7.7.6
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
The same as in my previous mail applies here.
Re: [PATCH] Btrfs: send - pre-allocated file support for btrfs-send mechanism
Hi,
and also, I think, for PREALLOC extents, it is not enough just to skip sending the data, but possibly need to perform preallocation on the receive side, so a new SEND command might be needed.
Alex.
Quoted 58 lines · Alexander Block
Quoted 47 lines · Chen Yang
From
4222aa3a32cf4db161fcbbb87ad29b714f1bfaf2Mon Sep 17 00:00:00 2001
From: Chen Yang <[email protected]>
Date: Thu, 29 Nov 2012 13:22:09 +0800
Subject: [PATCH] Btrfs: send - pre-allocated file support for btrfs-send
mechanismwhen send or receive a file which has pre-allocated part, the pre-allocated parts of the file will be sent as ZERO streams, and it's unnecessary.
So we can improve this by skip these EMPTY parts.
Signed-off-by Cheng Yang <[email protected]>---
this patch is based on
"Btrfs: send - sparse file support for btrfs-send mechanism"--- fs/btrfs/send.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)fs/btrfs/send.c +6
-2@@ -3732,13 +3732,18 @@ static int send_write_or_clone(struct send_ctx *sctx,* sure to send the whole thing */ len =PAGE_CACHE_ALIGN (len); - }else { + }else if (type ==BTRFS_FILE_EXTENT_REG ) { bytenr =btrfs_file_extent_disk_bytenr (path->nodes[0 ], ei);if (bytenr ==0 ) { ret =0 ; goto out; } len =btrfs_file_extent_num_bytes (path->nodes[0 ], ei); + }else if (type ==BTRFS_FILE_EXTENT_PREALLOC ) { + ret =0 ; + goto out; + }else { +BUG (); }if (offset + len > sctx->cur_inode_size) --1.7.7.6
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlThe same as in my previous mail applies here.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 38 lines · Chen Yang
when send/receive a sparse file, the holes of the original file will be filled with zero. The holes will be sent as ZERO streams, and it's unnecessary.
So, I improved this by skipping the hole of file while sending.
Signed-off-by Cheng Yang <[email protected]>--- fs/btrfs/send.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)fs/btrfs/send.c +6
-1@@ -3718,6 +3718,7 @@ static int send_write_or_clone(struct send_ctx *sctx,u64 pos =0 ;u64 len;u32 l; +u64 bytenr;u8 type; ei = btrfs_item_ptr(path->nodes[0], path->slots[0],@@ -3732,6 +3733,11 @@ static int send_write_or_clone(struct send_ctx *sctx,*/ len =PAGE_CACHE_ALIGN (len); }else { + bytenr =btrfs_file_extent_disk_bytenr (path->nodes[0 ], ei); +if (bytenr ==0 ) { + ret =0 ; + goto out; + } len =btrfs_file_extent_num_bytes (path->nodes[0 ], ei); } --1.7.7.6
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
This won't work for incremental sends if I understand it correctly. If the hole got punched into the file after the initial send, the data will be unchanged on the receiving side when receiving incrementally.
Hi Chen, Alexander, Jan,
how about the following patch: it works for disknr==0 extents (to handle PREALLOC extents correctly, we need to add a new command, which will break existing btrfs-progs, because they are updated slower than the kernel side).
fs/btrfs/send.c +13 -0
@@ -3946,16 +3946,38 @@ static int process_extent(struct send_ctx *sctx,if (sctx->parent_root && !sctx->cur_inode_new) { ret =is_extent_unchanged (sctx, path, key);if (ret <0 ) goto out;if (ret) { ret =0 ; goto out; } + }else { +struct extent_buffer *eb; +struct btrfs_file_extent_item *ei; +u8 extent_type; +u64 extent_disknr; + + eb = path->nodes[0 ]; + ei =btrfs_item_ptr (eb, path->slots[0 ], + struct btrfs_file_extent_item); + + extent_type =btrfs_file_extent_type (eb, ei); + extent_disknr =btrfs_file_extent_disk_bytenr (eb, ei); + if (extent_type == BTRFS_FILE_EXTENT_REG &&
extent_disknr == 0) {
+ /*
+ * This is disknr=0 extent in a full-send or a new inode + * in a diff-send. Since we will send truncate command + * in finish_inode_if_needed anyways, the
inode size will be
+ * correct, and we don't have to send all-zero data.
+ */
+ ret = 0;
+ goto out;
+ }
}
ret = find_extent_clone(sctx, path, key->objectid, key->offset,
sctx->cur_inode_size, &found_clone);
if (ret != -ENOENT && ret < 0)
goto out;
ret = send_write_or_clone(sctx, path, key, found_clone);Thanks,
Alex.
Quoted 51 lines · Alexander Block
Quoted 38 lines · Chen Yang
when send/receive a sparse file, the holes of the original file will be filled with zero. The holes will be sent as ZERO streams, and it's unnecessary.
So, I improved this by skipping the hole of file while sending.
Signed-off-by Cheng Yang <[email protected]>--- fs/btrfs/send.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)fs/btrfs/send.c +6
-1@@ -3718,6 +3718,7 @@ static int send_write_or_clone(struct send_ctx *sctx,u64 pos =0 ;u64 len;u32 l; +u64 bytenr;u8 type; ei = btrfs_item_ptr(path->nodes[0], path->slots[0],@@ -3732,6 +3733,11 @@ static int send_write_or_clone(struct send_ctx *sctx,*/ len =PAGE_CACHE_ALIGN (len); }else { + bytenr =btrfs_file_extent_disk_bytenr (path->nodes[0 ], ei); +if (bytenr ==0 ) { + ret =0 ; + goto out; + } len =btrfs_file_extent_num_bytes (path->nodes[0 ], ei); } --1.7.7.6
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlThis won't work for incremental sends if I understand it correctly. If the hole got punched into the file after the initial send, the data will be unchanged on the receiving side when receiving incrementally. --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi,
I believe there are two more cases, where we can safely avoid shipping disknr=0 extent. These are cases, in which the file grows between the two snapshots, and some of the new extents are no-data extents. So the whole patch becomes:
fs/btrfs/send.c +26 -2
@@ -3839,17 +3839,18 @@ static int is_extent_unchanged(struct send_ctx *sctx,/* * Handle special case where the right side has no extents at all. */ eb = path->nodes[0 ]; slot = path->slots[0 ];btrfs_item_key_to_cpu (eb, &found_key, slot);if (found_key.objectid != key.objectid || found_key.type != key.type) { - ret =0 ; +/* No need to send a no-data extent it in this case */ + ret = (left_disknr ==0 ) ?1 :0 ; goto out; }/* * We're now on 2a, 2b or 7. */ key = found_key;while (key.offset < ekey->offset + left_len) {
@@ -3865,17 +3866,18 @@ static int is_extent_unchanged(struct send_ctx *sctx, goto out; }/* * Are we at extent 8? If yes, we know the extent is changed. * This may only happen on the first iteration. */ if (found_key.offset + right_len <= ekey->offset) { - ret =0 ; +/* No need to send a no-data extent it in this case */ + ret = (left_disknr ==0 ) ?1 :0 ; goto out; } left_offset_fixed = left_offset;if (key.offset < ekey->offset) {/* Fix the right offset for 2a and 7. */ right_offset += ekey->offset - key.offset; }else {
@@ -3946,16 +3948,38 @@ static int process_extent(struct send_ctx *sctx,if (sctx->parent_root && !sctx->cur_inode_new) { ret =is_extent_unchanged (sctx, path, key);if (ret <0 ) goto out;if (ret) { ret =0 ; goto out; } + }else { +struct extent_buffer *eb; +struct btrfs_file_extent_item *ei; +u8 extent_type; +u64 extent_disknr; + + eb = path->nodes[0 ]; + ei =btrfs_item_ptr (eb, path->slots[0 ], + struct btrfs_file_extent_item); + + extent_type =btrfs_file_extent_type (eb, ei); + extent_disknr =btrfs_file_extent_disk_bytenr (eb, ei); +if (extent_type ==BTRFS_FILE_EXTENT_REG && extent_disknr ==0 ) { +/* +* This is disknr=0 extent in a full-send or a new inode +* in a diff-send. Since we will send truncate command +* in finish_inode_if_needed anyways, the inode size will be +* correct, and we don't have to send all-zero data. +*/ + ret =0 ; + goto out; + } } ret =find_extent_clone (sctx, path, key->objectid, key->offset, sctx->cur_inode_size, &found_clone);if (ret != -ENOENT && ret <0 ) goto out; ret =send_write_or_clone (sctx, path, key, found_clone);
Alex.
Quoted 108 lines · Alex Lyakas
Hi Chen, Alexander, Jan,
how about the following patch: it works for disknr==0 extents (to handle PREALLOC extents correctly, we need to add a new command, which will break existing btrfs-progs, because they are updated slower than the kernel side).fs/btrfs/send.c +17
-0@@ -3946,16 +3946,38 @@ static int process_extent(struct send_ctx *sctx,if (sctx->parent_root && !sctx->cur_inode_new) { ret =is_extent_unchanged (sctx, path, key);if (ret <0 ) goto out;if (ret) { ret =0 ; goto out; } + }else { +struct extent_buffer *eb; +struct btrfs_file_extent_item *ei; +u8 extent_type; +u64 extent_disknr; + + eb = path->nodes[0 ]; + ei =btrfs_item_ptr (eb, path->slots[0 ], + struct btrfs_file_extent_item); + + extent_type =btrfs_file_extent_type (eb, ei); + extent_disknr =btrfs_file_extent_disk_bytenr (eb, ei); +if (extent_type ==BTRFS_FILE_EXTENT_REG && extent_disknr ==0 ) { +/* +* This is disknr=0 extent in a full-send or a new inode +* in a diff-send. Since we will send truncate command +* in finish_inode_if_needed anyways, the inode size will be
fs/btrfs/send.c +5
-0+* correct, and we don't have to send all-zero data. +*/ + ret =0 ; + goto out; + } } ret =find_extent_clone (sctx, path, key->objectid, key->offset, sctx->cur_inode_size, &found_clone);if (ret != -ENOENT && ret <0 ) goto out; ret =send_write_or_clone (sctx, path, key, found_clone);Thanks,
Alex.Quoted 51 lines · Alexander Block
Quoted 38 lines · Chen Yang
when send/receive a sparse file, the holes of the original file will be filled with zero. The holes will be sent as ZERO streams, and it's unnecessary.
So, I improved this by skipping the hole of file while sending.
Signed-off-by Cheng Yang <[email protected]>--- fs/btrfs/send.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)fs/btrfs/send.c +6
-1@@ -3718,6 +3718,7 @@ static int send_write_or_clone(struct send_ctx *sctx,u64 pos =0 ;u64 len;u32 l; +u64 bytenr;u8 type; ei = btrfs_item_ptr(path->nodes[0], path->slots[0],@@ -3732,6 +3733,11 @@ static int send_write_or_clone(struct send_ctx *sctx,*/ len =PAGE_CACHE_ALIGN (len); }else { + bytenr =btrfs_file_extent_disk_bytenr (path->nodes[0 ], ei); +if (bytenr ==0 ) { + ret =0 ; + goto out; + } len =btrfs_file_extent_num_bytes (path->nodes[0 ], ei); } --1.7.7.6
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlThis won't work for incremental sends if I understand it correctly. If the hole got punched into the file after the initial send, the data will be unchanged on the receiving side when receiving incrementally. --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html