Skip to contentSkip to conversation

xfsrestore: fix inventory unpacking

PATCH Superseded Donald Douwsma 2 participants 12 messages Oct 13, 2022 – Oct 16, 2022

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover xfsrestore: fix inventory unpacking

When xfsrestore reads its inventory from tape it fails to convert the media record on bigendian systems, if the online inventory is unavailable this results in invalid data being writen to the online inventory and failure to restore non-directory files.

The series fixes the converstion and related issues.

---
v2
- Seperate out cleanup and content.c changes, fix whitespace.
- Show a full reproducer in the first patch.
V3
- Fix whitespace and Signed-off-by.
- Make for loop formatting consistent.
- Rename patches to make their intent clearer.
- Add xfsdump: fix on-media inventory stream packing.
- Add descriptions that reproduce the stream problems.

Donald Douwsma (4):
xfsrestore: fix on-media inventory media unpacking
xfsrestore: fix on-media inventory stream unpacking
xfsdump: fix on-media inventory stream packing
xfsrestore: untangle inventory unpacking logic

 inventory/inv_stobj.c | 42 +++++++++++++++---------------------------
 restore/content.c     | 13 +++++--------
 2 files changed, 20 insertions(+), 35 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v3 1/4 xfsrestore: fix on-media inventory media unpacking

When xfsrestore reads the inventory from tape media it fails to convert media file records from bigendian. If the xfsdump inventory is not available xfsrestore will write this invalid record to the on-line inventory.

[root@rhel8 ~]# xfsdump -L Test1 -M "" -f /dev/st0 /boot > /dev/null
[root@rhel8 ~]# xfsdump -L Test2 -M "" -f /dev/st0 /boot > /dev/null
[root@rhel8 ~]# rm -rf /var/lib/xfsdump/inventory/
[root@rhel8 ~]# mt -f /dev/nst0 asf 2
[root@rhel8 ~]# xfsrestore -f /dev/nst0 -L Test2 /tmp/test2
xfsrestore: using scsi tape (drive_scsitape) strategy
xfsrestore: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: preparing drive
xfsrestore: examining media file 3
xfsrestore: found dump matching specified label:
xfsrestore: hostname: rhel8
xfsrestore: mount point: /boot
xfsrestore: volume: /dev/sda1
xfsrestore: session time: Tue Sep 27 16:05:28 2022
xfsrestore: level: 0
xfsrestore: session label: "Test2"
xfsrestore: media label: ""
xfsrestore: file system id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
xfsrestore: session id: 62402423-7ae0-49ed-8ecb-9e5bc7642b91
xfsrestore: media id: 47ba45ca-3417-4006-ab10-3dc6419b83e2
xfsrestore: incorporating on-media session inventory into online inventory
xfsrestore: /var/lib/xfsdump/inventory created
xfsrestore: using on-media session inventory
xfsrestore: searching media for directory dump
xfsrestore: rewinding
xfsrestore: examining media file 0
xfsrestore: inventory session uuid (62402423-7ae0-49ed-8ecb-9e5bc7642b91) does not match the media header's session uuid (1771d9e8-a1ba-4e87-a61e-f6be97e41b45)
xfsrestore: examining media file 1
xfsrestore: inventory session uuid (62402423-7ae0-49ed-8ecb-9e5bc7642b91) does not match the media header's session uuid (1771d9e8-a1ba-4e87-a61e-f6be97e41b45)
xfsrestore: examining media file 2
xfsrestore: reading directories
xfsrestore: 9 directories and 320 entries processed
xfsrestore: directory post-processing
xfsrestore: restore complete: 0 seconds elapsed
xfsrestore: Restore Summary:
xfsrestore: stream 0 /dev/nst0 OK (success)
xfsrestore: Restore Status: SUCCESS
[root@rhel8 ~]# xfsdump -I
file system 0:
fs id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
session 0:
mount point: rhel8:/boot
device: rhel8:/dev/sda1
time: Tue Sep 27 16:05:28 2022
session label: "Test2"
session id: 62402423-7ae0-49ed-8ecb-9e5bc7642b91
level: 0
resumed: NO
subtree: NO
streams: 1
stream 0:
pathname: /dev/st0
start: ino 133 offset 0
end: ino 1572997 offset 0
interrupted: YES
media files: 1
media file 0:
mfile index: 33554432
mfile type: data
mfile size: 211187836911616
mfile start: ino 9583660007044415488 offset 0
mfile end: ino 9583686395323482112 offset 0
media label: ""
media id: 47ba45ca-3417-4006-ab10-3dc6419b83e2
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]#
[root@rhel8 ~]# ls /tmp/test2
efi grub2 loader

The invalid start and end inode information cause xfsrestore to consider that non-directory files do not reside in the current media and will fail to restore them.

The behaviour of an initial restore may succeed if the position of the tape is such that the data file is encountered before the inventory file, or if there is only one dump session on tape, xfsrestore is somewhat inconsistent in this regard. Subsequent restores will use the invalid on-line inventory and fail to restore files.

Fix this by correctly unpacking the inventory data.

---
 inventory/inv_stobj.c | 27 +++++++--------------------
 1 file changed, 7 insertions(+), 20 deletions(-)
inventory/inv_stobj.c +7 -20
index c20e71c..b461666 100644
@@ -1008,7 +1008,7 @@ stobj_unpack_sessinfo(
         size_t             bufsz,
 	invt_sessinfo_t   *s)
 {
-	uint 		 i;
+	uint 		 i, j;
 	char	         *tmpbuf;
 	char 		 *p = (char *)bufp;
 
@@ -1087,26 +1087,13 @@ stobj_unpack_sessinfo(
 
 	/* all the media files */
 	s->mfiles = (invt_mediafile_t *)p;
-
-#ifdef INVT_DELETION
-	{
-		int tmpfd = open("moids", O_RDWR | O_CREAT, S_IRUSR|S_IWUSR);
-		uint j;
-		invt_mediafile_t *mmf = s->mfiles;
-		for (i=0; i< s->ses->s_cur_nstreams; i++) {
-			for (j=0; j< s->strms[i].st_nmediafiles;
-			     j++, mmf++)
-				xlate_invt_mediafile((invt_mediafile_t *)mmf, (invt_mediafile_t *)tmpbuf, 1);
-				bcopy(tmpbuf, mmf, sizeof(invt_mediafile_t));
-				put_invtrecord(tmpfd, &mmf->mf_moid,
-					 sizeof(uuid_t), 0, SEEK_END, 0);
-		}
-		close(tmpfd);
-	}
-#endif
 	for (i = 0; i < s->ses->s_cur_nstreams; i++) {
-		p += (size_t) (s->strms[i].st_nmediafiles)
-			* sizeof(invt_mediafile_t);
+		for(j = 0; j < s->strms[i].st_nmediafiles; j++) {
+			xlate_invt_mediafile((invt_mediafile_t *)p,
+					     (invt_mediafile_t *)tmpbuf, 1);
+			bcopy(tmpbuf, p, sizeof(invt_mediafile_t));
+			p +=  sizeof(invt_mediafile_t);
+		}
 	}
 
 	/* sanity check the size of the buffer given to us vs. the size it
Signature · 1 line
2.31.1
#2 lore raw

PATCH v3 2/4 xfsrestore: fix on-media inventory stream unpacking

In reply to Donald Douwsma

xfsdump can create multiple streams, when restoring the online inventory with multiple streams we fail to process these and assert when the inventory buffer is not fully decoded.

[root@rhel8 ~]# xfsdump -L "Test1" -f /dev/nst0 -M tape1 -f /dev/nst1 -M tape2 /boot

xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsdump: level 0 dump of rhel8:/boot
xfsdump: dump date: Thu Oct  6 13:50:45 2022
xfsdump: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsdump: session label: "Test1"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: identifying stream starting points
xfsdump: stream 0: ino 133 offset 0 to ino 28839 offset 0
xfsdump: stream 1: ino 28839 offset 0 to end
xfsdump: ino map construction complete
xfsdump: estimated dump size: 328720704 bytes
xfsdump: estimated dump size per stream: 164375728 bytes
xfsdump: /var/lib/xfsdump/inventory created
xfsdump: drive 0: preparing drive
xfsdump: drive 1: preparing drive
xfsdump: drive 1: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 1: dumping ino map
xfsdump: drive 1: dumping non-directory files
xfsdump: drive 0: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 0: dumping ino map
xfsdump: drive 0: dumping directories
xfsdump: drive 0: dumping non-directory files
xfsdump: drive 1: ending media file
xfsdump: drive 1: media file size 166723584 bytes
xfsdump: drive 1: waiting for synchronized session inventory dump
xfsdump: drive 0: ending media file
xfsdump: drive 0: media file size 165675008 bytes
xfsdump: drive 0: waiting for synchronized session inventory dump
xfsdump: drive 0: dumping session inventory
xfsdump: drive 0: beginning inventory media file
xfsdump: drive 0: media file 1 (media 0, file 1)
xfsdump: drive 0: ending inventory media file
xfsdump: drive 0: inventory media file size 2097152 bytes
xfsdump: drive 0: writing stream terminator
xfsdump: drive 0: beginning media stream terminator
xfsdump: drive 0: media file 2 (media 0, file 2)
xfsdump: drive 0: ending media stream terminator
xfsdump: drive 0: media stream terminator size 1048576 bytes
xfsdump: drive 1: dumping session inventory
xfsdump: drive 1: beginning inventory media file
xfsdump: drive 1: media file 1 (media 0, file 1)
xfsdump: drive 1: ending inventory media file
xfsdump: drive 1: inventory media file size 2097152 bytes
xfsdump: drive 1: writing stream terminator
xfsdump: drive 1: beginning media stream terminator
xfsdump: drive 1: media file 2 (media 0, file 2)
xfsdump: drive 1: ending media stream terminator
xfsdump: drive 1: media stream terminator size 1048576 bytes
xfsdump: dump size (non-dir files) : 328189016 bytes
xfsdump: dump complete: 4 seconds elapsed
xfsdump: Dump Summary:
xfsdump:   stream 0 /dev/nst0 OK (success)
xfsdump:   stream 1 /dev/nst1 OK (success)
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# xfsdump -I
file system 0:
	fs id:		26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
	session 0:
		mount point:	rhel8:/boot
		device:		rhel8:/dev/sda1
		time:		Thu Oct  6 13:50:45 2022
		session label:	"Test1"
		session id:	aa25fa48-4493-45c7-9027-61e53e486445
		level:		0
		resumed:	NO
		subtree:	NO
		streams:	2
		stream 0:
			pathname:	/dev/nst0
			start:		ino 133 offset 0
			end:		ino 28839 offset 0
			interrupted:	NO
			media files:	2
			media file 0:
				mfile index:	0
				mfile type:	data
				mfile size:	165675008
				mfile start:	ino 133 offset 0
				mfile end:	ino 28839 offset 0
				media label:	"tape1"
				media id:	adb31f2a-f026-4597-a20a-326f28ecbaf1
			media file 1:
				mfile index:	1
				mfile type:	inventory
				mfile size:	2097152
				media label:	"tape1"
				media id:	adb31f2a-f026-4597-a20a-326f28ecbaf1
		stream 1:
			pathname:	/dev/nst1
			start:		ino 28839 offset 0
			end:		ino 1572997 offset 0
			interrupted:	NO
			media files:	2
			media file 0:
				mfile index:	0
				mfile type:	data
				mfile size:	166723584
				mfile start:	ino 28839 offset 0
				mfile end:	ino 1572997 offset 0
				media label:	"tape2"
				media id:	22224f02-b6c7-47d5-ad61-a61ba071c8a8
			media file 1:
				mfile index:	1
				mfile type:	inventory
				mfile size:	2097152
				media label:	"tape2"
				media id:	22224f02-b6c7-47d5-ad61-a61ba071c8a8

xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# mv /var/lib/xfsdump/inventory /var/lib/xfsdump/inventory_two_sessions [root@rhel8 ~]# xfsdump -I
xfsdump: Dump Status: SUCCESS

[root@rhel8 ~]# xfsrestore -L Test1 -f /dev/nst0 /tmp/test1/
xfsrestore: using scsi tape (drive_scsitape) strategy
xfsrestore: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: preparing drive
xfsrestore: examining media file 2
xfsrestore: found dump matching specified label:
xfsrestore: hostname: rhel8
xfsrestore: mount point: /boot
xfsrestore: volume: /dev/sda1
xfsrestore: session time: Thu Oct 6 13:50:45 2022
xfsrestore: level: 0
xfsrestore: session label: "Test1"
xfsrestore: media label: "tape1"
xfsrestore: file system id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
xfsrestore: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsrestore: media id: adb31f2a-f026-4597-a20a-326f28ecbaf1
xfsrestore: searching media for directory dump
xfsrestore: rewinding
xfsrestore: examining media file 0
xfsrestore: reading directories
xfsrestore: 9 directories and 320 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: examining media file 1
xfsrestore: inv_stobj.c:1119: stobj_unpack_sessinfo: Assertion `(size_t) ( p - (char *) bufp ) == bufsz' failed. Aborted (core dumped)

Make sure we unpack multiple streams when restoring the online inventory from media.

---
 inventory/inv_stobj.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
inventory/inv_stobj.c +7 -6
index b461666..025d431 100644
@@ -1065,25 +1065,26 @@ stobj_unpack_sessinfo(
 		return BOOL_FALSE;
 	}
 
+	/* get the seshdr and then, the remainder of the session */
 	xlate_invt_seshdr((invt_seshdr_t *)p, (invt_seshdr_t *)tmpbuf, 1);
 	bcopy(tmpbuf, p, sizeof(invt_seshdr_t));
-
-	/* get the seshdr and then, the remainder of the session */
 	s->seshdr = (invt_seshdr_t *)p;
 	s->seshdr->sh_sess_off = -1;
 	p += sizeof(invt_seshdr_t);
 
-
 	xlate_invt_session((invt_session_t *)p, (invt_session_t *)tmpbuf, 1);
 	bcopy (tmpbuf, p, sizeof(invt_session_t));
 	s->ses = (invt_session_t *)p;
 	p += sizeof(invt_session_t);
 
 	/* the array of all the streams belonging to this session */
-	xlate_invt_stream((invt_stream_t *)p, (invt_stream_t *)tmpbuf, 1);
-	bcopy(tmpbuf, p, sizeof(invt_stream_t));
 	s->strms = (invt_stream_t *)p;
-	p += s->ses->s_cur_nstreams * sizeof(invt_stream_t);
+	for (i = 0; i < s->ses->s_cur_nstreams; i++) {
+		xlate_invt_stream((invt_stream_t *)p,
+				  (invt_stream_t *)tmpbuf, 1);
+		bcopy(tmpbuf, p, sizeof(invt_stream_t));
+		p += sizeof(invt_stream_t);
+	}
 
 	/* all the media files */
 	s->mfiles = (invt_mediafile_t *)p;
Signature · 1 line
2.31.1
Reviewed-by on v3 2/4
#3 lore raw

Re: [PATCH 2/4] xfsrestore: fix on-media inventory stream unpacking

Quoted 148 lines · Donald Douwsma

xfsdump can create multiple streams, when restoring the online inventory with multiple streams we fail to process these and assert when the inventory buffer is not fully decoded.

[root@rhel8 ~]# xfsdump -L "Test1" -f /dev/nst0 -M tape1 -f /dev/nst1 -M tape2 /boot

xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsdump: level 0 dump of rhel8:/boot
xfsdump: dump date: Thu Oct  6 13:50:45 2022
xfsdump: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsdump: session label: "Test1"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: identifying stream starting points
xfsdump: stream 0: ino 133 offset 0 to ino 28839 offset 0
xfsdump: stream 1: ino 28839 offset 0 to end
xfsdump: ino map construction complete
xfsdump: estimated dump size: 328720704 bytes
xfsdump: estimated dump size per stream: 164375728 bytes
xfsdump: /var/lib/xfsdump/inventory created
xfsdump: drive 0: preparing drive
xfsdump: drive 1: preparing drive
xfsdump: drive 1: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 1: dumping ino map
xfsdump: drive 1: dumping non-directory files
xfsdump: drive 0: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 0: dumping ino map
xfsdump: drive 0: dumping directories
xfsdump: drive 0: dumping non-directory files
xfsdump: drive 1: ending media file
xfsdump: drive 1: media file size 166723584 bytes
xfsdump: drive 1: waiting for synchronized session inventory dump
xfsdump: drive 0: ending media file
xfsdump: drive 0: media file size 165675008 bytes
xfsdump: drive 0: waiting for synchronized session inventory dump
xfsdump: drive 0: dumping session inventory
xfsdump: drive 0: beginning inventory media file
xfsdump: drive 0: media file 1 (media 0, file 1)
xfsdump: drive 0: ending inventory media file
xfsdump: drive 0: inventory media file size 2097152 bytes
xfsdump: drive 0: writing stream terminator
xfsdump: drive 0: beginning media stream terminator
xfsdump: drive 0: media file 2 (media 0, file 2)
xfsdump: drive 0: ending media stream terminator
xfsdump: drive 0: media stream terminator size 1048576 bytes
xfsdump: drive 1: dumping session inventory
xfsdump: drive 1: beginning inventory media file
xfsdump: drive 1: media file 1 (media 0, file 1)
xfsdump: drive 1: ending inventory media file
xfsdump: drive 1: inventory media file size 2097152 bytes
xfsdump: drive 1: writing stream terminator
xfsdump: drive 1: beginning media stream terminator
xfsdump: drive 1: media file 2 (media 0, file 2)
xfsdump: drive 1: ending media stream terminator
xfsdump: drive 1: media stream terminator size 1048576 bytes
xfsdump: dump size (non-dir files) : 328189016 bytes
xfsdump: dump complete: 4 seconds elapsed
xfsdump: Dump Summary:
xfsdump:   stream 0 /dev/nst0 OK (success)
xfsdump:   stream 1 /dev/nst1 OK (success)
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# xfsdump -I
file system 0:
	fs id:		26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
	session 0:
		mount point:	rhel8:/boot
		device:		rhel8:/dev/sda1
		time:		Thu Oct  6 13:50:45 2022
		session label:	"Test1"
		session id:	aa25fa48-4493-45c7-9027-61e53e486445
		level:		0
		resumed:	NO
		subtree:	NO
		streams:	2
		stream 0:
			pathname:	/dev/nst0
			start:		ino 133 offset 0
			end:		ino 28839 offset 0
			interrupted:	NO
			media files:	2
			media file 0:
				mfile index:	0
				mfile type:	data
				mfile size:	165675008
				mfile start:	ino 133 offset 0
				mfile end:	ino 28839 offset 0
				media label:	"tape1"
				media id:	adb31f2a-f026-4597-a20a-326f28ecbaf1
			media file 1:
				mfile index:	1
				mfile type:	inventory
				mfile size:	2097152
				media label:	"tape1"
				media id:	adb31f2a-f026-4597-a20a-326f28ecbaf1
		stream 1:
			pathname:	/dev/nst1
			start:		ino 28839 offset 0
			end:		ino 1572997 offset 0
			interrupted:	NO
			media files:	2
			media file 0:
				mfile index:	0
				mfile type:	data
				mfile size:	166723584
				mfile start:	ino 28839 offset 0
				mfile end:	ino 1572997 offset 0
				media label:	"tape2"
				media id:	22224f02-b6c7-47d5-ad61-a61ba071c8a8
			media file 1:
				mfile index:	1
				mfile type:	inventory
				mfile size:	2097152
				media label:	"tape2"
				media id:	22224f02-b6c7-47d5-ad61-a61ba071c8a8

xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# mv /var/lib/xfsdump/inventory /var/lib/xfsdump/inventory_two_sessions [root@rhel8 ~]# xfsdump -I
xfsdump: Dump Status: SUCCESS

[root@rhel8 ~]# xfsrestore -L Test1 -f /dev/nst0 /tmp/test1/
xfsrestore: using scsi tape (drive_scsitape) strategy
xfsrestore: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: preparing drive
xfsrestore: examining media file 2
xfsrestore: found dump matching specified label:
xfsrestore: hostname: rhel8
xfsrestore: mount point: /boot
xfsrestore: volume: /dev/sda1
xfsrestore: session time: Thu Oct 6 13:50:45 2022
xfsrestore: level: 0
xfsrestore: session label: "Test1"
xfsrestore: media label: "tape1"
xfsrestore: file system id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
xfsrestore: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsrestore: media id: adb31f2a-f026-4597-a20a-326f28ecbaf1
xfsrestore: searching media for directory dump
xfsrestore: rewinding
xfsrestore: examining media file 0
xfsrestore: reading directories
xfsrestore: 9 directories and 320 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: examining media file 1
xfsrestore: inv_stobj.c:1119: stobj_unpack_sessinfo: Assertion `(size_t) ( p - (char *) bufp ) == bufsz' failed. Aborted (core dumped)

Quoted 4 lines

Make sure we unpack multiple streams when restoring the online inventory from media.

Much better now, though I hope there's an fstest coming to make sure that multistream restore/dump work properly.

--D

Quoted 44 lines
---
 inventory/inv_stobj.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
inventory/inv_stobj.c +7 -6
index b461666..025d431 100644
@@ -1065,25 +1065,26 @@ stobj_unpack_sessinfo(
 		return BOOL_FALSE;
 	}
 
+	/* get the seshdr and then, the remainder of the session */
 	xlate_invt_seshdr((invt_seshdr_t *)p, (invt_seshdr_t *)tmpbuf, 1);
 	bcopy(tmpbuf, p, sizeof(invt_seshdr_t));
-
-	/* get the seshdr and then, the remainder of the session */
 	s->seshdr = (invt_seshdr_t *)p;
 	s->seshdr->sh_sess_off = -1;
 	p += sizeof(invt_seshdr_t);
 
-
 	xlate_invt_session((invt_session_t *)p, (invt_session_t *)tmpbuf, 1);
 	bcopy (tmpbuf, p, sizeof(invt_session_t));
 	s->ses = (invt_session_t *)p;
 	p += sizeof(invt_session_t);
 
 	/* the array of all the streams belonging to this session */
-	xlate_invt_stream((invt_stream_t *)p, (invt_stream_t *)tmpbuf, 1);
-	bcopy(tmpbuf, p, sizeof(invt_stream_t));
 	s->strms = (invt_stream_t *)p;
-	p += s->ses->s_cur_nstreams * sizeof(invt_stream_t);
+	for (i = 0; i < s->ses->s_cur_nstreams; i++) {
+		xlate_invt_stream((invt_stream_t *)p,
+				  (invt_stream_t *)tmpbuf, 1);
+		bcopy(tmpbuf, p, sizeof(invt_stream_t));
+		p += sizeof(invt_stream_t);
+	}
 
 	/* all the media files */
 	s->mfiles = (invt_mediafile_t *)p;
Signature · 1 line
2.31.1
#4 lore raw

Re: [PATCH 2/4] xfsrestore: fix on-media inventory stream unpacking

Quoted 149 lines · Darrick J. Wong
Quoted 148 lines · Donald Douwsma

xfsdump can create multiple streams, when restoring the online inventory with multiple streams we fail to process these and assert when the inventory buffer is not fully decoded.

[root@rhel8 ~]# xfsdump -L "Test1" -f /dev/nst0 -M tape1 -f /dev/nst1 -M tape2 /boot

xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsdump: level 0 dump of rhel8:/boot
xfsdump: dump date: Thu Oct  6 13:50:45 2022
xfsdump: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsdump: session label: "Test1"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: identifying stream starting points
xfsdump: stream 0: ino 133 offset 0 to ino 28839 offset 0
xfsdump: stream 1: ino 28839 offset 0 to end
xfsdump: ino map construction complete
xfsdump: estimated dump size: 328720704 bytes
xfsdump: estimated dump size per stream: 164375728 bytes
xfsdump: /var/lib/xfsdump/inventory created
xfsdump: drive 0: preparing drive
xfsdump: drive 1: preparing drive
xfsdump: drive 1: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 1: dumping ino map
xfsdump: drive 1: dumping non-directory files
xfsdump: drive 0: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 0: dumping ino map
xfsdump: drive 0: dumping directories
xfsdump: drive 0: dumping non-directory files
xfsdump: drive 1: ending media file
xfsdump: drive 1: media file size 166723584 bytes
xfsdump: drive 1: waiting for synchronized session inventory dump
xfsdump: drive 0: ending media file
xfsdump: drive 0: media file size 165675008 bytes
xfsdump: drive 0: waiting for synchronized session inventory dump
xfsdump: drive 0: dumping session inventory
xfsdump: drive 0: beginning inventory media file
xfsdump: drive 0: media file 1 (media 0, file 1)
xfsdump: drive 0: ending inventory media file
xfsdump: drive 0: inventory media file size 2097152 bytes
xfsdump: drive 0: writing stream terminator
xfsdump: drive 0: beginning media stream terminator
xfsdump: drive 0: media file 2 (media 0, file 2)
xfsdump: drive 0: ending media stream terminator
xfsdump: drive 0: media stream terminator size 1048576 bytes
xfsdump: drive 1: dumping session inventory
xfsdump: drive 1: beginning inventory media file
xfsdump: drive 1: media file 1 (media 0, file 1)
xfsdump: drive 1: ending inventory media file
xfsdump: drive 1: inventory media file size 2097152 bytes
xfsdump: drive 1: writing stream terminator
xfsdump: drive 1: beginning media stream terminator
xfsdump: drive 1: media file 2 (media 0, file 2)
xfsdump: drive 1: ending media stream terminator
xfsdump: drive 1: media stream terminator size 1048576 bytes
xfsdump: dump size (non-dir files) : 328189016 bytes
xfsdump: dump complete: 4 seconds elapsed
xfsdump: Dump Summary:
xfsdump:   stream 0 /dev/nst0 OK (success)
xfsdump:   stream 1 /dev/nst1 OK (success)
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# xfsdump -I
file system 0:
	fs id:		26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
	session 0:
		mount point:	rhel8:/boot
		device:		rhel8:/dev/sda1
		time:		Thu Oct  6 13:50:45 2022
		session label:	"Test1"
		session id:	aa25fa48-4493-45c7-9027-61e53e486445
		level:		0
		resumed:	NO
		subtree:	NO
		streams:	2
		stream 0:
			pathname:	/dev/nst0
			start:		ino 133 offset 0
			end:		ino 28839 offset 0
			interrupted:	NO
			media files:	2
			media file 0:
				mfile index:	0
				mfile type:	data
				mfile size:	165675008
				mfile start:	ino 133 offset 0
				mfile end:	ino 28839 offset 0
				media label:	"tape1"
				media id:	adb31f2a-f026-4597-a20a-326f28ecbaf1
			media file 1:
				mfile index:	1
				mfile type:	inventory
				mfile size:	2097152
				media label:	"tape1"
				media id:	adb31f2a-f026-4597-a20a-326f28ecbaf1
		stream 1:
			pathname:	/dev/nst1
			start:		ino 28839 offset 0
			end:		ino 1572997 offset 0
			interrupted:	NO
			media files:	2
			media file 0:
				mfile index:	0
				mfile type:	data
				mfile size:	166723584
				mfile start:	ino 28839 offset 0
				mfile end:	ino 1572997 offset 0
				media label:	"tape2"
				media id:	22224f02-b6c7-47d5-ad61-a61ba071c8a8
			media file 1:
				mfile index:	1
				mfile type:	inventory
				mfile size:	2097152
				media label:	"tape2"
				media id:	22224f02-b6c7-47d5-ad61-a61ba071c8a8

xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# mv /var/lib/xfsdump/inventory /var/lib/xfsdump/inventory_two_sessions [root@rhel8 ~]# xfsdump -I
xfsdump: Dump Status: SUCCESS

[root@rhel8 ~]# xfsrestore -L Test1 -f /dev/nst0 /tmp/test1/
xfsrestore: using scsi tape (drive_scsitape) strategy
xfsrestore: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: preparing drive
xfsrestore: examining media file 2
xfsrestore: found dump matching specified label:
xfsrestore: hostname: rhel8
xfsrestore: mount point: /boot
xfsrestore: volume: /dev/sda1
xfsrestore: session time: Thu Oct 6 13:50:45 2022
xfsrestore: level: 0
xfsrestore: session label: "Test1"
xfsrestore: media label: "tape1"
xfsrestore: file system id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
xfsrestore: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsrestore: media id: adb31f2a-f026-4597-a20a-326f28ecbaf1
xfsrestore: searching media for directory dump
xfsrestore: rewinding
xfsrestore: examining media file 0
xfsrestore: reading directories
xfsrestore: 9 directories and 320 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: examining media file 1
xfsrestore: inv_stobj.c:1119: stobj_unpack_sessinfo: Assertion `(size_t) ( p - (char *) bufp ) == bufsz' failed. Aborted (core dumped)

Quoted 7 lines
Quoted 4 lines

Make sure we unpack multiple streams when restoring the online inventory from media.

Much better now, though I hope there's an fstest coming to make sure that multistream restore/dump work properly.

That could be a little tricky.

xfstests currently only supports one TAPE_DEV (and a remote RMT_TAPE_DEV). So we'd need to add support for multiple tapes, something like

TAPE_DEV1=/dev/nst0
TAPE_DEV2=/dev/nst1

Then it can be hard to get access to a tape drive, let alone two.

Probably the easiest way to address that is to provide instructions on how to setup mhvtl, https://github.com/markh794/mhvtl and load the tapes before a test run with

[root@rhel8 ~]# systemctl start mhvtl.target
[root@rhel8 ~]# mtx -f /dev/sch0 load 1 0; mtx -f /dev/sch load 2 1 Loading media from Storage Element 1 into drive 0...done Loading media from Storage Element 2 into drive 1...done [root@rhel8 ~]# ./check -g xfs/tape

Not sure there's many people who test the tape group, so this may be generally useful.

mhvtl requires a kernel module to be built, not sure if that would bother QE folk when doing general tests, but for xfsdump userspace testing it should be fine.

Don

Quoted 49 lines

--D

Quoted 44 lines
---
  inventory/inv_stobj.c | 13 +++++++------
  1 file changed, 7 insertions(+), 6 deletions(-)
inventory/inv_stobj.c +7 -6
index b461666..025d431 100644
@@ -1065,25 +1065,26 @@ stobj_unpack_sessinfo(
  		return BOOL_FALSE;
  	}
 
+	/* get the seshdr and then, the remainder of the session */
  	xlate_invt_seshdr((invt_seshdr_t *)p, (invt_seshdr_t *)tmpbuf, 1);
  	bcopy(tmpbuf, p, sizeof(invt_seshdr_t));
-
-	/* get the seshdr and then, the remainder of the session */
  	s->seshdr = (invt_seshdr_t *)p;
  	s->seshdr->sh_sess_off = -1;
  	p += sizeof(invt_seshdr_t);
 
-
  	xlate_invt_session((invt_session_t *)p, (invt_session_t *)tmpbuf, 1);
  	bcopy (tmpbuf, p, sizeof(invt_session_t));
  	s->ses = (invt_session_t *)p;
  	p += sizeof(invt_session_t);
 
  	/* the array of all the streams belonging to this session */
-	xlate_invt_stream((invt_stream_t *)p, (invt_stream_t *)tmpbuf, 1);
-	bcopy(tmpbuf, p, sizeof(invt_stream_t));
  	s->strms = (invt_stream_t *)p;
-	p += s->ses->s_cur_nstreams * sizeof(invt_stream_t);
+	for (i = 0; i < s->ses->s_cur_nstreams; i++) {
+		xlate_invt_stream((invt_stream_t *)p,
+				  (invt_stream_t *)tmpbuf, 1);
+		bcopy(tmpbuf, p, sizeof(invt_stream_t));
+		p += sizeof(invt_stream_t);
+	}
 
  	/* all the media files */
  	s->mfiles = (invt_mediafile_t *)p;
Signature · 1 line
2.31.1
#5 lore raw

Re: [PATCH 2/4] xfsrestore: fix on-media inventory stream unpacking

Quoted 158 lines · Donald Douwsma
Quoted 155 lines · Darrick J. Wong
Quoted 153 lines · Donald Douwsma

xfsdump can create multiple streams, when restoring the online inventory with multiple streams we fail to process these and assert when the inventory buffer is not fully decoded.

[root@rhel8 ~]# xfsdump -L "Test1" -f /dev/nst0 -M tape1 -f /dev/nst1 -M tape2 /boot
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsdump: level 0 dump of rhel8:/boot
xfsdump: dump date: Thu Oct  6 13:50:45 2022
xfsdump: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsdump: session label: "Test1"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: identifying stream starting points
xfsdump: stream 0: ino 133 offset 0 to ino 28839 offset 0
xfsdump: stream 1: ino 28839 offset 0 to end
xfsdump: ino map construction complete
xfsdump: estimated dump size: 328720704 bytes
xfsdump: estimated dump size per stream: 164375728 bytes
xfsdump: /var/lib/xfsdump/inventory created
xfsdump: drive 0: preparing drive
xfsdump: drive 1: preparing drive
xfsdump: drive 1: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 1: dumping ino map
xfsdump: drive 1: dumping non-directory files
xfsdump: drive 0: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 0: dumping ino map
xfsdump: drive 0: dumping directories
xfsdump: drive 0: dumping non-directory files
xfsdump: drive 1: ending media file
xfsdump: drive 1: media file size 166723584 bytes
xfsdump: drive 1: waiting for synchronized session inventory dump
xfsdump: drive 0: ending media file
xfsdump: drive 0: media file size 165675008 bytes
xfsdump: drive 0: waiting for synchronized session inventory dump
xfsdump: drive 0: dumping session inventory
xfsdump: drive 0: beginning inventory media file
xfsdump: drive 0: media file 1 (media 0, file 1)
xfsdump: drive 0: ending inventory media file
xfsdump: drive 0: inventory media file size 2097152 bytes
xfsdump: drive 0: writing stream terminator
xfsdump: drive 0: beginning media stream terminator
xfsdump: drive 0: media file 2 (media 0, file 2)
xfsdump: drive 0: ending media stream terminator
xfsdump: drive 0: media stream terminator size 1048576 bytes
xfsdump: drive 1: dumping session inventory
xfsdump: drive 1: beginning inventory media file
xfsdump: drive 1: media file 1 (media 0, file 1)
xfsdump: drive 1: ending inventory media file
xfsdump: drive 1: inventory media file size 2097152 bytes
xfsdump: drive 1: writing stream terminator
xfsdump: drive 1: beginning media stream terminator
xfsdump: drive 1: media file 2 (media 0, file 2)
xfsdump: drive 1: ending media stream terminator
xfsdump: drive 1: media stream terminator size 1048576 bytes
xfsdump: dump size (non-dir files) : 328189016 bytes
xfsdump: dump complete: 4 seconds elapsed
xfsdump: Dump Summary:
xfsdump:   stream 0 /dev/nst0 OK (success)
xfsdump:   stream 1 /dev/nst1 OK (success)
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# xfsdump -I
file system 0:
    fs id:        26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
    session 0:
        mount point:    rhel8:/boot
        device:        rhel8:/dev/sda1
        time:        Thu Oct  6 13:50:45 2022
        session label:    "Test1"
        session id:    aa25fa48-4493-45c7-9027-61e53e486445
        level:        0
        resumed:    NO
        subtree:    NO
        streams:    2
        stream 0:
            pathname:    /dev/nst0
            start:        ino 133 offset 0
            end:        ino 28839 offset 0
            interrupted:    NO
            media files:    2
            media file 0:
                mfile index:    0
                mfile type:    data
                mfile size:    165675008
                mfile start:    ino 133 offset 0
                mfile end:    ino 28839 offset 0
                media label:    "tape1"
                media id:    adb31f2a-f026-4597-a20a-326f28ecbaf1
            media file 1:
                mfile index:    1
                mfile type:    inventory
                mfile size:    2097152
                media label:    "tape1"
                media id:    adb31f2a-f026-4597-a20a-326f28ecbaf1 stream 1:
            pathname:    /dev/nst1
            start:        ino 28839 offset 0
            end:        ino 1572997 offset 0
            interrupted:    NO
            media files:    2
            media file 0:
                mfile index:    0
                mfile type:    data
                mfile size:    166723584
                mfile start:    ino 28839 offset 0
                mfile end:    ino 1572997 offset 0
                media label:    "tape2"
                media id:    22224f02-b6c7-47d5-ad61-a61ba071c8a8
            media file 1:
                mfile index:    1
                mfile type:    inventory
                mfile size:    2097152
                media label:    "tape2"
                media id:    22224f02-b6c7-47d5-ad61-a61ba071c8a8
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# mv /var/lib/xfsdump/inventory /var/lib/xfsdump/inventory_two_sessions [root@rhel8 ~]# xfsdump -I
xfsdump: Dump Status: SUCCESS

[root@rhel8 ~]# xfsrestore -L Test1 -f /dev/nst0 /tmp/test1/
xfsrestore: using scsi tape (drive_scsitape) strategy
xfsrestore: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: preparing drive
xfsrestore: examining media file 2
xfsrestore: found dump matching specified label:
xfsrestore: hostname: rhel8
xfsrestore: mount point: /boot
xfsrestore: volume: /dev/sda1
xfsrestore: session time: Thu Oct  6 13:50:45 2022
xfsrestore: level: 0
xfsrestore: session label: "Test1"
xfsrestore: media label: "tape1"
xfsrestore: file system id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
xfsrestore: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsrestore: media id: adb31f2a-f026-4597-a20a-326f28ecbaf1
xfsrestore: searching media for directory dump
xfsrestore: rewinding
xfsrestore: examining media file 0
xfsrestore: reading directories
xfsrestore: 9 directories and 320 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: examining media file 1
xfsrestore: inv_stobj.c:1119: stobj_unpack_sessinfo: Assertion `(size_t) ( p - (char *) bufp ) == bufsz' failed. Aborted (core dumped)

Make sure we unpack multiple streams when restoring the online inventory from media.

Quoted 8 lines
Quoted 2 lines

Much better now, though I hope there's an fstest coming to make sure that multistream restore/dump work properly.

That could be a little tricky.

xfstests currently only supports one TAPE_DEV (and a remote RMT_TAPE_DEV). So we'd need to add support for multiple tapes, something like

Thinking about this some more RMT_TAPE_DEV may be ok. If streaming to multiple targets is meant to enable an offsite dump it could be done via remote i.e.

# xfsdump -L "Test3" -f /dev/nst0 -M "tape1" -f backup.example.com:/dev/nst0 -M "tape2" /

The existing remote tests xfs/037 and xfs/038 just use the device for a simple backup.

Or I could be inventing a never useful use case.

Don

Quoted 71 lines

TAPE_DEV1=/dev/nst0
TAPE_DEV2=/dev/nst1

Then it can be hard to get access to a tape drive, let alone two.

Probably the easiest way to address that is to provide instructions on how to setup mhvtl, https://github.com/markh794/mhvtl and load the tapes before a test run with

[root@rhel8 ~]# systemctl start mhvtl.target
[root@rhel8 ~]# mtx -f /dev/sch0 load 1 0; mtx -f /dev/sch load 2 1 Loading media from Storage Element 1 into drive 0...done Loading media from Storage Element 2 into drive 1...done [root@rhel8 ~]# ./check -g xfs/tape

Not sure there's many people who test the tape group, so this may be generally useful.

mhvtl requires a kernel module to be built, not sure if that would bother QE folk when doing general tests, but for xfsdump userspace testing it should be fine.

Don

Quoted 45 lines

--D

Quoted 40 lines

---
  inventory/inv_stobj.c | 13 +++++++------

  1 file changed, 7 insertions(+), 6 deletions(-)
inventory/inv_stobj.c +7 -6
index b461666..025d431 100644
@@ -1065,25 +1065,26 @@ stobj_unpack_sessinfo(
           return BOOL_FALSE;
       }
+    /* get the seshdr and then, the remainder of the session */
       xlate_invt_seshdr((invt_seshdr_t *)p, (invt_seshdr_t *)tmpbuf, 1);
       bcopy(tmpbuf, p, sizeof(invt_seshdr_t));
-
-    /* get the seshdr and then, the remainder of the session */
       s->seshdr = (invt_seshdr_t *)p;
       s->seshdr->sh_sess_off = -1;
       p += sizeof(invt_seshdr_t);
-
       xlate_invt_session((invt_session_t *)p, (invt_session_t *)tmpbuf, 1);
       bcopy (tmpbuf, p, sizeof(invt_session_t));
       s->ses = (invt_session_t *)p;
       p += sizeof(invt_session_t);
       /* the array of all the streams belonging to this session */
-    xlate_invt_stream((invt_stream_t *)p, (invt_stream_t *)tmpbuf, 1);
-    bcopy(tmpbuf, p, sizeof(invt_stream_t));
       s->strms = (invt_stream_t *)p;
-    p += s->ses->s_cur_nstreams * sizeof(invt_stream_t);
+    for (i = 0; i < s->ses->s_cur_nstreams; i++) {
+        xlate_invt_stream((invt_stream_t *)p,
+                  (invt_stream_t *)tmpbuf, 1);
+        bcopy(tmpbuf, p, sizeof(invt_stream_t));
+        p += sizeof(invt_stream_t);
+    }
       /* all the media files */
       s->mfiles = (invt_mediafile_t *)p;
Signature · 1 line
2.31.1
#6 lore raw

Re: [PATCH 2/4] xfsrestore: fix on-media inventory stream unpacking

Quoted 178 lines · Donald Douwsma
Quoted 162 lines · Donald Douwsma
Quoted 159 lines · Darrick J. Wong
Quoted 157 lines · Donald Douwsma

xfsdump can create multiple streams, when restoring the online inventory with multiple streams we fail to process these and assert when the inventory buffer is not fully decoded.

[root@rhel8 ~]# xfsdump -L "Test1" -f /dev/nst0 -M tape1 -f /dev/nst1 -M tape2 /boot
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsdump: level 0 dump of rhel8:/boot
xfsdump: dump date: Thu Oct  6 13:50:45 2022
xfsdump: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsdump: session label: "Test1"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: identifying stream starting points
xfsdump: stream 0: ino 133 offset 0 to ino 28839 offset 0
xfsdump: stream 1: ino 28839 offset 0 to end
xfsdump: ino map construction complete
xfsdump: estimated dump size: 328720704 bytes
xfsdump: estimated dump size per stream: 164375728 bytes
xfsdump: /var/lib/xfsdump/inventory created
xfsdump: drive 0: preparing drive
xfsdump: drive 1: preparing drive
xfsdump: drive 1: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 1: dumping ino map
xfsdump: drive 1: dumping non-directory files
xfsdump: drive 0: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 0: dumping ino map
xfsdump: drive 0: dumping directories
xfsdump: drive 0: dumping non-directory files
xfsdump: drive 1: ending media file
xfsdump: drive 1: media file size 166723584 bytes
xfsdump: drive 1: waiting for synchronized session inventory dump
xfsdump: drive 0: ending media file
xfsdump: drive 0: media file size 165675008 bytes
xfsdump: drive 0: waiting for synchronized session inventory dump
xfsdump: drive 0: dumping session inventory
xfsdump: drive 0: beginning inventory media file
xfsdump: drive 0: media file 1 (media 0, file 1)
xfsdump: drive 0: ending inventory media file
xfsdump: drive 0: inventory media file size 2097152 bytes
xfsdump: drive 0: writing stream terminator
xfsdump: drive 0: beginning media stream terminator
xfsdump: drive 0: media file 2 (media 0, file 2)
xfsdump: drive 0: ending media stream terminator
xfsdump: drive 0: media stream terminator size 1048576 bytes
xfsdump: drive 1: dumping session inventory
xfsdump: drive 1: beginning inventory media file
xfsdump: drive 1: media file 1 (media 0, file 1)
xfsdump: drive 1: ending inventory media file
xfsdump: drive 1: inventory media file size 2097152 bytes
xfsdump: drive 1: writing stream terminator
xfsdump: drive 1: beginning media stream terminator
xfsdump: drive 1: media file 2 (media 0, file 2)
xfsdump: drive 1: ending media stream terminator
xfsdump: drive 1: media stream terminator size 1048576 bytes
xfsdump: dump size (non-dir files) : 328189016 bytes
xfsdump: dump complete: 4 seconds elapsed
xfsdump: Dump Summary:
xfsdump:   stream 0 /dev/nst0 OK (success)
xfsdump:   stream 1 /dev/nst1 OK (success)
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# xfsdump -I
file system 0:
    fs id:        26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8 session 0:
        mount point:    rhel8:/boot
        device:        rhel8:/dev/sda1
        time:        Thu Oct  6 13:50:45 2022
        session label:    "Test1"
        session id:    aa25fa48-4493-45c7-9027-61e53e486445 level:        0
        resumed:    NO
        subtree:    NO
        streams:    2
        stream 0:
            pathname:    /dev/nst0
            start:        ino 133 offset 0
            end:        ino 28839 offset 0
            interrupted:    NO
            media files:    2
            media file 0:
                mfile index:    0
                mfile type:    data
                mfile size:    165675008
                mfile start:    ino 133 offset 0
                mfile end:    ino 28839 offset 0
                media label:    "tape1"
                media id:    adb31f2a-f026-4597-a20a-326f28ecbaf1 media file 1:
                mfile index:    1
                mfile type:    inventory
                mfile size:    2097152
                media label:    "tape1"
                media id:    adb31f2a-f026-4597-a20a-326f28ecbaf1 stream 1:
            pathname:    /dev/nst1
            start:        ino 28839 offset 0
            end:        ino 1572997 offset 0
            interrupted:    NO
            media files:    2
            media file 0:
                mfile index:    0
                mfile type:    data
                mfile size:    166723584
                mfile start:    ino 28839 offset 0
                mfile end:    ino 1572997 offset 0
                media label:    "tape2"
                media id:    22224f02-b6c7-47d5-ad61-a61ba071c8a8 media file 1:
                mfile index:    1
                mfile type:    inventory
                mfile size:    2097152
                media label:    "tape2"
                media id:    22224f02-b6c7-47d5-ad61-a61ba071c8a8
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# mv /var/lib/xfsdump/inventory /var/lib/xfsdump/inventory_two_sessions
[root@rhel8 ~]# xfsdump -I
xfsdump: Dump Status: SUCCESS

[root@rhel8 ~]# xfsrestore -L Test1 -f /dev/nst0 /tmp/test1/
xfsrestore: using scsi tape (drive_scsitape) strategy
xfsrestore: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: preparing drive
xfsrestore: examining media file 2
xfsrestore: found dump matching specified label:
xfsrestore: hostname: rhel8
xfsrestore: mount point: /boot
xfsrestore: volume: /dev/sda1
xfsrestore: session time: Thu Oct  6 13:50:45 2022
xfsrestore: level: 0
xfsrestore: session label: "Test1"
xfsrestore: media label: "tape1"
xfsrestore: file system id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
xfsrestore: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsrestore: media id: adb31f2a-f026-4597-a20a-326f28ecbaf1
xfsrestore: searching media for directory dump
xfsrestore: rewinding
xfsrestore: examining media file 0
xfsrestore: reading directories
xfsrestore: 9 directories and 320 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: examining media file 1
xfsrestore: inv_stobj.c:1119: stobj_unpack_sessinfo: Assertion `(size_t) ( p - (char *) bufp ) == bufsz' failed. Aborted (core dumped)

Make sure we unpack multiple streams when restoring the online inventory from media.

Quoted 8 lines
Quoted 2 lines

Much better now, though I hope there's an fstest coming to make sure that multistream restore/dump work properly.

That could be a little tricky.

xfstests currently only supports one TAPE_DEV (and a remote RMT_TAPE_DEV). So we'd need to add support for multiple tapes, something like

Thinking about this some more RMT_TAPE_DEV may be ok. If streaming to multiple targets is meant to enable an offsite dump it could be done via remote i.e.

Quoted 7 lines
# xfsdump -L "Test3" -f /dev/nst0 -M "tape1" -f backup.example.com:/dev/nst0
-M "tape2" /

The existing remote tests xfs/037 and xfs/038 just use the device for a simple backup.

Or I could be inventing a never useful use case.

I wonder, does the "remote" tape device have to be remote, or is xfsdump flexible enough to let you set up a dummy file to dump into?

I bet any fstest that actually requires a real tape device hasn't been run in years, except maybe by the RH QA department. :/

--D

Quoted 75 lines

Don

Quoted 72 lines

TAPE_DEV1=/dev/nst0
TAPE_DEV2=/dev/nst1

Then it can be hard to get access to a tape drive, let alone two.

Probably the easiest way to address that is to provide instructions on how to setup mhvtl, https://github.com/markh794/mhvtl and load the tapes before a test run with

[root@rhel8 ~]# systemctl start mhvtl.target
[root@rhel8 ~]# mtx -f /dev/sch0 load 1 0; mtx -f /dev/sch load 2 1 Loading media from Storage Element 1 into drive 0...done Loading media from Storage Element 2 into drive 1...done [root@rhel8 ~]# ./check -g xfs/tape

Not sure there's many people who test the tape group, so this may be generally useful.

mhvtl requires a kernel module to be built, not sure if that would bother QE folk when doing general tests, but for xfsdump userspace testing it should be fine.

Don

Quoted 46 lines

--D

Quoted 41 lines

---
  inventory/inv_stobj.c | 13 +++++++------

  1 file changed, 7 insertions(+), 6 deletions(-)
inventory/inv_stobj.c +1 -3
index b461666..025d431 100644
@@ -1065,25 +1065,26 @@ stobj_unpack_sessinfo(
           return BOOL_FALSE;
       }
+    /* get the seshdr and then, the remainder of the session */
       xlate_invt_seshdr((invt_seshdr_t *)p, (invt_seshdr_t *)tmpbuf, 1);
       bcopy(tmpbuf, p, sizeof(invt_seshdr_t));
-
-    /* get the seshdr and then, the remainder of the session */
       s->seshdr = (invt_seshdr_t *)p;
       s->seshdr->sh_sess_off = -1;
       p += sizeof(invt_seshdr_t);
-

      xlate_invt_session((invt_session_t *)p, (invt_session_t *)tmpbuf, 1);
      bcopy (tmpbuf, p, sizeof(invt_session_t));
      s->ses = (invt_session_t *)p;
      p += sizeof(invt_session_t);
      /* the array of all the streams belonging to this session */ -    xlate_invt_stream((invt_stream_t *)p, (invt_stream_t *)tmpbuf, 1); -    bcopy(tmpbuf, p, sizeof(invt_stream_t));
      s->strms = (invt_stream_t *)p;
-    p += s->ses->s_cur_nstreams * sizeof(invt_stream_t);
+    for (i = 0; i < s->ses->s_cur_nstreams; i++) {
+        xlate_invt_stream((invt_stream_t *)p,
+                  (invt_stream_t *)tmpbuf, 1);
+        bcopy(tmpbuf, p, sizeof(invt_stream_t));
+        p += sizeof(invt_stream_t);
+    }
      /* all the media files */
      s->mfiles = (invt_mediafile_t *)p;

Signature · 1 line
2.31.1
#7 lore raw

Re: [PATCH 2/4] xfsrestore: fix on-media inventory stream unpacking

Quoted 182 lines · Darrick J. Wong
Quoted 181 lines · Donald Douwsma
Quoted 162 lines · Donald Douwsma
Quoted 159 lines · Darrick J. Wong
Quoted 157 lines · Donald Douwsma

xfsdump can create multiple streams, when restoring the online inventory with multiple streams we fail to process these and assert when the inventory buffer is not fully decoded.

[root@rhel8 ~]# xfsdump -L "Test1" -f /dev/nst0 -M tape1 -f /dev/nst1 -M tape2 /boot
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: using scsi tape (drive_scsitape) strategy
xfsdump: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsdump: level 0 dump of rhel8:/boot
xfsdump: dump date: Thu Oct  6 13:50:45 2022
xfsdump: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsdump: session label: "Test1"
xfsdump: ino map phase 1: constructing initial dump list
xfsdump: ino map phase 2: skipping (no pruning necessary)
xfsdump: ino map phase 3: identifying stream starting points
xfsdump: stream 0: ino 133 offset 0 to ino 28839 offset 0
xfsdump: stream 1: ino 28839 offset 0 to end
xfsdump: ino map construction complete
xfsdump: estimated dump size: 328720704 bytes
xfsdump: estimated dump size per stream: 164375728 bytes
xfsdump: /var/lib/xfsdump/inventory created
xfsdump: drive 0: preparing drive
xfsdump: drive 1: preparing drive
xfsdump: drive 1: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 1: dumping ino map
xfsdump: drive 1: dumping non-directory files
xfsdump: drive 0: creating dump session media file 0 (media 0, file 0)
xfsdump: drive 0: dumping ino map
xfsdump: drive 0: dumping directories
xfsdump: drive 0: dumping non-directory files
xfsdump: drive 1: ending media file
xfsdump: drive 1: media file size 166723584 bytes
xfsdump: drive 1: waiting for synchronized session inventory dump
xfsdump: drive 0: ending media file
xfsdump: drive 0: media file size 165675008 bytes
xfsdump: drive 0: waiting for synchronized session inventory dump
xfsdump: drive 0: dumping session inventory
xfsdump: drive 0: beginning inventory media file
xfsdump: drive 0: media file 1 (media 0, file 1)
xfsdump: drive 0: ending inventory media file
xfsdump: drive 0: inventory media file size 2097152 bytes
xfsdump: drive 0: writing stream terminator
xfsdump: drive 0: beginning media stream terminator
xfsdump: drive 0: media file 2 (media 0, file 2)
xfsdump: drive 0: ending media stream terminator
xfsdump: drive 0: media stream terminator size 1048576 bytes
xfsdump: drive 1: dumping session inventory
xfsdump: drive 1: beginning inventory media file
xfsdump: drive 1: media file 1 (media 0, file 1)
xfsdump: drive 1: ending inventory media file
xfsdump: drive 1: inventory media file size 2097152 bytes
xfsdump: drive 1: writing stream terminator
xfsdump: drive 1: beginning media stream terminator
xfsdump: drive 1: media file 2 (media 0, file 2)
xfsdump: drive 1: ending media stream terminator
xfsdump: drive 1: media stream terminator size 1048576 bytes
xfsdump: dump size (non-dir files) : 328189016 bytes
xfsdump: dump complete: 4 seconds elapsed
xfsdump: Dump Summary:
xfsdump:   stream 0 /dev/nst0 OK (success)
xfsdump:   stream 1 /dev/nst1 OK (success)
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# xfsdump -I
file system 0:
    fs id:        26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8 session 0:
        mount point:    rhel8:/boot
        device:        rhel8:/dev/sda1
        time:        Thu Oct  6 13:50:45 2022
        session label:    "Test1"
        session id:    aa25fa48-4493-45c7-9027-61e53e486445 level:        0
        resumed:    NO
        subtree:    NO
        streams:    2
        stream 0:
            pathname:    /dev/nst0
            start:        ino 133 offset 0
            end:        ino 28839 offset 0
            interrupted:    NO
            media files:    2
            media file 0:
                mfile index:    0
                mfile type:    data
                mfile size:    165675008
                mfile start:    ino 133 offset 0
                mfile end:    ino 28839 offset 0
                media label:    "tape1"
                media id:    adb31f2a-f026-4597-a20a-326f28ecbaf1 media file 1:
                mfile index:    1
                mfile type:    inventory
                mfile size:    2097152
                media label:    "tape1"
                media id:    adb31f2a-f026-4597-a20a-326f28ecbaf1 stream 1:
            pathname:    /dev/nst1
            start:        ino 28839 offset 0
            end:        ino 1572997 offset 0
            interrupted:    NO
            media files:    2
            media file 0:
                mfile index:    0
                mfile type:    data
                mfile size:    166723584
                mfile start:    ino 28839 offset 0
                mfile end:    ino 1572997 offset 0
                media label:    "tape2"
                media id:    22224f02-b6c7-47d5-ad61-a61ba071c8a8 media file 1:
                mfile index:    1
                mfile type:    inventory
                mfile size:    2097152
                media label:    "tape2"
                media id:    22224f02-b6c7-47d5-ad61-a61ba071c8a8
xfsdump: Dump Status: SUCCESS
[root@rhel8 ~]# mv /var/lib/xfsdump/inventory /var/lib/xfsdump/inventory_two_sessions
[root@rhel8 ~]# xfsdump -I
xfsdump: Dump Status: SUCCESS

[root@rhel8 ~]# xfsrestore -L Test1 -f /dev/nst0 /tmp/test1/
xfsrestore: using scsi tape (drive_scsitape) strategy
xfsrestore: version 3.1.8 (dump format 3.0) - type ^C for status and control
xfsrestore: searching media for dump
xfsrestore: preparing drive
xfsrestore: examining media file 2
xfsrestore: found dump matching specified label:
xfsrestore: hostname: rhel8
xfsrestore: mount point: /boot
xfsrestore: volume: /dev/sda1
xfsrestore: session time: Thu Oct  6 13:50:45 2022
xfsrestore: level: 0
xfsrestore: session label: "Test1"
xfsrestore: media label: "tape1"
xfsrestore: file system id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
xfsrestore: session id: aa25fa48-4493-45c7-9027-61e53e486445
xfsrestore: media id: adb31f2a-f026-4597-a20a-326f28ecbaf1
xfsrestore: searching media for directory dump
xfsrestore: rewinding
xfsrestore: examining media file 0
xfsrestore: reading directories
xfsrestore: 9 directories and 320 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: examining media file 1
xfsrestore: inv_stobj.c:1119: stobj_unpack_sessinfo: Assertion `(size_t) ( p - (char *) bufp ) == bufsz' failed. Aborted (core dumped)

Make sure we unpack multiple streams when restoring the online inventory from media.

Quoted 8 lines
Quoted 2 lines

Much better now, though I hope there's an fstest coming to make sure that multistream restore/dump work properly.

That could be a little tricky.

xfstests currently only supports one TAPE_DEV (and a remote RMT_TAPE_DEV). So we'd need to add support for multiple tapes, something like

Thinking about this some more RMT_TAPE_DEV may be ok. If streaming to multiple targets is meant to enable an offsite dump it could be done via remote i.e.

# xfsdump -L "Test3" -f /dev/nst0 -M "tape1" -f backup.example.com:/dev/nst0
-M "tape2" /
Quoted 7 lines
Quoted 4 lines

The existing remote tests xfs/037 and xfs/038 just use the device for a simple backup.

Or I could be inventing a never useful use case.

I wonder, does the "remote" tape device have to be remote, or is xfsdump flexible enough to let you set up a dummy file to dump into?

xfsdump identifies a remote device by looking for a : and will chose which drive target to use based on this. I cant see anywhere that xfstests checks for the format of RMT_TAPE_DEV, so should work.

Quoted 2 lines

I bet any fstest that actually requires a real tape device hasn't been run in years, except maybe by the RH QA department. :/

I dont know if anyone is running it regularly. I know some folk at sgi tested with it in 2008, and I've been running it a bit while trying to track this problem down. All the tests that are run still pass (apart from a couple of Irix tests that we cant run).

Don

Quoted 77 lines
Quoted 76 lines

Don

Quoted 72 lines

TAPE_DEV1=/dev/nst0
TAPE_DEV2=/dev/nst1

Then it can be hard to get access to a tape drive, let alone two.

Probably the easiest way to address that is to provide instructions on how to setup mhvtl, https://github.com/markh794/mhvtl and load the tapes before a test run with

[root@rhel8 ~]# systemctl start mhvtl.target
[root@rhel8 ~]# mtx -f /dev/sch0 load 1 0; mtx -f /dev/sch load 2 1 Loading media from Storage Element 1 into drive 0...done Loading media from Storage Element 2 into drive 1...done [root@rhel8 ~]# ./check -g xfs/tape

Not sure there's many people who test the tape group, so this may be generally useful.

mhvtl requires a kernel module to be built, not sure if that would bother QE folk when doing general tests, but for xfsdump userspace testing it should be fine.

Don

Quoted 46 lines

--D

Quoted 41 lines
---
   inventory/inv_stobj.c | 13 +++++++------
   1 file changed, 7 insertions(+), 6 deletions(-)
inventory/inv_stobj.c +7 -6
index b461666..025d431 100644
@@ -1065,25 +1065,26 @@ stobj_unpack_sessinfo(
           return BOOL_FALSE;
       }
+    /* get the seshdr and then, the remainder of the session */
       xlate_invt_seshdr((invt_seshdr_t *)p, (invt_seshdr_t *)tmpbuf, 1);
       bcopy(tmpbuf, p, sizeof(invt_seshdr_t));
-
-    /* get the seshdr and then, the remainder of the session */
       s->seshdr = (invt_seshdr_t *)p;
       s->seshdr->sh_sess_off = -1;
       p += sizeof(invt_seshdr_t);
-
       xlate_invt_session((invt_session_t *)p, (invt_session_t
 *)tmpbuf, 1);
       bcopy (tmpbuf, p, sizeof(invt_session_t));
       s->ses = (invt_session_t *)p;
       p += sizeof(invt_session_t);
       /* the array of all the streams belonging to this session */
-    xlate_invt_stream((invt_stream_t *)p, (invt_stream_t *)tmpbuf, 1);
-    bcopy(tmpbuf, p, sizeof(invt_stream_t));
       s->strms = (invt_stream_t *)p;
-    p += s->ses->s_cur_nstreams * sizeof(invt_stream_t);
+    for (i = 0; i < s->ses->s_cur_nstreams; i++) {
+        xlate_invt_stream((invt_stream_t *)p,
+                  (invt_stream_t *)tmpbuf, 1);
+        bcopy(tmpbuf, p, sizeof(invt_stream_t));
+        p += sizeof(invt_stream_t);
+    }
       /* all the media files */
       s->mfiles = (invt_mediafile_t *)p;
Signature · 1 line
2.31.1
#8 lore raw

PATCH v3 3/4 xfsdump: fix on-media inventory stream packing

In reply to Donald Douwsma

With the on-media inventory now being restored for multiple streams we can see that the restored streams both claim to be for /dev/nst0.

[root@rhel8 xfsdump-dev]# xfsdump -L "Test2" -f /dev/nst0 -M "tape1" -f /dev/nst1 -M "tape2" /boot
...
[root@rhel8 xfsdump-dev]# rm -rf /var/lib/xfsdump/inventory /tmp/test1/* [root@rhel8 xfsdump-dev]# restore/xfsrestore -L Test2 -f /dev/nst0 -f /dev/nst1 /tmp/test2 restore/xfsrestore: using scsi tape (drive_scsitape) strategy
restore/xfsrestore: using scsi tape (drive_scsitape) strategy
restore/xfsrestore: version 3.1.10 (dump format 3.0) - type ^C for status and control
...

restore/xfsrestore: Restore Summary:
restore/xfsrestore:   stream 0 /dev/nst0 OK (success)
restore/xfsrestore:   stream 1 /dev/nst1 ALREADY_DONE (another stream completed the operation)
restore/xfsrestore: Restore Status: SUCCESS

[root@rhel8 xfsdump-dev]# xfsdump -I
file system 0:
fs id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
session 0:
mount point: rhel8:/boot
device: rhel8:/dev/sda1
time: Wed Oct 12 15:36:55 2022
session label: "Test2"
session id: 50be3b17-d9f9-414d-885b-ababf660e189
level: 0
resumed: NO
subtree: NO
streams: 2
stream 0:
pathname: /dev/nst0
start: ino 133 offset 0
end: ino 28839 offset 0
interrupted: YES
media files: 1
media file 0:
mfile index: 2
mfile type: data
mfile size: 165675008
mfile start: ino 133 offset 0
mfile end: ino 28839 offset 0
media label: "test"
media id: e2e6978d-5546-4f1f-8c9e-307168071889
stream 1:
pathname: /dev/nst0
start: ino 133 offset 0
end: ino 28839 offset 0
interrupted: YES
media files: 1
media file 0:
mfile index: 0
mfile type: data
mfile size: 166723584
mfile start: ino 28839 offset 0
mfile end: ino 1572997 offset 0
media label: "tape2"
media id: 1ad6d919-7159-42fb-a20f-5a2c4e3e24b1
xfsdump: Dump Status: SUCCESS
[root@rhel8 xfsdump-dev]#

Fix this by indexing the stream being packed for the on-media inventory.

---
 inventory/inv_stobj.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
inventory/inv_stobj.c +1 -1
index 025d431..fb4d93a 100644
@@ -798,7 +798,7 @@ stobj_pack_sessinfo(int fd, invt_session_t *ses, invt_seshdr_t *hdr,
 	sesbuf += sizeof(invt_session_t);
 
 	for (i = 0; i < ses->s_cur_nstreams; i++) {
-		xlate_invt_stream(strms, (invt_stream_t *)sesbuf, 1);
+		xlate_invt_stream(&strms[i], (invt_stream_t *)sesbuf, 1);
 		sesbuf += sizeof(invt_stream_t);
 	}
 
Signature · 1 line
2.31.1
Reviewed-by on v3 3/4
#9 lore raw

Re: [PATCH 3/4] xfsdump: fix on-media inventory stream packing

Quoted 33 lines · Donald Douwsma

With the on-media inventory now being restored for multiple streams we can see that the restored streams both claim to be for /dev/nst0.

[root@rhel8 xfsdump-dev]# xfsdump -L "Test2" -f /dev/nst0 -M "tape1" -f /dev/nst1 -M "tape2" /boot
...
[root@rhel8 xfsdump-dev]# rm -rf /var/lib/xfsdump/inventory /tmp/test1/* [root@rhel8 xfsdump-dev]# restore/xfsrestore -L Test2 -f /dev/nst0 -f /dev/nst1 /tmp/test2 restore/xfsrestore: using scsi tape (drive_scsitape) strategy
restore/xfsrestore: using scsi tape (drive_scsitape) strategy
restore/xfsrestore: version 3.1.10 (dump format 3.0) - type ^C for status and control
...

restore/xfsrestore: Restore Summary:
restore/xfsrestore:   stream 0 /dev/nst0 OK (success)
restore/xfsrestore:   stream 1 /dev/nst1 ALREADY_DONE (another stream completed the operation)
restore/xfsrestore: Restore Status: SUCCESS

[root@rhel8 xfsdump-dev]# xfsdump -I
file system 0:
fs id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
session 0:
mount point: rhel8:/boot
device: rhel8:/dev/sda1
time: Wed Oct 12 15:36:55 2022
session label: "Test2"
session id: 50be3b17-d9f9-414d-885b-ababf660e189
level: 0
resumed: NO
subtree: NO
streams: 2
stream 0:
pathname: /dev/nst0
start: ino 133 offset 0
end: ino 28839 offset 0
interrupted: YES

Quoted 8 lines

media files: 1
media file 0:
mfile index: 2
mfile type: data
mfile size: 165675008
mfile start: ino 133 offset 0
mfile end: ino 28839 offset 0
media label: "test"

It's odd that you have -M tape1 above but this ends up labelled "test"? If that isn't just a munged patch message, then that might need fixing (separate patch) as well.

The code change looks correct though. Thanks for fixing dump.

--D

Quoted 42 lines

media id: e2e6978d-5546-4f1f-8c9e-307168071889
stream 1:
pathname: /dev/nst0
start: ino 133 offset 0
end: ino 28839 offset 0
interrupted: YES
media files: 1
media file 0:
mfile index: 0
mfile type: data
mfile size: 166723584
mfile start: ino 28839 offset 0
mfile end: ino 1572997 offset 0
media label: "tape2"
media id: 1ad6d919-7159-42fb-a20f-5a2c4e3e24b1
xfsdump: Dump Status: SUCCESS
[root@rhel8 xfsdump-dev]#

Fix this by indexing the stream being packed for the on-media inventory.

---
 inventory/inv_stobj.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
inventory/inv_stobj.c +1 -1
index 025d431..fb4d93a 100644
@@ -798,7 +798,7 @@ stobj_pack_sessinfo(int fd, invt_session_t *ses, invt_seshdr_t *hdr,
 	sesbuf += sizeof(invt_session_t);
 
 	for (i = 0; i < ses->s_cur_nstreams; i++) {
-		xlate_invt_stream(strms, (invt_stream_t *)sesbuf, 1);
+		xlate_invt_stream(&strms[i], (invt_stream_t *)sesbuf, 1);
 		sesbuf += sizeof(invt_stream_t);
 	}
 
Signature · 1 line
2.31.1
#10 lore raw

Re: [PATCH 3/4] xfsdump: fix on-media inventory stream packing

Quoted 43 lines · Darrick J. Wong
Quoted 33 lines · Donald Douwsma

With the on-media inventory now being restored for multiple streams we can see that the restored streams both claim to be for /dev/nst0.

[root@rhel8 xfsdump-dev]# xfsdump -L "Test2" -f /dev/nst0 -M "tape1" -f /dev/nst1 -M "tape2" /boot
...
[root@rhel8 xfsdump-dev]# rm -rf /var/lib/xfsdump/inventory /tmp/test1/* [root@rhel8 xfsdump-dev]# restore/xfsrestore -L Test2 -f /dev/nst0 -f /dev/nst1 /tmp/test2 restore/xfsrestore: using scsi tape (drive_scsitape) strategy
restore/xfsrestore: using scsi tape (drive_scsitape) strategy
restore/xfsrestore: version 3.1.10 (dump format 3.0) - type ^C for status and control
...

restore/xfsrestore: Restore Summary:
restore/xfsrestore:   stream 0 /dev/nst0 OK (success)
restore/xfsrestore:   stream 1 /dev/nst1 ALREADY_DONE (another stream completed the operation)
restore/xfsrestore: Restore Status: SUCCESS

[root@rhel8 xfsdump-dev]# xfsdump -I
file system 0:
fs id: 26dd5aa0-b901-4cf5-9b68-0c5753cb3ab8
session 0:
mount point: rhel8:/boot
device: rhel8:/dev/sda1
time: Wed Oct 12 15:36:55 2022
session label: "Test2"
session id: 50be3b17-d9f9-414d-885b-ababf660e189
level: 0
resumed: NO
subtree: NO
streams: 2
stream 0:
pathname: /dev/nst0
start: ino 133 offset 0
end: ino 28839 offset 0
interrupted: YES

Quoted 8 lines

media files: 1
media file 0:
mfile index: 2
mfile type: data
mfile size: 165675008
mfile start: ino 133 offset 0
mfile end: ino 28839 offset 0
media label: "test"

Quoted 3 lines

It's odd that you have -M tape1 above but this ends up labelled "test"? If that isn't just a munged patch message, then that might need fixing (separate patch) as well.

I looked back over the terminals where I tested this and they're all tape1, I think this was an earlier cut n pasta mistake when I prepared the commit message. Its probably worth fixing the message because not everyone will want to setup a virtual tape library to look at this.

Thanks for the reviews,
Don

Quoted 49 lines

The code change looks correct though. Thanks for fixing dump.

--D

Quoted 42 lines

media id: e2e6978d-5546-4f1f-8c9e-307168071889
stream 1:
pathname: /dev/nst0
start: ino 133 offset 0
end: ino 28839 offset 0
interrupted: YES
media files: 1
media file 0:
mfile index: 0
mfile type: data
mfile size: 166723584
mfile start: ino 28839 offset 0
mfile end: ino 1572997 offset 0
media label: "tape2"
media id: 1ad6d919-7159-42fb-a20f-5a2c4e3e24b1
xfsdump: Dump Status: SUCCESS
[root@rhel8 xfsdump-dev]#

Fix this by indexing the stream being packed for the on-media inventory.

---
  inventory/inv_stobj.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
inventory/inv_stobj.c +1 -1
index 025d431..fb4d93a 100644
@@ -798,7 +798,7 @@ stobj_pack_sessinfo(int fd, invt_session_t *ses, invt_seshdr_t *hdr,
  	sesbuf += sizeof(invt_session_t);
 
  	for (i = 0; i < ses->s_cur_nstreams; i++) {
-		xlate_invt_stream(strms, (invt_stream_t *)sesbuf, 1);
+		xlate_invt_stream(&strms[i], (invt_stream_t *)sesbuf, 1);
  		sesbuf += sizeof(invt_stream_t);
  	}
 
Signature · 1 line
2.31.1
#11 lore raw

PATCH v3 4/4 xfsrestore: untangle inventory unpacking logic

In reply to Donald Douwsma

stobj_unpack_sessinfo returns bool_t, fix logic in pi_addfile so errors can be properly reported.

---
 restore/content.c | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)
restore/content.c +5 -8
index b3999f9..04b6f81 100644
@@ -5463,17 +5463,14 @@ pi_addfile(Media_t *Mediap,
 			 * desc.
 			 */
 			sessp = 0;
-			if (!buflen) {
-				ok = BOOL_FALSE;
-			} else {
-			    /* extract the session information from the buffer */
-			    if (stobj_unpack_sessinfo(bufp, buflen, &sessinfo)<0) {
-				ok = BOOL_FALSE;
-			    } else {
+			ok = BOOL_FALSE;
+			/* extract the session information from the buffer */
+			if (buflen &&
+			    stobj_unpack_sessinfo(bufp, buflen, &sessinfo)) {
 				stobj_convert_sessinfo(&sessp, &sessinfo);
 				ok = BOOL_TRUE;
-			    }
 			}
+
 			if (!ok || !sessp) {
 				mlog(MLOG_DEBUG | MLOG_WARNING | MLOG_MEDIA, _(
 				      "on-media session "
Signature · 1 line
2.31.1