Skip to contentSkip to conversation

punching holes in files

DISCUSSION Steve French 4 participants 5 messages Nov 1, 2007 – Nov 7, 2007 VFS

lore
Share & tools
lore
#0 lore raw

madvise_remove (in Linux) is used to free the backing store associated
with pages (punching a hole in a file). This is one of the vfs
operations that we do not send over the wire to Samba (so this call
would return -ENOSYS locally). Any thoughts on whether this could be
done with an obscure SetFileInfo level or FCNTL or whether it is worth adding to the CIFS POSIX Extensions?

A second interesting question is whether to implement fallocate over the wire. sys_fallocate takes a range (and also a flag which indicates whether to extend the size beyond end of file). SET_FILE_ALLOCATION_INFO takes a size field rather than a range. For the case of the range starting at zero, we could simply call Trans2
set pathinfo (or file info) with SET_FILE_ALLOCATION_INFO. Should we
add a CIFS POSIX Extension operation for this too?

By the way it looks like OCFS2 (not just ext4) implement these.

Signature · 3 lines
Thanks,

Steve
#1 lore raw
Quoted 6 lines · Steve French

madvise_remove (in Linux) is used to free the backing store associated
with pages (punching a hole in a file). This is one of the vfs
operations that we do not send over the wire to Samba (so this call
would return -ENOSYS locally). Any thoughts on whether this could be
done with an obscure SetFileInfo level or FCNTL or whether it is worth adding to the CIFS POSIX Extensions?

Don't know about SMB/CIFS but on Windows locally you would need to first set the file sparse, and then to punch the hole. These two things are accomplished like so on Windows (STARTING_OFFSET is the first byte of the hole to be punched and END_OFFSET is the first byte after the hole to be punched):

	HANDLE f; // This is obtained from a CreateFile() call...
	DWORD bw;

	/* Set file sparse. */
	if (!DeviceIoControl(f, FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &bw,  
NULL)) {
		// Failed.
		return 1;
	}

	FILE_ZERO_DATA_INFORMATION z;

	/* Create a hole. */
	z.FileOffset.QuadPart = STARTING_OFFSET;
	z.BeyondFinalZero.QuadPart = END_OFFSET;
	if (!DeviceIoControl(f, FSCTL_SET_ZERO_DATA, &z, sizeof(z), NULL, 0,  
&bw, NULL)) {
		// Failed.
		return 1;
	}
Quoted 9 lines

A second interesting question is whether to implement fallocate over the wire. sys_fallocate takes a range (and also a flag which indicates whether to extend the size beyond end of file). SET_FILE_ALLOCATION_INFO takes a size field rather than a range. For the case of the range starting at zero, we could simply call Trans2
set pathinfo (or file info) with SET_FILE_ALLOCATION_INFO. Should we
add a CIFS POSIX Extension operation for this too?

By the way it looks like OCFS2 (not just ext4) implement these.

Again, I don't know about SMB/CIFS but Windows locally supports this, too. To pre-allocate some space without changing the file size (i.e. this only allocates space on disk, assigns it to the file, and changes the allocated_size on NTFS appropriately but does not change the data_size), one would call (note BYTE_OFFSET_IN_FILE_TO_PREALLOCATE_TO is the logical file offset to which to pre-allocate to):

	HANDLE f; // This is obtained from a CreateFile() call...
	FILE_ALLOCATION_INFO ai;

	ai.AllocationSize.QuadPart = BYTE_OFFSET_IN_FILE_TO_PREALLOCATE_TO;
	if (!SetFileInformationByHandle(f, FileAllocationInfo, &ai,  
sizeof(ai))) {
		// Failed.
		return 1;
	}

Hope this is of some use to use...

Best regards,

Anton

Signature · 3 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/
#2 lore raw
Quoted 8 lines · Anton Altaparmakov
Quoted 6 lines · Steve French

madvise_remove (in Linux) is used to free the backing store associated
with pages (punching a hole in a file). This is one of the vfs
operations that we do not send over the wire to Samba (so this call
would return -ENOSYS locally). Any thoughts on whether this could be
done with an obscure SetFileInfo level or FCNTL or whether it is worth adding to the CIFS POSIX Extensions?

Quoted 5 lines

Don't know about SMB/CIFS but on Windows locally you would need to first set the file sparse, and then to punch the hole. These two things are accomplished like so on Windows (STARTING_OFFSET is the first byte of the hole to be punched and END_OFFSET is the first byte after the hole to be punched):

It would be really great if you could try this with a file on a Windows server and send us the sniffs!

Volker

_______________________________________________
linux-cifs-client mailing list
[email protected]
https://lists.samba.org/mailman/listinfo/linux-cifs-client

#3 lore raw

Large SMBwriteX testing.

In reply to Steve French

Hi Steve,

I've finished adding the ability for smbd to support up to
16MB writeX calls in the latest git 3.2 tree.

To enable, set the parameter :

min receivefile size = XXX

where XXX is the smallest writeX you want to handle with recvfile.

The linux kernel doesn't yet support zerocopy from network to file (ie. splice only works one way currently) so it's emulated in userspace (with a 128k staging buffer) for now.

Also it must be an unsigned connection (for obvious reasons).

Once you've set this param smbd will start reporting CIFS_UNIX_LARGE_WRITE_CAP on a SMB_QUERY_CIFS_UNIX_INFO:
call and you should be good to go. You'll need to use a writeX call identical to Windows (14 wct with a 1 byte pad field) in order to trigger the new code.

Let me know if you get the chance to test it and if it makes a speed difference for CIFSFS.

Cheers,

Jeremy.

#4 lore raw

Re: Large SMBwriteX testing.

I have verified that it works for the case in which "min receivefile size" is under 128K. When I set it to 250000 and tried to read 148000 there were two or three problems (reply_write_and_X in Samba is calling smb_len instead of smb_len_large and it is looking for "req->unread_bytes" incorrectly in a few places in reply.c and fileio.c

Quoted 30 lines · Jeremy Allison

Hi Steve,

I've finished adding the ability for smbd to support up to
16MB writeX calls in the latest git 3.2 tree.

To enable, set the parameter :

min receivefile size = XXX

where XXX is the smallest writeX you want to handle with recvfile.

The linux kernel doesn't yet support zerocopy from network to file (ie. splice only works one way currently) so it's emulated in userspace (with a 128k staging buffer) for now.

Also it must be an unsigned connection (for obvious reasons).

Once you've set this param smbd will start reporting CIFS_UNIX_LARGE_WRITE_CAP on a SMB_QUERY_CIFS_UNIX_INFO:
call and you should be good to go. You'll need to use a writeX call identical to Windows (14 wct with a 1 byte pad field) in order to trigger the new code.

Let me know if you get the chance to test it and if it makes a speed difference for CIFSFS.

Cheers,

Jeremy.

Signature · 3 lines
Thanks,

Steve