Skip to contentSkip to conversation

current read implementation and POSIX

RFC In review Alexander Aring 2 participants 13 messages Feb 28, 2014 – Mar 3, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover current read implementation and POSIX

Hi all,

first I want to say "Sascha, please don't apply this patch serie!". I suppose I detected some incompatible POSIX read call behaviour.

History:

I am thinking about to add a procfs into barebox and I begun to hack this while I sitting in the train. I don't want to talk about the procfs idea right now, maybe we don't need something like this. But while I hacked it I detected some misbehaviour of read(2).

Issue:

read returns the number of bytes which are readed if there are no bytes it should return 0;

But if the file is zero read will return nothing, this is fixed in Patch 2/3 (please review carefully, I did this fix fast and check only if it working)

Some commands/functions check if the file is zero and then these functions calls no read but this is wrong. When a file is zero, read can read some bytes of course. A zero file not indicate that the file has no content. Maybe I should reference the ugly POSIX opengroup documentation, but you can see it yourself in linux:

ls -l /proc/version:
-r--r--r-- 1 root root 0 Feb 28 08:30 /proc/version

is a zero file and contains information with "cat /proc/version".

Patch 3/3 fix this issue for commands like edit which use the read_full function which should read contents with the return value of read and not with size value.

To test the behaviour I advice you this steps:

1. Apply patch 1/3 which adds a simple ugly hacked foofs.

2. Compile a sandbox and start it.

3. in barebox run: "mkdir /bar;mount none foofs /bar"

4. run "cat /bar/foobar" and "edit /bar/foobar" -> nothing will be displayed

5. Apply patch 2/3 and 3/3 and rerun steps 2-4

So why I don't fix this issue? Because I can't test everything and I think some filesystem and other commands can't deal with it and maybe we don't want to fix this issue and accept the current behaviour. :-(

Another hint:

Sascha had some problems with zero files in tftpfs maybe there is some relationship between this issue and the zero file tftpfs files... because the read of zero files is currently some kind of broken. :-)

- Alex

Alexander Aring (3):
ramfs: add foofs for testing
fs: read: handle zero files
libbb: read_full: use read return instead size

 fs/fs.c     |  3 ---
 fs/ramfs.c  | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/libbb.c |  2 +-
 3 files changed, 70 insertions(+), 4 deletions(-)
Signature · 7 lines
1.9.0


_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox
#1 lore raw

RFC 1/3 ramfs: add foofs for testing

---
 fs/ramfs.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)
fs/ramfs.c +69 -0
index f45a454..4b93c2e 100644
@@ -608,6 +608,48 @@ static int ramfs_probe(struct device_d *dev)
 	return 0;
 }
 
+static int foofs_read(struct device_d *_dev, FILE *f, void *buf, size_t insize)
+{
+	if (f->pos == strlen("Hello World!\n"))
+		return 0;
+
+	return sprintf(buf, "%s", "Hello World!\n");
+}
+
+static int foofs_probe(struct device_d *dev)
+{
+	struct ramfs_inode *n;
+	struct ramfs_priv *priv = xzalloc(sizeof(struct ramfs_priv));
+
+	dev->priv = priv;
+
+	priv->root.name = "/";
+	priv->root.mode = S_IFDIR | S_IRWXU | S_IRWXG | S_IRWXO;
+	priv->root.parent = &priv->root;
+	n = ramfs_get_inode();
+	n->name = strdup(".");
+	n->mode = S_IFDIR;
+	n->parent = &priv->root;
+	n->child = n;
+	priv->root.child = n;
+	n = ramfs_get_inode();
+	n->name = strdup("..");
+	n->mode = S_IFDIR | S_IRWXU | S_IRWXG | S_IRWXO;
+	n->parent = &priv->root;
+	n->child = priv->root.child;
+	priv->root.child->next = n;
+
+	/* just for test zero file read */
+	n = ramfs_get_inode();
+	n->name = strdup("foobar");
+	n->mode = S_IFREG | S_IRWXU | S_IRWXG | S_IRWXO;
+	n->parent = &priv->root;
+	n->child = priv->root.child;
+	priv->root.child->next = n;
+
+	return 0;
+}
+
 static void ramfs_remove(struct device_d *dev)
 {
 	free(dev->priv);
@@ -638,8 +680,35 @@ static struct fs_driver_d ramfs_driver = {
 	}
 };
 
+static struct fs_driver_d foofs_driver = {
+	.create    = ramfs_create,
+	.unlink    = ramfs_unlink,
+	.open      = ramfs_open,
+	.close     = ramfs_close,
+	.truncate  = ramfs_truncate,
+	.read      = foofs_read,
+	.write     = ramfs_write,
+	.lseek     = ramfs_lseek,
+	.mkdir     = ramfs_mkdir,
+	.rmdir     = ramfs_rmdir,
+	.opendir   = ramfs_opendir,
+	.readdir   = ramfs_readdir,
+	.closedir  = ramfs_closedir,
+	.stat      = ramfs_stat,
+	.symlink   = ramfs_symlink,
+	.readlink  = ramfs_readlink,
+	.flags     = FS_DRIVER_NO_DEV,
+	.drv = {
+		.probe  = foofs_probe,
+		.remove = ramfs_remove,
+		.name = "foofs",
+	}
+};
+
+
 static int ramfs_init(void)
 {
+	register_fs_driver(&foofs_driver);
 	return register_fs_driver(&ramfs_driver);
 }
 
Signature · 7 lines
1.9.0


_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox
#2 lore raw

Re: [RFC 1/3] ramfs: add foofs for testing

fs/ramfs.c static int ramfs_probe(struct device_d *dev) · line 648

Quoted 8 lines · Alexander Aring
fs/ramfs.c +8 -0
+
+	/* just for test zero file read */
+	n = ramfs_get_inode();
+	n->name = strdup("foobar");
+	n->mode = S_IFREG | S_IRWXU | S_IRWXG | S_IRWXO;
+	n->parent = &priv->root;
+	n->child = priv->root.child;
+	priv->root.child->next = n;

oops, I overwrite the ".." here, it should be
priv->root.child->next->next = n

sry, but you can test it anyway with this. :-)

- Alex

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

#3 lore raw

Re: [RFC 1/3] ramfs: add foofs for testing

In reply to Alexander Aring
Quoted 10 lines · Alexander Aring
---
 fs/ramfs.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)
fs/ramfs.c +0 -0
index f45a454..4b93c2e 100644

fs/ramfs.c static int ramfs_probe(struct device_d *dev) · line 617

Quoted 10 lines
fs/ramfs.c +7 -0
@@ -608,6 +608,48 @@ static int ramfs_probe(struct device_d *dev)
 	return 0;
 }
 
+static int foofs_read(struct device_d *_dev, FILE *f, void *buf, size_t insize)
+{
+	if (f->pos == strlen("Hello World!\n"))
+		return 0;
+
+	return sprintf(buf, "%s", "Hello World!\n");
+}

You should never read more bytes than insize. This is also true for insize == 0. Implement this correctly and you'll see that your patches do not solve the problem.

Sascha

Signature · 9 lines
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox
#4 lore raw

Re: [RFC 1/3] ramfs: add foofs for testing

Hi Sascha,

Quoted 14 lines · Sascha Hauer
Quoted 13 lines · Alexander Aring
---
 fs/ramfs.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)
fs/ramfs.c +0 -0
index f45a454..4b93c2e 100644
@@ -608,6 +608,48 @@ static int ramfs_probe(struct device_d *dev)
 	return 0;
 }
 
Quoted 12 lines

fs/ramfs.c static int ramfs_probe(struct device_d *dev) · line 617

Quoted 7 lines
fs/ramfs.c +7 -0
+static int foofs_read(struct device_d *_dev, FILE *f, void *buf, size_t insize)
+{
+	if (f->pos == strlen("Hello World!\n"))
+		return 0;
+
+	return sprintf(buf, "%s", "Hello World!\n");
+}

You should never read more bytes than insize. This is also true for insize == 0. Implement this correctly and you'll see that your patches do not solve the problem.

yes it's not correctly implemented. Maybe I implement a "testfs" filesystem for barebox with unit-tests which can be used to test the filesystem layer to see if something broken or not. This fs should be based on ramfs.

The fs tests be placed into commands -> testing and it's only interesting for some developers.

- Alex

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

#5 lore raw

RFC 2/3 fs: read: handle zero files

In reply to Alexander Aring
---
 fs/fs.c | 3 ---
 1 file changed, 3 deletions(-)
fs/fs.c +0 -3
index 1b43c61..9584495 100644
@@ -804,9 +804,6 @@ static ssize_t __read(FILE *f, void *buf, size_t count)
 	if (f->size != FILE_SIZE_STREAM && f->pos + count > f->size)
 		count = f->size - f->pos;
 
-	if (!count)
-		return 0;
-
 	ret = fsdrv->read(dev, f, buf, count);
 
 	if (ret < 0)
Signature · 7 lines
1.9.0


_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox
#6 lore raw

RFC 3/3 libbb: read_full: use read return instead size

In reply to Alexander Aring
---
 lib/libbb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/libbb.c +1 -1
index 189a170..c8d0835 100644
@@ -162,7 +162,7 @@ int read_full(int fd, void *buf, size_t size)
 	int now;
 	int total = 0;
 
-	while (size) {
+	while (now) {
 		now = read(fd, buf, size);
 		if (now == 0)
 			return total;
Signature · 7 lines
1.9.0


_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox
#7 lore raw

Re: [RFC 3/3] libbb: read_full: use read return instead size

Quoted 10 lines · Alexander Aring
---
 lib/libbb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/libbb.c +0 -0
index 189a170..c8d0835 100644

lib/libbb.c int read_full(int fd, void *buf, size_t size) · line 168

Quoted 8 lines
lib/libbb.c +1 -1
@@ -162,7 +162,7 @@ int read_full(int fd, void *buf, size_t size)
 	int now;
 	int total = 0;
 
-	while (size) {
+	while (now) {
 		now = read(fd, buf, size);
 		if (now == 0)
 			return total;

and this should be a:

do {
...
} while (now);

sry, it's only to demonstrate the issue.

- Alex

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

#8 lore raw

Re: [RFC 3/3] libbb: read_full: use read return instead size

Quoted 21 lines · Alexander Aring
Quoted 10 lines · Alexander Aring
---
 lib/libbb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/libbb.c +0 -0
index 189a170..c8d0835 100644

lib/libbb.c int read_full(int fd, void *buf, size_t size) · line 168

Quoted 8 lines
lib/libbb.c +1 -1
@@ -162,7 +162,7 @@ int read_full(int fd, void *buf, size_t size)
 	int now;
 	int total = 0;
 
-	while (size) {
+	while (now) {
 		now = read(fd, buf, size);
 		if (now == 0)
 			return total;

and this should be a:

Quoted 5 lines

do {
...
} while (now);

sry, it's only to demonstrate the issue.

'now' will never be 0 at the end of the loop, so you could equally well write while(1). With this change we try to read as long as we read something last time, even if there's nothing left to read (size is 0). What issue do you see with this function?

Sascha

Signature · 9 lines
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox
#9 lore raw

Re: [RFC 3/3] libbb: read_full: use read return instead size

Hi Sascha,

Quoted 22 lines · Sascha Hauer
Quoted 21 lines · Alexander Aring
Quoted 10 lines · Alexander Aring
---
 lib/libbb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/libbb.c +0 -0
index 189a170..c8d0835 100644

lib/libbb.c int read_full(int fd, void *buf, size_t size) · line 168

Quoted 8 lines
lib/libbb.c +1 -1
@@ -162,7 +162,7 @@ int read_full(int fd, void *buf, size_t size)
 	int now;
 	int total = 0;
 
-	while (size) {
+	while (now) {
 		now = read(fd, buf, size);
 		if (now == 0)
 			return total;

and this should be a:

Quoted 8 lines
Quoted 5 lines

do {
...
} while (now);

sry, it's only to demonstrate the issue.

'now' will never be 0 at the end of the loop, so you could equally well write while(1). With this change we try to read as long as we read

yes, I did it quickly to make something that command "edit" works in some way. The whole patches is only to demonstrate the issue. Also the foofs demo works if we read the whole thing at once, but that's okay for the test.

Quoted 2 lines

something last time, even if there's nothing left to read (size is 0). What issue do you see with this function?

The function read_full makes similar things like:

stat(fd, &statbuf);
...
read(fd, buf, statbuf.st_size);

this is wrong because a filesize can be zero and read can read something from this file.

For example procfs in linux, if you run this under linux for /proc/version it will do nothing.

a "while (size)" don't call read because at first we checked if the file is zero, which don't indicate that you can read something from a file.

So we have many function which works like this (I suppose). But there is something wrong with the whole read function because we can't read something from a file, when the filesize is zero. Patch 2/3 changes this.

Sascha, do you understand what I mean... in some way? :-/

I know it sounds a little bit crazy that read can read more than zero bytes if the filesize is zero.

- Alex

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

#10 lore raw

Re: [RFC 3/3] libbb: read_full: use read return instead size

Quoted 48 lines · Alexander Aring

Hi Sascha,

Quoted 22 lines · Sascha Hauer
Quoted 21 lines · Alexander Aring
Quoted 10 lines · Alexander Aring
---
 lib/libbb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/libbb.c +0 -0
index 189a170..c8d0835 100644

lib/libbb.c int read_full(int fd, void *buf, size_t size) · line 168

Quoted 8 lines
lib/libbb.c +1 -1
@@ -162,7 +162,7 @@ int read_full(int fd, void *buf, size_t size)
 	int now;
 	int total = 0;
 
-	while (size) {
+	while (now) {
 		now = read(fd, buf, size);
 		if (now == 0)
 			return total;

and this should be a:

Quoted 8 lines
Quoted 5 lines

do {
...
} while (now);

sry, it's only to demonstrate the issue.

'now' will never be 0 at the end of the loop, so you could equally well write while(1). With this change we try to read as long as we read

yes, I did it quickly to make something that command "edit" works in some way. The whole patches is only to demonstrate the issue. Also the foofs demo works if we read the whole thing at once, but that's okay for the test.

Quoted 2 lines

something last time, even if there's nothing left to read (size is 0). What issue do you see with this function?

The function read_full makes similar things like:

stat(fd, &statbuf);
...
read(fd, buf, statbuf.st_size);

Quoted 9 lines

this is wrong because a filesize can be zero and read can read something from this file.

For example procfs in linux, if you run this under linux for /proc/version it will do nothing.

a "while (size)" don't call read because at first we checked if the file is zero, which don't indicate that you can read something from a file.

I meant s/can read/can't read/

- Alex

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

#11 lore raw

Re: [RFC 3/3] libbb: read_full: use read return instead size

In reply to Alexander Aring
Quoted 39 lines · Alexander Aring

Hi Sascha,

Quoted 22 lines · Sascha Hauer
Quoted 21 lines · Alexander Aring
Quoted 10 lines · Alexander Aring
---
 lib/libbb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/libbb.c +0 -0
index 189a170..c8d0835 100644

lib/libbb.c int read_full(int fd, void *buf, size_t size) · line 168

Quoted 8 lines
lib/libbb.c +1 -1
@@ -162,7 +162,7 @@ int read_full(int fd, void *buf, size_t size)
 	int now;
 	int total = 0;
 
-	while (size) {
+	while (now) {
 		now = read(fd, buf, size);
 		if (now == 0)
 			return total;

and this should be a:

Quoted 8 lines
Quoted 5 lines

do {
...
} while (now);

sry, it's only to demonstrate the issue.

'now' will never be 0 at the end of the loop, so you could equally well write while(1). With this change we try to read as long as we read

yes, I did it quickly to make something that command "edit" works in some way. The whole patches is only to demonstrate the issue. Also the foofs demo works if we read the whole thing at once, but that's okay for the test.

Quoted 4 lines
Quoted 2 lines

something last time, even if there's nothing left to read (size is 0). What issue do you see with this function?

The function read_full makes similar things like:

s/read_full/read_file/

Quoted 10 lines

stat(fd, &statbuf);
...
read(fd, buf, statbuf.st_size);

this is wrong because a filesize can be zero and read can read something from this file.

For example procfs in linux, if you run this under linux for /proc/version it will do nothing.

So our read_file implementation doesn't work procfs like filesystems where all sizes are 0. Your patch doesn't change this though. In 2/3 you remove the if (!size) check and call the fs drivers read function with size 0. In this case the read function may return values, but never actually read something because the buffer size is 0.

Sascha

Sascha

Signature · 9 lines
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox
#12 lore raw

Re: [RFC 3/3] libbb: read_full: use read return instead size

Hi Sascha,

Quoted 7 lines · Sascha Hauer

So our read_file implementation doesn't work procfs like filesystems where all sizes are 0. Your patch doesn't change this though. In 2/3 you remove the if (!size) check and call the fs drivers read function with size 0. In this case the read function may return values, but never actually read something because the buffer size is 0.

exactly. The imporant question for me is "do you want to accept patches to changes this behaviour, so we can read zero file size files". I mean this would be a huge change in internal api of filesystem layer and I hope we doesn't break anything else.

But your are fine with me to say "there exists a problem with read of zero sized files?".

I will create some "better" patches for this, this RFC should only answer the question if you apply fixes or not.

- Alex

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox