Skip to contentSkip to conversation

rework access to /proc/net/rpc

PATCH David Härdeman 3 participants 8 messages Dec 9, 2014 – Dec 10, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

it seems that the "rework access to /proc/net/rpc" patchset removed dynamic buffers in favour of static, fixed size, buffers. That seems like a step backwards to me?

At least the readline() function could be implemented using read/write (instead of fread/fwrite) and a dynamic buffer...no?

//David

#1 lore raw
Quoted 3 lines · On Tue, 09 Dec 2014 09:16:59 +0100 David Härdeman <[email protected]> wrote:

it seems that the "rework access to /proc/net/rpc" patchset removed dynamic buffers in favour of static, fixed size, buffers. That seems like a step backwards to me?

Depends a bit on your view. On read() side, readline() like functionality is removed yes. Though, my understanding is so that this is not needed with the kernel API. Maybe someone can correct me if I'm wrong. The removal simplifies memory management, overall code size. As probably has a positive impact on speed too (probably not too big, but this communication is used all overall, so it might be useful).

On write() side the old code was completely wrong. It did several assumptions on how FILE buffering works, most of them being incorrect in general, but also glibc. It only worked because no large messages have been sent to kernel.

Quoted 2 lines

At least the readline() function could be implemented using read/write (instead of fread/fwrite) and a dynamic buffer...no?

It's extra complexity. I'd rather not add it unless it's required. My understanding about the communication mechanism with kernel is that it's not required. Why have code that would never be used?

Thanks,
Timo

#2 lore raw
Quoted 7 lines · Timo Teras
Quoted 3 lines · On Tue, 09 Dec 2014 09:16:59 +0100 David Härdeman <[email protected]> wrote:

it seems that the "rework access to /proc/net/rpc" patchset removed dynamic buffers in favour of static, fixed size, buffers. That seems like a step backwards to me?

Quoted 6 lines

Depends a bit on your view. On read() side, readline() like functionality is removed yes. Though, my understanding is so that this is not needed with the kernel API. Maybe someone can correct me if I'm wrong. The removal simplifies memory management, overall code size. As probably has a positive impact on speed too (probably not too big, but this communication is used all overall, so it might be useful).

And it makes the buffer size static, introducing an arbitrary limitation (although a rather large one...32KB allocated on the stack IIRC)

Quoted 4 lines

On write() side the old code was completely wrong. It did several assumptions on how FILE buffering works, most of them being incorrect in general, but also glibc. It only worked because no large messages have been sent to kernel.

I didn't really check the write() side, it was just the readline() that I was interested in actually...

Quoted 7 lines
Quoted 3 lines

At least the readline() function could be implemented using read/write (instead of fread/fwrite) and a dynamic buffer...no?

It's extra complexity. I'd rather not add it unless it's required. My understanding about the communication mechanism with kernel is that it's not required. Why have code that would never be used?

I agree that it depends on your view. I tend to be very sceptical of arbitrary limitations unless they have a very good reason (like measurable and relevant performance impact), I doubt that's the case here.

It's up to the maintainer though, I just wanted to point it out :)

Regards,
David

#3 lore raw
Quoted 28 lines · David Härdeman
Quoted 7 lines · Timo Teras
Quoted 3 lines · On Tue, 09 Dec 2014 09:16:59 +0100 David Härdeman <[email protected]> wrote:

it seems that the "rework access to /proc/net/rpc" patchset removed dynamic buffers in favour of static, fixed size, buffers. That seems like a step backwards to me?

Quoted 6 lines

Depends a bit on your view. On read() side, readline() like functionality is removed yes. Though, my understanding is so that this is not needed with the kernel API. Maybe someone can correct me if I'm wrong. The removal simplifies memory management, overall code size. As probably has a positive impact on speed too (probably not too big, but this communication is used all overall, so it might be useful).

And it makes the buffer size static, introducing an arbitrary limitation (although a rather large one...32KB allocated on the stack IIRC)

Quoted 4 lines

On write() side the old code was completely wrong. It did several assumptions on how FILE buffering works, most of them being incorrect in general, but also glibc. It only worked because no large messages have been sent to kernel.

I didn't really check the write() side, it was just the readline() that I was interested in actually...

Quoted 4 lines
Quoted 3 lines

At least the readline() function could be implemented using read/write (instead of fread/fwrite) and a dynamic buffer...no?

Quoted 7 lines
Quoted 3 lines

It's extra complexity. I'd rather not add it unless it's required. My understanding about the communication mechanism with kernel is that it's not required. Why have code that would never be used?

I agree that it depends on your view. I tend to be very sceptical of arbitrary limitations unless they have a very good reason (like measurable and relevant performance impact), I doubt that's the case here.

Your skeptical-ability of arbitrary limitations has become very clear in the last few hours... ;-) I guess I'm indifferent about it... From reading your gssd patch set, it is a bit more artful not to use fixed size buffers but again, I'm indifferent... That said... if patches appear removing these fixed buffers they definitely would be considered...

Quoted 2 lines

It's up to the maintainer though, I just wanted to point it out :)

My understanding these patches were needed to make nfs-utils compatible with the musl c-library. That is the case, correct?

steved.

#4 lore raw
Quoted 21 lines · Steve Dickson
Quoted 6 lines · David Härdeman
Quoted 5 lines · Timo Teras
Quoted 2 lines · On Tue, 09 Dec 2014 09:16:59 +0100 David Härdeman <[email protected]> wrote:

At least the readline() function could be implemented using read/write (instead of fread/fwrite) and a dynamic buffer...no?

Quoted 7 lines
Quoted 3 lines

It's extra complexity. I'd rather not add it unless it's required. My understanding about the communication mechanism with kernel is that it's not required. Why have code that would never be used?

I agree that it depends on your view. I tend to be very sceptical of arbitrary limitations unless they have a very good reason (like measurable and relevant performance impact), I doubt that's the case here.

Your skeptical-ability of arbitrary limitations has become very clear in the last few hours... ;-) I guess I'm indifferent about it... From reading your gssd patch set, it is a bit more artful not to use fixed size buffers but again, I'm indifferent... That said... if patches appear removing these fixed buffers they definitely would be considered...

Quoted 1 line
Quoted 3 lines
Quoted 1 line

It's up to the maintainer though, I just wanted to point it out :)

My understanding these patches were needed to make nfs-utils compatible with the musl c-library. That is the case, correct?

The fread/fwrite removal seems reasonable, yes. The removal of the readline() function though (which could be implemented using normal read/malloc/realloc) seems less so.....IMHO.

Signature · 1 line
David Härdeman
#5 lore raw
Quoted 22 lines · David Härdeman
Quoted 21 lines · Steve Dickson
Quoted 6 lines · David Härdeman
Quoted 5 lines · Timo Teras
Quoted 2 lines · On Tue, 09 Dec 2014 09:16:59 +0100 David Härdeman <[email protected]> wrote:

At least the readline() function could be implemented using read/write (instead of fread/fwrite) and a dynamic buffer...no?

Quoted 7 lines
Quoted 3 lines

It's extra complexity. I'd rather not add it unless it's required. My understanding about the communication mechanism with kernel is that it's not required. Why have code that would never be used?

I agree that it depends on your view. I tend to be very sceptical of arbitrary limitations unless they have a very good reason (like measurable and relevant performance impact), I doubt that's the case here.

Your skeptical-ability of arbitrary limitations has become very clear in the last few hours... ;-) I guess I'm indifferent about it... From reading your gssd patch set, it is a bit more artful not to use fixed size buffers but again, I'm indifferent... That said... if patches appear removing these fixed buffers they definitely would be considered...

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

It's up to the maintainer though, I just wanted to point it out :)

My understanding these patches were needed to make nfs-utils compatible with the musl c-library. That is the case, correct?

The fread/fwrite removal seems reasonable, yes. The removal of the readline() function though (which could be implemented using normal read/malloc/realloc) seems less so.....IMHO.

Patches welcome! 8-)

steved.

#6 lore raw
Quoted 28 lines · On Tue, 09 Dec 2014 16:30:52 -0500 Steve Dickson <[email protected]> wrote:
Quoted 25 lines · David Härdeman
Quoted 24 lines · Steve Dickson
Quoted 11 lines · David Härdeman
Quoted 9 lines · Timo Teras
Quoted 2 lines · On Tue, 09 Dec 2014 09:16:59 +0100 David Härdeman <[email protected]> wrote:

At least the readline() function could be implemented using read/write (instead of fread/fwrite) and a dynamic buffer...no?

It's extra complexity. I'd rather not add it unless it's required. My understanding about the communication mechanism with kernel is that it's not required. Why have code that would never be used?

Quoted 4 lines

I agree that it depends on your view. I tend to be very sceptical of arbitrary limitations unless they have a very good reason (like measurable and relevant performance impact), I doubt that's the case here.

Your skeptical-ability of arbitrary limitations has become very clear in the last few hours... ;-) I guess I'm indifferent about it... From reading your gssd patch set, it is a bit more artful not to use fixed size buffers but again, I'm indifferent... That said... if patches appear removing these fixed buffers they definitely would be considered...

Quoted 1 line
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

It's up to the maintainer though, I just wanted to point it out :)

My understanding these patches were needed to make nfs-utils compatible with the musl c-library. That is the case, correct?

Yes. It is because musl FILE implementation uses writev() and readv() with multiple buffers, and the kernel side does not handle that.

Also, the write side was very brittle even with glibc, it probably was broken with large messages.

Quoted 5 lines
Quoted 4 lines

The fread/fwrite removal seems reasonable, yes. The removal of the readline() function though (which could be implemented using normal read/malloc/realloc) seems less so.....IMHO.

Patches welcome! 8-)

Normally having upper limits is also a security feature. It means the other end cannot force the client code to receive so large messages that it runs out of memory. Though, in this case the other end is kernel and to be trusted.

Though, it probably helps catch potential errors. The messages from kernel really cannot exceed that length. If they do, it's an error anyway. The message structure pretty much ensures this.

In my opinion the dynamic allocation is a step backward, rather then forwards. It adds potential failure (out of memory), is not required, and it does not add any features either.

IMHO, "just because it used to be so" is a bad excuse. And it would just cause additional code making harder to debug and easier to fail. Why add complexity when it can be done simpler?

just my five cents,
Timo

#7 lore raw
Quoted 7 lines · Timo Teras
Quoted 2 lines · On Tue, 09 Dec 2014 16:30:52 -0500 Steve Dickson <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines

My understanding these patches were needed to make nfs-utils compatible with the musl c-library. That is the case, correct?

Yes. It is because musl FILE implementation uses writev() and readv() with multiple buffers, and the kernel side does not handle that.

I should probably note in that case that my patches to gssd include a call to fscanf, I'm guessing that'd be a problem for you?

Quoted 7 lines

In my opinion the dynamic allocation is a step backward, rather then forwards. It adds potential failure (out of memory), is not required, and it does not add any features either.

IMHO, "just because it used to be so" is a bad excuse. And it would just cause additional code making harder to debug and easier to fail. Why add complexity when it can be done simpler?

I think PATH_MAX is a good counter-example.

But I think we can at least agree that we're discussing coding style now, which is a bit like discussing Emacs vs vi, and I doubt we'll ever reach an agreement... :)

Regards,
David