Skip to contentSkip to conversation

horrible disk thorughput on itanium

DISCUSSION Andi Kleen 12 participants 24 messages Dec 7, 2001 – Dec 9, 2001 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Andi Kleen

author
#1 lore raw
Quoted 4 lines · [email protected] (Linus Torvalds) writes:

"putc()" is a standard function. If it sucks, let's get it fixed. And instead of changing bonnie, how about pinging the _real_ people who write sucky code?

It is easy to fix. Just do #define putc putc_unlocked
There is just a slight problem: it'll fail if your application is threaded and wants to use the same FILE from multiple threads.

It is a common problem on all OS that eventually got threadsafe stdio. I bet putc sucks on Solaris too.

-Andi

#2 lore raw
Quoted 6 lines · Andi Kleen
Quoted 4 lines · [email protected] (Linus Torvalds) writes:

"putc()" is a standard function. If it sucks, let's get it fixed. And instead of changing bonnie, how about pinging the _real_ people who write sucky code?

Quoted 6 lines

It is easy to fix. Just do #define putc putc_unlocked
There is just a slight problem: it'll fail if your application is threaded and wants to use the same FILE from multiple threads.

It is a common problem on all OS that eventually got threadsafe stdio. I bet putc sucks on Solaris too.

This kind of thing should be covered by _REENTRANT. If you don't compile with _REENTRANT (the default), then putc() should be the unlocked version.

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]
#3 lore raw
Quoted 3 lines · Richard Gooch

This kind of thing should be covered by _REENTRANT. If you don't compile with _REENTRANT (the default), then putc() should be the unlocked version.

The link to the mailing list post from bug-glibc says otherwise, that is the problem. Using the unlocked version isn't implied by not setting __REENTRANT.

Robert Love

#4 lore raw
Quoted 9 lines · Robert Love
Quoted 3 lines · Richard Gooch

This kind of thing should be covered by _REENTRANT. If you don't compile with _REENTRANT (the default), then putc() should be the unlocked version.

The link to the mailing list post from bug-glibc says otherwise, that is the problem. Using the unlocked version isn't implied by not setting __REENTRANT.

The bug is in glibc. An application shouldn't need to be changed to work around that bug. putc() is a well-known interface, and people shouldn't have to code around a change in that interface.

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]
#5 lore raw
Quoted 7 lines · Richard Gooch
Quoted 3 lines

The link to the mailing list post from bug-glibc says otherwise, that is the problem. Using the unlocked version isn't implied by not setting __REENTRANT.

The bug is in glibc. An application shouldn't need to be changed to work around that bug. putc() is a well-known interface, and people shouldn't have to code around a change in that interface.

Right. That's why I referenced a post on bug-glibc and called the issue a problem. I'm not defending the heaping mass known as glibc ... it should be fixed.

Robert Love

#6 lore raw
In reply to Andi Kleen
Quoted 7 lines · Andi Kleen
Quoted 4 lines · [email protected] (Linus Torvalds) writes:

"putc()" is a standard function. If it sucks, let's get it fixed. And instead of changing bonnie, how about pinging the _real_ people who write sucky code?

It is easy to fix. Just do #define putc putc_unlocked

Sure. And why don't you also do

	#define sin(x) (1)
	#define sqrt(x) (1)
	#define strlen(x) (1)
	...

to make other benchmarks happier?

bonnie is a _benchmark_. It's meant for finding bad performance. Changing it to make it work better when performance is bad is _pointless_. You've now made the whole point of bonnie go away.

Quoted 4 lines

There is just a slight problem: it'll fail if your application is threaded and wants to use the same FILE from multiple threads.

It is a common problem on all OS that eventually got threadsafe stdio.

It's a common problem with bad programming.

You can be thread-safe without sucking dead baby donkeys through a straw. I already mentioned two possible ways to fix it so that you have locking when you need to, and no locking when you don't.

Linus

#7 lore raw

In article <[email protected]>,

Quoted 3 lines · Linus Torvalds

bonnie is a _benchmark_. It's meant for finding bad performance. Changing it to make it work better when performance is bad is _pointless_. You've now made the whole point of bonnie go away.

It isn't just bonnie showing bad performance. My application shows it, bonnie shows it, and tiobench shows it. I think the focus on putc may be too intense. It isn't suprizing to me that either the kernel or glibc may not be optimised on ia64, I'm just trying to figure out how to get better io rates out of my itanium machine.

#8 lore raw
Quoted 11 lines · Greg Hennessy

In article <[email protected]>,

Quoted 3 lines · Linus Torvalds

bonnie is a _benchmark_. It's meant for finding bad performance. Changing it to make it work better when performance is bad is _pointless_. You've now made the whole point of bonnie go away.

It isn't just bonnie showing bad performance. My application shows it, bonnie shows it, and tiobench shows it. I think the focus on putc may be too intense. It isn't suprizing to me that either the kernel or glibc may not be optimised on ia64, I'm just trying to figure out how to get better io rates out of my itanium machine.

Does a simple 'dd' show the problem? I mean,

time dd if=/dev/zero of=/somelargefile count=somelargenumber bs=8k

is it much slower on the itanium, too? dd doesn't use putc(), I hope.

Just for comparison, I've run the following command here:

# time dd if=/dev/zero of=/u2/test count=250000 bs=8k
250000+0 records in
250000+0 records out
0.14user 12.95system 1:23.15elapsed 15%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (117major+18minor)pagefaults 0swaps

(Almost idle box, 256MB RAM, UDMA disk, 2.2.x kernel)

You may try with a bigger file, expecially if you have more RAM.

.TM.

Signature · 5 lines
      ____/  ____/   /
     /      /       /			Marco Colombo
    ___/  ___  /   /		      Technical Manager
   /          /   /			 ESI s.r.l.
 _____/ _____/  _/		       [email protected]
author
#9 lore raw
In reply to Andi Kleen
Quoted 3 lines

You can be thread-safe without sucking dead baby donkeys through a straw. I already mentioned two possible ways to fix it so that you have locking when you need to, and no locking when you don't.

Your proposals sound rather dangerous. They would silently break recompiled threaded programs that need the locking and don't use -D__REENTRANT (most people do not seem to use it). I doubt the possible pain from that is worth it for speeding up an basically obsolete interface like putc.

i.e. if someone wants speed they definitely shouldn't use putc()

-Andi

#10 lore raw
Quoted 3 lines · Andi Kleen

Your proposals sound rather dangerous. They would silently break recompiled threaded programs that need the locking and don't use -D__REENTRANT

No it wouldn't.

Once you do a pthread_create(), the locking is there.

Before you do a pthread_create(), it doesn't lock.

What's the problem? Before you do a pthread_create(), you don't _NEED_ locking, because there is only one thread that accesses the stdio data structures.

And there are no races - if there is only one thread, then another thread couldn't be suddenly doing a pthread_create() during a stdio operations.

Safe, and efficient. Yes, it adds a flag test or a indirect branch, but considering that you avoid a serialized locking instruction, the optimization sounds obvious.

Linus

#11 lore raw
Quoted 24 lines · Linus Torvalds
Quoted 3 lines · Andi Kleen

Your proposals sound rather dangerous. They would silently break recompiled threaded programs that need the locking and don't use -D__REENTRANT

No it wouldn't.

Once you do a pthread_create(), the locking is there.

Before you do a pthread_create(), it doesn't lock.

What's the problem? Before you do a pthread_create(), you don't _NEED_ locking, because there is only one thread that accesses the stdio data structures.

And there are no races - if there is only one thread, then another thread couldn't be suddenly doing a pthread_create() during a stdio operations.

Safe, and efficient. Yes, it adds a flag test or a indirect branch, but considering that you avoid a serialized locking instruction, the optimization sounds obvious.

Linus

#12 lore raw
In reply to Andi Kleen
Quoted 10 lines · Andi Kleen
Quoted 3 lines

You can be thread-safe without sucking dead baby donkeys through a straw. I already mentioned two possible ways to fix it so that you have locking when you need to, and no locking when you don't.

Your proposals sound rather dangerous. They would silently break recompiled threaded programs that need the locking and don't use -D__REENTRANT (most people do not seem to use it). I doubt the possible pain from that is worth it for speeding up an basically obsolete interface like putc.

i.e. if someone wants speed they definitely shouldn't use putc()

Threaded programs that need locking and don't define _THREAD_SAFE or _REENTRANT or whatever is appropriate are already broken -- they just don't know it yet.

FreeBSD #defines putc and getc to their unlocked versions unless _THREAD_SAFE is defined, and people don't seem to think its libc is broken. Many lightly threaded programs, in fact, wouldn't need or even want the locked variants to be the default. One app I've worked with only reads and writes any given FILE* from one thread, and I saw an 4x speedup by switching to the unlocked variants.

It's generally a bad idea to make people pay for a feature they don't ask for. FreeBSD's libc understands this; glibc apaprently doesn't.

-- Michael Poole

#13 lore raw
In reply to Andi Kleen
Quoted 8 lines · Andi Kleen
Quoted 4 lines

You can be thread-safe without sucking dead baby donkeys through a straw. I already mentioned two possible ways to fix it so that you have locking when you need to, and no locking when you don't.

Your proposals sound rather dangerous. They would silently break recompiled threaded programs that need the locking and don't use -D__REENTRANT (most people do not seem to use it).

I would worry about threaded progs that don't -D_REENTRANT as they are broken.

Quoted 3 lines

I doubt the possible pain from that is worth it for speeding up an basically obsolete interface like putc.

i.e. if someone wants speed they definitely shouldn't use putc()

It's not just putc, it's all of stdio.

Padraig.

#14 lore raw
In reply to Andi Kleen
Quoted 18 lines · Michael Poole
Quoted 10 lines · Andi Kleen
Quoted 4 lines

You can be thread-safe without sucking dead baby donkeys through a straw. I already mentioned two possible ways to fix it so that you have locking when you need to, and no locking when you don't.

Your proposals sound rather dangerous. They would silently break recompiled
threaded programs that need the locking and don't use -D__REENTRANT (most
people do not seem to use it). I doubt the possible pain from that is worth it for speeding up an basically obsolete interface like putc.

i.e. if someone wants speed they definitely shouldn't use putc()

Threaded programs that need locking and don't define _THREAD_SAFE or _REENTRANT or whatever is appropriate are already broken -- they just don't know it yet.

Quoted 6 lines

FreeBSD #defines putc and getc to their unlocked versions unless _THREAD_SAFE is defined, and people don't seem to think its libc is broken. Many lightly threaded programs, in fact, wouldn't need or even want the locked variants to be the default. One app I've worked with only reads and writes any given FILE* from one thread, and I saw an 4x speedup by switching to the unlocked variants.

This breaks for the case discussed @
http://sources.redhat.com/ml/bug-glibc/2001-11/msg00079.html
I.E. if you have a multithreaded lib being linked by single threaded apps (Note multithreaded lib, not just a threadsafe lib (I.E. the lib calls pthread_create())).

Padraig.

#15 lore raw
In reply to Andi Kleen

From: Andi Kleen <[email protected]>
Date: 07 Dec 2001 14:54:49 +0100

It is a common problem on all OS that eventually got threadsafe stdio. I bet putc sucks on Solaris too.

Nope, they even inline the full implementation it in their header files so they beat even our putc_unlocked() to the point it is embarassing.

#16 lore raw
In reply to Andi Kleen
Quoted 5 lines · Padraig Brady

This breaks for the case discussed @
http://sources.redhat.com/ml/bug-glibc/2001-11/msg00079.html
I.E. if you have a multithreaded lib being linked by single threaded apps (Note multithreaded lib, not just a threadsafe lib (I.E. the lib calls pthread_create())).

That's an interesting, but very contrived, example. Can you find a single multi-threaded lib that uses FILE*'s shared with the application using it?

Linus's suggestion to add hooks to pthread_create() gets around that problem, anyway. Alternatively, the multi-threaded library could require any application linking to it to define _REENTRANT.

After all, it's silly to talk about a 'multi-threaded' library linked to a 'single-threaded' application -- the application plus any libraries, as a whole, are either multithreaded or not. They have to be on the same page to deal with *any* locking issues.

-- Michael

#17 lore raw
In reply to Andi Kleen
Quoted 10 lines · Michael Poole
Quoted 6 lines · Padraig Brady

This breaks for the case discussed @
http://sources.redhat.com/ml/bug-glibc/2001-11/msg00079.html
I.E. if you have a multithreaded lib being linked by single threaded apps (Note multithreaded lib, not just a threadsafe lib (I.E. the lib calls pthread_create())).

Quoted 3 lines

That's an interesting, but very contrived, example. Can you find a single multi-threaded lib that uses FILE*'s shared with the application using it?

Well you have to deal with the general case. A single threaded

app linking against a multithreaded lib. It mightn't be just shared FILE*'s that could cause problems.

Quoted 2 lines

Linus's suggestion to add hooks to pthread_create() gets around that problem, anyway.

I don't think this will work as I said before current apps that use _unlocked() functions directly manipulate the stdio structures, hence a "new smarter locking stdio" would never get used by existing compiled apps.

Quoted 2 lines

Alternatively, the multi-threaded library could require any application linking to it to define _REENTRANT.

It could, but what if an existing interface (lib) is changed

from signle to multithreaded. You can't preclude this.

Quoted 6 lines

After all, it's silly to talk about a 'multi-threaded' library linked to a 'single-threaded' application -- the application plus any libraries, as a whole, are either multithreaded or not. They have to be on the same page to deal with *any* locking issues.

-- Michael

Padraig.

#18 lore raw
In reply to Andi Kleen
Quoted 3 lines · Padraig Brady

Well you have to deal with the general case. A single threaded app linking against a multithreaded lib. It mightn't be just shared FILE*'s that could cause problems.

The question was about putc(), and presumably other stdio functions. They deal with FILE*'s. They are also commonly used and small operations, so forcing locking on an app that doesn't ask for it can (and has in the past) cause a major performance degredation. I bet you could find very similar results for memory allocation in other applications.

Quoted 7 lines
Quoted 2 lines

Linus's suggestion to add hooks to pthread_create() gets around that problem, anyway.

I don't think this will work as I said before current apps that use _unlocked() functions directly manipulate the stdio structures, hence a "new smarter locking stdio" would never get used by existing compiled apps.

As Linus pointed out, you just need another test in those macros, so they can be forced to call functions rather than using inline code. When you call a function, it can use whatever new smarter locking library you want.

Quoted 5 lines
Quoted 2 lines

Alternatively, the multi-threaded library could require any application linking to it to define _REENTRANT.

It could, but what if an existing interface (lib) is changed from signle to multithreaded. You can't preclude this.

I certainly can preclude that. I'd consider adding threads and their locking considerations to the library a change in the API and ABI -- and quite a big one at that. If you change the ABI (rather than just extending it), you must increase the major version number, which prevents linking with those applications that expect the semantics of earlier versions.

(BSD variants have in the past had both libc and libc_r, where only libc_r makes you pay locking overhead. This is yet another way to enforce the ABI separation between single- and multi-threaded applications, with different tradeoffs than the others mentioned.)

-- Michael

#19 lore raw

[OT] fputc vs putc Re: horrible disk thorughput on itanium

In reply to Andi Kleen
Quoted 2 lines · David S. Miller

They do it also for putc() if you haven't defined the appropriate defines.

what exactly is the difference between putc() and fputc()? the man page is very vague.

Signature · 2 lines
Tom Vier <[email protected]>
DSA Key id 0x27371A2C
#20 lore raw

Re: [OT] fputc vs putc Re: horrible disk thorughput on itanium

In reply to Andi Kleen
Quoted 6 lines · Tom Vier
Quoted 2 lines · David S. Miller

They do it also for putc() if you haven't defined the appropriate defines.

what exactly is the difference between putc() and fputc()? the man page is very vague.

fputc is a function putc is a macro because the overhead of a function call in writing a character has always been a problem...

Eric

#21 lore raw

Re: [OT] fputc vs putc Re: horrible disk thorughput on itanium

In reply to Andi Kleen
Followup to:  <[email protected]>
By author:    [email protected] (Eric W. Biederman)
In newsgroup: linux.dev.kernel
Quoted 4 lines

fputc is a function putc is a macro because the overhead of a function call in writing a character has always been a problem...

putc() is frequently defined as

#define putc(__C) fputc((__C), stdout)

... or some equivalent; I think the best way to say it's that it's a shorthand.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#22 lore raw

Re: [OT] fputc vs putc Re: horrible disk thorughput on itanium

In reply to Andi Kleen
Quoted 6 lines · H. Peter Anvin

putc() is frequently defined as

#define putc(__C) fputc((__C), stdout)

... or some equivalent; I think the best way to say it's that it's a shorthand.

according to the putc man page in debian stable, it takes the same args as fputc. maybe it varies by glibc version (mine is 2.1.3-19). i guess anyone using putc better use autoconf. also, "unix system programming for SVr4" says the only difference is that putc is an inlined macro version of fputc.

Signature · 2 lines
Tom Vier <[email protected]>
DSA Key id 0x27371A2C
#23 lore raw

Re: [OT] fputc vs putc Re: horrible disk thorughput on itanium

In reply to Andi Kleen
Followup to:  <20011209141408.A17671@zero>
By author:    Tom Vier <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 6 lines

according to the putc man page in debian stable, it takes the same args as fputc. maybe it varies by glibc version (mine is 2.1.3-19). i guess anyone using putc better use autoconf. also, "unix system programming for SVr4" says the only difference is that putc is an inlined macro version of fputc.

Nevermind. I was thinking about putc() and putchar(), additionally confused by the annoying fact that fputs() is to putc() as puts() is to putchar()... makes no fscking sense.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#24 lore raw
In reply to a message not in the archive
Quoted 7 lines · Andi Kleen
Quoted 4 lines · [email protected] (Linus Torvalds) writes:

"putc()" is a standard function. If it sucks, let's get it fixed. And instead of changing bonnie, how about pinging the _real_ people who write sucky code?

Quoted 7 lines

It is easy to fix. Just do #define putc putc_unlocked
There is just a slight problem: it'll fail if your application is threaded and wants to use the same FILE from multiple threads.

It is a common problem on all OS that eventually got threadsafe stdio. I bet putc sucks on Solaris too.

-Andi

Interesting thread on this:
http://sources.redhat.com/ml/bug-glibc/2001-11/msg00079.html

for glibc 2.2.4 with the following program with input file of 354371 lines of text(/usr/share/doc/*), where the average line length was 37 chars.

getc/putc
real 2.181s
user 2.150s
sys 0.030s
getc_unlocked/putc_unlocked
real 0.326s
user 0.280s
sys 0.040s

I.E. 669% faster!

Padraig.

-------------------
#include <stdio.h>

#ifndef _REENTRANT

#    undef getc
#    define getc(x)   getc_unlocked(x)
#    undef putc
#    define putc(x,y) putc_unlocked(x,y)

#endif //_REENTRANT

void main(void)

{
    int c;
    while((c=getc(stdin))!=EOF) putc(c,stdout);
}