Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1Jqs42-0007Qk-LB for qemu-devel@nongnu.org; Tue, 29 Apr 2008 11:48:18 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1Jqs40-0007QP-EC for qemu-devel@nongnu.org; Tue, 29 Apr 2008 11:48:17 -0400 Received: from [199.232.76.173] (port=47825 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Jqs40-0007QM-93 for qemu-devel@nongnu.org; Tue, 29 Apr 2008 11:48:16 -0400 Received: from ecfrec.frec.bull.fr ([129.183.4.8]) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1Jqs3y-00069K-5m for qemu-devel@nongnu.org; Tue, 29 Apr 2008 11:48:15 -0400 Subject: Re: [Qemu-devel] [PATCH] Align file accesses with cache=off (O_DIRECT) From: Laurent Vivier In-Reply-To: <48173591.9010609@suse.de> References: <4807514B.9040607@suse.de> <4815EE89.70407@suse.de> <1209459667.4328.7.camel@frecb07144> <48173591.9010609@suse.de> Content-Type: text/plain; charset=utf-8 Date: Tue, 29 Apr 2008 17:48:24 +0200 Message-Id: <1209484105.4248.27.camel@frecb07144> Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Reply-To: qemu-devel@nongnu.org List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Kevin Wolf Cc: qemu-devel@nongnu.org Le mardi 29 avril 2008 =C3=A0 16:49 +0200, Kevin Wolf a =C3=A9crit : > Hi Laurent, >=20 > Laurent Vivier schrieb: > > But if we want too keep simplicity without memcpy(), we could only > > de-activate O_DIRECT on pread() or pwrite() : >=20 > You're right, this approach is simpler, better and makes the patch=20 > smaller. I've attached a new version of the patch. >=20 > However, now that you've pointed me to your patch I realize that my=20 > patch might be simple but it is incomplete as well. Normal read/write=20 > operation on qcow images should work fine (only metadata is unaligned=20 > and there pread is used). Snapshots don't work though because they end=20 > up in aio requests which are not routed to raw_pread. >=20 > Disabling O_DIRECT for a single aio request is impossible (after all,=20 > aio is asynchronous), and disabling it for at least one aio request is=20 Perhaps I'm wrong, but I think it is possible: the only consequence is the asynchronous I/O becomes synchronous... > going to be ugly. So maybe we better turn O_DIRECT off for snapsnot=20 > saving/loading, even if it's not the generic fix I wanted to have when = I=20 > started. I don't think it is a good idea: In linux world, there are three reasons to use O_DIRECT: 1- to use linux AIO (not POSIX AIO). 2- to avoid a buffer copy between user- and kernel- space (performance ?) 3- to increase reliability: by using O_DIRECT you are sure your data are on the disk when the write is over and your system can now crash (if it wants). And I think reliability is better when the snapshot is being saved... > I'm still undecided, though. What do you think? Is it possible to align the last AIO ? And see comments below > =EF=BB=BFIndex: block-raw-posix.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- block-raw-posix.c.orig > +++ block-raw-posix.c > @@ -77,6 +77,7 @@ > typedef struct BDRVRawState { > int fd; > int type; > + int flags; > unsigned int lseek_err_cnt; > #if defined(__linux__) > /* linux floppy specific */ > @@ -95,6 +96,7 @@ static int raw_open(BlockDriverState *bs > BDRVRawState *s =3D bs->opaque; > int fd, open_flags, ret; > =20 > + s->flags =3D flags; > s->lseek_err_cnt =3D 0; I think you should store open_flags instead of flags (see below). > open_flags =3D O_BINARY; > @@ -141,7 +143,14 @@ static int raw_open(BlockDriverState *bs > #endif > */ > =20 > -static int raw_pread(BlockDriverState *bs, int64_t offset, > +/*=20 > + * offset and count are in bytes, but must be multiples of 512 for > files=20 > + * opened with O_DIRECT. buf must be aligned to 512 bytes then. > + * > + * This function may be called without alignment if the caller > ensures > + * that O_DIRECT is not in effect. > + */ > +static int raw_pread_aligned(BlockDriverState *bs, int64_t offset, > uint8_t *buf, int count) > { > BDRVRawState *s =3D bs->opaque; > @@ -194,7 +203,14 @@ label__raw_read__success: > return ret; > } > =20 > -static int raw_pwrite(BlockDriverState *bs, int64_t offset, > +/*=20 > + * offset and count are in bytes, but must be multiples of 512 for > files=20 > + * opened with O_DIRECT. buf must be aligned to 512 bytes then. > + * > + * This function may be called without alignment if the caller > ensures > + * that O_DIRECT is not in effect. > + */ > +static int raw_pwrite_aligned(BlockDriverState *bs, int64_t offset, > const uint8_t *buf, int count) > { > BDRVRawState *s =3D bs->opaque; > @@ -230,6 +246,69 @@ label__raw_write__success: > return ret; > } > =20 > + > +#ifdef O_DIRECT > +/*=20 > + * offset and count are in bytes and possibly not aligned. For files > opened=20 > + * with O_DIRECT, necessary alignments are ensured before calling=20 > + * raw_pread_aligned to do the actual read. > + */ > +static int raw_pread(BlockDriverState *bs, int64_t offset, > + uint8_t *buf, int count) > +{ > + BDRVRawState *s =3D bs->opaque; > + > + if (unlikely((s->flags & BDRV_O_DIRECT) && > + (offset % 512 !=3D 0 || (uintptr_t) buf % 512))) { > + > + int flags, ret; > + > + // Temporarily disable O_DIRECT for unaligned access > + flags =3D fcntl(s->fd, F_GETFL); > + fcntl(s->fd, F_SETFL, flags & ~O_DIRECT); > + ret =3D raw_pread_aligned(bs, offset, buf, count); > + fcntl(s->fd, F_SETFL, flags); > + > + return ret; if you store open_flag instead of flags, you can do: if (unlikely((s->open_flags & O_DIRECT) &&=20 (offset % 512 || (uintptr_t) buf % 512))) { fcntl(s->fd, F_SETFL, s->open_flags & ~O_DIRECT); ret =3D raw_pread_aligned(bs, offset, buf, count); fcntl(s->fd, F_SETFL, s->open_flags); } > + } else { > + return raw_pread_aligned(bs, offset, buf, count); > + } > +} > + > +/*=20 > + * offset and count are in bytes and possibly not aligned. For files > opened=20 > + * with O_DIRECT, necessary alignments are ensured before calling=20 > + * raw_pwrite_aligned to do the actual write. > + */ > +static int raw_pwrite(BlockDriverState *bs, int64_t offset, > + const uint8_t *buf, int count) > +{ > + BDRVRawState *s =3D bs->opaque; > + > + if (unlikely((s->flags & BDRV_O_DIRECT) && > + (offset % 512 !=3D 0 || (uintptr_t) buf % 512))) { > + > + int flags, ret; > + > + // Temporarily disable O_DIRECT for unaligned access > + flags =3D fcntl(s->fd, F_GETFL); > + fcntl(s->fd, F_SETFL, flags & ~O_DIRECT); > + ret =3D raw_pwrite_aligned(bs, offset, buf, count); > + fcntl(s->fd, F_SETFL, flags); ditto > + return ret; > + } else { > + return raw_pwrite_aligned(bs, offset, buf, count); > + } > +} > + > +#else > +#define raw_pread raw_pread_aligned > +#define raw_pwrite raw_pwrite_aligned > +#endif > + > + > /***********************************************************/ > /* Unix AIO using POSIX AIO */ > =20 >=20 Regards, Laurent --=20 ------------- Laurent.Vivier@bull.net --------------- "The best way to predict the future is to invent it." - Alan Kay