Skip to contentSkip to conversation

Simple db with separable io layer for humfs metadata.

uml-devel Piotr Neuman 2 participants 2 messages Apr 9, 2004 User-mode Linux (UML)

lore
Share & tools
lore
#0 lore raw

As Jeff Dike had requested that humfs db should allow UML to do actuall IO, I have found 2 simple db-es suitable for such task. I was however unable to find one that would have clear distinction of IO operations, like for example using another lib to do it. My findings:

TDB: http://sourceforge.net/projects/tdb/ (it's used in samba project)

Need only reimplement 2 pretty simple functions:
static int tdb_write(TDB_CONTEXT *tdb, tdb_off off, void *buf, tdb_len len) static int tdb_read(TDB_CONTEXT *tdb,tdb_off off,void *buf,tdb_len len,int cv)

The whole db is just two files: tdb.c and tdb.h which weight around 49KB, so shipping it along with UML would be no problem imo.

For example (to show difficulty of reimplementing):

static int tdb_write(TDB_CONTEXT *tdb, tdb_off off, void *buf, tdb_len len)

{
	if (tdb_oob(tdb, off + len, 0) != 0)
		return -1;

	if (tdb->map_ptr)
		memcpy(off + (char *)tdb->map_ptr, buf, len);
#ifdef HAVE_PWRITE
	else if (pwrite(tdb->fd, buf, len, off) != (ssize_t)len) {
#else
	else if (lseek(tdb->fd, off, SEEK_SET) != off
		 || write(tdb->fd, buf, len) != (ssize_t)len) {
#endif
		TDB_LOG((tdb, 0,"tdb_write failed at %d len=%d (%s)\n",
			   off, len, strerror(errno)));
		return TDB_ERRCODE(TDB_ERR_IO, -1);
	}
	return 0;
}

QDBM: http://qdbm.sourceforge.net/

In this case need to reimplement 4 functions (also simple):
static int dpwrite(int fd, const void *buf, int size); static int dpseekwrite(int fd, int off, const void *buf, int size); static int dpread(int fd, void *buf, int size);
static int dpseekread(int fd, int off, void *buf, int size);

Basic setup is: depot.c, depot.h, myconf.c and myconf.h - ~86KB. But additional b+tree support allows for gracious recovery after crash, and smaller file size than tdb at the cost of increasing code size to ~308KB. qdbm is generally much more advanced and featurefull than tdb. It also has plenty of interfaces to other languages, like perl (not counted in that size approximation).

Example:

static int dpseekwrite(int fd, int off, const void *buf, int size){
char *lbuf;
assert(fd >= 0 && buf && size >= 0);
if(size < 1) return TRUE;
lbuf = (char *)buf;

  if(off < 0){
    if(lseek(fd, 0, SEEK_END) == -1){
      dpecode = DP_ESEEK;
      return FALSE;
    }
  } else {
    if(lseek(fd, off, SEEK_SET) != off){
      dpecode = DP_ESEEK;
      return FALSE;
    }
  }
  if(dpwrite(fd, lbuf, size) != size){
    dpecode = DP_EWRITE;
    return FALSE;

}
return TRUE;
}

Regards to UML developement team (Jeff Dike that is ;)

-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#1 lore raw
Quoted 6 lines · [email protected] said:

TDB: http://sourceforge.net/projects/tdb/ (it's used in samba project)

Need only reimplement 2 pretty simple functions:
static int tdb_write(TDB_CONTEXT *tdb, tdb_off off, void *buf, tdb_len len) static int tdb_read(TDB_CONTEXT *tdb,tdb_off off,void *buf,tdb_len len,int cv)

Quoted 7 lines

QDBM: http://qdbm.sourceforge.net/

In this case need to reimplement 4 functions (also simple):
static int dpwrite(int fd, const void *buf, int size); static int dpseekwrite(int fd, int off, const void *buf, int size); static int dpread(int fd, void *buf, int size);
static int dpseekread(int fd, int off, void *buf, int size);

That's exactly the sort of thing I was looking for.

The reason I want to do IO on behalf of any DB that gets linked into UML is that I have a long-term goal of UML never blocking for any reason, except in its own idle loop. This mostly includes waiting for disk IO, i.e. for normal IO, swapping or faulting in pages, populating mmap areas, etc. So, I especially want no blocking IO. I want UML to be able to schedule other things while it's waiting for IO.

So, when it comes to databases, I want to be able to do AIO on their behalf, do other things while the IO is happening, and let them know when it has finished.

Jeff

-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel