Skip to contentSkip to conversation

pNFS-related questions

DISCUSSION Jiri Kortus 2 participants 6 messages May 12, 2010 – May 13, 2010 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I'd like to ask two questions that are related to pNFS, I hope it's not a problem that I put them in one post even though they aren't related one to the other.

1. How to configure a pNFS+exofs setup with multiple OSD targets? I was able to use a setup with one OSD target, one OSD initiator+pNFS metadata server and one client with the help of pNFS mailing list and the instructions from here http://mailman.open-osd.org/pipermail/osd-dev/2009-July/001033.html which seems to be quite clear to me. But I've been thinking about how a setup with more OSDs would look like? (I mean mainly the metadata server). I suppose that I would configure more OSDs in the same way as I did with one. Then how about the pNFS MDS setup? I've been wondering if I could do it this way:
- login to the targets, then mount /dev/osdX devices for example in /exofs/osdX
- set up the /etc/exports file in this way:
/exofs *(....,fsid=0,pnfs)
/exofs/osd1 *(....)
/exofs/osd2 *(....)
...
/exofs/osdX *(....)

- then reload the exports by running "exportfs -r"
- and finally I would login to the OSDs on the client, load objlayoutdriver and mount the export from MDS with the "minorversion=1" option.
Could the content of /etc/exports file look like that? Or is it fundamentally wrong? I'd be glad for any advice. I can't test it right now but I have to know how to do it to describe it in my bachelor's thesis.

2. I'd like to know how well the IPv6 support in pNFS works - what works, what doesn't, if there are any problems to solve etc., just anything worth to know concerning the IPv6 support.

Thank you very much in advance.

Best regards,

Jiri Kortus

#1 lore raw
Quoted 16 lines · Jiri Kortus

Hi,

I'd like to ask two questions that are related to pNFS, I hope it's not a problem that I put them in one post even though they aren't related one to the other.

1. How to configure a pNFS+exofs setup with multiple OSD targets? I was able to use a setup with one OSD target, one OSD initiator+pNFS metadata server and one client with the help of pNFS mailing list and the instructions from here
http://mailman.open-osd.org/pipermail/osd-dev/2009-July/001033.html
which seems to be quite clear to me. But I've been thinking about how a setup with more OSDs would look like? (I mean mainly the metadata server). I suppose that I would configure more OSDs in the same way as I did with one. Then how about the pNFS MDS setup? I've been wondering if I could do it this way:

Quoted 9 lines

- login to the targets, then mount /dev/osdX devices for example in /exofs/osdX
- set up the /etc/exports file in this way:
/exofs *(....,fsid=0,pnfs)
/exofs/osd1 *(....)
/exofs/osd2 *(....)
...
/exofs/osdX *(....)

No not at all! ;-)

I'll be a paper about an experiment I did on Panasas labs with 8 osds and 8 clients 1 MDS. I'll be posting the complete scripts I used for setup and tests

So it is much more simple really.

- On X machines you just run ./up to load the osd-target.
You now have X ip addresses/names
- On MDS/client you change your ./do-osd script to look like this (I'm attaching the complete script) login_iscsi_osd()
{
echo login into: $1 expect $2
if $iscsiadm $ISCSI_DBG -m discovery -t sendtargets -p $1 --login; then
echo $iscsiadm $ISCSI_DBG -m discovery -t sendtargets -p $1 --login returned error ($?)! sleep 3;
fi

sleep 1;
ls $2
}
And then the do_start changes to:
start_iscsi_intiator

start_open_osd

login_iscsi_osd $IP_OSD0 $DEV_OSD0
login_iscsi_osd $IP_OSD1 $DEV_OSD1
login_iscsi_osd $IP_OSD2 $DEV_OSD2
login_iscsi_osd $IP_OSD3 $DEV_OSD3

login_iscsi_osd $IP_OSD4 $DEV_OSD4
login_iscsi_osd $IP_OSD5 $DEV_OSD5
login_iscsi_osd $IP_OSD6 $DEV_OSD6
login_iscsi_osd $IP_OSD7 $DEV_OSD7

So basically before hand on client and MDS you login to all osds

- On MDS at one time you do a "format" (see attached script) which mkfs.exofs of a file systems with all the OSDs included (Yes exofs is multy device that's the all point. mirrors strips and groups are all supported)

First do a format on MDS. Then do ./do-osd stop. Then do-osd start on MDS and all clients. Because format changes the osd-names and they must get updated in memory.

- Then you do the export just exactly as before just one /exp/exofs and exportfs -r. (Again this does not change from before)

- Finally you do the do-pnfs start on clients exactlly the same as before. There is a single file system. The pnfs/osd protocol takes care of all the multiple devices and file layouts.

So to sumerize the only thing different from before is that you need to iscsi login to more then one OSD. And that the mkfs.exofs command has lots of devices. (Or you can wait for my paper next week)

Boaz

Quoted 15 lines

- then reload the exports by running "exportfs -r"
- and finally I would login to the OSDs on the client, load objlayoutdriver and mount the export from MDS with the "minorversion=1" option.
Could the content of /etc/exports file look like that? Or is it fundamentally wrong? I'd be glad for any advice. I can't test it right now but I have to know how to do it to describe it in my bachelor's thesis.

2. I'd like to know how well the IPv6 support in pNFS works - what works, what doesn't, if there are any problems to solve etc., just anything worth to know concerning the IPv6 support.

Thank you very much in advance.

Quoted 8 lines

Best regards,

Jiri Kortus
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#!/bin/sh
#
# ect: Exofs Cluster Test

modprobe=/sbin/modprobe
service=/sbin/service
iscsiadm=/sbin/iscsiadm

# Name of iscsi service. .e.g at /etc/init.d/
ISCSI=iscsi

# 8 osds
DEV_OSD0=/dev/osd0
UUID0="8c0772c1-3b9b-44c0-bc28-dff56354ed39"

DEV_OSD1=/dev/osd1
UUID1="4680a768-e586-4d42-b4b6-43fdc5f02304"

DEV_OSD2=/dev/osd2
UUID2="f71a5dbd-49c8-4558-bba8-a1bb12c8bff6"

DEV_OSD3=/dev/osd3
UUID3="e432ae72-81d6-49ab-8b61-93b5bb0268e3"

DEV_OSD4=/dev/osd4
UUID4="7f518365-1c3b-4ce1-a511-40eabad1ce58"

DEV_OSD5=/dev/osd5
UUID5="93edbb0a-cf91-492a-a4af-931be496fd55"

DEV_OSD6=/dev/osd6
UUID6="fbd8966e-0ff0-4965-baa5-4ceff832d5f8"

DEV_OSD7=/dev/osd7
UUID7="c1462bab-999f-4a25-a95d-ebd158818bf1"

IP_OSD0=10.70.4.120:3260
IP_OSD1=10.70.4.121:3260
IP_OSD2=10.70.4.122:3260
IP_OSD3=10.70.4.123:3260

IP_OSD4=10.70.4.124:3260
IP_OSD5=10.70.4.125:3260
IP_OSD6=10.70.4.126:3260
IP_OSD7=10.70.4.127:3260

# mkexofs layout
OO_ROOT=`dirname $0`
EXOFS_DIR=$OO_ROOT/fs/exofs
USR_DIR=$OO_ROOT/usr
MOUNTDIR=/mnt/exofs

PID=0x10000
PID4=0x20004
PID2=0x20002
PID1=0x20001

FORMAT_SIZE=0
RAID=0
MIRRORS=0
STRIPE_PAGES=16
GROUP_WIDTH=0
GROUP_DEPTH=0

do_cmd()

{
	$* 2>&1 | logger &
}

start_iscsi_intiator()
{
echo start iscsi service

if ! $service $ISCSI status; then

echo iscsi not started running [$service $ISCSI start]

do_cmd $service $ISCSI start ;

until cat /sys/class/iscsi_transport/tcp/handle 2>/dev/null ; do
echo wait for iscsi-transport
sleep 1;
done
fi

sleep 7
echo start iscsi done
}
stop_iscsi_intiator()
{
echo stop iscsi

do_cmd $service $ISCSI stop

until [ -n $modprobe -l scsi_transport_iscsi ]; do
sleep 1;
done

echo stop iscsi done
}

ISCSIDB=/var/lib/iscsi
# ISCSIDB=/etc/iscsi
do_iscsi_clean()
{
rm -rfv $ISCSIDB/ifaces/*
rm -rfv $ISCSIDB/nodes/*
rm -rfv $ISCSIDB/send_targets/*
}

# ISCSI_DBG=--debug=8

# usage login_iscsi_osd $IP_OSDX $DEV_OSDX
login_iscsi_osd()
{
	echo login into: $1 expect $2
	if $iscsiadm $ISCSI_DBG -m discovery -t sendtargets -p $1 --login; then 
		echo $iscsiadm $ISCSI_DBG -m discovery -t sendtargets -p $1 --login returned error ($?)!
		sleep 3; 
	fi
	sleep 1; 
	ls $2 
}
# usage logout_iscsi_osd $IP_OSDX $DEV_OSDX
logout_iscsi_osd()
{
	echo login out of $1 expected on $2
	do_cmd $iscsiadm -m node -p $1 --logout

# while ls $3 2>/dev/null; do sleep 1; done

	sleep 1;
}

start_open_osd()
{
# In-tree

	echo start osd

	$modprobe osd;
}
stop_open_osd()
{
# In-tree
	echo stop osd

	$modprobe --remove osd;
}

do_start()
{
start_iscsi_intiator

start_open_osd

login_iscsi_osd $IP_OSD0 $DEV_OSD0
login_iscsi_osd $IP_OSD1 $DEV_OSD1
login_iscsi_osd $IP_OSD2 $DEV_OSD2
login_iscsi_osd $IP_OSD3 $DEV_OSD3

login_iscsi_osd $IP_OSD4 $DEV_OSD4
login_iscsi_osd $IP_OSD5 $DEV_OSD5
login_iscsi_osd $IP_OSD6 $DEV_OSD6
login_iscsi_osd $IP_OSD7 $DEV_OSD7
}
do_stop()
{
logout_iscsi_osd $IP_OSD7 $DEV_OSD7
logout_iscsi_osd $IP_OSD6 $DEV_OSD6
logout_iscsi_osd $IP_OSD5 $DEV_OSD5
logout_iscsi_osd $IP_OSD4 $DEV_OSD4

logout_iscsi_osd $IP_OSD3 $DEV_OSD3
logout_iscsi_osd $IP_OSD2 $DEV_OSD2
logout_iscsi_osd $IP_OSD1 $DEV_OSD1
logout_iscsi_osd $IP_OSD0 $DEV_OSD0

stop_iscsi_intiator

stop_open_osd
}

mkexofs_format()

{
	$USR_DIR/mkfs.exofs \
		--pid=$PID --raid=$RAID --mirrors=$MIRRORS \
		--stripe_pages=$STRIPE_PAGES \
		--group_width=$GROUP_WIDTH --group_depth=$GROUP_DEPTH \
		--dev=$DEV_OSD0 --format=$FORMAT_SIZE  --osdname=$UUID0 \
		--dev=$DEV_OSD1 --format=$FORMAT_SIZE  --osdname=$UUID1 \
		--dev=$DEV_OSD2 --format=$FORMAT_SIZE  --osdname=$UUID2 \
		--dev=$DEV_OSD3 --format=$FORMAT_SIZE  --osdname=$UUID3 \
		--dev=$DEV_OSD4 --format=$FORMAT_SIZE  --osdname=$UUID4 \
		--dev=$DEV_OSD5 --format=$FORMAT_SIZE  --osdname=$UUID5 \
		--dev=$DEV_OSD6 --format=$FORMAT_SIZE  --osdname=$UUID6 \
		--dev=$DEV_OSD7 --format=$FORMAT_SIZE  --osdname=$UUID7
}

mkexofs_mkfs4()

{
	$USR_DIR/mkfs.exofs \
		--pid=$PID4 --raid=$RAID --mirrors=$MIRRORS \
		--stripe_pages=$STRIPE_PAGES \
		--group_width=$GROUP_WIDTH --group_depth=$GROUP_DEPTH \
		--dev=$DEV_OSD0 \
		--dev=$DEV_OSD1 \
		--dev=$DEV_OSD2 \
		--dev=$DEV_OSD3
}

mkexofs_mkfs2()

{
	$USR_DIR/mkfs.exofs \
		--pid=$PID2 --raid=$RAID --mirrors=$MIRRORS \
		--stripe_pages=$STRIPE_PAGES \
		--group_width=$GROUP_WIDTH --group_depth=$GROUP_DEPTH \
		--dev=$DEV_OSD0 \
		--dev=$DEV_OSD1
}

mkexofs_mkfs1()

{
	$USR_DIR/mkfs.exofs \
		--pid=$PID1 --raid=$RAID --mirrors=$MIRRORS \
		--stripe_pages=$STRIPE_PAGES \
		--group_width=$GROUP_WIDTH --group_depth=$GROUP_DEPTH \
		--dev=$DEV_OSD0
}

do_mount()

{
	OPT="pid=$PID,_netdev"

	mount -t exofs -o $OPT $DEV_OSD0 $MOUNTDIR
}

do_umount()
{
umount $MOUNTDIR
}

PNFS_SRV=compute-4-18
PNFS_MNT=/mnt/pnfs

do_pnfs_start()
{
# service nfs start

mkdir -p $PNFS_MNT
modprobe objlayoutdriver
mount -t nfs4 -o minorversion=1 $PNFS_SRV:/ $PNFS_MNT
}
do_pnfs_stop()

{
	umount $PNFS_MNT
	modprobe --remove objlayoutdriver

# service nfs stop
}

do_pnfs_dd_test()

{
	# let in an IO for the grace period
	dd if=/dev/zero of=$PNFS_MNT/dd_$(hostname) bs=64k count=131072
}

prompt()

{
	read -p "$* >>> "
}

case $1 in
iscsi_clean)

	do_iscsi_clean
	;;

start)
echo $0 Starting | logger
do_start
echo $0 Initialized | logger
;;

stop)
echo $0 Stopping | logger
do_stop
echo $0 Stopped | logger
;;

format)

	prompt mkexofs_format
	mkexofs_format
	echo $0 format | logger
	;;

mkfs4)

	prompt mkexofs_mkfs4
	mkexofs_mkfs4
	echo $0 mkexofs_mkfs4 | logger
	;;
mkfs2)
	prompt mkexofs_mkfs2
	mkexofs_mkfs2
	echo $0 mkexofs_mkfs2 | logger
	;;
mkfs1)
	prompt mkexofs_mkfs1
	mkexofs_mkfs1
	echo $0 mkexofs_mkfs1 | logger
	;;

mount)
echo $0 mounting | logger
do_mount
echo $0 mounted | logger
;;

umount)
echo $0 umounting | logger
do_umount
echo $0 umounted | logger
;;

pnfs_start)
echo $0 mounting | logger
do_pnfs_start
echo $0 mounted | logger
;;

pnfs_stop)
echo $0 umounting | logger
do_pnfs_stop
echo $0 umounted | logger
;;

pnfs_dd_test)

	do_pnfs_dd_test
	;;

*)

	echo "!! iscsi_clean|start|stop|format|mount|umount|pnfs_start|pnfs_stop|pnfs_dd_test"
	;;

esac

#2 lore raw

Dne 13.5.2010 11:00, Boaz Harrosh napsal(a):

Quoted 80 lines · Jiri Kortus
Quoted 17 lines

Hi,

I'd like to ask two questions that are related to pNFS, I hope it's not a problem that I put them in one post even though they aren't related one to the other.

1. How to configure a pNFS+exofs setup with multiple OSD targets? I was able to use a setup with one OSD target, one OSD initiator+pNFS metadata server and one client with the help of pNFS mailing list and the instructions from here
http://mailman.open-osd.org/pipermail/osd-dev/2009-July/001033.html
which seems to be quite clear to me. But I've been thinking about how a setup with more OSDs would look like? (I mean mainly the metadata server). I suppose that I would configure more OSDs in the same way as I did with one. Then how about the pNFS MDS setup? I've been wondering if I could do it this way:

Quoted 10 lines

- login to the targets, then mount /dev/osdX devices for example in /exofs/osdX
- set up the /etc/exports file in this way:
/exofs *(....,fsid=0,pnfs)
/exofs/osd1 *(....)
/exofs/osd2 *(....)
...
/exofs/osdX *(....)

No not at all! ;-)

I'll be a paper about an experiment I did on Panasas labs with 8 osds and 8 clients 1 MDS. I'll be posting the complete scripts I used for setup and tests

So it is much more simple really.

- On X machines you just run ./up to load the osd-target.
You now have X ip addresses/names
- On MDS/client you change your ./do-osd script to look like this (I'm attaching the complete script) login_iscsi_osd()
{
echo login into: $1 expect $2
if $iscsiadm $ISCSI_DBG -m discovery -t sendtargets -p $1 --login; then
echo $iscsiadm $ISCSI_DBG -m discovery -t sendtargets -p $1 --login returned error ($?)! sleep 3;
fi

sleep 1;
ls $2
}
And then the do_start changes to:
start_iscsi_intiator

start_open_osd

login_iscsi_osd $IP_OSD0 $DEV_OSD0
login_iscsi_osd $IP_OSD1 $DEV_OSD1
login_iscsi_osd $IP_OSD2 $DEV_OSD2
login_iscsi_osd $IP_OSD3 $DEV_OSD3

login_iscsi_osd $IP_OSD4 $DEV_OSD4
login_iscsi_osd $IP_OSD5 $DEV_OSD5
login_iscsi_osd $IP_OSD6 $DEV_OSD6
login_iscsi_osd $IP_OSD7 $DEV_OSD7

So basically before hand on client and MDS you login to all osds

- On MDS at one time you do a "format" (see attached script) which mkfs.exofs of a file systems with all the OSDs included (Yes exofs is multy device that's the all point. mirrors strips and groups are all supported)

First do a format on MDS. Then do ./do-osd stop. Then do-osd start on MDS and all clients. Because format changes the osd-names and they must get updated in memory.

- Then you do the export just exactly as before just one /exp/exofs and exportfs -r. (Again this does not change from before)

- Finally you do the do-pnfs start on clients exactlly the same as before. There is a single file system. The pnfs/osd protocol takes care of all the multiple devices and file layouts.

Quoted 6 lines

So to sumerize the only thing different from before is that you need to iscsi login to more then one OSD. And that the mkfs.exofs command has lots of devices. (Or you can wait for my paper next week)

Boaz

Hi Boaz,

thank you for describing how it works :) It looks good and quite simple to grasp.
I'd like to ask you whether you would be also so kind as to write me some information about the IPv6 support?

Thank you very much.

Best regards,

Jiri

Quoted 25 lines
Quoted 24 lines

- then reload the exports by running "exportfs -r"
- and finally I would login to the OSDs on the client, load objlayoutdriver and mount the export from MDS with the "minorversion=1" option.
Could the content of /etc/exports file look like that? Or is it fundamentally wrong? I'd be glad for any advice. I can't test it right now but I have to know how to do it to describe it in my bachelor's thesis.

2. I'd like to know how well the IPv6 support in pNFS works - what works, what doesn't, if there are any problems to solve etc., just anything worth to know concerning the IPv6 support.

Thank you very much in advance.

Best regards,

Jiri Kortus
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
In reply to a message not in the archive
Quoted 7 lines · Jiri Kortus

Hi Boaz,

thank you for describing how it works :) It looks good and quite simple to grasp.
I'd like to ask you whether you would be also so kind as to write me some information about the IPv6 support?

Quoted 6 lines

Thank you very much.

Best regards,

Jiri

No idea! :-(

I don't have any ipv6 setup anywhere

Boaz

#5 lore raw
Quoted 12 lines · Boaz Harrosh
Quoted 11 lines · Jiri Kortus

Hi Boaz,

thank you for describing how it works :) It looks good and quite simple to grasp.
I'd like to ask you whether you would be also so kind as to write me some information about the IPv6 support?

Thank you very much.

Best regards,

Quoted 8 lines
Quoted 2 lines

Jiri

No idea! :-(

I don't have any ipv6 setup anywhere

Boaz

BTW tgtd does not support it yet as well. iscsi-initiator does. osd does not care it is all hidden under the iscsi stack.

Boaz

#6 lore raw

Dne 13.5.2010 17:35, Boaz Harrosh napsal(a):

Quoted 20 lines · Boaz Harrosh
Quoted 19 lines · Jiri Kortus
Quoted 8 lines

Hi Boaz,

thank you for describing how it works :) It looks good and quite simple to grasp.
I'd like to ask you whether you would be also so kind as to write me some information about the IPv6 support?

Quoted 7 lines

Thank you very much.

Best regards,

Jiri

No idea! :-(

Quoted 9 lines
Quoted 4 lines

I don't have any ipv6 setup anywhere

Boaz

BTW tgtd does not support it yet as well. iscsi-initiator does. osd does not care it is all hidden under the iscsi stack.

Boaz

Thanks, Boaz! Even this is good to know.

Would someone else who knows something about IPv6 and pNFS (in general, i. e. all kinds of layouts, just anything concerning the domain of IPv6 and pNFS) be so kind as to write me something about it or at least give some hint where I can find more?

Thanks in advance,

Jiri