mpt2sas logged messages
We have a new toy to test very large & slow storage with built up from 5 SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).
The basic idea is to build a cheap & slow test bed for file & storage system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1': File name too long
The second is that the mpt2sas driver spews various messages like:
Aug 17 06:55:17 megadeth kernel: mpt2sas0: log_info(0x31120102): originator(PL), code(0x12), sub_code(0x0102)
Any insight into whether these are issues worth pursuing is appreciated, thanks!
Ric
Quoted 11 lines · Ric Wheeler
We have a new toy to test very large & slow storage with built up from 5 SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).
The basic idea is to build a cheap & slow test bed for file & storage system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file '/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1': File name too long
Quoted 9 lines
The second is that the mpt2sas driver spews various messages like:
Aug 17 06:55:17 megadeth kernel: mpt2sas0: log_info(0x31120102): originator(PL), code(0x12), sub_code(0x0102)Any insight into whether these are issues worth pursuing is appreciated, thanks!
Ric
I left out the interesting part - this was an F11 system with a 2.6.31-rc6 kernel. I am updating the box to F12 rawhide to make sure that udev and related components are updated and will retest,
Ric
Quoted 10 lines · Ric Wheeler
We have a new toy to test very large & slow storage with built up from 5 SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).
The basic idea is to build a cheap & slow test bed for file & storage system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.
Quoted 8 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too long
Odd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
Quoted 16 lines · Greg KH
Quoted 15 lines · Ric Wheeler
We have a new toy to test very large & slow storage with built up from 5 SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).
The basic idea is to build a cheap & slow test bed for file & storage system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Quoted 6 lines
Quoted 3 lines
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longOdd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
In terms of tree depth, there's no real limit (although the deeper it goes, the slower the network). The examples in the docs usually go about 4 expanders deep.
James
Quoted 20 lines · Greg KH
Quoted 10 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5 SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).
The basic idea is to build a cheap& slow test bed for file& storage system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.
Quoted 8 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too long
Quoted 6 lines
Odd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
There are two dual-port SAS HBA's in the server that plug into the first of the 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy chained to the next shelf.... This test is running with just the first 4 shelves active (although that fifth shelf is plugged in, just not used/active).
Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both loops.
We could break up the shelves into two independent sets of devices which would limit the tree depth.
How can I get you the sysfs tree information in a useful way?
Ric
Quoted 33 lines · Ric Wheeler
Quoted 20 lines · Greg KH
Quoted 10 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5 SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).
The basic idea is to build a cheap& slow test bed for file& storage system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.
Quoted 8 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longQuoted 6 lines
Odd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
There are two dual-port SAS HBA's in the server that plug into the first of the 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy chained to the next shelf.... This test is running with just the first 4 shelves active (although that fifth shelf is plugged in, just not used/active).
Quoted 7 lines
Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both loops.
We could break up the shelves into two independent sets of devices which would limit the tree depth.
How can I get you the sysfs tree information in a useful way?
'tree /sys/devices/'
or
'find /sys/devices/'
would be good.
thanks,
greg k-h
Quoted 5 lines · Ric Wheeler
The second is that the mpt2sas driver spews various messages like:
Aug 17 06:55:17 megadeth kernel: mpt2sas0: log_info(0x31120102):originator(PL), code(0x12),
sub_code(0x0102)
Ric -
The code (0x12) means PL_LOGINFO_CODE_ABORT
The sub_code(0x102) means PL_LOGINFO_SUB_CODE_OPEN_FAILURE_PATHWAY_BLOCKED
Eric Moore
Quoted 44 lines · Greg KH
Quoted 34 lines · Ric Wheeler
Quoted 20 lines · Greg KH
Quoted 9 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.Quoted 9 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longQuoted 7 lines
Odd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
There are two dual-port SAS HBA's in the server that plug into the first of the 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy chained to the next shelf.... This test is running with just the first 4 shelves active (although that fifth shelf is plugged in, just not used/active).
Quoted 8 lines
Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both loops.
We could break up the shelves into two independent sets of devices which would limit the tree depth.
How can I get you the sysfs tree information in a useful way?
Quoted 9 lines
'tree
/sys/devices/'
or
'find/sys/devices/'
would be good.thanks,
greg k-h
Attached is the bzipped output from - the uncompressed output is quite large. Note that this same server has CCIS controllers and fibre HBA's as well,
ric
Quoted 50 lines · Ric Wheeler
Quoted 49 lines · Greg KH
Quoted 34 lines · Ric Wheeler
Quoted 20 lines · Greg KH
Quoted 9 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.Quoted 9 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longQuoted 7 lines
Odd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
There are two dual-port SAS HBA's in the server that plug into the first of the 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy chained to the next shelf.... This test is running with just the first 4 shelves active (although that fifth shelf is plugged in, just not used/active).
Quoted 8 lines
Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both loops.
We could break up the shelves into two independent sets of devices which would limit the tree depth.
How can I get you the sysfs tree information in a useful way?
'tree
/sys/devices/'
or
'find/sys/devices/'
would be good.
Quoted 8 lines
Quoted 4 lines
thanks,
greg k-h
Attached is the bzipped output from - the uncompressed output is quite large. Note that this same server has CCIS controllers and fibre HBA's as well,
It's perfectly legal the way you have it, but I will say you have an inefficient configuration. The way a configuration like this is supposed to look is that there should be a fanout expander at the top going to the expander in each tray, for a routing depth of two for every device.
The way you've got it: expander daisy chained off the next expander gives an unnecessary routing delay to the disks furthest away in the chain.
James
Quoted 11 lines · Eric
Quoted 5 lines · Ric Wheeler
The second is that the mpt2sas driver spews various messages like:
Aug 17 06:55:17 megadeth kernel: mpt2sas0: log_info(0x31120102):originator(PL), code(0x12),
sub_code(0x0102)Ric -
The code (
0x12) meansPL_LOGINFO_CODE_ABORT
Thesub_code(0x102)meansPL_LOGINFO_SUB_CODE_OPEN_FAILURE_PATHWAY_BLOCKED
To translate this according to the SAS 1.1 specification, a pathway open failed because a timeout fired before the path could be established.
You can mitigate this by increasing the expander partial pathway timeout value. (or just move to a more efficient configuration).
James
Quoted 16 lines · Ric Wheeler
Quoted 15 lines · Greg KH
Quoted 13 lines · Ric Wheeler
Quoted 11 lines · Greg KH
Quoted 9 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too long
Ok, it looks like the way that udev handles it's internal "database" can't handle paths that are larger than a single filename size.
Kay, I think this is all yours :)
thanks,
greg k-h
Quoted 4 lines · James Bottomley
Quoted 2 lines
The code (
0x12) meansPL_LOGINFO_CODE_ABORT
Thesub_code(0x102)means
PL_LOGINFO_SUB_CODE_OPEN_FAILURE_PATHWAY_BLOCKED
Quoted 8 lines
To translate this according to the SAS 1.1 specification, a pathway open
failed because a timeout fired before the path could be established.You can mitigate this by increasing the expander partial pathway timeout
value. (or just move to a more efficient configuration).
I was telling Ric it was something else in seperate email. The pathway was blocked due to two initiors establishing a connection to single port. That woudl be the case if he wasn't using a sata multiplexor.
anyways, if the pathway timeout is the issue, then I believe you can set the pathway timeout value using Dougs smp tools, smp_phy_control , setting the --pptv command line option.
Quoted 66 lines · James Bottomley
Quoted 50 lines · Ric Wheeler
Quoted 49 lines · Greg KH
Quoted 34 lines · Ric Wheeler
Quoted 20 lines · Greg KH
Quoted 9 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.Quoted 9 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longQuoted 7 lines
Odd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
There are two dual-port SAS HBA's in the server that plug into the first of the 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy chained to the next shelf.... This test is running with just the first 4 shelves active (although that fifth shelf is plugged in, just not used/active).
Quoted 8 lines
Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both loops.
We could break up the shelves into two independent sets of devices which would limit the tree depth.
How can I get you the sysfs tree information in a useful way?
'tree
/sys/devices/'
or
'find/sys/devices/'
would be good.Quoted 8 lines
Quoted 4 lines
thanks,
greg k-h
Attached is the bzipped output from - the uncompressed output is quite large. Note that this same server has CCIS controllers and fibre HBA's as well,
It's perfectly legal the way you have it, but I will say you have an inefficient configuration. The way a configuration like this is supposed to look is that there should be a fanout expander at the top going to the expander in each tray, for a routing depth of two for every device.
Quoted 5 lines
The way you've got it: expander daisy chained off the next expander gives an unnecessary routing delay to the disks furthest away in the chain.
James
I understand that this configuration is not optimal, but from what I saw with some commercial arrays, this is not an uncommon config (up to 4 shelves) when going for capacity over performance.
Do you have a particular SAS fanout expander in mind? I suppose that we could always add more hba's as well which would have other benefits...
Thanks!
Ric
Quoted 26 lines · Greg KH
Quoted 16 lines · Ric Wheeler
Quoted 15 lines · Greg KH
Quoted 13 lines · Ric Wheeler
Quoted 11 lines · Greg KH
Quoted 9 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too long
Quoted 8 lines
Ok, it looks like the way that udev handles it's internal "database" can't handle paths that are larger than a single filename size.
Kay, I think this is all yours :)
thanks,
greg k-h
Happy to try patches, also happy to try and work around this with a different configuration :-)
ric
Quoted 14 lines · Eric
Quoted 4 lines · James Bottomley
Quoted 2 lines
The code (
0x12) meansPL_LOGINFO_CODE_ABORT
Thesub_code(0x102)means
PL_LOGINFO_SUB_CODE_OPEN_FAILURE_PATHWAY_BLOCKEDQuoted 8 lines
To translate this according to the SAS 1.1 specification, a pathway open
failed because a timeout fired before the path could be established.You can mitigate this by increasing the expander partial pathway timeout
value. (or just move to a more efficient configuration).
Quoted 6 lines
I was telling Ric it was something else in seperate email. The pathway was blocked due to two initiors establishing a connection to single port. That woudl be the case if he wasn't using a sata multiplexor.
anyways, if the pathway timeout is the issue, then I believe you can set the pathway timeout value using Dougs smp tools,
smp_phy_control, setting the --pptv command line option.
Thanks, I will give that a try,
ric
Quoted 22 lines · Greg KH
Quoted 21 lines · Ric Wheeler
Quoted 20 lines · Greg KH
Quoted 18 lines · Ric Wheeler
Quoted 16 lines · Greg KH
Quoted 14 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5 SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).
The basic idea is to build a cheap& slow test bed for file& storage system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longOk, it looks like the way that udev handles it's internal "database" can't handle paths that are larger than a single filename size.
Kay, I think this is all yours :)
Yeah, seems so. Linux can not have filenames longer than 256. We'll need to change the way we store the DEVPATH in a filename. I'll look into it.
Thanks,
Kay
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 30 lines · Kay Sievers
Quoted 29 lines · Greg KH
Quoted 16 lines · Ric Wheeler
Quoted 15 lines · Greg KH
Quoted 13 lines · Ric Wheeler
Quoted 11 lines · Greg KH
Quoted 9 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longOk, it looks like the way that udev handles it's internal "database" can't handle paths that are larger than a single filename size.
Quoted 8 lines
Quoted 1 line
Kay, I think this is all yours :)
Yeah, seems so. Linux can not have filenames longer than 256. We'll need to change the way we store the DEVPATH in a filename. I'll look into it.
Thanks,
Kay
Hi Kay,
I think that we can reconfigure the storage to avoid this but will be happy to restore this configuration if you have something that you want to test later on...
Thanks!
Ric
Quoted 71 lines · Ric Wheeler
Quoted 70 lines · James Bottomley
Quoted 50 lines · Ric Wheeler
Quoted 49 lines · Greg KH
Quoted 34 lines · Ric Wheeler
Quoted 20 lines · Greg KH
Quoted 9 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.Quoted 9 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longQuoted 7 lines
Odd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
There are two dual-port SAS HBA's in the server that plug into the first of the 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy chained to the next shelf.... This test is running with just the first 4 shelves active (although that fifth shelf is plugged in, just not used/active).
Quoted 8 lines
Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both loops.
We could break up the shelves into two independent sets of devices which would limit the tree depth.
How can I get you the sysfs tree information in a useful way?
'tree
/sys/devices/'
or
'find/sys/devices/'
would be good.Quoted 8 lines
Quoted 4 lines
thanks,
greg k-h
Attached is the bzipped output from - the uncompressed output is quite large. Note that this same server has CCIS controllers and fibre HBA's as well,
It's perfectly legal the way you have it, but I will say you have an inefficient configuration. The way a configuration like this is supposed to look is that there should be a fanout expander at the top going to the expander in each tray, for a routing depth of two for every device.
The way you've got it: expander daisy chained off the next expander gives an unnecessary routing delay to the disks furthest away in the chain.
Quoted 5 lines
Quoted 1 line
James
I understand that this configuration is not optimal, but from what I saw with some commercial arrays, this is not an uncommon config (up to 4 shelves) when going for capacity over performance.
The config is fine ... it's the way the daisy chain routing is done which isn't. You can see that each shelf gets further away from the HBA by an expander as you go up.
Quoted 2 lines
Do you have a particular SAS fanout expander in mind? I suppose that we could always add more hba's as well which would have other benefits...
Not really ... I've never seen a real expander in the flesh. I've got a set of experimental ones LSI gave me (as bare circuit boards).
If you don't have the expanders, you can likely rig the first expander to act as a fanout since it must have table routed ports otherwise it wouldn't work in the daisy chain.
Failing that, as you suggest, multiple HBAs subbing for the fanout expander would be fine as well.
James
Quoted 88 lines · James Bottomley
Quoted 71 lines · Ric Wheeler
Quoted 70 lines · James Bottomley
Quoted 50 lines · Ric Wheeler
Quoted 49 lines · Greg KH
Quoted 34 lines · Ric Wheeler
Quoted 20 lines · Greg KH
Quoted 9 lines · Ric Wheeler
We have a new toy to test very large& slow storage with built up from 5
SAS expansion shelves (Promise Vtrak J-Class) with 60 S-ATA drives and 16 SAS drives (the S-ATA drives each have a Promise Vtrak S-ATA MUX adapter daughter card in the disk sled).The basic idea is to build a cheap& slow test bed for file& storage
system scalability. Collectively, we have about 120TB (raw) of capacity to play with in one server.Quoted 9 lines
As we work through various issues, a couple of oddities popped out.
The first is that udev grumbles during boot about "file name too long" like the following:
Aug 17 06:49:58 megadeth udevd-event[20447]: unable to create db file'/dev/.udev/db/\x2fdevices\x2fpci0000:00\x2f0000:00:04.0\x2f0000:17:00.0\x2f0000:18:0a.0\x2f0000:1f:00.0\x2fhost11\x2fport-11:0\x2fexpander-11:0\x2fport-11:0:0\x2fexpander-11:1\x2fport-11:1:0\x2fexpander-11:2\x2fport-11:2:17\x2fexpander-11:3\x2fport-11:3:1\x2fend_device-11:3:1\x2fbsg\x2fend_device-11:3:1':
File name too longQuoted 7 lines
Odd, what is the sysfs tree for this device? You have expanders attached to ports attached to expanders? How deep can you go?
thanks,
greg k-h
There are two dual-port SAS HBA's in the server that plug into the first of the 5 SAS expansion shelves. Each shelf has two internal SAS loops and is daisy chained to the next shelf.... This test is running with just the first 4 shelves active (although that fifth shelf is plugged in, just not used/active).
Quoted 8 lines
Each of the 60 S-ATA disks sits behind a MUX card which lets it appear on both loops.
We could break up the shelves into two independent sets of devices which would limit the tree depth.
How can I get you the sysfs tree information in a useful way?
'tree
/sys/devices/'
or
'find/sys/devices/'
would be good.Quoted 8 lines
Quoted 4 lines
thanks,
greg k-h
Attached is the bzipped output from - the uncompressed output is quite large. Note that this same server has CCIS controllers and fibre HBA's as well,
It's perfectly legal the way you have it, but I will say you have an inefficient configuration. The way a configuration like this is supposed to look is that there should be a fanout expander at the top going to the expander in each tray, for a routing depth of two for every device.
The way you've got it: expander daisy chained off the next expander gives an unnecessary routing delay to the disks furthest away in the chain.
Quoted 5 lines
Quoted 1 line
James
I understand that this configuration is not optimal, but from what I saw with some commercial arrays, this is not an uncommon config (up to 4 shelves) when going for capacity over performance.
The config is fine ... it's the way the daisy chain routing is done which isn't. You can see that each shelf gets further away from the HBA by an expander as you go up.
Quoted 2 lines
Do you have a particular SAS fanout expander in mind? I suppose that we could always add more hba's as well which would have other benefits...
Not really ... I've never seen a real expander in the flesh. I've got a set of experimental ones LSI gave me (as bare circuit boards).
Quoted 8 lines
If you don't have the expanders, you can likely rig the first expander to act as a fanout since it must have table routed ports otherwise it wouldn't work in the daisy chain.
Failing that, as you suggest, multiple HBAs subbing for the fanout expander would be fine as well.
James
Adding more HBA's (one per shelf or pair of shelves) would probably be the easiest way around this....
Thanks!
Ric
Quoted 3 lines · Ric Wheeler
I think that we can reconfigure the storage to avoid this but will be happy to restore this configuration if you have something that you want to test later on...
This hopefully addresses the issue:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168
Thanks,
Kay
Quoted 11 lines · Kay Sievers
Quoted 3 lines · Ric Wheeler
I think that we can reconfigure the storage to avoid this but will be happy to restore this configuration if you have something that you want to test later on...
This hopefully addresses the issue:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168Thanks,
Kay
Great, I will give these a shot - thanks!
Ric
Quoted 11 lines · Kay Sievers
Quoted 3 lines · Ric Wheeler
I think that we can reconfigure the storage to avoid this but will be happy to restore this configuration if you have something that you want to test later on...
This hopefully addresses the issue:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168Thanks,
Kay
Hi Kay,
This seems to resolve the issue on my test rig - no more "too long" error messages during boot.
Many thanks!
Ric
Quoted 12 lines · Ric Wheeler
Quoted 11 lines · Kay Sievers
Quoted 4 lines · Ric Wheeler
I think that we can reconfigure the storage to avoid this but will be happy
to restore this configuration if you have something that you want to test later on...This hopefully addresses the issue:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168
Quoted 7 lines
Quoted 2 lines
Thanks,
KayHi Kay,
This seems to resolve the issue on my test rig - no more "too long" error messages during boot.
Great! Thanks a lot for the testing, and letting us know. It's a pleasure when things work like this.
Kay
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 18 lines · Kay Sievers
Quoted 17 lines · Ric Wheeler
Quoted 7 lines · Kay Sievers
Quoted 4 lines · Ric Wheeler
I think that we can reconfigure the storage to avoid this but will be happy
to restore this configuration if you have something that you want to test later on...Quoted 6 lines
This hopefully addresses the issue:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;h=2ffc9cc1917b1bb6fe86881a94a47dce9aa15168
Thanks,
KayHi Kay,
Quoted 7 lines
Quoted 2 lines
This seems to resolve the issue on my test rig - no more "too long" error messages during boot.
Great! Thanks a lot for the testing, and letting us know. It's a pleasure when things work like this.
Kay
The pleasure is all mine - great to get such rapid fixes :-)
It has been a long time challenge for us to get developers on big systems, especially big and complicated storage boxes which is why we decided to try and build this box to help us stress scale in every way so we can track down exactly this kind of issue in a timely way.
It was not that expensive to build a 100+ TB box - 4 SAS expansion shelves, 60 2TB S-ATA disks and a couple of HBA's for a reasonable server. It won't be that fast, but is much cheaper than buying a similar array from a real vendor...
ric