Skip to contentSkip to conversation

st.c block limits

DISCUSSION ael 4 participants 7 messages Jul 30, 2002 – Aug 1, 2002 SCSI

lore
Share & tools
lore

ael

author
#0 lore raw

I have a scsi tape drive:-

Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: AIWA Model: TD-20001 Rev: 0159
Type: Sequential-Access ANSI SCSI revision: 02

It has worked perfectly for several years, but from kernel 2.4.10 st.c
started to report a 24-bit block limit: st0: Block limits 1 - 16777215 bytes. I am now on stock 2.4.18 with static char *verstr = "20020205".

The above drive is a NS-20 drive with default block size of 512 and 20G blocks: so a range of 25 bits for the block number is needed.

So when I attempt to write to the tape beyond block 2^24, I get an error like:

# tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
tar: /dev/ntape: Wrote only 0 of 10240 bytes
tar: Error is not recoverable: exiting now
At block 17832312.

With:
kernel: st0: Block limits 1 - 16777215 bytes.
in the kernel log.

Now I *think* that I used to write to the end of tapes without hitting this
2^24 block limit. I guess that I can work around by setting a larger block size. But is there a fix? Or is the 24-bit limit built into the scsi tape protocol? Since I used to fill 2 tapes to complete a backup, I must have written beyond block (2^24 -1).

Sorry if this is an FAQ. I did google and found nothing directly relevant.
When did st.c change and why? I did turn on DEBUG, and I append a few entries, although I am not sure that it helps much:

Jul 30 16:49:28 conquest3 kernel: st: Version 20020205, bufsize 32768, wrt
30720, max init. bufs 4, s/g segs 16
Jul 30 16:49:28 conquest3 kernel: Attached scsi tape st0 at scsi0, channel
0, id 0, lun 0
Jul 30 16:49:28 conquest3 kernel: st: Allocated tape buffer 0 (32768 bytes,
1 segments, dma: 1, a: c01e8000).
Jul 30 16:49:28 conquest3 kernel: st: segment sizes: first 32768, last
32768 bytes.
drwx------    2 ael      ael          4096 Nov  3  2001 Mail
Jul 30 16:49:28 conquest3 kernel: st0: Block limits 1 - 16777215 bytes.
Jul 30 16:49:28 conquest3 kernel: st0: Mode sense. Length 11, medium 85,
WBS 10, BLL 8
Jul 30 16:49:28 conquest3 kernel: st0: Density 0, tape length: 0, drv buffer: 1
Jul 30 16:49:28 conquest3 kernel: st0: Block size: 512, buffer size: 32768

(64 blocks).

After switching to 1K blocks and fsf'ing past existing data:-
--------------------------------------------------------------

Jul 30 17:28:50 conquest3 kernel: st0: Setting block size to 1024 bytes.
Jul 30 17:28:50 conquest3 kernel: st0: Rewinding tape.
Jul 30 17:28:52 conquest3 kernel: st0: Block limits 1 - 16777215 bytes.
Jul 30 17:28:52 conquest3 kernel: st0: Mode sense. Length 11, medium 85,
WBS 10, BLL 8
Jul 30 17:28:52 conquest3 kernel: st0: Density 0, tape length: 0, drv buffer: 1
Jul 30 17:28:52 conquest3 kernel: st0: Block size: 1024, buffer size: 32768
(32 blocks).
Jul 30 17:28:52 conquest3 kernel: st0: Rewinding tape.
Jul 30 17:29:36 conquest3 kernel: st0: Block limits 1 - 16777215 bytes.
Jul 30 17:29:36 conquest3 kernel: st0: Mode sense. Length 11, medium 85,
WBS 10, BLL 8
Jul 30 17:29:36 conquest3 kernel: st0: Density 0, tape length: 0, drv buffer: 1
Jul 30 17:29:36 conquest3 kernel: st0: Block size: 1024, buffer size: 32768
(32 blocks).
Jul 30 17:29:36 conquest3 kernel: st0: Spacing tape forward over 12 filemarks.
Jul 30 17:32:28 conquest3 kernel: st0: Block limits 1 - 16777215 bytes.
Jul 30 17:32:28 conquest3 kernel: st0: Mode sense. Length 11, medium 85,
WBS 10, BLL 8
Jul 30 17:32:28 conquest3 kernel: st0: Density 0, tape length: 0, drv buffer: 1
Jul 30 17:32:28 conquest3 kernel: st0: Block size: 1024, buffer size: 32768
(32 blocks).
Jul 30 17:32:28 conquest3 kernel: st0: Got tape pos. blk 17832312 part 0.

Thanks in advance for any help

ael

#1 lore raw
Quoted 6 lines · ael

I have a scsi tape drive:-

Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: AIWA Model: TD-20001 Rev: 0159
Type: Sequential-Access ANSI SCSI revision: 02

Quoted 7 lines

It has worked perfectly for several years, but from kernel 2.4.10 st.c started to report a 24-bit block limit: st0: Block limits 1 - 16777215 bytes. I am now on stock 2.4.18 with static char *verstr = "20020205".

The above drive is a NS-20 drive with default block size of 512 and 20G
blocks: so a range of 25 bits for the block number is needed.

The block limits tell the block size limits the drive reports, not limits to the amount of data. The driver writes data until the drive tells that there is no space. I.e., the driver does not limit the amount of data written to the tape. No changes affecting this behaviour has been made for a very long time.

Quoted 8 lines

So when I attempt to write to the tape beyond block 2^24, I get an error like:

# tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
tar: /dev/ntape: Wrote only 0 of 10240 bytes
tar: Error is not recoverable: exiting now
At block 17832312.

Looks like you are at end of tape. I think the 20 GB capacity is compressed and not real. This means that the actual amount of data you can write on the tape depends on the compressibility of data. 17.8 GB is actually not too bad in this case.

Kai

author
#2 lore raw
Quoted 33 lines · Kai Makisara
Quoted 6 lines · ael

I have a scsi tape drive:-

Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: AIWA Model: TD-20001 Rev: 0159
Type: Sequential-Access ANSI SCSI revision: 02

Quoted 7 lines

It has worked perfectly for several years, but from kernel 2.4.10 st.c started to report a 24-bit block limit: st0: Block limits 1 - 16777215 bytes. I am now on stock 2.4.18 with static char *verstr = "20020205".

The above drive is a NS-20 drive with default block size of 512 and 20G
blocks: so a range of 25 bits for the block number is needed.

The block limits tell the block size limits the drive reports, not limits to the amount of data. The driver writes data until the drive tells that there is no space. I.e., the driver does not limit the amount of data written to the tape. No changes affecting this behaviour has been made for a very long time.

Quoted 8 lines

So when I attempt to write to the tape beyond block 2^24, I get an error like:

# tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
tar: /dev/ntape: Wrote only 0 of 10240 bytes
tar: Error is not recoverable: exiting now
At block 17832312.
Quoted 4 lines

Looks like you are at end of tape. I think the 20 GB capacity is compressed and not real. This means that the actual amount of data you can write on the tape depends on the compressibility of data. 17.8 GB is actually not too bad in this case.

Thanks for the reply.

The tape has a real 10GB capacity. (I believe.) The 20BG blocks is correct for the default block size of 512. So compression is not an issue here. And the report above is actually 17.8/2 = 8.9 GB :-(

I did wonder whether some bug in decoding what the drive reports had crept in. I will see if I have kept any logs old enough to see what used to happen... Or maybe I have some substandard tapes. It looks as if I had better do some more checks.

Thanks anyway,

ael

author
#6 lore raw
In reply to Kai Makisara
Quoted 33 lines · Kai Makisara
Quoted 6 lines · ael

I have a scsi tape drive:-

Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: AIWA Model: TD-20001 Rev: 0159
Type: Sequential-Access ANSI SCSI revision: 02

Quoted 7 lines

It has worked perfectly for several years, but from kernel 2.4.10 st.c started to report a 24-bit block limit: st0: Block limits 1 - 16777215 bytes. I am now on stock 2.4.18 with static char *verstr = "20020205".

The above drive is a NS-20 drive with default block size of 512 and 20G
blocks: so a range of 25 bits for the block number is needed.

The block limits tell the block size limits the drive reports, not limits to the amount of data. The driver writes data until the drive tells that there is no space. I.e., the driver does not limit the amount of data written to the tape. No changes affecting this behaviour has been made for a very long time.

Quoted 8 lines

So when I attempt to write to the tape beyond block 2^24, I get an error like:

# tar -cf /dev/ntape blah_home.tar.bz2; mt  tell
tar: /dev/ntape: Wrote only 0 of 10240 bytes
tar: Error is not recoverable: exiting now
At block 17832312.
Quoted 6 lines

Looks like you are at end of tape. I think the 20 GB capacity is compressed and not real. This means that the actual amount of data you can write on the tape depends on the compressibility of data. 17.8 GB is actually not too bad in this case.

Kai

Following up my previous reply where I noted that I had only written 8.8Gb to a tape with uncompressed capacity of 10Gb, it then seems that the tape firmware is reporting a maximum drive capacity of ~ 16GB.

And perhaps a real tape limit of 17832312 /2 = 8.9GB which is supposedly on a 10GB tape : Imation NS-20, specified as 10GB uncompressed. Should I ask Imation for replacements :-) ?

ael

#3 lore raw
In reply to ael
Quoted 7 lines
Quoted 1 line
Quoted 1 line
# tar -cf /dev/ntape blah_home.tar.bz2; mt  tell

The tape has a real 10GB capacity. (I believe.) The 20BG blocks is correct
for the default block size of 512. So compression is not an issue here. And
the report above is actually 17.8/2 = 8.9 GB :-(

Some compression algorithms will actually expand data by a significant amount. For example, many implementations of ALDC compression will give a compression ratio of 0.8 on random data. In your example above you are writing a .bz2 file (which is already compressed), so it would probably expand if compressed again. Try turning drive internal compression off:

mt -f /dev/ntape compression off

This might help squeeze a little more data on your tape.

Anthony Battersby
Cybernetics

#4 lore raw

Tony Battersby wrote:
[Charset iso-8859-1 unsupported, filtering to ASCII...]

Quoted 8 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
# tar -cf /dev/ntape blah_home.tar.bz2; mt  tell

The tape has a real 10GB capacity. (I believe.) The 20BG blocks is correct
for the default block size of 512. So compression is not an issue here. And
the report above is actually 17.8/2 = 8.9 GB :-(

Quoted 5 lines

Some compression algorithms will actually expand data by a significant amount. For example, many implementations of ALDC compression will give a compression ratio of 0.8 on random data. In your example above you are writing a .bz2 file (which is already compressed), so it would probably expand if compressed again. Try turning drive internal compression off:

It's easy to prevent this from costing too much: Simply prepend a block of data with a "1" compressed or "0" uncompressed bit.

Max expansion is then 1 bit per block. (At a cost of one extra bit per block).

This technique was maybe unacceptable in the old days when you couldn't afford to buffe a WHOLE block in RAM while you were compressing to see if it would expand......

Nowadays you can easily buffer the data, and still get acceptable speed, at an acceptable cost in the form of a little higher RAM usage.

Roger.

Signature · 4 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* There are old pilots, and there are bold pilots. 
* There are also old, bald pilots. 
author
#5 lore raw
In reply to Tony Battersby
Quoted 16 lines · Tony Battersby
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
# tar -cf /dev/ntape blah_home.tar.bz2; mt  tell

The tape has a real 10GB capacity. (I believe.) The 20BG blocks is correct
for the default block size of 512. So compression is not an issue here. And
the report above is actually 17.8/2 = 8.9 GB :-(

Some compression algorithms will actually expand data by a significant amount. For example, many implementations of ALDC compression will give a compression ratio of 0.8 on random data. In your example above you are writing a .bz2 file (which is already compressed), so it would probably expand if compressed again. Try turning drive internal compression off:

Quoted 3 lines

mt -f /dev/ntape compression off

This might help squeeze a little more data on your tape.

Thanks for the suggestion. Various such thoughts had crossed what passes for my mind, including whether 10GB was before formatting. But I did assume that the tape drive/driver was reporting actual bytes on the tape regardless of compression.

I was indeed recording largely bz2 files, and I did realize that the default drive streaming compression would almost certainly expand again. I sometimes do remember to turn the drive compression off in such circumstances. But as I say, I assume that the actual data on the tape after compression was being reported. That has always seemed to be the case in the past, although I am not sure whether I checked very thoroughly.

I guess that I should browse the Imation web site and see exactly they specify for the tape itself. The packaging is clearly marked 10GB uncompressed. The drive has a read-while-write-and-rerecord-on-error circuit, but the tapes have not had much use, and I don't think that the data is being expanded by repeated blocks. I could be wrong, of course.
And in any case, I would expect the drive to report the total blaoc count including repeats.

ael