Prev: Simple regex question
Next: How to run multiple commands in -exec of find command
From: Name withheld by request on 23 Jan 2010 10:08
In article <5f7927-ijf.ln1(a)marcumbill.bellsouth.net>,
Bill Marcum <marcumbill(a)bellsouth.net> wrote:
>On 2010-01-16, Name withheld by request <anonb6e9(a)nyx3.nyx.net> wrote:
>> I wanted to tar files from a windows box back to a tape drive on the unix box,
>> but the cygwin rmt command was not being found so I thought I would use tar
>> gzip rsh and dd.
>>
--snip
>> I'm sharing it in hopes that it
>> will be useful, and that some one might both explain why it works and how it
>> might be simplified. Here is an example that works:
>>
>> # 09:03:10 Fri 100101 0j tty0 2404 ~
>> # neon root $ (tar -cf - /etc|gzip; dd if=/dev/zero count=1)|dd obs=1024k |
>> dd bs=1024k conv=sync|rsh 10.164.123.227 dd of=/dev/st0
>
>You probably don't need so many dd commands, just one "dd bs=xxxx
>conv=sync" with a block size greater than 512.

OK, Here's I tried your suggestion this morning ( in this case
both hosts are linux ):

# (tar -cf - /etc|gzip; dd if=/dev/zero count=1)| dd bs=1024k conv=sync|rsh 10.164.123.227 dd of=/dev/st0
tar: Removing leading `/' from member names
1+0 records in
1+0 records out
0+59 records in
59+0 records out
96594+30083 records in
120832+0 records out
# rsh 10.164.123.227 dd if=/dev/st0 |tar -ztf - |tail -3
tar: Skipping to next header
tar: Archive contains obsolescent base-64 headers

gzip: stdin: invalid compressed data--crc error

gzip: stdin: invalid compressed data--length error
tar: Child died with signal 13
tar: Error exit delayed from previous errors
etc/sysconfig/network-scripts/ifup-post
etc/sysconfig/network-scripts/ifup-ppp
etc/sysconfig/network-scripts/ifup-routes
#

So that failed.

Retrying the original approach on the same two linux hosts:

# (tar -cf - /etc|gzip; dd if=/dev/zero count=1)|dd obs=1024k| dd bs=1024k conv=sync|rsh 10.164.123.227 dd of=/dev/st0
tar: Removing leading `/' from member names
1+0 records in
1+0 records out
3308+1 records in
1+1 records out
1+1 records in
2+0 records out
3692+504 records in
4096+0 records out
# rsh 10.164.123.227 dd if=/dev/st0 |tar -ztf - |tail -3
4096+0 records in
4096+0 records out
etc/local.site,/RCS/
etc/local.site,/svars_all_shells
etc/Muttrc
#

So the original posted approach works - again I don't really understand
why..


>> And another question.. what does the notation:
>>
>> 2628+846 records in
>> 3474+0 records out
>>
>> indicate? Specifically what is the +846?
>>
>+846 bytes because the amount of input data was not a multiple of the block
>size.

I'm not convinced the 846 is in units of bytes. I need to think
about the records in/records out notation more.

--
thanks/regards


From: Name withheld by request on 23 Jan 2010 16:45
In article <1264259283.78143(a)irys.nyx.net>,
Name withheld by request <anonb6e9(a)nyx.net> wrote:
>In article <5f7927-ijf.ln1(a)marcumbill.bellsouth.net>,
>Bill Marcum <marcumbill(a)bellsouth.net> wrote:
>>On 2010-01-16, Name withheld by request <anonb6e9(a)nyx3.nyx.net> wrote:
>>> I wanted to tar files from a windows box back to a tape drive on the unix box,
>>> but the cygwin rmt command was not being found so I thought I would use tar
>>> gzip rsh and dd.
>>>
>--snip
>>> I'm sharing it in hopes that it
>>> will be useful, and that some one might both explain why it works and how it
>>> might be simplified. Here is an example that works:
>>>
>>> # 09:03:10 Fri 100101 0j tty0 2404 ~
>>> # neon root $ (tar -cf - /etc|gzip; dd if=/dev/zero count=1)|dd obs=1024k |
>>> dd bs=1024k conv=sync|rsh 10.164.123.227 dd of=/dev/st0
>>
>>You probably don't need so many dd commands, just one "dd bs=xxxx
>>conv=sync" with a block size greater than 512.
>
>OK, Here's I tried your suggestion this morning ( in this case
>both hosts are linux ):
>
> # (tar -cf - /etc|gzip; dd if=/dev/zero count=1)| dd bs=1024k conv=sync|rsh 10.164.123.227
>dd of=/dev/st0
> tar: Removing leading `/' from member names
> 1+0 records in
> 1+0 records out
> 0+59 records in
> 59+0 records out
> 96594+30083 records in
> 120832+0 records out
> # rsh 10.164.123.227 dd if=/dev/st0 |tar -ztf - |tail -3
> tar: Skipping to next header
> tar: Archive contains obsolescent base-64 headers
>
> gzip: stdin: invalid compressed data--crc error
>
> gzip: stdin: invalid compressed data--length error
> tar: Child died with signal 13
> tar: Error exit delayed from previous errors
> etc/sysconfig/network-scripts/ifup-post
> etc/sysconfig/network-scripts/ifup-ppp
> etc/sysconfig/network-scripts/ifup-routes
> #
>
>So that failed.
>
>Retrying the original approach on the same two linux hosts:
>
> # (tar -cf - /etc|gzip; dd if=/dev/zero count=1)|dd obs=1024k| dd bs=1024k conv=sync|rsh
>10.164.123.227 dd of=/dev/st0
> tar: Removing leading `/' from member names
> 1+0 records in
> 1+0 records out
> 3308+1 records in
> 1+1 records out
> 1+1 records in
> 2+0 records out
> 3692+504 records in
> 4096+0 records out
> # rsh 10.164.123.227 dd if=/dev/st0 |tar -ztf - |tail -3
> 4096+0 records in
> 4096+0 records out
> etc/local.site,/RCS/
> etc/local.site,/svars_all_shells
> etc/Muttrc
> #
>
>So the original posted approach works - again I don't really understand
>why..
>
>
>>> And another question.. what does the notation:
>>>
>>> 2628+846 records in
>>> 3474+0 records out
>>>
>>> indicate? Specifically what is the +846?
>>>
>>+846 bytes because the amount of input data was not a multiple of the block
>>size.
>
>I'm not convinced the 846 is in units of bytes. I need to think
>about the records in/records out notation more.
>
>--
>thanks/regards
>
>


From: Stephane CHAZELAS on 23 Jan 2010 17:33
2010-01-16, 03:11(+00), Name withheld by request:
[...]
> # tigris root # (tar -cf - /etc|gzip; dd if=/dev/zero count=1) |
> rsh 10.164.123.227 dd conv=sync of=/dev/st0
> tar: Removing leading `/' from member names
> 1+0 records in
> 1+0 records out
> 2628+846 records in
[...]
> And another question.. what does the notation:
>
> 2628+846 records in
> 3474+0 records out
>
> indicate? Specifically what is the +846?

It tells you 846 were partly read. The second dd's stdin will be
a socket, doing a read(0, buf, 512) will not always return 512,
it depends on a lot of things like size of packets in transit...
Using conv=sync was a mistake. It pads incomplete reads with
zeros, which just makes dd output garbage.

Here, you want:

tar cf - /etc | gzip -3 | rsh ... dd of=/dev/st0

Except that if the length of the output of gzip is not a
multiple of 512, the last write to /dev/st0 will not be 512
bytes large. You can use your dd if=/dev/zero count=1 trick for
that.

--
St�phane
From: Name withheld by request on 23 Jan 2010 19:47
--snip
>>> And another question.. what does the notation:
>>>
>>> 2628+846 records in
>>> 3474+0 records out
>>>
>>> indicate? Specifically what is the +846?
--snip
>I'm not convinced the 846 is in units of bytes. I need to think
>about the records in/records out notation more.

Take a look at these test cases and notice the pattern:

$ i=10 ;echo count: $i;yes|dd ibs=1 obs=5 count=$i 2>/dev/null|
dd bs=4 conv=sync |command od -Ad -c
count: 10
2+2 records in
4+0 records out
16 bytes (16 B) copied, 0.01579 s, 1.0 kB/s
0000000 y \n y \n y \0 \0 \0 \n y \n y \n \0 \0 \0
0000016
$ i=15 ;echo count: $i;yes|dd ibs=1 obs=5 count=$i 2>/dev/null|
dd bs=4 conv=sync |command od -Ad -c
count: 15
3+3 records in
6+0 records out
24 bytes (24 B) copied, 0.0161833 s, 1.5 kB/s
0000000 y \n y \n y \0 \0 \0 \n y \n y \n \0 \0 \0
0000016 y \n y \n y \0 \0 \0
0000024
$ i=17 ;echo count: $i;yes|dd ibs=1 obs=5 count=$i 2>/dev/null|
dd bs=4 conv=sync |command od -Ad -c
count: 17
4+1 records in
5+0 records out
20 bytes (20 B) copied, 0.02879 s, 0.7 kB/s
0000000 y \n y \n y \n y \n y \n y \n y \n y \n
0000016 y \0 \0 \0
0000020
$ i=20 ;echo count: $i;yes|dd ibs=1 obs=5 count=$i 2>/dev/null|
dd bs=4 conv=sync |command od -Ad -c
count: 20
4+4 records in
8+0 records out
32 bytes (32 B) copied, 0.0268962 s, 1.2 kB/s
0000000 y \n y \n y \0 \0 \0 \n y \n y \n \0 \0 \0
*
0000032
$ i=40 ;echo count: $i;yes|dd ibs=1 obs=5 count=$i 2>/dev/null|
dd bs=4 conv=sync |command od -Ad -c
count: 40
8+8 records in
16+0 records out
64 bytes (64 B) copied, 0.0321943 s, 2.0 kB/s
0000000 y \n y \n y \0 \0 \0 \n y \n y \n \0 \0 \0
*
0000064

The above were done on yet another (much newer centos 5.3 box).
The number after the + appears to be the count of input
*records* that dd could not entirely fill when it did a read.
The case with 17 chars has only one of these.

dd says nothing directly about the number of remainder bytes AFAIK.

--
Here's an interesting run, with my damn prompt showing the time:

# 15:36:56 Sat 100123 1j 1 7701 /etc/profile.d
# hera myname $ i=10 ;echo count: $i;yes|dd ibs=1 obs=5 count=$i 2>/dev/null|
dd bs=4 conv=sync |command od -Ad -c
count: 10
2+2 records in
4+0 records out
16 bytes (16 B) copied, 0.016554 s, 1.0 kB/s
0000000 y \n y \n y \0 \0 \0 \n y \n y \n \0 \0 \0
0000016
# 15:36:57 Sat 100123 1j 1 7701 /etc/profile.d
# hera myname $ i=10 ;echo count: $i;yes|dd ibs=1 obs=5 count=$i 2>/dev/null|
dd bs=4 conv=sync |command od -Ad -c
count: 10
2+1 records in
3+0 records out
12 bytes (12 B) copied, 0.000364284 s, 32.9 kB/s
0000000 y \n y \n y \n y \n y \n \0 \0
0000012

Same command, two differing results - due to asynchronous nature
of pipeline? Anyone care to explain?
From: Name withheld by request on 24 Jan 2010 08:48
In article <slrnhlmu9t.rd1.stephane.chazelas(a)spam.is.invalid>,
Stephane CHAZELAS <stephane_chazelas(a)yahoo.fr> wrote:
>2010-01-16, 03:11(+00), Name withheld by request:
>[...]
>> # tigris root # (tar -cf - /etc|gzip; dd if=/dev/zero count=1) |
>> rsh 10.164.123.227 dd conv=sync of=/dev/st0
>> tar: Removing leading `/' from member names
>> 1+0 records in
>> 1+0 records out
>> 2628+846 records in
>[...]
>> And another question.. what does the notation:
>>
>> 2628+846 records in
>> 3474+0 records out
>>
>> indicate? Specifically what is the +846?
>
>It tells you 846 were partly read. The second dd's stdin will be
>a socket, doing a read(0, buf, 512) will not always return 512,
>it depends on a lot of things like size of packets in transit...
>Using conv=sync was a mistake. It pads incomplete reads with
>zeros, which just makes dd output garbage.
>
>Here, you want:
>
>tar cf - /etc | gzip -3 | rsh ... dd of=/dev/st0
>
>Except that if the length of the output of gzip is not a
>multiple of 512, the last write to /dev/st0 will not be 512
>bytes large. You can use your dd if=/dev/zero count=1 trick for
>that.
>
>--
>St�phane

Thanks much! The previous post of mine (24 Jan 2010 00:47:59 GMT)
seems to confirm what you say above.

>Here, you want:
>
>tar cf - /etc | gzip -3 | rsh ... dd of=/dev/st0
>
>Except that if the length of the output of gzip is not a
>multiple of 512, the last write to /dev/st0 will not be 512
>bytes large. You can use your dd if=/dev/zero count=1 trick for
>that.
>
>--
>St�phane

Thanks, I think I got it..

I was assuming that one of my first approaches was *failing* (because
of 'dd: /dev/st0: Input/output error'); now I think that error is just
due to the final left over nulls, that don't need to go to tape.

I see no tar related errors here:

# tar -cf - /etc|(gzip; dd if=/dev/zero count=1)|rsh 10.164.123.227 dd of=/dev/st0
tar: Removing leading ' from member names
1+0 records in
1+0 records out
dd: /dev/st0: Input/output error
2669+788 records in
3308+0 records out
# rsh 10.164.123.227 dd if=/dev/st0 |tar -ztf - |tail -3
3308+0 records in
3308+0 records out
etc/local.site,/RCS/
etc/local.site,/svars_all_shells
etc/Muttrc
#

--
thanks to all that helped

 | 
Pages: 1
Prev: Simple regex question
Next: How to run multiple commands in -exec of find command