Redirect stdout and stderr preserving the original order [Shell]

Prev: list out thousand of files
Next: unzip | touch | re-zip

From: mop2 on 30 Dec 2009 15:32

On Wed, 30 Dec 2009 08:38:06 -0200, Radoulov, Dimitre <cichomitiko(a)gmail.com> wrote:

> On 30/12/2009 2.44, mop2 wrote:

>> The more important here, I think, is the program tail.
>> If the "-s" is available you can try it to see if behavior changes for
>> small values:
>> -s, --sleep-interval=N
>> with -f, sleep for approximately N seconds (default 1.0) between iterations
>>
>> I never tried this option, but in the past I had problems with
>> buffering, as said by others.

>
> I tried with values from 0.1 to 0.5, but I get the same result.
> Later I will try the test again on a different system.

When I saw this option, my first idea was the value zero or times very very
small, as .00001 sec or lower.

>> while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >o <fo &
>> while read;do printf "$REPLY\n";printf "$REPLY\n">>b;done >e <fe &

In the above proposal, 'printf "$REPLY\n">>b' before
'printf "$REPLY\n"', seems more logical.

From: bsh on 30 Dec 2009 21:21

"Dimitre Radoulov" <cichomitiko(a)gmail.com> wrote:
> ... how to redirect stdin and stderr into two separate files
> for both streams, with a third file containing both streams
> in their original order

To save stdout, stderr, and the original stream into three
separate files, try this:

# untested for your application
( ( ./myprogram 2>&1 1>&3 | tee ~/stderr.txt
) 3>&1 1>&2 |
tee ~/stdout.txt
) >~/stdmix.txt 2>&1

Icarus Sparry wrote:
> ... For typical programs, this is very hard. The reason is that
> almost all programs use the "stdio" (standard I/O) library.

This is why ksh93 utilizes the Kiem-Phong Vo's stdio emulation
library, sfio ("Safe and Fast I/O"). It is a component in AT&T's
AST library, which happens to be statically linked into ksh93,
which is why if the above doesn't work, to try to use this shell
instead. Pre-compiled version exist specifically for RHEL and many
other Unices.

John Koy <John....(a)example.com> wrote:
> All these tricks are useless. One cannot achieve synchronousness
> across process boundaries without explicit IPC synchronization,
> like using semaphores.

Yes and no. Since IRIX 3+, all the stdio functions and macroes
are written to preserve line-by-line serialization (in the old
sense of the term). Presumably other systems are written this
way, but I have no information. Like I indicated above, there
is a reason that sfio was created!

If indeed semaphores are your only resort, after ten years of
research and experimentation, I have come to the conclusion,
that at least until ksh has the builtin ability, the only decent
semaphore library and batch job frontend, are:

"semaphore.ksh"
http://www.unixreview.com/documents/s=9303/sam0408f/0408f.htm
http://www.samag.com/code/
ftp://ftp.mfi.com/pub/sysadmin/2004/aug2004.zip
TRACT: Schaefer, Ed and John Spurgeon. "Queuing Jobs with qjob".
2005-07. Sys Admin. 14(8). <Url:http://www.theillien.com/Sys_Admin_v12/
html/v14/i08/a8.htm>.

mop2 wrote:
> > tail -s
> When I saw this option, my first idea was the value zero
> or times very very small, as .00001 sec or lower.

Time granularity and accuracy is (very) system dependent --
if it even supports this level at all. A better idea, if
only more portable, is to try Daniel Bernstein's "tai64n":

"tai64n.c": put a precise timestamp on each line
http://cr.yp.to/daemontools/tai64n.html

In general, I _would_ have thought to have suggested looking
through his pipe-tools and/or conn-tools library, but that
now I see that its component software tools are either
deprecated or inapplicable to your use: try the above ideas
first, but if all else fails, look at:

http://www.skarnet.org/software/pipe-tools/
http://www.skarnet.org/software/conn-tools/

=Brian

From: Stephane CHAZELAS on 31 Dec 2009 05:50

2009-12-30, 18:21(-08), bsh:
[...]
> Icarus Sparry wrote:
>> ... For typical programs, this is very hard. The reason is that
>> almost all programs use the "stdio" (standard I/O) library.
>
> This is why ksh93 utilizes the Kiem-Phong Vo's stdio emulation
> library, sfio ("Safe and Fast I/O"). It is a component in AT&T's
> AST library, which happens to be statically linked into ksh93,
> which is why if the above doesn't work, to try to use this shell
> instead. Pre-compiled version exist specifically for RHEL and many
> other Unices.
[...]

In which way does that solve the problem. Are you saying that
commands run by those shells will use a different stdio by some
sort of LD_PRELOAD trick?

> John Koy <John....(a)example.com> wrote:
>> All these tricks are useless. One cannot achieve synchronousness
>> across process boundaries without explicit IPC synchronization,
>> like using semaphores.
>
> Yes and no. Since IRIX 3+, all the stdio functions and macroes
> are written to preserve line-by-line serialization (in the old
> sense of the term). Presumably other systems are written this
> way, but I have no information. Like I indicated above, there
> is a reason that sfio was created!

That has nothing to do with stdio here. With pipes, you'd need
some sort of syncronisation.

The problem is that you need to have a process read from 2
pipes, and you can't guarantee that this process will read the
data in the order they were written. You could only guarantee
that if writing to the pipe was blocking until the reading
process has read everything (and thanksfully, pipes don't work
like that) and if there aren't processes writing simultaneously
to stdout and stderr.

For instance, in

echo x
echo y >&2
echo z

which results in

write(1, "x\n", 2);
write(2, "y\n", 2);
write(1, "z\n", 2);

(echo doesn't use stdio, or fflushes afterwards if it does
anyway).

What is likely to happen is that that command will perform the 3
writes at once before any other process is scheduled.

Then your process that reads the other end of those 2 pipes will
read "x\nz\n" from the first one and "y\n" from the second in no
predictable order, and if it's 2 separate processes reading the
2 pipes, the result is even less likely to be predictable.

> If indeed semaphores are your only resort, after ten years of
> research and experimentation, I have come to the conclusion,
> that at least until ksh has the builtin ability, the only decent
> semaphore library and batch job frontend, are:
>
> "semaphore.ksh"

Here you need semaphores in between write(2) and read(2) system
calls, so I don't expect a shell solution to be of any use.

a LD_PRELOAD solution with a modified stdio that would
do the logging could work assuming all the commands called by
the shell use a dynamically linked stdio..

--
St�phane

From: Radoulov, Dimitre on 31 Dec 2009 07:23

On 31/12/2009 11.50, Stephane CHAZELAS wrote:
> 2009-12-30, 18:21(-08), bsh:
> [...]
>> Icarus Sparry wrote:
>>> ... For typical programs, this is very hard. The reason is that
>>> almost all programs use the "stdio" (standard I/O) library.
>>
>> This is why ksh93 utilizes the Kiem-Phong Vo's stdio emulation
>> library, sfio ("Safe and Fast I/O"). It is a component in AT&T's
>> AST library, which happens to be statically linked into ksh93,
>> which is why if the above doesn't work, to try to use this shell
>> instead. Pre-compiled version exist specifically for RHEL and many
>> other Unices.
> [...]
>
> In which way does that solve the problem. Are you saying that
> commands run by those shells will use a different stdio by some
> sort of LD_PRELOAD trick?
>
>
>> John Koy<John....(a)example.com> wrote:
>>> All these tricks are useless. One cannot achieve synchronousness
>>> across process boundaries without explicit IPC synchronization,
>>> like using semaphores.
>>
>> Yes and no. Since IRIX 3+, all the stdio functions and macroes
>> are written to preserve line-by-line serialization (in the old
>> sense of the term). Presumably other systems are written this
>> way, but I have no information. Like I indicated above, there
>> is a reason that sfio was created!
>
> That has nothing to do with stdio here. With pipes, you'd need
> some sort of syncronisation.
>
> The problem is that you need to have a process read from 2
> pipes, and you can't guarantee that this process will read the
> data in the order they were written. You could only guarantee
> that if writing to the pipe was blocking until the reading
> process has read everything (and thanksfully, pipes don't work
> like that) and if there aren't processes writing simultaneously
> to stdout and stderr.
>
> For instance, in
>
> echo x
> echo y>&2
> echo z
>
> which results in
>
> write(1, "x\n", 2);
> write(2, "y\n", 2);
> write(1, "z\n", 2);
>
> (echo doesn't use stdio, or fflushes afterwards if it does
> anyway).
>
> What is likely to happen is that that command will perform the 3
> writes at once before any other process is scheduled.
>
> Then your process that reads the other end of those 2 pipes will
> read "x\nz\n" from the first one and "y\n" from the second in no
> predictable order, and if it's 2 separate processes reading the
> 2 pipes, the result is even less likely to be predictable.
>
>> If indeed semaphores are your only resort, after ten years of
>> research and experimentation, I have come to the conclusion,
>> that at least until ksh has the builtin ability, the only decent
>> semaphore library and batch job frontend, are:
>>
>> "semaphore.ksh"
>
> Here you need semaphores in between write(2) and read(2) system
> calls, so I don't expect a shell solution to be of any use.
>
> a LD_PRELOAD solution with a modified stdio that would
> do the logging could work assuming all the commands called by
> the shell use a dynamically linked stdio..

Thank you all for your valuable input.

Best regards
Dimitre

From: bsh on 1 Jan 2010 03:44

On Dec 31 2009, 6:00 am, John Koy <John....(a)example.com> wrote:
> bsh wrote:
> > "Dimitre Radoulov"<cichomit...(a)gmail.com> wrote:
> > > ...

> This has nothing to do with serialization[sic] of user-mode data structures.
> A pipe is a kernel data structure. At any time the kernel could be
> handling 100s of pipes between dozens of processes. No user-mode library
> can control the order and time these pipes are served, unless the kernel
> has some real-time extensions.

Like I said: yes and no. In (at least) IRIX, which I can presume
doesn't apply here, stdio functions and macros _are_ embedded in
line-oriented locking, which to my recollection is such a user-level
library that is serialized(*) on a system-level, not a process per
process level. If I remember correctly, this makes it true that all
processes' read(2)s and write(2)s are atomic -- as indeed I understand
these system calls are in BSD-derived Unices, if not (older?) Linux
kernels. Properly implemented I/O libraries (that is, sfio in lieu of
stdio) can (or will?) then (potentially?) be formally atomic.

(*) Java etal has subsumed the notion of serialization to mean data
persistence. I intend the old meaning of locking, as per use of
SysV message queues, synchronization, shared memory, RPC,
or whatever other IPC facility.

> These are all beyond the scope of shell programming.

I completely agree! and this was my thesis: ksh93 with its vastly
improved statically linked I/O -- and maybe it is also necessary that
the underlying Unix implement atomic I/O as well... -- supports my
suggestion to at least give that shell a chance; it was the potential
solution of least effort.

=Brian