Redirect stdout and stderr preserving the original order
in [Shell]
|
Prev: list out thousand of files
Next: unzip | touch | re-zip
From: Radoulov, Dimitre on 29 Dec 2009 12:11 Hi all, There was a question on a web forum on how to redirect stdin and stderr in the following manner: - two separate files for both streams - a third file containing both streams in their original order We haven't come up with a solution and I'm curious to hear your opinion. For instance, let's say we have the following shell script: % cat s #!/bin/sh for i in 1 2; do for j in 1 2; do printf '%s\n' "out $i" done for k in 1 2; do printf '%s\n' "err $i" >&2 done done The script produces the following output on the terminal (which of course includes both stdout and stderr streams): % ./s out 1 out 1 err 1 err 1 out 2 out 2 err 2 err 2 Note that we use out/err n only for clarity and easy debugging. We assume that in the real world both the content and the order of the stdout and stderr streams are unknown. Now the question is, is it possible to end up with the following files after running the script: out.log out 1 out 1 out 2 out 2 err.log err 1 err 1 err 2 err 2 both.log out 1 out 1 err 1 err 1 out 2 out 2 err 2 err 2 Note the order of the entries in both.log. We also assume that it's not possible to modify the script/program that generates the output in any way. Best regards Dimitre
From: Seebs on 29 Dec 2009 12:57 On 2009-12-29, Radoulov, Dimitre <cichomitiko(a)gmail.com> wrote: > Hi all, > There was a question on a web forum on how to redirect stdin and stderr > in the following manner: > > - two separate files for both streams > - a third file containing both streams in their original order > > We haven't come up with a solution and I'm curious to hear your opinion. Hmm. I suspect you can, but would have to write a pretty specialized program. Here's my idea. You write a program which creates a single unix domain socket, and allows multiple connections to it. You then open it twice -- not using 2>&1, but >socket 2>socket and then that program handles the interleaving, duplicating, and so on. You might be able to fake something up by simply prepending microsecond timestamps to both streams, logging them, then sorting those streams together to get the "original order". -s -- Copyright 2009, all wrongs reversed. Peter Seebach / usenet-nospam(a)seebs.net http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
From: Radoulov, Dimitre on 29 Dec 2009 14:33 On 29/12/2009 18.57, Seebs wrote: > On 2009-12-29, Radoulov, Dimitre<cichomitiko(a)gmail.com> wrote: >> Hi all, >> There was a question on a web forum on how to redirect stdin and stderr >> in the following manner: >> >> - two separate files for both streams >> - a third file containing both streams in their original order >> >> We haven't come up with a solution and I'm curious to hear your opinion. [...] > I suspect you can, but would have to write a pretty specialized program. > Here's my idea. You write a program which creates a single unix domain > socket, and allows multiple connections to it. You then open it twice -- > not using 2>&1, but > >socket 2>socket > and then that program handles the interleaving, duplicating, and so on. > > You might be able to fake something up by simply prepending microsecond > timestamps to both streams, logging them, then sorting those streams together > to get the "original order". Interesting, I'll try it. Thank you! Dimitre
From: mop2 on 29 Dec 2009 15:12 On Tue, 29 Dec 2009 15:11:01 -0200, Radoulov, Dimitre <cichomitiko(a)gmail.com> wrote: > Hi all, > There was a question on a web forum on how to redirect stdin and stderr > in the following manner: > > - two separate files for both streams > - a third file containing both streams in their original order > > We haven't come up with a solution and I'm curious to hear your opinion. > > For instance, let's say we have the following shell script: > > > % cat s > #!/bin/sh > > for i in 1 2; do > for j in 1 2; do > printf '%s\n' "out $i" > done > for k in 1 2; do > printf '%s\n' "err $i" >&2 > done > done > > > The script produces the following output on the terminal (which of > course includes both stdout and stderr streams): > > % ./s > out 1 > out 1 > err 1 > err 1 > out 2 > out 2 > err 2 > err 2 > > Note that we use out/err n only for clarity and easy debugging. We > assume that in the real world both the content and the order of the > stdout and stderr streams are unknown. > > Now the question is, is it possible to end up with the following files > after running the script: > > out.log > > out 1 > out 1 > out 2 > out 2 > > err.log > > err 1 > err 1 > err 2 > err 2 > > both.log > > out 1 > out 1 > err 1 > err 1 > out 2 > out 2 > err 2 > err 2 > > > Note the order of the entries in both.log. > > We also assume that it's not possible to modify the script/program that > generates the output in any way. > > > > Best regards > Dimitre > > > > Just an idea: $ cat s #!/bin/sh for i in 1 2; do for j in 1 2; do printf '%s\n' "out $i" done for k in 1 2; do printf '%s\n' "err $i" >&2 done done $ cat test printf '' >b printf '' >o printf '' >e tail -n0 -f o >>b & To=$! tail -n0 -f e >>b & Te=$! ../s >o 2>e read -t2; kill $To $Te $ . ./test [1]- Terminated tail -n0 -f o >> b [2]+ Terminated tail -n0 -f e >> b $ cat o out 1 out 1 out 2 out 2 $ cat e err 1 err 1 err 2 err 2 $ cat b out 1 out 1 err 1 err 1 out 2 out 2 err 2 err 2 $
From: Seebs on 29 Dec 2009 15:19
On 2009-12-29, mop2 <invalid(a)mail.address> wrote: > tail -n0 -f o >>b & To=$! Nice! My worry would be that I don't have very high confidence that it'll respond immediately to updates in the files -- meaning that you could end up with things getting put in unexpected orders. There is also the question of whether output going to a file will result in buffering behavior, especially for stdout. -s -- Copyright 2009, all wrongs reversed. Peter Seebach / usenet-nospam(a)seebs.net http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated! |