Reading stdout & stderr from a pipe with popen3
in [Ruby]
|
From: Matt Mencel on 3 Jul 2008 16:07 I originally posted this to a Ruby forum at railsforum.com, but I just found this list so thought I'd try here since I haven't gotten a response at railsforum. OK...so I'm using popen3 to run some command line stuff and be able to get back stdout/stderr... $cmdin, $cmdout, $cmderr = Open3.popen3("zmprov") zmprov opens a new command prompt to which I can send more commands to and get back output... $cmdin.puts("sm username") $cmdin.puts("sm") count = 0 $cmdout.each |line| count += 1 if line.include?("\n") puts line break if count == 2 # IF I FIND 2 BLANK LINES IN stdout, BREAK THE LOOP end Which works just fine. However, I've run into a snag. $cmdout and $cmderr are pipes that never close until I send the "quit" command...so in order to break out of my loops I've got to jump through some extra hoops to look for specific patterns in the output. In the above example I have to send an extra "sm" command without the username, and in that output I can find the two blank lines...so I know it's done and can drop out of the loop. It's a pain, but I can do that. The real problem becomes what to do when a command returns something in stderr? Because stdout returns nothing...there's nothing to check for to drop out of that loop before I check for stderr....check the example below. $cmdin.puts("sm BADUSERNAME") # PRODUCES AN ERROR $cmdout.each |line| puts line end $cmderr.each |line| puts line end If the command produces an error....I never get out of the stdout loop to grab the error because stdout is returning me nothing...it's just an open pipe with no data in it until I run a command that produces stdout. So I never get where I can read stderr. A coworker suggested I run the original "zmprov" command as "zmprov 2>&1" so that stderr is returned in stdout. Which will work...but I was just wondering if I could get the code to work as designed. If stdout is returning data...read it and do something....if stderr is returning data then do something with that. I'm wanting to keep one from blocking my ability to read the other. After some further research and a little testing, I've found that using IO.readpartial (http://www.noobkit.com/show/ruby/ruby/ruby-core/io/readpartial.html) might be what I'm looking for, but I'm not sure. So I can do something like... $cmdin.readpartial(4096) ...and it will grab all the data in the pipe (up to 4096 bytes I think) and when it runs out of stuff to read it returns control back to the program. What happens though if there is more data waiting in the pipe than I call for with my IO.readpartial(maxlen) argument? Will it still return control or will it keep reading 4096 blocks until it runs out and THEN return control? I think that is how I would want it to handle. There is almost no examples out there about IO.readpartial, so I'm hoping someone here can help point me in the right direction. Thanks, Matt
From: Matt Mencel on 7 Jul 2008 09:28 bump... Just trying to see if anyone knows IO.readpartial better than me. Matt ----- Original Message ----- From: "Matt Mencel" <MR-Mencel(a)wiu.edu> To: "ruby-talk ML" <ruby-talk(a)ruby-lang.org> Sent: Thursday, July 3, 2008 3:07:15 PM GMT -06:00 US/Canada Central Subject: Reading stdout & stderr from a pipe with popen3 I originally posted this to a Ruby forum at railsforum.com, but I just found this list so thought I'd try here since I haven't gotten a response at railsforum. OK...so I'm using popen3 to run some command line stuff and be able to get back stdout/stderr... $cmdin, $cmdout, $cmderr = Open3.popen3("zmprov") zmprov opens a new command prompt to which I can send more commands to and get back output... $cmdin.puts("sm username") $cmdin.puts("sm") count = 0 $cmdout.each |line| count += 1 if line.include?("\n") puts line break if count == 2 # IF I FIND 2 BLANK LINES IN stdout, BREAK THE LOOP end Which works just fine. However, I've run into a snag. $cmdout and $cmderr are pipes that never close until I send the "quit" command...so in order to break out of my loops I've got to jump through some extra hoops to look for specific patterns in the output. In the above example I have to send an extra "sm" command without the username, and in that output I can find the two blank lines...so I know it's done and can drop out of the loop. It's a pain, but I can do that. The real problem becomes what to do when a command returns something in stderr? Because stdout returns nothing...there's nothing to check for to drop out of that loop before I check for stderr....check the example below. $cmdin.puts("sm BADUSERNAME") # PRODUCES AN ERROR $cmdout.each |line| puts line end $cmderr.each |line| puts line end If the command produces an error....I never get out of the stdout loop to grab the error because stdout is returning me nothing...it's just an open pipe with no data in it until I run a command that produces stdout. So I never get where I can read stderr. A coworker suggested I run the original "zmprov" command as "zmprov 2>&1" so that stderr is returned in stdout. Which will work...but I was just wondering if I could get the code to work as designed. If stdout is returning data...read it and do something....if stderr is returning data then do something with that. I'm wanting to keep one from blocking my ability to read the other. After some further research and a little testing, I've found that using IO.readpartial (http://www.noobkit.com/show/ruby/ruby/ruby-core/io/readpartial.html) might be what I'm looking for, but I'm not sure. So I can do something like... $cmdin.readpartial(4096) ...and it will grab all the data in the pipe (up to 4096 bytes I think) and when it runs out of stuff to read it returns control back to the program. What happens though if there is more data waiting in the pipe than I call for with my IO.readpartial(maxlen) argument? Will it still return control or will it keep reading 4096 blocks until it runs out and THEN return control? I think that is how I would want it to handle. There is almost no examples out there about IO.readpartial, so I'm hoping someone here can help point me in the right direction. Thanks, Matt
From: ara.t.howard on 7 Jul 2008 11:12 On Jul 3, 2008, at 2:07 PM, Matt Mencel wrote: > A coworker suggested I run the original "zmprov" command as "zmprov > 2>&1" so that stderr is returned in stdout. Which will work...but I > was just wondering if I could get the code to work as designed. If > stdout is returning data...read it and do something....if stderr is > returning data then do something with that. I'm wanting to keep one > from blocking my ability to read the other. the issue is even worse than you describe, the program can easily become blocked if it's stdout or stderr pipes get full - to fix the situation you need to use threads, one processing both stdout and stderr asynchronously where each may have to trigger actions on stdin. the general pattern is q = Queue.new err = Thread.new do Thread.current.abort_on_exception = true while(( line = stderr.gets )) ... q.push :somthing if some_condition_on(line) end q.push :stderr_done end out = Thread.new do Thread.current.abort_on_exception = true while(( line = stdout.gets )) ... q.push :something if some_condition_on(line) end q.push :stdout_done end in = Thread.new do Thread.current.abort_on_exception while(( command = q.pop )) ... break if stdout_done and stderr_done end end in.join so basically have one thread sending commands down stdin. start a thread each for stdout and stderr, each doing their own processing, if they encounter something which means input needs to be send push it onto a queue to allow the stdin thread to do it on their behave. this of course ignores exceptional conditions and coordination between the stdout and stderr threads, but it's one approach. the big conditions any solution needs to handle are having no output on either stderr or stdout or being blocked on a write to either due a full pipe, which is why this cannot work safely: loop do handle stdout.gets handle stderr.gets end check out open4 and session for examples of using threads to process both stdout and stderr concurrently. http://codeforpeople.com/lib/ruby/ # gem install open4 session cheers. a @ http://codeforpeople.com/ -- we can deny everything, except that we have the possibility of being better. simply reflect on that. h.h. the 14th dalai lama
From: Matt Mencel on 17 Jul 2008 12:53 OK...using what ara.t.howard suggested, I've rewritten a block of my code and it looks something like this... >>>>CODE require "open3" require "thread" def runner(stdin, stdout, stderr, cmd) queue = Queue.new stdin.puts(cmd) errthd = Thread.new do Thread.current.abort_on_exception = true while(( line = stderr.gets )) queue.push(line) end queue.push :stderr_done end outthd = Thread.new do Thread.current.abort_on_exception = true while(( line = stdout.gets )) queue.push(line) end queue.push :stdout_done end inthd = Thread.new do Thread.current.abort_on_exception = true while(( stuff = queue.pop )) puts stuff break if :stdout_done and :stderr_done end end inthd.join errthd.exit outthd.exit end stdin, stdout, stderr = Open3.popen3("zmprov") runner(stdin, stdout, stderr, "selectMailbox shares") runner(stdin, stdout, stderr, "getAllFolders") <<<<CODE It almost works. I found that if I didn't add the errthd.exit and outthd.exit lines, the program would get stuck on inthd.join(waiting for the break conditions?) most of the time...very rarely would it complete. But the output is always incomplete. If I run the commands directly from the command line it look something like this... >>>>GOOD OUTPUT $ zmprov prov> selectMailbox shares mailbox: shares(a)domain.com, size: 174.36 KB, messages: 115, unread: 59 mbox shares(a)domain.com> getAllFolders Id View Unread Msg Count Path ---------- ---- ---------- ---------- ---------- 1 conv 0 0 / 16 docu 0 0 /Briefcase 10 appo 0 0 /Calendar 14 mess 0 0 /Chats 7 cont 0 0 /Contacts 6 mess 0 0 /Drafts 13 cont 0 2 /Emailed Contacts 257 appo 0 0 /evite 2 mess 0 0 /Inbox 4 mess 0 0 /Junk 12 wiki 0 0 /Notebook 5 mess 0 2 /Sent 15 task 0 0 /Tasks 3 conv 0 0 /Trash <<<<GOOD OUTPUT When I run my program though it looks more like this... >>>>>BAD OUTPUT prov> mailbox: shares(a)domain.com, size: 174.36 KB, messages: 115, unread: 59 mbox shares(a)domain.com> Id View Unread Msg Count Path <<<<<BAD OUTPUT The first line always seems to come out right... but the 'getAllFolders' command never comes out correctly. Maybe 1 in 10 times it will return a line or two more than just the 'mbox' line, but most of the time it's just that line of the output...the first line returned from the 'getAllFolders' command. So the thread for standard out doesn't seem to be grabbing all the data out of the pipe? I tried adding a short pause 'sleep 2' above the threads to see if maybe there was some slow response from them getting in the way, but that didn't really change anything. Any help or ideas anyone can offer? Thanks, Matt ----- "ara.t.howard" <ara.t.howard(a)gmail.com> wrote: > > the issue is even worse than you describe, the program can easily > become blocked if it's stdout or stderr pipes get full - to fix the > situation you need to use threads, one processing both stdout and > stderr asynchronously where each may have to trigger actions on > stdin. the general pattern is > > > q = Queue.new > > err = Thread.new do > Thread.current.abort_on_exception = true > > while(( line = stderr.gets )) > ... > q.push :somthing if some_condition_on(line) > end > q.push :stderr_done > end > > out = Thread.new do > Thread.current.abort_on_exception = true > > while(( line = stdout.gets )) > ... > q.push :something if some_condition_on(line) > end > q.push :stdout_done > end > > in = Thread.new do > Thread.current.abort_on_exception > > while(( command = q.pop )) > ... > break if stdout_done and stderr_done > end > end > > in.join > > so basically have one thread sending commands down stdin. start a > thread each for stdout and stderr, each doing their own processing, if > > they encounter something which means input needs to be send push it > onto a queue to allow the stdin thread to do it on their behave. this > > of course ignores exceptional conditions and coordination between the > > stdout and stderr threads, but it's one approach. > > > the big conditions any solution needs to handle are having no output > > on either stderr or stdout or being blocked on a write to either due a > > full pipe, which is why this cannot work safely: > > loop do > handle stdout.gets > handle stderr.gets > end > > check out open4 and session for examples of using threads to process > > both stdout and stderr concurrently. > > http://codeforpeople.com/lib/ruby/ > # gem install open4 session > > cheers. > > > a @ http://codeforpeople.com/ > -- > we can deny everything, except that we have the possibility of being > > better. simply reflect on that. > h.h. the 14th dalai lama
From: ara.t.howard on 17 Jul 2008 13:01
On Jul 17, 2008, at 10:53 AM, Matt Mencel wrote: > def runner(stdin, stdout, stderr, cmd) > queue = Queue.new > stdin.puts(cmd) stdin.flush > > errthd = Thread.new do > Thread.current.abort_on_exception = true > while(( line = stderr.gets )) > queue.push(line) > end > queue.push :stderr_done > end > > outthd = Thread.new do > Thread.current.abort_on_exception = true > while(( line = stdout.gets )) > queue.push(line) > end > queue.push :stdout_done > end > > inthd = Thread.new do > Thread.current.abort_on_exception = true > while(( stuff = queue.pop )) > puts stuff > break if :stdout_done and :stderr_done this test reads "break if true and true" !?! you are testing truth on two symbols. you need to keep vars and set them when :stdout_done and :stderr_done are seen on the queue. if stuff == :stdout_done stdout_done == true next end if stuff == :stderr_done stderr_done = true next end if stdout_done and stderr_done end etc... btw - you are simply re-writing the open4 gem - why not use it? > > end > end > > inthd.join > errthd.exit > outthd.exit > end a @ http://codeforpeople.com/ -- we can deny everything, except that we have the possibility of being better. simply reflect on that. h.h. the 14th dalai lama |