Prev: Light-weight/very-simple version control under Windows using Python?
Next: Socket performance
From: Nobody on 23 Jul 2010 17:27
On Fri, 23 Jul 2010 10:45:32 +0200, Thomas Guettler wrote:

> I use non-blocking io to check for timeouts. Sometimes I get EAGAIN
> (Resource temporarily unavailable) on write(). My working code looks
> like this. But I am unsure how many bytes have been written to the pipe
> if I get an EAGAIN IOError.

It should be zero; that's how the underlying system calls behave. If any
bytes are read/written, the number of bytes are returned. The call
only blocks (or fails with EAGAIN if non-blocking I/O is enabled) if it
isn't possible to read/write any data.

From: Kushal Kumaran on 24 Jul 2010 00:32
On Fri, Jul 23, 2010 at 2:15 PM, Thomas Guettler <hv(a)tbz-pariv.de> wrote:
> Hi,
>
> I use non-blocking io to check for timeouts. Sometimes I get EAGAIN (Resource temporarily unavailable)
> on write(). My working code looks like this. But I am unsure how many bytes have been written to the
> pipe if I get an EAGAIN IOError. Up to now I retry with the same chunk.
>
> If I get EAGAIN can I just sleep, and then retry with the same data chunk?
>
> pipe=subprocess.Popen(cmd, stdin=subprocess.PIPE, bufsize=-1)
> fcntl.fcntl(pipe.stdin, fcntl.F_SETFL, os.O_NONBLOCK)
> ....
> chunk_size=1024
>        while select.select([], [pipe.stdin], [], 5):

Please read the documentation of the select.select function. You are
not using it correctly. After the call returns, you need to check
whether the descriptor you are interested in is actually ready. If
select reaches your timeout, it will simply return three empty lists,
and your write will get EAGAIN. In general, after select has told you
a descriptor is ready, the first write after that should always
succeed.

BTW, from the documentation of file objects, it seems write always
wants to write the entire string you give it, since it does not return
anything. In that case, you might want to get the descriptor for the
file object (using pipe.fileno()) and use that for writing.

> <snipped remaining code>

--
regards,
kushal
From: Roy Smith on 24 Jul 2010 07:25
In article <mailman.1105.1279945954.1673.python-list(a)python.org>,
Kushal Kumaran <kushal.kumaran+python(a)gmail.com> wrote:

> In general, after select has told you a descriptor is ready, the
> first write after that should always succeed.

I used to think that too. Over the last few years, I've been
maintaining a large hunk of cross-platform C++ code which makes heavy
use of select(), with both UDP and TCP sockets. I've seen lots of
strange behavior.

For the moment, assume we're talking about a single-threaded program.
This simplifies things a lot.

If you write (pseudo-code):

select(fd)
write(fd)

when the select indicates fd is ready, it's not really saying, "The
following i/o call will succeed". What it's saying is, "The following
i/o call won't block". It could return an error, as long as it returns
it immediately.

Consider, for example, a write on a TCP connection. You are sitting in
a select(), when the other side closes the connection. The select()
should return, and the write should then immediately fail. If you're
tempted to say that the select() should return some sort of error,
consider the case where the remote end closes the connection after the
select() returns but before your process gets to execute the following
write() call.

We also saw a case where (due to what we consider a kernel bug), a
received UDP packet with a checksum error would cause the select() to
wake up, *then* notice the checksum error and discard the packet, and
thus the following read() would block.

The bottom line is if you really want to make sure you never block in an
I/O call, put your descriptor into non-blocking mode, and treat select()
as a *hint*. A way to ask the kernel, "Tell me when you think it might
be a good idea to try polling this descriptor again".
From: John Nagle on 24 Jul 2010 14:23
On 7/23/2010 1:45 AM, Thomas Guettler wrote:
> Hi,
>
> I use non-blocking io to check for timeouts. Sometimes I get EAGAIN (Resource temporarily unavailable)
> on write(). My working code looks like this. But I am unsure how many bytes have been written to the
> pipe if I get an EAGAIN IOError.

At the OS level, if you get EAGAIN, no bytes have been written. If
you get EOK from a non-blocking request, you must check the number of
bytes written to see if everything was written. You may have to write
more after an EOK.

Ref: http://www.opengroup.org/onlinepubs/009695399/functions/write.html

"If the O_NONBLOCK flag is set, write() shall not block the thread. If
some data can be written without blocking the thread, write() shall
write what it can and return the number of bytes written. Otherwise, it
shall return -1 and set errno to [EAGAIN]."

At the Python level, it's different.

http://docs.python.org/library/stdtypes.html#file-objects

says that "file.write(s)" returns nothing. If a non-blocking write
can't complete, you get an "io.BlockingIOError" exception

(ref "http://docs.python.org/release/3.0.1/library/io.html")

from which you can retrieve the number of bytes written.
This is only in Python 3.x, and may not be working right
(ref "http://bugs.python.org/issue7786"). It's unclear
what happens in this situation in Python 2.x, but it's
probably not what you wanted to happen.

However, "socket.send(s)" returns the number of bytes sent.
"send" and "recv" do work on pipes (with problems; see
"http://bugs.python.org/issue5573"). So
use "send", not "write", and pay attention to the number of bytes
sent.

> If I get EAGAIN can I just sleep, and then retry with the same data chunk?

Yes. But if you've filled the pipe, you may have to wait until the
program reading from the pipe reads more. This can take however long
the other program needs.

Incidentally, "select" apparently doesn't work on pipes under
Windows.

Since your code isn't doing anything else while waiting for a
write to complete on the pipe, why use non-blocking I/O at all?

(I know, the Python I/O timeout logic is broken in some versions.
You're probably working around that.)

John Nagle


From: Thomas Guettler on 26 Jul 2010 04:56
John Nagle wrote:
> On 7/23/2010 1:45 AM, Thomas Guettler wrote:
>> Hi,
>>
>> I use non-blocking io to check for timeouts. Sometimes I get EAGAIN
>> (Resource temporarily unavailable)
>> on write(). My working code looks like this. But I am unsure how many
>> bytes have been written to the
>> pipe if I get an EAGAIN IOError.

.....


> Since your code isn't doing anything else while waiting for a
> write to complete on the pipe, why use non-blocking I/O at all?
>
> (I know, the Python I/O timeout logic is broken in some versions.
> You're probably working around that.)

I want to handle timeouts. The app runs on linux, but I don't want to use
signals, since it is in a wsgi context:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues
> .. As a general rule therefore, no WSGI application component should
> attempt to register its own signal handlers.
>

The hint of Kushal was right: The timeout was reached, and I didn't check
the result of the select call.

--
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de
 |  Next  |  Last
Pages: 1 2
Prev: Light-weight/very-simple version control under Windows using Python?
Next: Socket performance