From: Matt Triviski on
Hello All,

I am having a very annoying issue. This is the situation:

I am reading a file on one machine that is continuosly being written to
and writing out to a socket. On another machine I am reading from the
socket and writing it to a file. The problem is that when I tail the
original file and the file I am creating, the new file has a delay of
around 1 minute until I see the new data. It seems that the OS (Linux
Redhat 5) is waiting until it gets a block of data before it actually
writes to the file. I want it to write immediately.

Here is a code snippet of how I am reading from the socket and writing
to the file.

file = File.new(fname, "w")
file.sync = true

buff = ""

loop{
begin
ios = IO.select([server.socket], [file],[server.socket, file],
0.0001)
if ios[0].length > 0
buff = buff + ios[0].first.readpartial(8192)
end

if ios[1].length > 0
file.write buff
buff = ""
end

rescue Errno::ECONNRESET, SocketError
puts "ERROR: Lost connection to server"
exit 1
rescue EOFError
end

sleep 0.01
}
}

I first used just a regular readline from the socket and then did a
file.puts line, but I was having the same issue sot that is why I tried
to go to select. I am wondering if this is a Linux issue, since I
turned the file sync to true. However, I am not sure, so I am hoping
that other code heads out there could help me out.
--
Posted via http://www.ruby-forum.com/.

From: Eric Wong on
Matt Triviski <matt.triviski(a)gmail.com> wrote:
> Hello All,
>
> I am having a very annoying issue. This is the situation:
>
> I am reading a file on one machine that is continuosly being written to
> and writing out to a socket. On another machine I am reading from the
> socket and writing it to a file. The problem is that when I tail the
> original file and the file I am creating, the new file has a delay of
> around 1 minute until I see the new data. It seems that the OS (Linux
> Redhat 5) is waiting until it gets a block of data before it actually
> writes to the file. I want it to write immediately.
>
> Here is a code snippet of how I am reading from the socket and writing
> to the file.
>
> file = File.new(fname, "w")
> file.sync = true

Hi Matt,

"file.sync = true" looks good already.

Any chance you're writing to a file on NFS or some other remote
filesystem?

Since you're on Linux, try stracing both the Ruby process and your tail
process to see what it's doing.

> buff = ""
>
> loop{
> begin
> ios = IO.select([server.socket], [file],[server.socket, file],
> 0.0001)

There's no point in ever calling IO.select on a regular file,
they're always ready for reading/writing.

Since you're down to one socket, I wouldn't bother with select at all:

file.write(server.socket.readpartial(8192))
>
> rescue Errno::ECONNRESET, SocketError
> puts "ERROR: Lost connection to server"
> exit 1
> rescue EOFError
> end
>
> sleep 0.01
> }
> }
>
> I first used just a regular readline from the socket and then did a
> file.puts line, but I was having the same issue sot that is why I tried
> to go to select. I am wondering if this is a Linux issue, since I
> turned the file sync to true. However, I am not sure, so I am hoping
> that other code heads out there could help me out.

Whatever it is, there's a high likelyhood that strace-ing one of the
processes involved will tell you what the problem is :>

--
Eric Wong

From: Joel VanderWerf on
Eric Wong wrote:
> Matt Triviski <matt.triviski(a)gmail.com> wrote:
>> Hello All,
>>
>> I am having a very annoying issue. This is the situation:
>>
>> I am reading a file on one machine that is continuosly being written to
>> and writing out to a socket. On another machine I am reading from the
>> socket and writing it to a file. The problem is that when I tail the
>> original file and the file I am creating, the new file has a delay of
>> around 1 minute until I see the new data. It seems that the OS (Linux
>> Redhat 5) is waiting until it gets a block of data before it actually
>> writes to the file. I want it to write immediately.
>>
>> Here is a code snippet of how I am reading from the socket and writing
>> to the file.
>>
>> file = File.new(fname, "w")
>> file.sync = true
>
> Hi Matt,
>
> "file.sync = true" looks good already.

You could try calling fsync as well (it's not a writer, just call
"file.fsync"):

Implementation from IO
------------------------------------------------------------------------------
ios.fsync -> 0 or nil

------------------------------------------------------------------------------

Immediately writes all buffered data in ios to disk. Note that
fsync differs from using IO#sync=. The latter ensures that data
is flushed from Ruby's buffers, but doesn't not guarantee that the
underlying
operating system actually writes it to disk.

NotImplementedError is raised if the underlying operating system does
not support fsync(2).



From: Matt Triviski on
Joel VanderWerf wrote:
> Eric Wong wrote:
>>> Redhat 5) is waiting until it gets a block of data before it actually
>> "file.sync = true" looks good already.
> You could try calling fsync as well (it's not a writer, just call
> "file.fsync"):
>
> Implementation from IO
> ------------------------------------------------------------------------------
> ios.fsync -> 0 or nil
>
> ------------------------------------------------------------------------------
>
> Immediately writes all buffered data in ios to disk. Note that
> fsync differs from using IO#sync=. The latter ensures that data
> is flushed from Ruby's buffers, but doesn't not guarantee that the
> underlying
> operating system actually writes it to disk.
>
> NotImplementedError is raised if the underlying operating system does
> not support fsync(2).


Hey fellas, thanks for the response.

I tried file.fsync, and checked to make sure that the return was not
nil, and it wasn't. However, I am seeing the same activity. I took out
the select call and as Eric suggested and just used
"file.write(server.socket.readpartial(8k))" with and that did not seem
to help either.

I am writing to NFS, so I am thinking that this may be the issue. I'm
starting to think that there is no way for me to get what I want, except
to rewrite this in C or Assembly :(
--
Posted via http://www.ruby-forum.com/.

From: Eric Wong on
Matt Triviski <matt.triviski(a)gmail.com> wrote:
> I am writing to NFS, so I am thinking that this may be the issue. I'm
> starting to think that there is no way for me to get what I want, except
> to rewrite this in C or Assembly :(

NFS is your problem, not Ruby. See the nfs(5) manpage and see if you
(or your friendly systems administrator) can tune the NFS mount
options to invalidate the cache more frequently.

Ruby IO methods are already thin wrappers around corresponding C
functions (and thinner if you use the sys{write,read,...} variants).
You normally don't need to go lower-level than that. Heck, with enough
time and effort, you even could write your own NFS client implementation
in Ruby (not a serious suggestion :)

--
Eric Wong