Prev: clients, was Re: Details about how process scheduler works?
Next: processor time
From: Chris Friesen on 15 Apr 2010 12:57
On 04/14/2010 10:21 PM, Rachit Agrawal wrote:

> Multiple processes will be writing to the file, there will be no
> reading. And all will be appending at the end of the file. And the
> access to this file to write can be radom,

You've just contradicted yourself. Are the file accesses random or are
they appending to the end of the file?

> I have already selected the file system, it is ext3 and OS is CentOS.
> Physical Disk won't be a problem, as I am working on a server, with
> 750 GB hard-disk and 8GB RAM.

Okay, so you've got a linux system, ext3, unspecified processor, and a
single spindle rotating media hard disk.

From your other email you suggested you wanted to write 7 bytes every
100 instructions or so. Assuming 2 billion instructions per second,
that works out to about 140MB/sec that you want to write. That's pretty
much at the limit of consumer-grade hardware.

Do you have any rules about how the chunks of 7 bytes need to be
ordered? Is it enough that each chunk is atomic or do they need to be
ordered somehow as well?

Chris
From: Rachit Agrawal on 15 Apr 2010 14:52
On Apr 15, 9:57 pm, Chris Friesen <cbf...(a)mail.usask.ca> wrote:
> On 04/14/2010 10:21 PM, Rachit Agrawal wrote:
>
> > Multiple processes will be writing to the file, there will be no
> > reading. And all will be appending at the end of the file. And the
> > access to this file to write can be radom,
By random I meant, the order in which different process writes is not
fixed, anyone can write first. And they will be appending at the end
only.

>
> You've just contradicted yourself.  Are the file accesses random or are
> they appending to the end of the file?
>
> > I have already selected the file system, it is ext3 and OS is CentOS.
> > Physical Disk won't be a problem, as I am working on a server, with
> > 750 GB hard-disk and 8GB RAM.
>
> Okay, so you've got a linux system, ext3, unspecified processor, and a
> single spindle rotating media hard disk.
Intel Xeon 24-core machine, forgot to mention that.

>
> From your other email you suggested you wanted to write 7 bytes every
> 100 instructions or so.  Assuming 2 billion instructions per second,
> that works out to about 140MB/sec that you want to write.  That's pretty
> much at the limit of consumer-grade hardware.
>
> Do you have any rules about how the chunks of 7 bytes need to be
> ordered?  Is it enough that each chunk is atomic or do they need to be
> ordered somehow as well?
Each chunk need to be atomic.

>
> Chris

From: David Schwartz on 15 Apr 2010 16:18
On Apr 14, 11:24 pm, Rachit Agrawal <rachitsw...(a)gmail.com> wrote:

> It would be 1 long, 1 short and 1 char so in total around (7 bytes).
> They will be written very often, say 1 write after every 100
> instructions. No process would be writing most of the time.

My intuition is that the best way to do this is either:

1) If it's supported on your platform, just have all the processes
append the file. On most platforms, small appends will be atomic.
(However, this is not absolutely guaranteed.)

2) Otherwise, have all the processes write to a pipe. (It can be a
named pipe if that's convenient.) Small writes to a pipe are
guaranteed atomic by POSIX. Then have one process that copies from the
pipe to the file.

DS
From: Rachit Agrawal on 15 Apr 2010 17:04
On Apr 16, 1:18 am, David Schwartz <dav...(a)webmaster.com> wrote:
> On Apr 14, 11:24 pm, Rachit Agrawal <rachitsw...(a)gmail.com> wrote:
>
> > It would be 1 long, 1 short and 1 char so in total around (7 bytes).
> > They will be written very often, say 1 write after every 100
> > instructions. No process would be writing most of the time.
>
> My intuition is that the best way to do this is either:
>
> 1) If it's supported on your platform, just have all the processes
> append the file. On most platforms, small appends will be atomic.
> (However, this is not absolutely guaranteed.)
>
> 2) Otherwise, have all the processes write to a pipe. (It can be a
> named pipe if that's convenient.) Small writes to a pipe are
> guaranteed atomic by POSIX. Then have one process that copies from the
> pipe to the file.
>
> DS
But in that case, won't it be better to use Message Queues? As

http://www.bluedonkey.org/cgi-bin/twiki/bin/view/Books/VxWorksCookbookTheIOSystem#Pipes_vs_i_Message_Queues

says pipes are a wrapper to Message queues and are bit slower than
them.
From: Chris Friesen on 15 Apr 2010 17:06
On 04/15/2010 03:04 PM, Rachit Agrawal wrote:

> But in that case, won't it be better to use Message Queues? As
>
> http://www.bluedonkey.org/cgi-bin/twiki/bin/view/Books/VxWorksCookbookTheIOSystem#Pipes_vs_i_Message_Queues
>
> says pipes are a wrapper to Message queues and are bit slower than
> them.

That is specific to vxworks.

Chris
First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: clients, was Re: Details about how process scheduler works?
Next: processor time