Prev: clients, was Re: Details about how process scheduler works?
Next: processor time
From: Rachit Agrawal on 14 Apr 2010 07:07
Hi,

I have a scenario where multiple process wants to write to a single
file. I was wondering what would be the best method to achieve this?

1. Message Queues.
2. Memory Mapped Files
3. Or any other method.

Speed is the main concern. Can someone please throw some light on
this?

Thank you.
From: Ersek, Laszlo on 14 Apr 2010 09:24
On Wed, 14 Apr 2010, Rachit Agrawal wrote:

> I have a scenario where multiple process wants to write to a single
> file. I was wondering what would be the best method to achieve this?
>
> 1. Message Queues.
> 2. Memory Mapped Files
> 3. Or any other method.
>
> Speed is the main concern.

(I have no idea which is faster, I'd just like to list some options for
further discussion and myth busting by others.)

write offsets blocks are ordered, but
are available their sizes are unknown
independently in advance

regular file output 1 2

pipe output 3 4

1.a. pwrite()
1.b. mmap()
1.c. reordering by way of a priority queue and sequential writing

2. pq + sequential writing

3. pq + sequential writing

4. pq + sequential writing

If there is a good IO scheduler and/or the disk (subsystem) can service a
number of "parallel write streams", the order of preference is probably
(i) mmap(), (ii) pwrite(), (iii) pq. I guess the kernel can take hints for
accessing the underlying object for both pwrite() and mmap()
(posix_fadvise() and posix_madvise()). Also, a pq will block if the
sequentially next block is not yet available even in case 1, which is
wrong.

On the other hand, the pq solution seems to be applicable to all four
cases. You would rely on the IO scheduler and disk (subsystem) to
reorganize writes to a lesser extent, decreasing the risk of thrashing.
For most disks, sequential writes are faster than random writes. You
shouldn't lose the benefit of striping though, if striping is supported
underneath. (<http://en.wikipedia.org/wiki/RAID#Standard_levels>.)
Supposing the maximum sequential write throughput will be your actual
bottleneck, the contention on the shared queue should be negligible.

I have no idea.
lacos
From: Eric Sosman on 14 Apr 2010 09:44
On 4/14/2010 7:07 AM, Rachit Agrawal wrote:
> Hi,
>
> I have a scenario where multiple process wants to write to a single
> file. I was wondering what would be the best method to achieve this?
>
> 1. Message Queues.
> 2. Memory Mapped Files
> 3. Or any other method.
>
> Speed is the main concern. Can someone please throw some light on
> this?

Just open the file in all the processes and have them
write to it. To improve speed, write less.

... or are there issues and details you didn't mention?

--
Eric Sosman
esosman(a)ieee-dot-org.invalid
From: Chris Friesen on 14 Apr 2010 15:05
On 04/14/2010 05:07 AM, Rachit Agrawal wrote:
> Hi,
>
> I have a scenario where multiple process wants to write to a single
> file. I was wondering what would be the best method to achieve this?
>
> 1. Message Queues.
> 2. Memory Mapped Files
> 3. Or any other method.
>
> Speed is the main concern. Can someone please throw some light on
> this?

The obvious solution is just having them all open the file and write to
it using the normal file operations. You will want to use file locking
to ensure that you don't get multiple processes trampling each other's data.

Do you have any indications that performance is a problem with the
obvious solution?

Chris
From: Rachit Agrawal on 14 Apr 2010 17:42
On Apr 15, 12:05 am, Chris Friesen <cbf...(a)mail.usask.ca> wrote:
> On 04/14/2010 05:07 AM, Rachit Agrawal wrote:
>
> > Hi,
>
> > I have a scenario where multiple process wants to write to a single
> > file. I was wondering what would be the best method to achieve this?
>
> > 1. Message Queues.
> > 2. Memory Mapped Files
> > 3. Or any other method.
>
> > Speed is the main concern. Can someone please throw some light on
> > this?
>
> The obvious solution is just having them all open the file and write to
> it using the normal file operations.   You will want to use file locking
> to ensure that you don't get multiple processes trampling each other's data.
>
> Do you have any indications that performance is a problem with the
> obvious solution?
>
> Chris

@Ersek: Sorry I couldn't understand your point much.

@Chris, @Eric: I have read that File locking is generally slow. So,
instead of implementing all the possible ways, I wanted to know if
there is any known faster method than the traditional file locking.
Without implementing all I don't know how to compare the performance.

P.S: The amount of data I will be writing will be huge. Around 4-5 GB
of data.
 |  Next  |  Last
Pages: 1 2 3 4
Prev: clients, was Re: Details about how process scheduler works?
Next: processor time