Prev: show a jpg picture on a dialog
Next: the shell can't find it when i double-click one file
From: Rahul on 14 May 2008 02:12
Hi,

I have a VC++(2005) application (2 "Generator" threads creating a data
buffer, and a "Writer" thread to write it).
The application creates and writes about 400 files (containt ~20MB
data each)
The Generator threads creates next data file while the Writer is
writing the previous (Generator waits until writer completes and
starts next file only when the current is handed over to Writer)
The writer thread does


My Problem is as follows
When the Writer thread creates and writer date into 400 different
files then the end to end jobtime is ~ 280 seconds (cpu usage time
remains same with teh below case)

Instead of creating 400 different files if I overwrite in a single
file always then the end to end jobtime is ~ 200 seconds.

So the slowdows is happening because of creating and writing into 400
different files.
I am using fopen/fwrite.

I want to know why is writing into multiple files slow.

There were no other apps running at the time of test (I killed all
network drivers also)

From: Giovanni Dicanio on 14 May 2008 03:58

"Rahul" <rahulsharma(a)lucent.com> ha scritto nel messaggio
news:70302021-03de-48cb-90a6-81ccc0c61838(a)a9g2000prl.googlegroups.com...

> My Problem is as follows
> When the Writer thread creates and writer date into 400 different
> files then the end to end jobtime is ~ 280 seconds (cpu usage time
> remains same with teh below case)
>
> Instead of creating 400 different files if I overwrite in a single
> file always then the end to end jobtime is ~ 200 seconds.
>
> So the slowdows is happening because of creating and writing into 400
> different files.
> I am using fopen/fwrite.
>
> I want to know why is writing into multiple files slow.

If my understanding is correct, when you write data to multiple files, the
total time is ~280 seconds.
Instead, if you write all your data to a single file, the total jobtime is
~200 seconds.

Well, I think that each time you call fopen you have an overhead (the
operating system must create a new file), so the file-opening overheads (for
each file you create) add up and are the main cause of the 80 seconds more
time.

Giovanni


From: Rahul on 14 May 2008 06:03
On May 14, 12:58 pm, "Giovanni Dicanio" <giovanni.dica...(a)invalid.com>
wrote:
> "Rahul" <rahulsha...(a)lucent.com> ha scritto nel messaggionews:70302021-03de-48cb-90a6-81ccc0c61838(a)a9g2000prl.googlegroups.com...
>
> > My Problem is as follows
> > When the Writer thread creates and writer date into 400 different
> > files then the end to end jobtime is ~ 280 seconds (cpu usage time
> > remains same with teh below case)
>
> > Instead of creating 400 different files if I overwrite in a single
> > file always then the end to end jobtime is ~ 200 seconds.
>
> > So the slowdows is happening because of creating and writing into 400
> > different files.
> > I am using fopen/fwrite.
>
> > I want to know why is writing into multiple files slow.
>
> If my understanding is correct, when you write data to multiple files, the
> total time is ~280 seconds.
> Instead, if you write all your data to a single file, the total jobtime is
> ~200 seconds.
>
> Well, I think that each time you call fopen you have an overhead (the
> operating system must create a new file), so the file-opening overheads (for
> each file you create) add up and are the main cause of the 80 seconds more
> time.
>
> Giovanni


No That is not the case,
As I mentioned previously, for the single file also I was doing fopen/
fclose 400 times, once for each data item created by the generated
So the number of fopen/fclose/fwrite calls are same in both cases
(case 1: multiple files and case 2: single file)
But the time differ by 80 seconds, Why???
From: Giovanni Dicanio on 14 May 2008 06:07

"Rahul" <rahulsharma(a)lucent.com> ha scritto nel messaggio
news:dae53aac-8142-4c85-adb0-b5e99b814e87(a)i76g2000hsf.googlegroups.com...

> As I mentioned previously, for the single file also I was doing fopen/
> fclose 400 times, once for each data item created by the generated
> So the number of fopen/fclose/fwrite calls are same in both cases
> (case 1: multiple files and case 2: single file)
> But the time differ by 80 seconds, Why???

Maybe in the single file scenario, the operating system has some "cache"
feature, so when you reopen the file again, it is faster than opening a
completely new file...

Giovanni


From: Giovanni Dicanio on 14 May 2008 06:09

"Rahul" <rahulsharma(a)lucent.com> ha scritto nel messaggio
news:dae53aac-8142-4c85-adb0-b5e99b814e87(a)i76g2000hsf.googlegroups.com...

> As I mentioned previously, for the single file also I was doing fopen/
> fclose 400 times, once for each data item created by the generated
> So the number of fopen/fclose/fwrite calls are same in both cases
> (case 1: multiple files and case 2: single file)
> But the time differ by 80 seconds, Why???

BTW: I think you may ask to the microsoft.public.win32.kernel newsgroup,
too...

Maybe this is a low-level thing, that the guys programming at the kernel
level know well.

Giovanni



 |  Next  |  Last
Pages: 1 2
Prev: show a jpg picture on a dialog
Next: the shell can't find it when i double-click one file