Performance issue: Writing multiple files vs single file using fopen/fwrite
in [Visual C]
|
Prev: Calling inherited protected method fails to compile
Next: Performance issue: Writing multiple files vs single file using fopen/fwrite
From: Rahul on 14 May 2008 02:18 Hi, I have a VC++(2005) application (2 "Generator" threads creating a data buffer, and a "Writer" thread to write it). The application creates and writes about 400 files (containt ~20MB data each) The Generator threads creates next data file while the Writer is writing the previous (Generator waits until writer completes and starts next file only when the current is handed over to Writer) The writer thread does fopen () fwrite in a loop fclose() My Problem is as follows When the Writer thread creates and writer date into 400 different files then the end to end jobtime is ~ 280 seconds (cpu usage time remains same with teh below case) Instead of creating 400 different files if I overwrite in a single file always then the end to end jobtime is ~ 200 seconds. So the slowdows is happening because of creating and writing into 400 different files. I am using fopen/fwrite. I want to know why is writing into multiple files slow. for the single file also we are opening and closing it 400 times so why is this running 80 seconds faster. There were no other apps running at the time of test (I killed all network drivers also) I have a 2 core machine (2.1 GHz each) with 2 GB RAM and 50 GB free HDD space (VM space = 4GB)
From: Carl Daniel [VC++ MVP] on 14 May 2008 09:40 Rahul wrote: > I want to know why is writing into multiple files slow. for the single > file also we are opening and closing it 400 times so why is this > running 80 seconds faster. First try a simple test: Make your program create and close 400 files, writing nothing to each file. By your description, you'd expect that to take 80 seconds. If it does, then you have your answer: file creation is slow. If it doesn't, then you know that there's something else different between the two configurations of your program that accounts for the difference. As a second experiment, you might try using CreateFile/WriteFile instead of fopen/fwrite to see if that makes a difference. -cd
From: Rahul on 14 May 2008 10:33 On May 14, 6:40 pm, "Carl Daniel [VC++ MVP]" <cpdaniel_remove_this_and_nos...(a)mvps.org.nospam> wrote: > Rahul wrote: > > I want to know why is writing into multiple files slow. for the single > > file also we are opening and closing it 400 times so why is this > > running 80 seconds faster. > > First try a simple test: Make your program create and close 400 files, > writing nothing to each file. By your description, you'd expect that to > take 80 seconds. If it does, then you have your answer: file creation is > slow. If it doesn't, then you know that there's something else different > between the two configurations of your program that accounts for the > difference. > > As a second experiment, you might try using CreateFile/WriteFile instead of > fopen/fwrite to see if that makes a difference. > > -cd The application behaves differently on different systems. on one computer I saw the fopen time (total for all 400 files) varying from 20 seconds to 100 seconds on different run's, while on the other the fopen time remained same but the fwrite time (total for all files) changed from 80 to 140 seconds. I know there can be variations, but 60-80 seconds variation seems very un realistic to me when the system virtaully free (only few process other then my app are doing I/O at the time of testing and their amount is quite less, ~ kb per minute) I tried using _open / _write but they also gave teh same results is _open different from CreateFile, i guess it will call CreateFile only (internally).
From: Igor Tandetnik on 14 May 2008 12:09 Rahul <rahulsharma(a)lucent.com> wrote: > On May 14, 6:40 pm, "Carl Daniel [VC++ MVP]" > <cpdaniel_remove_this_and_nos...(a)mvps.org.nospam> wrote: >> Rahul wrote: >>> I want to know why is writing into multiple files slow. for the >>> single file also we are opening and closing it 400 times so why is >>> this running 80 seconds faster. >> >> First try a simple test: Make your program create and close 400 >> files, writing nothing to each file. By your description, you'd >> expect that to take 80 seconds. If it does, then you have your >> answer: file creation is slow. If it doesn't, then you know that >> there's something else different between the two configurations of >> your program that accounts for the difference. >> >> As a second experiment, you might try using CreateFile/WriteFile >> instead of fopen/fwrite to see if that makes a difference. > > The application behaves differently on different systems. on one > computer I saw the fopen time (total for all 400 files) varying from > 20 seconds to 100 seconds on different run's, while on the other the > fopen time remained same but the fwrite time (total for all files) > changed from 80 to 140 seconds. Do these machines have different brands of antivirus software running, by any chance? Do the numbers change if you temporarily disable such software? -- With best wishes, Igor Tandetnik With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead. -- RFC 1925
From: Rahul on 14 May 2008 14:14
On May 14, 9:09 pm, "Igor Tandetnik" <itandet...(a)mvps.org> wrote: > Rahul <rahulsha...(a)lucent.com> wrote: > > On May 14, 6:40 pm, "Carl Daniel [VC++ MVP]" > > <cpdaniel_remove_this_and_nos...(a)mvps.org.nospam> wrote: > >> Rahul wrote: > >>> I want to know why is writing into multiple files slow. for the > >>> single file also we are opening and closing it 400 times so why is > >>> this running 80 seconds faster. > > >> First try a simple test: Make your program create and close 400 > >> files, writing nothing to each file. By your description, you'd > >> expect that to take 80 seconds. If it does, then you have your > >> answer: file creation is slow. If it doesn't, then you know that > >> there's something else different between the two configurations of > >> your program that accounts for the difference. > > >> As a second experiment, you might try using CreateFile/WriteFile > >> instead of fopen/fwrite to see if that makes a difference. > > > The application behaves differently on different systems. on one > > computer I saw the fopen time (total for all 400 files) varying from > > 20 seconds to 100 seconds on different run's, while on the other the > > fopen time remained same but the fwrite time (total for all files) > > changed from 80 to 140 seconds. > > Do these machines have different brands of antivirus software running, > by any chance? Do the numbers change if you temporarily disable such > software? > -- > With best wishes, > Igor Tandetnik > > With sufficient thrust, pigs fly just fine. However, this is not > necessarily a good idea. It is hard to be sure where they are going to > land, and it could be dangerous sitting under them as they fly > overhead. -- RFC 1925- Hide quoted text - > > - Show quoted text - There is no antivirus software on the systems as they are performance testing machines. and I had killed all other network drivers also, so the only proceses running were windows essential services e.g. svchost, exloprer, etc. I also killed the other non essential drivers (e.g. java updater, sound manager ect. ) But there is a mojor difference in the machines, one has windows 2003 SP2 and other has windows XP Though the area of slowdown was different in both machine (fopen vs fwrite) yet both showed a slowdown in case of different files. I did 2 quick tests 1) using a simple cpp program for writing data into same file (20 MB data 200 times, opening and closing the file in each of the 200 iteration) 2) writing 20 MB in 200 different files and got following results same_File.exe Total time_ fopen for 200 files in seconds = 1.829000 Total time_ fwrite for 20480 Kbytes per file in seconds = 11.123000 different_File.exe Total time_ fopen for 200 files in seconds = 17.984000 Total time_ fwrite for 20480 Kbytes per file in seconds = 86.746000 on multiple runs i got tsimilar timings on different machines. Is there any way to make make multiple file write as efficient as single file write. By explicitly caching some directory structure or something?? |