From: Tom Serface on
I think you need to be really careful not to use up all the real memory as
well so that you don't start swapping to disk. That is a killer that you
don't even see coming, although 50MB shouldn't be a problem on most modern
computers.

Tom

"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message
news:686vk59l8chun24uceekvdc8pt2uj4n811(a)4ax.com...
> By the way, did anyone really notice that ReadFile and WriteFile in Win64
> cannot read or
> write more than 4.2GB? Seems really, really strange the length and bytes
> read did not
> become DWORD_PTR values...
> joe
>
> On Thu, 14 Jan 2010 16:37:26 -0500, Joseph M. Newcomer
> <newcomer(a)flounder.com> wrote:
>
>>Yes, but the file size was given as 50MB.
>> joe
>>
>>On Thu, 14 Jan 2010 14:24:30 -0600, Stephen Myers
>><""StephenMyers\"@discussions(a)microsoft.com"> wrote:
>>
>>>Just to verify my (admittedly limited) understanding...
>>>
>>>I assume that the code posted will fail for files greater than 2GB or so
>>>with a 32 bit OS due to available address space.
>>>
>>>Steve
>>>
>>>Joseph M. Newcomer wrote:
>>>> See below...
>>>> On Thu, 14 Jan 2010 09:01:47 -0600, "Peter Olcott"
>>>> <NoSpam(a)SeeScreen.com> wrote:
>>>>
>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
>>>>> news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl...
>>>>>> Peter Olcott wrote:
>>>>>>
>>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in
>>>>>>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl...
>>>>>>>> Peter Olcott wrote:
>>>>>>>>
>>>>>>>> By File Copy, you mean DOS copy command or the
>>>>>>>> CopyFile() API?
>>>>>>> I am using the DOS command prompt's copy command. This
>>>>>>> is fast.
>>>>>>>
>>>>>>>
>>>>>>> The problem is the contradiction formed by the fact that
>>>>>>> reading and writng the file is fast, while reading and
>>>>>>> not wrting this same file is slow.
>>>>>>> I am currently using fopen() and fread(); I am using
>>>>>>> Windows XP.
>>>>>> True, if the DOS copy command is fast,then I believe the
>>>>>> code you are using is not optimal. The DOS Copy is using
>>>>>> the same CreateFile() API which fopen() also finally uses
>>>>>> in the RTL. So you should be able to match the same
>>>>>> performance of the DOS Copy command.
>>>>>>
>>>>>> Have you tried using setvbuf to set a buffer cache?
>>>>>>
>>>>>> Here is a small test code that opens a 50 meg file:
>>>>>>
>>>>>> // File: V:\wc7beta\testbufsize.cpp
>>>>>> // Compile with: cl testbufsize.cpp
>>>>>>
>>>>>> #include <stdio.h>
>>>>>> #include <windows.h>
>>>>>>
>>>>>> void main(char argc, char *argv[])
>>>>>> {
>>>>>> char _cache[1024*16] = {0}; // 16K cache
>>>>>> BYTE buf[1024*1] = {0}; // 1K buffer
>>>> ****
>>>> Reading a 50MB file, why such an incredibly tiny buffer?
>>>> ****
>>>>>> FILE *fv = fopen("largefile.dat","rb");
>>>>>> if (fv) {
>>>>>> int res = setvbuf(fv, _cache, _IOFBF,
>>>>>> sizeof(_cache));
>>>>>> DWORD nTotal = 0;
>>>>>> DWORD nDisks = 0;
>>>>>> DWORD nLoops = 0;
>>>>>> DWORD nStart = GetTickCount();
>>>>>> while (!feof(fv)) {
>>>>>> nLoops++;
>>>>>> memset(&buf,sizeof(buf),0);
>>>> ****
>>>> The memset is silly. Wastes time, accomplishes nothing. You are
>>>> setting a buffer to 0
>>>> right before completely overwriting it! This is like writing
>>>> int a;
>>>>
>>>> a = 0; // make sure a is 0 before assigning b
>>>> a = b;
>>>> ****
>>>>>> int nRead = fread(buf,1,sizeof(buf),fv);
>>>>>> nTotal +=nRead;
>>>>>> if (nRead > 0 && !fv->_cnt) nDisks++;
>>>>>> }
>>>>>> fclose(fv);
>>>>>> printf("Time: %d | Size: %d | Reads: %d | Disks:
>>>>>> %d\n",
>>>>>> GetTickCount()-nStart,
>>>>>> nTotal,
>>>>>> nLoops,
>>>>>> nDisks);
>>>>>> }
>>>>>> }
>>>> ****
>>>> If I were reading a small 50MB file, I would do
>>>>
>>>> void tmain(int argc, _TCHAR * argv[])
>>>> {
>>>> HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL,
>>>> OPEN_EXISTING,
>>>> FILE_ATTRIBUTE_NORMAL, NULL);
>>>>
>>>> LARGE_INTEGER size;
>>>>
>>>> GetFileSizeEx(h, &size);
>>>>
>>>> // This code assumes file is < 4.2GB!
>>>> LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT,
>>>> PAGE_READWRITE);
>>>> DWORD bytesRead;
>>>> ReadFile(h, p, size.LowPart, &bytesRead, NULL);
>>>> ... process data
>>>> VirtualFree(p, (SIZE_T)size.LowPart, MEM_DECOMMIT);
>>>> return 0;
>>>> }
>>>>
>>>> Note that the above does not do any error checking; the obvious error
>>>> checking is left as
>>>> an Exercise For The Reader. No read loops, no gratuitous memsets, just
>>>> simple code that
>>>> does exactly ONE ReadFile.
>>>> joe
>>>>
>>>>>> What this basically shows is the number of disk hits it
>>>>>> makes
>>>>>> by checking the fv->_cnt value. It shows that as long as
>>>>>> the cache size is larger than the read buffer size, you
>>>>>> get the same number of disk hits. I also spit out the
>>>>>> milliseconds. Subsequent runs, of course, is faster since
>>>>>> the OS API CreateFile() is used by the RTL in buffer mode.
>>>>>>
>>>>>> Also do you know what protocol you have Samba using?
>>>>> I am guessing that the code above will work with a file of
>>>>> any size?
>>>>> If that is the case, then you solved my problem.
>>>>> The only Samba protocol that I am aware of is smb.
>>>>>
>>>>>>
>>>>>> --
>>>>>> HLS
>>>> Joseph M. Newcomer [MVP]
>>>> email: newcomer(a)flounder.com
>>>> Web: http://www.flounder.com
>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm
>>Joseph M. Newcomer [MVP]
>>email: newcomer(a)flounder.com
>>Web: http://www.flounder.com
>>MVP Tips: http://www.flounder.com/mvp_tips.htm
> Joseph M. Newcomer [MVP]
> email: newcomer(a)flounder.com
> Web: http://www.flounder.com
> MVP Tips: http://www.flounder.com/mvp_tips.htm

From: Tom Serface on
I still had the problem on XP and Vista. I haven't tried testing it on Win7
yet, but I should do that since managing mapped drives is a holy nightmare
especially if you are using services that can't access them.

Tom

"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
news:#230xRPlKHA.5728(a)TK2MSFTNGP06.phx.gbl...
> Tom Serface wrote:
>
>> Not sure if this helps or not, but I've also noticed that network access
>> speeds up if you map a drive letter to the share rather than using a UNC.
>> I've seen as much as 3x speed up. I'm not sure why.
>
>
> Thats a good point. That was differently something I had to do under NT
> and always keep that in mind with other OSes. But after running out of
> drive letters and using UNC in certain new scenarios, I have not seen a
> slow down. I figured MS addressed this old problem. I seem to recall
> this was explained once back in the day.
>
> --
> HLS

From: Tom Serface on
Well, as someone pointed out, it may not be that big of a difference these
days. I know that Microsoft is not a big fan of supporting mapped drives
any longer so they may not be putting a lot of effort into improving it. It
was worth mentioning.

Tom

"Peter Olcott" <NoSpam(a)SeeScreen.com> wrote in message
news:AOydnRtcXMeGutLWnZ2dnUVZ_sKdnZ2d(a)giganews.com...
>
> "Tom Serface" <tom(a)camaswood.com> wrote in message
> news:uWT%23vsOlKHA.1536(a)TK2MSFTNGP06.phx.gbl...
>> Not sure if this helps or not, but I've also noticed that network access
>> speeds up if you map a drive letter to the share rather than using a UNC.
>> I've seen as much as 3x speed up. I'm not sure why.
>>
>> Tom
>
> I am already doing that.
>
>>
>> "Peter Olcott" <NoSpam(a)SeeScreen.com> wrote in message
>> news:ut-dnZaUeubM69PWnZ2dnUVZ_vWdnZ2d(a)giganews.com...
>>> I can copy a file to and from my Samba network in 20 seconds, yet it
>>> takes 80 seconds to compute its MD5 value. The delay is not related to
>>> the MD5 processing time because it can compute this in 12 seconds from
>>> the local drive.
>>>
>>> What is going on here?
>>>
>
>
From: Tom Serface on
That's what my test showed me as well. To be fair, I am often copying
files from optical media (CD, DVD, Blu-ray) so my buffer (16K) works well in
that environment.

Tom

"Peter Olcott" <NoSpam(a)SeeScreen.com> wrote in message
news:yt2dnXnWr6fyANLWnZ2dnUVZ_h2dnZ2d(a)giganews.com...
>
> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message
> news:blruk59utgoph0al9saun1gu93dooccr60(a)4ax.com...
>> See below...
>> On Wed, 13 Jan 2010 23:55:56 -0600, "Peter Olcott" <NoSpam(a)SeeScreen.com>
>> wrote:
>>
>>>
>>>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
>>>news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl...
>>>> Peter Olcott wrote:
>>>>
>>>>>
>>>>> I am doing block I/O, and it is very fast on the local
>>>>> drive and much slower on the 1.0 gb LAN, yet file copies
>>>>> to and from the LAN are still fast.
>>>>>
>>>>> (1) File copy to and from the LAN is faster than local
>>>>> drive copies, 20 seconds for LAN, 25 seconds for local.
>>>>> (2) Block I/O is fast on the local drive, 12 seconds for
>>>>> 632 MB.
>>>>> (3) Block I/O is slow on the LAN, 80 seconds for 632 MB.
>>>>> I also tried changing the block size from 4K to 1500
>>>>> bytes and 9000 bytes (consistent with Ethernet frame
>>>>> size), this did not help.
>>>>
>>>> By File Copy, you mean DOS copy command or the CopyFile()
>>>> API?
>>>I am using the DOS command prompt's copy command. This is
>>>fast.
>>>
>>>>
>>>> To me, the above appears to be consistent with a caching
>>>> issue that your code is not enabling when the file is
>>>> first open. The "File Copy" is doing it, but you are not.
>>>> Probably showing how you are opening the file will help,
>>>> i.e. the CreateFile() function or fopen().
>>>>
>>>> Another thing is maybe to check google
>>>>
>>>> Search: SAMBA Slow File Copy
>>>
>>>The problem is the contradiction formed by the fact that
>>>reading and writng the file is fast, while reading and not
>>>wrting this same file is slow.
>>>I am currently using fopen() and fread(); I am using
>>>Windows XP.
>> ****
>> Use of fopen/fread would certainly indicate that you are not doing this
>> in anything like
>> an optimal fashion.
>>
>> If you want to read a file that is under 100MB, it is usually best just
>> to allocate a
>> buffer the size of the file,
>
> I tested this and increasing the buffer size beyond 64K and 4K
> respectively does not measurably increase speed. Also in my case I must
> process files of arbitrary sizes to compute their MD5.
>
>> CreateFile, do a single ReadFile, do your computation, do a
>> WriteFile, and you are done. You are comparing two completely unrelated
>> concepts:
>> fopen/fread and a copy command; what you didn't ask was "what is the
>> fastest way to read a
>> file"; instead, you observe that two completely different technologies
>> have different
>> performance. You did not actually state this in your original question;
>> you just used a
>> generic concept of "copy". Details matter!
>>
>> Note that fread, called thousands of times, is amazingly slow in
>> comparison to a single
>> ReadFile.
>>
>> By failing to supply all the critical information, you essentially asked
>> "Why is it that I
>> can get from city A to city B in 20 minutes, but my friend takes two
>> hours?" and neglected
>> to mention you took the high-speed train while your friend went by
>> bicycle.
>> joe
>> ****
>>>
>>>>
>>>> There are some interesting hits, in particular if you are
>>>> using Vista, this MS Samba hotfix for Vista,
>>>>
>>>> http://support.microsoft.com/kb/931770
>>>>
>>>> There was another 2007 thread that the original poster
>>>> said turning off indexing improved the speed.
>>>>
>>>> --
>>>> HLS
>>>
>> Joseph M. Newcomer [MVP]
>> email: newcomer(a)flounder.com
>> Web: http://www.flounder.com
>> MVP Tips: http://www.flounder.com/mvp_tips.htm
>
>
From: Joseph M. Newcomer on
Of course, this clearly represents a failure of imagination during the original design.
joe

On Thu, 14 Jan 2010 19:43:07 -0800, "Alexander Grigoriev" <alegr(a)earthlink.net> wrote:

>Historically, I/O sizes in the kernel drivers has been stored in ULONG. And
>Length in IO_STACK_LOCATION is ULONG, too. That would be a bit too much
>hassle to convert everything to SIZE_T...
>
>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message
>news:686vk59l8chun24uceekvdc8pt2uj4n811(a)4ax.com...
>> By the way, did anyone really notice that ReadFile and WriteFile in Win64
>> cannot read or
>> write more than 4.2GB? Seems really, really strange the length and bytes
>> read did not
>> become DWORD_PTR values...
>> joe
>>
>> On Thu, 14 Jan 2010 16:37:26 -0500, Joseph M. Newcomer
>> <newcomer(a)flounder.com> wrote:
>>
>>>Yes, but the file size was given as 50MB.
>>> joe
>>>
>>>On Thu, 14 Jan 2010 14:24:30 -0600, Stephen Myers
>>><""StephenMyers\"@discussions(a)microsoft.com"> wrote:
>>>
>>>>Just to verify my (admittedly limited) understanding...
>>>>
>>>>I assume that the code posted will fail for files greater than 2GB or so
>>>>with a 32 bit OS due to available address space.
>>>>
>>>>Steve
>>>>
>>>>Joseph M. Newcomer wrote:
>>>>> See below...
>>>>> On Thu, 14 Jan 2010 09:01:47 -0600, "Peter Olcott"
>>>>> <NoSpam(a)SeeScreen.com> wrote:
>>>>>
>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
>>>>>> news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl...
>>>>>>> Peter Olcott wrote:
>>>>>>>
>>>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in
>>>>>>>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl...
>>>>>>>>> Peter Olcott wrote:
>>>>>>>>>
>>>>>>>>> By File Copy, you mean DOS copy command or the
>>>>>>>>> CopyFile() API?
>>>>>>>> I am using the DOS command prompt's copy command. This
>>>>>>>> is fast.
>>>>>>>>
>>>>>>>>
>>>>>>>> The problem is the contradiction formed by the fact that
>>>>>>>> reading and writng the file is fast, while reading and
>>>>>>>> not wrting this same file is slow.
>>>>>>>> I am currently using fopen() and fread(); I am using
>>>>>>>> Windows XP.
>>>>>>> True, if the DOS copy command is fast,then I believe the
>>>>>>> code you are using is not optimal. The DOS Copy is using
>>>>>>> the same CreateFile() API which fopen() also finally uses
>>>>>>> in the RTL. So you should be able to match the same
>>>>>>> performance of the DOS Copy command.
>>>>>>>
>>>>>>> Have you tried using setvbuf to set a buffer cache?
>>>>>>>
>>>>>>> Here is a small test code that opens a 50 meg file:
>>>>>>>
>>>>>>> // File: V:\wc7beta\testbufsize.cpp
>>>>>>> // Compile with: cl testbufsize.cpp
>>>>>>>
>>>>>>> #include <stdio.h>
>>>>>>> #include <windows.h>
>>>>>>>
>>>>>>> void main(char argc, char *argv[])
>>>>>>> {
>>>>>>> char _cache[1024*16] = {0}; // 16K cache
>>>>>>> BYTE buf[1024*1] = {0}; // 1K buffer
>>>>> ****
>>>>> Reading a 50MB file, why such an incredibly tiny buffer?
>>>>> ****
>>>>>>> FILE *fv = fopen("largefile.dat","rb");
>>>>>>> if (fv) {
>>>>>>> int res = setvbuf(fv, _cache, _IOFBF,
>>>>>>> sizeof(_cache));
>>>>>>> DWORD nTotal = 0;
>>>>>>> DWORD nDisks = 0;
>>>>>>> DWORD nLoops = 0;
>>>>>>> DWORD nStart = GetTickCount();
>>>>>>> while (!feof(fv)) {
>>>>>>> nLoops++;
>>>>>>> memset(&buf,sizeof(buf),0);
>>>>> ****
>>>>> The memset is silly. Wastes time, accomplishes nothing. You are
>>>>> setting a buffer to 0
>>>>> right before completely overwriting it! This is like writing
>>>>> int a;
>>>>>
>>>>> a = 0; // make sure a is 0 before assigning b
>>>>> a = b;
>>>>> ****
>>>>>>> int nRead = fread(buf,1,sizeof(buf),fv);
>>>>>>> nTotal +=nRead;
>>>>>>> if (nRead > 0 && !fv->_cnt) nDisks++;
>>>>>>> }
>>>>>>> fclose(fv);
>>>>>>> printf("Time: %d | Size: %d | Reads: %d | Disks:
>>>>>>> %d\n",
>>>>>>> GetTickCount()-nStart,
>>>>>>> nTotal,
>>>>>>> nLoops,
>>>>>>> nDisks);
>>>>>>> }
>>>>>>> }
>>>>> ****
>>>>> If I were reading a small 50MB file, I would do
>>>>>
>>>>> void tmain(int argc, _TCHAR * argv[])
>>>>> {
>>>>> HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL,
>>>>> OPEN_EXISTING,
>>>>> FILE_ATTRIBUTE_NORMAL, NULL);
>>>>>
>>>>> LARGE_INTEGER size;
>>>>>
>>>>> GetFileSizeEx(h, &size);
>>>>>
>>>>> // This code assumes file is < 4.2GB!
>>>>> LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT,
>>>>> PAGE_READWRITE);
>>>>> DWORD bytesRead;
>>>>> ReadFile(h, p, size.LowPart, &bytesRead, NULL);
>>>>> ... process data
>>>>> VirtualFree(p, (SIZE_T)size.LowPart, MEM_DECOMMIT);
>>>>> return 0;
>>>>> }
>>>>>
>>>>> Note that the above does not do any error checking; the obvious error
>>>>> checking is left as
>>>>> an Exercise For The Reader. No read loops, no gratuitous memsets, just
>>>>> simple code that
>>>>> does exactly ONE ReadFile.
>>>>> joe
>>>>>
>>>>>>> What this basically shows is the number of disk hits it
>>>>>>> makes
>>>>>>> by checking the fv->_cnt value. It shows that as long as
>>>>>>> the cache size is larger than the read buffer size, you
>>>>>>> get the same number of disk hits. I also spit out the
>>>>>>> milliseconds. Subsequent runs, of course, is faster since
>>>>>>> the OS API CreateFile() is used by the RTL in buffer mode.
>>>>>>>
>>>>>>> Also do you know what protocol you have Samba using?
>>>>>> I am guessing that the code above will work with a file of
>>>>>> any size?
>>>>>> If that is the case, then you solved my problem.
>>>>>> The only Samba protocol that I am aware of is smb.
>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> HLS
>>>>> Joseph M. Newcomer [MVP]
>>>>> email: newcomer(a)flounder.com
>>>>> Web: http://www.flounder.com
>>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm
>>>Joseph M. Newcomer [MVP]
>>>email: newcomer(a)flounder.com
>>>Web: http://www.flounder.com
>>>MVP Tips: http://www.flounder.com/mvp_tips.htm
>> Joseph M. Newcomer [MVP]
>> email: newcomer(a)flounder.com
>> Web: http://www.flounder.com
>> MVP Tips: http://www.flounder.com/mvp_tips.htm
>
Joseph M. Newcomer [MVP]
email: newcomer(a)flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm