From: Joseph M. Newcomer on
See below...
On Thu, 14 Jan 2010 09:01:47 -0600, "Peter Olcott" <NoSpam(a)SeeScreen.com> wrote:

>
>"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
>news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl...
>> Peter Olcott wrote:
>>
>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in
>>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl...
>>>> Peter Olcott wrote:
>>>>
>>>> By File Copy, you mean DOS copy command or the
>>>> CopyFile() API?
>>
>> >
>>
>>> I am using the DOS command prompt's copy command. This
>>> is fast.
>>>
>>>
>>> The problem is the contradiction formed by the fact that
>>> reading and writng the file is fast, while reading and
>>> not wrting this same file is slow.
>>> I am currently using fopen() and fread(); I am using
>>> Windows XP.
>>
>> True, if the DOS copy command is fast,then I believe the
>> code you are using is not optimal. The DOS Copy is using
>> the same CreateFile() API which fopen() also finally uses
>> in the RTL. So you should be able to match the same
>> performance of the DOS Copy command.
>>
>> Have you tried using setvbuf to set a buffer cache?
>>
>> Here is a small test code that opens a 50 meg file:
>>
>> // File: V:\wc7beta\testbufsize.cpp
>> // Compile with: cl testbufsize.cpp
>>
>> #include <stdio.h>
>> #include <windows.h>
>>
>> void main(char argc, char *argv[])
>> {
>> char _cache[1024*16] = {0}; // 16K cache
>> BYTE buf[1024*1] = {0}; // 1K buffer
****
Reading a 50MB file, why such an incredibly tiny buffer?
****
>>
>> FILE *fv = fopen("largefile.dat","rb");
>> if (fv) {
>> int res = setvbuf(fv, _cache, _IOFBF,
>> sizeof(_cache));
>> DWORD nTotal = 0;
>> DWORD nDisks = 0;
>> DWORD nLoops = 0;
>> DWORD nStart = GetTickCount();
>> while (!feof(fv)) {
>> nLoops++;
>> memset(&buf,sizeof(buf),0);
****
The memset is silly. Wastes time, accomplishes nothing. You are setting a buffer to 0
right before completely overwriting it! This is like writing
int a;

a = 0; // make sure a is 0 before assigning b
a = b;
****
>> int nRead = fread(buf,1,sizeof(buf),fv);
>> nTotal +=nRead;
>> if (nRead > 0 && !fv->_cnt) nDisks++;
>> }
>> fclose(fv);
>> printf("Time: %d | Size: %d | Reads: %d | Disks:
>> %d\n",
>> GetTickCount()-nStart,
>> nTotal,
>> nLoops,
>> nDisks);
>> }
>> }
****
If I were reading a small 50MB file, I would do

void tmain(int argc, _TCHAR * argv[])
{
HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL, OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL, NULL);

LARGE_INTEGER size;

GetFileSizeEx(h, &size);

// This code assumes file is < 4.2GB!
LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT, PAGE_READWRITE);
DWORD bytesRead;
ReadFile(h, p, size.LowPart, &bytesRead, NULL);
... process data
VirtualFree(p, (SIZE_T)size.LowPart, MEM_DECOMMIT);
return 0;
}

Note that the above does not do any error checking; the obvious error checking is left as
an Exercise For The Reader. No read loops, no gratuitous memsets, just simple code that
does exactly ONE ReadFile.
joe

>>
>> What this basically shows is the number of disk hits it
>> makes
>> by checking the fv->_cnt value. It shows that as long as
>> the cache size is larger than the read buffer size, you
>> get the same number of disk hits. I also spit out the
>> milliseconds. Subsequent runs, of course, is faster since
>> the OS API CreateFile() is used by the RTL in buffer mode.
>>
>> Also do you know what protocol you have Samba using?
>
>I am guessing that the code above will work with a file of
>any size?
>If that is the case, then you solved my problem.
>The only Samba protocol that I am aware of is smb.
>
>>
>>
>> --
>> HLS
>
Joseph M. Newcomer [MVP]
email: newcomer(a)flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Stephen Myers on
Just to verify my (admittedly limited) understanding...

I assume that the code posted will fail for files greater than 2GB or so
with a 32 bit OS due to available address space.

Steve

Joseph M. Newcomer wrote:
> See below...
> On Thu, 14 Jan 2010 09:01:47 -0600, "Peter Olcott" <NoSpam(a)SeeScreen.com> wrote:
>
>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
>> news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl...
>>> Peter Olcott wrote:
>>>
>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in
>>>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl...
>>>>> Peter Olcott wrote:
>>>>>
>>>>> By File Copy, you mean DOS copy command or the
>>>>> CopyFile() API?
>>>> I am using the DOS command prompt's copy command. This
>>>> is fast.
>>>>
>>>>
>>>> The problem is the contradiction formed by the fact that
>>>> reading and writng the file is fast, while reading and
>>>> not wrting this same file is slow.
>>>> I am currently using fopen() and fread(); I am using
>>>> Windows XP.
>>> True, if the DOS copy command is fast,then I believe the
>>> code you are using is not optimal. The DOS Copy is using
>>> the same CreateFile() API which fopen() also finally uses
>>> in the RTL. So you should be able to match the same
>>> performance of the DOS Copy command.
>>>
>>> Have you tried using setvbuf to set a buffer cache?
>>>
>>> Here is a small test code that opens a 50 meg file:
>>>
>>> // File: V:\wc7beta\testbufsize.cpp
>>> // Compile with: cl testbufsize.cpp
>>>
>>> #include <stdio.h>
>>> #include <windows.h>
>>>
>>> void main(char argc, char *argv[])
>>> {
>>> char _cache[1024*16] = {0}; // 16K cache
>>> BYTE buf[1024*1] = {0}; // 1K buffer
> ****
> Reading a 50MB file, why such an incredibly tiny buffer?
> ****
>>> FILE *fv = fopen("largefile.dat","rb");
>>> if (fv) {
>>> int res = setvbuf(fv, _cache, _IOFBF,
>>> sizeof(_cache));
>>> DWORD nTotal = 0;
>>> DWORD nDisks = 0;
>>> DWORD nLoops = 0;
>>> DWORD nStart = GetTickCount();
>>> while (!feof(fv)) {
>>> nLoops++;
>>> memset(&buf,sizeof(buf),0);
> ****
> The memset is silly. Wastes time, accomplishes nothing. You are setting a buffer to 0
> right before completely overwriting it! This is like writing
> int a;
>
> a = 0; // make sure a is 0 before assigning b
> a = b;
> ****
>>> int nRead = fread(buf,1,sizeof(buf),fv);
>>> nTotal +=nRead;
>>> if (nRead > 0 && !fv->_cnt) nDisks++;
>>> }
>>> fclose(fv);
>>> printf("Time: %d | Size: %d | Reads: %d | Disks:
>>> %d\n",
>>> GetTickCount()-nStart,
>>> nTotal,
>>> nLoops,
>>> nDisks);
>>> }
>>> }
> ****
> If I were reading a small 50MB file, I would do
>
> void tmain(int argc, _TCHAR * argv[])
> {
> HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL, OPEN_EXISTING,
> FILE_ATTRIBUTE_NORMAL, NULL);
>
> LARGE_INTEGER size;
>
> GetFileSizeEx(h, &size);
>
> // This code assumes file is < 4.2GB!
> LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT, PAGE_READWRITE);
> DWORD bytesRead;
> ReadFile(h, p, size.LowPart, &bytesRead, NULL);
> ... process data
> VirtualFree(p, (SIZE_T)size.LowPart, MEM_DECOMMIT);
> return 0;
> }
>
> Note that the above does not do any error checking; the obvious error checking is left as
> an Exercise For The Reader. No read loops, no gratuitous memsets, just simple code that
> does exactly ONE ReadFile.
> joe
>
>>> What this basically shows is the number of disk hits it
>>> makes
>>> by checking the fv->_cnt value. It shows that as long as
>>> the cache size is larger than the read buffer size, you
>>> get the same number of disk hits. I also spit out the
>>> milliseconds. Subsequent runs, of course, is faster since
>>> the OS API CreateFile() is used by the RTL in buffer mode.
>>>
>>> Also do you know what protocol you have Samba using?
>> I am guessing that the code above will work with a file of
>> any size?
>> If that is the case, then you solved my problem.
>> The only Samba protocol that I am aware of is smb.
>>
>>>
>>> --
>>> HLS
> Joseph M. Newcomer [MVP]
> email: newcomer(a)flounder.com
> Web: http://www.flounder.com
> MVP Tips: http://www.flounder.com/mvp_tips.htm

From: Joseph M. Newcomer on
Yes, but the file size was given as 50MB.
joe

On Thu, 14 Jan 2010 14:24:30 -0600, Stephen Myers
<""StephenMyers\"@discussions(a)microsoft.com"> wrote:

>Just to verify my (admittedly limited) understanding...
>
>I assume that the code posted will fail for files greater than 2GB or so
>with a 32 bit OS due to available address space.
>
>Steve
>
>Joseph M. Newcomer wrote:
>> See below...
>> On Thu, 14 Jan 2010 09:01:47 -0600, "Peter Olcott" <NoSpam(a)SeeScreen.com> wrote:
>>
>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
>>> news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl...
>>>> Peter Olcott wrote:
>>>>
>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in
>>>>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl...
>>>>>> Peter Olcott wrote:
>>>>>>
>>>>>> By File Copy, you mean DOS copy command or the
>>>>>> CopyFile() API?
>>>>> I am using the DOS command prompt's copy command. This
>>>>> is fast.
>>>>>
>>>>>
>>>>> The problem is the contradiction formed by the fact that
>>>>> reading and writng the file is fast, while reading and
>>>>> not wrting this same file is slow.
>>>>> I am currently using fopen() and fread(); I am using
>>>>> Windows XP.
>>>> True, if the DOS copy command is fast,then I believe the
>>>> code you are using is not optimal. The DOS Copy is using
>>>> the same CreateFile() API which fopen() also finally uses
>>>> in the RTL. So you should be able to match the same
>>>> performance of the DOS Copy command.
>>>>
>>>> Have you tried using setvbuf to set a buffer cache?
>>>>
>>>> Here is a small test code that opens a 50 meg file:
>>>>
>>>> // File: V:\wc7beta\testbufsize.cpp
>>>> // Compile with: cl testbufsize.cpp
>>>>
>>>> #include <stdio.h>
>>>> #include <windows.h>
>>>>
>>>> void main(char argc, char *argv[])
>>>> {
>>>> char _cache[1024*16] = {0}; // 16K cache
>>>> BYTE buf[1024*1] = {0}; // 1K buffer
>> ****
>> Reading a 50MB file, why such an incredibly tiny buffer?
>> ****
>>>> FILE *fv = fopen("largefile.dat","rb");
>>>> if (fv) {
>>>> int res = setvbuf(fv, _cache, _IOFBF,
>>>> sizeof(_cache));
>>>> DWORD nTotal = 0;
>>>> DWORD nDisks = 0;
>>>> DWORD nLoops = 0;
>>>> DWORD nStart = GetTickCount();
>>>> while (!feof(fv)) {
>>>> nLoops++;
>>>> memset(&buf,sizeof(buf),0);
>> ****
>> The memset is silly. Wastes time, accomplishes nothing. You are setting a buffer to 0
>> right before completely overwriting it! This is like writing
>> int a;
>>
>> a = 0; // make sure a is 0 before assigning b
>> a = b;
>> ****
>>>> int nRead = fread(buf,1,sizeof(buf),fv);
>>>> nTotal +=nRead;
>>>> if (nRead > 0 && !fv->_cnt) nDisks++;
>>>> }
>>>> fclose(fv);
>>>> printf("Time: %d | Size: %d | Reads: %d | Disks:
>>>> %d\n",
>>>> GetTickCount()-nStart,
>>>> nTotal,
>>>> nLoops,
>>>> nDisks);
>>>> }
>>>> }
>> ****
>> If I were reading a small 50MB file, I would do
>>
>> void tmain(int argc, _TCHAR * argv[])
>> {
>> HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL, OPEN_EXISTING,
>> FILE_ATTRIBUTE_NORMAL, NULL);
>>
>> LARGE_INTEGER size;
>>
>> GetFileSizeEx(h, &size);
>>
>> // This code assumes file is < 4.2GB!
>> LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT, PAGE_READWRITE);
>> DWORD bytesRead;
>> ReadFile(h, p, size.LowPart, &bytesRead, NULL);
>> ... process data
>> VirtualFree(p, (SIZE_T)size.LowPart, MEM_DECOMMIT);
>> return 0;
>> }
>>
>> Note that the above does not do any error checking; the obvious error checking is left as
>> an Exercise For The Reader. No read loops, no gratuitous memsets, just simple code that
>> does exactly ONE ReadFile.
>> joe
>>
>>>> What this basically shows is the number of disk hits it
>>>> makes
>>>> by checking the fv->_cnt value. It shows that as long as
>>>> the cache size is larger than the read buffer size, you
>>>> get the same number of disk hits. I also spit out the
>>>> milliseconds. Subsequent runs, of course, is faster since
>>>> the OS API CreateFile() is used by the RTL in buffer mode.
>>>>
>>>> Also do you know what protocol you have Samba using?
>>> I am guessing that the code above will work with a file of
>>> any size?
>>> If that is the case, then you solved my problem.
>>> The only Samba protocol that I am aware of is smb.
>>>
>>>>
>>>> --
>>>> HLS
>> Joseph M. Newcomer [MVP]
>> email: newcomer(a)flounder.com
>> Web: http://www.flounder.com
>> MVP Tips: http://www.flounder.com/mvp_tips.htm
Joseph M. Newcomer [MVP]
email: newcomer(a)flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on
By the way, did anyone really notice that ReadFile and WriteFile in Win64 cannot read or
write more than 4.2GB? Seems really, really strange the length and bytes read did not
become DWORD_PTR values...
joe

On Thu, 14 Jan 2010 16:37:26 -0500, Joseph M. Newcomer <newcomer(a)flounder.com> wrote:

>Yes, but the file size was given as 50MB.
> joe
>
>On Thu, 14 Jan 2010 14:24:30 -0600, Stephen Myers
><""StephenMyers\"@discussions(a)microsoft.com"> wrote:
>
>>Just to verify my (admittedly limited) understanding...
>>
>>I assume that the code posted will fail for files greater than 2GB or so
>>with a 32 bit OS due to available address space.
>>
>>Steve
>>
>>Joseph M. Newcomer wrote:
>>> See below...
>>> On Thu, 14 Jan 2010 09:01:47 -0600, "Peter Olcott" <NoSpam(a)SeeScreen.com> wrote:
>>>
>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
>>>> news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl...
>>>>> Peter Olcott wrote:
>>>>>
>>>>>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in
>>>>>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl...
>>>>>>> Peter Olcott wrote:
>>>>>>>
>>>>>>> By File Copy, you mean DOS copy command or the
>>>>>>> CopyFile() API?
>>>>>> I am using the DOS command prompt's copy command. This
>>>>>> is fast.
>>>>>>
>>>>>>
>>>>>> The problem is the contradiction formed by the fact that
>>>>>> reading and writng the file is fast, while reading and
>>>>>> not wrting this same file is slow.
>>>>>> I am currently using fopen() and fread(); I am using
>>>>>> Windows XP.
>>>>> True, if the DOS copy command is fast,then I believe the
>>>>> code you are using is not optimal. The DOS Copy is using
>>>>> the same CreateFile() API which fopen() also finally uses
>>>>> in the RTL. So you should be able to match the same
>>>>> performance of the DOS Copy command.
>>>>>
>>>>> Have you tried using setvbuf to set a buffer cache?
>>>>>
>>>>> Here is a small test code that opens a 50 meg file:
>>>>>
>>>>> // File: V:\wc7beta\testbufsize.cpp
>>>>> // Compile with: cl testbufsize.cpp
>>>>>
>>>>> #include <stdio.h>
>>>>> #include <windows.h>
>>>>>
>>>>> void main(char argc, char *argv[])
>>>>> {
>>>>> char _cache[1024*16] = {0}; // 16K cache
>>>>> BYTE buf[1024*1] = {0}; // 1K buffer
>>> ****
>>> Reading a 50MB file, why such an incredibly tiny buffer?
>>> ****
>>>>> FILE *fv = fopen("largefile.dat","rb");
>>>>> if (fv) {
>>>>> int res = setvbuf(fv, _cache, _IOFBF,
>>>>> sizeof(_cache));
>>>>> DWORD nTotal = 0;
>>>>> DWORD nDisks = 0;
>>>>> DWORD nLoops = 0;
>>>>> DWORD nStart = GetTickCount();
>>>>> while (!feof(fv)) {
>>>>> nLoops++;
>>>>> memset(&buf,sizeof(buf),0);
>>> ****
>>> The memset is silly. Wastes time, accomplishes nothing. You are setting a buffer to 0
>>> right before completely overwriting it! This is like writing
>>> int a;
>>>
>>> a = 0; // make sure a is 0 before assigning b
>>> a = b;
>>> ****
>>>>> int nRead = fread(buf,1,sizeof(buf),fv);
>>>>> nTotal +=nRead;
>>>>> if (nRead > 0 && !fv->_cnt) nDisks++;
>>>>> }
>>>>> fclose(fv);
>>>>> printf("Time: %d | Size: %d | Reads: %d | Disks:
>>>>> %d\n",
>>>>> GetTickCount()-nStart,
>>>>> nTotal,
>>>>> nLoops,
>>>>> nDisks);
>>>>> }
>>>>> }
>>> ****
>>> If I were reading a small 50MB file, I would do
>>>
>>> void tmain(int argc, _TCHAR * argv[])
>>> {
>>> HANDLE h = CreateFile(_T("largefile.dat"), GENERIC_READ, 0, NULL, OPEN_EXISTING,
>>> FILE_ATTRIBUTE_NORMAL, NULL);
>>>
>>> LARGE_INTEGER size;
>>>
>>> GetFileSizeEx(h, &size);
>>>
>>> // This code assumes file is < 4.2GB!
>>> LPVOID p = VirtualAlloc(NULL, (SIZE_T)size.LowPart, MEM_COMMIT, PAGE_READWRITE);
>>> DWORD bytesRead;
>>> ReadFile(h, p, size.LowPart, &bytesRead, NULL);
>>> ... process data
>>> VirtualFree(p, (SIZE_T)size.LowPart, MEM_DECOMMIT);
>>> return 0;
>>> }
>>>
>>> Note that the above does not do any error checking; the obvious error checking is left as
>>> an Exercise For The Reader. No read loops, no gratuitous memsets, just simple code that
>>> does exactly ONE ReadFile.
>>> joe
>>>
>>>>> What this basically shows is the number of disk hits it
>>>>> makes
>>>>> by checking the fv->_cnt value. It shows that as long as
>>>>> the cache size is larger than the read buffer size, you
>>>>> get the same number of disk hits. I also spit out the
>>>>> milliseconds. Subsequent runs, of course, is faster since
>>>>> the OS API CreateFile() is used by the RTL in buffer mode.
>>>>>
>>>>> Also do you know what protocol you have Samba using?
>>>> I am guessing that the code above will work with a file of
>>>> any size?
>>>> If that is the case, then you solved my problem.
>>>> The only Samba protocol that I am aware of is smb.
>>>>
>>>>>
>>>>> --
>>>>> HLS
>>> Joseph M. Newcomer [MVP]
>>> email: newcomer(a)flounder.com
>>> Web: http://www.flounder.com
>>> MVP Tips: http://www.flounder.com/mvp_tips.htm
>Joseph M. Newcomer [MVP]
>email: newcomer(a)flounder.com
>Web: http://www.flounder.com
>MVP Tips: http://www.flounder.com/mvp_tips.htm
Joseph M. Newcomer [MVP]
email: newcomer(a)flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on

"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message
news:OzySgEPlKHA.2132(a)TK2MSFTNGP05.phx.gbl...
> Peter Olcott wrote:
>
>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in
>> message news:%23OQCOfNlKHA.1824(a)TK2MSFTNGP04.phx.gbl...
>>> Peter Olcott wrote:
>>>
>>> By File Copy, you mean DOS copy command or the
>>> CopyFile() API?
>
> >
>
>> I am using the DOS command prompt's copy command. This
>> is fast.
>>
>>
>> The problem is the contradiction formed by the fact that
>> reading and writng the file is fast, while reading and
>> not wrting this same file is slow.
>> I am currently using fopen() and fread(); I am using
>> Windows XP.
>
> True, if the DOS copy command is fast,then I believe the
> code you are using is not optimal. The DOS Copy is using
> the same CreateFile() API which fopen() also finally uses
> in the RTL. So you should be able to match the same
> performance of the DOS Copy command.
>
> Have you tried using setvbuf to set a buffer cache?
>
> Here is a small test code that opens a 50 meg file:
>
> // File: V:\wc7beta\testbufsize.cpp
> // Compile with: cl testbufsize.cpp
>
> #include <stdio.h>
> #include <windows.h>
>
> void main(char argc, char *argv[])
> {
> char _cache[1024*16] = {0}; // 16K cache
> BYTE buf[1024*1] = {0}; // 1K buffer

char _cache[1024*64] = {0}; // 64K cache
BYTE buf[1024*4] = {0}; // 4K buffer

These buffer sizes match the DOS copy speed, and provide the
best performance.

>
> FILE *fv = fopen("largefile.dat","rb");
> if (fv) {
> int res = setvbuf(fv, _cache, _IOFBF,
> sizeof(_cache));
> DWORD nTotal = 0;
> DWORD nDisks = 0;
> DWORD nLoops = 0;
> DWORD nStart = GetTickCount();
> while (!feof(fv)) {
> nLoops++;
> memset(&buf,sizeof(buf),0);
> int nRead = fread(buf,1,sizeof(buf),fv);
> nTotal +=nRead;
> if (nRead > 0 && !fv->_cnt) nDisks++;
> }
> fclose(fv);
> printf("Time: %d | Size: %d | Reads: %d | Disks:
> %d\n",
> GetTickCount()-nStart,
> nTotal,
> nLoops,
> nDisks);
> }
> }
>
> What this basically shows is the number of disk hits it
> makes
> by checking the fv->_cnt value. It shows that as long as
> the cache size is larger than the read buffer size, you
> get the same number of disk hits. I also spit out the
> milliseconds. Subsequent runs, of course, is faster since
> the OS API CreateFile() is used by the RTL in buffer mode.
>
> Also do you know what protocol you have Samba using?
>
>
> --
> HLS