Heap management?
in [Win32 API]
|
Prev: WMI compiing problem
Next: linker problem: __iob
From: rivenburgh on 24 Feb 2006 14:48 Hi. I have an application in which I need to dynamically allocate multiple instances of large amounts of memory (used to store large datasets). They can vary in size, but 250 MB is typical. On a machine with 2 GB of memory, I can generally read in three of them before I get an out of memory error. The app is written in C++ and uses standard "new" to allocate the memory. I have Memory Validator, a nice tool along the lines of Purify. It lets me see the memory space as my program runs. It looks like a single 250 MB allocation actually tries to allocate double that behind the scenes, for efficiency's sake. At some seemingly premature point, I guess because of memory fragmentation, it fails. (Not much else is running on the machine at the time.) I've been told that instead of "new" I should use Windows routines like HeapCreate and HeapAlloc to prevent it from trying to allocate so much memory. I've been playing around with them in a dummy program, and I'm seeing something strange. If I create a growable, 200 MB heap space with HeapCreate, and then I try to allocate 20 MB of space from that heap using HeapAlloc, Memory Validator shows that the 20 MB is being taken from the free space OUTSIDE my 200 MB heap. My question, then, is whether HeapAlloc is guaranteed to use the space from the specified heap. Is it possible that the tool I'm using to watch what's going on in memory is just getting it wrong? I'm also afraid that even HeapCreate/Alloc are temporarily using more than the exact amount of memory I need and failing prematurely (double again, in fact). Does anyone have any suggestions for maximum efficiency when allocating huge chunks of memory? The exact amount I'll need isn't something I know ahead of time, unfortunately, so I can't just do something like grab 1.5 GB when the program starts. (Well, I guess I could, but that might be unnecessarily greedy in some cases.) I understand fragmentation is an unpredictable issue. Also, I can't use anything like windbg in this environment; just what comes with Visual Studio and the Memory Validator tool. I'm running Windows 2000 SP4. I may move to XP someday.... Thanks for any info! I'm obviously something of a novice when it comes to low-level memory management.... Thanks, Reid P.S. I have no connection with the Memory Validator folks; they've been very helpful and it seems like a nice tool. http://www.softwareverify.com/memoryValidator/
From: Kellie Fitton on 24 Feb 2006 15:13 Hi, Well, use the following APIs to get some clues on how much actual memory, the virtual memory manager wants you to use within your application program: GetProcessWorkingSetSizeEx() SetProcessWorkingSetSizeEx() http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/getprocessworkingsetsizeex.asp http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/setprocessworkingsetsizeex.asp Hope these information helps, Kellie.
From: Alf P. Steinbach on 24 Feb 2006 15:15 * rivenburgh(a)gmail.com: > Hi. I have an application in which I need to dynamically allocate > multiple instances of large amounts of memory (used to store large > datasets). They can vary in size, but 250 MB is typical. On a machine > with 2 GB of memory, I can generally read in three of them before I get > an out of memory error. Is it possible for you to use memory-mapped files? -- A: Because it messes up the order in which people normally read text. Q: Why is it such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?
From: Grzegorz on 24 Feb 2006 15:26 rivenburgh(a)gmail.com wrote: > Hi. I have an application in which I need to dynamically allocate > multiple instances of large amounts of memory (used to store large > datasets). They can vary in size, but 250 MB is typical. On a machine > with 2 GB of memory, I can generally read in three of them before I get > an out of memory error. The app is written in C++ and uses standard > "new" to allocate the memory. I have Memory Validator, a nice tool > along the lines of Purify. It lets me see the memory space as my > program runs. It looks like a single 250 MB allocation actually tries > to allocate double that behind the scenes, for efficiency's sake. At You kidding. What kind of efficiency would be that? You should have no problem with allocating 2gb memory, if you tried more it could be slow because of page swapping. Show us the code you must be doing something wrong. Memory fragmentation shouldn't be a problem, you deal with virtual address space and os cares about translating it into physical one. You might want to check whether your machine is not configured that way (server mode) it leaves only 1GB address space for applications, reserving 3 upper GB for system usage. -- 677265676F727940346E6575726F6E732E636F6D
From: Stephen Kellett on 24 Feb 2006 15:54
In message <dtnpjf$p4v$1(a)atlantis.news.tpi.pl>, Grzegorz Wr?bel </dev/null(a)localhost.localdomain> writes >You kidding. What kind of efficiency would be that? >You should have no problem with allocating 2gb memory, You are mistaken. Try this code. char *p = new char [1024 * 1024 * 1024 * 2]; Guaranteed to fail and return NULL on Windows NT/W2K/XP Workstation. >Memory fragmentation shouldn't be a problem, you deal with virtual >address space and os cares about translating it into physical one. Correct, but it needs to find a 2GB contiguous memory space. You won't find a space that large on Windows NT/2K/XP (you may on a /3GB machine). On a non /3GB machine 2GB is the max space for the DLLs that form your application, the C heap, program stack and two 4KB guard pages at 0x00000000 and 0x7ffff000 and the workspace from which you wish to allocate memory. By definition there is not 2GB contiguous space available in that 2GB block as some of it is already used. You can check that with VM Validator (free) or Memory Validator's virtual tab which graphically shows you the memory space. You can find these tools at http://www.softwareverify.com Stephen -- Stephen Kellett Object Media Limited http://www.objmedia.demon.co.uk/software.html Computer Consultancy, Software Development Windows C++, Java, Assembler, Performance Analysis, Troubleshooting |