Heap management?
in [Win32 API]
|
Prev: WMI compiing problem
Next: linker problem: __iob
From: Alf P. Steinbach on 24 Feb 2006 17:25 * rivenburgh(a)gmail.com: > Unfortunately no, the data needs to be read from a file and processed a > bit. All of this data management is actually in an external library > that I don't want to modify too much. Swapping out "new" for HeapAlloc > sounds pretty manageable. > > Thanks for the idea, though! Not sure: are your requirements for processing are to allocate and deallocate a lot of small chunks? Otherwise, when you call MapViewOfFile you get a memory pointer that you can pass to your external library. -- A: Because it messes up the order in which people normally read text. Q: Why is it such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail?
From: rivenburgh on 24 Feb 2006 17:31 No, the processing involved doesn't allocate and deallocate small chunks. It pretty much allocates the one big chunk and reads the data from the file, processing the input and filling the memory. The library just wants a file name and handles all of the memory management itself, so it doesn't really lend itself to those kinds of modifications.... Thanks, Reid
From: Scott McPhillips [MVP] on 24 Feb 2006 17:45 rivenburgh(a)gmail.com wrote: > The ideal thing for me would be if the OS memory manager gave me only > what I ask for (e.g. 250 MB) and used no more, and did as good a job as > possible of minimizing memory fragmentation, improving my chances for > finding a contiguous chunk of memory of that size.... > > Reid > The heap manager can't do anything useful about memory fragmentation in such a case. > Does anyone have any suggestions for maximum > efficiency when allocating huge chunks of memory? The exact amount > I'll need isn't something I know ahead of time, unfortunately, so I > can't just do something like grab 1.5 GB when the program starts. > (Well, I guess I could, but that might be unnecessarily greedy in some > cases.) No, it's not greedy. Grabbing everything you can get ahead of time is stealing nothing except your own app's memory space. It does not impact other apps until and unless you actually access it. If you do your own suballocation of the large data sets from a huge buffer like this you have the power to do something about fragmentation, because you can move arrays if necessary. -- Scott McPhillips [VC++ MVP]
From: Reid Rivenburgh on 24 Feb 2006 18:16 "Scott McPhillips [MVP]" <org-dot-mvps-at-scottmcp> writes: > rivenburgh(a)gmail.com wrote: >> The ideal thing for me would be if the OS memory manager gave me only >> what I ask for (e.g. 250 MB) and used no more, and did as good a job as >> possible of minimizing memory fragmentation, improving my chances for >> finding a contiguous chunk of memory of that size.... > > The heap manager can't do anything useful about memory fragmentation > in such a case. Well, I just meant in general that it would be nice if fragmentation could be minimized as much as possible. Put the small chunks together, for example, freeing up more big spaces. > > Does anyone have any suggestions for maximum > > efficiency when allocating huge chunks of memory? The exact amount > > I'll need isn't something I know ahead of time, unfortunately, so I > > can't just do something like grab 1.5 GB when the program starts. > > (Well, I guess I could, but that might be unnecessarily greedy in some > > cases.) > > No, it's not greedy. Grabbing everything you can get ahead of time > is stealing nothing except your own app's memory space. It does not > impact other apps until and unless you actually access it. If you > do your own suballocation of the large data sets from a huge buffer > like this you have the power to do something about fragmentation, > because you can move arrays if necessary. Hm, interesting, I wasn't looking at it like that. It looks to me in Memory Validator, however, as if HeapCreate tries to grab the chunk of memory requested and call it "commit", even before trying to do anything with it. I figured the heap space would be unusable by other apps. Maybe I'm misunderstanding you; are you talking about using C++ "new" to grab 1.5 GB and managing that space in the app, NOT using HeapCreate? That I can believe, though trying to modify the library I'm using to work that way would be a big hassle. That brings me back to my original question: Even when I used HeapCreate and HeapAlloc in a dummy program, it looked as if the HeapAlloc space wasn't being taken from the heap I created. Is that possible? If I grab 1.5 GB of heap for use by my datasets, it'd be kind of pointless if the actual memory allocated comes from somewhere else.... Thanks, Reid
From: Alf P. Steinbach on 24 Feb 2006 18:23
* rivenburgh(a)gmail.com: > No, the processing involved doesn't allocate and deallocate small > chunks. It pretty much allocates the one big chunk and reads the data > from the file, processing the input and filling the memory. The > library just wants a file name and handles all of the memory management > itself, so it doesn't really lend itself to those kinds of > modifications.... I don't understand what the problem is. If you can replace 'new' with HeapAlloc, and there is essentially one big chunk of memory that's the problem, why can't you replace 'new' with MapViewOfFile? They both yield memory pointers, and the use of those pointers are the same, no difference. -- A: Because it messes up the order in which people normally read text. Q: Why is it such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet and in e-mail? |