Advice on how to keep track of allocated memory
in [Fortran]
|
Prev: Summing array elements
Next: comp.lang.fortran charter
From: Lorents on 20 Apr 2008 08:50 I'd like to ask for a piece of advice. I would like, every time I allocate or deallocate arrays: 1) To check allocation or deallocation flag and to stop the program printing a useful error message is there is an error 2) To print the amount of memory allocated/deallocated and the total system memory used 3) To do these things in the neatest way possible :-) In a preliminary test I did something like this: !======================= (...) allocate( xd(npot), stat=alloc_status) array_name = "xd(npot)" call mem_usage("A", "R", real(npot, fp), alloc_status) (...) subroutine mem_usage(action, type, size, alloc_status) ! action="A"=> allocate, ="D" deallocate ! type = "R" = Real, "I" integer ! size = number of elements in the array (check that action and type are OK; check IOSTATUS) if( action == "A") then write(unit=istdout, fmt='(A, F10.3, A)', advance="no") "Successfully allocated ", array_memory, ' MB of ram. ' total_memory = total_memory + array_memory write(unit=istdout, fmt='(A, F10.3, A)', advance="yes") "TOTAL allocated memory is now", total_memory, ' MB.' else write(unit=istdout, fmt='(A, F10.3, A)', advance="no") "Successfully deallocated ", array_memory, ' MB of ram. ' total_memory = total_memory - array_memory write(unit=istdout, fmt='(A, F10.3, A)', advance="yes") "TOTAL allocated memory is now", total_memory, ' MB.' end if !======================= but it doesn't look very elegant to me... Anyone can suggest a general way of keeping track of these things ? Lorenzo
From: David Flower on 20 Apr 2008 12:11 On Apr 20, 1:50�pm, Lorents <lore...(a)amp.te> wrote: > I'd like to ask for a piece of advice. > I would like, every time I allocate or deallocate arrays: > > 1) To check allocation or deallocation flag and to stop the program > printing a useful error message is there is an error > 2) To print the amount of memory allocated/deallocated and the total > system memory used > 3) To do these things in the neatest way possible :-) > > In a preliminary test I did something like this: > > !======================= > (...) > allocate( xd(npot), stat=alloc_status) > array_name = "xd(npot)" > call mem_usage("A", "R", real(npot, fp), alloc_status) > (...) > > subroutine mem_usage(action, type, size, alloc_status) > ! action="A"=> allocate, ="D" deallocate > ! type �= "R" = Real, "I" integer > ! size = number of elements in the array > > (check that action and type are OK; check IOSTATUS) > > if( action == "A") then > � write(unit=istdout, fmt='(A, F10.3, A)', advance="no") "Successfully > allocated ", array_memory, ' MB of ram. ' > � total_memory = total_memory + array_memory > � write(unit=istdout, fmt='(A, F10.3, A)', advance="yes") "TOTAL > allocated memory is now", total_memory, ' MB.' > else > � write(unit=istdout, fmt='(A, F10.3, A)', advance="no") "Successfully > deallocated ", array_memory, ' MB of ram. ' > � total_memory = total_memory - array_memory > � write(unit=istdout, fmt='(A, F10.3, A)', advance="yes") "TOTAL > allocated memory is now", total_memory, ' MB.' > end if > > !======================= > > but it doesn't look very elegant to me... Anyone can suggest a general > way of keeping track of these things ? > > Lorenzo I think that you are on the right track, but how about adding a unique identifier to each call to mem_usage, so that it is easy to identify an erronoeus call ? Dave Flower
From: Lorents on 20 Apr 2008 13:25 Lorents wrote: > I'd like to ask for a piece of advice. [...] Another possibility I am contemplating is is to create module containing subroutines "my_allocate" and "my_deallocate" that allocate, do the checks and print information if requested, something like: ================ subroutine my_allocate(A, N, M) implicit none double precision, allocatable :: A(:,:) (other declarations) allocate( A(N,M) , status=alloc_stat) if(alloc_status \= 0 ) then (manage error) end if memory_array = real(N)*real(M)/1.e6 write(*,*) "I succesfully allocated this amount of memory in MB: ", memory_array (...) ================ In this way it may be easier to keep track of everything and appear neater in the main program. Are there reasons why this could cause a performance penality? Lorenzo
From: Bil Kleb on 20 Apr 2008 14:31 Lorents wrote: > Another possibility I am contemplating is is to create > module containing subroutines "my_allocate" and "my_deallocate" > that allocate, do the checks and print information if requested, > something like: This is what we have done for FUN3D, but we have found that the checks didn't work reliably, we never got around to the "print information" part, and we found valgrind^1 more convenient for tracking and debugging memory use. IMHO, we failed to heed the YAGNI principle^2. Regards, -- Bil Kleb http://fun3d.larc.nasa.gov [1] http://en.wikipedia.org/wiki/Valgrind [2] http://en.wikipedia.org/wiki/YAGNI
From: Colin Watters on 20 Apr 2008 17:01
"Bil Kleb" <Bil.Kleb(a)NASA.gov> wrote in message news:fug25u$5b7$1(a)aioe.org... > Lorents wrote: >> Another possibility I am contemplating is is to create >> module containing subroutines "my_allocate" and "my_deallocate" >> that allocate, do the checks and print information if requested, >> something like: > > This is what we have done for FUN3D, but we have found that > the checks didn't work reliably, we never got around to > the "print information" part, and we found valgrind^1 more > convenient for tracking and debugging memory use. > (...snip) We considered doing the same. What stopped us was: (a) use of CVF 6.6b, where un-allocated allocatable arrays as arguments just don't work relyably (b) we couldn't figure out how to allocate an arbitrary structure in a general-purpose routine Eventually my colleague wrote a Perl script to add explicit CALL statements to bespoke logging routines at the correct points in the code to gather the required data for all allocate and deallocate statements. So every ALLOCATE statement gets followed by a call to LOG_ALLOCATE, arguments to which are (1) a unique string to identify it, (b) the SIZE or SIZEOF of the array or structure just allocated, (c) the LOC() of the array or structure. Every DEALLOCATE gets preceeded by a call to LOG_DEALLOCATE, with much the same arguments. We do not use STAT= in the allocate statements, we prefer a hard failure if anything goes wrong. Reason for all this is to trap memory leaks. Its amazing how tiny leaks mount up when people run your code for 2 days solid. The log_ routines use the address of the allocated array or structure (as provided by the LOC()) to match allocates and deallocates. Knowledge of what the code is trying to do then allows stuff that doesn't get deallocated to be flagged as "probable leaks". We dump a synopsis of "stuff still allocated that didn't ought to be" every hour or so to a text file, and this can be examined while the run is in progress, which with 2-day runs is pretty much essential. Using this technique we identified a number of leaks. Turns out repeated allocation of a pointer is a leak, I had assumed if it was already allocated the existing memory would be re-used. I also learnt that constructor routines should not allocate pointer components of new structures when being called from copy routines that blindly reallocate new pointer components. And so on (there are A LOT of ways to leak memory when using pointers!) Useful as the Perl script is, I can't help thinking there ought to be an easier way. Anyone out there got one? -- Qolin Email: my qname at domain dot com Domain: qomputing |