Heavy malloc()/free() load and threads
in [Unix Programming]
|
From: Ian Collins on 11 Mar 2010 15:02 On 03/12/10 07:44 AM, Rainer Weikusat wrote: > Chris Friesen<cbf123(a)mail.usask.ca> writes: >> On 03/11/2010 08:01 AM, Nicolas George wrote: >>> Urs Thuermann wrote in message >>> <ygfzl2fxh5k.fsf(a)janus.isnogud.escape.de>: >>>> 1. Will typical implementations of malloc()/free() in libc handle this >>>> load well? Or should I implement my own memory management? >>> >>> Code it, debug it, and test it. If you have performances problems, profile. >>> If the profiling shows that memory allocation is the bottleneck, then you >>> can think about changing it. >> >> Absolutely. There's no point messing with it if it's already fast enough. > > Except that 'it' doesn't presently exist. And once 'it' has been > written with ad hoc usage of the malloc-interface in mind, the amount > of work which was necessary to change to another approach will likely > be prohibitive. Considering the other approach is likely to be use an alternative allocator, the amount of work could be minimal; link a different library. -- Ian Collins
From: Rainer Weikusat on 11 Mar 2010 15:32 Ian Collins <ian-news(a)hotmail.com> writes: > On 03/12/10 07:44 AM, Rainer Weikusat wrote: >> Chris Friesen<cbf123(a)mail.usask.ca> writes: >>> On 03/11/2010 08:01 AM, Nicolas George wrote: >>>> Urs Thuermann wrote in message >>>> <ygfzl2fxh5k.fsf(a)janus.isnogud.escape.de>: >>>>> 1. Will typical implementations of malloc()/free() in libc handle this >>>>> load well? Or should I implement my own memory management? >>>> >>>> Code it, debug it, and test it. If you have performances problems, profile. >>>> If the profiling shows that memory allocation is the bottleneck, then you >>>> can think about changing it. >>> >>> Absolutely. There's no point messing with it if it's already fast enough. >> >> Except that 'it' doesn't presently exist. And once 'it' has been >> written with ad hoc usage of the malloc-interface in mind, the amount >> of work which was necessary to change to another approach will likely >> be prohibitive. > > Considering the other approach is likely to be use an alternative > allocator, the amount of work could be minimal; link a different > library. I wrote 'written with ad hoc usage of the malloc-interface in mind' for a reason. And this 'approach' doesn't change when swapping malloc implementations. An interesting, related question would be: How to determine which 'other malloc' to use because of what reasons, eg, who is going to read through all the code and understand its behaviour fully enough to make an educated judgement in this respect? The guy who desired to avoid writing the less-than-200-LOC in the first place? Fat chance ...
From: Andrew Poelstra on 11 Mar 2010 15:42 On 2010-03-11, Rainer Weikusat <rweikusat(a)mssgmbh.com> wrote: > > I wrote 'written with ad hoc usage of the malloc-interface in mind' > for a reason. And this 'approach' doesn't change when swapping > malloc implementations. An interesting, related question would be: How > to determine which 'other malloc' to use because of what reasons, eg, > who is going to read through all the code and understand its behaviour > fully enough to make an educated judgement in this respect? The guy > who desired to avoid writing the less-than-200-LOC in the first place? > Fat chance ... > Writing less than 200 LOC is not only costly up front in terms of extra coding effort, but also requires testing and maintenance, possibly refactoring and documentation, and is likely to be a blight on your otherwise-unrelated-to-memory-management source. Whereas looking through a few memory allocators to find out which one is good at rapidfire alloc/free of tiny chunks of memory, requires a one-time lookup and one-time comment. -- Andrew Poelstra http://www.wpsoftware.net/andrew
From: Rainer Weikusat on 11 Mar 2010 16:10 Andrew Poelstra <apoelstra(a)localhost.localdomain> writes: > On 2010-03-11, Rainer Weikusat <rweikusat(a)mssgmbh.com> wrote: >> I wrote 'written with ad hoc usage of the malloc-interface in mind' >> for a reason. And this 'approach' doesn't change when swapping >> malloc implementations. An interesting, related question would be: How >> to determine which 'other malloc' to use because of what reasons, eg, >> who is going to read through all the code and understand its behaviour >> fully enough to make an educated judgement in this respect? The guy >> who desired to avoid writing the less-than-200-LOC in the first place? >> Fat chance ... > > Writing less than 200 LOC is not only costly up front in terms of > extra coding effort, but also requires testing and maintenance, > possibly refactoring and documentation, and is likely to be a > blight on your otherwise-unrelated-to-memory-management source. 'Writing less than 200 LOC' is something like half a days worth of work (in extreme cases, most I had to deal with so far were much simpler than that). Since the code is going to be used by the other parts of the program, separate testing isn't really necessary. It presumably wouldn't hurt to spend something like half an hour on that, too. 'Maintenance' means 'making code changes'. I do not quite understand which type of code changes might be required here. So far, I only ever had to 'maintain' infrastructure code if I got the initial design wrong completely. Which usually doesn't hapen. If it does, that's a one time effort (IIRC, I did do this exactly once in the last 6.x years). > Whereas looking through a few memory allocators to find out > which one is good at rapidfire alloc/free of tiny chunks of > memory, requires a one-time lookup and one-time comment. As I already wrote above: 'Linking another few thousands or even tenthousands lines of code to something' is only as simple as that if the 'serious effort in ensuring that it actually works well' you consider to be unavoidable even for really tiny amounts of code, as you wrote above, is simply skipped. Your universe must be a very magical place if you of all programmers are the only one which ever makes programming errors.
From: Ian Collins on 11 Mar 2010 16:18
On 03/12/10 09:32 AM, Rainer Weikusat wrote: > Ian Collins<ian-news(a)hotmail.com> writes: >> On 03/12/10 07:44 AM, Rainer Weikusat wrote: >>> Chris Friesen<cbf123(a)mail.usask.ca> writes: >>>> On 03/11/2010 08:01 AM, Nicolas George wrote: >>>>> Urs Thuermann wrote in message >>>>> <ygfzl2fxh5k.fsf(a)janus.isnogud.escape.de>: >>>>>> 1. Will typical implementations of malloc()/free() in libc handle this >>>>>> load well? Or should I implement my own memory management? >>>>> >>>>> Code it, debug it, and test it. If you have performances problems, profile. >>>>> If the profiling shows that memory allocation is the bottleneck, then you >>>>> can think about changing it. >>>> >>>> Absolutely. There's no point messing with it if it's already fast enough. >>> >>> Except that 'it' doesn't presently exist. And once 'it' has been >>> written with ad hoc usage of the malloc-interface in mind, the amount >>> of work which was necessary to change to another approach will likely >>> be prohibitive. >> >> Considering the other approach is likely to be use an alternative >> allocator, the amount of work could be minimal; link a different >> library. > > I wrote 'written with ad hoc usage of the malloc-interface in mind' > for a reason. And this 'approach' doesn't change when swapping > malloc implementations. An interesting, related question would be: How > to determine which 'other malloc' to use because of what reasons, eg, > who is going to read through all the code and understand its behaviour > fully enough to make an educated judgement in this respect? The guy > who desired to avoid writing the less-than-200-LOC in the first place? > Fat chance ... I doubt anyone would. I've written numerous allocators, but I can't say I've studied the code of many others (which may not be open). Programmers being pragmatists with a deadline are more likely to profile their application with different libraries and observe the results. They (we) trust the skills of the library implementers, who will have spent considerable time refining their algorithms and code. By the way, what are those "less-than-200-LOC"? -- Ian Collins |