Memory overcommit
in [Kernel]
|
Prev: [PATCH -v6 00/13] ftrace for MIPS
Next: [Bug #14372] ath5k wireless not working after suspend-resume - eeepc
From: Andrea Arcangeli on 30 Oct 2009 10:20 On Fri, Oct 30, 2009 at 02:53:33PM +0100, Vedran Furač wrote: > % free -m > total used free shared buffers cached > Mem: 3458 3429 29 0 102 1119 > -/+ buffers/cache: 2207 1251 > > There's plenty of memory available. Shouldn't cache be automatically > dropped (this question was in my original mail, hence the subject)? This is not about cache, cache amount is physical, this about virtual amount that can only go in ram or swap (at any later time, current time is irrelevant) vs "ram + swap". In short add more swap if you don't like overcommit and check grep Commit /proc/meminfo in case this is accounting bug... -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Vedran Furač on 30 Oct 2009 10:50 Andrea Arcangeli wrote: > On Fri, Oct 30, 2009 at 02:53:33PM +0100, Vedran Furač wrote: >> % free -m >> total used free shared buffers cached >> Mem: 3458 3429 29 0 102 1119 >> -/+ buffers/cache: 2207 1251 >> >> There's plenty of memory available. Shouldn't cache be automatically >> dropped (this question was in my original mail, hence the subject)? > > This is not about cache, cache amount is physical, this about > virtual amount that can only go in ram or swap (at any later time, > current time is irrelevant) vs "ram + swap". Oh... so this is because apps "reserve" (Committed_AS?) more then they currently need. > In short add more swap if > you don't like overcommit and check grep Commit /proc/meminfo in case > this is accounting bug... A the time of "malloc: Cannot allocate memory": CommitLimit: 3364440 kB Committed_AS: 3240200 kB So probably everything is ok (and free is misleading). Overcommit is unfortunately necessary if I want to be able to use all my memory. Btw. http://www.redhat.com/advice/tips/meminfo.html says Committed_AS is a (gu)estimate. Hope it is a good (not to high) guesstimate. :) Regards, Vedran -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Vedran Furač on 30 Oct 2009 11:20 Thomas Fjellstrom wrote: >> malloc: Cannot allocate memory /* Great, no OOM, but: */ >> >> % free -m total used free shared buffers cached >> Mem: 3458 3429 29 0 102 1119 >> -/+ buffers/cache: 2207 1251 >> >> There's plenty of memory available. Shouldn't cache be >> automatically dropped (this question was in my original mail, hence >> the subject)? >> > > I think this is the MOST serious issue related to the oom killer. For > some reason it refuses to drop pages before trying to kill. When it > should drop cache, THEN kill if needed. This isn't about OOM, but situation when you turn off overcommit. I was jumping to conclusion here. You can drop caches manually with: # echo 1 > /proc/sys/vm/drop_caches but you still get: "malloc: Cannot allocate memory" even if almost nothing is cached: total used free shared buffers cached Mem: 3458 2210 1248 0 3 90 -/+ buffers/cache: 2116 1342 As for not dropping pages by kernel before killing, I don't know nothing about it. It happens so fast and I never tried to measure it. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andrea Arcangeli on 30 Oct 2009 11:20 On Fri, Oct 30, 2009 at 03:41:12PM +0100, Vedran Furač wrote: > Oh... so this is because apps "reserve" (Committed_AS?) more then they > currently need. They don't actually reserve, they end up "reserving" if overcommit is set to 2 (OVERCOMMIT_NEVER)... Apps aren't reserving, more likely they simply avoid a flood of mmap when a single one is enough to map an huge MAP_PRIVATE region like shared libs that you may only execute partially (this is why total_vm is usually much bigger than real ram mapped by pagetables represented in rss). But those shared libs are 99% pageable and they don't need to stay in swap or ram, so overcommit-as greatly overstimates the actual needs even if shared lib loading wouldn't be 64bit optimized (i.e. large and a single one). > A the time of "malloc: Cannot allocate memory": > > CommitLimit: 3364440 kB > Committed_AS: 3240200 kB > > So probably everything is ok (and free is misleading). Overcommit is > unfortunately necessary if I want to be able to use all my memory. Add more swap. > Btw. http://www.redhat.com/advice/tips/meminfo.html says Committed_AS is > a (gu)estimate. Hope it is a good (not to high) guesstimate. :) It is a guess in the sense to guarantee no ENOMEM it has to take into account the worst possible case, that is all shared lib MAP_PRIVATE mappings are cowed, which is very far from reality. Other than that the overcommitas should exactly match all mmapped possibly writeable space that can only fit in ram+swap, so from that point of view it's not a guessed number (modulo the smp read out of order). The only guess is how much slab, cache and other stuff is freeable, which doesn't provide true perfection to OVERCOMMIT_NEVER. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Hugh Dickins on 30 Oct 2009 12:30
On Fri, 30 Oct 2009, Andrea Arcangeli wrote: > > It is a guess in the sense to guarantee no ENOMEM it has to take into > account the worst possible case, that is all shared lib MAP_PRIVATE > mappings are cowed, which is very far from reality. A MAP_PRIVATE area is only counted into Committed_AS when it is or has in the past been PROT_WRITE. I think it's up to the ELF header of the shared library whether a section is PROT_WRITE or not; but it looks like many are not, so Committed_AS should be (a little) nearer reality than you fear. Though we do account for Committed_AS, even while allowing overcommit, we do not at present account for Committed_AS per mm. Seeing David and KAMEZAWA-san debating over total_vm versus rss versus anon_rss, I wonder whether such a "commit" count might be a better measure for OOM choices (but shmem is as usual awkward: though accounted just once in Committed_AS, it would probably have to be accounted to every mm that maps it). Just an idea to throw into the mix. Hugh -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |