Transparent Hugepage Support #29 [Kernel]

Prev: drivers/net/bfin_mac.c: Misc function cleanups, neatening
Next: [GIT PULL] x86 platform drivers for 2.6.36

From: Andrea Arcangeli on 3 Aug 2010 10:00

From: Ed Tomlinson on 4 Aug 2010 08:20

Andrea,

I have been tracking & testing this for a while. It looks very much like Linus wants to merge (or a big chunk of it):

git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin.git vfs-scale-working

When I pull an aa-29 on top of this there is one conflict in vmscan.c that git cannot resolve.
It would be really nice to have a version of the THS patchs that can be tested with the vfs scaling
tree.

TIA
Ed Tomlinson

On Tuesday 03 August 2010 09:56:15 you wrote:
> http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=shortlog
>
> first: git clone git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git
> or first: git clone --reference linux-2.6 git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git
> later: git fetch; git checkout -f origin/master
>
> The tree is rebased and git pull won't work.
>
> http://www.kernel.org/pub/linux/kernel/people/andrea/patches/v2.6/2.6.35/transparent_hugepage-29/
> http://www.kernel.org/pub/linux/kernel/people/andrea/patches/v2.6/2.6.35/transparent_hugepage-29.gz
>
> Diff #28 -> #29:
>
> THP-disable-on-small-systems | 41 ++++++++
>
> Disable THP on <500M systems, from Rik.
>
> compaction-kswapd | 194 ++++++++++++++++++++++++++++++++++++++++
>
> Add a compaction mode for kswapd to fully replace obsolete blind lumpy
> reclaim.
>
> free_pages-count | 60 ++++++++++++
> free_pages-drain_all_pages | 62 ++++++++++++
> free_pages-vmstat | 156 ++++++++++++++++++++++++++++++++
>
> Make free page statistics more accurate from Mel.
>
> ksmd-khugepaged-freeze | 97 ++++++++++++++++++++
>
> kswapd uses set_freezing, so shall ksmd and khugepaged.
>
> transparent-hugepage-nr_rotated | 36 +++++++
> transparent-hugepage-stat | 166 ++++++++++++++++++++++++++++++++++
>
> Fix inactive/active stats and nr_rotated, from Rik.
>
> Documentation/vm/transhuge.txt | 283 ++++
> arch/alpha/include/asm/mman.h | 2
> arch/mips/include/asm/mman.h | 2
> arch/parisc/include/asm/mman.h | 2
> arch/powerpc/mm/gup.c | 12
> arch/x86/include/asm/kvm_host.h | 1
> arch/x86/include/asm/paravirt.h | 23
> arch/x86/include/asm/paravirt_types.h | 6
> arch/x86/include/asm/pgtable-2level.h | 9
> arch/x86/include/asm/pgtable-3level.h | 23
> arch/x86/include/asm/pgtable.h | 149 ++
> arch/x86/include/asm/pgtable_64.h | 28
> arch/x86/include/asm/pgtable_types.h | 3
> arch/x86/kernel/paravirt.c | 3
> arch/x86/kernel/vm86_32.c | 1
> arch/x86/kvm/mmu.c | 68 -
> arch/x86/kvm/paging_tmpl.h | 6
> arch/x86/mm/gup.c | 28
> arch/x86/mm/pgtable.c | 66 +
> arch/xtensa/include/asm/mman.h | 2
> fs/Kconfig | 2
> fs/exec.c | 44
> fs/proc/meminfo.c | 14
> fs/proc/page.c | 14
> include/asm-generic/mman-common.h | 2
> include/asm-generic/pgtable.h | 130 +
> include/linux/compaction.h | 13
> include/linux/gfp.h | 14
> include/linux/huge_mm.h | 151 ++
> include/linux/khugepaged.h | 66 +
> include/linux/ksm.h | 20
> include/linux/kvm_host.h | 4
> include/linux/memory_hotplug.h | 14
> include/linux/mm.h | 114 +
> include/linux/mm_inline.h | 19
> include/linux/mm_types.h | 3
> include/linux/mmu_notifier.h | 66 +
> include/linux/mmzone.h | 1
> include/linux/page-flags.h | 36
> include/linux/res_counter.h | 12
> include/linux/rmap.h | 33
> include/linux/sched.h | 1
> include/linux/swap.h | 2
> include/linux/vmstat.h | 4
> kernel/fork.c | 12
> kernel/futex.c | 55
> mm/Kconfig | 38
> mm/Makefile | 1
> mm/compaction.c | 48
> mm/huge_memory.c | 2212 ++++++++++++++++++++++++++++++++++
> mm/hugetlb.c | 69 -
> mm/ksm.c | 86 -
> mm/madvise.c | 8
> mm/memcontrol.c | 169 +-
> mm/memory-failure.c | 2
> mm/memory.c | 241 +++
> mm/memory_hotplug.c | 14
> mm/mempolicy.c | 14
> mm/migrate.c | 22
> mm/mincore.c | 7
> mm/mmap.c | 57
> mm/mmu_notifier.c | 20
> mm/mprotect.c | 20
> mm/mremap.c | 8
> mm/oom_kill.c | 1
> mm/page_alloc.c | 58
> mm/pagewalk.c | 1
> mm/rmap.c | 228 ++-
> mm/sparse.c | 4
> mm/swap.c | 117 +
> mm/swap_state.c | 6
> mm/swapfile.c | 2
> mm/vmscan.c | 102 -
> mm/vmstat.c | 31
> virt/kvm/iommu.c | 2
> virt/kvm/kvm_main.c | 56
> 76 files changed, 4680 insertions(+), 527 deletions(-)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo(a)vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

From: Andrea Arcangeli on 4 Aug 2010 08:50

Hi Ed,

On Wed, Aug 04, 2010 at 08:15:02AM -0400, Ed Tomlinson wrote:
> Andrea,
>
> I have been tracking & testing this for a while. It looks very much like Linus wants to merge (or a big chunk of it):
>
> git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin.git vfs-scale-working
>
> When I pull an aa-29 on top of this there is one conflict in vmscan.c that git cannot resolve.
> It would be really nice to have a version of the THS patchs that can be tested with the vfs scaling
> tree.

please try:

zcat transparent-hugepage-29.gz |patch -p1

It didn't spawn rejects here, maybe something's wrong in some
intra-patch or git is too picky. If it doesn't build let me know and I
will look into it, thanks!

Andrea
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

From: Ed Tomlinson on 5 Aug 2010 07:30

On Wednesday 04 August 2010 08:44:42 Andrea Arcangeli wrote:
> Hi Ed,
>
> On Wed, Aug 04, 2010 at 08:15:02AM -0400, Ed Tomlinson wrote:
> > Andrea,
> >
> > I have been tracking & testing this for a while. It looks very much like Linus wants to merge (or a big chunk of it):
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin.git vfs-scale-working
> >
> > When I pull an aa-29 on top of this there is one conflict in vmscan.c that git cannot resolve.
> > It would be really nice to have a version of the THS patchs that can be tested with the vfs scaling
> > tree.
>
> please try:
>
> zcat transparent-hugepage-29.gz |patch -p1
>
> It didn't spawn rejects here, maybe something's wrong in some
> intra-patch or git is too picky. If it doesn't build let me know and I
> will look into it, thanks!

On a tree with:

commit 9d9cec888c3c4a2d5b5f01c3f3984fc4bc602681
Merge: dd6c927 9edd35f
Author: Ed <edt(a)aei.ca>
Date: Wed Aug 4 15:18:53 2010 -0400

Merge branch 'vfs-scale-working' of git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin into vfs-scaling

merged on top of

commit 9fe6206f400646a2322096b56c59891d530e8d51
Author: Linus Torvalds <torvalds(a)linux-foundation.org>
Date: Sun Aug 1 15:11:14 2010 -0700

Linux 2.6.35

doing: bzcat ../transparent-hugepage-29.bz2 | patch -p1

I get one reject:

patching file mm/vmscan.c
Hunk #3 succeeded at 612 (offset 43 lines).
Hunk #4 succeeded at 968 (offset 43 lines).
Hunk #5 succeeded at 1026 (offset 43 lines).
Hunk #6 succeeded at 1063 (offset 43 lines).
Hunk #7 succeeded at 1168 (offset 43 lines).
Hunk #8 succeeded at 1179 (offset 43 lines).
Hunk #9 succeeded at 1198 (offset 43 lines).
Hunk #10 succeeded at 1236 (offset 43 lines).
Hunk #11 succeeded at 1263 (offset 43 lines).
Hunk #12 succeeded at 1333 (offset 43 lines).
Hunk #13 succeeded at 1401 (offset 43 lines).
Hunk #14 succeeded at 1658 (offset 43 lines).
Hunk #15 FAILED at 1629.
Hunk #16 succeeded at 2112 with fuzz 2 (offset 51 lines).
Hunk #17 FAILED at 2093.
Hunk #18 succeeded at 2169 (offset 46 lines).
2 out of 18 hunks FAILED -- saving rejects to file mm/vmscan.c.rej
patching file mm/vmstat.c

---
cat mm/vmscan.c.rej
--- mm/vmscan.c
+++ mm/vmscan.c
@@ -1629,8 +1586,6 @@

get_scan_count(zone, sc, nr, priority);

- set_lumpy_reclaim_mode(priority, sc);
-
while (nr[LRU_INACTIVE_ANON] || nr[LRU_ACTIVE_FILE] ||
nr[LRU_INACTIVE_FILE]) {
for_each_evictable_lru(l) {
@@ -2093,9 +2049,25 @@
lru_pages);
sc.nr_reclaimed += reclaim_state->reclaimed_slab;
total_scanned += sc.nr_scanned;
+ compaction = 0;
+
+ if (order &&
+ zone_watermark_ok(zone, 0,
+ high_wmark_pages(zone),
+ end_zone, 0) &&
+ !zone_watermark_ok(zone, order,
+ high_wmark_pages(zone),
+ end_zone, 0)) {
+ compact_zone_order(zone,
+ order,
+ sc.gfp_mask,
+ COMPACT_MODE_KSWAPD);
+ compaction = 1;
+ }
+
if (zone->all_unreclaimable)
continue;
- if (nr_slab == 0 &&
+ if (!compaction && nr_slab == 0 &&
zone->pages_scanned >= (zone_reclaimable_pages(zone) * 6))
zone->all_unreclaimable = 1;
/*

--

Ideas?
Ed Tomlinson
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

From: Andrea Arcangeli on 5 Aug 2010 07:50

On Thu, Aug 05, 2010 at 07:27:24AM -0400, Ed Tomlinson wrote:
> On a tree with:
>
> commit 9d9cec888c3c4a2d5b5f01c3f3984fc4bc602681
> Merge: dd6c927 9edd35f
> Author: Ed <edt(a)aei.ca>
> Date: Wed Aug 4 15:18:53 2010 -0400
>
> Merge branch 'vfs-scale-working' of git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin into vfs-scaling
>
> merged on top of
>
> commit 9fe6206f400646a2322096b56c59891d530e8d51
> Author: Linus Torvalds <torvalds(a)linux-foundation.org>
> Date: Sun Aug 1 15:11:14 2010 -0700
>
> Linux 2.6.35

Well I didn't realize you also merged 2.6.35 on top of it. This is no
standard tree, so I suggest you to checkout -f origin/master of Nick's
tree until he updates to 2.6.35 himself (then I can port to it).
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

| Next | Last
Pages: 1 2
Prev: drivers/net/bfin_mac.c: Misc function cleanups, neatening
Next: [GIT PULL] x86 platform drivers for 2.6.36