32GB SSD on USB1.1 P3/700 == ___HELL___ (2.6.34-rc3)
in [Kernel]
|
Prev: [PATCH 4/4] Staging: batman-adv: fix whitespace style issues
Next: 2.6.34-rc3: simple du (on a big xfs tree) triggers oom killer
From: Wu Fengguang on 7 Apr 2010 05:00 > >> active_anon:34886 inactive_anon:41460 isolated_anon:1 > >> active_file:13576 inactive_file:27884 isolated_file:65 > >> unevictable:0 dirty:4788 writeback:5675 unstable:0 > >> free:1198 slab_reclaimable:1952 slab_unreclaimable:2594 > >> mapped:10152 shmem:56 pagetables:742 bounce:0 > >> DMA free:2052kB min:84kB low:104kB high:124kB active_anon:940kB inactive_anon:3876kB active_file:212kB inactive_file:8224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15804kB mlocked:0kB dirty:3448kB writeback:752kB mapped:80kB shmem:0kB slab_reclaimable:160kB slab_unreclaimable:124kB kernel_stack:40kB pagetables:48kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:20096 all_unreclaimable? yes > >> lowmem_reserve[]: 0 492 492 > >> Normal free:2740kB min:2792kB low:3488kB high:4188kB active_anon:138604kB inactive_anon:161964kB active_file:54092kB inactive_file:103312kB unevictable:0kB isolated(anon):4kB isolated(file):260kB present:503848kB mlocked:0kB dirty:15704kB writeback:21948kB mapped:40528kB shmem:224kB slab_reclaimable:7648kB slab_unreclaimable:10252kB kernel_stack:1632kB pagetables:2920kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:73056 all_unreclaimable? no > >> lowmem_reserve[]: 0 0 0 > >> DMA: 513*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2052kB > >> Normal: 685*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2740kB > >> 56122 total pagecache pages > >> 14542 pages in swap cache > >> Swap cache stats: add 36404, delete 21862, find 8669/10118 > >> Free swap = 671696kB > >> Total swap = 755048kB > >> 131034 pages RAM > >> 3214 pages reserved > >> 94233 pages shared > >> 80751 pages non-shared > >> Out of memory: kill process 3462 (kdeinit4) score 95144 or a child > > > > shmem=56 is ignorable, and > > active_file+inactive_file=13576+27884=41460 < 56122 total pagecache pages. > > > > Where are the 14606 file pages gone? > > swapcache? Ah exactly! Thanks, Fengguang -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andreas Mohr on 7 Apr 2010 07:20 Hi, On Wed, Apr 07, 2010 at 03:00:50PM +0800, Wu Fengguang wrote: > Many applications (this one and below) are stuck in > wait_on_page_writeback(). I guess this is why "heavy write to > irrelevant partition stalls the whole system". They are stuck on page > allocation. Your 512MB system memory is a bit tight, so reclaim > pressure is a bit high, which triggers the wait-on-writeback logic. "Your 512MB system memory is a bit tight". Heh, try to survive making such a statement 15 years ago ;) (but you likely meant this in the context of inducing a whopping 300MB write) Thank you for your reply, I'll test the patch ASAP (with large writes and Firefox sync mixed in), maybe this will improve things already. Andreas Mohr -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andreas Mohr on 8 Apr 2010 15:50 On Wed, Apr 07, 2010 at 01:17:02PM +0200, Andreas Mohr wrote: > Thank you for your reply, I'll test the patch ASAP (with large writes > and Firefox sync mixed in), maybe this will improve things already. Indeed, AFAICS this definitely seems MUCH better than before. I had to do a full kernel rebuild (due to CONFIG_LOCALVERSION_AUTO changes; with no changed configs though). I threw some extra load into the mix (read 400MB instead of 300 through USB1.1, ran gimp, grepped over the entire /usr partition etc.pp.), so far not nearly as severe as before, and no OOMs either. Launched Firefox some time after starting 400MB creation, pretty ok still. Some annoying lags sometimes of course, but nothing absolutely earth-shattering as experienced before. Things really appear to be a LOT better. OK, so which way to go? Thanks a lot, Andreas Mohr -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Bill Davidsen on 8 Apr 2010 16:20 Andreas Mohr wrote: > [CC'd some lucky candidates] > > Hello, > > I was just running > mkfs.ext4 -b 4096 -E stride=128 -E stripe-width=128 -O ^has_journal > /dev/sdb2 > on my SSD18M connected via USB1.1, and the result was, well, > absolutely, positively _DEVASTATING_. > > The entire system became _FULLY_ unresponsive, not even switching back > down to tty1 via Ctrl-Alt-F1 worked (took 20 seconds for even this key > to be respected). > > Once back on ttys, invoking any command locked up for minutes > (note that I'm talking about attempted additional I/O to the _other_, > _unaffected_ main system HDD - such as loading some shell binaries -, > NOT the external SSD18M!!). > > Having an attempt at writing a 300M /dev/zero file to the SSD's filesystem > was even worse (again tons of unresponsiveness), combined with multiple > OOM conditions flying by (I/O to the main HDD was minimal, its LED was > almost always _off_, yet everything stuck to an absolute standstill). > > Clearly there's a very, very important limiter somewhere in bio layer > missing or broken, a 300M dd /dev/zero should never manage to put > such an onerous penalty on a system, IMHO. > You are using a USB 1.1 connection, about the same speed as a floppy. If you have not tuned your system to prevent all of the memory from being used to cache writes, it will be used that way. I don't have my notes handy, but I believe you need to tune the "dirty" parameters of /proc/sys/vm so that it makes better use of memory. Of course putting a fast device like SSD on a super slow connection makes no sense other than as a test of system behavior on misconfigured machines. > > I've got SysRq-W traces of these lockup conditions if wanted. > > > Not sure whether this is a 2.6.34-rc3 thing, might be a general issue. > > Likely the lockup behaviour is a symptom of very high memory pressure. > But this memory pressure shouldn't even be allowed to happen in the first > place, since the dd submission rate should immediately get limited by the kernel's > bio layer / elevators. > > Also, I'm wondering whether perhaps additionally there are some cond_resched() > to be inserted in some places, to try to improve coping with such a > broken situation at least. > > Thanks, > > Andreas Mohr -- Bill Davidsen <davidsen(a)tmr.com> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andreas Mohr on 8 Apr 2010 16:40
On Thu, Apr 08, 2010 at 04:12:41PM -0400, Bill Davidsen wrote: > Andreas Mohr wrote: >> Clearly there's a very, very important limiter somewhere in bio layer >> missing or broken, a 300M dd /dev/zero should never manage to put >> such an onerous penalty on a system, IMHO. >> > You are using a USB 1.1 connection, about the same speed as a floppy. If Ahahahaaa. A rather distant approximation given a speed of 20kB/s vs. 987kB/s ;) (but I get the point you're making here) I'm not at all convinced that USB2.0 would fare any better here, though: after all we are buffering the file that is written to the device - after the fact! (plus there are many existing complaints of people that copying of large files manages to break entire machines, and I doubt many of those were using USB1.1) https://bugzilla.kernel.org/show_bug.cgi?id=13347 https://bugzilla.kernel.org/show_bug.cgi?id=7372 And many other reports. > you have not tuned your system to prevent all of the memory from being > used to cache writes, it will be used that way. I don't have my notes > handy, but I believe you need to tune the "dirty" parameters of > /proc/sys/vm so that it makes better use of memory. Hmmmm. I don't believe that there should be much in need of being tuned, especially in light of default settings being so problematic. Of course things here are similar to the shell ulimit philosophy, but IMHO default behaviour should be reasonable. > Of course putting a fast device like SSD on a super slow connection makes > no sense other than as a test of system behavior on misconfigured > machines. "because I can" (tm) :) And because I like to break systems that happen to work moderately wonderfully for the mainstream(?)(?!?) case of quad cores with 16GB of RAM ;) [well in fact I don't, but of course that just happens to happen...] Thanks for your input, Andreas Mohr -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |