direct-io regression [Was: How to track down abysmal performance ata - raid1 - crypto - vg/lv - xfs]
in [Kernel]
|
Prev: [PATCH] stop_machine: struct cpu_stopper, remove alignment padding on 64 bits
Next: stop_machine: struct cpu_stopper, remove alignment padding on 64 bits
From: Chris Mason on 5 Aug 2010 07:40 On Thu, Aug 05, 2010 at 11:31:00AM +0200, Dominik Brodowski wrote: > Hey, > > when attempting to track down insufficient I/O performance, I found the > following reression relating to direct-io on my notebook, where an > ata device, which consists of several partitions, is combined to a lvm > volume, and one logical volume is then encrypted using dm-crypt. Test case > was the following command: > > $ dd if=/dev/mapper/vg0-root_crypt of=/dev/zero iflag=direct bs=8k count=131072 > > 2.6.34 results in ~16 MB/s, > 2.6.35 results in ~ 3.1 MB/s > > The regression was bisected down to the follwoing commit: > > commit c2c6ca417e2db7a519e6e92c82f4a933d940d076 > Author: Josef Bacik <josef(a)redhat.com> > Date: Sun May 23 11:00:55 2010 -0400 > > direct-io: do not merge logically non-contiguous requests > > ... > > How to fix this? I do not use btrfs, but ext3 (and the access was down on > the block level, not on the fs level, so this btrs-related commit should not > cause such a regression). Well, you've already bisected down to an offending if statement, that's a huge help. I'll try to reproduce this and fix it up today. But, I'm surprised your drive is doing 8K dio reads at 16MB/s, that seems a little high. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Dominik Brodowski on 5 Aug 2010 11:40 Hey, On Thu, Aug 05, 2010 at 07:32:40AM -0400, Chris Mason wrote: > But, I'm surprised your drive is doing 8K dio reads at 16MB/s, that > seems a little high. Well, that's what it does: # $ dd if=/dev/mapper/vg0-home_crypt of=/dev/zero iflag=direct bs=8k count=131072 seek=131072 # 131072+0 records in # 131072+0 records out # 1073741824 bytes (1.1 GB) copied, 62.0177 s, 17.3 MB/s On Thu, Aug 05, 2010 at 08:36:49AM -0400, Josef Bacik wrote: > Hrm, I made sure there were no perf regressions when I wast testing this stuff, > though I think I only tested xfs and ext4. For this test, I'm not doing dio on filesystem level, but on block level (/dev/mapper/vg0-*_crypt). It seems that dm-crypt creates such offending holes, which cause this huge performance drop. > Originally I had a test where if we > provided our own submit_io, so maybe as a workaround just make > > if (dio->final_block_in_bio != dio->cur_page_block || > cur_offset != bio_next_offset) > > look like this > > if (dio->final_block_in_bio != dio->cur_page_block || > (dio->submit_io && cur_offset != bio_next_offset)) Tested-by: Dominik Brodowski <linux(a)dominikbrodowski.net> With this fix, I get proper speeds when doing dio reads from /dev/mapper/vg0-*_crypt; see the 17.3 MB/s above. Most strangely, also accesing /dev/mapper/vg0-* (un-encrypted) and the raw device at /dev/sda* speeds up (to up to 28 MB/s). Was only seeing around 16 to 18 MB/s without this patch for unencrypted access. > I know why it could cause a problem, but this change shouldn't be > causing a 400% regression. Well, it seems to cause -- at least on my notebook -- a 150% regression on unencrypted LVM2 access; and this > 400% on encrypted LVM2 access... > I suspect something else is afoot here. There is, probably. But the fix you propose helps a lot, already. Thanks & best, Dominik -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Chris Mason on 5 Aug 2010 11:50 On Thu, Aug 05, 2010 at 05:35:19PM +0200, Dominik Brodowski wrote: > Hey, > > On Thu, Aug 05, 2010 at 07:32:40AM -0400, Chris Mason wrote: > > But, I'm surprised your drive is doing 8K dio reads at 16MB/s, that > > seems a little high. > > Well, that's what it does: > > # $ dd if=/dev/mapper/vg0-home_crypt of=/dev/zero iflag=direct bs=8k count=131072 seek=131072 > # 131072+0 records in > # 131072+0 records out > # 1073741824 bytes (1.1 GB) copied, 62.0177 s, 17.3 MB/s Can I ask you to do the test directly to the real honest to goodness drive? If it were an SSD I'd be less surprised, but then the extra submits shouldn't hurt the ssd that much either. Thanks for testing the patch, I'll send it in. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Dominik Brodowski on 5 Aug 2010 12:00 On Thu, Aug 05, 2010 at 11:39:43AM -0400, Chris Mason wrote: > On Thu, Aug 05, 2010 at 05:35:19PM +0200, Dominik Brodowski wrote: > > On Thu, Aug 05, 2010 at 07:32:40AM -0400, Chris Mason wrote: > > > But, I'm surprised your drive is doing 8K dio reads at 16MB/s, that > > > seems a little high. > > > > Well, that's what it does: > > > > # $ dd if=/dev/mapper/vg0-home_crypt of=/dev/zero iflag=direct bs=8k count=131072 seek=131072 > > # 131072+0 records in > > # 131072+0 records out > > # 1073741824 bytes (1.1 GB) copied, 62.0177 s, 17.3 MB/s > > Can I ask you to do the test directly to the real honest to goodness > drive? If it were an SSD I'd be less surprised, but then the extra > submits shouldn't hurt the ssd that much either. From lower the chain up to the device: # LANG=EN dd if=/dev/mapper/vg0-root_crypt of=/dev/zero bs=8k count=131072 seek=393300 iflag=direct 131072+0 records in 131072+0 records out 1073741824 bytes (1.1 GB) copied, 63.1217 s, 17.0 MB/s # LANG=EN dd if=/dev/mapper/vg0-root of=/dev/zero bs=8k count=131072 seek=393300 iflag=direct 131072+0 records in 131072+0 records out 1073741824 bytes (1.1 GB) copied, 43.2335 s, 24.8 MB/s # LANG=EN dd if=/dev/sda5 of=/dev/zero bs=8k count=131072 seek=393300 iflag=direct 131072+0 records in 131072+0 records out 1073741824 bytes (1.1 GB) copied, 42.0868 s, 25.5 MB/s Best, Dominik -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Dominik Brodowski on 5 Aug 2010 12:40
Small correction: On Thu, Aug 05, 2010 at 05:35:19PM +0200, Dominik Brodowski wrote: > With this fix, I get proper speeds when doing dio reads from > /dev/mapper/vg0-*_crypt; see the 17.3 MB/s above. Most strangely, > also accesing /dev/mapper/vg0-* (un-encrypted) and the raw > device at /dev/sda* speeds up (to up to 28 MB/s). Was only seeing around > 16 to 18 MB/s without this patch for unencrypted access. The speed-up of the unencrypted access (18 -> 28 MB/s) is caused by using a different configuration for kernel 2.6.35; and seems to be unrelated to your patch. Will try to track down which config option is the culprit. kernel, dmcrypt? | good config | bad config ------------------------------------------------- patched 2.6.35, dmcrypt ~ 18 MB/s ~ 13 MB/s patched 2.6.35 ~ 28 MB/s ~ 18 MB/s ------------------------------------------------- plain 2.6.35, dmcrypt ~ 3 MB/s ~ 3 MB/s plain 2.6.35 <not tested> ~ 16 MB/s Best, Dominik -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |