RAID/block regression starting from 2.6.32, bisected
in [Kernel]
|
Prev: AppArmor: update Maintainer and Documentation/kernel-parameters.txt
Next: [PATCH 3/3] frv: clean up arch/frv/Makefile
From: Vladislav Bolkhovitin on 28 Jul 2010 14:20 Hello, In recent kernels we are experiencing a problem that in our setup using SCST BLOCKIO backend some BIOs are finished, i.e. the finish callback called for them, with error -EIO. It happens quite often, much more often than one would expect to have an actual IO error. (BLOCKIO backend just converts all incoming SCSI commands to the corresponding block requests.) After some investigation, we figured out, that, most likely, raid5.c::make_request() for some reason sometimes calls bio_endio() with not BIO_UPTODATE bios. We bisected it to commit: commit a82afdfcb8c0df09776b6458af6b68fc58b2e87b Author: Tejun Heo <tj(a)kernel.org> Date: Fri Jul 3 17:48:16 2009 +0900 block: use the same failfast bits for bio and request bio and request use the same set of failfast bits. This patch makes the following changes to simplify things. * enumify BIO_RW* bits and reorder bits such that BIOS_RW_FAILFAST_* bits coincide with __REQ_FAILFAST_* bits. * The above pushes BIO_RW_AHEAD out of sync with __REQ_FAILFAST_DEV but the matching is useless anyway. init_request_from_bio() is responsible for setting FAILFAST bits on FS requests and non-FS requests never use BIO_RW_AHEAD. Drop the code and comment from blk_rq_bio_prep(). * Define REQ_FAILFAST_MASK which is OR of all FAILFAST bits and simplify FAILFAST flags handling in init_request_from_bio(). Signed-off-by: Tejun Heo <tj(a)kernel.org> Signed-off-by: Jens Axboe <jens.axboe(a)oracle.com> After looking at it I can't see how it can lead to the effect we are experiencing. Could anybody comment on this, please? Is it a known problem? The error can be only reproduced when running RAID 5. The general layout is: Disks --> RAID5 --> LVM --> BLOCKIO VDISK The problem is easy to reproduce by forcing the RAID 5 array to re-sync its members, eg just fail out one member and add it back into the array and then generate some IO using dd. In fact, just writing out to the partition table on the exported block device is usually enough to provoke the error. The complete thread about the topic you can find in http://sourceforge.net/mailarchive/forum.php?thread_name=20100727220110.GF31152%40atlantis.cc.ndsu.nodak.edu&forum_name=scst-devel If any additional information is needed we would be glad to provide it. Thanks, Vlad -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Tejun Heo on 30 Jul 2010 06:40 Hello, On 07/28/2010 08:16 PM, Vladislav Bolkhovitin wrote: > In recent kernels we are experiencing a problem that in our setup > using SCST BLOCKIO backend some BIOs are finished, i.e. the finish > callback called for them, with error -EIO. It happens quite often, > much more often than one would expect to have an actual IO > error. (BLOCKIO backend just converts all incoming SCSI commands to > the corresponding block requests.) > > After some investigation, we figured out, that, most likely, > raid5.c::make_request() for some reason sometimes calls bio_endio() > with not BIO_UPTODATE bios. > > We bisected it to commit: > > commit a82afdfcb8c0df09776b6458af6b68fc58b2e87b > Author: Tejun Heo <tj(a)kernel.org> > Date: Fri Jul 3 17:48:16 2009 +0900 > > block: use the same failfast bits for bio and request That commit doesn't (or at least isn't supposed to) make any behavior difference. It's just repositioning flag bits. If the commit is actually causing the problem, I think one possibility is that whatever code could be using hard coded constants which now are mapped to different flags. The mixed merge changes have been in mainline for quite some time and shipping in all major distros too and this is the first time this is reported, so I don't think it could be a widespread problem. Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Neil Brown on 1 Aug 2010 20:50
On Fri, 30 Jul 2010 12:29:30 +0200 Tejun Heo <tj(a)kernel.org> wrote: > Hello, > > On 07/28/2010 08:16 PM, Vladislav Bolkhovitin wrote: > > In recent kernels we are experiencing a problem that in our setup > > using SCST BLOCKIO backend some BIOs are finished, i.e. the finish > > callback called for them, with error -EIO. It happens quite often, > > much more often than one would expect to have an actual IO > > error. (BLOCKIO backend just converts all incoming SCSI commands to > > the corresponding block requests.) > > > > After some investigation, we figured out, that, most likely, > > raid5.c::make_request() for some reason sometimes calls bio_endio() > > with not BIO_UPTODATE bios. > > > > We bisected it to commit: > > > > commit a82afdfcb8c0df09776b6458af6b68fc58b2e87b > > Author: Tejun Heo <tj(a)kernel.org> > > Date: Fri Jul 3 17:48:16 2009 +0900 > > > > block: use the same failfast bits for bio and request > > That commit doesn't (or at least isn't supposed to) make any behavior > difference. It's just repositioning flag bits. If the commit is > actually causing the problem, I think one possibility is that whatever > code could be using hard coded constants which now are mapped to > different flags. The mixed merge changes have been in mainline for > quite some time and shipping in all major distros too and this is the > first time this is reported, so I don't think it could be a widespread > problem. > > Thanks. > The problem is that md/raid5 tests bio->bi_rw against RWA_MASK, which used to align with BIO_RW_AHEAD, and now doesn't. However the definition of bio_rw() in fs.h seems to justify that RWA_MASK should align with BIO_RW_AHEAD, as does the definition of READA. Given the current definitions, any WRITE request with BIO_RW_FAILFAST_DEV set is going to confused a number of drives which test bio_rw(bio) == WRITE I guess RWA_MASK needs to be changed to (1<<BIO_RW_AHEAD), and READA need to be change to that value too. Can I leave that to you Tejun? Thanks, NeilBrown -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |