Btrfs: Add hot data tracking functionality
in [Kernel]
|
Prev: [PATCH 4/4] Checkpatch: warn about unexpectedly long msleep's
Next: [RFC][PATCH] PCI / PCIe: Ask BIOS for control of all native services at once (v2)
From: Mingming Cao on 28 Jul 2010 17:30 On Tue, 2010-07-27 at 15:29 -0700, Tracy Reed wrote: > On Tue, Jul 27, 2010 at 05:00:18PM -0500, bchociej(a)gmail.com spake thusly: > > The long-term goal of these patches, as discussed in the Motivation > > section at the end of this message, is to enable Btrfs to perform > > automagic relocation of hot data to fast media like SSD. This goal has > > been motivated by the Project Ideas page on the Btrfs wiki. > > With disks being so highly virtualized away these days is there any > way for btrfs to know which are the fast outer-tracks vs the slower > inner-tracks of a physical disk? If so not only could this benefit SSD > owners but it could also benefit the many more spinning platters out > there. If not (which wouldn't be surprising) then disregard. Even just > having that sort of functionality for SSD would be excellent. If I > understand correctly not only would this work for SSD but if I have a > SAN full of many large 7200rpm disks and a few 15k SAS disks I could > effectively utilize that disk by allowing btrfs to place hot data on > the 15k SAS. I understand Compellent does this as well. > This certainly possible. The disk to store hot data does not has to limit to SSDs, thought current implementation detecting "fast" device by checking the SSD rotation flag. This could be easily extended if btrfs is able to detect the relatively fast devices. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Mingming Cao on 28 Jul 2010 18:10 On Wed, 2010-07-28 at 01:38 +0200, Christian Stroetmann wrote: > At the 28.07.2010 00:00, Ben Chociej wrote: > > INTRODUCTION: > > > > This patch series adds experimental support for tracking data > > temperature in Btrfs. Essentially, this means maintaining some key > > stats (like number of reads/writes, last read/write time, frequency of > > reads/writes), then distilling those numbers down to a single > > "temperature" value that reflects what data is "hot." > > > > The long-term goal of these patches, as discussed in the Motivation > > section at the end of this message, is to enable Btrfs to perform > > automagic relocation of hot data to fast media like SSD. This goal has > > been motivated by the Project Ideas page on the Btrfs wiki. > > > > Of course, users are warned not to run this code outside of development > > environments. These patches are EXPERIMENTAL, and as such they might > > eat your data and/or memory. > > > > > > MOTIVATION: > > > > The overall goal of enabling hot data relocation to SSD has been > > motivated by the Project Ideas page on the Btrfs wiki at > > https://btrfs.wiki.kernel.org/index.php/Project_ideas. It is hoped that > > this initial patchset will eventually mature into a usable hybrid > > storage feature set for Btrfs. > > > > This is essentially the traditional cache argument: SSD is fast and > > expensive; HDD is cheap but slow. ZFS, for example, can already take > > advantage of SSD caching. Btrfs should also be able to take advantage > > of hybrid storage without any broad, sweeping changes to existing code. > > > > Wouldn't this feature be useful for other file systems as well, so that > a more general and not an only Btrfs related solution is preferable? > Would certainly nice to add this feature to all filesystem, but right now btrfs is the only fs which have multiple device support in itself. Mingming > > With Btrfs's COW approach, an external cache (where data is *moved* to > > SSD, rather than just cached there) makes a lot of sense. Though these > > patches don't enable any relocation yet, they do lay an essential > > foundation for enabling that functionality in the near future. We plan > > to roll out an additional patchset introducing some of the automatic > > migration functionality in the next few weeks. > > > > > > With all the best > Christian Stroetmann > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo(a)vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Dave Chinner on 29 Jul 2010 08:20 On Wed, Jul 28, 2010 at 03:00:48PM -0700, Mingming Cao wrote: > On Wed, 2010-07-28 at 01:38 +0200, Christian Stroetmann wrote: > > At the 28.07.2010 00:00, Ben Chociej wrote: > > Wouldn't this feature be useful for other file systems as well, so that > > a more general and not an only Btrfs related solution is preferable? > > > > Would certainly nice to add this feature to all filesystem, but right > now btrfs is the only fs which have multiple device support in itself. Why does it even need multiple devices in the filesystem? All the filesystem needs to know is the relative speed of regions of it's block address space and to be provided allocation hints. everything else is just movement of data. You could keep the speed information in the device mapper table and add an interface for filesystems to query it, and then you've got infrastructure that all filesystems could hook into. The tracking features dont' appear to have anything btrfs specific in them, so t iseems wrong to implement it there just because you're only looking at btrfs' method of tracking multiple block devices and moving blocks.... Cheers, Dave. -- Dave Chinner david(a)fromorbit.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Christian Stroetmann on 29 Jul 2010 09:20 Aloha Mingming, Aloha Dave; On the 29.07.2010 14:17, Dave Chinner wrote: > On Wed, Jul 28, 2010 at 03:00:48PM -0700, Mingming Cao wrote: > >> On Wed, 2010-07-28 at 01:38 +0200, Christian Stroetmann wrote: >> >>> At the 28.07.2010 00:00, Ben Chociej wrote: >>> Wouldn't this feature be useful for other file systems as well, so that >>> a more general and not an only Btrfs related solution is preferable? >>> >>> >> Would certainly nice to add this feature to all filesystem, but right >> now btrfs is the only fs which have multiple device support in itself. >> > Thanks for your explanation, Mingming. And I had further questions to this point, but didn't know exactly how to formulate them in a short way. But luckily Dave has a possible solution that is related with my questions around the multiple device feature of Btrfs and hot data handling. > Why does it even need multiple devices in the filesystem? Yes, that was the point I asked myself after reading about this in the Btrfs wiki (https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices) and how ZFS does it. > All the > filesystem needs to know is the relative speed of regions of it's > block address space and to be provided allocation hints. everything > else is just movement of data. You could keep the speed information > in the device mapper table and add an interface for filesystems to > query it, and then you've got infrastructure that all filesystems > could hook into. > Yes indeed, something like this general solution that doesn't need the multiple device feature at all. > The tracking features dont' appear to have anything btrfs specific > in them, so t iseems wrong to implement it there just because you're > only looking at btrfs' method of tracking multiple block devices > and moving blocks.... > > Cheers, > > Dave. > Thank you very much for being creative. :D Cheerio Christian *<:o) O>-< -(D)>-< -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Mingming Cao on 4 Aug 2010 13:50
On Thu, 2010-07-29 at 22:17 +1000, Dave Chinner wrote: > On Wed, Jul 28, 2010 at 03:00:48PM -0700, Mingming Cao wrote: > > On Wed, 2010-07-28 at 01:38 +0200, Christian Stroetmann wrote: > > > At the 28.07.2010 00:00, Ben Chociej wrote: > > > Wouldn't this feature be useful for other file systems as well, so that > > > a more general and not an only Btrfs related solution is preferable? > > > > > > > Would certainly nice to add this feature to all filesystem, but right > > now btrfs is the only fs which have multiple device support in itself. > > Why does it even need multiple devices in the filesystem? All the > filesystem needs to know is the relative speed of regions of it's > block address space and to be provided allocation hints. everything > else is just movement of data. You could keep the speed information > in the device mapper table and add an interface for filesystems to > query it, and then you've got infrastructure that all filesystems > could hook into. > > The tracking features dont' appear to have anything btrfs specific > in them, so t iseems wrong to implement it there just because you're > only looking at btrfs' method of tracking multiple block devices > and moving blocks.... > I agree hot data tracking could be done at vfs layer. The current hot data temperature calculation and indexing code is very self-contained, and could be reuse to other fs or move up to vfs. We could define a common interface to export to hot data tempreture out. The relocation eventually has to be filesystem specific. btrfs does cow and knows where is the data on/off SSD directly makes the relocation to and from very straightforward. Mingming -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |