Ideas for distributed Samba servers
in [Samba]
|
Prev: [Samba] Upgrading 3.2.15 to 3.3.12 sernet package on opensuse 10.2
Next: RHEL 5 compilation of Samba 3.5.2, termcap library problem, use '--no-as-needed'
From: Eric Shubert on 10 Apr 2010 13:30 Robert LeBlanc wrote: > I'm trying to think about how to setup a Samba system and would like to pick > the brains of some experts. We are looking up put a large amount of storage > ~75TB in a central data center. We have some remote (ok, not remote, but > across slower links, ok if you consider several hundred clients over 1Gb to > be slow) locations that we would like to set up samba servers that 'cache' > the file system and serve it up to the clients in the building and sync with > the main data center storage. The idea is have a couple of TB that are > located in the building that serve up the Samba share. When a client > requests a file, if it's in the local cache it is served up from there, if > not then the Samba server grabs the file from the main data center and > serves it to the client. When a file is written, something like rsync is > used to transfer only difference back to the main data center. The problem > is that I'm not sure of a file system that does this. We are using Lustre on > our HPC, but this won't do what we want. > > Any suggestions are welcome. > > Robert LeBlanc > Life Sciences & Undergraduate Education Computer Support > Brigham Young University I'm curious to know what you came up with for this. Care you share? TIA. -- -Eric 'shubes' -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
From: Nico Kadel-Garcia on 10 Apr 2010 16:00 On Sat, Apr 10, 2010 at 1:14 PM, Eric Shubert <ejs(a)shubes.net> wrote: > Robert LeBlanc wrote: >> >> I'm trying to think about how to setup a Samba system and would like to >> pick >> the brains of some experts. We are looking up put a large amount of >> storage >> ~75TB in a central data center. We have some remote (ok, not remote, but >> across slower links, ok if you consider several hundred clients over 1Gb >> to >> be slow) locations that we would like to set up samba servers that 'cache' >> the file system and serve it up to the clients in the building and sync >> with >> the main data center storage. The idea is have a couple of TB that are >> located in the building that serve up the Samba share. When a client >> requests a file, if it's in the local cache it is served up from there, if >> not then the Samba server grabs the file from the main data center and >> serves it to the client. When a file is written, something like rsync is >> used to transfer only difference back to the main data center. The problem >> is that I'm not sure of a file system that does this. We are using Lustre >> on >> our HPC, but this won't do what we want. >> >> Any suggestions are welcome. >> >> Robert LeBlanc >> Life Sciences & Undergraduate Education Computer Support >> Brigham Young University > > I'm curious to know what you came up with for this. Care you share? > TIA. It's a historically tricky problem, and takes serious thought about who is allowed to make changes, and how they propagate to remote locations, and about how synchronization is scheduled. One of my favorites has become "git". Use local repositories, much as git is used for the Linux kernel, and submit changes to a central repository for users who are so authorized. The merging capabilities are pretty good, and changes can be made locally without committing them to the central repository. And by the way, it run on Samba one heck of a lot more efficiently and effectively than Subversion does. -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
From: Robert LeBlanc on 10 Apr 2010 18:10 On Sat, Apr 10, 2010 at 11:14 AM, Eric Shubert <ejs(a)shubes.net> wrote: > Robert LeBlanc wrote: > >> I'm trying to think about how to setup a Samba system and would like to >> pick >> the brains of some experts. We are looking up put a large amount of >> storage >> ~75TB in a central data center. We have some remote (ok, not remote, but >> across slower links, ok if you consider several hundred clients over 1Gb >> to >> be slow) locations that we would like to set up samba servers that 'cache' >> the file system and serve it up to the clients in the building and sync >> with >> the main data center storage. The idea is have a couple of TB that are >> located in the building that serve up the Samba share. When a client >> requests a file, if it's in the local cache it is served up from there, if >> not then the Samba server grabs the file from the main data center and >> serves it to the client. When a file is written, something like rsync is >> used to transfer only difference back to the main data center. The problem >> is that I'm not sure of a file system that does this. We are using Lustre >> on >> our HPC, but this won't do what we want. >> >> Any suggestions are welcome. >> >> Robert LeBlanc >> Life Sciences & Undergraduate Education Computer Support >> Brigham Young University >> > > I'm curious to know what you came up with for this. Care you share? > TIA. > > We haven't come up with anything yet. We are still thinking this over. It's not pressing yet as we don't have the storage yet. Robert LeBlanc Life Sciences & Undergraduate Education Computer Support Brigham Young University -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
From: Adam Tauno Williams on 11 Apr 2010 10:50 On Sat, 2010-04-10 at 10:14 -0700, Eric Shubert wrote: > Robert LeBlanc wrote: > > I'm trying to think about how to setup a Samba system and would like to pick > > the brains of some experts. We are looking up put a large amount of storage > > ~75TB in a central data center. We have some remote (ok, not remote, but > > across slower links, ok if you consider several hundred clients over 1Gb to > > be slow) locations that we would like to set up samba servers that 'cache' > > the file system and serve it up to the clients in the building and sync with > > the main data center storage. a.) I don't think you can really do that with a 'file server' b.) I believe what you describe is almost exactly how AFS works. <http://www.openafs.org/> "OpenAFS is the world's foremost location independent file system." c.) Most SAN vendors provide a block-level replication solution for their products. > The idea is have a couple of TB that are > > located in the building that serve up the Samba share. When a client > > requests a file, if it's in the local cache it is served up from there, if > > not then the Samba server grabs the file from the main data center and > > serves it to the client. When a file is written, something like rsync is > > used to transfer only difference back to the main data center. The problem > > is that I'm not sure of a file system that does this. We are using Lustre on > > our HPC, but this won't do what we want. With all the fun of file locking, concurrent access, etc... I think what you describe just won't work, or at least will never work well. Why not just you a groupware server that supports document check-out and check-in; that seems like the correct solution to me. Or possibly something like iFolder <http://ifolder.com/ifolder> -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba
From: ravi channavajhala on 11 Apr 2010 11:10
WAFS (Wide Area File System) appliances can be very well deployed for this sort of thing precisely. Unfortunately, I don't know of any opensource project for WAFS. However, commercial solutions such as Riverbed, Expand Networks, CISCO/WAFS, Juniper/Peribit do exist. Regards, /rkc CTO DCiEra (P) Ltd -----Original Message----- From: samba-bounces(a)lists.samba.org [mailto:samba-bounces(a)lists.samba.org] On Behalf Of Adam Tauno Williams Sent: Sunday, April 11, 2010 8:15 PM To: samba(a)lists.samba.org Subject: Re: [Samba] Ideas for distributed Samba servers On Sat, 2010-04-10 at 10:14 -0700, Eric Shubert wrote: > Robert LeBlanc wrote: > > I'm trying to think about how to setup a Samba system and would like to pick > > the brains of some experts. We are looking up put a large amount of storage > > ~75TB in a central data center. We have some remote (ok, not remote, but > > across slower links, ok if you consider several hundred clients over 1Gb to > > be slow) locations that we would like to set up samba servers that 'cache' > > the file system and serve it up to the clients in the building and sync with > > the main data center storage. a.) I don't think you can really do that with a 'file server' b.) I believe what you describe is almost exactly how AFS works. <http://www.openafs.org/> "OpenAFS is the world's foremost location independent file system." c.) Most SAN vendors provide a block-level replication solution for their products. > The idea is have a couple of TB that are > > located in the building that serve up the Samba share. When a client > > requests a file, if it's in the local cache it is served up from there, if > > not then the Samba server grabs the file from the main data center and > > serves it to the client. When a file is written, something like rsync is > > used to transfer only difference back to the main data center. The problem > > is that I'm not sure of a file system that does this. We are using Lustre on > > our HPC, but this won't do what we want. With all the fun of file locking, concurrent access, etc... I think what you describe just won't work, or at least will never work well. Why not just you a groupware server that supports document check-out and check-in; that seems like the correct solution to me. Or possibly something like iFolder <http://ifolder.com/ifolder> -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba -- To unsubscribe from this list go to the following URL and read the instructions: https://lists.samba.org/mailman/options/samba |