design strategy on handling large DB
in [Perl]
|
Prev: Server Socket programming with Perl
Next: FAQ 9.6 How do I download a file from the user's machine? How do I open a file on another machine?
From: Ela on 11 Apr 2008 08:46 except in using mysql or related programs, how can perl be used to deal with large databases? Furthermore, how can I load that large db into cache in advance and later use another perl to access/search that db?
From: Ted Zlatanov on 11 Apr 2008 10:31 On Fri, 11 Apr 2008 20:46:13 +0800 "Ela" <ela(a)yantai.org> wrote: E> except in using mysql or related programs, how can perl be used to deal with E> large databases? In order (you may already know some of these): 1) learn SQL and at least some RDBMS design theory (select+join, update, delete, indexing, foreign keys, normalization, data types, etc.) 2) read the DBI documentation 3) try Rose::DB::Object or Class::DBIx to see if they will work for you instead of direct DBI queries. I hope by "large" you mean at most a hundred gigabytes of data (which is considered a mid-size database by most people). Large databases are usually in the hundreds of gigabytes - terabytes range and very different beasts. E> Furthermore, how can I load that large db into cache in advance and E> later use another perl to access/search that db? I don't understand exactly what you want; there are 3 possibilities I can think of and I'd rather not guess. Can you please explain what you want to accomplish in more detail? Ted
From: Ela on 11 Apr 2008 11:03 > I hope by "large" you mean at most a hundred gigabytes of data (which is > considered a mid-size database by most people). Large databases are > usually in the hundreds of gigabytes - terabytes range and very > different beasts. It's only 3GB... but I find when I run the current java program with the mysql, it uses up all the 2GB memory... > > E> Furthermore, how can I load that large db into cache in advance and > E> later use another perl to access/search that db? > > I don't understand exactly what you want; there are 3 possibilities I > can think of and I'd rather not guess. Can you please explain what you > want to accomplish in more detail? > > Ted In MySQL, the db is already structured and therefore runtime prog can access the structued DB. But in perl, I don't know whether I can put the DB flat file into some data structures in advance, so users can query the DB whenever they want instead of dynamically create the DB every time.
From: RedGrittyBrick on 11 Apr 2008 11:40 Ela wrote: > > It's only 3GB... but I find when I run the current java program with the > mysql, it uses up all the 2GB memory... 3 is usually greater than 2. Unless the DBMS is very space-inefficient at storing data and the Java program is very space-efficient at storing data. >> E> Furthermore, how can I load that large db into cache in advance cache usually means memory doesn't it? There must be something I haven't understood that explains how 3 GB data can be expected to fit into less than 2GB of memory. -- RGB
From: xhoster on 11 Apr 2008 13:55
"Ela" <ela(a)yantai.org> wrote: > except in using mysql or related programs, how can perl be used to deal > with large databases? Many ways. Flat files which you write your own access to, flat files used with some DBD module to access, special formats like DBM::Deep, etc. > Furthermore, how can I load that large db into > cache in advance To my mind, a DB that can be loaded into cache is by definition not large. > and later use another perl to access/search that db? I don't know of any way of reliably doing this that is less annoying than just running mysql or related programs. Afterall, that is what they are designed for. Xho -- -------------------- http://NewsReader.Com/ -------------------- The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact. |