Prev: character substitution
Next: how to find the second column is a digital number and biger than 50000
From: alexus on 12 Apr 2008 01:23
i have few huge flat files, for example one of them is ~150Mb.zip or
~610Mb uncompressed and it contains bunch of records seperated by |
(pipe), between pipes there is a text, some of the fields contains
spaces and I need to import all of it into MySQL in most optimized way
in terms of shell side, so first things first i need to parse it, what
would be the best way to read file line by line?

for i in `cat /path/to/file`; do
echo $i
done

wouldn't really work here, as some of the lines contains spaces, and
echo $i would show everything before space and i cant break line into
multiple peaces.

is there an easy way to read by line or seperate parametrs in loop by
line instead of spaces?
From: www.isp2dial.com on 12 Apr 2008 01:56
On Fri, 11 Apr 2008 22:23:54 -0700 (PDT), alexus <alexus(a)gmail.com>
wrote:

>i have few huge flat files, for example one of them is ~150Mb.zip or
>~610Mb uncompressed and it contains bunch of records seperated by |
>(pipe), between pipes there is a text, some of the fields contains
>spaces and I need to import all of it into MySQL in most optimized way

>is there an easy way to read by line or seperate parametrs in loop by
>line instead of spaces?

cat flatfile |

while read -d '|'; do
echo "$REPLY"
done


--
Webmail for Dialup Users
http://www.isp2dial.com/freeaccounts.html

From: pk on 12 Apr 2008 05:40
alexus wrote:

> is there an easy way to read by line or seperate parametrs in loop by
> line instead of spaces?

Please provide sample input and expected output.

--
All the commands are tested with bash and GNU tools, so they may use
nonstandard features. I try to mention when something is nonstandard (if
I'm aware of that), but I may miss something. Corrections are welcome.
From: Maxwell Lol on 12 Apr 2008 06:54
alexus <alexus(a)gmail.com> writes:

> is there an easy way to read by line or seperate parametrs in loop by
> line instead of spaces?

Generally it's best to have a single process read the file and parse
it a line at a time. You can use a posix shell, AWK, perl or whatever
depending upon the complexity.

From: Chris F.A. Johnson on 12 Apr 2008 11:17
On 2008-04-12, alexus wrote:
> i have few huge flat files, for example one of them is ~150Mb.zip or
> ~610Mb uncompressed and it contains bunch of records seperated by |
> (pipe), between pipes there is a text, some of the fields contains
> spaces and I need to import all of it into MySQL in most optimized way
> in terms of shell side, so first things first i need to parse it, what
> would be the best way to read file line by line?
>
> for i in `cat /path/to/file`; do
> echo $i
> done
>
> wouldn't really work here, as some of the lines contains spaces, and
> echo $i would show everything before space and i cant break line into
> multiple peaces.
>
> is there an easy way to read by line or seperate parametrs in loop by
> line instead of spaces?

They way to read a file in a shell loop is:

while IFS= read -r line
do
printf "%s\n" "$line"
done < /path/to/file

The 'while ... read ... line' can be adjusted to break the line
into parts.

However, for files as large as you are dealing with, it is better
to use awk.

--
Chris F.A. Johnson, author <http://cfaj.freeshell.org/shell/>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence
 |  Next  |  Last
Pages: 1 2 3 4 5 6
Prev: character substitution
Next: how to find the second column is a digital number and biger than 50000