From: marc on
Hello,

I gave a big zipped file (foo.gz) and I would want to split it into n
small files, but in memory, because I haven't enough space on my FS.
I tried with mkfifo, gunzip and split, but it splits first all the
files on disk and it stops logically with
"No space left on device" error.
Is there another way ?

Thanks.
From: Stephane Chazelas on
2009-12-15, 05:55(-08), marc:
[...]
> I gave a big zipped file (foo.gz) and I would want to split it into n
> small files, but in memory, because I haven't enough space on my FS.
> I tried with mkfifo, gunzip and split, but it splits first all the
> files on disk and it stops logically with
> "No space left on device" error.
> Is there another way ?
[...]

If you mean load the file in memory, then remove it and then
create the split files, then with zsh (other shells can't cope
with binary data).:

print -rn -- "$(cat foo.gz && rm foo.gz") | split -b 10m - foo.gz.

However note that if foo.gz ends in newline characters, they
will be removed. It's probably not gonna be very efficient

You could use perl instead:

perl -0777 -pe 'unlink $ARGV' foo.gz | split -b10m - foo.gz.

One of the other things you could do that would be less resource
intensive if you have space for the whole file and one chunk is
to process the file from the end backward: start creating the
last chunk, then truncate the original big file, and proceed
backward like this.


--
St�phane
From: bb on
On 2009-12-15 14:55, marc wrote:
> Hello,
>
> I gave a big zipped file (foo.gz) and I would want to split it into n
> small files, but in memory, because I haven't enough space on my FS.
> I tried with mkfifo, gunzip and split, but it splits first all the
> files on disk and it stops logically with
> "No space left on device" error.
> Is there another way ?
>
> Thanks.

move it to /tmp

/bb
From: Kaz Kylheku on
On 2009-12-15, marc <marc.tessis(a)caramail.com> wrote:
> Hello,
>
> I gave a big zipped file (foo.gz) and I would want to split it into n
> small files, but in memory, because I haven't enough space on my FS.

Stick an inexpensive storage device into your USB port.