Prev: ActivePerl Module Install
Next: Setting permissions on remote directory using Net::SFTP
From: himagauri on 18 Oct 2006 17:37
Hi
My perl script is a very big script and performs a lot of complex
tasks. One of the tasks within the script is to calulate the size of a
file as follows:

$size = -s $filename;
$size = -1 unless defined($size);

When the file size is larger than 1GB for example 16376862124 bytes or
16 GB, the script returns $size = -1
For a file size less than 1 GB for example 775752944 bytes,
the correct file size is returned.

When I run another sample script performing just the above simple task
of calculating file size, the correct size of 16376862124 bytes is
returned for the larger file after a long time-about 40 minutes. The
problem arises when the above task is within the larger script.

I'm using Perl Version 5.8.3
If I do a perl -V, I do get the following option in the summary:
uselargefiles=define

So where could the problem lie?
Please suggest a solution to my problem.

-Thanks,
Regards,
Gauri

From: xhoster on 18 Oct 2006 18:07
himagauri(a)gmail.com wrote:
> Hi
> My perl script is a very big script and performs a lot of complex
> tasks. One of the tasks within the script is to calulate the size of a
> file as follows:
>
> $size = -s $filename;

# ask perl what went wrong:
warn $! unless defined $size;

> $size = -1 unless defined($size);
>
> When the file size is larger than 1GB for example 16376862124 bytes or
> 16 GB, the script returns $size = -1
> For a file size less than 1 GB for example 775752944 bytes,
> the correct file size is returned.
>
> When I run another sample script performing just the above simple task
> of calculating file size, the correct size of 16376862124 bytes is
> returned for the larger file after a long time-about 40 minutes. The
> problem arises when the above task is within the larger script.

I would consider it taking 40 minutes to -s a file to be a problem in
and of itself. Maybe these problems are related. Can you strace (or
whatever the local equivalent of that is) the simple program and see what
the last 10 or so lines of the trace are?

>
> I'm using Perl Version 5.8.3
> If I do a perl -V, I do get the following option in the summary:
> uselargefiles=define

What OS?

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
From: Big and Blue on 18 Oct 2006 20:17
himagauri(a)gmail.com wrote:
>
> $size = -s $filename;
> $size = -1 unless defined($size);
>
> When the file size is larger than 1GB for example 16376862124 bytes or
> 16 GB, the script returns $size = -1

So you've only tested (or at least only reported) < 1GB and 16GB. What
about some intermediate values, say 1.5GB, 3GB, 6GB?

> When I run another sample script performing just the above simple task
> of calculating file size, the correct size of 16376862124 bytes is
> returned for the larger file after a long time-about 40 minutes. The
> problem arises when the above task is within the larger script.

So, your simple 2 line script takes 40 minutes to size one file, but you
don't consider *that* to be a problem?

> So where could the problem lie?
> Please suggest a solution to my problem.

As has been noted - please give more detail, particularly what OS this
is on. It reminds me of a problem sizing files on VAX systems when they
were beinf accessed as NFS servers. That OS had various types of file,
including variable length record files (IIRC). So in order to find out the
"real" size of the file you actually did have to read each record
individually and add up all of the record lengths. This made "ls -l" from a
Unix system takes *ages* for such files. (FWIW: eventually the server code
was changed to allow approximate sizing...)


--
Just because I've written it doesn't mean that
either you or I have to believe it.
From: himagauri on 18 Oct 2006 22:48
The OS is Suse Linux and perl version is 5.8.3.

Point is that I would later have to work on files >16 GB. Hence I
haven't tested for intermediate values.
Initially I thought the '-s' operator doesn't work for file sizes >1
GB. Hence I worked out a sample program and noticed that it works.

Then I thought probably I'm running out of memory..but not sure about
it.

The file in question is being transferred from MVS to Linux using FTP.
The file on MVS has a record format of variable block (RECFM=VB).
I need to calculate the file size on Linux to check if the number of
bytes transferred to linux is equal to the file size on MVS.
does Linux OS have variable length files? How do I define them? how
else can I calculate the file size in perl besides -s operator?
what about File::Stat? is it more efficient as compared to -s?

-Gauri

Big and Blue wrote:
> himagauri(a)gmail.com wrote:
> >
> > $size = -s $filename;
> > $size = -1 unless defined($size);
> >
> > When the file size is larger than 1GB for example 16376862124 bytes or
> > 16 GB, the script returns $size = -1
>
> So you've only tested (or at least only reported) < 1GB and 16GB. What
> about some intermediate values, say 1.5GB, 3GB, 6GB?
>
> > When I run another sample script performing just the above simple task
> > of calculating file size, the correct size of 16376862124 bytes is
> > returned for the larger file after a long time-about 40 minutes. The
> > problem arises when the above task is within the larger script.
>
> So, your simple 2 line script takes 40 minutes to size one file, but you
> don't consider *that* to be a problem?
>
> > So where could the problem lie?
> > Please suggest a solution to my problem.
>
> As has been noted - please give more detail, particularly what OS this
> is on. It reminds me of a problem sizing files on VAX systems when they
> were beinf accessed as NFS servers. That OS had various types of file,
> including variable length record files (IIRC). So in order to find out the
> "real" size of the file you actually did have to read each record
> individually and add up all of the record lengths. This made "ls -l" from a
> Unix system takes *ages* for such files. (FWIW: eventually the server code
> was changed to allow approximate sizing...)
>
>
> --
> Just because I've written it doesn't mean that
> either you or I have to believe it.

From: himagauri on 18 Oct 2006 22:52
OS is Suse Linux

xhoster(a)gmail.com wrote:
> himagauri(a)gmail.com wrote:
> > Hi
> > My perl script is a very big script and performs a lot of complex
> > tasks. One of the tasks within the script is to calulate the size of a
> > file as follows:
> >
> > $size = -s $filename;
>
> # ask perl what went wrong:
> warn $! unless defined $size;
>
> > $size = -1 unless defined($size);
> >
> > When the file size is larger than 1GB for example 16376862124 bytes or
> > 16 GB, the script returns $size = -1
> > For a file size less than 1 GB for example 775752944 bytes,
> > the correct file size is returned.
> >
> > When I run another sample script performing just the above simple task
> > of calculating file size, the correct size of 16376862124 bytes is
> > returned for the larger file after a long time-about 40 minutes. The
> > problem arises when the above task is within the larger script.
>
> I would consider it taking 40 minutes to -s a file to be a problem in
> and of itself. Maybe these problems are related. Can you strace (or
> whatever the local equivalent of that is) the simple program and see what
> the last 10 or so lines of the trace are?
>
> >
> > I'm using Perl Version 5.8.3
> > If I do a perl -V, I do get the following option in the summary:
> > uselargefiles=define
>
> What OS?
>
> Xho
>
> --
> -------------------- http://NewsReader.Com/ --------------------
> Usenet Newsgroup Service $9.95/Month 30GB

 |  Next  |  Last
Pages: 1 2 3
Prev: ActivePerl Module Install
Next: Setting permissions on remote directory using Net::SFTP