Speed of reading some MB of data using qx(...)
in [Perl]
|
From: Wolfram Humann on 23 Jul 2010 05:05 On Jul 22, 11:14 pm, Ilya Zakharevich <nospam-ab...(a)ilyaz.org> wrote: > On 2010-07-22, Uri Guttman <u...(a)StemSystems.com> wrote: > > > WH> But the effect is the same: fast on linux, slow on win32. Besides > > WH> bashing win32 :-) and ideas for reason and (possibly) cure? > > > you also bashed File::Slurp in a bug report you sent me. obviously it is > > a winblows issue. one possibility is that winblows does poor process > > context switching and piping between processes causes lots of > > that. there may be other reasons but i stay away from redmond. > > Disable your network adapters, and switch off virus checking. Recheck > speed. > > Hope this helps, > Ilya Ilya, I tried that and it makes no difference. Also, when enabled, I don't see any CPU usage in the virus check process -- it all goes to perl.exe. I also tried things using cygwin perl on the same win32 pc: cygwin perl also runs <1 second ... Thanks for the suggestion, Wolfram
From: Ben Morrow on 25 Jul 2010 18:10 Quoth Wolfram Humann <w.c.humann(a)arcor.de>: > I have a program that processes PDF files by converting them to > Postscript, read the ps and do something with it. I use pdftops (from > xpdf) for the pdf->ps conversion and retrieve the result like this: > > $ps_text = qx( pdftops $infile - ); > > On win32 using strawberry perl (tried 5.10 and 5.12) this takes much > more time than I expected so I did a test and first converted the PDF > to Postscript, then read the Postscript (about 12 MB) like this (cat > on win32 provided by cygwin): > > perl -E" $t = qx(cat psfile.ps); say length $t " > > This takes about 16 seconds on win32 but only <1 seconds on Linux. I > was afraid that this might be a 'binmode' problem so I also tried > this: > > perl -E" open $in,'cat psfile.ps |'; binmode $in; local $/; $t=<$in>; > say length $t " > > But the effect is the same: fast on linux, slow on win32. Besides > bashing win32 :-) and ideas for reason and (possibly) cure? Win32's pipes are *really* slow. Write it to a temporary and then read the file normally in perl. Ben
From: Wolfram Humann on 26 Jul 2010 03:24 On Jul 26, 12:10 am, Ben Morrow <b...(a)morrow.me.uk> wrote: > > Win32's pipes are *really* slow. Write it to a temporary and then read > the file normally in perl. After further experiments I am now convinced that pipes are not the bottleneck (even on win32 piping 10MB can't take 20 seconds...). The problem seems to be Straberry's memory management for long strings. I need to do some more benchmarking and will report this issue separately. Does anybody know of the appropriate place to report Strawberry Perl specific bugs? Wolfram
From: Ben Morrow on 26 Jul 2010 05:39 Quoth Wolfram Humann <w.c.humann(a)arcor.de>: > On Jul 26, 12:10�am, Ben Morrow <b...(a)morrow.me.uk> wrote: > > > > Win32's pipes are *really* slow. Write it to a temporary and then read > > the file normally in perl. > > After further experiments I am now convinced that pipes are not the > bottleneck (even on win32 piping 10MB can't take 20 seconds...). The > problem seems to be Straberry's memory management for long strings. I > need to do some more benchmarking and will report this issue > separately. > > Does anybody know of the appropriate place to report Strawberry Perl > specific bugs? I seriously doubt the issue is with Strawberry specifically; almost certainly any issue applies Win32 perl in general and should be reported to p5p with perlbug. If you can confirm it is specific to Strawberry (so, e.g., a self-compiled mingw perl *doesn't* have the problem) then I think the correct place is the Perl::Dist::Strawberry queue on rt.cpan.org (mail bug-Perl-Dist-Strawberry(a)rt.cpan.org). (IIRC malloc on Win32 is *also* known to be deadly slow, and also IIRC it's impossible to use perl's malloc without breaking things...) Ben
From: Wolfram Humann on 26 Jul 2010 06:07
On Jul 26, 11:39 am, Ben Morrow <b...(a)morrow.me.uk> wrote: > > I seriously doubt the issue is with Strawberry specifically; almost > certainly any issue applies Win32 perl in general and should be reported > to p5p with perlbug. If you can confirm it is specific to Strawberry > (so, e.g., a self-compiled mingw perl *doesn't* have the problem) then I > think the correct place is the Perl::Dist::Strawberry queue on > rt.cpan.org (mail bug-Perl-Dist-Strawbe...(a)rt.cpan.org). > > (IIRC malloc on Win32 is *also* known to be deadly slow, and also IIRC > it's impossible to use perl's malloc without breaking things...) > > Ben Thanks for the pointers. My comparison is Strawberry Perl (5.10 and 5.12) against Cygwin Perl on the same machine. The latter (as well as Perl on Linux) doesn't have the issues I see. Is that a sufficient "proof" for the issues being Strawberry specific? Wolfram |