From: aminer on

Skybuck wrote:
> What if people wanna roll there own versions ? ;)
> They would much better be "served" by algorithms/pseudo
> code than real code which could be system/language specific ;)

It's easy to EXTRACT algorithms from Object Pascal code...

Look for example inside pbzip.pas, i am using this in the
main body of my program:

name:='msvcr100.dll';

It's the 'test' file that i am using - it's inside the
zip file also - one you compile and execute pbzip.pas it
will generate a file msvcr100.dll.bz. And as you have
noticed i am using a - portable - compound filesystem,
look at ParallelStructuredStorage.pas inside the zip file.


After that i am opening it with:

fstream1:=TFileStream.create(name, fmOpenReadWrite);

and i am reading chunks of streams and 'distributing' them
to my Thread Pool Engine to be compressed - in parallel -
by myobj.BZipcompress method, look at:


for i:=0 to e
do
begin

if (i=e) and (r=0) then break;
stream1:=TMemoryStream.create;
if (r > 0) and (i=e)
then stream1.copyfrom(fstream1,r)
else stream1.copyfrom(fstream1,d);
stream1.position:=0;
obj:=TJob.create;
obj.stream:=stream1;
obj.directory:=directory;
obj.compressionlevel:=9;
obj.streamindex:=inttostr(i);
obj.r:=r;
obj.number:=e;

TP.execute(myobj.BZipcompress,pointer(obj));

end;


I am doing the same thing in PZlib.pas...


http://pages.videotron.com/aminer/



Sincerely,
Amine Moulay Ramdane.

From: aminer on

Hello again,

And after that i am reading those compressed files
from the compound filesystem - look inside pzlib.pas -
and i am 'distributing' those compressed files, as streams,
to my Thread Pool Engine to be decompressed - look inside
pzlib.pas - by myobj.Zlibdecompress method, look at:


--------------------------------------------

names:=TStringlIST.create;

storage.foldernames('/',names);



len:=strtoint(names[0]);

if r=0 then len:=len+ 1 ;

for i:=0 to len

do

begin

if (i=len) and (r=0) then break;



obj:=TJob.create;

obj.directory:=directory;

obj.streamindex:=inttostr(i);

obj.index:=i;

obj.number:=e;

obj.r:=r;



TP.execute(myobj.Zlibdecompress,pointer(obj));



end;

--------------------------------------------------



http://pages.videotron.com/aminer/



Sincerely,
Amine Moulay Ramdane.

From: aminer on

I wrote:
> And as you have noticed i am using a portable
> compound filesystem, look at ParallelStructuredStorage.pas
> inside the zip file.

Why ?

Cause you can parallel compress your files and store
those compound filesystem .zlb (zlib) or .bz (bzip)
compressed files in a portable compound filesystem
and after that you can distribute your compound filesystem...

And of course you can uncompress files - or all the
content of your compound file system - from your compound
file system.


And of course that's easy with Parallel Compression 1.0 :)

http://pages.videotron.com/aminer/


Sincerely,
Amine Moulay Ramdane.



From: aminer on

Skybuvk wrote:
>[...] an algorithm really ;)
>What's so special about it ?

Parallel bzip and zlib is not just pbzip.pas and pzlib.pas
the parallel bzip and zlib algorithm include my Thread Pool Engine
algorithm + Parallel Queue algorithm ...

I am calling it algorithm cause it uses a finite number of
instructions and rules to resolve a problem - parallel compression
and decompression -

Do you understand ?

And as i said you can parallel compress your files and store
those compound filesystem .zlb (zlib) or .bz (bzip)
compressed files in a portable compound filesystem
and after that you can distribute your compound filesystem...
And of course you can uncompress files - or all the
content of your compound file system - from your compound
file system.

> I see a whole bunch of pascal/delphi files thrown together,
>a whole bunch of dll's and god-forbid ms*.dll files...

Those dlls are mandatory for now...

and you can easily write a batch file etc. and reorganize ...

> I see some "test programs" which are described as "modules" which they
> simply are not...

That's VERY easy to convert those pzlib.pas and pbzip.pas
to units, and that's what i will do in the next step...

Parallel Compression will still be enhanced in the future...

> It shouldn't be that hard... set your editor to "use tab character" (turn
> tabs to spaces off)

I am not using the delphi editor, just the notpad.exe or write.exe...
and i am compiling from the dos prompt...

>So far it seems like you are inserting your
>threads/syncronizations
>everywhere in single-thread-design algorithms ?

No, it's not just insertting threads/syncronizations ..

I have reasoned - and used logic - look for example at
parallelhashlist.pas inside the zip file, i am using MEWs etc.
carefully in the right places and i have also a little bit
modified the serial code... and it uses a hash based method ,
with an array of MREW...

The Thread Pool Engine Engine i have constructued it from zero
- and i have used my ParallelQueue - an efficent lock-free queue -
etc....

The parallel bzip and zlib, i have constructed it by using
also my Thread Pool Engine construction etc...
etc.

That's not just 'inserting' threads/syncronizations.

>But my estimate would be that for now on low core systems... the
>"compression" would take far more time...

No. pbzlib.pas takes for example 3.3x on 4 cores...

http://pages.videotron.com/aminer/ParallelCompression/parallelbzip.htm

Skybuck also wrote
> [...] or anything extraordinary...

Don't be stupid Skybuck.

It's in fact:

1- Usefull
2 - A good thing for educational purpose.


Amine Moulay Ramdane.


From: aminer on

Skybuck wrote:
>The thread pool concept is retarded.
>Any good delphi programmer is capable of creating an array of threads.
>So my advice to you:
>1. Delete your thread pool, because it's junk.
>2. Write a serious/big application that uses many threads,
>and simply derive from TThread to see how easy it is.


How can you be so stupid ?

My Thread Pool Engine is not just an array of threads,
it uses effient lock-free queues - example lock-free ParalleQueue -
for each worker thread and it uses work-stealing - for more
efficiency - etc ...

And it easy the work for you - you can 'reuse' the TThreadPool
Class... -
and it is very useful...


Please read again:

http://pages.videotron.com/aminer/threadpool.htm



Amine Moulay Ramdane.