Prev: screamer does not work
Next: postmodern thread safety in prepared statements functionality
From: Bob Felts on 23 Apr 2008 10:39
I have a friend who took a Perl script and recoded it in C for a huge
gain in performance. I've been telling him how much I like Lisp and, in
an attempt to do a little bit of Lisp evangelism, I offered to recode
his C into Lisp.

And I've hit a brick wall. The program needs four large arrays: two
arrays of (* 128 1024 1024) elements of unsigned bytes, and two arrays
of (* 128 1024 1024) elements of type bit.

LispWorks Personal 5.0.2 on Mac OS X won't even create the arrays;
they're too large.

sbcl 1.0.15.13 on Mac OS X (10.5.2, Intel) gives inconsistent behavior.

If I do:

(defun main ( )
(let* ((size (* 128 1024 1024)))
(array1 (make-array size
:element-type 'unsigned-byte
:initial-element 0))
(array2 (make-array size
:element-type 'bit
:initial-element 0))
(array3 ...
(array4 ...))
...))

then the code essentially hangs (it ran for 40 minutes before I killed
it.)

Ok, so maybe there's a problem creating large arrays inside a function.
Experimentation from the sbcl prompt showed that:

(defparameter *array1* (make-array size ...))

worked. All four arrays could be made.

So I moved the arrays outside of main like this:

(defparameter *array1* nil)
...
(defun main ...
(setf *array1* (make-array size ...)))

[The arrays are really dynamically sized, based on a parameter to main.]

This runs, once.

If I invoke main again, it hangs. Ok, maybe it's collecting garbage.
Let's see if we can help it out:

(defun main ...
(setf *array1* nil) ; for all 4 arrays
(setf *array1* (make-array size ...))
...)

Still no luck. Inserting a call to (gc) doesn't help.

At this point, the only way I can develop this code is to incrementally
develop the next part, run it, quit Aquamacs/slime, possibly kill sbcl
from the terminal, restart Aquamacs/slime, lather, rinse, repeat.

This isn't helping me to sing the praises of Lisp. Anyone been through
this before and have any helpful suggestions?



From: Richard M Kreuter on 23 Apr 2008 10:52
wrf3(a)stablecross.com (Bob Felts) writes:

> I have a friend who took a Perl script and recoded it in C for a huge
> gain in performance. I've been telling him how much I like Lisp and, in
> an attempt to do a little bit of Lisp evangelism, I offered to recode
> his C into Lisp.
>
> And I've hit a brick wall. The program needs four large arrays: two
> arrays of (* 128 1024 1024) elements of unsigned bytes, and two arrays
> of (* 128 1024 1024) elements of type bit.
>
> LispWorks Personal 5.0.2 on Mac OS X won't even create the arrays;
> they're too large.
>
> sbcl 1.0.15.13 on Mac OS X (10.5.2, Intel) gives inconsistent behavior.
>
> If I do:
>
> (defun main ( )
> (let* ((size (* 128 1024 1024)))
> (array1 (make-array size
> :element-type 'unsigned-byte
> :initial-element 0))
> (array2 (make-array size
> :element-type 'bit
> :initial-element 0))
> (array3 ...
> (array4 ...))
> ...))

It's not clear what MAIN does, but if it tries to print any of these
arrays (possibly by returning them), then Lisp may have to construct
string representations of each array (and for an array with 128
Mibiobjects, the printed representation will be no less than 128
Mibicharacters; if the implementation represents a character with 4
octets, that's a 512 MB string); additionally, when working under
SLIME, Lisp also has to ship that data over to Emacs, which can also
be a problematic bottleneck. Have you tried setting *PRINT-ARRAY* to
NIL (or *PRINT-LENGTH* to some small nonnegative integer)?

--
RmK
From: Brian on 23 Apr 2008 11:10
For me on SBCL 1.0.6 on Ubuntu, SBCL doesn't have enough heap to
allocate a 512MiB array by default. I needed to use the --dynamic-
space-size parameter.

You are going to be needing around 1.03GiB of memory for the arrays
alone; perhaps you should start SBCL with a larger dynamic space size?
From: Rainer Joswig on 23 Apr 2008 11:51
In article <1ifuhu4.av99ve1t4nyg0N%wrf3(a)stablecross.com>,
wrf3(a)stablecross.com (Bob Felts) wrote:

> I have a friend who took a Perl script and recoded it in C for a huge
> gain in performance. I've been telling him how much I like Lisp and, in
> an attempt to do a little bit of Lisp evangelism, I offered to recode
> his C into Lisp.
>
> And I've hit a brick wall. The program needs four large arrays: two
> arrays of (* 128 1024 1024) elements of unsigned bytes, and two arrays
> of (* 128 1024 1024) elements of type bit.
>
> LispWorks Personal 5.0.2 on Mac OS X won't even create the arrays;
> they're too large.

You can always use a 64bit Common Lisp. Some 32bit Lisps have
small array sizes (due to smaller fixnums than 32bit).
You can check for the max number of elements - there is
a variable that gives that number.

Also:

* make sure that the Lisp starts with enough memory (or is able
to extend its memory at runtime)

* don't try to print those large arrays in the repl. There
variables to control the printer, don't call functions
in the REPL that return those arrays, etc.




>
> sbcl 1.0.15.13 on Mac OS X (10.5.2, Intel) gives inconsistent behavior.
>
> If I do:
>
> (defun main ( )
> (let* ((size (* 128 1024 1024)))
> (array1 (make-array size
> :element-type 'unsigned-byte
> :initial-element 0))
> (array2 (make-array size
> :element-type 'bit
> :initial-element 0))
> (array3 ...
> (array4 ...))
> ...))
>
> then the code essentially hangs (it ran for 40 minutes before I killed
> it.)
>
> Ok, so maybe there's a problem creating large arrays inside a function.
> Experimentation from the sbcl prompt showed that:
>
> (defparameter *array1* (make-array size ...))
>
> worked. All four arrays could be made.
>
> So I moved the arrays outside of main like this:
>
> (defparameter *array1* nil)
> ...
> (defun main ...
> (setf *array1* (make-array size ...)))
>
> [The arrays are really dynamically sized, based on a parameter to main.]
>
> This runs, once.
>
> If I invoke main again, it hangs. Ok, maybe it's collecting garbage.
> Let's see if we can help it out:
>
> (defun main ...
> (setf *array1* nil) ; for all 4 arrays
> (setf *array1* (make-array size ...))
> ...)
>
> Still no luck. Inserting a call to (gc) doesn't help.
>
> At this point, the only way I can develop this code is to incrementally
> develop the next part, run it, quit Aquamacs/slime, possibly kill sbcl
> from the terminal, restart Aquamacs/slime, lather, rinse, repeat.
>
> This isn't helping me to sing the praises of Lisp. Anyone been through
> this before and have any helpful suggestions?
>
>
>

--
http://lispm.dyndns.org/
From: Juho Snellman on 23 Apr 2008 11:56
wrf3(a)stablecross.com (Bob Felts) writes:
> (defun main ( )
> (let* ((size (* 128 1024 1024)))
> (array1 (make-array size
> :element-type 'unsigned-byte
> :initial-element 0))

Do you really mean to use UNSIGNED-BYTE here? That upgrades to T on
SBCL, and probably on all other CLs too. If you want an array of
octets, use (UNSIGNED-BYTE 8) as the element type.

--
Juho Snellman
 |  Next  |  Last
Pages: 1 2
Prev: screamer does not work
Next: postmodern thread safety in prepared statements functionality