Prev: How to run an external application on a different computer from MATLAB
Next: Querying MATLAB Memory
From: TJ Ulrich on 18 Jun 2008 14:32
Peter Boettcher <boettcher(a)ll.mit.edu> wrote in message
<muyhcbq1xbv.fsf(a)G99-Boettcher.llan.ll.mit.edu>...
> "TJ Ulrich" <tjuii(a)yahoo.com> writes:
>
> So after it's read in, and A is a 3006x255x255 vector of uint8s, if you
> say plot(A(:,1,1)) you get another out-of-memory error?
>
> And B = A(1,1,1) fails with the same error?

Yes. ... but I have a solution that I don't understand. I will relate it here for
posterity, and maybe someone will understand what is going on. This will
also address the other post about memmapfile.

The documentation for my file format (it is coming from a commercial
ultrasound system) says the data is recorded and written to file as uint8.
Upon receiving the data file I used the memmapfile to get some info and
verified that (according to memmapfile) the data was uint8. I proceeded with
that information and found myself here not understanding my out of memory
problems.

I thought about what the data looked like and what it should look like and
realized that uint8 was NOT correct. Using int16 (e.g., A = fread(fid,'*int16'))
gives me the data that I expected and solves my memory problems. I
understand that the number of array elements is now half of what it was for
uint8 data, but the amount of contiguous memory must be the same (i.e. 2
bytes/element now instead of 1), right? So why such a dramatic difference in
performance?

As for memmapfile, why did it tell me that this was uint8 data, when it clearly
is not? I can understand the documentation being wrong (I won't go there
now), but shouldn't memmapfile tell me what is really in there? Also,
memmapfile does look like a powerful function that may help me in the
future, but the documentation (untypical of Matlab documentation) is cryptic
enough to not be very useful without spending sometime on it. But maybe
that is just me trying to skim through it to quickly.

Thanks for all your help,

TJ


From: Peter Boettcher on 18 Jun 2008 15:09

"TJ Ulrich" <tjuii(a)yahoo.com> writes:
> Yes. ... but I have a solution that I don't understand. I will relate it here for
> posterity, and maybe someone will understand what is going on. This will
> also address the other post about memmapfile.
>
> The documentation for my file format (it is coming from a commercial
> ultrasound system) says the data is recorded and written to file as uint8.
> Upon receiving the data file I used the memmapfile to get some info and
> verified that (according to memmapfile) the data was uint8. I proceeded with
> that information and found myself here not understanding my out of memory
> problems.
>
> I thought about what the data looked like and what it should look like and
> realized that uint8 was NOT correct. Using int16 (e.g., A = fread(fid,'*int16'))
> gives me the data that I expected and solves my memory problems. I
> understand that the number of array elements is now half of what it was for
> uint8 data, but the amount of contiguous memory must be the same (i.e. 2
> bytes/element now instead of 1), right? So why such a dramatic difference in
> performance?
>
> As for memmapfile, why did it tell me that this was uint8 data, when it clearly
> is not?

A file on disk is simply a stream of bytes. NOTHING, repeat NOTHING,
out there can know what datatypes these bytes are intended to represent
outside of external information (or a specification for a file format).

memmapfile is used to "map" a file into memory, where it can accessed
without ever explicitly reading. Instead, the operating system loads a
page (usually 4KB) at a time when the user accesses that area of the
file. It does not try to identify the contents of the file. Maybe it
assumes uint8 if told nothing else, and this is what you saw?

I have no suggestion regarding '*uint16' working but '*uint8' not, from
the point of view of memory. Unless you're doing something with the
data, like extracting a size from a header, then creating a matrix that
size. Then the format mismatch would be enough to confuse things.

> I can understand the documentation being wrong (I won't go there
> now), but shouldn't memmapfile tell me what is really in there? Also,
> memmapfile does look like a powerful function that may help me in the
> future, but the documentation (untypical of Matlab documentation) is cryptic
> enough to not be very useful without spending sometime on it. But maybe
> that is just me trying to skim through it to quickly.


-Peter
From: us on 18 Jun 2008 15:17
Peter Boettcher:
<SNIP down to smart answer...

> "TJ Ulrich" <tjuii(a)yahoo.com> writes:
> > As for memmapfile, why did it tell me that this was
uint8 data, when it clearly is not?

> A file on disk is simply a stream of bytes. NOTHING,
repeat NOTHING, out there can know what datatypes these
bytes are intended to represent outside of external
information (or a specification for a file format).
> memmapfile ... does not try to identify the contents of
the file. Maybe it assumes uint8 if told nothing else, and
this is what you saw...

and this is exactly what the docs say:

M = MEMMAPFILE(FILENAME) constructs a memmapfile object
that maps file FILENAME to memory, using default(ed,!)
property values.
....
Format: Char array or Nx3 cell array (defaults to 'uint8').
Format of the contents of the mapped region.
....

as usual, just a thought...
us
From: Peter Boettcher on 18 Jun 2008 15:44
"us " <us(a)neurol.unizh.ch> writes:

> Peter Boettcher:
> <SNIP down to smart answer...
>
>> "TJ Ulrich" <tjuii(a)yahoo.com> writes:
>> > As for memmapfile, why did it tell me that this was
> uint8 data, when it clearly is not?
>
>> A file on disk is simply a stream of bytes. NOTHING,
> repeat NOTHING, out there can know what datatypes these
> bytes are intended to represent outside of external
> information (or a specification for a file format).
>> memmapfile ... does not try to identify the contents of
> the file. Maybe it assumes uint8 if told nothing else, and
> this is what you saw...
>
> and this is exactly what the docs say:
>
> M = MEMMAPFILE(FILENAME) constructs a memmapfile object
> that maps file FILENAME to memory, using default(ed,!)
> property values.
> ...
> Format: Char array or Nx3 cell array (defaults to 'uint8').
> Format of the contents of the mapped region.
> ...

One more note on memmapfile: even though it doesn't consume physical
memory until each page is accessed, the entire file DOES consume virtual
memory space. That is, every byte of the file has a corresponding
address in the mapping. So an active memmapfile can take a huge chunk
of contiguous space in the MATLAB process.

-Peter
From: TJ Ulrich on 18 Jun 2008 16:20
Thanks for your help. Indeed this was a case of a default value from matlab
(uint8) unfortunately being what I expected it to be ... but not corresponding to
reality ... as usual, resulting in user error (my most common errors). As for the
int16 vs uint8 contiguous space issue, I'm still clueless, but just happy that it
works now.

I'll have to spend some time learning how to use memmapfile properly. I think
it will be advantageous to get at portions of these filesinstead of grabbing the
whole thing.

Thanks again,

TJ



 |  Next  |  Last
Pages: 1 2
Prev: How to run an external application on a different computer from MATLAB
Next: Querying MATLAB Memory