Prev: How to display a popup showing just items list view?
Next: Run a package from the command line without launching a notebook?
From: Aaron Bramson on 10 Aug 2010 04:01
Hello Everybody,

I am parsing a large data file and naturally this is memory intensive so I
am breaking it into the smallest chunks possible. But I'm still having a
problem with memory because I can't seem to clear up my RAM between chunks.
So far there is NO output except writing a csv file, and I've cleared the
main data-holding variable I use, and all of it is done inside a
module...but still when the module finishes the memory isn't cleared.
Quitting the kernel does clear the memory, but I can't do that in between
calls of this function. Can anybody tell me what I need to change to get
Mathematica to clear the memory at the end of the function?

Here's the function:

GetData[popdynamics_, simultaneous_, learning_] :=
Module[{ThisData, DataLine, DataFile},
ThisData = {};
DataLine = {};
DataFile =
OpenRead[
"Experiment-table.csv"];
Do[DataLine = ReadList[DataFile, Record, 1], {7}];
DataLine =
Map[StringTake[#, {2, StringLength[#] - 1}] &,
ReadList[DataFile, Record, 36,
RecordSeparators -> {",", "\n"}]];
While[DataLine != {},
If[(DataLine[[6]] == popdynamics) && (DataLine[[8]] ==
simultaneous) && (DataLine[[10]] == learning) ,
AppendTo[ThisData, DataLine]];
DataLine =
Map[StringTake[#, {2, StringLength[#] - 1}] &,
ReadList[DataFile, Record, 36, RecordSeparators -> {",", "\n"}]];
];
If[ThisData != {},
Export["Test Data 2e.csv", ThisData]];
Close[DataFile];
ClearAll[ThisData];
ClearSystemCache[];
]

Here's the function call: GetData["true", "false", "true"];

Thanks in advance for your assistance,
Aaron Bramson


From: Vince Virgilio on 11 Aug 2010 04:44
On Aug 10, 4:01 am, Aaron Bramson <aaronbram...(a)gmail.com> wrote:
> Hello Everybody,
>
> I am parsing a large data file and naturally this is memory intensive so I
> am breaking it into the smallest chunks possible. But I'm still having a
> problem with memory because I can't seem to clear up my RAM between chunks.
> So far there is NO output except writing a csv file, and I've cleared the
> main data-holding variable I use, and all of it is done inside a
> module...but still when the module finishes the memory isn't cleared.
> Quitting the kernel does clear the memory, but I can't do that in between
> calls of this function. Can anybody tell me what I need to change to get
> Mathematica to clear the memory at the end of the function?

Aaron,

Mathematica is a bottomless pit when it comes to memory consumption.
Sooner or later, you will hit this wall. There is no way around it,
except to save intermediate results to file, restart the kernel(s),
then continue the computation.

Vince

From: David Bailey on 12 Aug 2010 05:27
On 11/08/10 09:44, Vince Virgilio wrote:
> On Aug 10, 4:01 am, Aaron Bramson<aaronbram...(a)gmail.com> wrote:
>> Hello Everybody,
>>
>> I am parsing a large data file and naturally this is memory intensive so I
>> am breaking it into the smallest chunks possible. But I'm still having a
>> problem with memory because I can't seem to clear up my RAM between chunks.
>> So far there is NO output except writing a csv file, and I've cleared the
>> main data-holding variable I use, and all of it is done inside a
>> module...but still when the module finishes the memory isn't cleared.
>> Quitting the kernel does clear the memory, but I can't do that in between
>> calls of this function. Can anybody tell me what I need to change to get
>> Mathematica to clear the memory at the end of the function?
>
> Aaron,
>
> Mathematica is a bottomless pit when it comes to memory consumption.
> Sooner or later, you will hit this wall. There is no way around it,
> except to save intermediate results to file, restart the kernel(s),
> then continue the computation.
>
> Vince
>
I don't think I agree with this! I think Vince should provide the
smallest possible example of his problem as a working example (complete
with data). Also remember that Mathematica will not return memory when
it does not need it (I imagine), it will just reuse that memory as and
when it is required.

Vince, have you actually run out of memory doing this calculation?

David Bailey

http://www.dbaileyconsultancy.co.uk

From: peter on 12 Aug 2010 05:25
you could try
Clear["Global`*"]

On 11 August 2010 09:45, Vince Virgilio <blueschi(a)gmail.com> wrote:
> On Aug 10, 4:01 am, Aaron Bramson <aaronbram...(a)gmail.com> wrote:
>> Hello Everybody,
>>
>> I am parsing a large data file and naturally this is memory intensive so I
>> am breaking it into the smallest chunks possible. But I'm still having a
>> problem with memory because I can't seem to clear up my RAM between chunks.
>> So far there is NO output except writing a csv file, and I've cleared the
>> main data-holding variable I use, and all of it is done inside a
>> module...but still when the module finishes the memory isn't cleared.
>> Quitting the kernel does clear the memory, but I can't do that in between
>> calls of this function. Can anybody tell me what I need to change to get
>> Mathematica to clear the memory at the end of the function?
>
> Aaron,
>
> Mathematica is a bottomless pit when it comes to memory consumption.
> Sooner or later, you will hit this wall. There is no way around it,
> except to save intermediate results to file, restart the kernel(s),
> then continue the computation.
>
> Vince
>
>

From: Sébastien Roy on 12 Aug 2010 05:31
On Aug 11, 4:44 am, Vince Virgilio <blues...(a)gmail.com> wrote:
> On Aug 10, 4:01 am, Aaron Bramson <aaronbram...(a)gmail.com> wrote:
>
> > Hello Everybody,
>
> > I am parsing a large data file and naturally this is memory intensive so I
> > am breaking it into the smallest chunks possible. But I'm still having a
> > problem with memory because I can't seem to clear up my RAM between chunks.
> > So far there is NO output except writing a csv file, and I've cleared the
> > main data-holding variable I use, and all of it is done inside a
> > module...but still when the module finishes the memory isn't cleared.
> > Quitting the kernel does clear the memory, but I can't do that in between
> > calls of this function. Can anybody tell me what I need to change to get
> > Mathematica to clear the memory at the end of the function?
>
> Aaron,
>
> Mathematica is a bottomless pit when it comes to memory consumption.
> Sooner or later, you will hit this wall. There is no way around it,
> except to save intermediate results to file, restart the kernel(s),
> then continue the computation.
>
> Vince

One thing that helps a lot in practice is:

$HistoryLength = 0

The fact that it is set to infinity by default probably contributes to
the "bottomless pit"...


 | 
Pages: 1
Prev: How to display a popup showing just items list view?
Next: Run a package from the command line without launching a notebook?