write a 20GB file
in [Python]
|
Prev: d-cm Controll Manager
Next: cmd app and xml
From: Nobody on 14 May 2010 15:23 On Fri, 14 May 2010 10:50:49 -0400, J wrote: > someone smarter than me can correct me, but file.write() will write when > it's buffer is filled, or close() or flush() are called. And, in all probability, seek() will either flush it immediately or cause the next write() to flush it before writing anything.
From: J on 14 May 2010 18:38 On Fri, May 14, 2010 at 15:23, Nobody <nobody(a)nowhere.com> wrote: > On Fri, 14 May 2010 10:50:49 -0400, J wrote: > >> someone smarter than me can correct me, but file.write() will write when >> it's buffer is filled, or close() or flush() are called. > > And, in all probability, seek() will either flush it immediately or cause > the next write() to flush it before writing anything. Ahhh... I didn't know that... I thought seek() just moved the pointer through the file a little further.... Cool.
From: Jackie Lee on 14 May 2010 22:45 Thanks to y'all. I should have be more careful reading the documentation. Cheers On Fri, May 14, 2010 at 10:07 PM, Martin v. Loewis <martin(a)v.loewis.de> wrote: >> The code works fine. I just don't know how f.write works. It says that >> file.write won't write the file until file.close or file.flush. > > You are misinterpreting the documentation. It certainly won't keep the > entire file in memory. Instead, it has a fixed-size buffer (something > like 8kiB or 32kiB) in which it writes and which it flushes when that > buffer is full. > > The comment about flush and close merely refers to the problem that some > data may still be in the buffer at any point in time, unless you just > called close or flush. > > HTH, > Martin > -- Jackie
From: Nobody on 15 May 2010 10:05 On Fri, 14 May 2010 18:38:55 -0400, J wrote: >>> someone smarter than me can correct me, but file.write() will write when >>> it's buffer is filled, or close() or flush() are called. >> >> And, in all probability, seek() will either flush it immediately or cause >> the next write() to flush it before writing anything. > > Ahhh... I didn't know that... I thought seek() just moved the pointer > through the file a little further.... Think about how this affects buffering. write() writes at the current file position. If you write, then seek, then write, it can't just concatenate the two sets of data, as that would "lose" the seek. Either the buffer has to contain multiple, distinct sets of data, each with an associated position, or (far more likely), the original data must be written to the correct location before the second set of data can be stored.
From: Dave Angel on 15 May 2010 20:09
Nathan Rice wrote: > This is precisely the situation mmap was made for :) It has almost the same > methods as a file so it should be an easy replacement. > > <snip> Only on a 64bit system, and I'm not sure it's even possible there in every case. On a 32bit system, it would be impossible to mmap a 20gb file. You only have 4gb of address space to play with, total. DaveA |