Prev: use of recordset
Next: missing message when using TRACE output in time-critical apps
From: Joseph M. Newcomer on 5 May 2008 16:47
One of the problems that arises with this paradigm is how do you update a file atomically,
that is, how do you insert or remove text from an existing file that might be concurrently
accessed by some other process. In this case, there are too many synchronization holes in
the copy-and-rename to work right, and the trick is to open the file in exclusive mode,
create a memory mapping (extend the mapping if doing an insertion), do in-memory moves
(memmove) to rearrange the contents, and then call SetFilePosition and SetEndOfFile, to
update the information. That way there is no chance of concurrency interfering. (I use
this example in my Systems Programming Course to show that memory-mapped files are useful
for more than just doing a fancy interprocess shared memory segment).
joe

On Mon, 5 May 2008 12:45:46 -0700, "Tom Serface" <tom.nospam(a)camaswood.com> wrote:

>I mostly write temp files when I'm trying to insert of exclude something
>from a file. The best way I've found to do it is to create a new file, copy
>the stuff I want from the previous file, then delete the original file and
>rename the temp file to the original file name. I do a lot of this in
>memory if the file isn't too big, but it's lots easier to just do it in temp
>files (imo).
>
>Tom
>
>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message
>news:n5nu141dl5pqqj4focudo92n2hcb8hba0k(a)4ax.com...
>> Also, I've discovered in some cases that where people were writing
>> "temporary files" it
>> was because "the data is too big to keep in memory, so I have to write a
>> file", and in no
>> case that I found for this was the file > 1MB! So unless you need to pass
>> these files on
>> to a subsequent processing program, consider carefully if you need to
>> write one at all.
>> joe
>>
Joseph M. Newcomer [MVP]
email: newcomer(a)flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Tom Serface on 5 May 2008 19:37
Good points of course, but the ones I'm talking about are mostly just flat
files that need to be updated. I understand your point though and I agree.
I tend to use a database for data that changes dynamically and from multiple
sources. It's nice that you can even use something like SQL Server CE for
very small applications without my trouble.

Tom

"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message
news:fbsu14l7qvm4h9okfuqr4tqraks80nkmde(a)4ax.com...
> One of the problems that arises with this paradigm is how do you update a
> file atomically,
> that is, how do you insert or remove text from an existing file that might
> be concurrently
> accessed by some other process. In this case, there are too many
> synchronization holes in
> the copy-and-rename to work right, and the trick is to open the file in
> exclusive mode,
> create a memory mapping (extend the mapping if doing an insertion), do
> in-memory moves
> (memmove) to rearrange the contents, and then call SetFilePosition and
> SetEndOfFile, to
> update the information. That way there is no chance of concurrency
> interfering. (I use
> this example in my Systems Programming Course to show that memory-mapped
> files are useful
> for more than just doing a fancy interprocess shared memory segment).
> joe

From: Joseph M. Newcomer on 5 May 2008 23:04
One of the encouraging trends is that SQL Server is now part of the standard environment;
over the years, I've had so many problems that were fundamentally database problems, and
for years I used dBASE or Paradox engines, but with the demise of Win16, and the lack of a
decent and standard database on Win32, I'd largely gotten away from this kind of
programming.

Mostly, if I have database problems, I find that XML-based databases work well. The other
reason for writing files (not necessarily temporary files) is data retention. I wrote a
flat-file XML database some years ago in about three days. Since I ended up not getting
paid for that project (the company went into Chapter 7 after my last invoice was
sent...only a few thousand, but the database was part of that project) I have often
thought of putting the code on my Web site. It requires some cleanup, so I haven't
bothered too much about doing it. It used memory mappings to update and delete records,
and had a simple single-field flat index. That's all we needed for the project.

Right now I'm in the midst of writing a term paper (silly me, going back to school...) so
the only program I'm working on now is an update to my Font Explorer, and only enough of
it is getting revised to produce the figures to illustrate that paper. But I'm becoming
more of an expert on Windows font technology than I'd thought I could be. There are some
weird and wondrous aspects that have caused me days of debugging.
joe
On Mon, 5 May 2008 16:37:46 -0700, "Tom Serface" <tom.nospam(a)camaswood.com> wrote:

>Good points of course, but the ones I'm talking about are mostly just flat
>files that need to be updated. I understand your point though and I agree.
>I tend to use a database for data that changes dynamically and from multiple
>sources. It's nice that you can even use something like SQL Server CE for
>very small applications without my trouble.
>
>Tom
>
>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message
>news:fbsu14l7qvm4h9okfuqr4tqraks80nkmde(a)4ax.com...
>> One of the problems that arises with this paradigm is how do you update a
>> file atomically,
>> that is, how do you insert or remove text from an existing file that might
>> be concurrently
>> accessed by some other process. In this case, there are too many
>> synchronization holes in
>> the copy-and-rename to work right, and the trick is to open the file in
>> exclusive mode,
>> create a memory mapping (extend the mapping if doing an insertion), do
>> in-memory moves
>> (memmove) to rearrange the contents, and then call SetFilePosition and
>> SetEndOfFile, to
>> update the information. That way there is no chance of concurrency
>> interfering. (I use
>> this example in my Systems Programming Course to show that memory-mapped
>> files are useful
>> for more than just doing a fancy interprocess shared memory segment).
>> joe
Joseph M. Newcomer [MVP]
email: newcomer(a)flounder.com
Web: http://www.flounder.com
MVP Tips: http://www.flounder.com/mvp_tips.htm
From: David Ching on 6 May 2008 00:04
"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message
news:98iv14pfl1khfda44c8quv2e2nlud97sem(a)4ax.com...
> One of the encouraging trends is that SQL Server is now part of the
> standard environment;

It's not standard on XP/Vista AFAIK. Even the free SQL Server Express is
not installed by default. And even if it is, there are myriads of setup
options and confusing things that make it unapproachable by the average end
user.

Far easier to use SQL Server Compact Edition. This is great, no server to
install or configure, the entire database is stored in one file that you
freely can xcopy around. The only disadvantage is it's meant to be used
only by one app at a time (no concurrency).

-- David


From: Tom Serface on 6 May 2008 01:34
Yeah, but it sets up your application to grow into bigger things too. It's
a great addition to the SQL family imo.

Tom

"David Ching" <dc(a)remove-this.dcsoft.com> wrote in message
news:ENQTj.15881$2g1.12148(a)nlpi068.nbdc.sbc.com...
> "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message
> news:98iv14pfl1khfda44c8quv2e2nlud97sem(a)4ax.com...
>> One of the encouraging trends is that SQL Server is now part of the
>> standard environment;
>
> It's not standard on XP/Vista AFAIK. Even the free SQL Server Express is
> not installed by default. And even if it is, there are myriads of setup
> options and confusing things that make it unapproachable by the average
> end user.
>
> Far easier to use SQL Server Compact Edition. This is great, no server to
> install or configure, the entire database is stored in one file that you
> freely can xcopy around. The only disadvantage is it's meant to be used
> only by one app at a time (no concurrency).
>
> -- David
>
>

First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: use of recordset
Next: missing message when using TRACE output in time-critical apps