From: John Paine on
Hi Glen,

Thanks for the comments. I wasn't too sure what to put as the subject, so
just opted for the flavor of the task and did not intend it to be
interpreted as relating to the access='direct' clause. (Besides, that's also
how it is referred to in the Intel documentation, though that's not of a
defence).

I did include a remark about the possibility of using access='direct', but
would prefer not to do that for a number of reasons. Primarily, I'd like the
flexibility of rewriting various portions of the binary files I use and
frequently the actual position will depend on the data that precedes the
location where the data is to be written. I know that I can keep track of
this position (and can even use fseek to directly tell me where I am in the
file), but wanted to check first that there wasn't some obvious option for
the open statement that I was missing that would allow me to rewrite a
portion of the file without truncating it. Also, having to have a fixed
record length would make it much harder to keep track of where I actually
want to write the data and I'd rather revamp my interface to the relevant
system calls than rewrite my code to manage extraneous task of counting
records.

John


"glen herrmannsfeldt" <gah(a)ugcs.caltech.edu> wrote in message
news:hknlej$4tk$1(a)naig.caltech.edu...
> John Paine <johnpaine1(a)optusnet.com.au> wrote:
>
>> I'm struggling with a problem in trying to perform direct
>> access read/write to a file using Intel 11.1.
>
> (snip)
>> 4. Once all of the data blocks have been written, I then want
>> to rewrite the header block with the updated statistics and
>> data block sizes etc
> (snip)
>
>> open(10,file='test.dat',form='binary')
> (snip)
>
>> write(10)numy,zmin,zmax
>> write(10)(numx(j),j=1,numy)
>
> Your subject says "Direct Access" but your sample code doesn't.
>
> You need the ACCESS='DIRECT' and RECL= options on the OPEN,
> and the REC= on the READ/WRITE statements. RECL specifies
> the record length that all records will have.
>
> If your data doesn't naturally have a fixed length then you
> have to do some work to divide it up into fixed length units.
> (Possibly wasting the end of some of the records.)
>
> Note that in the case of statements like:
>
> write(10)numy,zmin,zmax
> write(10)(numx(j),j=1,numy)
>
> Two records will be used, while it can be written:
>
> write(10)numy,zmin,zmax,(numx(j),j=1,numy)
>
> and use only one record. It can then be read with:
>
>
> read(10)numy,zmin,zmax,(numx(j),j=1,numy)
>
> (But note that these are not direct access I/O statements
> without the REC= option.
>
> -- glen

From: John Paine on
Hi Louis,

Thanks for the response.

I did consider this route as it is akin to the separate header/data files,
but avoids the problem of separate files. While it is certainly a feasible
solution, it's not really one I'd opt for. As per my comments to Glen's
response, I was hoping for something that would allow me to simply rewrite
data within an existing file using standard fortran and (although unstated
in my oriiginal email) with as little rewriting as possible. The task is
indeed somewhat similar to the PDF problem as in that case (if I remember
correctly) you need to know where the bytes defining the file structure are
at the end of the file, and you don't know that position until you've
finished writing the contents file. It is a pretty common task to rewrite
information in a header block, so I was hoping that Fortran would allow me
to do it directly without having to resort to the Windows API or writing the
functionality as a C library.

Many thanks
John


"Louis Krupp" <lkrupp_nospam(a)indra.com.invalid> wrote in message
news:eZqdnTGRr5zM-PLWnZ2dnUVZ_gednZ2d(a)indra.net...
> John Paine wrote:
>> Hi all,
>>
>> I'm struggling with a problem in trying to perform direct access
>> read/write to a file using Intel 11.1.
>>
>> What I want to do is:
>>
>> 1. Open a data file
>> 2. Write a header block to the file defining the data in the file (this
>> block is of known size, but unknown content when I open the file)
>> 3. Write multiple blocks of binary data of various types sequentially to
>> the file and accumulate statistics about the data such as range of
>> values, size of data blocks etc
>> 4. Once all of the data blocks have been written, I then want to rewrite
>> the header block with the updated statistics and data block sizes etc
>> 5. Close the file
> <snip>
>
> You may be overlooking the easy way to do this:
>
> 1. Write your multiple blocks of binary data to a scratch file,
> accumulating statistics as you go.
>
> 2. Rewind the scratch file.
>
> 3. Open the output data file and write the header with everything you
> need.
>
> 4. Read records from the scratch file and write them to the output file.
>
> 5. Close the output file. Remove the scratch file.
>
> Unless your files are really, really big (for some definition of "big"),
> this is likely to be fast enough *and* easier to code. Plus, it will let
> the header size vary, which might be convenient at some point.
>
> A couple more thoughts:
>
> If your header size and/or format change, make sure that new versions of
> the file that read the file can read old versions or at least fail
> gracefully. Including a version number in the first few bytes of the
> header is probably a good way to do this. You'll probably also want to
> make sure that old versions of the program know which version(s) of the
> file they can read.
>
> I'm not sure I'd recommend this practice, but as you may know, PDF uses a
> cross-reference and a trailer record at the end of the file. Programs
> that read PDF files start by reading a bunch of bytes at the end of the
> file and scanning for a magic marker string.
>
> Louis

From: John Paine on

"James Van Buskirk" <not_valid(a)comcast.net> wrote in message
news:hknqce$h1m$1(a)news.eternal-september.org...
> "John Paine" <johnpaine1(a)optusnet.com.au> wrote in message
> news:4b6f518f$0$32133$afc38c87(a)news.optusnet.com.au...
>
>> I'm struggling with a problem in trying to perform direct access
>> read/write to a file using Intel 11.1.
>
> C:\gfortran\clf\streamtest>type streamtest.f90
> program streamtest
> implicit none
> type head
> integer i
> real x
> character(3) c
> end type head
> type(head) header
> integer array(10)
> integer i
>
> array = [(i,i=1,size(array))]
> header = head(0,0,'0')
> open(10,file='streamtest.dat',access='stream')
> write(10) header, array
> header = head(2,3.14,'CAT')
> write(10,POS=1) header
> close(10)
> open(10,file='streamtest.dat',access='stream')
> header = head(0,0,'0')
> array = 0
> read(10), header, array
> write(*,*) 'header = ', header
> write(*,*) 'array = ', array
> end program streamtest
>
> C:\gfortran\clf\streamtest>gfortran streamtest.f90 -ostreamtest
>
> C:\gfortran\clf\streamtest>streamtest
> header = 2 3.1400001 CAT
> array = 1 2 3 4 5
> 6 7 8 9 10
>
> --
> write(*,*) transfer((/17.392111325966148d0,6.5794487871554595D-85, &
> 6.0134700243160014d-154/),(/'x'/)); end
>
>

Hi James,

I haven't had any experience with stream files, so didn't try using that
type of access as the Intel documentation didn't sound very promising.
However your sample looks like it does the job, so I rewrote the code as
below and it does exactly what I want.

The one point that troubles me is that the "pos=" keyword uses a number that
"indicates a file position in file storage units in a stream file ". This
seems to be bytes, but I thought that I saw a comment in the Intel
documentation (which I can't locate at the moment), that the "storage unit"
is system dependant. Also, I can't find any way to inquire what the unit
might be. I'm happy to ignore this for now as it allows me to implement a
simple replacement of the API calls with my own equivalents for a stream
file, but I'm sure that at some stage the unit size will come back to bite
me.

Many thanks
John





implicit none

integer*4 i,j
integer*4 numy,numx(40)

real*4 zmin,zmax
real*4 zval(40)

c 1: open the file

open(10,file='test_stream.dat',access='stream')

c 2: initialise and write out the header

numy=40
zmin=1e32
zmax=-1e32
do j=1,numy
numx(j)=0
end do

write(10)numy,zmin,zmax,(numx(j),j=1,numy)

c 3: loop over the data

do j=1,numy

c do stuff to create numx(j) data values in array zval
c NB numx(j) is determined as part of the calculation

numx(j)=j
do i=1,numx(j)
zval(i)=i
zmin=min(zmin,zval(i))
zmax=max(zmax,zval(i))
end do

c write out the data

write(10)(zval(i),i=1,numx(j))

end do

c 4: rewrite the changed avlues in the header

write(10,pos=5)zmin,zmax,(numx(j),j=1,numy)

c 5: close the file

close(10)

c reopen the file and read the data

open(10,file='test_stream',access='stream')

c this step works ok and the header is read correctly

read(10)numy,zmin,zmax,(numx(j),j=1,numy)

c this loop correctly reads the data from the file

do j=1,numy
read(10)(zval(i),i=1,numx(j))
end do
close(10)

end



From: John Paine on

"Richard Maine" <nospam(a)see.signature> wrote in message
news:1jdk4w1.162yhg4tspypsN%nospam(a)see.signature...
> John Paine <johnpaine1(a)optusnet.com.au> wrote:
>
>> What I want to do is:
> ....
>> 4. Once all of the data blocks have been written, I then want to rewrite
>> the
>> header block with the updated statistics and data block sizes etc
> ...
>> In CVF, I used to do this using the dfwin interface
> ..
>> Since I am now migrating my code to the Intel
>> compiler, I'm cleaning up my code base and would like to eliminate the
>> Windows routines and use standard Fortran IO.
> [example using form='binary']
>
> Well form='binary' is not standard Fortran, so if that is the objective,
> this would not achieve it even if it did act like you wanted.
>

I knew I was vulnerable on that front and was hoping that nobody would
notice. I know it's not standard, but I'm assuming that it'd be a brave
vendor who left it out in the forseeable future.


>> This all works fine, except that step 4 truncates the file after the
>> write
>> so all of the data records are lost....
>> The behavior of the 'binary' write truncating the file is documentated by
>> Intel. They also include a mention of "direct access" which suggests that
>> direct access be used.
>
> Yes, you could certainly do it with direct access, but there are lots of
> complications - more than you mentioned. For a start,
>
>> step 1 with access='direct',recl=1 specified.
>
> The use of recl=1 with direct access is a hack recognized by some
> compilers, but it is not standard. Direct access is standard, but the
> common special-case interpretation of recl=1 is not. There are other
> complications as well.
>

I agree, and that was one of the considerations for not going down that
route. Though if it had worked (and I did try it), I wouldn't have been
posting.

>> Thanks in advance for any suggestions.
>
> I'd recommend against going with direct access. It certainly can be
> done; I've done that kind of thing in the past. But the many gotchas of
> direct access are a large part of why I was a big pusher for stream
> access in f2003.
>
> I recommend using stream access. James gave some presumably fine example
> code (I didn't check in detail, but I suspect it is fine). I just
> thought I'd supply some English to supplement his Fortran. :-)
>
> The form='binary' and access='direct',recl=1 are both nonstandard
> variants of stream access. They date from before stream was
> standardized. Now that stream is standardized, I recommend using it
> instead of those nonstandard variants. As an additional "side" benefit,
> the standard requires that it act like you want (no truncation), at
> least as long as you stick to unformatted stream. (Formatted is a
> different story).
>
> The approach that Louis mentioned (using a temporary intermediate file)
> can also work well; that's your choice to make.

I will be going down the stream route and James' code was more than adequate
to put me on the right path. As an added benefit, it now seems that I will
be using standard fortran to do what I want. Very pleasing!


>
> --
> Richard Maine | Good judgment comes from experience;
> email: last name at domain . net | experience comes from bad judgment.
> domain: summertriangle | -- Mark Twain

From: Richard Maine on
John Paine <johnpaine1(a)optusnet.com.au> wrote:

> "Richard Maine" <nospam(a)see.signature> wrote in message
> news:1jdk4w1.162yhg4tspypsN%nospam(a)see.signature...

> > Well form='binary' is not standard Fortran, so if that is the objective,
> > this would not achieve it even if it did act like you wanted.
> >
> I knew I was vulnerable on that front and was hoping that nobody would
> notice. I know it's not standard, but I'm assuming that it'd be a brave
> vendor who left it out in the forseeable future.

Lots of vendors already don't have it. Pretty much all of them have some
kind of stream-like functionality, but the details do vary already. If a
large majority of vendors already did it the same way, that way probably
would have been adopted as the standard instead of introducing stream.
But they don't, so it wasn't.

Now if you are talking about the odds of Intel in specific dropping it,
I'd agree those odds would be low. But the odds of some other random
vendor happening to have it are much more questionable.

(I see you are happy with the suggestion to use stream, so these points
are perhaps a bit academic).

--
Richard Maine | Good judgment comes from experience;
email: last name at domain . net | experience comes from bad judgment.
domain: summertriangle | -- Mark Twain
First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4 5 6
Prev: Mac OS X and FORTRAN
Next: Problem with abstract interface