From: Mathieu Desnoyers on
* Linus Torvalds (torvalds(a)linux-foundation.org) wrote:
>
>
> On Wed, 19 May 2010, Mathieu Desnoyers wrote:
> >
> > Good point. This discard flag might do the trick and let us keep things simple.
> > The major concern here is to keep the page cache disturbance relatively low.
> > Which of new page allocation or stealing back the page has the lowest overhead
> > would have to be determined with benchmarks.
>
> We could probably make it easier somehow to do the writeback and discard
> thing, but I have had _very_ good experiences with even a rather trivial
> file writer that basically used (iirc) 8MB windows, and the logic was very
> trivial:
>
> - before writing a new 8M window, do "start writeback"
> (SYNC_FILE_RANGE_WRITE) on the previous window, and do
> a wait (SYNC_FILE_RANGE_WAIT_AFTER) on the window before that.
>
> in fact, in its simplest form, you can do it like this (this is from my
> "overwrite disk images" program that I use on old disks):
>
> for (index = 0; index < max_index ;index++) {
> if (write(fd, buffer, BUFSIZE) != BUFSIZE)
> break;
> /* This won't block, but will start writeout asynchronously */
> sync_file_range(fd, index*BUFSIZE, BUFSIZE, SYNC_FILE_RANGE_WRITE);
> /* This does a blocking write-and-wait on any old ranges */
> if (index)
> sync_file_range(fd, (index-1)*BUFSIZE, BUFSIZE, SYNC_FILE_RANGE_WAIT_BEFORE|SYNC_FILE_RANGE_WRITE|SYNC_FILE_RANGE_WAIT_AFTER);
> }
>
> and even if you don't actually do a discard (maybe we should add a
> SYNC_FILE_RANGE_DISCARD bit, right now you'd need to do a separate
> fadvise(FADV_DONTNEED) to throw it out) the system behavior is pretty
> nice, because the heavy writer gets good IO performance _and_ leaves only
> easy-to-free pages around after itself.

Great! I just implemented it in LTTng and it works very well !

A faced a small counter-intuitive fadvise behavior though.

posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);

only seems to affect the parts of a file that already exist. So after each
splice() that appends to the file, I have to call fadvise again. I would have
expected the "0" len parameter to tell the kernel to apply the hint to the whole
file, even parts that will be added in the future. I expect we have this
behavior because fadvise() was initially made with read behavior in mind rather
than write.

For the records, I do a fadvice+async range write after each splice(). Also,
after each subbuffer write, I do a blocking write-and-wait on all pages that are
in the subbuffer prior to the one that has just been written, instead of using
the fixed 8MB window.

Thanks,

Mathieu

--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds on


On Wed, 19 May 2010, Mathieu Desnoyers wrote:
>
> A faced a small counter-intuitive fadvise behavior though.
>
> posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
>
> only seems to affect the parts of a file that already exist.

POSIX_FADV_DONTNEED does not have _any_ long-term behavior. So when you do
a

posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);

it only affects the pages that are there right now, it has no effect on
any future actions.

> So after each splice() that appends to the file, I have to call fadvise
> again. I would have expected the "0" len parameter to tell the kernel to
> apply the hint to the whole file, even parts that will be added in the
> future.

It's not a hint about future at all. It's a "throw current pages away".

I would also suggest against doing that kind of thing in a streaming write
situation. The behavior for dirty page writeback is _not_ welldefined, and
if you do POSIX_FADV_DONTNEED, I would suggest you do it as part of that
writeback logic, ie you do it only on ranges that you have just waited on.

IOW, in my example, you'd couple the

sync_file_range(fd, (index-1)*BUFSIZE, BUFSIZE, SYNC_FILE_RANGE_WAIT_BEFORE|SYNC_FILE_RANGE_WRITE|SYNC_FILE_RANGE_WAIT_AFTER);

with a

posix_fadvise(fd, (index-1)*BUFSIZE, BUFSIZE, POSIX_FADV_DONTNEED);

afterwards to throw out the pages that you just waited for.

Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Mathieu Desnoyers on
* Linus Torvalds (torvalds(a)linux-foundation.org) wrote:
>
>
> On Wed, 19 May 2010, Mathieu Desnoyers wrote:
> >
> > A faced a small counter-intuitive fadvise behavior though.
> >
> > posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
> >
> > only seems to affect the parts of a file that already exist.
>
> POSIX_FADV_DONTNEED does not have _any_ long-term behavior. So when you do
> a
>
> posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
>
> it only affects the pages that are there right now, it has no effect on
> any future actions.

Hrm, someone should tell the author of posix_fadvise(2) about the benefit of
some clarifications (I'm CCing the manpage maintainer)

Quoting man posix_fadvise, annotated:


Programs can use posix_fadvise() to announce an intention to access
file data in a specific pattern in the future, thus allowing the kernel
to perform appropriate optimizations.

This only talks about future accesses, not past. From what I understand, you are
saying that in the writeback case it's better to think of posix_fadvise() as
applying to pages that have been written in the past too.


The advice applies to a (not necessarily existent) region starting at
offset and extending for len bytes (or until the end of the file if len
is 0) within the file referred to by fd. The advice is not binding; it
merely constitutes an expectation on behalf of the application.

This could be enhanced by saying that it applies up to the current file size if
0 is specified, and does not extend as the file grows. The formulation as it is
currently stated is a bit misleading.

> > So after each splice() that appends to the file, I have to call fadvise
> > again. I would have expected the "0" len parameter to tell the kernel to
> > apply the hint to the whole file, even parts that will be added in the
> > future.
>
> It's not a hint about future at all. It's a "throw current pages away".
>
> I would also suggest against doing that kind of thing in a streaming write
> situation. The behavior for dirty page writeback is _not_ welldefined, and
> if you do POSIX_FADV_DONTNEED, I would suggest you do it as part of that
> writeback logic, ie you do it only on ranges that you have just waited on.
>
> IOW, in my example, you'd couple the
>
> sync_file_range(fd, (index-1)*BUFSIZE, BUFSIZE, SYNC_FILE_RANGE_WAIT_BEFORE|SYNC_FILE_RANGE_WRITE|SYNC_FILE_RANGE_WAIT_AFTER);
>
> with a
>
> posix_fadvise(fd, (index-1)*BUFSIZE, BUFSIZE, POSIX_FADV_DONTNEED);
>
> afterwards to throw out the pages that you just waited for.

OK, so it's better to do the writeback as part of sync_file_range rather than
relying on the dirty page writeback to do it for us. I guess the I/O scheduler
will have more room to ensure that writes are contiguous.

Thanks for the feedback,

Mathieu

--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds on


On Wed, 19 May 2010, Mathieu Desnoyers wrote:
>
> Programs can use posix_fadvise() to announce an intention to access
> file data in a specific pattern in the future, thus allowing the kernel
> to perform appropriate optimizations.

It's true for some of them. The random-vs-linear behavior is a flag for
the future, for example (relevant for prefetching).

In fact, it's technically true even for DONTNEED. It's true that we won't
need the pages in the future! So we throw the pages away. But that means
that we throw the _current_ pages away.

If we actually touch pages later, than that obviously invalidates the fact
that we said 'DONTNEED' - we clearly needed them.

Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/