Prev: CONFIDENTIAL
Next: asm-generic: add NEED_SG_DMA_LENGTH to define sg_dma_len()
From: Zan Lynx on 15 Apr 2010 19:50
On 4/15/10 4:53 PM, Andrew Morton wrote:

> This just isn't an interesting case. World-wide, the number of people
> who compile their own web browser and execute it from the file which ld
> produced is, umm, seven.

Gentoo users? Linux From Scratch?

There are many more than 7 of us. Unless you are talking about the build
environments always running some tool after ld which I am not aware of.


--
Zan Lynx
zlynx(a)acm.org

"Knowledge is Power. Power Corrupts. Study Hard. Be Evil."
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Andrew Morton on 15 Apr 2010 19:50
On Thu, 15 Apr 2010 17:21:25 -0600 Zan Lynx <zlynx(a)acm.org> wrote:

> On 4/15/10 4:53 PM, Andrew Morton wrote:
>
> > This just isn't an interesting case. World-wide, the number of people
> > who compile their own web browser and execute it from the file which ld
> > produced is, umm, seven.
>
> Gentoo users? Linux From Scratch?
>
> There are many more than 7 of us. Unless you are talking about the build
> environments always running some tool after ld which I am not aware of.
>

OK, eight then.

But I still don't think it's the case we should optimise for. Not if
it impacts the common case even the slightest. It'd be far far better
to change those distros to perform the very cheap, once-off step of
straightening out their executables (including shared libraries).

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Taras Glek on 15 Apr 2010 20:50
On 04/15/2010 03:53 PM, Andrew Morton wrote:
> On Mon, 05 Apr 2010 15:43:02 -0700
> Taras Glek<tglek(a)mozilla.com> wrote:
>
>
>> To make matters worse,
>> the compile-time linker + gcc lay out code in a manner that does not
>> correspond to how the resulting executable will be executed(ie the
>> layout is basically random).
>>
> Yes, the linker scrambles the executable's block ordering.
>
> This just isn't an interesting case. World-wide, the number of people
> who compile their own web browser and execute it from the file which ld
> produced is, umm, seven.
>
I'm sorry that you don't find this interesting. I did not suggest that
people compile their own browser to get a perfect layout. This is
something that Mozilla can do when preparing builds and it's also
something distributions can do. It just so happens that large parts of
startup will be very similar for every single firefox install, might as
well layout the binary accordingly.
> So I'd suggest that you always copy the executable to a temp file and
> mv it back before running any timing tests.
>
You mean to get it into a cache or to hope to avoid fragmentation? If
you are suggesting this to avoid measuring the startup overhead of
paging the binary in, I strongly disagee. It is the slowest part of
firefox startup and needs to be addressed.

Taras


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Andrew Morton on 15 Apr 2010 21:30
On Thu, 15 Apr 2010 17:41:48 -0700 Taras Glek <tglek(a)mozilla.com> wrote:

> On 04/15/2010 03:53 PM, Andrew Morton wrote:
> > On Mon, 05 Apr 2010 15:43:02 -0700
> > Taras Glek<tglek(a)mozilla.com> wrote:
> >
> >
> >> To make matters worse,
> >> the compile-time linker + gcc lay out code in a manner that does not
> >> correspond to how the resulting executable will be executed(ie the
> >> layout is basically random).
> >>
> > Yes, the linker scrambles the executable's block ordering.
> >
> > This just isn't an interesting case. World-wide, the number of people
> > who compile their own web browser and execute it from the file which ld
> > produced is, umm, seven.
> >
>
> I'm sorry that you don't find this interesting.

It's not a case we should optimise for. It's perfectly reasonable for
the kernel to assume that the executable is reasonably well-laid-out on
disk. And if is _isn't_ well-laid-out than that should be fixed in
userspace, because for simple locality-of-reference reasons, that's
always going to produce the fastest result.

Plus it's the common case as well - the executable was copied from DVD
or over the network or whatever.

Plus it's so utterly trivial for people who compile-their-own to
straighten the file out - just run cp! These people have gone and
screwed up their file layout - they should fix that, rather than trying
to get the kernel to perform the impossible for them. See?

> I did not suggest that
> people compile their own browser to get a perfect layout. This is
> something that Mozilla can do when preparing builds and it's also
> something distributions can do. It just so happens that large parts of
> startup will be very similar for every single firefox install, might as
> well layout the binary accordingly.
> > So I'd suggest that you always copy the executable to a temp file and
> > mv it back before running any timing tests.
> >
> You mean to get it into a cache or to hope to avoid fragmentation? If
> you are suggesting this to avoid measuring the startup overhead of
> paging the binary in, I strongly disagee. It is the slowest part of
> firefox startup and needs to be addressed.

No, nothing like that at all.

What I'm saying is that you shouldn't be testing or attempting to
optimise for files which were laid out by ld. Because those files are
an utter mess - the block ordering is simply all over the place. And
the great majority of people aren't using executables which were laid out
on disk by ld!

Instead, straighten out the block layout with `cp', then go and do the
testing and the optimisation. Because if you're not taking this first
step then you're just not serious about performance at all!

Here's a small executable, as laid out by ld:

File offset disk blocks
0-0: 18383385-18383385 (1)
1-1: 18383389-18383389 (1)
2-3: 18383392-18383393 (2)
4-4: 18383400-18383400 (1)
5-7: 18383430-18383432 (3)
8-11: 18383450-18383453 (4)
12-12: 18383423-18383423 (1)
13-14: 18383447-18383448 (2)
15-16: 18383474-18383475 (2)
17-17: 18383390-18383390 (1)
18-18: 18383398-18383398 (1)
19-20: 18383418-18383419 (2)
21-21: 18383421-18383421 (1)
22-22: 18383397-18383397 (1)
23-23: 18383399-18383399 (1)
24-24: 18383407-18383407 (1)
25-25: 18383391-18383391 (1)
26-26: 18383396-18383396 (1)
27-28: 18383394-18383395 (2)
29-34: 18383401-18383406 (6)
35-38: 18383425-18383428 (4)
39-39: 18383433-18383433 (1)
40-40: 18383463-18383463 (1)
41-44: 18383490-18383493 (4)
45-45: 18383409-18383409 (1)
46-46: 18383422-18383422 (1)
47-47: 18383442-18383442 (1)
48-48: 18383410-18383410 (1)
49-49: 18383420-18383420 (1)
50-50: 18383424-18383424 (1)
51-51: 18383429-18383429 (1)
52-54: 18383411-18383413 (3)
55-56: 18383416-18383417 (2)
57-64: 18383434-18383441 (8)
65-66: 18383458-18383459 (2)
67-68: 18383414-18383415 (2)
69-70: 18383387-18383388 (2)
71-71: 18383408-18383408 (1)
72-74: 18383443-18383445 (3)

Not only is it fragmented, it's also in jumbled-up order.

And here it is after I did `cp':

0-11: 18391043-18391054 (12)
12-15: 18391056-18391059 (4)
16-74: 18391064-18391122 (59)

Trying to get the kernel to fix up the first case is daft, when it is
so easy to fix and so obviously _needs_ fixing.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Taras Glek on 15 Apr 2010 22:40
On 04/15/2010 03:21 PM, Andrew Morton wrote:
> On Thu, 15 Apr 2010 17:41:48 -0700 Taras Glek<tglek(a)mozilla.com> wrote:
>
>
>> On 04/15/2010 03:53 PM, Andrew Morton wrote:
>>
>>> On Mon, 05 Apr 2010 15:43:02 -0700
>>> Taras Glek<tglek(a)mozilla.com> wrote:
>>>
>>>
>>>
>>>> To make matters worse,
>>>> the compile-time linker + gcc lay out code in a manner that does not
>>>> correspond to how the resulting executable will be executed(ie the
>>>> layout is basically random).
>>>>
>>>>
>>> Yes, the linker scrambles the executable's block ordering.
>>>
>>> This just isn't an interesting case. World-wide, the number of people
>>> who compile their own web browser and execute it from the file which ld
>>> produced is, umm, seven.
>>>
>>>
>> I'm sorry that you don't find this interesting.
>>
> It's not a case we should optimise for. It's perfectly reasonable for
> the kernel to assume that the executable is reasonably well-laid-out on
> disk. And if is _isn't_ well-laid-out than that should be fixed in
> userspace, because for simple locality-of-reference reasons, that's
> always going to produce the fastest result.
>
> Plus it's the common case as well - the executable was copied from DVD
> or over the network or whatever.
>
> Plus it's so utterly trivial for people who compile-their-own to
> straighten the file out - just run cp! These people have gone and
> screwed up their file layout - they should fix that, rather than trying
> to get the kernel to perform the impossible for them. See?
>
>
>> I did not suggest that
>> people compile their own browser to get a perfect layout. This is
>> something that Mozilla can do when preparing builds and it's also
>> something distributions can do. It just so happens that large parts of
>> startup will be very similar for every single firefox install, might as
>> well layout the binary accordingly.
>>
>>> So I'd suggest that you always copy the executable to a temp file and
>>> mv it back before running any timing tests.
>>>
>>>
>> You mean to get it into a cache or to hope to avoid fragmentation? If
>> you are suggesting this to avoid measuring the startup overhead of
>> paging the binary in, I strongly disagee. It is the slowest part of
>> firefox startup and needs to be addressed.
>>
> No, nothing like that at all.
>
> What I'm saying is that you shouldn't be testing or attempting to
> optimise for files which were laid out by ld. Because those files are
> an utter mess - the block ordering is simply all over the place. And
> the great majority of people aren't using executables which were laid out
> on disk by ld!
>
> Instead, straighten out the block layout with `cp', then go and do the
> testing and the optimisation. Because if you're not taking this first
> step then you're just not serious about performance at all!
>
> Here's a small executable, as laid out by ld:
>
> File offset disk blocks
> 0-0: 18383385-18383385 (1)
> 1-1: 18383389-18383389 (1)
> 2-3: 18383392-18383393 (2)
> 4-4: 18383400-18383400 (1)
> 5-7: 18383430-18383432 (3)
> 8-11: 18383450-18383453 (4)
> 12-12: 18383423-18383423 (1)
> 13-14: 18383447-18383448 (2)
> 15-16: 18383474-18383475 (2)
> 17-17: 18383390-18383390 (1)
> 18-18: 18383398-18383398 (1)
> 19-20: 18383418-18383419 (2)
> 21-21: 18383421-18383421 (1)
> 22-22: 18383397-18383397 (1)
> 23-23: 18383399-18383399 (1)
> 24-24: 18383407-18383407 (1)
> 25-25: 18383391-18383391 (1)
> 26-26: 18383396-18383396 (1)
> 27-28: 18383394-18383395 (2)
> 29-34: 18383401-18383406 (6)
> 35-38: 18383425-18383428 (4)
> 39-39: 18383433-18383433 (1)
> 40-40: 18383463-18383463 (1)
> 41-44: 18383490-18383493 (4)
> 45-45: 18383409-18383409 (1)
> 46-46: 18383422-18383422 (1)
> 47-47: 18383442-18383442 (1)
> 48-48: 18383410-18383410 (1)
> 49-49: 18383420-18383420 (1)
> 50-50: 18383424-18383424 (1)
> 51-51: 18383429-18383429 (1)
> 52-54: 18383411-18383413 (3)
> 55-56: 18383416-18383417 (2)
> 57-64: 18383434-18383441 (8)
> 65-66: 18383458-18383459 (2)
> 67-68: 18383414-18383415 (2)
> 69-70: 18383387-18383388 (2)
> 71-71: 18383408-18383408 (1)
> 72-74: 18383443-18383445 (3)
>
> Not only is it fragmented, it's also in jumbled-up order.
>
> And here it is after I did `cp':
>
> 0-11: 18391043-18391054 (12)
> 12-15: 18391056-18391059 (4)
> 16-74: 18391064-18391122 (59)
>
> Trying to get the kernel to fix up the first case is daft, when it is
> so easy to fix and so obviously _needs_ fixing.
>
Yeah ok. We are talking about different things. I meant the linker lays
out the program badly, ie within the executable itself. Turns out that
naively concatenating various compilation units makes for binaries that
load slowly due to excessive seeking within the file. I wasn't talking
about filesystem fragmentation. I agree that filesystem bustage caused
by executing the linker isn't interesting.

Taras
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4 5 6 7
Prev: CONFIDENTIAL
Next: asm-generic: add NEED_SG_DMA_LENGTH to define sg_dma_len()