Prev: [GIT PULL] perf fixlet for 2.6.34
Next: perf list documentation changes
From: Nigel Cunningham on 8 May 2010 01:30
Hi all.

I'm seeing a huge number of load balancing tick wakeups, with both
2.6.33 and current git (f1c448e0a9e99c76f4ece368714fb35a40a8daba).

This has been reported as a bug in some distros:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281

Regards,

Nigel

PowerTOP 1.12 (C) 2007, 2008 Intel Corporation

Collecting data for 15 seconds


Your CPU supports the following C-states : C1 C2 C3 C4
Your BIOS reports the following C-states : C1 C2 C4
Cn Avg residency
C0 (cpu running) (40.8%)
polling 0.7ms ( 0.1%)
C1 mwait 0.0ms ( 0.0%)
C2 mwait 0.1ms ( 3.2%)
C4 mwait 0.4ms (56.0%)
P-states (frequencies)
Turbo Mode 7.3%
2.21 Ghz 0.7%
1.60 Ghz 0.8%
1200 Mhz 0.9%
800 Mhz 90.3%
Wakeups-from-idle per second : 1727.3 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
53.0% (1983.1) [kernel scheduler] Load balancing tick
27.7% (1035.1) vmware-vmx
8.8% (330.6) [extra timer interrupt]
2.3% ( 85.5) [TLB shootdowns] <kernel IPI>
1.6% ( 61.3) totem
1.2% ( 45.9) [Rescheduling interrupts] <kernel IPI>
1.1% ( 41.1) [iwl3945] <interrupt>
1.0% ( 36.2) firefox-bin
0.4% ( 16.2) [ata_piix] <interrupt>
0.3% ( 11.6) vmware
0.3% ( 11.3) pulseaudio
0.3% ( 11.1) vmware-unity-he
0.3% ( 10.7) skype
0.2% ( 8.0) [kernel core] usb_hcd_poll_rh_status (rh_timer_func)
0.1% ( 5.4) [hda_intel] <interrupt>
0.1% ( 5.3) gnome-do
0.1% ( 5.1) compiz
0.1% ( 4.1) python
0.1% ( 4.0) avant-window-na
0.1% ( 2.3) liferea
0.1% ( 2.1) vmware-tray
0.1% ( 2.1) thunderbird-bin
0.1% ( 2.0) mysqld
0.1% ( 2.0) emerald
0.1% ( 1.9) gnome-terminal
0.0% ( 1.3) [kernel core] sk_reset_timer (tcp_delack_timer)
0.0% ( 1.3) [ahci] <interrupt>
0.0% ( 1.2) cc1
0.0% ( 1.0) [nvidia] <interrupt>
0.0% ( 1.0) vmware-usbarbit
0.0% ( 1.0) vmnet-natd
0.0% ( 1.0) gvfs-afc-volume
0.0% ( 1.0) [kernel core] enqueue_task_rt (sched_rt_period_timer)
0.0% ( 1.0) [kernel module] HostIF_InitUptime
(HostIFUptimeResyncMono)
0.0% ( 0.8) cli
0.0% ( 0.5) events/1
0.0% ( 0.5) events/0
0.0% ( 1.0) vmware-usbarbit
0.0% ( 1.0) vmnet-natd
0.0% ( 1.0) gvfs-afc-volume
0.0% ( 1.0) [kernel core] enqueue_task_rt (sched_rt_period_timer)
0.0% ( 1.0) [kernel module] HostIF_InitUptime
(HostIFUptimeResyncMono)
0.0% ( 0.8) cli
0.0% ( 0.5) events/1
0.0% ( 0.5) events/0
0.0% ( 0.5) hald-addon-stor
0.0% ( 0.5) udisks-daemon
0.0% ( 0.3) tomboy
0.0% ( 0.3) awn-applet
0.0% ( 0.2) rtkit-daemon
0.0% ( 0.2) Thunar
0.0% ( 0.2) padevchooser
0.0% ( 0.2) xfdesktop
0.0% ( 0.1) gnome-power-man
0.0% ( 0.1) ssh-agent
0.0% ( 0.1) mono
0.0% ( 0.1) NetworkManager
0.0% ( 0.1) hald

An audio device is active 100.0% of the time:
hwC0D2 SigmaTel STAC9228

A USB device is active 0.0% of the time:
USB device 2-2 : Biometric Coprocessor (STMicroelectronics)

Suggestion: Enable SATA ALPM link power management via:
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
or press the S key.

Suggestion: Enable wireless power saving mode by executing the following
command:
iwconfig wlan0 power timeout 500ms
This will sacrifice network performance slightly to save power.

Suggestion: Enable the CONFIG_USB_SUSPEND kernel configuration option.
This option will automatically disable UHCI USB when not in use, and may
save approximately 1 Watt of power.

Recent USB suspend statistics
Active Device name
0.0% USB device 5-2 : Generic RNDIS (HTC)
0.0% USB device 2-2 : Biometric Coprocessor (STMicroelectronics)
0.0% USB device 1-1 : Laptop Integrated Webcam (OmniVision
Technologies, Inc. -2640-07.07.20.3)
0.0% USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
0.0% USB device usb6 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
0.0% USB device usb5 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
0.0% USB device usb4 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)
0.0% USB device usb3 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
0.0% USB device usb2 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
0.0% USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)

Recent audio activity statistics
Active Device name
100.0% hwC0D2 SigmaTel STAC9228

Recent SATA AHCI link activity statistics
Active Partial Slumber Device name
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Len Brown on 8 May 2010 15:20
(linux-pm cc'd)

> I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and
> current git (f1c448e0a9e99c76f4ece368714fb35a40a8daba).
>
> This has been reported as a bug in some distros:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281

Unclear if your sighting is the same as those two.

What do you see if vmware is not present?
What do you see if you boot single user mode,
how about with maxcpusx=1?

cheers,
Len Brown, Intel Open Source Technology Center

> PowerTOP 1.12 (C) 2007, 2008 Intel Corporation
>
> Collecting data for 15 seconds
>
>
> Your CPU supports the following C-states : C1 C2 C3 C4
> Your BIOS reports the following C-states : C1 C2 C4
> Cn Avg residency
> C0 (cpu running) (40.8%)
> polling 0.7ms ( 0.1%)
> C1 mwait 0.0ms ( 0.0%)
> C2 mwait 0.1ms ( 3.2%)
> C4 mwait 0.4ms (56.0%)
> P-states (frequencies)
> Turbo Mode 7.3%
> 2.21 Ghz 0.7%
> 1.60 Ghz 0.8%
> 1200 Mhz 0.9%
> 800 Mhz 90.3%
> Wakeups-from-idle per second : 1727.3 interval: 15.0s
> no ACPI power usage estimate available
> Top causes for wakeups:
> 53.0% (1983.1) [kernel scheduler] Load balancing tick
> 27.7% (1035.1) vmware-vmx
> 8.8% (330.6) [extra timer interrupt]
> 2.3% ( 85.5) [TLB shootdowns] <kernel IPI>
> 1.6% ( 61.3) totem
> 1.2% ( 45.9) [Rescheduling interrupts] <kernel IPI>
> 1.1% ( 41.1) [iwl3945] <interrupt>
> 1.0% ( 36.2) firefox-bin
> 0.4% ( 16.2) [ata_piix] <interrupt>
> 0.3% ( 11.6) vmware
> 0.3% ( 11.3) pulseaudio
> 0.3% ( 11.1) vmware-unity-he
> 0.3% ( 10.7) skype
> 0.2% ( 8.0) [kernel core] usb_hcd_poll_rh_status (rh_timer_func)
> 0.1% ( 5.4) [hda_intel] <interrupt>
> 0.1% ( 5.3) gnome-do
> 0.1% ( 5.1) compiz
> 0.1% ( 4.1) python
> 0.1% ( 4.0) avant-window-na
> 0.1% ( 2.3) liferea
> 0.1% ( 2.1) vmware-tray
> 0.1% ( 2.1) thunderbird-bin
> 0.1% ( 2.0) mysqld
> 0.1% ( 2.0) emerald
> 0.1% ( 1.9) gnome-terminal
> 0.0% ( 1.3) [kernel core] sk_reset_timer (tcp_delack_timer)
> 0.0% ( 1.3) [ahci] <interrupt>
> 0.0% ( 1.2) cc1
> 0.0% ( 1.0) [nvidia] <interrupt>
> 0.0% ( 1.0) vmware-usbarbit
> 0.0% ( 1.0) vmnet-natd
> 0.0% ( 1.0) gvfs-afc-volume
> 0.0% ( 1.0) [kernel core] enqueue_task_rt (sched_rt_period_timer)
> 0.0% ( 1.0) [kernel module] HostIF_InitUptime (HostIFUptimeResyncMono)
> 0.0% ( 0.8) cli
> 0.0% ( 0.5) events/1
> 0.0% ( 0.5) events/0
> 0.0% ( 1.0) vmware-usbarbit
> 0.0% ( 1.0) vmnet-natd
> 0.0% ( 1.0) gvfs-afc-volume
> 0.0% ( 1.0) [kernel core] enqueue_task_rt (sched_rt_period_timer)
> 0.0% ( 1.0) [kernel module] HostIF_InitUptime (HostIFUptimeResyncMono)
> 0.0% ( 0.8) cli
> 0.0% ( 0.5) events/1
> 0.0% ( 0.5) events/0
> 0.0% ( 0.5) hald-addon-stor
> 0.0% ( 0.5) udisks-daemon
> 0.0% ( 0.3) tomboy
> 0.0% ( 0.3) awn-applet
> 0.0% ( 0.2) rtkit-daemon
> 0.0% ( 0.2) Thunar
> 0.0% ( 0.2) padevchooser
> 0.0% ( 0.2) xfdesktop
> 0.0% ( 0.1) gnome-power-man
> 0.0% ( 0.1) ssh-agent
> 0.0% ( 0.1) mono
> 0.0% ( 0.1) NetworkManager
> 0.0% ( 0.1) hald


> An audio device is active 100.0% of the time:
> hwC0D2 SigmaTel STAC9228
>
> A USB device is active 0.0% of the time:
> USB device 2-2 : Biometric Coprocessor (STMicroelectronics)
>
> Suggestion: Enable SATA ALPM link power management via:
> echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
> or press the S key.
>
> Suggestion: Enable wireless power saving mode by executing the following
> command:
> iwconfig wlan0 power timeout 500ms
> This will sacrifice network performance slightly to save power.
>
> Suggestion: Enable the CONFIG_USB_SUSPEND kernel configuration option.
> This option will automatically disable UHCI USB when not in use, and may
> save approximately 1 Watt of power.
>
> Recent USB suspend statistics
> Active Device name
> 0.0% USB device 5-2 : Generic RNDIS (HTC)
> 0.0% USB device 2-2 : Biometric Coprocessor (STMicroelectronics)
> 0.0% USB device 1-1 : Laptop Integrated Webcam (OmniVision Technologies,
> Inc. -2640-07.07.20.3)
> 0.0% USB device usb7 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
> 0.0% USB device usb6 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
> 0.0% USB device usb5 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
> 0.0% USB device usb4 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)
> 0.0% USB device usb3 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
> 0.0% USB device usb2 : UHCI Host Controller (Linux 2.6.34-rc6 uhci_hcd)
> 0.0% USB device usb1 : EHCI Host Controller (Linux 2.6.34-rc6 ehci_hcd)
>
> Recent audio activity statistics
> Active Device name
> 100.0% hwC0D2 SigmaTel STAC9228
>
> Recent SATA AHCI link activity statistics
> Active Partial Slumber Device name

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Nigel Cunningham on 8 May 2010 19:30
Hi Len.

On 09/05/10 05:16, Len Brown wrote:
> (linux-pm cc'd)
>
>> I'm seeing a huge number of load balancing tick wakeups, with both 2.6.33 and
>> current git (f1c448e0a9e99c76f4ece368714fb35a40a8daba).
>>
>> This has been reported as a bug in some distros:
>>
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=521944
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/524281
>
> Unclear if your sighting is the same as those two.
>
> What do you see if vmware is not present?

I still see a huge number, just not as huge. The number of load
balancing ticks seems to increase with the number of ticks from other
sources. Without vmware running I got:

Cn Avg residency
C0 (cpu running) (14.5%)
polling 0.2ms ( 0.0%)
C1 mwait 0.0ms ( 0.0%)
C2 mwait 0.1ms ( 1.7%)
C4 mwait 1.1ms (83.8%)
P-states (frequencies)
Turbo Mode 1.2%
2.21 Ghz 0.0%
1.60 Ghz 0.0%
1200 Mhz 0.1%
800 Mhz 98.7%
Wakeups-from-idle per second : 907.4 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
42.1% (435.6) [kernel scheduler] Load balancing tick
21.2% (219.6) [extra timer interrupt]
10.2% (105.6) plugin-containe
9.5% ( 98.1) pulseaudio
1.7% ( 17.9) [Rescheduling interrupts] <kernel IPI>
1.7% ( 17.1) [iwl3945] <interrupt>
1.4% ( 14.8) [kernel core] hrtimer_start (tick_sched_timer)
1.4% ( 14.3) [ata_piix] <interrupt>

> What do you see if you boot single user mode,
> how about with maxcpusx=1?

I'll try those later today and get back to you with more info.

Regards,

Nigel

> cheers,
> Len Brown, Intel Open Source Technology Center
>
>> PowerTOP 1.12 (C) 2007, 2008 Intel Corporation
>>
>> Collecting data for 15 seconds
>>
>>
>> Your CPU supports the following C-states : C1 C2 C3 C4
>> Your BIOS reports the following C-states : C1 C2 C4
>> Cn Avg residency
>> C0 (cpu running) (40.8%)
>> polling 0.7ms ( 0.1%)
>> C1 mwait 0.0ms ( 0.0%)
>> C2 mwait 0.1ms ( 3.2%)
>> C4 mwait 0.4ms (56.0%)
>> P-states (frequencies)
>> Turbo Mode 7.3%
>> 2.21 Ghz 0.7%
>> 1.60 Ghz 0.8%
>> 1200 Mhz 0.9%
>> 800 Mhz 90.3%
>> Wakeups-from-idle per second : 1727.3 interval: 15.0s
>> no ACPI power usage estimate available
>> Top causes for wakeups:
>> 53.0% (1983.1) [kernel scheduler] Load balancing tick
>> 27.7% (1035.1) vmware-vmx
>> 8.8% (330.6) [extra timer interrupt]
>> 2.3% ( 85.5) [TLB shootdowns]<kernel IPI>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Nigel Cunningham on 9 May 2010 19:50
Hi again.

On 09/05/10 05:16, Len Brown wrote:
> What do you see if vmware is not present?
> What do you see if you boot single user mode,
> how about with maxcpusx=1?

Repeated the tests, just booting into X (xfce) - except for the init S
test. No programs were started and wifi was turned off.

1) No vmware (etc)

Cn Avg residency
C0 (cpu running) ( 3.7%)
polling 0.0ms ( 0.0%)
C1 mwait 0.0ms ( 0.0%)
C2 mwait 0.1ms ( 0.3%)
C4 mwait 5.4ms (96.0%)
P-states (frequencies)
Turbo Mode 1.4%
2.21 Ghz 0.0%
1.60 Ghz 0.0%
1200 Mhz 0.0%
800 Mhz 98.6%
Wakeups-from-idle per second : 202.7 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
56.1% (128.5) [kernel scheduler] Load balancing tick
13.1% ( 30.1) [extra timer interrupt]
6.9% ( 15.9) [ata_piix] <interrupt>

2) Above + maxcpus=1

Cn Avg residency
C0 (cpu running) ( 4.0%)
polling 0.0ms ( 0.0%)
C1 mwait 0.0ms ( 0.0%)
C2 mwait 0.4ms ( 0.2%)
C4 mwait 16.0ms (95.9%)
P-states (frequencies)
Turbo Mode 6.0%
2.21 Ghz 0.0%
1.60 Ghz 0.0%
1200 Mhz 0.0%
800 Mhz 94.0%
Wakeups-from-idle per second : 64.5 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
39.6% ( 46.3) [kernel scheduler] Load balancing tick
12.3% ( 14.3) [ata_piix] <interrupt>

(No idea why you need a load balancing tick with one core!)

3) Init S, without maxcpus=1

Cn Avg residency
C0 (cpu running) ( 0.0%)
polling 0.0ms ( 0.0%)
C1 mwait 0.0ms ( 0.0%)
C2 mwait 0.1ms ( 0.0%)
C4 mwait 48.9ms (100.0%)
P-states (frequencies)
Turbo Mode 0.0%
2.21 Ghz 0.0%
1.60 Ghz 0.0%
1200 Mhz 0.0%
800 Mhz 100.0%
Wakeups-from-idle per second : 21.4 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
33.5% ( 10.3) [kernel core] hrtimer_start (tick_sched_timer)
16.1% ( 4.9) [kernel core] cursor_timer_handler (cursor_timer_handler)
15.9% ( 4.9) [kernel scheduler] Load balancing tick
12.8% ( 3.9) [kernel core] usb_hcd_poll_rh_status (rh_timer_func)
9.3% ( 2.9) [ahci] <interrupt>

I also tried the patch Dominik provided, and it didn't seem to make a
difference:

P-states (frequencies)
Turbo Mode 3.7%
2.21 Ghz 0.0%
1.60 Ghz 0.0%
1200 Mhz 0.6%
800 Mhz 95.7%
Wakeups-from-idle per second : 236.7 interval: 15.0s
no ACPI power usage estimate available
Top causes for wakeups:
55.4% (119.7) [kernel scheduler] Load balancing tick
9.7% ( 20.9) skype
7.7% ( 16.5) [ata_piix] <interrupt>

Regards,

Nigel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
 | 
Pages: 1
Prev: [GIT PULL] perf fixlet for 2.6.34
Next: perf list documentation changes