From: Lin Ming on

> * Peter Zijlstra <peterz(a)infradead.org> wrote:
>
> > On Thu, 2010-05-20 at 16:12 -0700, Greg KH wrote:
>
> > > How deep in the device tree are you really going to be
> > > caring about? It sounds like the large majority of
> > > events are only going to be coming from the "system"
> > > type objects (cpu, nodes, memory, etc.) and very few
> > > would be from things that we consider a 'struct
> > > device' today (like a pci, usb, scsi, or input, etc.)
> >
> > The general noise I hear from the hardware people is
> > that we'll see more and more device-level stuff - bus
> > bridges/controller and actual devices (GPUs, NICs etc.)
> > will be wanting to export performance metrics.
>
> There's (much) more:
>
> - laptops want to provide power level/usage metrics,
>
> - we could express a lot of special, lower level
> (transport specific) disk IO stats via events as well -
> without having to push those stats to a higher level
> (where it might not make sense). Currently such kinds
> of stats/metrics are very device/subsystem specific
> way, if they are provided at all.
>
> Also, we already have quite a few per device tracepoints
> upstream. Here are a few examples:
>
> - GPU tracepoints (trace_i915_gem_request_submit(), etc.)
> - WIFI tracepoints (trace_iwlwifi_dev_ioread32(), etc.)
> - block tracepoints (trace_block_bio_complete())
>
> So these would be attached to:
>
> # GEM events of drm/card0:
> /sys/devices/pci0000:00/0000:00:02.0/drm/card0/events/i915_gem_request_submit/
>
> # Wifi-ioread events of wlan0:
> /sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlan0/events/iwlwifi_dev_ioread32/
>
> # whole sdb disk events:
> /sys/block/sdb/events/block_bio_complete/
>
> # sdb1 partition events:
> /sys/block/sdb/sdb1/events/block_bio_complete/
>

The difficulty is how to know where each event should be attached to.

struct ftrace_event_call *call;

for_each_event(call, __start_ftrace_events, __stop_ftrace_events) {
/* where will this event be attached to? */
}

Any idea?

Thanks,
Lin Ming


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar on

* Lin Ming <ming.m.lin(a)intel.com> wrote:

>
> > * Peter Zijlstra <peterz(a)infradead.org> wrote:
> >
> > > On Thu, 2010-05-20 at 16:12 -0700, Greg KH wrote:
> >
> > > > How deep in the device tree are you really going to be
> > > > caring about? It sounds like the large majority of
> > > > events are only going to be coming from the "system"
> > > > type objects (cpu, nodes, memory, etc.) and very few
> > > > would be from things that we consider a 'struct
> > > > device' today (like a pci, usb, scsi, or input, etc.)
> > >
> > > The general noise I hear from the hardware people is
> > > that we'll see more and more device-level stuff - bus
> > > bridges/controller and actual devices (GPUs, NICs etc.)
> > > will be wanting to export performance metrics.
> >
> > There's (much) more:
> >
> > - laptops want to provide power level/usage metrics,
> >
> > - we could express a lot of special, lower level
> > (transport specific) disk IO stats via events as well -
> > without having to push those stats to a higher level
> > (where it might not make sense). Currently such kinds
> > of stats/metrics are very device/subsystem specific
> > way, if they are provided at all.
> >
> > Also, we already have quite a few per device tracepoints
> > upstream. Here are a few examples:
> >
> > - GPU tracepoints (trace_i915_gem_request_submit(), etc.)
> > - WIFI tracepoints (trace_iwlwifi_dev_ioread32(), etc.)
> > - block tracepoints (trace_block_bio_complete())
> >
> > So these would be attached to:
> >
> > # GEM events of drm/card0:
> > /sys/devices/pci0000:00/0000:00:02.0/drm/card0/events/i915_gem_request_submit/
> >
> > # Wifi-ioread events of wlan0:
> > /sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlan0/events/iwlwifi_dev_ioread32/
> >
> > # whole sdb disk events:
> > /sys/block/sdb/events/block_bio_complete/
> >
> > # sdb1 partition events:
> > /sys/block/sdb/sdb1/events/block_bio_complete/
> >
>
> The difficulty is how to know where each event should be attached to.
>
> struct ftrace_event_call *call;
>
> for_each_event(call, __start_ftrace_events, __stop_ftrace_events) {
> /* where will this event be attached to? */
> }
>
> Any idea?

Well, it cannot be automatic - for each subsystem it's a different sysfs
point. So it has to be specified in the TRACE_EVENT() definition or so.

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Lin Ming on

> From: Ingo Molnar <mingo(a)elte.hu>
>
> * Peter Zijlstra <peterz(a)infradead.org> wrote:
>
> > On Thu, 2010-05-20 at 16:12 -0700, Greg KH wrote:
>
> > > How deep in the device tree are you really going to be
> > > caring about? It sounds like the large majority of
> > > events are only going to be coming from the "system"
> > > type objects (cpu, nodes, memory, etc.) and very few
> > > would be from things that we consider a 'struct
> > > device' today (like a pci, usb, scsi, or input, etc.)
> >
> > The general noise I hear from the hardware people is
> > that we'll see more and more device-level stuff - bus
> > bridges/controller and actual devices (GPUs, NICs etc.)
> > will be wanting to export performance metrics.
>
> There's (much) more:
>
> - laptops want to provide power level/usage metrics,
>
> - we could express a lot of special, lower level
> (transport specific) disk IO stats via events as well -
> without having to push those stats to a higher level
> (where it might not make sense). Currently such kinds
> of stats/metrics are very device/subsystem specific
> way, if they are provided at all.
>
> Also, we already have quite a few per device tracepoints
> upstream. Here are a few examples:
>
> - GPU tracepoints (trace_i915_gem_request_submit(), etc.)
> - WIFI tracepoints (trace_iwlwifi_dev_ioread32(), etc.)
> - block tracepoints (trace_block_bio_complete())
>
> So these would be attached to:
>
> # GEM events of drm/card0:
> /sys/devices/pci0000:00/0000:00:02.0/drm/card0/events/i915_gem_request_submit/
>
> # Wifi-ioread events of wlan0:
> /sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlan0/events/iwlwifi_dev_ioread32/
>
> # whole sdb disk events:
> /sys/block/sdb/events/block_bio_complete/
>
> # sdb1 partition events:
> /sys/block/sdb/sdb1/events/block_bio_complete/
>
> And we also have 'software nodes' in /sys that have events
> upstream here and today. For example for SLAB we already
> have kmalloc/kfree tracepoints (trace_kmalloc() and
> trace_kfree()):
>
> # all kmalloc events:
> /sys/kernel/slab/events/
>
> # kmalloc events for sighand_cache:
> /sys/kernel/slab/sighand_cache/events/kmalloc/
>
> # kfree events for sighand_cache:
> /sys/kernel/slab/sighand_cache/events/kfree/
>
> In general the set of events we have upstream is growing
> along an exponential curve (there's over a hundred now,
> via tracepoints).
>
> They are either logically attached to the hardware
> topology of the system (as in the first set of examples
> above), or ae attached to the software/subsystem object
> topology of the kernel (some examples of which are
> described in the second set of examples above).
>
> Sometimes there are aliasing/filtering relationship
> between events, which is expressed very well via the
> hierarchy and granularity of /sysfs.
>
> New events would go into that topology there in a natural
> way.
>
> For example general hugepage tracepoints (should we
> introduce any) would go into the existing hugepage node:
>
> /sys/kernel/mm/hugepages/events/...
>
> All in one, all these existing and future events, both of
> hardware and software type, are literally begging to be
> attached to nodes in /sys :-)

(sorry for late response...)

There are so many events via tracepoints, how about below sysfs point
for each events?

1. kvm events
/sys/devices/system/kvm/kvm0/events/kvm_set_irq/
....
/sys/devices/system/kvm/kvm0/events/kvm_emulate_insn/

2. kvm_mmu events
/sys/devices/system/kvm/kvm0/events/kvm_mmu_pagetable_walk/
....
/sys/devices/system/kvm/kvm0/events/kvm_mmu_zap_page/

3. raw_syscalls events
/sys/kernel/events/sys_enter/
/sys/kernel/events/sys_exit/

4. mce events
/sys/devices/system/machinecheck/machinecheck0/mce_record/
....
/sys/devices/system/machinecheck/machinecheckN/mce_record/

5. sched events
/sys/kernel/events/sched_kthread_stop/
....
/sys/kernel/events/sched_stat_runtime/

6. irq events
/sys/kernel/events/irq_handler_entry/
....
/sys/kernel/events/softirq_exit/

7. timer events
/sys/kernel/events/timer_init/
....
/sys/kernel/events/itimer_expire/

8. signal events
/sys/kernel/events/signal_generate/
....
/sys/kernel/events/signal_lose_info/

9. workqueue events
/sys/kernel/events/workqueue_insertion/
....
/sys/kernel/events/workqueue_destruction/

10. lock events
/sys/kernel/events/lock_acquire/
....
/sys/kernel/events/lock_acquired/

11. module events
/sys/module/events/module_load/
....
/sys/module/events/module_request/

12. power events
/sys/power/events/power_start/
....
/sys/power/events/power_end/

13. kmem events
/sys/kernel/mm/events/kmalloc/
....
/sys/kernel/mm/events/mm_page_alloc_extfrag/

14. ext4 events
/sys/fs/events/ext4_free_inode/
....
/sys/fs/events/ext4_mb_buddy_bitmap_load/

15. jbd2 events
/sys/fs/events/jbd2_checkpoint/
....
/sys/fs/events/jbd2_cleanup_journal_tail/

16. xfs events
/sys/fs/events/xfs_attr_list_sf/
....
/sys/fs/events/xfs_log_recover_inode_skip/

17. gfs2 events
/sys/fs/events/gfs2_glock_state_change/
....
/sys/fs/events/gfs2_block_alloc/

18. block events
/sys/block/sda/events/block_rq_abort/
....
/sys/block/sdb/events/block_rq_remap/

19. bkl events
/sys/kernel/events/lock_kernel/
....
/sys/kernel/events/unlock_kernel/

20. scsi events
/sys/bus/scsi/devices/0:0:0:0/events/scsi_dispatch_cmd_start/
....
/sys/bus/scsi/devices/0:0:0:0/events/scsi_eh_wakeup/

21. iwlwifi_io events
/sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlan0/events/iwlwifi_dev_ioread32/
....
/sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlan0/events/iwlwifi_dev_ucode_event/

22. iwm events
<where should iwm events be in sysfs?>

23. skb events
/sys/class/net/events/kfree_skb/
/sys/class/net/events/skb_copy_datagram_iovec/

24. napi events
/sys/class/net/events/napi/

25. mac80211 events
/sys/class/net/events/drv_start/
....
/sys/class/net/events/stop_queue







--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Lin Ming on
On Tue, 2010-06-22 at 14:59 +0800, Johannes Berg wrote:
> On Tue, 2010-06-22 at 14:25 +0800, Lin Ming wrote:
> > On Mon, 2010-06-21 at 17:34 +0800, Johannes Berg wrote:
> > > On Mon, 2010-06-21 at 16:55 +0800, Lin Ming wrote:
> > >
> > > > 21. iwlwifi_io events
> > > > /sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlan0/events/iwlwifi_dev_ioread32/
> > > > ...
> > > > /sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/net/wlan0/events/iwlwifi_dev_ucode_event/
> > >
> > >
> > > That doesn't work, you could have multiple PCI devices in your system.
> >
> > Understood. This is just a "demo".
> >
> > Actually, I mean
> >
> > net/wlan0/events/
> > net/waln1/events/
> > ....
> > net/walnN/events/
>
> That's not appropriate either though since you may have multiple network
> interfaces on the same hardware :)

Doesn't net/wlan0...wlanN mean multiple network interfaces on the same
hardware?

If my understanding is wrong, would you show me an example in sysfs?

>
> > > > 23. skb events
> > > > /sys/class/net/events/kfree_skb/
> > > > /sys/class/net/events/skb_copy_datagram_iovec/
> > >
> > > > 25. mac80211 events
> > > > /sys/class/net/events/drv_start/
> > > > ...
> > > > /sys/class/net/events/stop_queue
> > >
> > > It doesn't really seem right to mix all these.
> >
> > Well, agree, skb events are totally different with mac80211 events.
> > Any idea?
>
> I suppose the most appropriate thing would
> be /sys/class/ieee80211/phyN/events/... for almost all of them.

Good idea.

Thanks,
Lin Ming

>
> johannes
>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Johannes Berg on
On Tue, 2010-06-22 at 15:22 +0800, Lin Ming wrote:

> > > net/wlan0/events/
> > > net/waln1/events/
> > > ....
> > > net/walnN/events/
> >
> > That's not appropriate either though since you may have multiple network
> > interfaces on the same hardware :)
>
> Doesn't net/wlan0...wlanN mean multiple network interfaces on the same
> hardware?

Yes, but the trace points aren't per network interface but rather per
hardware piece.

johannes

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/