Is there a way to change irq numbers of pcie cards ?
in [Kernel]
|
Prev: [RFC,PATCH 0/2] Common struct clk implementation, v3
Next: X86:reboot.c Add some dmi entries to pci_reboot_dmi_table.
From: Bert dd on 2 Jun 2010 08:10 Hi, Is there a way to change irq numbers of pcie cards ? I have a supermicro machine with 2 raid cards (lsi 9260-8i and lsi 9260-8e) and one nvidia GTX480 graphics card. The problem is that all these pcie cards are assigned the same irq number, and thus all the interrupts are routed to the same cpu which can not handle this. If I replace the nvidia GTX480 with an older graphics card(too old for our purposes (GTX295)), the graphics card gets a different irq number than the raid cards, the interrups of the graphics card are handled by a different cpu than those of the raid cards, and the read/write speed of the raid increases. I am reading/writing at 2.5 GB/s to the raid and I am uploading textures at 0.5 GB/s to the graphics hardware. Is there a way to change the irq numbers of the pcie cards ? I am using IOAPIC for interrupt routing. I am using ubuntu - kernel version 2.6.31-14-generic. thanks, Bert De Decker bert.dedecker(a)uhasselt.be -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Bjorn Helgaas on 2 Jun 2010 15:30 [I added linux-pci, where more PCI-savvy people will see it.] On Wednesday, June 02, 2010 06:08:33 am Bert dd wrote: > Hi, > Is there a way to change irq numbers of pcie cards ? > > I have a supermicro machine with 2 raid cards (lsi 9260-8i and lsi 9260-8e) and > one nvidia GTX480 graphics card. > The problem is that all these pcie cards are assigned the same irq number, > and thus all the interrupts are routed to the same cpu which can not > handle this. > If I replace the nvidia GTX480 with an older graphics card(too old for > our purposes (GTX295)), the graphics card gets a different irq number > than the raid cards, the interrups of the graphics card are handled by > a different cpu than those of the raid cards, and the read/write speed > of the raid increases. > I am reading/writing at 2.5 GB/s to the raid and I am uploading > textures at 0.5 GB/s to the graphics hardware. > Is there a way to change the irq numbers of the pcie cards ? > I am using IOAPIC for interrupt routing. I would think performance-oriented PCIe devices would be using some sort of MSI, but I'm not really an expert in that area. Can you post your /proc/interrupts and complete dmesg log in case nobody jumps in with an obvious answer? Bjorn > I am using ubuntu - kernel version 2.6.31-14-generic. > > thanks, > Bert De Decker > bert.dedecker(a)uhasselt.be -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Robert Hancock on 2 Jun 2010 19:20 On 06/02/2010 06:08 AM, Bert dd wrote: > Hi, > Is there a way to change irq numbers of pcie cards ? > > I have a supermicro machine with 2 raid cards (lsi 9260-8i and lsi 9260-8e) and > one nvidia GTX480 graphics card. > The problem is that all these pcie cards are assigned the same irq number, > and thus all the interrupts are routed to the same cpu which can not > handle this. > If I replace the nvidia GTX480 with an older graphics card(too old for > our purposes (GTX295)), the graphics card gets a different irq number > than the raid cards, the interrups of the graphics card are handled by > a different cpu than those of the raid cards, and the read/write speed > of the raid increases. > I am reading/writing at 2.5 GB/s to the raid and I am uploading > textures at 0.5 GB/s to the graphics hardware. > Is there a way to change the irq numbers of the pcie cards ? > I am using IOAPIC for interrupt routing. > I am using ubuntu - kernel version 2.6.31-14-generic. In general there's no way to change what interrupts get allocated to what slots, unless the BIOS has a setting somewhere to change it. If you can get MSI enabled for some of those cards, though, that should prevent them from sharing interrupts. What driver are you using for the nvidia cards? You can turn on MSI on the binary nvidia driver with the NVreg_EnableMSI=1 module parameter.. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Clemens Ladisch on 3 Jun 2010 03:50
Robert Hancock wrote: > On 06/02/2010 06:08 AM, Bert dd wrote: > > I have a supermicro machine with 2 raid cards (lsi 9260-8i and lsi 9260-8e) and > > one nvidia GTX480 graphics card. > > The problem is that all these pcie cards are assigned the same irq number, > > If you can get MSI enabled for some of those cards, though, that should > prevent them from sharing interrupts. The PCIe spec says that all PCIe devices must support MSI interrupts. (Whether they do in practice is another question ...) Regardless of the graphics card, enabling MSI for the RAID cards will prevent their interrupts from interfering with each other. Bert, you might want to try this completely untested patch: --- a/drivers/scsi/megaraid/megaraid_sas.c +++ b/drivers/scsi/megaraid/megaraid_sas.c @@ -3169,6 +3169,8 @@ megasas_probe_one(struct pci_dev *pdev, const struct pci_device_id *id) if (megasas_init_mfi(instance)) goto fail_init_mfi; + pci_enable_msi(pdev); + /* * Register IRQ */ @@ -3218,6 +3220,7 @@ megasas_probe_one(struct pci_dev *pdev, const struct pci_device_id *id) pci_set_drvdata(pdev, NULL); instance->instancet->disable_intr(instance->reg_set); free_irq(instance->pdev->irq, instance); + pci_disable_msi(pdev); megasas_release_mfi(instance); @@ -3354,6 +3357,7 @@ megasas_suspend(struct pci_dev *pdev, pm_message_t state) pci_set_drvdata(instance->pdev, instance); instance->instancet->disable_intr(instance->reg_set); free_irq(instance->pdev->irq, instance); + pci_disable_msi(pdev); pci_save_state(pdev); pci_disable_device(pdev); @@ -3416,6 +3420,8 @@ megasas_resume(struct pci_dev *pdev) tasklet_init(&instance->isr_tasklet, megasas_complete_cmd_dpc, (unsigned long)instance); + pci_enable_msi(pdev); + /* * Register IRQ */ @@ -3519,6 +3525,7 @@ static void __devexit megasas_detach_one(struct pci_dev *pdev) instance->instancet->disable_intr(instance->reg_set); free_irq(instance->pdev->irq, instance); + pci_disable_msi(pdev); megasas_release_mfi(instance); -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |