feeding a FIFO from PCI
in [Linux Hardware]
|
From: John Larkin on 12 Apr 2008 12:38 Hi, I'm working on a proposal to design a box that will control a scientific gadget. Our box will output frequency sweeps, arbitrary waveforms, a couple of dozen voltages that can be changed/ramped per user desires, and some discrete logic levels and triggers. One architecture would pack an Intel-cpu SBC and a custom board in a 2U rack box. The SBC would talk gigabit ethernet to the customer's system and PCI to our board. Something like this, maybe: http://us.kontron.com/index.php?id=226&cat=527&productid=1726 Our board would have a PCI interface driving a biggish FIFO, say 8k deep by 48 bits wide, inside an FPGA. A simple state machine/latch/mux thing repacks the 32-bit pci transfers into the input of the 48-bit wide fifo. The output side of the FIFO would be driving a fairly simple state machine; each fifo word has an opcode field and a data field, with different opcodes feeding various devices connected to the physics... dds synthesizers, ttl outputs, whatever. The state machine that unloads the fifo would run at 128 MHz, but one opcode is WAIT, so we can slow down operations to match the realtime needs of the experiment and reduce the average fifo feed rate. OK, we finally get to a question: If we run some flavor of Linux on the SBC, what's a good strategy for keeping the fifo loaded? Assuming that we have the recipe for an entire experimental shot in program ram, some tens of megabytes maybe, we could... 1. Have the fifo logic interrupt the cpu when the fifo is, say, half empty. The isr would compute how empty the fifo actually is at that instant and set up a short dma transfer to top it off. 2. A task (or isr) would be run periodically, a thousand times per second might work, and it would be responsible for topping off the fifo, either dma or maybe just poking in the data in a loop. 3. Best, if possible: set up a single DMA transfer to do the entire shot. That involves a dma controller that understands that the target is sometimes busy, and retries after getting bounced. I know the pci bus has hooks for split transfers, but I don't know if standard Intel-type dma controllers can work in this mode. 4. If it's a dual-core cpu, is it hard (under Linux) to assign one cpu to just do the fifo transfers? 5. Other ideas? The problem is not unlike feeding a sound card. How does that work? Maybe we could start with a sound card driver and hack that? Any suggestions for resources? Books, model drivers, references, people we could hire to write the drivers for us? Being a hardware guy, mostly analog at that, I don't know much about this stuff. Right now, I only have to write up a plausible proposed architecture, but if we get the job, p>=0.5 maybe, we'll have to do actually it. Thanks! John
From: Jamie on 12 Apr 2008 13:35 John Larkin wrote: > Hi, > > I'm working on a proposal to design a box that will control a > scientific gadget. Our box will output frequency sweeps, arbitrary > waveforms, a couple of dozen voltages that can be changed/ramped per > user desires, and some discrete logic levels and triggers. > > One architecture would pack an Intel-cpu SBC and a custom board in a > 2U rack box. The SBC would talk gigabit ethernet to the customer's > system and PCI to our board. > > Something like this, maybe: > > http://us.kontron.com/index.php?id=226&cat=527&productid=1726 > > Our board would have a PCI interface driving a biggish FIFO, say 8k > deep by 48 bits wide, inside an FPGA. A simple state machine/latch/mux > thing repacks the 32-bit pci transfers into the input of the 48-bit > wide fifo. The output side of the FIFO would be driving a fairly > simple state machine; each fifo word has an opcode field and a data > field, with different opcodes feeding various devices connected to the > physics... dds synthesizers, ttl outputs, whatever. The state machine > that unloads the fifo would run at 128 MHz, but one opcode is WAIT, so > we can slow down operations to match the realtime needs of the > experiment and reduce the average fifo feed rate. > > OK, we finally get to a question: If we run some flavor of Linux on > the SBC, what's a good strategy for keeping the fifo loaded? Assuming > that we have the recipe for an entire experimental shot in program > ram, some tens of megabytes maybe, we could... > > 1. Have the fifo logic interrupt the cpu when the fifo is, say, half > empty. The isr would compute how empty the fifo actually is at that > instant and set up a short dma transfer to top it off. > > 2. A task (or isr) would be run periodically, a thousand times per > second might work, and it would be responsible for topping off the > fifo, either dma or maybe just poking in the data in a loop. > > 3. Best, if possible: set up a single DMA transfer to do the entire > shot. That involves a dma controller that understands that the target > is sometimes busy, and retries after getting bounced. I know the pci > bus has hooks for split transfers, but I don't know if standard > Intel-type dma controllers can work in this mode. > > 4. If it's a dual-core cpu, is it hard (under Linux) to assign one cpu > to just do the fifo transfers? > > 5. Other ideas? > > The problem is not unlike feeding a sound card. How does that work? > Maybe we could start with a sound card driver and hack that? > > Any suggestions for resources? Books, model drivers, references, > people we could hire to write the drivers for us? Being a hardware > guy, mostly analog at that, I don't know much about this stuff. > > Right now, I only have to write up a plausible proposed architecture, > but if we get the job, p>=0.5 maybe, we'll have to do actually it. > > Thanks! > > John > > If you want my opinion, I've been down that road using DMA controllers and trying to keep up with the technology as it changes.. I found that having a self contained Buffer on board (stand Alone) is the best, using a common link like RJ-45/ Ethernet via a TCP to access the information from the buffers a much better approach! that is, if you time critical isn't a problem to synchronize something? Using a 1GB or even the 100 Mb links should yield good results.. With that, You can support any platform that has a Ethernet interface. if you want to go the route of linux or freeBSD, you can compile those as embedded operations in which case the full attention of the system can stay with your PCI card how ever, I would still use your own On board cache. The better sound cards for example do a min of 512 bytes and up on their own onboard cache before transporting that to the system because other hardware activities can stall the board. That is just my opinion of course. http://webpages.charter.net/jamie_5"
From: rickman on 12 Apr 2008 13:34 On Apr 12, 12:38 pm, John Larkin <jjlar...(a)highNOTlandTHIStechnologyPART.com> wrote: > Hi, > > I'm working on a proposal to design a box that will control a > scientific gadget. Our box will output frequency sweeps, arbitrary > waveforms, a couple of dozen voltages that can be changed/ramped per > user desires, and some discrete logic levels and triggers. > > One architecture would pack an Intel-cpu SBC and a custom board in a > 2U rack box. The SBC would talk gigabit ethernet to the customer's > system and PCI to our board. > > Something like this, maybe: > > http://us.kontron.com/index.php?id=226&cat=527&productid=1726 > > Our board would have a PCI interface driving a biggish FIFO, say 8k > deep by 48 bits wide, inside an FPGA. A simple state machine/latch/mux > thing repacks the 32-bit pci transfers into the input of the 48-bit > wide fifo. The output side of the FIFO would be driving a fairly > simple state machine; each fifo word has an opcode field and a data > field, with different opcodes feeding various devices connected to the > physics... dds synthesizers, ttl outputs, whatever. The state machine > that unloads the fifo would run at 128 MHz, but one opcode is WAIT, so > we can slow down operations to match the realtime needs of the > experiment and reduce the average fifo feed rate. > 5. Other ideas? I don't see the real need to tightly couple the FIFO to your Intel computer. Why not give the FPGA its own memory, either DRAM or Flash with some "offline" interface to load the entire experiment into it? You can still put your board in the 2U box with the Intel computer, but give it a USB port or something similar which wouldn't be too hard to program up via an MCU (making the MCU and DRAM look like a storage device should be a good solution). Or if you have any reason to separate your board and the Intel computer, you could have the data transferred via an SD card or similar. Certainly a PCI interface is not rocket science, but talking to your board in real time is likely a bit more work than other solutions. I did the FPGA design for a similar project to record and playback high speed data. We used a DRAM buffer on the card and had to stream data to/from a RAID hard drive. The software *was* the long pole in the tent, even using a real time version of NT. So speaking from experience, you might be better off giving the FPGA plenty of RAM/ Flash and a less direct connection to the Intel computer. Are your experiments likely to need more than a few gig of instructions?
From: Fred Bloggs on 12 Apr 2008 14:12 John Larkin wrote: > Hi, > > I'm working on a proposal to design a box that will control a > scientific gadget. Our box will output frequency sweeps, arbitrary > waveforms, a couple of dozen voltages that can be changed/ramped per > user desires, and some discrete logic levels and triggers. > > One architecture would pack an Intel-cpu SBC and a custom board in a > 2U rack box. The SBC would talk gigabit ethernet to the customer's > system and PCI to our board. > > Something like this, maybe: > > http://us.kontron.com/index.php?id=226&cat=527&productid=1726 > > Our board would have a PCI interface driving a biggish FIFO, say 8k > deep by 48 bits wide, inside an FPGA. A simple state machine/latch/mux > thing repacks the 32-bit pci transfers into the input of the 48-bit > wide fifo. The output side of the FIFO would be driving a fairly > simple state machine; each fifo word has an opcode field and a data > field, with different opcodes feeding various devices connected to the > physics... dds synthesizers, ttl outputs, whatever. The state machine > that unloads the fifo would run at 128 MHz, but one opcode is WAIT, so > we can slow down operations to match the realtime needs of the > experiment and reduce the average fifo feed rate. > > OK, we finally get to a question: If we run some flavor of Linux on > the SBC, what's a good strategy for keeping the fifo loaded? Assuming > that we have the recipe for an entire experimental shot in program > ram, some tens of megabytes maybe, we could... > > 1. Have the fifo logic interrupt the cpu when the fifo is, say, half > empty. The isr would compute how empty the fifo actually is at that > instant and set up a short dma transfer to top it off. > > 2. A task (or isr) would be run periodically, a thousand times per > second might work, and it would be responsible for topping off the > fifo, either dma or maybe just poking in the data in a loop. > > 3. Best, if possible: set up a single DMA transfer to do the entire > shot. That involves a dma controller that understands that the target > is sometimes busy, and retries after getting bounced. I know the pci > bus has hooks for split transfers, but I don't know if standard > Intel-type dma controllers can work in this mode. > > 4. If it's a dual-core cpu, is it hard (under Linux) to assign one cpu > to just do the fifo transfers? > > 5. Other ideas? > > The problem is not unlike feeding a sound card. How does that work? > Maybe we could start with a sound card driver and hack that? > > Any suggestions for resources? Books, model drivers, references, > people we could hire to write the drivers for us? Being a hardware > guy, mostly analog at that, I don't know much about this stuff. > > Right now, I only have to write up a plausible proposed architecture, > but if we get the job, p>=0.5 maybe, we'll have to do actually it. > > Thanks! > > John > > You're in effect putting a microprogrammed CPU in that FPGA so what's the big deal with it fetching the new FIFO data for itself from an external store, and you can design the architecture for that any way you want, within the constraints of the OS that is, some kind of dual port job. Seems to be better than the micromanaging you propose...
From: Didi on 12 Apr 2008 14:59
John Larkin wrote: > ... >... Our box will output frequency sweeps, arbitrary > waveforms, a couple of dozen voltages that can be changed/ramped per > user desires, and some discrete logic levels and triggers. > > One architecture would pack an Intel-cpu SBC and a custom board in a > 2U rack box. The SBC would talk gigabit ethernet to the customer's > system and PCI to our board. > > ... > ... > > 5. Other ideas? > Since the echo you are getting so far indicates the latency using a x86 may well be too high under linux or whatever, I can suggest doing some tiny DPS thing for you - with PCI and Ethernet. The latency then is no issue, tcp/ip etc. comes with it, filesystem/disk etc. If you settle for 100 MbpS Ethernet, it is quite easy for me - I can reuse some of the MPC5200 designs I have. 1 GbpS it will take some other part and more than 3-4 months, though. I am not sure I can beat the cost&time of someone writing the thing for you under linux while I do the whole thing, but I am willing to try hard to do so, the time has come when I want to make all that stuff I have more popular than it is now. Dimiter ------------------------------------------------------ Dimiter Popoff Transgalactic Instruments http://www.tgi-sci.com ------------------------------------------------------ http://www.flickr.com/photos/didi_tgi/sets/72157600228621276/ Original message: http://groups.google.com/group/comp.arch.embedded/msg/3b61a5c2043b8477?dmode=source |