Prev: Grub Legacy or Grub 2
Next: what's the difference between SUSE and openSUSE?
From: Bob Bob on 15 Jul 2008 22:08
Well its on a SuSE system...

I am trying to squeeze a bit more I/O bandwidth out of a video
processing box w/out changing the hardware. Have a look at this dreadful
procinfo.

linux 2.6.11.4-21.17-default (geeko(a)buildhost) (gcc 3.3.5 20050117) #1
Fri Apr 6 08:42:34 UTC 2007 1CPU [wcethopes3.]

Memory: Total Used Free Shared Buffers
Mem: 255416 138680 116736 0 90028
Swap: 522072 8384 513688

Bootup: Sat Jun 7 12:11:32 2008 Load average: 3.02 5.18 4.48 1/60 9517

user: 8d 7:49:32.61 21.7% page in: 662448590 disk 1: 2396196r3256073w
nice: 1d 9:19:12.01 3.6% page out: 1239648452 disk 2:
21758237r144550637w
system: 3d 9:19:51.07 8.8% page act: 338016275 disk 3:
21515776r116912910w
IOwait: 14d 7:12:00.73 37.3% page dea: 267210034
hw irq: 9:34:16.62 1.0% page flt: 1424451211
sw irq: 6:19:26.88 0.7% swap in : 41230
idle : 10d 6:54:50.34 26.8% swap out: 19971
uptime: 38d 8:29:33.50 context : 2432943939

irq 0:3314254056 timer irq 9: 1 acpi, uhci_hcd
irq 1: 96 i8042 irq 11:1893463035 eth0
irq 2: 0 cascade [4] irq 12: 3
irq 6: 33 irq 14: 5652276 ide0
irq 7: 1 parport0 [3] irq 15: 377573222 ide1
irq 8: 2 rtc

(Sorry about the formatting)

Notice the huge amount of I/O wait time. There are two basic main
processes that run on the box, one is a capture system and the other
creates MPEG's. They are pretty I/O intensive. Nicing helps one over the
other (the capture is kind of time critical) but not dramatically so. I
think it is I/O bound!

There are three disks. hda is a ATA33 and hdc/hdd are ATA6 RAID0. It was
a cruel choice having to put both ATA6 drives on the secondary channel
but I didn't want one slowed by the ATA33.

I have done the usual things with hdparm, turning off acoustic
management 32 bit sync etc etc. Still have problems.

Was considering playing with buffer flushing times or maybe the file
system itself. It is running vanilla reiser.

Its a P3/933 with an Intel 815 chipset. It also has two other boxes
talking NFS to it doing the same MPG processing. (Kind of like
clustering. We originally had a cpu/grunt/time to process problem that
has now gone away in favour of the I/O congestion!)

I am looking for any useful speedup suggestions. If I cant find a
solution I'll have to do some pgm rewrite/rethinks. Hardware changes are
not an option. I also wonder if there is a way to "nice" a process for
less I/O use rather than cpu slices...

TIA Bob
From: Nikos Chantziaras on 16 Jul 2008 09:24
Bob Bob wrote:
> I am trying to squeeze a bit more I/O bandwidth out of a video
> processing box w/out changing the hardware. [...]
> [...]
> Was considering playing with buffer flushing times or maybe the file
> system itself. It is running vanilla reiser.

I'm not an expert, but over the years the usual thing I read about
Reiser vs ext3 is that Reiser is very fast with *small* files while ext3
is very fast and uses less CPU with *big* files.
From: Bob Bob on 16 Jul 2008 13:48
Hi Nikos

Okay thanks for that, something I didnt know!

I am mostly manipulating 80-100KByte files so I guess that qualifies as
small.

I note there is disk elevator adjustment that I may look at. (ie wait
for a list of read and writes to gather before making the heads move and
do it)

OBTW already using a RAM disk for preprocessing.

Cheers Bob

Nikos Chantziaras wrote:
>
>
> Bob Bob wrote:
>> I am trying to squeeze a bit more I/O bandwidth out of a video
>> processing box w/out changing the hardware. [...]
>> [...]
>> Was considering playing with buffer flushing times or maybe the file
>> system itself. It is running vanilla reiser.
>
> I'm not an expert, but over the years the usual thing I read about
> Reiser vs ext3 is that Reiser is very fast with *small* files while ext3
> is very fast and uses less CPU with *big* files.
From: Nikos Chantziaras on 16 Jul 2008 14:55
Will Honea wrote:
> Bob Bob wrote:
>
>> Hi Nikos
>>
>> Okay thanks for that, something I didnt know!
>>
>> I am mostly manipulating 80-100KByte files so I guess that qualifies as
>> small.
>>
>> I note there is disk elevator adjustment that I may look at. (ie wait
>> for a list of read and writes to gather before making the heads move and
>> do it)
>>
>> OBTW already using a RAM disk for preprocessing.
>
> With 256Mb RAM, I'm surprised that it will stagger along even without the
> RAM disk!

I didn't pay enough attention and missed that there's only 256MB RAM!
You are right; with anything less than 1GB you can't expect to get
performance on a video-editing box. Also, I'm a bit surprised to hear
"mostly manipulating 80-100KByte files". What kind of video processing
are we talking about here exactly? Videos usually mean multi-GB file sizes.
From: Bob Bob on 16 Jul 2008 17:51
Will/Nikos

Tnxs for your feedback. I tried to keep the question simple, but see I
need to elaborate!

I have certainly allocated 2x RAM to swap but note that it is barely
used. To repeat the numbers;

Memory: Total Used Free Shared Buffers
Mem: 255416 138680 116736 0 90028
Swap: 522072 8384 513688

Only a very small amount of swap is used and there is still 90MB of
buffer space. Depending on the weather that buffer space goes up to
about 120MB. This will of course all be disk cache.

I am sorry I didnt explain the system well enough. It isnt video editing
and it isnt interactive. It is more a surviellance system. I capture
from 6 IP cameras, jpg images at around 5-6FPS and 640x480. Each jpg
compressed filesize is in the order of 50-100K. The current capture uses
wget and works surprisingly well. Since I use noclobber I then have to
rename the files to a timestamped name. This obviously adds time to
processing that has to happen in real time, so I shell/fork out & to do
this. The right way is of course to write code that captures and writes
direct to a stamped filename. (I am aware that some GPL projects area
available already do this) That will be the next step if I can't resolve
the I/O bottleneck. The renaming process is set to inhibit the wget
process if it runs overtime, which it does as the I/O load gets higher.
The result is that their are gaps in the capture stream. (I use an input
and output directory and swap them when renaming is finished)

After capturing I use mjpegtools (mpeg2enc etc) to create 1FPS MPEG2
videos from the individual images. (It also does motion detect) These
are used to rough catch thieving events after which the 5FPS images are
checked and/or a 5FPS MPEG is created for the date/time in question. We
have some classic shots of people tucking shop items into their shorts,
pants, pockets and even socks! (Its a non profit org "thrift" store)
Whether talking a 1FPS or 5FPS MPEG video they are between 20 and 50MB
in size. (They are of varying time lengths)

So it isnt as badly loaded as you might think. At any one time there are
a lot of images on the server, maybe 1-2 million for 2-3 days. They are
broken up by camera to make find/sorts a little faster and I think NFS
also suffers with the large number of files per directory. I cant do an
ls for example as shell expansion runs out of space. (I use find and
xargs) During the development (one might say playing) process I hit the
mpeg creation limit problem first. I needed 30 hours to process the days
data. I then clustered two more machines in and read up/implemented
reducing the cpu grunt required for mpeg2enc. This was all originally at
1FPS. I now find I have hit an I/O rather than cpu limit trying to
increase the frame rate.

Apologies for the length.

I dont think RAM will help all that much. The actual I/O rate numbers
look like;

The capture side is maybe 3Mbytes/sec but that goes to the ramdisk. The
moving/renaming will be around the same rate but of course writes to the
real RAID0 disk. The 1FPS mpeg creation for an hours worth of data per
cameras takes say 15 minutes. Thats about 350MB of reads or
0.5Mbytes/sec. There are three of these running plus a bit of other I/O
that wouild on average maybe an extra 50% (I have to recreate reference
images and that uses "convert") Say all up 3MBytes/sec. About the same
as the capture rate. hdparm -tT for one of the RAID0 devices shows about
20Mbytes/sec buffered disk reads. Maybe there is a lot more I/O going on
than I thought? Have to look at the mpeg creation script some more..

Would still like to hear your ideas.

Cheers Bob

Nikos Chantziaras wrote:
>> With 256Mb RAM, I'm surprised that it will stagger along even without the
>> RAM disk!
> What kind of video processing
> are we talking about here exactly? Videos usually mean multi-GB file
> sizes.
 |  Next  |  Last
Pages: 1 2 3 4
Prev: Grub Legacy or Grub 2
Next: what's the difference between SUSE and openSUSE?