DCO-O-Matic Batch processor

[Carsten]

It's hard for me to judge the virtual server thing. I did many many benchmarks over time, and, from my experience with some dual-CPU xeon machines, you should be able to get slightly faster speeds. I haven't seen much difference from the specific OS versions.

I am still a fan of the dual xeon CPU workstations like the 2*x5660 z600, or dual E5-2670 workstations (z620).

Benchmarks are here:

http://dcpomatic.com/benchmarks/fps
http://dcpomatic.com/benchmarks/input.php?id=1

10fps is not THAT much given that you have three machines running, and two of them are 8core xeons. But if you have such a slow main machine, and the two xeons networked, this may be in the ballpark what to expect from this setup. Definitely you are not far away from the theoretical maximum if you compare it to the above benchmarks.

I don't know how much footage you have to convert, but from my personal experience, I would feel you need more/faster machines or more time.

In the end, installing other OSs to get machines running with DOM will eat up time as well. So I wouldn't think about that and try to get as many machines going as quickly as possible, as the differences between different OSs will be small.

If you see that your machines have enough free RAM, you may increase the numer of encode threads on the servers to 12 or even 16. Though, with the recent versions of DCP-o-matic, this has shown little effect. But as long as the machines do not get into swapping, it doesn't hurt if you increase the number of computer threads by up to a factor of two. You definitely need a faster machine for the main GUI.

- Carsten

[Tudortech]

Hi,
I have setup some higher powered servers with dual E5-2430 6 core 12 thread. The dcp encoder server picks up that we have 24 threads available but only seems to be getting 12fps . should I be getting approx 1fps per thread or is that not possible.
Thanks
Alan

[Carsten]

What is the master GUI/batch processor running on?


The E5-2430 has many cores, but is only running at 2.2GHz. From my experience with dual Xeon x5660 running at 2.8GHz, you are probably in the right ballpark.

Can you get all these servers/virtual machines working together? In that case, the machine running master and batch processor needs to be fast as well to feed all the remote encoding threads.

- Carsten

[Tudortech]

Hi
Is it possible to run more than one master to feed the rendering farm. I am finding that when run individually my 12 core Xeon machines will give me 12-16fps but when run in the farm with other similar servers that they drop down to 4-6fps. I did notice that my Xeon based master was maxing out the gigabit link so I did nic teaming on the machine , do now have 2g bandwidth to layer2 switch. As we have many movies to render before the festival it would be helpful to have two masters feeding the farm. Pros/cons? Can the encoders handle input from multiple masters?
Thanks

[carl]

I'm not sure of the performance implications but the encoders can handle input from as many masters as you like (in theory...)

[Tudortech]

Hi,
when I run one of my encoders as a master I see a drop off in processing speed to the original master machine.
with four server running 12 core /24 thread and plenty of memory and hard drive space I am only getting 15-20fps when running from a master that has two E5-2690 @2.4ghz with 32GB memory and two gigabit teamed nics. I am not even reaching 50% on the networking and 50-60% on cpu use.
what could be slowing down the processing?
thanks

[Carsten]

Start the task manager on the master machine and have a look at the CPU load and thread balancing. I think 'being master' means running most things single-threaded. So, in that case, having the master only feeding the encode servers will not give the best performance. You should probably also allow the master to encode. Carl may comment on this, but only the J2K conversion (plus some initial prep like scaling, etc.) will actually benefit from multiple cores. And that will happen on the remote encoders.

So, a fast Desktop CPU, like 3-4GHz i7 etc. should be better suited as a master.

That said, it looks as if the 12 iMAC network you can see in the benchmark section had no trouble with a single iMac feeding close to 50 encode threads/s. The master and encoders were i7/3.4GHz.

http://dcpomatic.com/benchmarks/fps


Again, I suggest you look at the CPU load of the master and check how many threads are actually busy during 'mastering'.

I guess, a dual E5-2690 machine is better kept busy as an encode server.

Admittedly, I ran many benchmarks on local machine setups, but not that often in setups like you describe.

Are you working 2k or 4k, what type of source material do you convert?

- Carsten