According to Apple's specs listed under Tech Specs for the Mac Pro it should be able to handle 3 displays at up to
2560 by 1600 pixels over the digital connections. I think the trouble is coming from a little known about techology from AMD called Eyefinity. Which seems to require "Active" Adapters in order to run more than 2 displays at a time.
According to AMD who makes the ATI 5770 card.
What are the basic requirements for connecting multiple displays?
Connecting multiple monitors for AMD Eyefinity technology couldn’t be simpler. There are four easy rules to remember:
The first two monitors can connect to the graphics card with any display output on your product: HDMI, VGA, DVI or DisplayPort.
The third (or greater) display must be connected to the graphics card via DisplayPort.
If your monitor does not have a DisplayPort connection, you will need an inexpensive active DisplayPort adapter for it. DVI to DP adapters can be had for less than $30 USD.2
Every family of GPUs supports a different maximum number of displays.
It looks like you may have the passive adapters for the MiniDP to DVI adapters. and hence the trouble and why only 2 displays work when all DVI. Note that the MiniDP to VGA are active adapters.
With the 1920 X 1200 resoultion you will be fine with Single Link adapters, you just need active mini display port to DVI adapters. For some guidance on obtaining compatible adapters see AMD Eyefinity Validated Dongles
As noted in a similar question in the Apple Store:
Q:To use three monitors, does this work with the recently-available active single-link DP to DVI adapters (NOT the simple passive one that Apple [sold])?
Late last year, active single-link DVI adapters became available that function in the same way as the active dual-link adapter that Apple sells, but is intended for use with smaller, single-link monitors. These provide the same added intelligence that the dual-link adapters provide, but only support single-link resolutions. This makes them much cheaper (typically around $30) and they don't don't require the extra power of a USB port.
Can these adapters be used with this card in a mid-2010 (5,1) Mac Pro to drive the second and third monitors in a three-monitor setup?
Answer: To run 3 monitors with this card - 2 of them need to be "active" on the display port connections.
I run the card with 2 display port monitors (an HP and DELL) and 1 DVI monitor - (A 1080p projector with an hdmi/dvi passive adapter). I'm doing this in both a 2010 and 2008 mac pro.
I assume because of this, you can buy the active dual-link dvi/dp adapter and get the same results.
But I can confirm you can't do this with a passive adapter.