After several careful readings of your question, something stood out:
And on top of the huge amounts on 3G, this phone spent very little time outside Wifi.
Your iPhone, when it's in sleep mode, turns off the WiFi radio to conserve battery power. So if you have data hungry apps that are doing things in the background when your phone is sleeping/locked, they're doing it over 3G. Note that the WiFi-off-in-sleep-mode behaviour is somewhat inconsistent. But you can ensure 3G is never used by your phone while you're performing these tests by shutting off cellular data. That way, when it sleeps, it will continue to use WiFi and not 3G for data and you'll be certain to capture all the traffic in your monitoring net.
You can cobble together some technologies that might help you track down the culprit.
As a quick first pass you could try monitoring the phone when it's on a WiFi network using mitmproxy. From this excellent mclov.in blog post, you can setup a proxy for your phone on your Mac and filter all http/s traffic through it:
- Download the mitmproxy tool and set it up by going to the folder of mitmproxy and running sudo python setup.py install. If all goes well, mitmproxy must be available in your $PATH.
- Start mitmproxy by running mitmproxy.
- Obtain the IP address of your computer by running ifconfig en1 (or whatever is the interface that you are using).
- Set the proxy on your iPhone by going to your wireless settings, setting the proxy to be “Manual”, and set the IP to be your computer’s IP and the port as 8080.
If you set mitmproxy to record http/s calls to a file you can analyze it after some period of time and look for trends. You'll also be able to inspect the payloads on the http/s calls so you might able to spot unusually large requests this way.
Of course: this approach assumes the culprit is using http/s for communication. If it's not, you won't see the problem via mitmproxy and you have to resort to a lower-level solution.
You need to monitor all the traffic on a WiFi network you control to help you hunt down at least the end points for the data transfers. Bandwidth monitoring isn't hard to set up, but keeping a per-IP, per-endpoint log is really tough to do with consumer-grade hardware because it can be A MASSIVE amount of data. Every packet has to be checked for origin, destination, logged for time and size -- it adds up really quick.
But you can get some decent numbers and narrow down the search space so it's something potentially human-parseable.
First: you need to setup a network with bandwidth monitoring on a per-IP level. For that I recommend a Linksys WRT54GL consumer-grade WiFi router running Tomato and the TeamanIPTraffic add-on for IP-level bandwidth tracking.
Second: set your DNS on the router to use OpenDNS. In your OpenDNS account to track DNS lookups:
- Log in to https://dashboard.opendns.com/
- Click the 'Settings' tab
- Click the IP address for the network associated with your router
- Click 'Stats and Logs' on the left side
- Make sure 'Enable stats and logs' is checked and click the 'Apply' button
Now any DNS lookups performed on your network will be logged.
Assign your phone a static IP on this network so it's easier to track.
And...wait and watch.
You're going to monitor the activity for the IP address assigned to your phone in your router's per-IP activity tables. When the data activity for this IP address spikes you'll head over to your OpenDNS console and take a look at the stats and logs for your DNS lookups for the period of time when data movement was high for your phone.
That should give a short list [sic] of hostnames that were being resolved at the time. And armed with that list you might able to narrow it down to an application.
