You don't need any special tools like xcode or a debugger to pick apart the virtual memory system in terms of allocation and summary statistics.
I would start with
sysdiagnose from the command line and perhaps pass in the PID or process name you care about. It samples that app, but you also have the PID and information on that suspect program with all the diagnostic information. Passing in the process ID will run vmmap, heap and (perhaps) other good diagnostics or added details on the program in question.
You'll want to focus on the zprint.txt and allmemory.txt first. The sysctl.txt has some vm information as well and fs_usage.txt shows file access, but generally not swapping.
There are unix manual pages for most of these tools as well as the sysdiagnose script is readable so you can see what it does or change it to collect. You might want to enable thorough mode on the script in case that added information on the PID helps you isolate the cause.