It could be pure out of memory, but it may be that both products are fighting over the OpenGL implementation that Adobe uses.
Try this - it may need some fiddling to get the best balance
[this is for Photoshop, idk the precise locations for other Adobe apps]
In Prefs > Performance, reduce the overall memory requirement - it ought to handle this OK by itself, running into pagefile if needed, but just to see how it goes with this reduced…
Either disable "Use Graphics Processor" or open Advanced Settings… & selectively disable sections until you find which is causing the issue.
If you decide to leave "Use Graphics Processor" on, then look in Prefs > 3D & drop the memory requirement in there to half your card's total memory, less a little for the system.
All these tweaks will require you to relaunch each app, so it may take some time to finally settle at something that works best for you.
The alternative, as you first surmised, would be to get a more powerful graphics card with greater memory - if that would be even possible on a Macbook, which I somehow doubt.
Everymac states [for the high-end i7 machine]:
NVIDIA GeForce GT 330M with 512 MB of dedicated GDDR3 memory and Intel HD Graphics with 256 MB of RAM shared with the system -- that Mac OS X automatically switches between for better graphics performance and better battery life, respectively, depending on application use.
One thing you should definitely check is that it is at least switching over to the NVidia rather than using the Intel card.
gfxCardStatus would be the go-to tool for that kind of task.