2

I'm loading a 4GB csv in Python using pandas, and activity monitor says I'm using more RAM than I physically have in my laptop (15GB out of 8GB). How's this possible internally?

I need to download more RAM

2

This guy explains it well http://www.chriswrites.com/what-is-virtual-memory-on-the-mac/

So What is Virtual Memory?

Your Mac comes with a limited amount of physical memory (RAM) installed by default. Memory is critical to any computer to run applications and carry out all the operations commanded by the user. Memory must be as fast as possible (and it helps to have as much as possible) to maximise your Mac’s performance; thankfully, prices have drastically fallen in recent years. Nevertheless, every computer has a finite amount of memory so Mac OS X might actually need to use more than is physically available on your system.

This is where virtual memory comes in – you can think of it as a kind of extension to the real memory on your system, albeit much slower because it is in fact physical space on your hard disk (or SSD, if you have one installed). Let’s say that you have 4 GB of main memory on your Mac and a 512 GB hard disk. If Mac OS X needs to use all 4 GB of main memory, it can still make use of any free space on your hard disk as an extension to that memory. Unfortunately, virtual memory is much slower than the physical RAM installed in your computer, because hard disks cannot match the speed of memory chips.

  • got it, thanks. As an aside, if I wanted to manipulate 100GB csv files, could I install linux with a gigantic swap partition? – Boris Nov 28 '14 at 15:05
  • 2
    Yes but your system will slow down because some of the system procs will also swap. The right technique is to manipulate big files efficiently ;) eg striping your reading. – f01 Nov 28 '14 at 15:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .