I've been a fan of Apple's Hardware Test (AHT), built into newer Macs and available on restore DVDs for older Macs, for several years now. It allows for both a quick test (basic memory tests, basic test for other components and a quick HD scan) and an extended test. The extended test runs full verification tests on the RAM, the basic hardware tests and a full disk scan of the hard drive.
Apple has a knowledge base article devoted to AHT: Using Apple Hardware Test -HT1509
Basically depending on the age of your Mac you have two options...
Older Macs that shipped with a gray restore disc/s or MacBook Airs that shipped with a USB recovery disk:
- Insert the gray restore DVD (or USB recovery disk if MacBook Air)
- Power on your Mac while holding D
It may take several (15+) seconds for AHT to load, once loading though you should see this icon appear on the screen. If you see the Apple logo then you will need to shutdown the Mac and try again.
Select your desired language
There are three tabs. About explains the tool. Hardware tests runs the tests. Hardware Profile shows the Macs configuration. The middle tab, Hardware tests, is the one you'll use
You have two options, quick test (the default setting) or a checkbox to run an extended test. The extended test takes considerably longer (typically an hour although this will vary based on hard drive size and amount of installed RAM) but is more thorough.
If the tests pass you will receive a "No trouble found" message. If the tests fail you will receive a somewhat cryptic error code. This Cnet article does a great job of explaining the error codes. For what you're interested in though you'd be looking for a failure beginning with 4/MEM, this would indicate a failure with the memory test, either with the DIMM or the DIMM slot on the logic board.
Newer Macs that shipped with 10.7 or above, or without restore media, will follow the same steps as above however they can ignore the very first step since there is no media to insert in the Mac.
Newer, post June 2013, Macs will instead use Apple Diagnostics. While similar to AHT in it's capabilities it lacks the option for an extended test. You invoke it in the same manner as AHT.
If you're instead looking for a tool to run from within your OS then my tool of choice has always been TechTool Pro. It's memory tests usually confirm suspected RAM issues.
Hopefully that helps!