As someone who has to test RAM every day (Apple Certified Macintosh Technician for 2.5 years), I've found that the best RAM testing utility hands down is Memtest.
It's really easy to use too! First, copy it to
/usr/bin then boot to single-user mode. Next, type in
mount -uw / to mount your internal drive (for logging purposes) and finally type in
memtest all -L 3 to start the Memtest suite.
If you want to perform more loops, change the 3 to a number you desire (e.g.
memtest all -L 5), I reckon 3 to 5 loops is the best because it gives the RAM time to heat up (intermittent RAM failures can be heat related).
edit: Just wanted to add that Apple Hardware Test is useless, so never use it for testing hardware, especially RAM.