The core of OS X is not garbage collected; IOKit's libkern C++ Runtime requires developers to manage their own memory.
Mac Memory Management
From How does memory management work in Mac OS X?
Apple documents the lowest levels of the Mach Kernel and the virtual memory subsystem fairly well on the web as part of it's developer documentation.
Since that kernel was developed by Carnegie Mellon University, you can find dozens of papers describing it quite easily.
Wikipedia also discusses Mac OS memory management.
Garbage Collection
Garbage collection exists at the user or application layer. Even at this layer, garbage collection only helps if the application has released all claims to the memory. A circular dependancy can defeat garbage collection. Garbage collection itself is an evolving area of research and difficult to get right.
Report Bugs and Memory Leaks
Bugs within OS X will be leaking memory. Given the size of the code base, this is almost certain.
Please report reproducible bugs directly to Apple. Every bug report helps and maybe your example will be the one that helps Apple's engineers pin down the cause.