No, Time Machine does not use the Modified Date timestamps.
Ordinarily, it uses the File System Event Store, a log of changes to the file system. (That's also how it can usually figure out the differences so quickly, since it doesn't have to look at every file and folder on your Mac.)
But in some cases, it does have to do that -- after a failed backup, for example. In that case it still doesn't use the dates; it compares everything on your system to the most recent backup, so will be in the "Preparing" or "Calculating Changes" phase for a while, and there will be a message in your system.log about a "deep traversal" (prior to Lion) or a "deep scan."
See #D4 in Time Machine - Troubleshooting for some of the common culprits, and ways to see what's going on.