The software you are looking for is called "Digital Rights Management" software, often abbreviated to "DRM".
Searching for this term will find a wide range of third party tools and solutions for protecting your videos. The offerings will range from massive companies like Adobe with their Access solution to numerous smaller companies.
Protecting Streamed/Online Content
Controlling access to a video file to be played on viewer's computer is a difficult problem. Your best option is to provide online only access to your video content. By serving the video content yourself, you can more easily control access.
Have you considered using YouTube and their paid channel service?
Protecting Local Content
There is currently no built-in DRM mechanism included with Mac OS X. Requiring a user to install a third party video codec is likely to limit your audience.
However, Flash remains a popular plug-in and can be used to provide digital rights management. Flash works both in the web browser and for stand alone applications.
Searching for flash protect local video returns software claiming to support multiple platforms, including Apple:
We protect your software, video and content effectively against
illegal copies and sharing. On CD/DVD/BD, USB-Stick, in local networks
or via the Internet - We have the best solution for copy protection.
Robust content protection across
every screen is an essential part of any video monetization strategy.
Brightcove helps you protect your content and your business model with
the most advanced encryption and DRM technologies, as well as
geographic, time-based and user-based restrictions, to prevent
unauthorized access, downloads, and copying of your valuable content.
Multimedia OwnerGuard is designed to support a wide range of
Multimedia Files including (swf, flv, mp4, f4v, 3gp, avi, rm, mkv,
3gg, 3g2, mov, m4v, m4a, f4p, f4a, f4b, mp3, mpg, asf, wma, wmv, aac,
ogg, aif, flac). Now you can use full features of OwnerGuard DRM
Technology to protect and distribute your Flash SWF, FLV and many
other media files while OwnerGuard protects your digital ownership
Encrypt and protect Flash SWF files from decompilers. Protect not only Actionscripts, but also all images, sounds, buttons and sprites resources in your SWF files.
Try a Flash Based Solution
Try one of the Flash based video protection programs returned by the search above; I have no experience with any, so will not directly recommend one. The list above is extracted from the first page of Google results.