The Apple FAQ has this same question, but doesn't really have an answer. To quote:
What is the Mac App Store refund policy?
If you experience a technical issue with an app, first contact the developer of the app for assistance. Their support contact information can be found on the app's page in the Mac App Store. If that does not resolve the issue, contact Mac App Store Customer Support.
So the policy is that I contact someone for help... but can I actually get a refund? Suppose I buy an app that claims to play avi files, but it only works with three out of four files I throw at it. Would that kind of problem typically result in a refund?