I have been having the same issue. It must be a bug. Please report it to Apple here:

http://www.apple.com/feedback/watch.html