When trying to edit the mask, I am stuck with this blue dot that only rounds the corners of the mask and I can't get rid of it. How do I go about editing the mask borders?
You should also see the white squares defining the borders of the mask. Those are the handles for resizing borders. (It's also possible to select the picture behind the mask, in which case the handles surround the picture itself.)
Pixelcorps has a terrific MacBreak Work video tutorial on masks in iWork '09.