What do these dashed boxes mean and how to use them?
Your image seems to have disappeared so I added one myself.
The dashed boxes represent the space that a normal character would occupy so you can see how the characters you're typing will be positioned. On the middle row shown, the first key puts a mark in the lower-left position, the second key has the same position, the third shows that the dot will be in the middle of the character space and the last one shows the line will be beneath the normal character space, almost like an underscore
The Japanese characters on the depicted white keys are hiragana, these are what is used during direct input. So, the K key will produce の. The characters are arranged slightly differently on my keys; see my image. Using shift (or caps) as a modifier inputs the character in the right position. For instance, Shift < produces 、 and Shift > produces 。. Also, using option as a modifier will input dotted boxed characters in the upper position (only ＿ exist for me).
If you are using Mac OS X 10.6, you should have an IME menu on your menubar. In this OS X Japanese writing tutorial reference the steps to reach the utilitiy in Fig. 4 Kotoeri to change Input style to Kana typing.
I hope this helps.