The full procedure to delete a file from the desktop, without the mouse, that should work almost anywhere in OS X, includes the following.
Command-Tab to switch applications until the Finder is the active application.
If any Finder windows are currently open, press Command-Shift-D to open a new window containing the Desktop folder, or Option-Command-W to close all active Finder windows and make the Finder's desktop active;
To highlight the icon, type the first few letters of the icon's name, or press the arrow keys until it is selected (highlighted);
Command-Delete to send the icon to the trash;
Optionally, Command-Shift-Delete to empty the trash.
Learning the keyboard shortcuts takes time, but many of the steps in the Finder can become natural, almost "automatic" with enough practice.
(This answer gives the key names in U.S. English. The Command key is also labeled ⌘; The Option key is also labeled ⌥; The Shift key is also labeled ⇧; and as mentioned by Styfle, the "delete" key is called "Backspace" on other operating systems.)