The issue is caused by Gatekeeper. It uses "path randomization" to run applications, until the user moves the app in it's final destination (/Applications for example). This causes the app to be run from `/private/var/..` and Dock treats this as a new app.

To remove the quarantine attribute execute (removes recursively com.apple.quarantine):

`sudo xattr -dr com.apple.quarantine /Application/SomeApp.app`

Recursion (the -r flag) may be needed for apps that contain helpers.

To list the current attributes for an app execute:

`sudo xattr /Application/SomeApp.app`

or 

    ls -l@d /Application/SomeApp.app

Sources:

[On Gatekeeper and the Lifespan of Little Snitch Rules](https://blog.obdev.at/gatekeeper-and-little-snitch-rules/)