When I try to copy 50 GB to external HDD it succeed only when copy dialog is active. If I activate another app it fails at about 10gb with error "the operation can't be completed because an unexpected error occurred (error code -8084)". If I switch between apps often enough it also succeed. I tried it with 2 different external usb3 drives. It looks like mavericks decides to lower priority or freeze inactive copy process which causes copy error. The error tends to appear only when copy dialog is not visible (fully obscured by another window, minimized, another fullscreen app is active). How to fix this?
I think your hunch is correct. You seem to have run into a problem with App Nap. To save energy, a background task with no visible UI gets throttled down and eventually stopped. Each time you bring the dialog to the front, it has a "visible UI" and gets a new lease on life.
Depending on which app is doing the copying, you might be able to disable App Nap for that application. Get Info on the app, and put a checkmark in front of "Disable App Nap". You can't do that with Finder, though, because Finder's Get Info window doesn't show that checkbox.
You might try disabling App Nap system-wide. To do that, go to Terminal and enter the command:
defaults write NSGlobalDomain NSAppSleepDisabled -bool YES
Well, not quite system-wide. Only within your account. Repeat while logged in as each user that you want App Nap disabled for.
Bear in mind that disabling App Nap may make your computer consume more energy. If it's a portable, that means shorter battery life. But no worse than the way things were before the App Nap feature was introduced in Mavericks.