I have a 2010 Mac Pro 5,1. It has an ATI Radeon HD 5770.
I bought a Sapphire Pulse Radeon RX 560 to upgrade.
How can I use RX 560 on OS X El Capitan and Mojave?
According to various reports around the internet, to get to Mojave there are a couple of precautions to take so you don't lock yourself out of the machine.
I would always be certain to have not only an up to date Time Machine backup, but I would also have a full clone of the working system before starting out... you never know when something might go wrong.
I'm assuming the card you have is not specifically Mac-flashed.
My card is flashed, so some of these precautions are learned from others' experience, not my own.
First of all, to be safe - it you have File Vault enabled, disable it & wait for it to finish decrypting before moving on.
Another thing to check is whether your new card is 6-pin or 8-pin. If it's 8-pin you'll need to get adaptors. From memory, the HD5770 takes a single 6-pin, but there should be a second cable already in situ. The new card will need both.
You could then test by just swapping in the card & booting up. If you can see the boot screen, it's flashed. If you see nothing until you get to login, it's not.
If it's not flashed you cannot use File Vault once it's installed.
Other than that, on El Capitan it should 'just work', no drivers or firmware update is required.
Assuming all went well so far, then you will need to update to High Sierra 10.13.6 before you can go to Mojave. I don't know why this is, & I was already at 10.13.6 anyway before Mojave, but general consensus is that this is a necessary step.
I vaguely remember having to do a firmware update to get to High Sierra, though I don't remember the details. I think it was a simple 'Click OK & wait' type of structure.
The firmware update to load Mojave is different.
It will tell you to hold the power button until the light flashes or you hear a beep. Having done this many times now (all the way through the beta phase you had to do it every time, so once a week for 2 months ;) I can say with certainty that you need to wait until you hear the beep before letting go of the button. The Mac will immediately reboot at that point.
Once that's done, it's just a case of waiting until the install is complete & you can use your new card on Mojave.
 Not being able to see the boot screen will limit anything you ever need to do before login; change boot drive, dis/enable SIP, use Recovery etc - so it's best to always keep your old card for just in case.
However, as Mojave may not even boot with the old card in [I haven't tested this] you ought to hang onto the clone you made at the start. Belt & braces. Better safe than sorry.