There’s an easy way to do this from your iPhone :
Open the Maps app
Tap on ⓘ to open the settings
Select “Report an issue” and “Home”
You can then move the pin on the map without changing your address.
Dont click on “Report to Apple” and select “Save on my device”. It won’t work otherwise.
macOS Catalina (10.15.7)
Unchecking and renabling birthdays on the calendars sidebar will only toggle visibility and did not remove the duplicates, but unchecking and renabling 'Show birthday calendar' in the calendars settings seems to reload it - that's what worked for me.