Sounds like an opportunity to create an app!
Google Maps allows you to save your own maps (My Maps) where you can add locations and place short notes about them. You can not categorize or sort.
With Yelp, you can save locations and make notes, but its limited beyond those features.
The closest I have seen to what you are seeking is an app and website called Findery. It is a mapping app that allows you to save and create notes on specific locations. you can include links, videos etc if you like. While these notes are meant to be 'found' by others, you can make them private. In addition, it has a feature called 'Notemaps' that allows you to categorize your notes in separate maps, like 'Been here', or 'favorites' etc.
I am not affiliated with any of these, just a user.