EDIT2: I found the solution, see my answer below.
EDIT: The question may be a bit misleading I don't have it in the favorites bar, the problem is that finder always tries to connect to a server when I start my machine or log in.
I once saved an ftp server as a "favorite" in finder (so finder would automatically try to connect to it at startup). I no longer use this service, but finder still tries to connect to it automatically at every startup, I'm not sure how to delete this server from the list of servers finder tries to connect to automatically. How do I do that?
Some things I have tried: It has been removed from favorite servers and I also cleared the recent servers list in the "Connect to Server" menu. This had no effect.