I am not sure what all hosts I have accumulated over time under my
~/.ssh/known_hosts file are.
Is there any risks in keeping all hosts' addresses or doesn't it make any difference?
Is there were any risks in keeping hosts addresses or if it doesn't make any difference?
Yes, it is. This file creates your database of trusted host keys. If you do not maintain it in any way, there might be a lot of them that are expired or possibly leaked from the original host. In that case, you are exposing yourself to MitM attack, where potential attacker can spoof your DNS, redirect your SSH connection to his server (presenting previously valid host keys) and ask you for your password.
This is a big disadvantage of not using any public key infrastructure (PKI) with central certificate authority, which would take care of the expired certificates by distributing certificate revocation lists (CRL) to all its users.