RAM is there to be used. It is a place for your computer to keep the most frequently used data so it doesn't have to fetch it from disk which is much slower (could be 50-60 times slower). The RAM in your Mac Mini is 1600MHz DDR3 if you upgraded at time of purchasing. This is capable of about 12800MB/s transfer. The HDD if it's a physical one is a 5400rpm drive capable of 200-300MB/s. So you want your RAM to be fully utilised. It's no use to you when it's free.
With that in mind, your problem is not how much RAM is in use but rather, why your computer is slow. The best way to determine that is to see which programs are accessing the HDD. You can read all about how OSX manages memory on Apples support site.
As pointed out in this Superuser answer:
Mavericks introduced "memory compression" which is, more or less,
another layer of swap. Active pages now get classified inactive, then
compressed (which might show up as Kernel memory depending on what
tools you are using,) then written to swap as memory usage increases.
Mavericks has also has stopped showing separate numbers for active and
inactive in Activity Monitor, since it turns out not to be a useful
thing to look at, especially given the misconceptions surrounding it.
As suggested by other people, a hard-drive index or full system backup would be a major culprit of disk activity slowing your machine so I'd look there first. You can also disable the spotlight indexing in Terminal if you wish like so:
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
I would try turning off indexing temporarily, perform a full backup, when you've done both of those then re-assess the performance of your Mac and if you still have issues then use Activity Monitor to see what else is performing Disk I/O.