You will need to figure out what is causing a processing bottleneck on the Mac.
You'll need some understudying of basics like being able to tell if the virtual memory system is swapping (
vm_stat 5 will show paging when you reproduce the issue) as well as Activity Monitor to see what the overall RAM and disk IO are as well as look for processes that are not responding or taking a lot of CPU time.
The fact that a reboot cures things shows that it's not a hardware issue or a software conflict, but instead a resource issue. Experience in doing this leads me to guess you either have some apps that have memory leaks and once the system starts swapping, you have enough CPU busy things that on occasion that slows the GPU / graphics subsystem.
You could also try quitting all apps when you notice the slowness and then just starting one or two apps to see if that resolves things. It seems odd that you always have 4 GB free, since I would expect you to reach a state over time where almost all the "free" memory converts to inactive to cache things that were used, but now out of active memory use.
Hopefully this process helps you find the culprit since there isn't enough detail to guess what's really happening. Good luck hunting this down.