Program packages also contain information of their capabilities. Files like info.plist have to be scanned and added to the LaunchServices database. If that grows for example a right-click on a file can take quite a bit longer to make this noticeable enough.
Sometimes an application installs a bucket full of fonts, that nobody wants, mainly certain office applications.
Caches are nice, if they are indeed small. The macOS caches grow and self-corrupt over time, mandating a deletion as there is no on-board mechanism to rectify this.
LaunchDaemons and LaunchAgents (at /System/Library/, ~/Library/ and /Library/)
are also found within application packages and in any case: "not running at the time" doesn't mean they do not run "sometime", eg when you're not really looking. These are for LaunchDaemons, LoginItems and XPCServices. Usually these are moved on app-launch to their 'proper' places, but they can be also active from within the app (or by a proxy copied to a nested container in ~/Library).
Some examples:
- really pestering: VOX.app/Contents/Library/LoginItems/Loop.app & VOX Agent
- Another example might be DefaultFolderX: every time a file dialog opens this app adds its own routines from 9 frameworks to the once process
- some audio software installs plugins into /Library/Audio/Plug-Ins which then hook into the audio system and are not visible from Activity Monitor, despite adding to the system load when audio plays, trashing the app leaves the plugin behind
- HP Printer Software has a Device Monitor running, that eats away CPU and is hard to find on disk
All programs leave behind cruft. Caches, receipts, preferences, database entries no longer valid etc. Everything within an application package goes if you drag it to the trash. But I know of no single de-installer that really catches all the cruft that gets littered around by the majority of applications.
Therefore the more programs are installed the slower the system gets. Compared to 90s Windows this shouldn't get out of hand that easily.
Some of the top locations to really watch out for other than /Applications:
/Library/Extensions
/Library/Audio/Plug-Ins
/Library/Bundles
/Library/Caches
/Library/Frameworks
/Library/Input\ Methods
/Library/Internet\ Plug-Ins
/Library/LoginHook
/Library/QuickLook
/Library/ScriptingAdditions
/private/var/db
/private/var/folders
That list is far from complete and also applies to the corresponding folders under /Users.
Uncontrolled – and often quite unnecessary – installers, writing files all over the place are one problem. Unless you can inspect them with eg Pacifist, they should be considered "untrusted", and monitored closely.
There is no uninstaller in macOS. Once a program gets installed you have to trust the developer that the program doesn't write files all over the place and adds login-hooks, startup-programs, extensions, frameworks, fonts, daemons, input methods or plugins – to the system (or user Library)
The sheer number of applications/executables presents a slow down factor. One quite small one, compared to the above, but it nevertheless adds to the problem.
Actual malware or PUPA. Downloading and trying out lots of programs increases the chance that a bad apple gets onto the system and ruins the badge.
If the system is installed on HFSplus then of course the good old inefficiency of said system comes into play, to a degree. For example the directory structures get more inefficient over time, one reason why products like DiskWarrior exist. On APFS things are different, but there are not enough reports out there to say empirically whether it got better or worse overall. Generally all SSDs slow down when getting full and with age. The more files on it the more decisions the controller has to make on where to write the next bits and in the background all those bits have to be shuffled around for TRIM, GC and wear-levelling (not to mention plain bugs like in Samsung 840EVOs).
After a fresh install you may realise that isn't necessarily a really clean install: Sometimes a problem or slowdown factor was your reason to re-install the system, but then you used Migration Assistant, which is not always intelligent enough to not also migrate the problem to the new installation as well.
Psychology –– After re-installing a clean system it will certainly feel great to be able to do stuff with it again, after sitting through the long wait and reboots and rebuilds of things like caches and Spotlight indexes etc.
When your Mac runs slow, give it a tuneup
Speed Up Your Mac
7 Common Mistakes That Slow Your Mac Down
Why is my Mac Running Slow?
Why Is My Mac Running Slow? 5 Quick Tips to Speed Up Your Mac!
26 Reasons Why a Mac Computer is Running Slow [Infographic]
Keep apps and extensions to a bare minimum
How-To: Clean and speed up your Mac with free, trustworthy downloads
How-To – Unclutter your hard drive
As @jksoegaard rightly comments: a single 1KiB file that never gets used will not slow down the system. The above is just addressing the general bloat that badly written and maintained software unloads without benefit to any user on small but expensive SSDs, where these files and bigger ones are numbering into the thousands. A few kilobytes here a few megs there, after a while you realise that some gigabytes added up and are nothing but pure waste.
Unfortunately, most of the software for macOS falls now into that category (prime example: /System/iOSSupport/System/Library/PrivateFrameworks/MobileIcons.framework/Versions/A/Resources/MessagesDefaultIcon-32.png -> 32x24 pixels, but occupying the modest amount of 10,265,058 bytes on disk! Or the latest cruft by design. ) It's not worth hunting down these bloaters daily, but you paid for this waste. So, once in a while, a good scrubbing, wherever possible, is in order.