I was faced with the same lingering preference pane (specifically for fuse-ext2).
I took two steps to finally be able to remove the preference pane. All of the preference pane locations mentioned in the Apple documentation contained no mention of the preference pane in question.
What I found was one potentially relevant preference pane in a cache and the preference pane cache itself.
To clear the preference pane cache, just move the
~/Library/Caches/com.apple.preferencepanes.cache file to the trash or delete it by some other means. This may be all that you need to fix the issue. If it's not...
I also removed a file (
~/Library/Caches/com.github.osxfuse.OSXFUSEPrefPane) that could have been relevant, but may not have been. I removed the file before clearing the preference pane cache, so I'm unable to say which definitively solved the issue.