If you've enabled iCloud for iBooks [1], any books you've uploaded are stored in:
~/Library/Mobile Documents/iCloud~com~apple~iBooks/Documents/
Although you can't see it when you browse iCloud Drive in the Finder, the are two easy ways to get to it:
Open any PDF; when it opens in Preview, right-click the titlebar and click "iBooks (iCloud)".
or
In the Finder, press ⌘ command⇧ shiftG and paste the path.
Note that books you've purchased from the iBooks Store are stored in a different location:
~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks
You can open this folder the same way (⌘⇧G), but there's not much you can do here. If you want to add books to your library, add them to the iCloud folder, not here.
Just checked and confirmed locations are still accurate in macOS High Sierra (10.13.4).
[1] If you're not sure if iCloud for iBooks is enabled (understandable, since you only see it the first time you open iBooks, and there's no setting within the app), go to System Preferences → iCloud, then click "Options" next to iCloud Drive. Look for iBooks in the list. If it's checked, you're using iCloud for iBooks.