I'll take a shot at answering your question...
It sounds like your current Octave app is more or less a convenient wrapper for the scripts and such that you would otherwise have to run from the terminal to get things up and running. In other words, it's created for people that are trying to avoid interacting with the terminal directly (and since you seem to enjoy working directly from the command line, the app has lost a lot of its convenience).
Anyway, in my opinion (for what it's worth) your best bet is to ditch the Octave app and instead take a more hands-on approach and install Octave via a package manager like Homebrew (here are some instructions you can follow).
However, a quick warning: Installing octave this way is occasionally a real pain-in-the-neck (and same goes for getting Octave to work with GNUplot). Though the links I've provided should be enough to help you get Octave set up the way you want, the installation process itself is not always, well, particularly pleasant (after all, those "convenience wrappers" exist for a reason). Just giving you a heads-up...
Hope this helps!
Another helpful tutorial which is similar to the first link.
And still another (the title of this post is pretty revealing...).