Is there a way to install an app, which is not available on the App Store, without doing a jailbreak?
If the app was available you can restore the .ipa file. The app will show in the list of apps in iTunes on your iOS device and you choose to install.
Note I have installed an app which is no longer in the App Store but which I had got from App Store some time before. I did this after I did a factory reset. I think these were the steps but this is from memory so the details might be wrong but you definitely can do it
The only way you will be able to install it is if the application you want is open source.
This is because Apple has released Xcode 7 that now allows you to install applications to your iOS device without needing a developers account. The only down side is you cannot install IPAs or .apps this way as far as I know. This will NOT allow you to install jailbreak tweaks however.
You can do it via Profiles, in setting if you install a new profile, you would be able to install apps that are not there in AppStore That's how a Semi Jailbreak works, it doesn't exploit the iOS version completely. It installs a profile with which it installs Jailbreaking Package.
To know about Semi Jailbreak visit, http://m.pangu8.com/83.html