I have few app that was downloaded to my Mac, now there are not available on the iOS App Store anymore, so can I install them when I get a new phone even those app are not available on the App Store?
Yes. You can synchronise them by connecting your iPhone to iTunes on your Mac. It will synchronise all the apps that are in your iTunes library with your new iPhone.
That's why it's handy to download apps in iTunes as well. Also updates and keeping older versions from the app around. So you can go back to an older version when there is a bug introduced in the newer version.
To preface this with a little fyi: Mac Apps and iOS apps are not related. The stores are similar and there are many apps that have been made for both stores but they make separate apps for each platform and one will not work on the other.
Because of this, what I think you mean is that you made an iTunes backup of your phone and want to restore it to your new phone with Apps that have been removed from the store still installed. It should work if you are able to restore directly from the backup, but be warned that that will only work as long as your backup's iOS version is relatively current. As it gets older, iTunes will perform an update pass that will likely remove it during flashing.