You can't restore a backup from a newer iOS version to an older iOS version.
You should have an iTunes backup from before you updated iOS. If not, you will have to manually extract the data from your iOS 11 backup and add it back to your device after setting up your iPhone as new with iOS 10.3. How you do this depends on the individual app — many use iCloud such as iCloud backup in WhatsApp which will help you do this automatically, but otherwise it will be a case of manually editing a fresh backup of your iOS 10.3 device with an app such as iBackupBot. For things which are incompatible with older iOS such as your messages history, this will be unable to be restored to the older iOS.
Unfortunately all I can say is that you are warned of the perils of installing beta iOS prior to acceptance of the beta program. My suggestion is to put up with iOS 11 and keep updating to the latest beta as it becomes available, which will fix many of the bugs. You'll be updating to iOS 11 anyway in the future.