Obviously installing gcc from source will require a compiler to build it; is it possible to install gcc without Xcode?
You can find several versions of gcc for different OS X flavors here: HPC Mac OS X.
Download my binaries, and
cdto the download folder. Then
gunzip gcc-5.3-bin.tar.gz(if your browser didn't do so already) and then
sudo tar -xvf gcc-5.3-bin.tar -C /. It installs everything in /usr/local. You can invoke the Fortran 95 compiler by simply typing gfortran.
[To compile other sources...] you will also need to have Apple's XCode Tools installed from the Mac App Store. With XCode 4 or better you will need to download the command-line tools as an additional step. You will find the option to download the command-line tools in XCode's Preferences. On 10.9 Mavericks or higher, you can get the command-line tools by simply typing xcode-select --install.
Addition in square brackets by me...
Please also visit Information Security Stack Exchange and search for questions about the security risks associated with precompiled compilers from "unknown" sources.