The accurate measurement of three-dimensional (3D) fiber orientation in the brain is crucial for reconstructing fiber pathways and studying their involvement in neurological diseases. Comprehensive reconstruction of axonal tracts and small fascicles requires high-resolution technology beyond the ability of current in vivo imaging (e.g. diffusion magnetic resonance imaging). Optical imaging methods such as polarization-sensitive optical coherence tomography (PS-OCT) and polarization microscopy can quantify fiber orientation at micrometer resolution but have been limited to two-dimensional in-plane orientation or thin slices, preventing the comprehensive study of connectivity in 3D. In this work we present a novel method to quantify volumetric 3D orientation in full angular space with PS-OCT. We measure the polarization contrasts of the brain sample from two illumination angles of 0 and 15 degrees and apply a computational method that yields the 3D optic axis orientation and true birefringence. We further present 3D fiber orientation maps of entire coronal cerebrum sections and brainstem with 10 μm in-plane resolution, revealing unprecedented details of fiber configurations. We envision that our method will open a promising avenue towards large-scale 3D fiber axis mapping in the human brain as well as other complex fibrous tissues at microscopic level.