The terahertz radiation emitted by asymmetrically ionized wavepackets in two-color strong-field tunneling ionization is essential for detecting the system's associated electron dynamics and structural properties. We propose to characterize and control tunneling ionization using a phase-dependent terahertz polarization (PTP) spectrum, analyzed through a combination of the classical trajectory Monte Carlo method, an analytical model based on the virial theorem, and the rigorous solution of the time-dependent Schrödinger equation as a benchmark. Our results demonstrate that the PTP spectrum offers a high-precision measure of the Coulomb effect through the relative phase of the two-color laser. Comparisons of PTP results calculated using different methods suggest how the electron can be manipulated by controlling the relative phase and laser intensity. In particular, the PTP spectrum can be used to calibrate the relative phase and provides a convenient and robust reconstruction of the time-averaging of tunneling positions with high precision using the analytical model. These insights reveal that the PTP spectrum as a whole can be a new and useful tool for the all-optical characterization of ultrafast atomic and molecular ionization.