The light input to a semiconductor optical modulator can constitute an electrical energy supply through the photovoltaic effect, which is unexploited in conventional modulators. In this work, we leverage this effect to demonstrate a silicon modulator with sub-aJ/bit electrical energy consumption at sub-GHz speeds, relevant for massively parallel input/output systems such as neural interfaces. We use the parasitic photovoltaic current to self-charge the modulator and a single transistor to modulate the stored charge. This way, the electrical driver only needs to charge the nano-scale gate of the transistor, with attojoule-scale energy dissipation. We implement this 'photovoltaic modulator' in a monolithic CMOS platform. This work demonstrates how close integration and co-design of electronics and photonics offers a path to optical switching with as few as 500 injected electrons and electrical energy consumption as low as 20 zJ/bit, achieved only by recovering the absorbed optical energy that is wasted in conventional modulation.