Methane leaks in natural gas systems are low-hanging fruit for near-term, locally driven climate policy. Recent work suggests this emissions source is larger than previously believed and that repairing a small number of high emitters can cost-effectively reduce system-wide leakage. How successful are these repairs on the ground? Here, we assess the effectiveness of repair policies in the Massachusetts distribution system. Our analysis leverages state-wide utility data, on-site empirical measurements, stakeholder interviews, and document and legal analysis. We use these mixed methods to investigate the rate of repair failure, where a gas utility identifies and fixes a leak, but on-site emissions are not eliminated. We find that repair failures are relatively common, yet they are repeatedly neglected in policy. By not accounting for repair failures, policy may overestimate the effectiveness of distribution system repairs in meeting local greenhouse gas reduction targets. These results also underscore the importance of data transparency for monitoring and verifying subnational climate policies.