We analyze the impact of a proposed tidal instability coupling p modes and g modes within neutron stars on GW170817. This nonresonant instability transfers energy from the orbit of the binary to internal modes of the stars, accelerating the gravitational-wave driven inspiral. We model the impact of this instability on the phasing of the gravitational wave signal using three parameters per star: an overall amplitude, a saturation frequency, and a spectral index. Incorporating these additional parameters, we compute the Bayes factor (lnB_{!pg}^{pg}) comparing our p-g model to a standard one. We find that the observed signal is consistent with waveform models that neglect p-g effects, with lnB_{!pg}^{pg}=0.03_{-0.58}^{+0.70} (maximum a posteriori and 90% credible region). By injecting simulated signals that do not include p-g effects and recovering them with the p-g model, we show that there is a ≃50% probability of obtaining similar lnB_{!pg}^{pg} even when p-g effects are absent. We find that the p-g amplitude for 1.4 M_{⊙} neutron stars is constrained to less than a few tenths of the theoretical maximum, with maxima a posteriori near one-tenth this maximum and p-g saturation frequency ∼70 Hz. This suggests that there are less than a few hundred excited modes, assuming they all saturate by wave breaking. For comparison, theoretical upper bounds suggest ≲10^{3} modes saturate by wave breaking. Thus, the measured constraints only rule out extreme values of the p-g parameters. They also imply that the instability dissipates ≲10^{51} erg over the entire inspiral, i.e., less than a few percent of the energy radiated as gravitational waves.