We introduce several improvements to the penalty-based variational quantum Monte Carlo (VMC) algorithm for computing electronic excited states of Entwistle et al. [Nat. Commun. 14, 274 (2023)] and demonstrate that the accuracy of the updated method is competitive with other available excited-state VMC approaches. A theoretical comparison of the computational aspects of these algorithms is presented, where several benefits of the penalty-based method are identified. Our main contributions include an automatic mechanism for tuning the scale of the penalty terms, an updated form of the overlap penalty with proven convergence properties, and a new term that penalizes the spin of the wave function, enabling the selective computation of states with a given spin. With these improvements, along with the use of the latest self-attention-based ansatz, the penalty-based method achieves a mean absolute error below 1 kcal/mol for the vertical excitation energies of a set of 26 atoms and molecules, without relying on variance matching schemes. Considering excited states along the dissociation of the carbon dimer, the accuracy of the penalty-based method is on par with that of natural-excited-state (NES) VMC, while also providing results for additional sections of the potential energy surface, which were inaccessible with the NES method. Additionally, the accuracy of the penalty-based method is improved for a conical intersection of ethylene, with the predicted angle of the intersection agreeing well with both NES-VMC and multireference configuration interaction.