Vesicular traffic and membrane contact sites between organelles enable the exchange of proteins, lipids, and metabolites. Recruitment of tethers to contact sites between the endoplasmic reticulum (ER) and the plasma membrane is often triggered by calcium. Here we reveal a function for calcium in the repression of cholesterol export at membrane contact sites between the ER and the Golgi complex. We show that calcium efflux from ER stores induced by inositol-triphosphate [IP3] accumulation upon loss of the inositol 5-phosphatase INPP5A or receptor signaling triggers depletion of cholesterol and associated Gb3 from the cell surface, resulting in a blockade of clathrin-independent endocytosis (CIE) of Shiga toxin. This phenotype is caused by the calcium-induced dissociation of oxysterol binding protein (OSBP) from the Golgi complex and from VAP-containing membrane contact sites. Our findings reveal a crucial function for INPP5A-mediated IP3 hydrolysis in the control of lipid exchange at membrane contact sites.