Scientific machine learning has opened new avenues of research to enable efficient outer loop analysis through learned models of physical systems. However, these learned models are imperfect representations of complex physical processes. The discrepancy between such models and the underlying truth may be amplified by outer loop analysis such as optimization. We present a novel approach to compute the sensitivity of optimization problems with respect to model discrepancy and use this information to improve the solution obtained using learned models. By posing a Bayesian inverse problem to calibrate the discrepancy, we compute a posterior discrepancy distribution and then propagate it through post-optimality sensitivities to compute a posterior distribution on the optimal solution.