How scientific HPC applications perform in production, on real workloads and at scale, is mostly unknown. Monitoring and performance analysis tools for development and testing runs exist, but production execution performance is mostly opaque and unknown. This talk will explore ideas and experiments in production-side monitoring and analysis of HPC systems and applications, and will dive into a view of applications based on phases, which are unique periods of behavior during the application execution. Ideas and open questions regarding monitoring and analysis from a phase-oriented view will be presented, with feedback and discussion welcome.