Yimin Zhao
- Program & Year: PhD, fifth year
- Interned with: Microsoft, Applied Science and Research Team, Copilot Studio
At his Microsoft internship, PhD student Yimin Zhao’s statistical training helped him balance the quick-paced iteration of developing software features, while also maintaining rigor in the process.
As a data scientist intern on Microsoft Copilot Studio’s Applied Science and Research team, Yimin Zhao worked to improve an agentic retrieval-augmented generation (RAG) System, a robust process that helps AI answer questions.
Zhao said the experience gave him insight into the different ways production and academic environments operate as well as an understanding of how rapid iteration and careful experimentation can support, rather than compete with each other.
“One of the most surprising things I learned was how increasingly fluid the boundaries between different technical roles have become because of AI. As an applied scientist or data scientist, it is no longer sufficient to only design experiments and analyze results. We also need to understand the underlying systems, implement our ideas, and help launch features directly into production,” said Zhao.
Q&A
What were your responsibilities as a data science intern?
I worked on improving Copilot Studio’s agentic retrieval-augmented generation (RAG) system. My responsibilities included analyzing agent behavior and failure modes, designing and evaluating system improvements, and developing features to improve answer quality and efficiency. My primary contribution was File Evidence, a Copilot software development kit (SDK) tool that reduced latency for simple fact-retrieval tasks by approximately 80%. Previously, the system often routed these tasks through multiple general-purpose analysis skills, creating unnecessary orchestration and repeated processing. File Evidence introduced a direct, lightweight retrieval path for extracting facts from attached files.
What did you learn from your internship experience?
I had two major takeaways from my internship.
First, self-evolving agentic skills and harnesses do not transfer automatically from academic benchmarks to production environments. Academic benchmarks usually provide clearly defined tasks and feedback signals, so an error can often be connected directly to a potential improvement. Production traffic, however, is much noisier and more ambiguous. It can therefore be difficult for a self-improvement system to determine what it should learn from an interaction and what it should change.
Second, production agentic AI systems evolve extremely quickly. Building reliable systems in this environment requires balancing two priorities: moving fast enough to keep up with product development while maintaining the rigor needed to measure whether a change actually improves the system. I learned that rapid iteration and careful experimentation are not competing goals—they need to support each other.
What did you enjoy most about your internship?
I was fortunate that the World Cup took place during my internship. Our shared enthusiasm for soccer helped me connect with many people across Microsoft, including both fellow interns and full-time employees. Watching and discussing the games together was one of the most enjoyable parts of my experience, and it helped me form friendships beyond my immediate team.
What did you find most challenging or surprising?
One of the most surprising things I learned was how increasingly fluid the boundaries between different technical roles have become because of AI. As an applied scientist or data scientist, it is no longer sufficient to only design experiments and analyze results. We also need to understand the underlying systems, implement our ideas, and help launch features directly into production.
This was challenging, but also exciting. It encouraged me to develop a broader skill set and think about the full lifecycle of a feature—from identifying a problem and designing an experiment to implementation, evaluation, and product deployment.
How did your Biostatistics training prepare you for the internship?
My Biostatistics training gave me a valuable mindset for designing and interpreting experiments. It taught me to define outcomes carefully, choose appropriate comparison groups, identify possible sources of bias and confounding, and distinguish a true improvement from random variation or changes in the experimental environment.
These skills were especially useful when evaluating rapidly changing agentic AI systems. Because models, prompts, tools, and runtime environments can all evolve, experimental results must be interpreted carefully. My statistical training helped me approach these evaluations rigorously and communicate both the findings and their limitations clearly.
By Deb Nelson
Learn more about applying to the Biostatistics PhD program at UW.
