This post was co-authored by Ryan Allen and Loyal Tingley.
At Zocdoc, we’re constantly improving our products to make it easier for patients to access care and for providers to grow their practices. Rather than relying on intuition alone, we use controlled experiments to measure the impact and success of new products or technical features with real-world data.
Not every experiment succeeds. When a test doesn’t perform as expected, we can simply discard it and move on. But sometimes an experiment contains promising code, infrastructure, or design improvements even if the overall results fall slightly short. Often, in those cases, we iterate on the experiment instead of starting over. We’ve found that a few thoughtful refinements can turn an underperforming change into one that matches or exceeds the original experience while also improving developer experience and reducing technical debt.
We tend to invest more time and effort into improving something that initially underperformed, but a winning experiment is simply concluded as successful without the same level of additional consideration. The time and effort required to run an experiment did not support working on smaller modifications. Evaluating results in particular required time and effort that was best reserved for larger changes. But in late 2025, during a large redesign of the mobile apps, we had an opportunity to run a tightly-scoped test clarifying the copy on our pre-appointment page. This seemingly trivial change was a measurable success for our patient experience.
We could not help but wonder if there are other simple changes that could be applied on top of the redesigned pages. Our use of coding agents at Zocdoc gave us the opportunity to try. We had seen AI agents one-shot tasks like writing unit tests for existing code or performing small tech debt cleanups. We decided to see if agents could do as well with small incremental improvements like the copy change above.
We identified three stages to building out this workflow: product ideation, writing code, and evaluating the experiment results. Each of these steps had potential to be automated by an agent with appropriate human supervision, especially if we targeted small changes localized to individual screens on the app. Since writing code for small changes had already proven to work well, we began testing out if agents would be able to produce actionable ideas.
Product Ideation
A human product manager requires a deep understanding of users, the product, and the creativity to pull those two together into a story that makes a meaningful improvement in business metrics and user experience.
So, understandably, our coding agent failed fast. After telling our agent that Zocdoc’s iOS app primarily relies on patient bookings to drive business metrics, and giving it access to the code that represents our pre-appointment page, the first answer it gave was clear: “Remove the cancellation button, it’ll drive more appointments”. This quickly reminded us that we’re not working with humans here.
But this was salvageable. If you work at Zocdoc, you’ll hear about a lot of metrics, but almost everything comes back to our value of being patients first. In practice this means empowering the patients and providers using our services. So, the AI reviewing and providing suggested improvements needs to embody these values as well.
We did some back and forth refinement of our AI workflow skill by including company values, instructions on exploring the codebase, and categories of improvements we’d like to see: easing of friction (like confusing flows, unnecessary steps, or actions without feedback), copy improvements (like vague button text, jargon, or missing instructions), opportunities for reminders (like notifications, post-appointment suggestions, calendar events), and so on. With this we have reached a place where the agent suggests something promising around half the time. With every run, we generate enough ideas to produce several viable candidates, all small, targeted product optimizations.
Implementation
From this initial attempt, we grew the skill into a full product refinement workflow to support the process from idea generation through implementation.
First, we can have the agent create items with our internal ticketing system for each improvement we have chosen to work on. Because the workflow can use the selected idea as context, the contents of the ticket can be automatically populated with a clear summary and rationale. During the ideation step, we also generate a structured prompt that can be directly pasted during implementation.
Because AI can misunderstand, overcorrect, and make mistakes, like the cancellation example, we are deliberately including some human touchpoints. We want a person to review the generated ideas to ensure they are aligned with our values and our users’ needs. We also want a few experts to perform a light touch review of the output: a developer reviewing the code, a product manager approving the idea, and a designer approving the UI (User Interface). These are necessary as final checks for blocking issues, but we want to avoid them becoming red tape which slows the process down. Part of the workflow produces written overviews with before / after screenshots to pass on, allowing for very simple review from stakeholders.
Outside of the feature itself, the workflow can also help with experiment setup, consistency with our design system, and test coverage. With only minimal human review, the agent can perform most of the implementation work for these small changes.
Launch and Readout
One of our major wins so far has not been a direct change, but instead working with our data teams to create AI-assisted experiment readouts. This allows us to evaluate our changes through controlled experiments and human-reviewed analysis, but removes a significant bottleneck asking data analysts to do automatable lookups.
Because we operate in healthcare, AI-assisted workflows are designed with privacy, data protection, and human accountability in mind. The goal of readout automation is to make experiment analysis easier to understand and act on, while still protecting patient and provider privacy, and preserving right human judgment around whether an experiment succeeded or failed. This is tricky, but follows a lot of the same concepts as building our product ideation skill. The skill needs to embody the same principles that we would expect a Zocdoc product analyst to act on, confirming user satisfaction with quantifiable and non-identifying data.
The readout workflow is one of the areas where we see the most potential for future exploration. With that being said, we’ve already built a useful workflow that can surface the most important experiment signals. With a quick exchange, the AI agent has enough information to respond with how much longer the experiment needs to run or how well it performed.
Conclusion
We’re excited to see that some of the experiments we have run are already proving to help patients find care. In addition, the toolkit created around this has enabled us to trial good ideas that otherwise would have been hard to prioritize. Not only is each of these experiments an opportunity to learn what works best for our patients and providers, but we are now better equipped to act directly on their feedback as well.