The Weekly Habit That Keeps a Director’s AI Intuition Sharp

The leaders making the best AI decisions aren’t reading more. They’re running their own experiments every week.

Ailian Gan runs model evaluations every week. Not her team. Her.

She’s Head of Agents at Superhuman now. Before that, she was Director of Product at Grammarly and Lead PM of AI at Zoom, where she shipped Zoom AI Companion. She has a team. She has the seniority to delegate. And she still opens ChatGPT and Claude every week to test something new herself.

When I asked her why on Episode 15 of The AI Product Leader, her answer landed like a gut punch:

“There’s always this pull to just trust the team, hand it off. But with AI, if I’m not in it, I lose the intuition. And once you lose the intuition, you lose the ability to make good calls fast.”

That word. Intuition. Not knowledge. Not information. The thing that lets you walk into a meeting where something has gone sideways and know, in your body, what happened before anyone explains it.

Why intuition decays so fast

AI moves differently than any technology wave before it. A model update can change your product’s behavior overnight. A capability that didn’t exist last month is suddenly table stakes. The gap between where the technology is and where your understanding is grows quickly if you stop paying attention.

Ailian feels that pressure viscerally. She described the discipline of staying current:

“Every week I try something new. Even if it’s just playing with a new feature in ChatGPT or running a test in Claude. Because if I stop, I fall behind.”

Most senior leaders I talk to on the podcast know they should be doing this. They understand the argument intellectually. But the calendar fills up, the team seems to have it handled, and weeks pass. Then months. Then you’re in a product review asking questions you already know the answers to because you read about the update. You didn’t feel it. You didn’t watch it break.

That’s the gap Ailian refuses to let open.

The quiet fear nobody talks about

This isn’t about looking uninformed in front of your team. Ailian has the resume and the track record to coast on credibility for a long time. The fear that drives her weekly practice is more personal than that.

It’s the fear of making a bad call. Approving an approach that sounds right but falls apart in edge cases you would have caught if you’d been closer to the models. Greenlighting a timeline based on a capability you assumed still worked the way it did three months ago. Sitting in an architecture review and not being able to push back on something that feels off, because you’ve lost the feel for what “off” even means.

“You can’t just read about what the model does. You have to feel the failure modes yourself. When you’re in the meeting with engineering and something goes wrong, you have to know exactly what you’re talking about.”

That’s the player-coach discipline in one sentence. You stay hands-on so your judgment stays sharp. The alternative is slow, invisible erosion. Nobody tells you it’s happening. You just start making slightly worse decisions, slightly more often, and you’re the last person to notice.

What the weekly practice actually looks like

Ailian’s approach is deliberately low-friction. She picks something small. A new feature. A different model. A prompt she rewrites to see how the output shifts. It takes maybe an hour. Sometimes less.

The point is not to build something production-ready. The point is to keep her hands dirty enough that when a decision lands on her desk, she has recent experience to pull from. She’s not relying on a briefing doc or a vendor demo. She has her own data.

This is what separates a leader who understands AI from a leader who manages AI. Managing means reviewing what your team produces. Understanding means you’ve personally felt what happens when you push the model past its limits, when you change the temperature, when you feed it messy data and watch how it recovers or doesn’t.

A practice, not a phase

The temptation is to treat hands-on time as a ramp-up period. Get smart on the tools, build confidence, then graduate back to pure leadership. Ailian’s career proves that’s backward. She’s been in AI product leadership for years. Shipped one of the most visible AI products in the world. And she still runs weekly experiments.

Because the technology doesn’t slow down when you stop. Your intuition starts to erode the moment you pull back, and it compounds. A week is nothing. A month is noticeable. A quarter and you’re leading from secondhand information.

The weekly habit is the discipline. Small, consistent, unglamorous. Nobody sees you doing it. Nobody applauds it. But the next time you’re in a meeting and something goes wrong, you’ll know exactly what you’re talking about.

That’s worth an hour a week.

Polly Allen is the founder of AI Career Boost and host of The AI Product Leader podcast. She spent years leading AI at Amazon Alexa before building the AI Career Boost Blueprint, an 8-week program for Director+ product leaders becoming indispensable AI player-coaches. Subscribe to The AI Player-Coach newsletter →