Building the course takes a week. You then own it for three years.
Every AI feature in this market is aimed at that first week. Generate the outline, draft the lessons, write the quiz, make the images. Then the tool goes quiet, and the part that actually determines whether the training worked – the thirty-four months after launch – is back to being a human with a spreadsheet and no time.
Running courses with AI agents is the other half, and it’s the half almost nobody has shipped. This is what it looks like in practice: the prompts, the data it needs, and the two things you should never hand over.
Try Mini Course Generator free – build a course, then ask your assistant how it’s doing.
Key Takeaways
– Courses rot in four specific ways: the facts change, the quiz stops discriminating, a lesson starts losing people, and the cohort changes. Each one is detectable from data the platform already has.
– A connected agent can find all four by being asked, in plain language, in under a minute – because it reads live results rather than a report somebody exported last quarter.
– The highest-value single move is quiz hardening: a question 95%+ of learners pass first time is measuring nothing and is hiding the gap you built the course to close.
– Automate detection and drafting. Do not automate publishing changes to compliance content, or sending anything to learners, without a human in between.
– A workable cadence is one fifteen-minute review per course per month, which is roughly the effort level at which this actually gets done.
Why Courses Rot, Specifically
“Content goes stale” is too vague to act on. There are four separate failure modes and they need different fixes.
The facts changed. A price, a policy, a screenshot, a step in a process. This is the most obvious kind and the easiest to fix, and it still sits broken for months because the person who changed the policy isn’t the person who owns the course.
The quiz stopped discriminating. A question everybody gets right isn’t an assessment, it’s a formality. This one is invisible without looking at the data, because a 98% pass rate reads as good news on every dashboard ever built.
A lesson started losing people. Completion drops at a specific point. Something in that lesson – length, a confusing example, a broken embed – is doing the damage, and the aggregate completion number hides which one.
The cohort changed. The course was written for new hires with retail backgrounds and now half of them come from hospitality. Nothing is factually wrong. It’s just aimed at the wrong reader now.
The common property: all four are visible in data the platform already holds, and none of them announce themselves. Somebody has to go and look. That’s the job that never gets done, and it’s the job an agent is genuinely good at.
What the Agent Can Actually Do
Four moves, with the prompts. These assume an assistant connected to the platform over an MCP server, so it’s reading live results rather than guessing.
1. Find the quiz questions that aren’t testing anything
“Look at the quizzes in the Customer Onboarding course. List any question where more than 90% of learners answered correctly on the first attempt, and tell me which ones are testing something that matters.”
The first half is arithmetic. The second half is the useful part: separating questions that are easy because the concept is easy from questions that are easy because the phrasing gives it away.
Then the follow-up:
“Rewrite question 4 so it tests whether they can apply the refund rule to an edge case, rather than whether they read the sentence. Keep the same learning objective.”
This one move has the highest payoff of the four, because a broken quiz is worse than no quiz. It produces a completion record that says people learned something when the evidence says nothing of the kind.
2. Find the lesson that’s losing people
“For each lesson in this course, show me the completion rate and the average time on page. Flag anything where completion drops more than 15 points from the previous lesson.”
The drop-off point is nearly always more specific than “the course is too long”. Once you know it’s lesson six, ask what’s in lesson six – a fourteen-minute video, three concepts crammed together, an embed that doesn’t load on mobile.
3. Keep the facts current
“Our refund window changed from 14 days to 30. Find every lesson and quiz question across all published courses that references the old window, list them, and draft the corrections.”
Note the shape: find and draft, then stop. The agent produces a list and proposed edits. A human approves. That boundary matters and the next section explains why.
4. Nudge the people who stalled
“Who started this course more than three weeks ago and hasn’t finished? Group them by which lesson they stopped at.”
The grouping is what makes this worth doing. Twenty people stuck at the same lesson isn’t a motivation problem, it’s a content problem, and the fix is to repair lesson six rather than to email twenty people about their commitment.
The Data This Needs
None of the above works on a generated course sitting in a document. It needs three things, and it’s worth checking a platform has all three before assuming any of this is available.
| What’s needed | Why | What to check |
|---|---|---|
| Per-question results | Quiz hardening is impossible from a pass/fail score alone | Can you see first-attempt accuracy per question? |
| Per-lesson engagement | Finding the drop-off point needs lesson-level data | Completion and time-on-page per lesson, not per course |
| Agent read access to both | An agent that can’t read results can only guess | Does the connection expose analytics, or only authoring? |
That third row is the one that catches people. Plenty of tools now let an agent create a course. Far fewer let it read what happened afterwards, and a connection that only writes can do exactly one of the four moves above.
What to Automate and What Not To
The genuinely useful line isn’t “automate everything” or “trust nothing”. It runs between detection and consequence.
Automate freely: finding the stale facts, flagging the questions that don’t discriminate, spotting the drop-off lesson, drafting the rewrite, summarising a course’s health. All of these produce information, and the worst case is that a human ignores it.
Keep a human in the middle for two things:
Publishing changes to compliance content. If a course exists because a regulator, an auditor or a contract requires it, the version history is evidence. An agent silently improving the wording of a policy course is a problem for whoever has to prove what people were told and when. Draft, review, publish deliberately. Where a requirement renews on a cycle, the state you actually need to track belongs in a training matrix rather than a completion report.
Anything that reaches a learner. Nudges, reminders, emails. Not because the agent writes badly, but because volume is invisible from inside a prompt. “Email everyone who hasn’t finished” is a reasonable instruction that becomes four messages in a week to somebody who’s been on parental leave.
Everything else can run on the agent’s judgement, and treating the whole category as high-risk is how teams end up doing none of it.
A Cadence That Actually Gets Done
Ambitious maintenance schedules don’t survive contact with a quarter. This one is short enough to survive.
Monthly, per active course, about fifteen minutes:
- Ask for a health summary: completion, quiz performance per question, the biggest drop-off point.
- Fix one thing. The single highest-drop lesson, or the single least discriminating question. One, not a list.
- Ask what changed in the source material since last month, if the course was built from documentation.
Quarterly, across the library:
- Ask which courses haven’t been touched in six months and have live enrolments.
- Ask for a scan for facts that reference anything you know has changed – pricing, policy, tooling, org structure.
- Retire something. A library nobody prunes becomes a library nobody trusts.
The reason to write the cadence down is that this work has no deadline attached to it. Nothing breaks visibly when a course goes stale, which is exactly why it stays stale.
Why Most Tools Stop at Publish
Worth understanding, because it explains why this is hard to buy rather than being a feature anyone could add.
Post-publish work needs three things at once: the content, the learner data, and an agent with permission to act on both. Most AI course tools are authoring tools – they generate content and hand it over, so there are no learners and no data to read. Most LMS platforms have the learners and the data but treat AI as a feature inside their own interface rather than something an outside agent can drive.
The combination requires the platform to expose its own operations to an agent, which is an architecture decision rather than a feature. That’s the distinction underneath the term agentic LMS, and it’s why the third stage of the loop is where the category actually thins out. It’s also the capability we rated most under-used in our maturity map of AI in learning and development.
The Short Version
The week you spend building a course is the small part. The value comes from the years afterwards, and that’s the stretch every AI course tool leaves you to handle alone.
A connected agent changes the economics of maintenance, because the expensive part was never the fixing – it was noticing. Quiz questions that stopped discriminating, the lesson where completion falls off a cliff, the policy that changed in March: all of it is visible in data the platform already has, and all of it stays invisible until somebody asks.
Automate the noticing and the drafting. Keep a human between the agent and two things: compliance content going live, and anything landing in a learner’s inbox. Then run fifteen minutes a month per course, which is short enough that it’ll actually happen.
If you want to see what the loop feels like, connect an assistant to a course you’ve already published and ask it which quiz question is doing the least work. The answer usually arrives faster than the argument about whether to try it.
Frequently Asked Questions
Can an AI agent update a course after it’s published?
Yes, if the platform exposes both its authoring and its analytics to the agent over a connection like MCP. The agent can read live learner results, propose changes, and apply them. A tool that only generates content at the start cannot, because it has no visibility into what happened next.
What should I ask an agent to check on a live course?
Three things cover most of the value: which quiz questions more than 90% of learners pass on the first attempt, where completion drops most sharply between lessons, and whether any content references facts that have since changed. Each maps to a distinct failure mode with a distinct fix.
Is it safe to let an agent change published training?
For general content, yes, with review. For compliance training, keep a human approval step: the version history of a mandatory course is evidence, and silent edits undermine it. Also keep a human between the agent and anything sent to learners, because message volume is hard to judge from inside a prompt.
How often should training content be reviewed?
Monthly per active course is realistic if the review is fifteen minutes and fixes one thing. Quarterly, scan the whole library for untouched courses with live enrolments and retire what’s no longer used. Longer intervals sound reasonable and reliably become never.
Why do quiz questions stop being useful?
Usually because the phrasing gives the answer away, or because the concept became familiar through the lesson immediately before it. Either way a 95%+ first-attempt pass rate means the question isn’t discriminating, and the completion record it produces overstates what learners can actually do.
Does this need a special LMS?
It needs a platform that lets an outside agent both author and read analytics. Many platforms now have AI features inside their own interface; far fewer let an agent you control drive those operations. That capability is the practical difference, and it’s worth confirming in a demo rather than from a feature list.
