An LMS comparison settles a three-vendor shortlist when you set your category weights before the first demo and score every vendor on the same matrix while the demo is still on screen. Your director will ask why the winner won, and “it had more checkmarks” doesn’t survive that question when two vendors tick the same boxes. Below are the matrix to copy, a way to set your own weights, dealbreaker rows, and a worked example scored to the last point.
Key Takeaways
- A vendor can score 1 on a category weighted 25 and still total 80 out of 100, so count any 1 in a category weighted 20 or more as a Fail.
- Cap every cell by its evidence: 2 if the vendor only said it, 3 if you saw it on their demo account, up to 5 on your own file.
- Keep every weight between 5 and 35, because a category weighted 5 moves the total by 4 points at most.
- Before you recommend, move 10 weight points from the leader’s biggest score advantage to the runner-up’s; in the worked example, the lead is gone at 6 points and reverses past that.
Before you score anyone’s course building, turn one policy document you already own into a short course in an AI LMS and time it. That’s your own “what a 5 looks like” for that row, set before a vendor sets it for you.
Why an LMS Comparison Chart of Checkmarks Ends in a Tie
The usual LMS features comparison is a chart: features down the side and a tick under each vendor that has them. It answers “does it have SSO?” but not the two questions your recommendation depends on.
The first is how well. A tick for SSO looks the same whether single sign-on is in the plan you were quoted or two tiers above it. A tick for reporting looks the same whether a manager sees their own team’s overdue list or needs the vendor’s services team to build it.
The second is how much it matters. On a chart every row weighs the same, so automatic recertification, which decides your next audit, counts as much as leaderboard badges. Three platforms with 30 ticks each come out equal, and the choice falls back on whichever demo felt smoothest.
The matrix below scores how well each vendor did the job, from 1 to 5, and weights how much that job matters to your company. A features checklist still earns its place by finding the lines a vendor can fail outright, and those become your dealbreakers.
The LMS Comparison Matrix to Copy
The matrix has three parts: dealbreaker rows a vendor passes or fails, five to seven weighted categories, and a total. Weights add up to 100. Each vendor gets a score from 1 to 5 per category, and the weighted score is the weight times the score, divided by 5.
A vendor that scores 5 everywhere totals 100. One that scores 1 everywhere totals 20, not zero, so read a 60 as weaker than it sounds.
| Row | Weight (%) | What a 5 looks like | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|---|
| Dealbreaker 1: [a test you can watch] | Pass/Fail | [what passing looks like on screen] | |||
| Dealbreaker 2: [a test you can watch] | Pass/Fail | [what passing looks like on screen] | |||
| [Category 1] | [ ] | [ ] | [score · evidence] | [score · evidence] | [score · evidence] |
| [Category 2] | [ ] | [ ] | |||
| [Category 3] | [ ] | [ ] | |||
| [Category 4] | [ ] | [ ] | |||
| [Category 5] | [ ] | [ ] | |||
| [Category 6, optional] | [ ] | [ ] | |||
| [Category 7, optional] | [ ] | [ ] | |||
| Weighted total | 100 |
Write two things in every vendor cell: the score and a few words of evidence, such as “4 · our PDF, live, one fix needed”. The evidence is what lets a colleague check your score next week.
The 1-to-5 scale
| Score | What you saw |
|---|---|
| 5 | The job done on your own file or test account, in the plan you were quoted, with no workaround |
| 4 | Done on your data, with one workaround you can live with |
| 3 | Shown working on the vendor’s own demo account, or on your data with a workaround you’d resent every week |
| 2 | Described but not shown, or shown only in part |
| 1 | Missing, sold only on a plan above your quote, or the vendor couldn’t show it |
The scale caps each cell by its evidence. A feature the rep only described can’t score above 2, however good it sounds. A feature shown on the vendor’s prepared account can’t score above 3, because that account was built not to fail. Only your file, your test login or your trial earns a 4 or a 5.
Categories to choose from
Pick five to seven. Every “what a 5 looks like” has to be something you can watch happen during a demo or a trial.
| Category | What a 5 looks like |
|---|---|
| Compliance assignment and recertification | Courses assign by role or start date, due dates repeat, and an expired certificate reassigns the course with no admin click |
| Reporting managers will use | A manager logs in, sees only their team’s overdue list, and you schedule a monthly report without vendor help |
| Building courses from your documents | Your policy PDF becomes a published course with a quiz during the call, and an edit keeps finished learners’ records |
| Content you already own | Your own SCORM package plays and writes a status and a score to the report |
| SSO and account provisioning | Learners sign in through your identity provider, and accounts are created and removed automatically, in the quoted plan |
| Learner experience on a phone | A new learner goes from the invite email to the first assigned course on a phone, with no app to install |
| Three-year cost at your headcount | The lowest three-year total, quoted on the same learner count and the same definition of a user |
| Contract and exit terms | A printed notice period and renewal cap, a user count that can go down, and completion history exported as a file at no charge |
| Support after go-live | A named contact and response times by severity written into the contract |
| Security review | A current independent audit report and your hosting region, answered in writing |
Fewer than five categories and one row tends to decide everything. More than seven and you won’t fill every cell in a one-hour demo. For the content-you-already-own row, what a SCORM file contains shows where a package’s version is written. That’s worth knowing before a vendor says “fully compliant.”
How to Set Your Own Weights
Weights are a decision about your company, not a fact about LMSs. Two companies scoring the same three vendors can land on different winners, and both can be right. Set them in this order:
- Name the three failures you’d be blamed for a year from now, such as a failed audit, a rollout nobody finishes or a renewal that doubles. Give the matching categories at least 60 points between them.
- Spread the rest, with no category under 5. At 5, a category moves the total by 4 points at most (from 1 × 5 ÷ 5 up to 5 × 5 ÷ 5). A row worth less than that should go.
- Stop at 35. At 35, one category swings the total by up to 28 points. Anything bigger is a pass-or-fail question for the dealbreaker rows.
- Check the sum is 100 and get your director to agree before the first demo. Weights agreed before anyone saw a score are far easier to defend.
- Don’t change them after scoring. If you have to, rescore all three vendors and write down why.
Here’s how far apart two honest splits can be. Both are examples, with seven categories each.
| Category | Company whose training records get audited, 600 staff, yearly recertification | 40-person company, no IT team, no audit |
|---|---|---|
| Compliance assignment and recertification | 30 | 10 |
| SSO and account provisioning | 20 | 5 |
| Reporting managers will use | 15 | 10 |
| Building courses from your documents | 10 | 25 |
| Three-year cost at your headcount | 10 | 25 |
| Contract and exit terms | 10 | 5 |
| Learner experience on a phone | 5 | 20 |
| Total | 100 | 100 |
The audited company puts half its points on recertification and identity, which an auditor and an IT team will test. The small company puts 70 points on building, cost and the phone. Nobody there has time to build courses by hand, and most of its people work away from a desk. If you sit between the two, start from the closer column and move points one row at a time.
Add Dealbreaker Rows That Override the Total
A weighted total hides a failure. Take a vendor that scores 1 on a category you weighted 25 and 5 on everything else. It totals 80 out of 100 (5 points from that row, 75 from the rest), and it can beat a vendor with no gaps at all.
Dealbreaker rows stop that: they sit above the weighted categories, and a Fail removes the vendor whatever its total. Set them with two rules.
Pick one to three, before the first demo. They come from the musts on your LMS requirements checklist, the lines where a No ends the evaluation. Keep only the ones a vendor on your shortlist might actually fail, because a row all three will obviously pass wastes demo time.
Write each one as a test you can watch. “Supports SSO” passes every vendor. “Learners sign in through our identity provider on the plan in the quote” doesn’t.
If IT also needs accounts created and removed automatically, write that into the test too. Name SCIM, the IETF protocol for managing user identities across systems (RFC 7644), or a native connector to your HR system. A nightly CSV upload is neither, and what an LMS integration involves covers the difference before IT asks.
Then add one rule for the weighted rows: a 1 in any category weighted 20 or more counts as a Fail. A category that heavy matters too much to be bought back with points elsewhere.
Fill In the Matrix During the Demo, Not After
A score written an hour later is a score of the demo you remember, and the smoothest presenter wins that contest. Fill each cell while the task is still on screen.
The LMS demo questions make a ready script, and each of their seven groups feeds a row of this matrix:
| Demo question group | Matrix row it feeds |
|---|---|
| Learner view | Learner experience on a phone |
| Building a course | Building courses from your documents, and content you already own |
| Reporting | Reporting managers will use |
| Admin and automation | Compliance assignment and recertification |
| Integrations and SSO | SSO and account provisioning |
| Data export and exit | Contract and exit terms, plus your export dealbreaker |
| Commercial | Three-year cost, and contract and exit terms |
That guide marks each answer 2 if you saw it on your data, 1 if you saw it on the vendor’s demo account and 0 if you only heard it. Here, a 0 caps the cell at 2, a 1 caps it at 3, and a 2 lets you score 4 or 5 on how well the job went.
Run each vendor the same way:
- Send the vendor your script and one real file 48 hours ahead.
- Bring a second scorer, score separately, and write each score and its evidence before the next task starts.
- Compare scores within the hour, and talk through any cell where you’re 2 or more points apart.
- Rescore after the trial, which counts as your own data and lifts the cap on anything you first saw on the vendor’s account.
A Worked Example: Three Vendors, One Matrix
This is an example, not a record of real platforms. A 400-person company agreed the weights and the two dealbreakers below before its first demo.
Cost is scored by formula so nobody argues about it. The lowest three-year total gets a 5, and every other vendor gets 5 × the lowest total ÷ its own total, rounded to the nearest whole number and never below 1. Vendor C’s $48,000 is lowest, so C scores 5. Vendor B’s $60,000 scores 5 × 48,000 ÷ 60,000 = 4, and Vendor A’s $120,000 scores 5 × 48,000 ÷ 120,000 = 2.
| Row | Weight | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Dealbreaker: SSO through our identity provider, quoted plan | Pass/Fail | Pass | Pass | Pass |
| Dealbreaker: completion history as a file, no paid request | Pass/Fail | Pass | Pass | Fail · export by ticket, billed hours |
| Compliance assignment and recertification | 25 | 5 · rule by start date, our data → 25 | 4 · expired certificates reassigned by hand → 20 | 4 → 20 |
| Building courses from our documents | 20 | 2 · described, not shown → 8 | 5 · our PDF published live → 20 | 4 → 16 |
| Reporting managers will use | 15 | 4 → 12 | 3 · demo account only → 9 | 5 → 15 |
| SSO and account provisioning | 15 | 5 → 15 | 3 · SSO on demo account, provisioning described → 9 | 4 → 12 |
| Three-year cost at 400 learners | 15 | 2 · $120,000 → 6 | 4 · $60,000 → 12 | 5 · $48,000 → 15 |
| Contract and exit terms | 10 | 3 → 6 | 4 → 8 | 2 · no renewal cap printed → 4 |
| Weighted total | 100 | 72 | 78 | 82, removed |
Vendor C has the highest total and the lowest price, and it’s out. Getting completion history back takes a support ticket and billed hours, and this company agreed before any demo that its audit records can’t depend on a paid request. Without the dealbreaker row, C would have won by 4 points.
That leaves B ahead of A by 6. B picks up 20 points over A: 12 on building, 6 on cost and 2 on contract terms. A picks up 14 over B: 6 on identity, 5 on compliance and 3 on reporting.
Check Whether the Weights Decided the Winner
A 6-point lead looks like an answer. Test it before you put it in front of your director.
Take 10 weight points from the row where the leader outscores the runner-up by the most, and give them to the row where the runner-up outscores the leader by the most. In the example, that’s building courses, where B scored 5 and A scored 2, and SSO and account provisioning, where A scored 5 and B scored 3. Moving 10 points from building to SSO sets those weights at 10 and 25:
| Weights | Vendor A | Vendor B | Winner |
|---|---|---|---|
| As agreed: building 20, SSO 15 | 72 | 78 | B by 6 |
| Shifted: building 10, SSO 25 | 78 | 74 | A by 4 |
The winner flips. Each point moved adds 0.6 to A’s total and takes 0.4 off B’s, so B’s lead shrinks by 1 for every point moved and is gone at 6.
To find that rate in your own matrix, add the two score gaps and divide by 5: (3 + 2) ÷ 5 = 1 here. If your lead divided by the rate is more than 10, the winner survives the shift.
When it doesn’t, the matrix hasn’t failed. Three demos have come down to one question for your director: is building courses from your own documents worth more than account setup IT never has to touch? If building is, B wins. If IT’s side is, A wins.
The rule: if the winner survives a 10-point shift, recommend it. If it flips, bring your director the question the shift exposed, with both totals, not a single number.
Run one more check on close results. One scorer’s 1-point disagreement on your heaviest row moves the total by that weight divided by 5, which is 5 points at a weight of 25. A lead smaller than that is a tie, so settle it in the pilot.
Where the Matrix Sits in the Buying Process
This learning management system comparison covers the last three vendors. It should be the one sheet you carry from the first demo to the signature.
Before the demos. If you ran an RFP, bring its answers, not its scores. The LMS RFP template weights its categories for narrowing a wider field, and those weights don’t carry over. A written “Yes” is something the vendor said, so it enters this matrix at 2 at most until a demo shows it.
The cost row. Score a three-year total at the same learner count and the same definition of a user, and date each quote in its cell. If your decision runs past a quote’s validity date, ask for a fresh one. What an LMS costs sets the pricing models side by side, which helps when one quote counts active users and another counts everyone registered. Once the matrix has a winner, calculating LMS ROI turns that cost row into the number a budget owner signs off on.
After the demos. A job that breaks with real learners in the pilot drops a 4 as surely as a clean run lifts a 3 to a 5, but the weights stay put. The finished matrix, with its dealbreakers, evidence notes and the result of the 10-point shift, is your recommendation. The seven steps in how to choose an LMS show where the pilot and the contract review come after it.
Agree the Weights This Week, Before Any Demo
Copy the matrix, pick five to seven categories, and spend 100 points on them starting from the three failures you’d be blamed for. Add one to three dealbreakers you can watch pass or fail, and get your director to sign off on both before the first call. In each demo, score the cell before the next task starts, cap it by what you saw, and run the 10-point shift before you recommend anyone.
Mini Course Generator is our product, and if it’s on your shortlist, score it on the same sheet. Its identity row reads plainly: SSO for learners and the REST API are on the Custom plan, with no SCIM provisioning or native HR system connector. Its other connections are Zapier, Make, webhooks, a WordPress plugin, the MCG App for Intercom and MCP clients such as ChatGPT and Claude. The plans are on the pricing page.
Every plan opens with 14 days of full access and no credit card. Courses export as PDF or SCORM at any time, so you can score both the building row and the exit row on your own material.
FAQ
What should an LMS comparison include?
Five to seven weighted categories that add up to 100, a 1-to-5 score with a few words of evidence for each vendor, and one to three pass-or-fail dealbreakers. A learning management system features comparison with ticks helps you pick the categories, as does knowing what a learning management system does for your team, but neither is the comparison itself.
How is this different from an RFP scoring sheet?
An RFP scoring sheet ranks written answers from five to eight vendors, before anyone demos. The RFP is worth running when policy requires a bid, the contract runs longer than a year or auto-renews, or the vendors you want publish no price. It hands you three finalists, and this matrix scores them on screen. If none of those apply and your three vendors publish their prices and terms, skip the RFP and start here.
How many vendors should go into the matrix?
Three. Past that you can’t fill every cell live, and the scores blur. If you have more, narrow them first on minimum user counts and your dealbreakers, and the best LMS platforms is one place to start that list.
Should cost be a weighted category or a dealbreaker?
Both, in different forms. A hard budget ceiling is a dealbreaker: a three-year total above it fails. Below the ceiling, cost is a weighted category scored by formula, so a cheaper vendor gains points without winning on price alone.
What if two vendors finish within a few points?
Run both through a pilot with 10 to 20 real learners from one team on one real course, inside the vendor’s trial window. If a trial is short, get an extension to one full assignment cycle in writing. Log the admin hours each course took to build and run, read both completion reports with your director, and export the records to confirm another system could read them. Then rescore the cells those results touch, keep the weights, and recompute.
Sources
- IETF RFC 7644, System for Cross-domain Identity Management: Protocol
- Mini Course Generator pricing page
