A candidate interviews brilliantly. The hiring panel likes their energy, communication is smooth, and everyone leaves the room saying some version of, “They just feel right.”
Three months later, the same hire avoids feedback, clashes with the manager’s style, and frustrates teammates who expected more collaboration than self-promotion. The problem usually isn’t that the team missed a hidden personality flaw. The problem is that they assessed the wrong things, or assessed the right things too loosely to trust the result.
That’s where an example of content validity becomes useful. Not in an academic, journal-article way. In a practical hiring way. If your assessment claims to measure values alignment, culture fit, judgment, or collaboration, the actual questions need to represent those domains well enough that the score means something.
Most HR teams understand this in principle. What they often don’t have is a workflow that scales. Manual SME review is slow. Spreadsheet-based item tracking breaks down fast. And the guidance people find online often stops at theory. One reason this topic matters now is that practical guidance on using automation to support SME feedback, CVR calculations, and cohort comparison in culture assessments is still thin, despite clear demand for more scalable workflows, as noted in Scribbr’s overview of content validity methodology.
Why 'Gut Feel' Hiring Fails and What to Do Instead
A “gut-feel” hire usually looks strong at the point of selection because interviews reward fluency, confidence, similarity, and timing. None of those are the same as fit.
When HR teams tell me a culture-fit process isn’t working, I usually find one of two issues. Either they never defined what “fit” means in observable terms, or they built an assessment that samples only a narrow slice of the construct. A collaboration assessment might really be a communication test. A values screen might really be a likability filter.
What gut feel gets wrong
Hiring by intuition tends to overvalue signals that are easy to notice and undervalue signals that are job-critical but less visible in a short interview.
Common failure patterns include:
- Charm over substance: Candidates answer smoothly, but the questions never test whether they handle conflict, ambiguity, or accountability in the way your culture requires.
- One-domain overreach: A manager sees strong initiative and assumes the person will also collaborate well, take feedback well, and model company values.
- Undefined culture language: Teams say they hire for “ownership” or “candor,” but each interviewer uses a different private definition.
Practical rule: If two interviewers can’t describe the same construct in the same way, they aren’t measuring it. They’re reacting to it.
What to use instead
The alternative isn’t more complexity for its own sake. It’s structured content validity.
That means you define the construct first, break it into domains, write items that represent those domains, and ask subject matter experts whether each item is essential. Instead of trusting whether a question “sounds good,” you test whether knowledgeable reviewers agree it belongs.
In practice, this shifts the conversation from “Do we like this question?” to “Does this question represent the job or value domain we say we care about?”
That’s a better standard. It’s more defensible, more repeatable, and much easier to improve over time.
What Content Validity Really Means for Hiring Assessments
A simple way to understand content validity is to think about a certification exam for a chef. If the exam mostly asks about baking terminology, but the job requires menu planning, food safety, prep timing, line coordination, and sauce execution, the exam doesn’t represent the full job. It lacks content validity.
The same logic applies in hiring.
If your assessment claims to measure collaboration, but the items mostly ask whether someone likes group work, you’re not covering collaboration. You’re measuring a preference. If it claims to measure values alignment, but all the items focus on customer service while ignoring accountability and feedback behavior, the score is incomplete by design.
The practical definition
Content validity is the degree to which an assessment covers the full domain it claims to measure.
That sounds abstract until you apply it to a real hiring tool:
- For a customer success role, the domain might include de-escalation, expectation setting, follow-through, and internal coordination.
- For a values alignment assessment, the domain might include how someone responds to feedback, handles trade-offs, raises concerns, and prioritizes team norms.
- For a manager assessment, the domain might include coaching, decision quality, fairness, communication, and accountability.
A useful primer on the broader concept appears in MyCulture.ai’s article on the meaning of content validity.
How expert judgment becomes a metric
The most widely known method here is Lawshe’s Content Validity Ratio, or CVR. The formula is CVR = (nₑ - N/2) / (N/2), where nₑ is the number of experts who rate an item as “essential” and N is the total number of experts. The result ranges from -1 to +1, which gives you a clear way to quantify expert agreement about whether an item belongs in the assessment. The same source also notes an important correction to Lawshe’s original critical values table, which had been mislabeled as one-tailed when the values reflected a two-tailed binomial basis, improving the accuracy of significance decisions (Wikipedia summary of content validity and CVR).wikipedia.org/wiki/Content_validity)).
That matters because hiring teams often confuse agreement with evidence. A room full of managers nodding at a question is not the same as a documented item review process.
A good assessment item doesn’t just sound relevant. It survives expert scrutiny against a defined content domain.
A quick example
Suppose your company values “radical candor.” If your item asks, “I enjoy open communication,” it may feel aligned, but it’s too broad. Enjoying openness isn’t the same as giving direct, respectful feedback when stakes are high.
A stronger item would target the actual construct more closely, such as behavior under tension, upward feedback, or correction in real work settings. Content validity starts there. Not with elegant wording, but with representativeness.
How to Build an Assessment Blueprint from Job Roles
Most bad assessments are written too early. Teams jump into question writing before they’ve built the blueprint.
That sequence causes drift. The items end up reflecting whatever is easiest to phrase, not what the role requires. A content-valid assessment starts with construct definition, not item brainstorming.
Start with the construct
Establishing content validity begins with a deductive process. You define the construct in operational terms, generate an item pool from literature and SME input, and then have 3 to 5 or more independent experts rate how well each item represents the construct. Assessments that skip this foundation often miss important behavioral dimensions and end up with poor construct representation (Psychology on content validity in research).org/content-validity-in-research-definition-examples.html)).org/content-validity-in-research-definition-examples.html)).org/content-validity-in-research-definition-examples.html))....org/content-validity-in-research-definition-examples.html)).
If the construct is “values alignment,” don’t leave it at that. Define what it means in your organization.
For example:
| Construct | Weak definition | Better operational definition |
|---|---|---|
| Values alignment | Shares our values | Acts in ways consistent with stated company values during trade-offs, feedback, and team decisions |
| Customer obsession | Cares about customers | Proactively identifies customer friction, responds to issues, and incorporates feedback into future work |
| Accountability | Takes ownership | Follows through on commitments, surfaces risks early, and accepts responsibility without deflection |
Build domains and behaviors
Once the construct is defined, break it into domains. Then convert each domain into observable behavior.
A simple blueprint looks like this:
- Role or value domain
Example: Customer obsession.
- Sub-domains
Proactive support, problem resolution, feedback integration.
- Observable behaviors
Anticipates issues before escalation. Closes loops with stakeholders. Adjusts process after recurring feedback.
- Assessment item targets
Situational judgment prompts, behavioral statements, or manager-rating items tied to those behaviors.
Teams can borrow from instructional design principles that emphasize alig...com/blog/instructional-design-principle) that emphasize alignment between learning goals, activities, and evaluation. The same logic works in hiring. If the target behavior isn’t clear, the assessment won’t measure it consistently.
Use job analysis, not slogans
The blueprint should be grounded in actual work. Review job descriptions, performance expectations, onboarding docs, and manager input. Then compare those materials against what high performers do.
If you need a starting point, MyCulture.ai has a practical example of a job analysis that shows how role requirements can be translated into measurable domains.
The blueprint is your control document. Once it exists, every item either maps to it or gets cut.
That one discipline prevents a lot of weak item writing.
The Expert Review Method to Validate Your Questions
Once you have a blueprint, the next job is brutal in the best way. Put every item in front of people who know the role or construct well enough to reject weak questions.
Many teams find this uncomfortable. They’ve invested time in writing items, and SME review often reveals that several questions are vague, redundant, or only loosely connected to the intended domain. That’s not a failure. That’s the point.
Who should count as an SME
For hiring assessments, a strong SME panel usually mixes different kinds of expertise:
- High-performing incumbents: They know what competent behavior looks like in the flow of work.
- Experienced managers: They see patterns across hires and understand which behaviors matter under pressure.
- People with assessment expertise: I/O psychologists or trained HR specialists can spot construct drift, double-barreled wording, and avoidable bias.
You don’t need a panel full of people with the same perspective. You need informed reviewers who can judge whether an item is essential to the domain.
What SMEs should rate
The classic Lawshe approach uses a three-point rating scale:
| Rating option | What it means |
|---|---|
| Essential | The item is necessary to represent the construct |
| Useful, but not necessary | The item may help, but the assessment could still cover the domain without it |
| Not necessary | The item doesn’t belong or is too peripheral |
Keep the briefing tight. Give SMEs the construct definition, domain definitions, and clear instructions to judge representativeness, not style preferences.
How CVR works in practice
The content validity workflow involves collecting expert ratings, calculating item-level CVRs, and averaging those values into a Content Validity Index, or CVI. It’s also sensitive to panel size. For example, with 39 experts, an item may need 26 “essential” ratings, about 67%, while with 40 experts, the threshold can still be 26 agreements, about 65%, because the significance criterion changes with panel size (Dovetail’s explanation of content validity and CVR thresholds).
Here’s the practical takeaway: don’t compare raw agreement counts across panels of different sizes without checking the threshold logic.
Suppose you have an item reviewed by an 8-person panel. If 7 reviewers call it essential, that’s strong support. If only half do, the item likely needs revision or removal. The formula turns that judgment into a documented decision trail instead of a debate.
For a broader discussion of validity in HR tools, MyCulture.ai’s overview of validity and assessment is a useful reference.
Don’t ask SMEs whether they “like” an item. Ask whether the assessment loses meaningful construct coverage if that item disappears.
What works and what doesn’t
What works:
- Tight item-to-domain mapping
Turn this into a candidate assessment
Build a culture-fit assessment that compares values, work style, personality, and culture profile signals before the interview.
Create a culture fit assessment- Clear reviewer instructions
- Decision records for revise, retain, or delete
- A second review round for rewritten items
What doesn’t:
- Crowdsourcing feedback from anyone available
- Letting seniority override evidence
- Keeping weak items because they sound modern or brand-friendly
- Treating SME disagreement as noise instead of diagnostic data
Concrete Examples of Validated Assessment Items
Teams usually learn fastest when they can see the difference between a question that sounds relevant and one that maps to the construct. Below are practical examples you can adapt.
Good versus bad items for a value like radical candor
| Item | Better or worse | Why |
|---|---|---|
| “I enjoy honest communication.” | Worse | Too general. It captures preference, not behavior under tension. |
| “When I disagree with a teammate’s approach, I address the issue directly and respectfully rather than avoid it.” | Better | Targets direct feedback behavior in a realistic work context. |
| “I believe transparency is important.” | Worse | Measures abstract belief, not enacted behavior. |
| “If a manager’s decision creates risk for the team, I raise the concern with evidence even when the conversation is uncomfortable.” | Better | Closer to the domain because it captures upward candor and judgment. |
The strongest items usually do three things well. They name a context, signal a behavior, and map cleanly to one domain.
Example of content validity for an acceptable behavior matrix
A useful HR pattern is an acceptable behavior matrix. Instead of measuring only attitudes, it defines what aligned and misaligned behavior look like in the role.
Here’s a simple version for a sales role:
| Competency | Positive indicator | Negative indicator |
|---|---|---|
| Customer integrity | Sets realistic expectations and documents commitments clearly | Overpromises to secure agreement |
| Team collaboration | Shares account context with onboarding and support partners | Hoards information to protect ownership |
| Coachability | Applies feedback to the next customer interaction | Dismisses feedback as style preference |
| Accountability | Flags deal risk early and updates stakeholders | Hides slippage until a deadline is missed |
This kind of matrix helps SMEs judge whether your items cover the behaviors the role depends on.
Use quantitative cutoffs carefully
For the full instrument, S-CVI/Ave scores above 0.90 are considered adequate, and one cited study reduced a 216-item instrument to 133 items after only those items met the stricter S-CVI/Ave = 1.00 standard, which meant a 38.4% reduction to improve precision. The same source notes that for an 8-expert panel, an individual item needs a minimum CVR of 0.75 to be statistically significant (EIMJ paper on CVI and CVR thresholds).
That’s an important lesson for HR teams. More items does not mean better measurement. Well-targeted items beat bloated item banks.
If you want more item inspiration, this MyCulture.ai library of example assessment questions can help with drafting.
Content Validity Review Checklist
Checklist Item | Yes/No | Notes
| Construct is defined in operational terms | | |
|---|---|
| Domains are separated clearly | | |
| Each item maps to one primary domain | | |
| Items use observable behavior or realistic situations | | |
| Double-barreled wording has been removed | | |
| SME instructions define “essential” clearly | | |
| Ratings are documented item by item | | |
| Low-support items are revised or removed | | |
| Overall scale coverage is balanced across domains | | |
| Final decisions are recorded for auditability | | |
Weak items often survive because nobody owns the cut decision. Put one person in charge of the decision log.
How Modern HR Platforms Streamline Content Validity
A familiar failure point shows up around the third or fourth custom assessment. The HR team starts with a clean spreadsheet and a sensible review process. Then business units ask for role-specific versions, hiring managers request item tweaks, and nobody can say with confidence which version was reviewed by which SMEs, or why a weak item stayed in.
That is the point where content validity stops being just a psychometric question and becomes an operating model question.
Manual review still works for a single role or a small item bank. It breaks down when teams need to maintain consistent standards across culture assessments, values tools, behavioral screens, and role-specific variants at the same time. The hard part is rarely writing one good item. The hard part is preserving traceability as the library grows.
Software helps when it handles the work that usually falls through the cracks:
- Blueprint management: Store constructs, domains, and item mappings in one place before any assessment goes live.
- SME review workflows: Give reviewers a standard rating form, keep comments attached to each item, and avoid feedback getting buried in email threads.
5 minutes
to create your first hiring assessment
Use the assessment landing page to choose the right modules and see what the candidate report looks like.
See the assessment builder- Version control: Show which item wording changed, when it changed, and who approved the revision.
- Decision logs: Record why an item was kept, edited, retired, or moved to another domain.
- Cross-role reporting: Compare coverage across roles, teams, or hiring streams without rebuilding the whole process each time.
This matters even more for culture and values assessments. Teams often want local customization for different functions or business units. That can be reasonable. It can also create drift, where each group ends up measuring a slightly different construct under the same label. A platform should make that drift visible early.
MyCulture.ai is one example of a platform used for customizable assessments across values alignment, culture profile, acceptable behaviors, AI readiness, human skills, and Big Five workflows. The practical benefit is not automatic validity. The benefit is tighter process control around construct setup, item libraries, reviewer input, reporting, and downstream use in hiring or onboarding.
That trade-off is worth stating plainly. Tech can scale administration and documentation. Human experts still have to define the construct, judge item relevance, and make the cut decisions. The best platforms support that judgment at scale instead of replacing it.
Frequently Asked Questions About Content Validity
What’s the difference between content validity and face validity
Content validity is about whether experts judge an assessment to represent the full construct adequately.
Face validity is about whether the test appears relevant to the people taking it.
Both matter, but they’re not interchangeable. A test can look polished and job-related to candidates while still missing major parts of the construct. That’s why expert review matters.
How many SMEs should a smaller company use
Use as many qualified reviewers as you can recruit without lowering quality. In practice, smaller firms often start with a focused panel of knowledgeable managers, top performers, and someone with assessment expertise if available.
The key is not collecting opinions from a crowd. The key is getting informed judgments from people who understand the role or construct well enough to spot weak coverage.
Can AI help with content validity
Yes, but in a supporting role.
AI can help organize item banks, detect duplicate wording, summarize SME comments, and speed up review workflows. It can also help teams compare item sets across roles or cohorts.
AI should not be treated as the final judge of whether an item is essential to a construct. That decision still depends on human expertise and documented review.
What is a simple example of content validity in hiring
If a company says it hires for “accountability,” a content-valid assessment won’t ask only whether candidates value responsibility. It will include items that sample behaviors such as follow-through, risk escalation, ownership after mistakes, and response to feedback. If SMEs agree those items are essential to the construct, the assessment is closer to content-valid.
If your team is building culture, values, or behavior assessments and wants a more structured process, MyCulture.ai is worth evaluating. It gives HR teams a practical way to organize constructs, build custom assessments, and bring more consistency to hiring, onboarding, and team development without relying on gut feel alone.