Don | TIELEC Blog

When creating a questionnaire to assess reflective capacity, I became convinced that this issue needed to be addressed head-on.

"Won't this questionnaire just make people with poor metacognition say 'yes' more, making it an inaccurate assessment?"

This structural contradiction is well-known to anyone involved in psychometrics. While I understood it theoretically, actually being on the design side made me see its weight differently.

What is the Basic Assessment?

For the reflective capacity assessment, I designed two approaches:

Basic Assessment: Questionnaire format to understand your tendencies (20 questions)
Advanced Assessment: Free-form writing to reflect on a single event (LLM analysis)

This article focuses on the structural contradiction I faced when designing the Basic Assessment (questionnaire type) and its solutions.

The actual tool:

🔗 https://metacog-assess.tool.tielec.blog

Turning ORIMD into a Questionnaire

First, as a premise, I was attempting to measure the ORIMD (Observation → Reaction → Interpretation → Meaning → Decision) reflection framework by breaking it down into five abilities:

O: Observation Capacity - The ability to grasp facts without evaluation
R: Emotional Recognition - The ability to notice your own emotions
I: Interpretation Visualization - The ability to recognize your assumptions
M: Meaning Construction - The ability to transform experiences into learning
D: Decision Conversion - The ability to translate insights into action

Four questions per ability, 20 questions total. Five-point scale (strongly disagree to strongly agree).

For example, questions like these:

I-1: I notice when my thoughts are "interpretations" rather than "facts"

M-1: When I reflect on events, I often find my own learnings and insights

Technically, I thought this was a sound design. I avoided licensing issues with existing psychological scales. While referencing evidence-based theories (Flavell, Schön, etc.), the questions were entirely original.

But there was a fundamental flaw here.

For those interested in learning about metacognition systematically, this book is highly recommended.

[📦 商品リンク: moshimo-card-6tFbF]

Direct Hit Zone of the Dunning-Kruger Effect

"People with poor metacognition will just say 'yes' more"

This is a well-known problem in psychometrics. It's in the direct hit zone of the Dunning-Kruger effect (cognitive bias).

People with weaker metacognition
"assume they're doing well and press Yes"

Why does this happen?

People with weak metacognition can't objectively view their own state
They don't know the standards for reflection, so they misidentify themselves as "doing well"
They tend to overestimate their abilities

Conversely, people with higher metacognition:

Recognize their own limitations
Have strict standards, so they answer conservatively
Are more likely to feel "I'm not there yet"

In other words, people with lower abilities in what you're trying to measure score higher, and those with higher abilities score lower — a reversal phenomenon.

This is fatal.

For those interested in deepening their understanding of cognitive biases, this book is a valuable reference.

[📦 商品リンク: moshimo-card-RiXKq]

The Root Problem: Directly Asking About Ability

Looking back at the questions carefully, many of them were:

"can do ~"
"do ~"

These are direct self-assessments of ability.

Example:

When reflecting on events, I can organize just "what happened" without mixing in evaluations or emotions

This directly asks about "observation capacity."

But people with low observation capacity can't even notice that they're "mixing in evaluations or emotions." So they answer "I can do it."

This is the greatest challenge in questionnaire design.

Solution ①: Shifting to Behavioral-Based Questions

The standard solution in psychometrics is to ask about behaviors and specifics, not abilities directly.

❌ Bad Approach (Direct Ability Assessment)

I can objectively grasp events

✅ Good Approach (Behavioral-Based)

When reflecting on events, I often start by writing "what happened" in bullet points

Can you see the difference?

The latter asks "what do you do?" rather than "can you do it?" This allows people to answer based on whether they actually engage in that behavior.

Behavioral-Based Question Examples for Each ORIMD Dimension

O (Observation):

Specific behavior of "writing without evaluation"
Specific behavior of "organizing chronologically"

R (Reaction):

Specific behavior of "using two or more emotion words"
Specific behavior of "referring to bodily sensations"

I (Interpretation):

Specific behavior of "writing 'I thought...'"
Specific behavior of "listing alternative perspectives"

M (Meaning):

Specific behavior of "using abstract terms (learning, patterns)"
Specific behavior of "referring to other situations"

D (Decision):

Specific behavior of "writing what to do next time in one sentence"
Specific behavior of "deciding in If-Then format"

This is the most critical point. Ask about specific behaviors where abilities manifest, not the abilities themselves. This significantly reduces self-assessment bias.

Solution ②: Reverse Items (Calibration)

Another technique is introducing reverse items.

This involves deliberately including items that people with weak metacognition are more likely to say "yes" to.

Example (I: Interpretation Visualization)

I generally think my thoughts are correct

People with high metacognition answer "disagree" to this question (because they understand their thoughts are merely interpretations).

But people with low metacognition are more likely to answer "agree."

By reverse scoring this response, we can detect self-assessment distortion.

Other Examples

M (Meaning Construction):

I often feel that even when I reflect, I gain little from it

D (Decision):

Even when I think I've done good reflection, my behavior often doesn't change

By mixing in items like these, we can correct for the tendency to unconsciously press Yes repeatedly.

Solution ③: Other Considered Techniques

I also considered a technique of designing so that perfect scores are impossible.

What does this mean?

It's a method of deliberately making question pairs contradictory.

Example (O: Observation Capacity)

Question A:

When reflecting on events, I separate facts from emotions

Question B:

When reflecting on events, I integrate facts and emotions

Both are advanced forms of reflection. But answering "strongly agree" to both is contradictory.

People with truly high metacognition are aware of which one they use, or whether they switch depending on the situation, so they answer carefully.

But people with low metacognition are more likely to answer "agree" to both.

This technique of detecting such contradictions to correct unconscious high self-ratings also exists.

However, I ultimately prioritized keeping the question count down and didn't adopt this technique.

Implementation Plan Going Forward

Based on these techniques, I plan to revise the Basic Assessment questionnaire.

Specifically, while maintaining the structure of four questions per ability:

Center on behavioral-based questions
Appropriately place reverse items
Reduce direct ability assessment questions

I believe this will allow us to correct self-assessment bias from multiple angles.

But This Still Isn't Perfect

Even with all this, it's not perfect.

As long as it's self-assessment, there are inevitable limitations.

Particularly:

Misalignments in self-awareness can't be completely eliminated
Social desirability bias (wanting to look good) remains
Cultural background affects evaluation standards

That's why how we present diagnostic results becomes important.

Don't treat scores as absolute; present them as tendencies.
Not "high/low" but "this pattern is observed."
Design so that perfection isn't expected.

Summary of Technical Decisions

Questionnaires measuring metacognition — Basic Assessments — have a structural contradiction:

People with lower abilities tend to rate themselves higher

This contradiction can't be completely solved, but can be significantly improved with these techniques:

Don't ask about abilities directly - Ask about behaviors and specifics
Mix in reverse items - Detect unconscious Yes-pressing

And most importantly, designing while being aware of these limitations.

Questionnaires are not omnipotent.
Self-assessments have distortions.
Yet they still have value in functioning as a mirror.

With that premise, how do we create them?

Continuing to hold this question is what I now think is most important when creating questionnaire-type diagnostic tools like the Basic Assessment.