When building an assessment tool, there's a moment when the results screen becomes a "report card."
Scores appear. Ranks are displayed. Weaknesses are pointed out.
Technically, it's simple. But the moment you do that, the assessment becomes an "evaluation machine."
When implementing the advanced version of the reflection assessment, I struggled considerably with how to confront this problem.
Context: The Roles of Two Assessments
For the reflection capacity assessment, I designed two approaches:
Basic Assessment: Questionnaire format to understand your tendencies (20 questions)
Advanced Assessment: Free-form writing to reflect on a single event (LLM analysis)
Since I covered the Basic Assessment in Part 1, this article focuses on the implementation decisions for the Advanced Assessment (LLM analysis type).
The actual tool:
🔗 https://metacog-assess.tool.tielec.blog
This is the mechanism of the Advanced Assessment, where the LLM analyzes the user's actual reflective writing to observe "how they use each step of ORIMD."
The First Decision: Observation, Not Evaluation
Before starting implementation, there was something I wanted to make clear.
The role of LLM is "observer," not "evaluator"
Evaluator Model (What I Didn't Want to Do)
User's reflection text
↓
LLM analyzes
↓
O: 8 points, R: 6 points, I: 4 points, M: 7 points, D: 9 points
↓
"Your interpretation visualization capacity is low. You should improve."
This is easy to understand. Easy to implement.
But the moment you evaluate with scores, the assessment becomes a report card.
Observer Model (What I Wanted to Do)
User's reflection text
↓
LLM observes
↓
"In this reflection, we see a flow of organizing the event, naming emotions,
then moving to interpretation and action.
Meaning-making was less prominent, but this seems natural given that
this event required practical decision-making."
Not scores, but process description.
Not "good/bad," but a record of "how it was traversed."
However, "making it behave as an observer" was different in implementation than in theory.
Prompt Design: Three Revisions
Simply telling the LLM "write as an observer" didn't work.
First Attempt (Failed)
Analyze the following text and evaluate to what degree
each ORIMD step is being used.
Generated comments:
"Observation is sufficiently present"
"Emotional recognition is lacking"
"The interpretation perspective is weak"
...No good. This is completely evaluator language.
Second Attempt (Still Insufficient)
Observe the following text and describe how
each ORIMD step is being used.
Describe as process observation, not ability evaluation.
This was somewhat better, but still had problems.
"This perspective was not used this time"
This expression seems neutral. But there's a high probability that users will read it as "you couldn't do it".
Third Attempt (Explicit Forbidden Vocabulary)
Ultimately, I needed to explicitly specify the direction of forbidden vocabulary.
# Conceptual example (actual prompt simplified for documentation)
system_prompt = """
You are an observer of reflection processes.
Critical constraints:
1. Do not evaluate abilities
- Do not use expressions that indicate abilities
2. Direction of forbidden vocabulary:
- Expressions indicating lack or deficiency
- Expressions indicating superiority or strength
3. Direction of vocabulary to use:
- State descriptions (foreground/background, etc.)
- Expressions indicating context-dependence
4. Include "context-dependent" vocabulary in all observation comments
"""
With this, "observation-like" language finally started to be generated.
"In this reflection, this perspective was less prominent.
Given the nature of the event, other perspectives may have taken priority."
This is the most critical point. Simply telling the LLM to "write neutrally" is insufficient; concrete vocabulary-level constraints are necessary.
For those interested in more practical knowledge about product development with LLM, the following book is a valuable reference.
[📦 商品リンク: moshimo-book-BKl55]
UI Design: Why I Abandoned Radar Charts
Next, I faced UI design for the results screen.
The Temptation of Radar Charts
Initially, I envisioned a screen like this.
A 5-axis radar chart displaying scores for each ability (O, R, I, M, D).
Technically simple. Visually clear. Easily implemented with Chart.js or Recharts.
But radar charts inherently encourage "comparison".
They almost inevitably create readings like "R is high" and "I is low."
Switch to Flow Display
Ultimately, I changed to this display:
Observation → Reaction → Interpretation → Meaning → Decision
● ● ○ ○ ●
● Focus in this reflection
○ Perspective in background this time
This is not "scores" but a UI showing "the focus of this particular process."
And at the beginning and end of the screen, I wrote:
This is a record of how you traversed this particular event.
It is not an evaluation of your abilities or tendencies.
This level of explicit framing may seem excessive, but is actually about right, I believe.
If a reflection tool hurts users, it defeats the purpose.
Unexpected Problem: R/D Emphasis Bias
When implementation was nearly complete, I noticed something while reviewing the results screen.
In the UI, R and D were highlighted with color, while O/I/M were more subdued.
This could create an unconscious hierarchy reading of "good at emotion and decision" versus "weak in others".
The Structure of the Problem
When analyzing reflection text, actually:
- R (Reaction): "I was happy," "I felt anxious" - appears easily in text
- D (Decision): "Next time I'll do this" - written explicitly
- O (Observation)/I (Interpretation)/M (Meaning): Embedded in text, harder to observe
As a result, the LLM's observation comments had a structural bias toward being more affirming of R and D.
Solution: Emphasize Context-Dependence
I addressed this with prompt adjustments too:
"""
5. Regarding R (Reaction) and D (Decision):
- Given that these appear more easily in text
- Include expressions indicating contextual naturalness
Examples:
"In this event, this perspective naturally came to the foreground"
"This perspective emerged due to the nature of the event"
"""
This allowed promoting a reading that R and D being prominent is due to the nature of the event, not the person's abilities.
Implementation Details (Code Example)
For reference, here's the conceptual structure of the actual prompt:
def generate_observation_comment(user_text: str, step: str) -> str:
"""
Generate observation comment about a specific ORIMD step
from the user's reflection text
Note: This is a conceptual example; actual implementation is simplified
"""
system_prompt = """
You are an observer of reflection processes.
【Prohibitions】
- Expressions indicating ability evaluation
- Scoring or graded evaluation
- Expressions indicating superiority or strength
【Required expressions】
- Expressions indicating context-dependence
- Neutral state descriptions
- Expressions indicating possibilities
【Special rule: Regarding frequently appearing elements】
For elements that appear easily in text,
explain their nature as context
"""
user_prompt = f"""
Observe how the {step} step is being used
in the following reflection text.
【Reflection text】
{user_text}
"""
response = call_llm(system_prompt, user_prompt)
return response
This prompt structure reached this form through three revisions.
Initially, it was just "write neutrally," but that was completely insufficient.
Summary of Technical Decisions
What I learned from implementing the Advanced Assessment:
1. Making LLM an "Observer" Requires Concrete Vocabulary Constraints
Abstract instructions ("neutrally," "without evaluation") are insufficient.
I needed to explicitly write in the prompt which vocabulary to avoid and which to use.
2. UI Embodies Philosophy
Radar charts are technically simple but encourage "comparison."
Flow display allows visual expression of the philosophy of "process traversal."
3. Structural Biases Always Exist
In text analysis, some elements (R, D) appear more easily than others (O, I, M).
I needed to be aware of this structural bias and correct for it in the prompt.
4. Excessive Framing is Just Right
I included the phrase "This is not evaluation" at both the beginning and end.
It may seem excessive, but this is about right to prevent misreading.
Trade-offs: What I Chose and What I Gave Up
This design has clear trade-offs.
What I Chose
- User psychological safety
- Philosophical consistency
- Prevention of misreading
What I Gave Up
- Easy-to-understand score display
- Clear superiority/inferiority evaluation
- Gamification-like "sense of growth"
As long as we use the word "assessment," users easily feel they're being "evaluated".
Given that premise, considerable care was needed to avoid it becoming evaluation.
How to Use Basic vs. Advanced Assessment
The two assessments have different purposes:
| Basic Assessment | Advanced Assessment | |
|---|---|---|
| Format | Questionnaire (20 questions) | Free writing + LLM analysis |
| Target | Your tendencies | One specific event |
| Role | Map | Record |
| Focus | Distribution of abilities | Process traversal |
| Output | Radar chart | Flow display |
Basic Assessment is a map for understanding "what tendencies do I have?"
Advanced Assessment is a record for seeing "how did I traverse this time?"
Both share the philosophy of not evaluating.
The Intersection of Technology and Ethics
What I felt most strongly during this implementation process was that technical decisions become ethical decisions.
Radar chart or flow display.
Write "was not used" or "was less prominent."
These seem like trivial implementation differences.
But these choices can influence users' self-perception.
Therefore, I maintained the decision criterion of "when in doubt, prioritize user psychological safety" from start to finish.
Reaching a Releasable State
After these adjustments, the Advanced Assessment reached a releasable state.
It's not perfect. There's still a possibility of misreading, and there might be better expressions.
But I believe I succeeded in maintaining consistency in the philosophy of "not evaluating" across prompts, UI, and wording.
Conclusion
Building a reflection assessment, especially an LLM-based analysis type like the Advanced Assessment, means creating a tool that influences people's self-perception.
That responsibility must be carried in every single word of the prompt, every single element of the UI.
LLMs make it easy to create "assessment-like things."
But continuing to ask whether it might hurt people or instill incorrect self-perception—that is what's most important, I believe.
References
For those looking to deepen their reflective capacity, systematic learning through books is also recommended.
[📦 商品リンク: moshimo-book-reflection-kumakura]
Additionally, engaging in dialogue with a professional coach can be effective for deepening self-awareness.
[📦 商品リンク: moshimo-coachee-banner-728x90]
The actual tool: