Why AI Cannot Misgrade a Pipetting Technique

A student writes that a chemical reaction requires "at least one catalyst." The AI grading system interprets this as "only one catalyst" and deducts points. The student appeals. The teacher reviews the response. The answer was correct. The AI misread it.

This is not a hypothetical scenario. Reports of AI grading errors in text-based assessments have emerged across educational institutions, from standardised testing to university coursework. The pattern is consistent: AI systems trained on language patterns struggle with the inherent ambiguity of written expression. They penalise correct answers that use unexpected phrasing. They reward confident-sounding nonsense. They fail precisely where human judgment succeeds.

Meanwhile, a different approach to assessment is proving far more reliable. At Arizona State University, the Dreamscape Learn program grades students not on what they write, but on the reasoning steps they take within immersive VR experiences. Early studies show students who participate in these process-based assessments achieve dramatically higher lab grades than peers in conventional courses. The reason is straightforward: when you assess what students do rather than what they say, ambiguity disappears.

Why Does AI Struggle with Written Assessment?

Language is inherently ambiguous. The phrase "at least one" can mean "one or more" (correct) or be misread as implying "exactly one" by a system looking for precise terminology. The phrase "the reaction requires heat" might be marked wrong because the rubric specified "thermal energy" or "elevated temperature." A student who writes "the liquid changed colour" might lose points because they spelled it the British way rather than the American "color."

Research on automated essay scoring systems has documented these failures extensively. A 2023 study in Educational Technology Research and Development found that AI grading systems showed significant variance in scoring identical content when presented with different phrasings. The systems penalised unconventional sentence structures, even when the scientific content was accurate. They rewarded verbose, confident-sounding prose over concise, accurate responses.

The fundamental problem is that AI text interpretation operates on pattern matching. The system learns what "correct" answers typically look like and scores submissions based on similarity to those patterns. When a student expresses correct understanding in an unexpected way, the system fails to recognise it.

What Does Process-Based Assessment Look Like?

Process-based assessment inverts the traditional model. Instead of asking students to describe what they would do and parsing their language, you watch what they actually do and measure it directly.

In a virtual chemistry lab, this means tracking whether a student:

• Pipetted the correct volume of reagent (not whether they wrote "2.5 mL" or "2.5 millilitres")
• Swirled the flask to mix the solution (not whether they mentioned "agitation")
• Read the meniscus at eye level (not whether they described proper measurement technique)
• Added reagent dropwise near the endpoint (not whether they used the term "incremental addition")

There is no ambiguity in these measurements. Either the student added 2.5 mL or they did not. Either they positioned the pipette correctly or they did not. Either they observed proper safety protocols or they did not. The assessment system does not need to interpret language because it is measuring physical actions directly.

Arizona State University's work with Dreamscape Learn demonstrates this principle at scale. In their VR biology courses, students solve problems in immersive environments where their reasoning process is captured through their actions, not their written explanations. The results have been striking: students in VR-based sections achieved higher grades and showed better retention of concepts than those in traditional sections.

The Deeper Problem with Text-Based AI Assessment

The issues with AI grading of written work go beyond occasional misinterpretation. There is a more fundamental problem: text-based assessment incentivises writing skill over scientific competence.

Consider two students in a chemistry course. Student A understands titration deeply but writes awkwardly, using imprecise language and run-on sentences. Student B has a superficial understanding but writes beautifully, using technical terminology fluently and constructing grammatically perfect paragraphs. In a text-based AI assessment, Student B will likely score higher. In a process-based assessment where both students perform an actual titration, Student A's superior understanding becomes immediately apparent.

This is not merely a fairness issue. It is a validity issue. The purpose of science assessment is to measure scientific competence, not writing ability. When we conflate the two, we systematically disadvantage students who are strong scientists but weak writers, while rewarding students who are strong writers but weak scientists.

Research on science assessment has consistently shown that performance-based measures correlate more strongly with later scientific success than written measures (Hamilton et al., 2003). A student who can successfully execute a multi-step experimental procedure is demonstrating the skills that matter in actual scientific work. A student who can eloquently describe the procedure is demonstrating writing skills.

How WhimsyLabs Implements Process-Based Assessment

We designed WhimsyLabs around process-based assessment from the beginning because we understood that the real value of virtual labs lies not in simulating physical appearances, but in capturing the procedural knowledge that defines scientific practice.

When a student performs an experiment in WhimsyLabs, our system captures:

• Action sequences: Did the student follow the correct order of steps? Did they skip critical safety procedures?
• Technique quality: How steadily did they control the pipette? Did they approach endpoints appropriately?
• Error recovery: When something went wrong, did they recognise it? What did they do about it?
• Scientific reasoning: Based on intermediate observations, did they adjust their approach appropriately?

None of this requires language interpretation. A student from Germany, Japan, or Spain performs the same pipetting motion. A student who speaks English as a second language demonstrates the same understanding by swirling a flask at the right moment. The universal language of scientific procedure transcends the ambiguities of written expression.

Our AI tutor, WhimsyCat, uses this process data to provide personalised feedback. When a student makes an error, WhimsyCat identifies exactly what went wrong, not based on parsing an ambiguous written explanation, but based on observing the specific action that deviated from correct technique. The feedback is precise because the assessment is precise.

The Implications for AI in Assessment

None of this means AI has no role in educational assessment. It means that AI works best when applied to unambiguous data. Natural language processing is hard because natural language is inherently ambiguous. Measuring physical actions is comparatively easy because physical actions are definite.

This has implications for how we should deploy AI in education:

• Use AI for what it does well: Pattern recognition in structured data, procedural tracking, identifying specific technique errors
• Avoid AI for what it does poorly: Interpreting ambiguous language, evaluating creative expression, scoring open-ended written responses
• Design assessments around AI strengths: Rather than forcing AI to handle text ambiguity, create assessments that produce unambiguous data

The push to use AI for grading written work often comes from a desire to reduce teacher workload. This is a legitimate concern. Teachers are overwhelmed, and assessment takes enormous time. But the solution is not to apply AI to tasks it performs poorly. The solution is to redesign assessments so that AI can assist effectively.

Virtual labs offer exactly this redesign. Instead of reading thirty written lab reports, a teacher reviews a dashboard showing which students struggled with specific techniques. Instead of trying to interpret whether a student understands titration from a paragraph of prose, the teacher sees data showing exactly where each student's pipetting technique deviated from the correct procedure.

Beyond Grading: What Process Data Reveals

Process-based assessment does more than avoid the pitfalls of text interpretation. It reveals information that text-based assessment fundamentally cannot capture.

Consider a student who arrives at the correct answer through an incorrect process. In a text-based assessment asking for the result of a calculation, this student scores full marks. In a process-based assessment tracking how they reached that result, the conceptual gap becomes visible. The teacher can intervene before the misunderstanding causes problems in more advanced work.

Consider a student who understands the concept perfectly but makes a procedural error under time pressure. Text-based assessment might penalise this as incorrect understanding. Process-based assessment distinguishes between conceptual errors and execution errors, allowing targeted remediation.

Consider a student who consistently hesitates before a specific type of action, indicating uncertainty. Process data captures this hesitation. Written responses cannot.

The richness of process data enables a kind of educational insight that traditional assessment methods simply cannot provide. When we know not just what students concluded but how they got there, we can teach more effectively.

The Future of Science Assessment

The current moment in educational technology presents a choice. We can continue trying to make AI interpret written language, accepting occasional grading errors as the cost of automation. Or we can redesign assessment around tasks that produce unambiguous data, eliminating the errors entirely.

Virtual laboratories represent one path toward this redesign. By translating scientific procedures from written descriptions into actual performed actions, they create assessment contexts where AI excels rather than struggles. The student who correctly pipettes 2.5 mL demonstrates the same competence regardless of whether they would describe it as "transferring two point five millilitres," "pipetting 2.5 mL," or "adding the required volume." The ambiguity of language is bypassed entirely.

This is why we built WhimsyLabs around process-based assessment. Not because we wanted to avoid AI, but because we wanted to deploy AI where it works best. The result is assessment that is simultaneously more accurate, more fair, and more educationally valuable than traditional approaches.

AI cannot misgrade a pipetting technique because there is nothing to misinterpret. The action either happened correctly or it did not. In that clarity lies the future of science assessment.

Related Articles

• AI and the Assessment Crisis: How Process Data Changes Everything
• New VR Research Confirms: Technology Without Pedagogy Falls Flat
• The Critics Are Right: Most EdTech Is Useless