Viola Kempa Say hello

All work Case 01 · Bachelor thesis · 2025

Deceptive Pattern Detector

The checkout wireframe selected on the canvas beside the shipped plugin panel, redrawn to scale from thesis fig. 14: step 1, two frames detected; step 2, the frames Home Page and Date; step 3, the analysis as text, a confidence score of 5, the patterns Possible Misdirection and Hidden Costs, the disclaimer that the AI can be wrong, and the evidence for each pattern.
A Figma plugin that flags deceptive patterns in a design and says how sure it thinks it is.

Result

Models beat the rules. Almost every one also overrated itself.

My part
Everything, alone: the study, 147 ratings, interface, plugin, backend and the evaluation.
Time
Bachelor thesis, CODE Berlin, submitted 7 August 2025. Local prototype, no user testing yet. [Figma Community: intended in the thesis, publication unconfirmed.]
Constraint
The plugin only sends words: frame names and text layers, no pixels.

[Try it] GitHub

A designer asks a model whether a screen is deceptive, and the model answers with confidence. I tested 12 language models on 12 deceptive patterns from the Hall of Shame repository, 147 evaluations in all, and rated every answer myself against a four-point rubric, with a written reason for each score.

Before: manual audits, slow and inconsistent, or rule-based checkers, which miss context and new variants. Neither sits inside the design tool.

It never gives a bare verdict.

My first prompt, on 26 July, asked for a list of patterns and the evidence for each. No score, no warning. It read well, and the models were beating the rule-based systems.

Then the study came in. I scored 147 of the models’ answers from 0 to 10, and each model scored its own answer on the same scale. Ten of the eleven models that rated themselves came out above my rating, and the gap grew as the answers got worse. A plain answer would hand that overconfidence straight to the designer.

Repo · first prompt 26 Jul 2025 · score and disclaimer added 6 Aug 2025, the day before submission · thesis §4.4 and §5.2

Fig. 1 · Average rating, 0 to 1, the four best models by human score · Source: thesis fig. 8, p. 24

5.9/10 Average human score for the models’ answers

39 Scored 8 or more

28 Scored 4 or less

23 Returned no reasoning at all

15.6% of all responses, almost all from Mistral Large 2411 and DeepSeek R1, which could not take screenshots. Every one scored 0.

  1. GPT-4.1.78 → .92
  2. GPT-4o.74 → .89
  3. o1.71 → .88
  4. Claude 3.7.66 → .86
  5. Nova Pro.59 → .85
  6. Llama 3.3 70B.26 → .06
Human vs self-rating · six of eleven models · thesis, 2025

  1. Rejected

    A bare verdict

    Fast to read, and exactly as sure of itself as the model, which the study showed is too sure.

  2. Rejected · my first prompt

    Patterns, no doubt

    In the repo from 26 July to 6 August. It named patterns with evidence and never said it could be wrong.

  3. Chosen

    Score, evidence, disclaimer

    The designer sees how sure the model says it is and why, and keeps the final call.

I chose

A score from 0 to 10, named patterns with evidence, and “the AI can be wrong” every time.

It cost

It is slower to read.

Words are easy. Pictures are not.

Input Selected frames
Plugin · Figma API Reads frame names and text layers
Model, behind Express GPT-4, temperature 0.3
Check The designer keeps the final call
No model · the pixels are never sent Patterns carried by the picture: a designer decides
What the designer gets Score, patterns, evidence, disclaimer

Where it does not decide

The models were good at patterns carried by text and weak at misleading colour hierarchies, spatial misalignment and graphical affordances, because they are built for language first. A misleading layout has no words to read, so a language model should not have the last word there.

147

Answers I scored from 0 to 10. Each model scored its own answer on the same scale.

What I measured
The gap between a model’s own rating and the human one.
Method
One rater, me, with a four-criteria rubric and a written reason for every score. The thesis names that subjectivity as a limitation.
The set
147 scored answers: 12 models on 12 patterns from the Hall of Shame repository, three missing because DeepSeek R1 could not take screenshots.

Where it still breaks.

The case What it did What I did about it
A visual pattern on the canvas Cannot reach the model at all: no pixels are sent. The next version. Until then a designer decides.
A model with a poor answer Rates itself high anyway, up to +0.26 over my score (Nova Pro). The panel says every time that the AI can be wrong.
Mistral Large 2411 and DeepSeek R1 Almost all of the 23 responses (15.6%) with no reasoning. The plugin runs on GPT-4 only and sends text, which every model can read.
The shipped plugin panel beside three wireframes, checkout, cart and pricing, redrawn to scale from thesis fig. 14. The checkout frame is selected (1). In step 3 the analysis is text: the confidence score of 5 (2), the disclaimer that the AI can be wrong (4), and the patterns Possible Misdirection and Hidden Costs with the evidence for each (3). Purple is the plugin's own colour.
  1. Pick the frames Select frames on the canvas. The plugin lists them and reads every text layer.
  2. A score, not a verdict 0 to 10, the model’s own certainty. The study is why you read it with doubt.
  3. Named, with evidence Every pattern comes with the reason it was flagged.
  4. It admits it can be wrong Every time. The study is the reason why.
[MP4 · Select → Review → Analysis]

What it does for the designer

Three steps on the canvas, and the API key never leaves the Express server.

The prompt’s output contract · server/index.js · 6 Aug 2025 
Your output must start with:
1. Confidence Score (0–10): <score>
2. Detected Patterns: <comma-separated
   list, or "No deceptive patterns
   detected">
3. Disclaimer: ... It is important to
   still make your own judgement
   because the AI can be wrong.

What to notice

The order is the decision: score, patterns and disclaimer come before any explanation.

GitHub

+0.14 to +0.20

How far the four best models overrated themselves against my ratings, on a 0 to 1 scale. The weaker models went up to +0.26. [Installs or use: a number, or “not measured”.]

The score on screen is the model rating its own certainty, and that is the number the study found inflated, by 0.14 to 0.26. So it can’t tell a designer how far to trust the answer. [What you would do about it, concrete enough to be wrong.]