CRO Echo Report - 2026-10-25

33 new studies · 6 disputed

In this report
Grade	Study	Skeptic	Red flags	Original
Tier D	Kameleoon high-performer survey: 90% run both web and feature experiments	—	correlation causation, no baseline	no primary source
Tier C	Comparison tool on product listing pages: apparel brand reports a $16M revenue increase	—	correlation causation, no baseline, confounded window, selection bias	no primary source
Tier C	Research-based UX changes to a restaurant app: over 40% digital membership growth since the partnership began	—	correlation causation, no baseline, confounded window, selection bias	no primary source
Tier B	Mass Merchant UX benchmark of 10 sites: only 1 rated good across 5,600+ UX scores	—	vendor metric	no primary source
Tier B	E-cigarette & Vaping UX benchmark of 7 sites: none rated decent or better across 3,600+ UX scores	—	vendor metric	no primary source
Tier B	B2B electronic components UX benchmark of 6 sites: only 1 rated decent across 2,600+ UX scores	—	vendor metric	no primary source
Tier B	Event ticketing usability testing: 463 sessions surfaced 2,800+ medium-to-severe issues	—	none	no primary source
Tier B	Synthetic vs human survey respondents: same conclusion, but 14 points more UX terms and far less variety	—	selection bias	no primary source
Tier B	Digital twins of business-software users overestimated SUS scores by about 10.5 points	—	none	no primary source
Tier D	ChatGPT and Gemini vs a human researcher on a failed-task video: 47% and 39% agreement	—	none	no primary source
Tier C	ChatGPT-5.6 coding open-ended survey comments: kappa 0.749 vs 0.704 human-to-human	Tier B	none	no primary source
Tier B	Digital twins of AI-chatbot users overestimated SUS scores by 4.4-6.7 points	—	none	no primary source
Tier A	AI vs human interview moderators (randomized, n=20): participants spoke 45% less to AI	—	none	no primary source
Tier B	Streamlined UX-of-AI questionnaire on 420 chatbot users: reliability alpha 0.81-0.87	—	none	no primary source
Tier A	Simplified homepage (checklist + shop button) for The Raw Honey Shop: +24% sales vs control	—	cherry pick, multiple metrics	no primary source
Tier A	Free included services listed in the checkout order table: +9% orders vs control	—	cherry pick, anecdote generalisation	no primary source
Tier A	Redesigned Breakfast Train page with live calendar and menus: +10% bookings vs original	—	cherry pick	no primary source
Tier A	Tutor sign-up page reframed as a job posting with a real student request: +15% applications	—	cherry pick	no primary source
Tier A	Gift add-ons moved to their own funnel step: +28% revenue per order, no conversion loss	—	cherry pick	no primary source
Tier C	Kameleoon PBX across 17 tests: test creation 89% faster; 94% success on simple vs 63% on complex tests	—	selection bias	no primary source
Tier C	Kameleoon PBX at Best Western France: twice as many tests, launch in 20 minutes to 4 hours	—	cherry pick, vendor metric	no primary source
Tier C	Kameleoon PBX at Cedars-Sinai: up to 83% faster test builds, 67% more tests per year projected	—	headline overclaim, vendor metric	no primary source
Tier C	CTA text change on Alternative Airlines: +34% purchases	—	cherry pick	no primary source
Tier A	Promo codes shown only to AI-detected hesitant visitors at Orange: +11.6% conversion vs no promo	—	none	no primary source
Tier C	Kameleoon PBX at VeryChic: 87% less time per test build; +31% promo banner clicks	—	cherry pick, confounded window	no primary source
Tier B	Third-party automotive websites (11 sites, 561 users): average NPS +6%, trust at the 34th percentile	—	correlation causation, vendor metric	no primary source
Tier C	CRO programme at Churnet Valley Railway: capacity sold rose from ~40% to 80-100%	—	cherry pick, confounded window	no primary source
Tier A	Trial button 'Try it free for 14 days' vs 'Start free trial' at Exclaimer: +39% trial sign-ups	—	cherry pick	no primary source
Tier D	AutoTune subscription pricing test: +27% revenue per visitor, +24% AOV (client testimonial)	—	cherry pick, multiple metrics	no primary source
Tier B	Average documented cart abandonment rate across 50 studies: 70.22%	—	stale data, headline overclaim, vendor metric	no primary source
Tier D	Kameleoon PBX in five moderated sessions: usable variants reported 5-10x faster	—	anecdote generalisation, no baseline	no primary source
Tier C	Kameleoon PBX at Graham & Brown: test creation time cut ~90%; sticky add-to-bag +58%	—	vendor metric, multiple metrics	no primary source
Tier C	Papier's experimentation programme on Kameleoon: 4-5 tests a month, winning tests average +5-6% conversion	—	cherry pick	no primary source
Tier C	Baymard checkout guidelines across a store portfolio: checkout conversion reported doubled (+100%)	C	headline overclaim, no baseline, confounded window, selection bias	no primary source
Tier D	Baymard-informed checkout changes at a SaaS/digital services company: customer reports +$1M in value	D	headline overclaim, no baseline, selection bias	no primary source
Tier A	Personalization vs shipping the best single variant: +18% in one field study, +4% in another	D	none	numbers not found
Tier D	Literature review of AI interview moderators: mixed evidence, only two head-to-head studies	B	none	no primary source
Tier C	AI predictive targeting of Toyota.fr's test-drive pop-in: leads reported doubled	C	headline overclaim, no baseline	no primary source
Tier D	Forrester survey of 107 healthcare CX leaders: insight-driven organisations '5x more likely' to grow revenue	B	correlation causation	no primary source

How to read this table

Tier A Tier B Tier C Tier D strongest to weakest evidenceTier ? ungradedSkeptic: the blind second grade, — when it agrees≠disputed#numbers not found in the original∅original not reachable

New studies

Tier Dvendor

Kameleoon high-performer survey: 90% run both web and feature experiments

Kameleoon reports, in its 2025 experiment-led growth report, that 90% of high-performing companies run both web and feature experiments and 74% of leaders regularly check cross-team metric effects. No sample size or method is stated, and the link between team structure and growth is correlational; treat the figures as indicative.

Red flags: correlation causation, no baseline

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: share running web and feature experiments

Research: 2025

Part of: Kameleoon experiment-led growth report (editions 2025)

Subtopic: Experimentation tooling & velocity

Question: What share of high-performing companies run both web and feature experiments?

Coverage: Kameleoon Blog

Tier Cvendor

Comparison tool on product listing pages: apparel brand reports a $16M revenue increase

Baymard Institute reports that a premium apparel brand added a product-list comparison tool (guideline #495) on web and app and reported a $16M revenue increase. There is no baseline, time window, control group or attribution method, and the story is published by the vendor credited. The timeframe is unstated.

Red flags: correlation causation, no baseline, confounded window, selection bias

Conflict of interest: Baymard sells the research and audits credited with the result.

Measured: revenue

Research: unstated

Subtopic: Product pages

Question: Does adding a comparison tool to product lists raise revenue?

Published: 2026-10-07

Coverage: Baymard Institute

Tier Cvendor

Research-based UX changes to a restaurant app: over 40% digital membership growth since the partnership began

Baymard Institute reports that a global restaurant brand saw over 40% growth in app membership since the start of its Baymard partnership, alongside changes such as cart quantity controls. The growth is measured before/after over an unstated period with no control, so other changes may explain it.

Red flags: correlation causation, no baseline, confounded window, selection bias

Conflict of interest: Baymard sells the research and audits credited with the result.

Measured: Sign-ups & leads · digital membership

Research: unstated (since start of partnership)

Subtopic: Other

Question: Do research-based UX changes to an app grow digital membership?

Published: 2026-10-06

Coverage: Baymard Institute

Tier Bindependent

Mass Merchant UX benchmark of 10 sites: only 1 rated good across 5,600+ UX scores

Baymard Institute manually assessed 10 mass merchant and department store sites on 500+ UX parameters (5,600+ weighted scores). Most sites ranged from mediocre to decent, with only one rated good; common issues were search handling, missing unit prices and late order-cost disclosure. Scores are Baymard's own expert ratings, not measured conversions; the assessment date is unstated.

Red flags: vendor metric

Conflict of interest: Baymard sells audits of the benchmarked industries.

Measured: Task success & usability · UX performance score

Research: unstated

Subtopic: Other

Question: How well do ecommerce sites in an industry score on Baymard UX benchmarks? · 1 study · 3 articles

Published: 2026-10-22

Coverage: Baymard Institute

Tier Bindependent

E-cigarette & Vaping UX benchmark of 7 sites: none rated decent or better across 3,600+ UX scores

Baymard Institute assessed 7 e-cigarette and vaping ecommerce sites on 500+ UX parameters (3,600+ weighted scores): 5 rated mediocre and 2 poor. Missing critical filters and unclear order costs and delivery times were common. Scores are Baymard's expert ratings, not conversion data; the assessment date is unstated.

Red flags: vendor metric

Conflict of interest: Baymard sells audits of the benchmarked industries.

Measured: Task success & usability · UX performance score

Research: unstated

Subtopic: Other

Question: How well do ecommerce sites in an industry score on Baymard UX benchmarks? · 1 study · 3 articles

Published: 2026-10-07

Coverage: Baymard Institute

Tier Bindependent

B2B electronic components UX benchmark of 6 sites: only 1 rated decent across 2,600+ UX scores

Baymard Institute assessed 6 B2B electronic components and machinery sites on 420+ UX parameters (2,600+ scores). Most performed poorly; about a third lacked B2B shipping details in the cart and more than half offered no way to share or save cart contents. Results are expert ratings, not measured conversions; the assessment date is unstated.

Red flags: vendor metric

Conflict of interest: Baymard sells audits of the benchmarked industries.

Measured: Task success & usability · UX performance score

Research: unstated

Subtopic: Other

Question: How well do ecommerce sites in an industry score on Baymard UX benchmarks? · 1 study · 3 articles

Published: 2026-10-01

Coverage: Baymard Institute

Tier Bindependent

Event ticketing usability testing: 463 sessions surfaced 2,800+ medium-to-severe issues

Baymard Institute ran 463 think-aloud usability sessions (2,400+ hours of research) on US and UK ticketing marketplaces and venue sites such as Ticketmaster and StubHub, logging 2,800+ medium-to-severe issues. Hidden prices and late fees, late sold-out signals and costly seat-view evaluation drove abandonment in test sessions. The study is qualitative and reports no conversion rates; the testing period is unstated.

Conflict of interest: Baymard sells audits of the benchmarked industries.

Measured: Task success & usability · usability issues

Research: unstated

Subtopic: Other

Question: How many usability issues do event ticketing sites cause in usability testing?

About: Ticketmaster, StubHub, SeatGeek

Published: 2026-09-24

Coverage: Baymard Institute

Tier Bindependent

Synthetic vs human survey respondents: same conclusion, but 14 points more UX terms and far less variety

Nielsen Norman Group put one open-ended question to 120 synthetic respondents (Synthetic Users platform, February 2026) and compared them with 604 human professionals from an earlier survey. Both reached the same conclusion, but synthetic respondents used UX terminology 14 percentage points more often and collapsed onto a dozen labels versus 50+ for humans. The human sample was recruited via NN/g's own channels and the platform reworded the question, both possible confounds.

Red flags: selection bias

Measured: share of UX-related terminology; distinct labels

Baseline: human survey responses

Research: February 2026 (synthetic); earlier human survey

Subtopic: UX research methods

Question: Do synthetic respondents answer open-ended questions like human respondents?

About: Synthetic Users

Published: 2026-10-16

Coverage: Nielsen Norman Group

Tier Bindependent

Digital twins of business-software users overestimated SUS scores by about 10.5 points

MeasuringU built Claude Sonnet 5 digital twins of 289 participants from its January-February 2025 business software benchmark (6 products) and had them predict held-out SUS scores. Twins scored 10.6 (summary) and 10.4 (full) points higher than humans on average, enough to shift grades by up to two letters, though product rank order was largely preserved (r = .94-.95).

Measured: SUS score error

Baseline: human SUS scores

Research: January-February 2025 (human data)

Subtopic: UX research methods

Question: Do digital twins give the same SUS scores as the humans they model? · 1 study · 2 articles

About: Claude Sonnet 5

Published: 2026-10-20

Coverage: MeasuringU

Tier Dindependent

ChatGPT and Gemini vs a human researcher on a failed-task video: 47% and 39% agreement

MeasuringU reports that, in one usability video (from a 2019 benchmark) analysed four times by each AI, ChatGPT reached 47% and Gemini 39% any-2 agreement with an experienced human researcher. The AIs found 8 of the researcher's 11 problems but added three false alarms and three hallucinations. This is a single-participant case study, so the figures are indicative only.

Measured: any-2 agreement

Baseline: human researcher problem list

Research: unstated (video from a 2019 benchmark)

Subtopic: UX research methods

Question: Can AI find the usability problems a human researcher finds in test videos?

About: ChatGPT, Gemini

Published: 2026-10-07

Coverage: MeasuringU

Tier Cindependent · skeptic: B

ChatGPT-5.6 coding open-ended survey comments: kappa 0.749 vs 0.704 human-to-human

MeasuringU re-ran its 2023 dataset of SUPR-Q open-ended comments through ChatGPT-5.6 (three runs on August 11, 2026) and compared the results with three human coders. Human-to-ChatGPT-5.6 agreement (kappa 0.749) was statistically indistinguishable from human-to-human agreement (0.704), and the model's self-agreement was much higher. The new runs were aligned to the original themes by hand, which differs from the 2023 method.

Measured: interrater agreement (kappa)

Baseline: human-to-human kappa 0.704

Research: August 11, 2026 (AI runs); 2023 baseline

Subtopic: UX research methods

Question: Can an LLM code open-ended survey responses as reliably as human coders?

About: ChatGPT

Published: 2026-09-30

Coverage: MeasuringU

Tier Bindependent

Digital twins of AI-chatbot users overestimated SUS scores by 4.4-6.7 points

MeasuringU built Claude Sonnet 5 digital twins from 420 participants in its 2026 AI-chatbot benchmark (ChatGPT, Claude, Gemini, Grok) and compared their predicted SUS scores with the humans' held-out scores. Summary twins were 4.4 points and full twins 6.7 points higher on average, significant for all four products and enough to inflate grades. The authors note that all four products scored high, so a ceiling may have limited the error.

Measured: SUS score error

Baseline: human SUS scores

Research: May 2026 (human data)

Subtopic: UX research methods

Question: Do digital twins give the same SUS scores as the humans they model? · 1 study · 2 articles

About: Claude Sonnet 5

Published: 2026-09-23

Coverage: MeasuringU

Tier Aindependent

AI vs human interview moderators (randomized, n=20): participants spoke 45% less to AI

MeasuringU randomly assigned 20 participants to AI- or human-moderated in-depth interviews with the same script. AI sessions were 37% shorter, participants spoke 45% less per minute to AI moderators, and AI moderators took 64% of the words. The sample is small and no inferential statistics are reported; the study date is unstated.

Measured: participant words per minute

Baseline: human-moderated sessions

Research: unstated

Subtopic: UX research methods

Question: Do participants say less to AI moderators than to human moderators? · 1 study · 2 articles

Published: 2026-09-15

Coverage: MeasuringU

Tier Bindependent

Streamlined UX-of-AI questionnaire on 420 chatbot users: reliability alpha 0.81-0.87

MeasuringU tested 34 candidate items on 420 US users of ChatGPT, Claude, Gemini and Grok (May 2026). Factor analysis confirmed six constructs (such as AI Productivity and AI Trust), and the streamlined two-to-three-item scales reached coefficient alpha of 0.81-0.87. Claude led on AI Productivity and ChatGPT lagged on AI Trust.

Measured: scale reliability (coefficient alpha)

Research: May 2026

Subtopic: UX research methods

Question: Does a streamlined questionnaire measure the UX of AI chatbots reliably?

About: ChatGPT, Claude, Gemini, Grok

Published: 2026-08-18

Coverage: MeasuringU

Tier Aagency

Simplified homepage (checklist + shop button) for The Raw Honey Shop: +24% sales vs control

Conversion Rate Experts tested a homepage that replaced long brand storytelling with a claims checklist and a shop button against the original, after scrollmaps showed 50% of desktop users never reached the products. The variation lifted sales 24%, with a larger gain in revenue per visitor. No sample size, duration or significance is reported, and the result comes from a wins-only series.

Red flags: cherry pick, multiple metrics

Conflict of interest: The agency that ran the test publishes it to win clients.

Measured: Conversion rate · sales

Baseline: original homepage (control)

Research: unstated

Subtopic: Landing pages

Question: Does simplifying homepage copy and layout raise sales?

About: The Raw Honey Shop

Published: 2026-10-19

Coverage: Conversion Rate Experts

Tier Aagency

Free included services listed in the checkout order table: +9% orders vs control

Conversion Rate Experts added two rows to a gift retailer's order table naming services already included (premium service, satisfaction guarantee) priced FREE, and tested it against the original checkout. Orders rose 9%. No sample size, duration or significance is given, and the timeframe is unstated.

Red flags: cherry pick, anecdote generalisation

Conflict of interest: The agency that ran the test publishes it to win clients.

Measured: Conversion rate · orders

Baseline: original checkout page (control)

Research: unstated

Subtopic: Checkout & cart

Question: Does listing free included services in the order summary raise orders?

Published: 2026-10-06

Coverage: Conversion Rate Experts

Tier Aagency

Redesigned Breakfast Train page with live calendar and menus: +10% bookings vs original

Conversion Rate Experts tested a rebuilt Churnet Valley Railway dining-train page (price and next date upfront, availability calendar, detailed menus, repeated booking buttons) against the original. Bookings rose 10%. Many changes were bundled, so no single element can be credited, and no sample size or significance is reported.

Red flags: cherry pick

Conflict of interest: The agency that ran the test publishes it to win clients.

Measured: Conversion rate · bookings

Baseline: original page (control)

Research: unstated

Subtopic: Product pages

Question: Does a product page redesign with upfront booking info raise bookings?

About: Churnet Valley Railway

Published: 2026-09-01

Coverage: Conversion Rate Experts

Tier Aagency

Tutor sign-up page reframed as a job posting with a real student request: +15% applications

Conversion Rate Experts tested a tutoring marketplace's recruitment page rewritten as a job ad (subject and city headline, a real student request, pay range) against the plain registration form. Tutor applications rose 15%, about 20,000 more in the first year by the agency's extrapolation. No sample size or significance is stated.

Red flags: cherry pick

Conflict of interest: The agency that ran the test publishes it to win clients.

Measured: Sign-ups & leads · tutor applications

Baseline: original registration page (control)

Research: unstated

Subtopic: Forms & sign-up

Question: Does framing a supply-side sign-up page as a job posting raise applications?

Published: 2026-08-28

Coverage: Conversion Rate Experts

Tier Aagency

Gift add-ons moved to their own funnel step: +28% revenue per order, no conversion loss

Conversion Rate Experts moved a gift retailer's add-ons from the bottom of the product page to an interstitial step and tested it against the original. Revenue per order rose 28%, with no impact on overall conversion. No sample size, duration or significance is reported.

Red flags: cherry pick

Conflict of interest: The agency that ran the test publishes it to win clients.

Measured: Average order value · revenue per order

Baseline: add-ons at bottom of product page (control)

Research: unstated

Subtopic: Checkout & cart

Question: Does moving add-ons into their own funnel step raise order value?

Published: 2026-08-18

Coverage: Conversion Rate Experts

Tier Cindependent

Kameleoon PBX across 17 tests: test creation 89% faster; 94% success on simple vs 63% on complex tests

Henkan & Partners, in a study published by Kameleoon, evaluated Kameleoon's Prompt-Based Experimentation on 17 tests. Build time fell from about 9 hours to under one, but success was 94% on simple tests and only 63% on complex ones, with non-deterministic output. There is no control and the test set was chosen by the evaluators; the timeframe is unstated.

Red flags: selection bias

Conflict of interest: Published and promoted by Kameleoon, the vendor of PBX.

Measured: test creation time

Baseline: 9 hours per test

Research: unstated

Subtopic: Experimentation tooling & velocity

Question: Does prompt-based experimentation cut the time to build and launch a test? · ✓ consistent (3 independent studies) · corroborated by 2 other independent studies

About: Kameleoon PBX

Coverage: Kameleoon Blog

Tier Cvendor

Kameleoon PBX at Best Western France: twice as many tests, launch in 20 minutes to 4 hours

Kameleoon reports that Best Western Hotels & Resorts France doubled the number of experiments run with the same team after adopting its PBX tool, launching tests in 20 minutes to 4 hours instead of several days and saving up to EUR 2,000 per test. The figures are before/after from a vendor success story with no control; the timeframe is unstated.

Red flags: cherry pick, vendor metric

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: experiment velocity

Baseline: several days per test launch

Research: 2026 (unstated)

Subtopic: Experimentation tooling & velocity

Question: Does prompt-based experimentation cut the time to build and launch a test? · ✓ consistent (3 independent studies) · corroborated by 2 other independent studies

About: Kameleoon PBX

Coverage: Kameleoon Blog

Tier Cvendor

Kameleoon PBX at Cedars-Sinai: up to 83% faster test builds, 67% more tests per year projected

Kameleoon reports that Cedars-Sinai's single optimization engineer cut test build time by up to 83% with a hybrid PBX workflow and projects 60 tests a year instead of 36. The body gives 67% (simple) and 20% (complex) time reductions, and the volume gain is a projection, not a measured result. Data are from 2025-2026.

Red flags: headline overclaim, vendor metric

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: test build time

Baseline: 36 tests per year

Research: 2025-2026

Subtopic: Experimentation tooling & velocity

Question: Does prompt-based experimentation cut the time to build and launch a test? · ✓ consistent (3 independent studies) · corroborated by 2 other independent studies

About: Kameleoon PBX

Coverage: Kameleoon Blog

Tier Cvendor

CTA text change on Alternative Airlines: +34% purchases

Kameleoon reports that Alternative Airlines increased purchases by 34% by changing CTA text on a product, one of several highlighted wins from 100+ tests a year. No control details, sample size or significance are given, and the timeframe is unstated.

Red flags: cherry pick

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: Conversion rate · purchases

Research: unstated

Subtopic: CTAs & buttons

Question: Does rewriting CTA button text raise conversions? · ✓ consistent (2 independent studies) · corroborated by 1 other independent study

About: Alternative Airlines

Coverage: Kameleoon Blog

Tier Avendor

Promo codes shown only to AI-detected hesitant visitors at Orange: +11.6% conversion vs no promo

Kameleoon reports a three-arm test at Orange: no promo code (control), a promo code for everyone, and a promo code only for visitors its AI flagged as hesitant. The AI-targeted arm raised conversion 11.6% vs control and 7.6% vs promo-for-all, while the share of purchases using a code fell from 68% to 25%. No sample size or significance is reported; the timeframe is unstated.

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: Conversion rate · conversion rate

Baseline: no promo code (control)

Research: unstated

Subtopic: Personalization

Question: Does showing promo codes only to AI-detected hesitant visitors raise conversion?

About: Kameleoon AI Predictive Targeting, Orange

Coverage: Kameleoon Blog

Tier Cvendor

Kameleoon PBX at VeryChic: 87% less time per test build; +31% promo banner clicks

Kameleoon reports that VeryChic, a premium travel flash-sale platform, cut test build time by 87% with PBX from summer 2025 and saw 31% more clicks on promotional banners and 7% more quote-page visits. The banner result compares a summer run with an earlier test, so seasonality is uncontrolled, and the page also quotes 7% time saved.

Red flags: cherry pick, confounded window

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: test build time

Baseline: half a day per test

Research: summer 2025

Subtopic: Experimentation tooling & velocity

Question: Does prompt-based experimentation cut the time to build and launch a test? · ✓ consistent (3 independent studies) · corroborated by 2 other independent studies

About: Kameleoon PBX

Coverage: Kameleoon Blog

Tier Bindependent

Third-party automotive websites (11 sites, 561 users): average NPS +6%, trust at the 34th percentile

MeasuringU surveyed 561 US users of 11 third-party automotive sites in March-April 2026 with the SUPR-Q, UX-Lite and NPS. The industry averaged the 51st SUPR-Q percentile and an NPS of +6%, but trust sat at the 34th percentile; Kelley Blue Book led on NPS (+34%) and Carvana trailed (-23%). Results are self-reported ratings, not behavioural data.

Red flags: correlation causation, vendor metric

Measured: SUPR-Q percentile and NPS

Baseline: SUPR-Q database of ~200 websites

Research: March-April 2026

Subtopic: Other

Question: How do third-party automotive websites score on UX quality and NPS?

About: Kelley Blue Book, Carvana, CARFAX, cars.com, carweek.com

Published: 2026-10-13

Coverage: MeasuringU

Tier Cagency

CRO programme at Churnet Valley Railway: capacity sold rose from ~40% to 80-100%

Conversion Rate Experts publishes a client testimonial in which Churnet Valley Railway's managers say trains that used to run at 40% capacity on some days now regularly run at 80-100%, and income passed GBP 1 million for the first time in 2026. These are loose before/after figures over a multi-year engagement that also included SEO and other marketing, with no control.

Red flags: cherry pick, confounded window

Conflict of interest: The agency that ran the test publishes it to win clients.

Measured: train capacity filled

Baseline: 40% capacity some days

Research: 2026 season

Subtopic: Other

Question: Does a CRO programme raise how full a visitor attraction sells?

About: Churnet Valley Railway

Published: 2026-10-06

Coverage: Conversion Rate Experts

Tier Aagency

Trial button 'Try it free for 14 days' vs 'Start free trial' at Exclaimer: +39% trial sign-ups

Conversion Rate Experts tested a sitewide header button rewritten to state the trial length against the original 'Start free trial' copy. Trial conversions rose 39%, confirmed by the client. No sample size, duration or significance is given, and it comes from a wins-only series.

Red flags: cherry pick

Conflict of interest: The agency that ran the test publishes it to win clients.

Measured: Sign-ups & leads · trial sign-ups

Baseline: "Start free trial" button (control)

Research: unstated

Subtopic: CTAs & buttons

Question: Does rewriting CTA button text raise conversions? · ✓ consistent (2 independent studies) · corroborated by 1 other independent study

About: Exclaimer

Published: 2026-09-28

Coverage: Conversion Rate Experts

Tier Dagency · judge: C

AutoTune subscription pricing test: +27% revenue per visitor, +24% AOV (client testimonial)

AutoTune's CMO reports, in a Conversion Rate Experts testimonial, that one pricing test on a subscription product raised revenue per visitor 27%, average order value 24% and annual-plan conversion 32%, with 17-18% higher LTV. The test design and control are not described, and it is one favourable result from hundreds of tests; the timing is unstated.

Red flags: cherry pick, multiple metrics

Conflict of interest: The agency that ran the test publishes it to win clients.

Measured: Revenue per visitor · revenue per visitor

Research: unstated (last two and a half years)

Subtopic: Pricing & plans

Question: Does a new subscription pricing variant raise revenue per visitor?

About: AutoTune

Published: 2026-08-13

Coverage: Conversion Rate Experts

Tier Bindependent

Average documented cart abandonment rate across 50 studies: 70.22%

Baymard Institute averages 50 published studies of online cart abandonment to reach 70.22%. The sources, mostly cart-recovery vendors, use differing methods and go back many years, so the figure is a rough benchmark rather than a current measurement.

Red flags: stale data, headline overclaim, vendor metric

Measured: Cart abandonment · cart abandonment rate

Research: studies from 2006-2025

Subtopic: Checkout & cart

Question: What is the average online shopping cart abandonment rate?

Coverage: Baymard Institute

Tier Dindependent

Kameleoon PBX in five moderated sessions: usable variants reported 5-10x faster

Guess the Test reports, in a study published by Kameleoon, that across five moderated usability sessions with practitioners PBX produced usable variants 5-10 times faster and got users '90% of the way there'. The baseline workflow and timing method are not stated, so the claim rests on a very small sample.

Red flags: anecdote generalisation, no baseline

Conflict of interest: Published and promoted by Kameleoon, the vendor of PBX.

Measured: variant production speed

Research: unstated

Subtopic: Experimentation tooling & velocity

Question: Does prompt-based experimentation cut the time to build and launch a test? · ✓ consistent (3 independent studies) · corroborated by 2 other independent studies

About: Kameleoon PBX

Coverage: Kameleoon Blog

Tier Cvendor

Kameleoon PBX at Graham & Brown: test creation time cut ~90%; sticky add-to-bag +58%

Kameleoon reports that Graham & Brown cut test creation time by about 90% with PBX, an estimate from the customer, and that a PBX-built sticky add-to-bag test lifted add-to-bag 58%. No control, sample size or significance is given for the test; the timeframe is unstated.

Red flags: vendor metric, multiple metrics

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: test creation time

Baseline: days to weeks per test

Research: unstated

Subtopic: Experimentation tooling & velocity

Question: Does prompt-based experimentation cut the time to build and launch a test? · ✓ consistent (3 independent studies) · corroborated by 2 other independent studies

About: Kameleoon PBX

Coverage: Kameleoon Blog

Tier Cvendor

Papier's experimentation programme on Kameleoon: 4-5 tests a month, winning tests average +5-6% conversion

Kameleoon reports that Papier, a UK stationery brand with about 1.4M monthly visits, went from 1-2 experiments a quarter to 4-5 a month, with winning tests averaging a 5-6% conversion lift (up to 8%). The average covers winning tests only and there are no control or sample details; the timeframe is unstated.

Red flags: cherry pick

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: Conversion rate · conversion lift of winning tests

Baseline: 1–2 experiments per quarter

Research: unstated

Subtopic: Experimentation tooling & velocity

Question: Does adopting an experimentation platform raise test velocity?

About: Kameleoon

Coverage: Kameleoon Blog

Disputed (verdict and skeptic disagree; both grades shown)

Tier Cvendor

disputed

Baymard checkout guidelines across a store portfolio: checkout conversion reported doubled (+100%)

Baymard Institute relays a customer's report that checkout conversion doubled after the retailer applied Baymard's checkout usability guide across all its stores. No baseline rate, store count, timeframe or control group is given, and the claim comes from the vendor whose guidelines are credited. The research timeframe is unstated.

Disputed: no quoted number found in the text.
Skeptic's doubts: A vendor-published customer story says checkout conversion doubled after the guidelines were applied, but it gives no baseline rate, no time window and no control, and it does not account for anything else that changed at the same time.

Red flags: headline overclaim, no baseline, confounded window, selection bias

Conflict of interest: Baymard sells the research and audits credited with the result.

Measured: Conversion rate · checkout conversion rate

Research: unstated

Subtopic: Checkout & cart

Question: Does applying checkout usability guidelines raise checkout conversion?

Published: 2026-10-16

Coverage: Baymard Institute

Tier Dvendor

disputed

Baymard-informed checkout changes at a SaaS/digital services company: customer reports +$1M in value

Baymard Institute reports that a digital services customer credited checkout and buying-flow changes informed by Baymard research with more than $1 million in commercial value. The figure is the customer's own assessment, with no stated method, baseline or test; treat it as an anecdotal claim. The timeframe is unstated.

Disputed: no quoted number found in the text.
Skeptic's doubts: A vendor testimonial credits the changes with '+$1M commercial value' based only on the customer's own assessment, with no stated method, baseline or test results.

Red flags: headline overclaim, no baseline, selection bias

Conflict of interest: Baymard sells the research and audits credited with the result.

Measured: commercial value

Research: unstated

Subtopic: Checkout & cart

Question: Do research-informed checkout changes create commercial value?

Published: 2026-10-08

Coverage: Baymard Institute

Tier Askeptic: D · independent

disputed

Personalization vs shipping the best single variant: +18% in one field study, +4% in another

Covered on the VWO blog, two Wharton researchers tested five personalization methods in two large field studies (~20 variations each) and found personalization beat rolling out the single best version by 18% in one and only 4% in the other. The article adds a cited Yahoo study (observational lift of 870-1,200% vs 5.4% in a randomized holdout) and one practitioner test that won by 6% but lost money after costs. A primary-source check found the figures match the original abstract; the research timeframe is unstated.

Disputed: grades two tiers apart.
Skeptic's doubts: A methods essay on personalization break-even, built on cited studies (the Yahoo ad holdout and the Wharton field studies), one practitioner test that won by 6% but was unprofitable, and a hypothetical worked example.

Conflict of interest: Published on VWO's blog, which sells personalization tools.

Original: numbers not found — 5.4%

Measured: incremental lift from personalization

Baseline: single best version for everyone

Research: unstated

Subtopic: Personalization

Question: Does personalization beat shipping the single best variant to everyone?

Published: 2026-09-21

Primary source: https://arxiv.org/abs/2411.16552

Coverage: VWO Blog

Tier Dskeptic: B · independent

disputed

Literature review of AI interview moderators: mixed evidence, only two head-to-head studies

MeasuringU reviewed ten sources on AI-led interviewing published between 2024 and mid-2026. Evidence on whether participants say less or give shallower answers to AI was mixed, and only two studies compared AI and human moderators directly (5 vs 6 and 30 vs 30 interviews). The authors conclude AI matches humans on procedure but trails on judgment-dependent moderation.

Disputed: grades two tiers apart.
Skeptic's doubts: Structured literature review of about ten AI-interview studies with stated sample sizes (including a 60-participant RCT and samples of 381-399), tagging findings pro/neutral/anti; it contains no original data and its own conclusions are suitably hedged.

Measured: moderation quality

Research: 2024 to mid-2026 publications

Subtopic: UX research methods

Question: Do participants say less to AI moderators than to human moderators? · 1 study · 2 articles

Published: 2026-09-09

Coverage: MeasuringU

Tier Cvendor

disputed

AI predictive targeting of Toyota.fr's test-drive pop-in: leads reported doubled

Kameleoon reports that Toyota.fr doubled the number of test-drive leads after switching its pop-in from manual scoring to Kameleoon AI Predictive Targeting. No control group, absolute numbers or lead-quality measures (show-ups, sales) are given, though lead quality was the stated problem. The timeframe is unstated.

Disputed: no quoted number found in the text.
Skeptic's doubts: Says AI targeting 'doubled' leads against earlier manual scoring, with no control group or absolute numbers, and never measures lead quality (show-ups or sales), which was the stated problem.

Red flags: headline overclaim, no baseline

Conflict of interest: Kameleoon sells the platform credited with the result.

Measured: Sign-ups & leads · test drive leads

Baseline: manual-scoring pop-in targeting

Research: unstated

Subtopic: Personalization

Question: Does AI predictive targeting of a lead pop-in raise leads?

About: Kameleoon AI Predictive Targeting, Toyota

Coverage: Kameleoon Blog

Tier Dskeptic: B · vendor

disputed

Forrester survey of 107 healthcare CX leaders: insight-driven organisations '5x more likely' to grow revenue

Forrester, commissioned by Kameleoon, surveyed 107 marketing and CX managers at North American healthcare organisations and reports that those applying customer insights are 5x more likely to grow revenue. The link is correlational, the survey date is unstated (published 2021), and the vendor sponsored the study.

Disputed: grades two tiers apart.
Skeptic's doubts: Kameleoon-commissioned Forrester survey of 107 healthcare managers; the line that insight-driven firms are '5x more likely to grow revenue' is correlation presented as a causal benefit.

Red flags: correlation causation

Conflict of interest: Commissioned by Kameleoon, which sells optimization software.

Measured: revenue growth likelihood

Research: unstated (published 2021)

Subtopic: Other

Question: Are organisations that apply customer insights more likely to grow revenue?

Coverage: Kameleoon Blog