# Avouch Study 2: method and data

[Read the article](/insights/how-ai-shortlists-differ-between-buying-briefs)

## Method, dates and data

The comparison data and counting decisions are available below. Results were analysed on 19 October 2026. Percentages describe companies in these saved answers, not shares of buyers, market demand or repeated tests.

### Sample, selection and collection

We first checked the available records across client and market research. The main technology inventory contained 359 non-empty answers to 120 questions across IT support, charity app development and UK penetration testing. This article analyses a selected subset, not the full inventory.

The primary comparisons use 21 distinct answers across seven question texts. The three general IT answers are each reused for the size and combined-service comparisons. The supplementary software pair adds four answers to two question texts. Total: 25 distinct answers, nine question texts and 14 engine comparisons.

Selection followed an exploratory feasibility review in which some answers had already been inspected. The selection was not random or fixed before seeing any results. We cannot use this sample to estimate how frequently buyer details change recommendations.

IT and app answers were collected on 30 July 2026; penetration-testing answers on 21 August 2026; supplementary software answers on 17 October 2026. We compare answers within each engine, not scores pooled across engines or a time trend.

Historical ChatGPT labels include Codex collection runs. July records label the models gpt-5.6-terra, gemini-2.5-flash and sonnet. August ChatGPT is recorded as gpt-5.6 through Codex, and Claude as claude-sonnet-5 with web search. The August Gemini collection-level label conflicts with its per-answer gemini-2.5-flash label. Exact Claude versions are not consistently verified. These are stored labels, not a verified comparison of current consumer apps.

The supplementary Gemini record is labelled gemini-2.5-flash. The UK and US collections were separate; their stored location field identifies a runner rather than a verified user location. The software suppliers’ buyers and the buyers in the other cases should not all be described as technology companies.

### What counted as a shortlist place

We counted each unique supplier affirmatively offered as a candidate for the requested purchase, including explicit conditional alternatives. We excluded incidental mentions, customers in case studies and names appearing only as sources. Supplier names in the results describe model suggestions; they do not identify our clients or imply an endorsement.

Repeated mentions counted once. Explicit name variants were reconciled, including LRQA/Nettitude. Product and operator references were counted once at supplier level. These mappings reconcile the stored text and do not assert current corporate ownership.

An explicit overall preference was coded separately. Being first in a list did not establish that a company was the engine’s preferred choice. Retention equals suppliers in both answers divided by suppliers in the first answer. Newcomer share equals suppliers only in the second answer divided by suppliers in that second answer. These percentages have different denominators and need not add to 100%.

### All percentages and their denominators

Exact bases for every comparison

| Buying comparison / recorded engine | First list retained | Second list new |

| --- | --- | --- |

| IT support: 30-person outsourcing / ChatGPT / Codex | 43% (3 of 7) | 25% (1 of 4) |

| IT support: 30-person outsourcing / Gemini | 67% (6 of 9) | 57% (8 of 14) |

| IT support: 30-person outsourcing / Claude | 83% (5 of 6) | 0% (0 of 5) |

| IT support: IT and phones together / ChatGPT / Codex | 14% (1 of 7) | 83% (5 of 6) |

| IT support: IT and phones together / Gemini | 44% (4 of 9) | 50% (4 of 8) |

| IT support: IT and phones together / Claude | 33% (2 of 6) | 60% (3 of 5) |

| App development: £200k charity brief / ChatGPT / Codex | 0% (0 of 7) | 100% (3 of 3) |

| App development: £200k charity brief / Gemini | 12% (2 of 17) | 71% (5 of 7) |

| App development: £200k charity brief / Claude | 71% (5 of 7) | 17% (1 of 6) |

| Pen testing: software buyer on a budget / ChatGPT / Codex | 0% (0 of 5) | 100% (4 of 4) |

| Pen testing: software buyer on a budget / Gemini | 22% (2 of 9) | 86% (12 of 14) |

| Pen testing: software buyer on a budget / Claude | 20% (1 of 5) | 83% (5 of 6) |

| AI governance: UK to US (supplementary) / Gemini | 27% (3 of 11) | 79% (11 of 14) |

| AI governance: UK to US (supplementary) / Google AI Mode | 50% (2 of 4) | 60% (3 of 5) |

### Checks, alternative counts and limitations

One AI analyst read and coded all 25 answers, reconciled names, checked supporting text and recalculated the overlaps. A further challenge of the same analysis examined conditional candidates, secondary lists and competing explanations. This is a same-analyst review, not independent coding or second-model verification.

How alternative counting choices affect the findings

| Counting choice | Main result | Alternative result |

| --- | --- | --- |

| Omit Cumulus from the general ChatGPT / Codex IT list because it sits outside the main table | 43% retained for the size comparison; 14% for IT and phones | 50%; 17% |

| Count only Gemini’s 11 main app-agency bullets, excluding six additional positive alternatives | 12% retained | 18% retained |

| Omit Claude’s two conditional pen-testing options from the second list | 20% retained; 83% of the second list new | 0% retained; 100% new |

| Include four separately labelled consultancy/framework providers in Gemini’s UK software list | 27% retained | 20% retained |

The main count includes conditional candidates consistently. The supplementary software comparison excludes its separately labelled consultancy section. These alternatives test interpretation of the same answers; they are not additional observations.

There were no identical-question repeat controls. Changes in wording, buying situation, collection conditions and ordinary answer variation cannot be separated. The study does not measure ranking quality, verified supplier suitability, buyer behaviour, conversion or a causal content effect.

Client and account identities and private source exports are withheld. The public supplement contains the question pairs, dated coded supplier lists, exact calculations and counting notes. The full original answers and source-file checksums remain in the private research archive.

[Download the comparison data](/research-data/2026-10-19-tech-shortlists/comparisons.json) · [Read the public method](/research-data/2026-10-19-tech-shortlists/method.md) · [Download the chart](/img/insights/tech-shortlists/shortlist-retention.svg)