What we encountered
Generative systems do not always return the same answer to the same question. Wording changes, the set of firms named can change, and the order can change. Providers also update models and retrieval over time.
Small differences in how a question is phrased — “best”, “experienced”, or a description of a specific legal problem — can plausibly change what comes back.
Why it matters
A single answer can be unrepresentative in either direction. A firm can look strongly positioned on one run and absent on the next. Decisions about marketing budget or strategy should not rest on one draw.
It also means “we’re number one in ChatGPT” is not a stable claim unless it is backed by repeated observation under controlled conditions.
How FirmRanker handles it
FirmRanker is designed around controlled prompts, repeated runs of equivalent conditions, and preservation of every raw answer. Visibility is then described in terms such as appearance frequency, recommendation frequency and stability across runs — rather than a single position.
The approach is outlined in the methodology and discussed in Why “ranking in ChatGPT” may be the wrong mental model.
What we still don’t know
How many repetitions are needed before a pattern is reliable for a given market and practice area.
How quickly visibility patterns change over weeks and months, and how much of any change is caused by model updates rather than anything a firm did.