What we encountered
When you read AI answers to legal questions closely, the way a firm appears varies a great deal. Sometimes the answer names several firms and invites the user to consider them. Sometimes it singles one out and says, in substance, “this one”. Often it hedges — suggesting the user speak to a lawyer and offering names only as starting points.
The prompt does not settle which of these happens. A user can ask “which firm would you recommend?” and receive a neutral list. The wording of the question and the treatment in the answer are separate things.
Why it matters
If every appearance is counted as a “recommendation”, a firm that is routinely listed fifth among eight looks identical to a firm that is repeatedly endorsed. Those are not the same position in a prospective client’s decision.
It also makes comparisons unreliable. Two reports can both say a firm “appeared in ChatGPT” while describing very different outcomes.
How FirmRanker handles it
FirmRanker classifies how each answer treats each firm, rather than recording a simple yes/no appearance. The working categories are set out in What actually counts as an AI recommendation?: mentioned, suggested (placed in a consideration set), recommended (explicitly endorsed), source-only and negative/cautionary.
Classification is based on the response, not the prompt, and the original answer is preserved so the classification can be checked or revised.
What we still don’t know
Where exactly the boundary between “suggested” and “recommended” should sit in borderline answers — for example, a list where one firm receives noticeably warmer language.
How consistently human reviewers draw that boundary, and how closely automated classification can match them. That is part of what validation has to establish.