AcademicarXiv, 25 Sep 2026
More capable AI models made more consistent hotel choices, but did not agree on what to prefer
Strategy review & optimization
- Reader Note:
- The benchmark tested 28 LLMs from 8 providers on 3,600 hotel booking tasks built from 179 real New York City hotels. More capable models gave more stable, predictable answers, but they did not share a common lean toward cheaper or better-reviewed hotels. The comparison is across models at one point in time; the paper notes that architecture, alignment and training data may drive both capability and consistency.
- Evidence:
- Experimental
- Vendor interest:
- None
- Sample:
- 28 LLMs from 8 providers tested on 3,600 hotel choice tasks built from 179 real New York City hotel properties; each task was scored in both original and swapped orderings.
Good for
Backing budget and pricing decisions in similar situations.
Not for
- Situations far from the one tested, without checking again.
- This was measured in one market. It may not hold where you are.
- The source never says when this data was collected. It could be considerably older than the publication date suggests.
- The two groups were not split at random — they picked their own side. The difference may have been there before anything was measured.




