Skip to content

More capable AI models made more consistent hotel choices, but did not agree on what to prefer

Strategy review & optimization


Reader Note:
The benchmark tested 28 LLMs from 8 providers on 3,600 hotel booking tasks built from 179 real New York City hotels. More capable models gave more stable, predictable answers, but they did not share a common lean toward cheaper or better-reviewed hotels. The comparison is across models at one point in time; the paper notes that architecture, alignment and training data may drive both capability and consistency.
Evidence:
Experimental
Vendor interest:
None
Sample:
28 LLMs from 8 providers tested on 3,600 hotel choice tasks built from 179 real New York City hotel properties; each task was scored in both original and swapped orderings.

Good for

Backing budget and pricing decisions in similar situations.

Not for

  • Situations far from the one tested, without checking again.
  • This was measured in one market. It may not hold where you are.
  • The source never says when this data was collected. It could be considerably older than the publication date suggests.
  • The two groups were not split at random — they picked their own side. The difference may have been there before anything was measured.

Coming soon

This feature isn’t available yet.