GenRank: A Methodology for Measuring Entity Visibility in Large Language Model

1. Introduction

1.1 Background

The proliferation of large language models (LLMs) as primary information discovery tools has fundamentally altered how consumers identify and evaluate products, services, and brands. Unlike traditional search engines that display ranked results based on explicit relevance signals, LLMs generate synthesized recommendations that reflect patterns learned during training and reinforcement learning processes.¹

This shift presents both opportunities and challenges. Entities that achieve favorable positioning in LLM outputs may benefit from increased visibility and consideration, while those absent from recommendations face potential market invisibility—a phenomenon we term AI-mediated discovery bias.²

1.2 Research Objectives

GenRank addresses the following research objectives:

1. Establish a reproducible framework for measuring entity visibility across multiple LLM platforms
2. Develop a market-weighted scoring system that reflects real-world AI usage patterns
3. Create transparent, open-access datasets for academic and commercial research
4. Enable longitudinal analysis of recommendation patterns and temporal trends

1.3 Definitions

Entity

A distinct brand, product, service, organization, or concept that can be identified and tracked across LLM outputs.

Visibility Score

A normalized metric (0–100) representing an entity's prominence in LLM recommendations within a specific category.

Market Weight

A coefficient assigned to each LLM reflecting its estimated share of global AI assistant usage.

2. Data Collection

2.1 Query Design

Queries are designed to elicit ranked recommendation lists from LLMs. Each query undergoes validation against the following criteria:

• Specificity — Queries must target a defined category or use case
• Neutrality — Queries must not contain leading language or brand mentions
• Reproducibility — Queries must consistently elicit list-format responses
• Temporal stability — Queries should remain relevant across update cycles

2.2 Polling Parameters

Approved queries are submitted to each active LLM via official API endpoints. We deliberately do not override decoding parameters: a poll is a sample of what the model actually tells a user, and pinning temperature would measure something no user receives. Variance is acknowledged rather than suppressed: in a standard cycle each query-model pair is polled once, and the pipeline supports repeated sampling — in a sampled cycle each pair is polled multiple times and the resulting spread is published as a confidence interval (§4.5) rather than hidden.

Web search.Polling is executed with the model's native web search enabled. Each polled model is chosen specifically because it supports real-time web search, so recommendations reflect current information rather than training-cutoff knowledge. This constrains the active model set (see §4.2) and is versioned in the Changelog (§6).

2.3 Entity Resolution

Raw LLM outputs undergo entity resolution to map surface-form variations to canonical entity identifiers. This process employs:

1. Lexical normalization — Case folding, punctuation removal, whitespace normalization
2. Alias mapping — Maintained database of known aliases and abbreviations
3. Semantic clustering — LLM-assisted disambiguation for novel entity mentions
4. Human validation — Manual review of low-confidence mappings

3. Scoring Methodology

3.1 Theoretical Foundation

The GenRank scoring system is grounded in information retrieval theory, specifically adapting the logarithmic discounting principles used in normalized discounted cumulative gain (nDCG) metrics.³ We employ logarithmic decay to model the diminishing marginal value of lower-ranked positions while ensuring all mentions contribute non-zero utility.

3.2 Mathematical Formulation

3.3 Score Decay

The logarithmic decay function produces the following position-relative values:

3.4 Example Calculation

Consider entity "Notion" receiving the following rankings for the query "What are the best productivity applications?":

4. Model Weight Allocation

4.1 Methodology

Model weights are derived from a composite index incorporating three primary data sources, each addressing different aspects of real-world AI influence:

• Consumer market share (40%) — Monthly active users and web traffic data from Statcounter, SimilarWeb, and company disclosures
• Enterprise adoption (35%) — Deployment metrics from industry reports and API revenue estimates⁵
• Developer API usage (25%) — Token throughput from aggregated gateways⁶

4.1.1 Weight Derivation (October 2026)

Each component below is the provider's share normalized within the three-provider active set, from the most recent public figures at recalibration time. Values marked ~ are estimates where a source publishes partial breakdowns. Providers whose composite falls below a 3% materiality threshold are excluded from the active set entirely (see the note below the sources).

Sources: consumer — Similarweb web-visit share of major generative-AI chatbots, May–August 2026 (ChatGPT 53.9%, Gemini 27.9%, Claude 9.2%, Grok 2.4%, Perplexity 1.3% of all tracked assistants, renormalized to the active set). Enterprise — enterprise LLM API usage share, mid-2026 industry reports (Anthropic 40%, OpenAI 27%, Google 21% of total enterprise usage; xAI and Perplexity estimated). Developer — model-author token share on public aggregated gateways (OpenRouter), June 2026; shares for providers primarily consumed via direct APIs are estimates and this component is capped at 25% accordingly. Composite = 0.40·consumer + 0.35·enterprise + 0.25·developer, rounded to two decimals.

Materiality threshold. A provider whose composite falls below 3% is excluded: at that weight its influence on published scores is within rounding noise, while its polling cost is not. Perplexity (composite 0.01 within the prior five-provider set) and xAI (0.03 — and roughly a fifth of per-cycle polling cost for under 3% of score influence) were retired from the active set in October 2026. Their historical poll data is preserved, and both are re-evaluated at each recalibration.

4.2 Current Allocation (Q4 2026)

The following table presents active model weights as of October 2026, derived in §4.1.1. The active set is restricted to one web-search-capable model per major provider so recommendations reflect current information rather than training-cutoff knowledge. Weights are recalibrated periodically; every change is recorded in the Changelog (§6).

4.3 Provider Distribution

Aggregate weights by provider:

4.5 Reliability & Disclosure

4.5.1 Sampling and confidence intervals

A language model does not return the same answer twice. Across the 994 questions polled in both the July and August 2026 cycles, an average of 5.12 of the top 10 entities persisted (51.9%), and with a single sample there is no way to separate genuine change from sampling noise. Per-question figures are therefore labeled directional, and the robust signals are the aggregated ones (§4.5.2).

From v3.0 the pipeline supports repeated sampling of each question–model pair. In a standard monthly cycle each pair is polled once, and scores publish without an interval — an absent interval is reported as absent, never as a zero-width claim of certainty. In a sampled cycle (n = 3; first run September 2026), model contributions are averaged before weighting, so scores remain comparable across all cycles, and the spread across samples yields a published confidence interval:

Where published, these intervals are wide — at the level of a single question, wide enough that per-question figures remain directional. They were always this wide; unsampled cycles simply cannot report them. Category rankings aggregate many questions, models and samples, and are correspondingly tighter.

4.5.2 Inclusion rate is the primary metric

Position within a model's answer is the fragile signal; whether a model names an entity at all is the robust one. GenRank therefore reports inclusion rate — the share of all (model × sample) trials mentioning the entity — as its primary measure, with position retained as a secondary quality signal.

4.5.3 What we measure, stated plainly

• Collection channel: provider APIs, not consumer chat interfaces. These surfaces differ substantially, and we do not claim to measure what a consumer sees in a chat window.
• Model tier: the small/fast tier of each provider, named with versions in §4.2 — not the flagship models.
• Grounding: native web search enabled, capped at 3 results per call.
• Decoding parameters: provider defaults, deliberately unset by us (§2.2).
• Version stamp: every score records the methodology version that produced it, so figures are never silently compared across revisions.

4.6 Independence

GenRank does not sell, and will not sell, any service to an entity it ranks that is intended to change that entity's rank. No fee depends on the level or direction of any score, and correction of a mismeasured entity is always free — never a paid service.

This follows the separation that credit rating, benchmark and audit regulation all treat as load-bearing: publishing data and diagnostics is legitimate; selling advice on how to move the measured attributes is not. The full policy, including our material-change and corrections procedures, is published separately.

5. Limitations

Users should consider the following methodological limitations when interpreting results:

6. Changelog

The methodology is version-controlled. Each revision below is immutable and dated; results should always be interpreted against the methodology version in effect at poll time. Versions follow semantic numbering (major.minor): a major bump denotes a change that breaks longitudinal comparability (e.g., scoring formula), a minor bump denotes the model set, cadence, or weights changing.

7. References

1. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.
2. Shah, C., & Bender, E. M. (2024). Envisioning Information Access Systems: What Makes for Good Tools and a Healthy Web? ACM Transactions on the Web.
3. Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422–446.
4. Wilkinson, M. D., et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018.
5. Menlo Ventures. (2025). 2025 Mid-Year LLM Market Update: Foundation Model Landscape + Economics.
6. OpenRouter. (2025). LLM Rankings: Model Usage Statistics.
7. Statcounter. (2025). AI Chatbot Market Share Worldwide.