Token Economics: What It Costs an AI Agent to Research Your Business

Every token is a cost decision

Large language models process text as tokens - roughly four characters each. When an AI agent researches your business, it fetches your homepage, extracts roughly 18 percent of the raw HTML as usable text, and converts that into tokens. Then it decides whether to visit more pages. Each additional page adds to the token budget.

At current API prices, input tokens cost between $0.75 and $3.00 per million for mid-tier models. That sounds cheap until you multiply it by the number of pages, the number of research sessions per day, and the number of vendors being compared in parallel. Agents are designed to be efficient. When one business costs fifty times more tokens to understand than a competitor with a structured profile, the agent allocates its budget accordingly.

The math behind your Site Scan

When you run a Site Scan on our homepage, we calculate your estimated agent research cost. Here is the model: an agent typically visits your homepage first, then follows up on three to four key pages in a discovery round. If the agent is still interested, it does a second round hitting pricing, services, and policy pages - up to ten pages total.

For each page, we measure the raw HTML weight, apply a content extraction ratio of 18 percent (the rest is CSS, JS, and markup), and convert to tokens. A business with a 300KB homepage and ten sitemap pages might cost an agent 8,000 to 12,000 tokens per research visit. At $3 per million tokens, that is roughly 3 cents per visit. Compare ten businesses and the agent has spent 30 cents just on raw parsing before any reasoning begins.

With an AI Website Profile: 99 percent reduction

A typical llms.txt file is 2,000 to 6,000 characters of clean, structured text. No HTML to strip, no JavaScript to skip, no navigation labels to filter out. That translates to roughly 500 to 1,500 tokens - a 90 to 99 percent reduction compared to a multi-page crawl.

The agent reads one file, gets a complete and verified understanding of your business, and moves on to reasoning and comparison. No second round needed. No guessing about pricing hidden in PDFs. No wasted tokens parsing cookie consent banners. This is why we call it infrastructure: it makes the agent's job fundamentally cheaper and faster.

Why this matters more as agents scale

Today's AI assistants handle millions of queries per day. Tomorrow's autonomous agents will handle billions of vendor evaluations, price comparisons, and booking decisions. The economic pressure on per-query cost will only increase. Businesses that are cheap to read will get more agent attention. Businesses that are expensive to parse will be summarized from cached snippets or skipped entirely.

SDF Protocol, an emerging web standard, demonstrates this trajectory: it reduces a typical 89KB webpage to approximately 750 tokens by pre-extracting entities and relationships. The entire industry is moving toward lighter, more structured data formats for machine consumption. Your AI Website Profile puts you ahead of this curve today.

Frequently asked questions