Algorithm Tour

We collect search signals from several sources: the client’s Search Console queries, autocomplete suggestions, questions asked to AI assistants, and what competitors already cover. Each point is a raw query — thousands of them, noisy and redundant.

Queries are turned into vectors with a language model. Different phrases with the same intent (“apartment in Botafogo”, “living in Botafogo”, “is Botafogo worth it?”) end up close together in the vector space.

Proximity is measured by the cosine similarity between the vectors.

A clustering algorithm groups the queries into intents. Each group gets an aggregate volume, a difficulty level and an intent type: informational, comparative, transactional.

The groups are cross-checked against the data the client already has. An intent group with supporting data becomes a candidate page type; a group without data becomes a gap to fill — or a discarded idea.

Imagine the type “Living in [neighborhood]” for a real estate network. The raw material is a table: one row per neighborhood, one column per fact — average price per m², number of properties, time to downtown, schools, parks, stations.

First, we classify each column. Key columns identify the row; distinctive columns genuinely vary between rows; cosmetic columns change little or not at all. Only distinctive ones can support a page.

Then we score each row: how many distinctive facts does it have that differ from the column’s most common value? A row with many values of its own becomes a rich page; a row that just repeats the mode would become a thin page.

Rows with identical facts are grouped into a single page, instead of generating copies.

The result is a three-level verdict: build all, build some (with the exact list of those that pass), or build just a hub page. It is a conservative filter by design — one page fewer costs little; a thousand thin pages can cost the whole domain.

Each approved page gets a specification: which blocks must come directly from the row’s data, what share of the text must vary between pages (our goal is above 40%), which filler blocks are forbidden, and what the title and heading pattern is.

A research agent gathers the row’s facts and the permitted external sources. A writing agent assembles the draft within the specification. A verification agent checks every number against its origin — a number without a source is flagged, never invented.

But not every page is equal. Some are routine; others touch sensitive subjects, have contradictory data or belong to a client with a strong editorial voice. A classifier estimates each draft’s risk and decides whether it goes straight through or passes by a human editor.

Editors are not interchangeable either. We assign drafts to editors with an optimization problem similar to assigning customers to warehouses: a cost per (editor, page) pair that combines expertise, history with the client and the size of each person’s queue — recalculated several times a day.

The editor works in an interface built to understand the page quickly: the row’s data on one side, the draft on the other, each claim linked to its source. We test variations of this interface all the time; what we learn about how editors decide flows back to improve the agents.

We take a sample of finished pages and replace each one’s key value with a placeholder. If “Living in Botafogo” and “Living in Tijuca” are identical after that, the variation was just the name — and that is exactly what search engines penalize.

Then we remove the sentences shared by most of the samples. What is left is the text that is truly each page’s own. We measure the variable share against the 40% goal and compare the pages with each other using five-word sequences.

Each page gets one of five labels — ok, near-duplicate, variation only by key, low variable share or thin — and an action: keep, merge with another, or publish with noindex until it gains content. None of this depends on someone reading a thousand pages.

First, the hierarchy: a hub, categories and leaf pages, each level linked to the one above. Sitemaps are split by type and by batch, within Google’s file limits, so we can observe each wave separately.

A small pilot goes live — large enough to be statistically readable, small enough not to hurt if it goes wrong. From there we track three counts per batch: pages published, pages indexed, pages indexed with no impressions at all.

The stop criteria are explicit rules on those ratios. If the indexing rate falls below the threshold, or the share of indexed pages with no impressions rises above it, the next wave does not open. Each cause has its own rollback action: strengthen internal links, enrich data, apply noindex or merge pages.

When a batch passes, the next one opens larger. The same mechanism that accelerates a healthy set brakes a sick one — and the decision is recorded, batch by batch, for anyone who wants to audit.

Once live, each page has a state: published, indexed, with impressions, with clicks, cited by an AI. We record every state change, together with everything that happened between one and the next — an edit, an updated data point, a new link.

With thousands of pages, the transitions become a matrix. It tells us, for example, the probability that an indexed page will be cited by an AI assistant within 30 days — and how that probability changes when the page has a data table, a direct answer at the top or a cited source.

This is a simple Markov chain model, and it is surprisingly useful: it lets us simulate the whole set months ahead, compare page types with each other and identify which page features move the citation probability most.

What the model learns flows back to the writing agents as specification rules, to the risk classifier as new variables and to viability as column weights. The loop closes — and the next wave already starts better than the previous one.

Pages age at different rates. A neighborhood guide changes slowly; a properties-for-sale page changes every day. Each page has a simple “freshness” model: how much its data changed, how much traffic it gets and how long ago it was revalidated.

This model decides the order of regeneration. Instead of rebuilding everything every hour, we regenerate first the pages where the expected gain is greatest — and the same signal feeds the crawl budget we ask search engines for.

Looking even further ahead, search demand is seasonal and predictable. With historical series by intent, we anticipate which sets of pages need to be ready — and reviewed by people — months before the peak, so editorial capacity is where the traffic will be.

In each neighborhood, the property portfolio works like an inventory: each sale or rental removes a unit, and the acquisition of new properties replenishes it. Without replenishment, the “properties for sale” pages empty out and brokers are left with nothing to offer.

The hard part is the timing and size of acquisition. Acquiring too much leaves properties sitting, listings aging and capital tied up; acquiring too little leaves the neighborhood with no supply. We want to minimize the sum of the acquisition cost and the cost of carrying inventory — the latter proportional to the area under the curve.

Adding the two costs, a batch size appears that balances them. Acquiring in larger batches dilutes the fixed cost of each campaign but inflates inventory; smaller batches do the opposite. The minimum of the curve is the economic order quantity.

Use the controls at the end of this chapter to see the minimum shift.

The real question is bigger: how much to acquire of each type, in which neighborhood, and which old listings to take down to make room. We answer with a system dynamics model, fitted to history and used in robust optimization, with quantified uncertainty in the demand forecasts.

x = inventory levels per neighborhood · u = acquisition, reallocation and removal · w = demand forecast. In practice we look at each property type in each neighborhood; here we stay with the simple case.

Client data comes in through connectors and becomes versioned tables: each row carries its source and origin date. That lineage is what lets the verification agent say “this number comes from here” and the editor trust it.

On top of that, a layer of internal services: viability, generation, quality control, launch and learning are independent services, each with its own queue and records. The algorithm teams write the logic; the platform takes care of scaling, repeating and observing.

At the edge, pages are served statically and regenerated incrementally, page by page, in the order the freshness model decides. For the visitor — and for the crawler — the page always exists and always responds fast; all the complexity sits behind it.