How Ad Networks Profile You — The Real-Time Bidding Pipeline

When you load a web page that contains advertising space, something remarkable happens in the gap between the page request and the page appearing on your screen. In a window of typically under 150 milliseconds — though actual latency varies by implementation — a fraction of the time it takes to blink — your browsing profile is broadcast to potentially hundreds of advertisers, auctioned to the highest bidder, and an ad is fetched and inserted into the page. You see the result; you do not see the infrastructure.

This process is called real-time bidding (RTB), and it is the dominant mechanism through which display advertising on the open web is bought and sold. RTB replaced the earlier model of direct deals between publishers and advertisers — in which a newspaper's website, for example, would negotiate directly with an advertiser for specific inventory at a fixed price. RTB made advertising buying automated, instantaneous, and audience-based rather than context-based: advertisers no longer buy space on a particular website, they buy audiences — people matching specific profile criteria — wherever those people happen to be browsing.

Why RTB developed

The shift to programmatic advertising via RTB accelerated in the late 2000s and early 2010s for straightforward economic reasons. Publishers had more advertising inventory than they could sell through direct deals; the remnant (unsold) space was auctioned through ad networks. Advertisers wanted to reach their target audiences at scale without negotiating individually with hundreds of publishers. Intermediary technology companies built the auction infrastructure and extracted transaction fees.

The result was a global, automated advertising market operating at billions of transactions per day. Google's DoubleClick, acquired in 2007, became the dominant infrastructure provider on both the publisher side and the buyer side. This dual-sided market position became the subject of antitrust investigations in the US, UK, and EU.

The scale problem

The ICO's 2019 report estimated that the UK RTB market alone processed tens of billions of bid requests per year, with each bid request transmitting a profile of the visiting user to multiple demand-side platforms and bidders (ICO, 2019). A single user loading five pages on a news website in a morning might generate hundreds of individual bid request events — each one transmitting data about that user to a different set of bidding parties.

The data transmitted in each bid request is not a simple name and address. It is a structured profile that can include dozens of data fields covering inferred demographics, location, device type, browsing category, and behavioural segments assigned by data brokers. Understanding what is in those fields — and who receives them — is the substance of this topic.