If what you need is company identity, filed financials, ownership structure, or validated tax IDs across a lot of markets, license a dataset. That is not a hedge. Those fields come from official registries, they are standardised, and thousands of buyers want the same records, so somebody has already done the hard part and will sell it to you cheaply. Building that yourself is a well-documented way to spend a year arriving where you could have started.
And yet every data team we work with keeps a list. Fields the dataset does not carry. A sub-segment where coverage thins out. A refresh cadence nobody sells. The list is never long, but it is never empty either, and it is where the arguments start.
This article is about that list. Four situations where commissioning extraction is the right call, four questions to check before you do, and the honest answer about when it is not.
Quick reference
| Use case | What it covers | When it is NOT the answer |
|---|---|---|
| Fields no dataset sells | Licence status, accreditation, product lines, facility counts | Two or more vendors already sell the field |
| Coverage that stops at your segment | Sub-segments a registry-led dataset thins out on | Your provider can close it under the contract you already have |
| A cadence the market does not sell | Event-driven watching of a defined account list | Monthly refresh is genuinely good enough |
| A schema only you use | Classification against an internal taxonomy | The taxonomy maps cleanly onto a standard one |
Use case
Fields no dataset sells
What it covers
Licence status, accreditation, product lines, facility counts
When it is NOT the answer
Two or more vendors already sell the field
Use case
Coverage that stops at your segment
What it covers
Sub-segments a registry-led dataset thins out on
When it is NOT the answer
Your provider can close it under the contract you already have
Use case
A cadence the market does not sell
What it covers
Event-driven watching of a defined account list
When it is NOT the answer
Monthly refresh is genuinely good enough
Use case
A schema only you use
What it covers
Classification against an internal taxonomy
When it is NOT the answer
The taxonomy maps cleanly onto a standard one
Where a licensed dataset ends
There is a clean economic line between what a dataset sells you and what it never will, and it is worth naming because it settles most arguments before they start.
The core of company data is registry-backed. Legal name, address, industry code, incorporation status, filed accounts, ownership, tax IDs. Those fields have three properties that make them cheap: an official body publishes them, the schema is stable, and every buyer in the market wants the same records. That shared demand is what pays for the standardisation. Platforms operating at this layer carry hundreds of millions of companies across 200 or more markets with hundreds of attributes each, and you get the benefit of maintenance you did not have to do.
The long tail has none of those properties. It is fragmented across regulators, specific to one jurisdiction or one industry, often access-controlled, and usually valuable to a handful of buyers rather than thousands. No vendor builds it because there is no resale market to fund it.
Here is what the long tail actually looks like in practice:
| Source class | What it yields | Why no dataset sells it |
|---|---|---|
| Sector licence and authorisation registers | Permit status, licence class, expiry, sanctions | Fragmented per regulator and per country |
| Trade and professional membership directories | Accreditation, certification, membership tier | Access-controlled, no redistribution rights |
| Company product and pricing pages | SKUs, plans, list prices, service lines | Changes constantly, and it is specific to the buyer |
| Facility and capacity pages | Plant count, site type, throughput | Industry-specific, no standard schema |
| Tender and procurement portals | Contract awards, buyer relationships | Per-jurisdiction and time-boxed |
| Careers and hiring pages | Team structure, technology signals | Volatile and inference-heavy |
| Event and exhibitor listings | Market presence, named roles, segment focus | Ephemeral and per-event |
Source class
Sector licence and authorisation registers
What it yields
Permit status, licence class, expiry, sanctions
Why no dataset sells it
Fragmented per regulator and per country
Source class
Trade and professional membership directories
What it yields
Accreditation, certification, membership tier
Why no dataset sells it
Access-controlled, no redistribution rights
Source class
Company product and pricing pages
What it yields
SKUs, plans, list prices, service lines
Why no dataset sells it
Changes constantly, and it is specific to the buyer
Source class
Facility and capacity pages
What it yields
Plant count, site type, throughput
Why no dataset sells it
Industry-specific, no standard schema
Source class
Tender and procurement portals
What it yields
Contract awards, buyer relationships
Why no dataset sells it
Per-jurisdiction and time-boxed
Source class
Careers and hiring pages
What it yields
Team structure, technology signals
Why no dataset sells it
Volatile and inference-heavy
Source class
Event and exhibitor listings
What it yields
Market presence, named roles, segment focus
Why no dataset sells it
Ephemeral and per-event
The rule that falls out of this is short. If a field has a resale market, buy it. If it does not, that is the only work worth commissioning.

The four cases where extraction earns its cost
Fields no dataset sells
This is the clearest case and the easiest to recognise, because the gap is categorical rather than partial. The field is not thin. It is absent.
Take an insurer qualifying broker partners. They need licence class, authorisation status, and expiry date for every broker on the list. That information exists, it is public, and a regulator publishes it. It just lives on each regulator's own register, in a different structure per country, behind a search form rather than a download. No company dataset carries it, because no dataset can justify maintaining forty regulator-specific parsers for a field that a dozen buyers want.
That is what fully managed firmographic data extraction is for: a provider builds and runs the collection against those sources, handles the structure changes, and delivers the field into your schema on an agreed cadence. You are not buying a dataset. You are buying the maintenance of a pipeline that produces one.
When it is not the answer. If two or more vendors already sell the field, buy it. Commissioning custom work for a commodity attribute is the most common and most expensive mistake in this category, and it usually happens because nobody checked the market first.
Coverage that stops at your segment
Here the field exists and the dataset carries it. The problem is that coverage thins exactly where your ICP sits.
Registry-led datasets are deep on registered entities, which is most of the economy but not all of it. If your buyers are sole traders, micro-operators, or businesses in a market where registration thresholds sit above their size, you can license an excellent dataset and still find your segment underrepresented in it. Nothing has gone wrong. You are simply asking a registry-shaped source for something that sits outside registries.
Before you commission anything here, do the boring thing first: ask your provider. Coverage gaps are frequently a contract conversation rather than an engineering project. Providers extend coverage for named segments more often than buyers assume, and a request costs you an email. We have watched teams spend real money building collection for a segment their existing agreement would have covered on request.
When it is not the answer. When the gap closes with a coverage request. Exhaust that before anything else.
A cadence the market does not sell
Sometimes the field and the coverage are both fine and the timing is wrong.
Subscription products refresh on a schedule that suits the vendor's economics: monthly is common, quarterly is not unusual. For most attributes that is fine. Company data does decay, at roughly 2.1% of records a month and around 22.5% over a year (HubSpot Database Decay Simulation), so a monthly cycle keeps you broadly current. But decay is an average across a whole database, and averages hide the fields you actually care about.
Consider a manufacturer watching 4,000 named suppliers for site closures and capacity changes. Annual churn is low. The events matter enormously and they matter the week they happen. A monthly refresh across the full universe is both too slow for the signal and far more data than the question needs. What that team wants is a narrow watch on a defined list, running often, triggering on change.
That is a cadence question, not a coverage question, and it is worth being precise about the difference. The argument for matching refresh to the field is not that licensed data is worse. It is that refresh should be matched to how fast the specific field moves, which for most of your attributes is slower than you think and for a few is much faster.
When it is not the answer. When monthly is genuinely good enough. Most of the time it is.
A schema only you use
The fourth case is the one teams most often misdiagnose, because it does not look like a data problem at all.
Every mature company eventually builds a taxonomy that is theirs alone. A risk tier. A servicing model. A segmentation that cuts across standard industry codes in a way NACE and NAICS were never going to express. The inputs are usually public and often already licensed. The work is deciding, consistently and at volume, which bucket each company belongs in.
This gets mistaken for enrichment, and it is not. Enrichment appends a field somebody else has already defined. Here nobody has defined it but you, so nobody can sell it to you, and the accuracy of the output depends entirely on how well the classification rules encode judgment your team currently carries in its head. The collection is the easy half.
When it is not the answer. When the taxonomy maps cleanly onto a standard one. If your six tiers are industry codes with new names, use the codes.
Managed extraction is not DIY scraping
There is a well-argued case that scraping brings risks and hidden costs, and that licensed, registry-based data is the better foundation. That case is correct, and everything above has agreed with it.
But it is an argument about DIY collection, and it gets read as an argument about extraction generally. Those are different things, and the difference is where most of the money is lost.
The technique is not what fails. Running it unmonitored is. And the failure mode is worse than people expect, because it is quiet.
The dangerous failure is not the crash. When a source page changes, a scraper rarely stops. The selector still matches something, just not the right node. So the job completes, the row count looks normal, and plausible, wrong values flow into your CRM and your models for weeks. A crash you notice on the first morning. Silent drift you notice in a QBR.
That is manageable at a handful of stable sources. It stops being manageable as the source count climbs. Practitioner reporting from teams running collection at scale suggests something like 40% of engineering time goes to maintenance and incident response, and that in change-heavy sectors 10-15% of crawlers need fixing in any given week. Somewhere around two hundred sources, structural changes start arriving faster than a small team fixes them, and the backlog never clears again.
None of that is an argument for scraping less. It is an argument for either keeping DIY collection small and stable, or running it as a monitored pipeline rather than a set of scripts, with change detection, validation against expected ranges, and someone accountable when a source moves. That second option is what "managed" means. It is the maintenance you are buying, not the code.

The four-question test
Before commissioning anything, run the gap through four questions. They take about ten minutes and they kill most projects, which is the point.
| # | Question | If yes |
|---|---|---|
| 1 | Is it already in a contract you hold? | Coverage request, not a project. Ask your provider. |
| 2 | Does any vendor sell it? | Buy it. Custom work on a commodity field is money lit on fire. |
| 3 | Does it change faster than a person can hand-maintain it? | If no, a spreadsheet and an afternoon a quarter is a legitimate answer. |
| 4 | Would a wrong value fail silently? | If yes, whatever you build needs monitoring, not just collection. |
#
1
Question
Is it already in a contract you hold?
If yes
Coverage request, not a project. Ask your provider.
#
2
Question
Does any vendor sell it?
If yes
Buy it. Custom work on a commodity field is money lit on fire.
#
3
Question
Does it change faster than a person can hand-maintain it?
If yes
If no, a spreadsheet and an afternoon a quarter is a legitimate answer.
#
4
Question
Would a wrong value fail silently?
If yes
If yes, whatever you build needs monitoring, not just collection.
Commission the work when the field is not in your contract, nobody sells it, it moves faster than hand-maintenance, and a wrong value would pass unnoticed. Any other combination has a cheaper answer, and the cheaper answer is usually an email to a provider you already pay.

What to specify when you commission
If the gap survives all four questions, the quality of what you get back is decided almost entirely by the brief. Most disappointing extraction engagements were underspecified, not badly executed.
Six things are worth settling in writing before work starts.
The source list, named. Not "licence registers in Europe" but the specific registers, with URLs. If the provider is choosing sources, that is a research engagement with a different price and a different timeline, and you should both know that going in.
The schema, with types. Field names, data types, units, and what an empty value means. Whether an absent licence means "no licence" or "not found" is the kind of ambiguity that quietly poisons a dataset for a year.
Match keys. How a collected record joins to the company records you already hold. Name matching alone will not do it, and resolving that later costs more than specifying it now.
Cadence and the trigger. Weekly, monthly, or on change. If on change, define which fields count as a change worth reporting.
Coverage expectation, stated as a number. What percentage of your list should return a value, and what happens when a source is unreachable. A provider who will not give you a target here is telling you something.
Validation rules. Ranges, allowed values, and what should happen when a record fails them: drop it, flag it, or deliver it marked. This is the clause that turns silent drift into a visible alert, and it is the one most often left out.
None of this is exotic. It is the same specification discipline any data contract deserves, and it takes an afternoon.
Start with one question
The teams that handle this well are not the ones with the biggest data budgets. They are the ones who stopped treating every gap as a build-or-buy decision and started asking a narrower question first: does this field have a resale market?
If it does, someone maintains it better and cheaper than you will, and that includes the registry-grade core that most company data problems turn out to be. If it does not, you are looking at the small, genuinely bespoke slice where commissioned extraction earns its cost, and the only real decision left is whether it gets monitored.
Most teams discover their list is shorter than they feared. That is usually good news, and it is worth finding out before the next quarter's roadmap gets written.
This is a guest article by Forage AI, which builds managed web and company data extraction pipelines for teams whose requirements sit outside what off-the-shelf datasets cover.



