MCPera · Docs Business & Feasibility Plan
Chapter 03 · Updated 2026-10-13

Technical Feasibility: Collection & Verification

Design principles (throughout the chapter): ① Being a pure conveyor of claims has no value; the value is in the verification layer — an upstream registration does not mean usable; ② every field is traceable (source_url + fetched_at + check_method + confidence), a hard requirement from A-013/A-015 and the compliance foundation for data licensing; ③ raw snapshots are immutable + derived layers are recomputable, so any verification-logic upgrade can backfill history; ④ narrow first, wide later: the MVP covers only the subset that can be automated reliably — prefer few-but-true fields over many-but-fake (A-003 "narrow but true").

4.1 Data-source matrix (fully sufficient for cold start, and compliant)

Source Access (verified reachable) Fields it provides Frequency/scale Compliance notes
Official MCP Registry API GET /v0/servers?limit=100&cursor= Public REST, cursor pagination (verified limit≤100, returns metadata.nextCursor) name/title/description/version/remotes(type+url)/packages/status/publishedAt/updatedAt/isLatest + schema URL Live API, several thousand records total Official metadata consumed per its terms; trust only its "registration facts", not "usability"
Smithery Registry API registry.smithery.ai/servers?page=&limit= Public REST, pagination includes pagination.totalCount (verified 19,932) useCount/verified/remote/isDeployed/createdAt/homepage/owner Live API, ~20k Public read-only; treat useCount etc. as signals and cite the source, not as our own rating
npm Registry/search API registry.npmjs.org/-/v1/search?text=keywords:mcp-server Public REST (verified 9,775 packages); plus a skimdb changes feed package name/version/maintainers/dependencies/downloads (via the npm downloads API)/README/release history Incremental via changes feed Package metadata is public; respect each README's own license when extracting
GitHub API (REST/GraphQL) Token auth; search repos/topics, read READMEs, releases, issues, commit activity stars/activity/latest commit/issue responsiveness/language/license/owner Rate-limited; needs caching + ETag Respect API terms and rate limits; use the license field for compliance grading
Docker Hub API hub.docker.com/v2/search/repositories/ Public REST images/pull counts/updated time Search is noisy (verified 400k+ including noise) Use only for secondary enrichment of entities already hit by other sources, not as a standalone base
Vendor sites / docs / pricing pages Controlled HTTP fetch (render: static first, headless browser fallback for SPAs) pricing, auth method, SLA, region, ToS, contact Scheduled by entity priority Respect robots/ToS, rate-limit, take only public pages, store source URL + fetch time; never touch login-gated content
The MCP endpoint itself (the most critical self-produced source) Direct handshake per the MCP protocol (see 4.4) liveness/latency/tools list/input schema/auth challenge/compatibility signals Periodic active probing Only protocol-level side-effect-free calls (initialize/tools/list), never trigger writes; respect the target's rate limits
UGC / vendor claims Form + review usage experience, config pitfalls, real pricing, client-compatibility feedback Ongoing Human+AI dual review, source labeled as community, credibility scored independently

Cold-start path (no anti-crawler arms race needed): official Registry (skeleton) → Smithery (add useCount/verified/homepage) → npm/GitHub (add code activity/package info/local transport) → endpoint self-check (add liveness/capabilities/auth) → site/pricing fetch (add commercial fields). The first four steps are all structured APIs — a credible base of several thousand to ten-thousand-plus entities can be built without scraping any HTML.

4.2 Overall architecture (startable by one person, smoothly scalable to enterprise grade)

Five layers with one-way dependencies, independently replaceable and independently scalable:

[L1 Ingestors]
  Official Registry / Smithery / npm / GitHub / DockerHub / site crawler / MCP probe / UGC
        |  each ingestor emits a "RawRecord + source metadata", written immutable
        v
[L2 Raw Store]  object storage/disk: raw JSON/HTML snapshots partitioned by source/entity/date (immutable)
        |
        v
[L3 Normalize / Enrich]
  (1) deterministic normalization: identity resolution, field mapping, unit/currency normalization
  (2) AI-assisted extraction: README/pricing/docs -> structured fields (pricing model, auth, capabilities) with evidence snippets + confidence
  (3) rule-based validation/tagging: category, transport, deprecated?, risk flags
        |  results written to
        v
[L4 Verification] (core moat, independently scheduled)
  liveness probe / tools list / auth challenge / client compatibility matrix / latency / historical-snapshot diff
        |
        v
[L5 Serve]
  normalized SQLite/Postgres DB + generated static JSON
   |- static site generator -> programmatic SEO pages (zero-build can continue; see chapter 7 compatibility)
   |- read-only data API (versioned /v1/, keys & quotas) -> B2B/OEM licensing
   +- alerts/webhooks (supply cuts, redesigns, price drops, new competitors)

Key engineering trade-offs:

  • Immutable raw + recomputable derived: extraction/verification algorithms keep iterating, so we must be able to "replay" historical snapshots — the precondition for backfilling A-003's time dividend.
  • Plug-in ingestors: one adapter per source, all emitting RawRecord{source, source_url, fetched_at, content_hash, payload}. Adding a source (PyPI, crates, HuggingFace, cloud marketplaces) is just a new plugin without touching upper layers.
  • Identity resolution as its own module: the same MCP often appears in the official Registry/Smithery/npm/GitHub (e.g. brave). Merge via "canonical id + multi-source aliases + evidence score"; when in doubt keep suspected duplicates rather than wrongly merging (a wrong merge pollutes data worse than a duplicate).
  • Phased storage: MVP uses SQLite + on-disk JSON snapshots (tens of thousands of entities, zero ops, naturally compatible with the current zero-build site); once the data API/multi-tenancy/licensing matures, migrate to Postgres (or DuckDB for analytics). Object storage can be a local directory now and an S3-compatible store later with an unchanged interface.

4.3 AI-assisted extraction: used only where it matters, tightly constrained against hallucination

AI handles only the unstructured→structured segment (READMEs, pricing pages, docs, web copy); all "hard facts" (version, URL, stars, downloads, protocol-handshake results) go through deterministic APIs — the LLM is not allowed to invent them.

  • Constrained target schema (constrained output): fixed function-calling/JSON-mode output, e.g. pricing={model: free|freemium|subscription|pay_per_call|contact, currency, price, unit, free_trial, evidence_quote, source_url}, auth={type: none|api_key|oauth|... , where_to_get_key, scopes?}, capabilities=[...], transport=[stdio|streamable-http|sse], region_hint.
  • Evidence mandatory: every extracted value must carry a verbatim source quote + source URL + fetch time + confidence. Fields without evidence are set to null, never guessed — aligned with A-013's "source URL + verification date".
  • Confidence tiers: high (explicit table/pricing page), medium (inferred from copy), low (merely hinted). Low-confidence fields are not shown publicly; they enter a "needs human / needs re-probe" queue.
  • Cross-validation to cut hallucination: when AI-extracted transport/auth conflicts with the "MCP probe's measured result", the measurement wins and the contradiction is logged; pricing is adjudicated with priority official pricing page > docs > README.
  • Assistance with classification & dedup: let the model assign multi-labels within a controlled taxonomy and propose "suspected duplicate entity" candidates (final merge is rule + human confirmation).
  • Cost control: first filter with rules/regex (prices, /mcp endpoints, OAuth keywords); only send rule-uncertain items to the model; cache/dedup by content_hash so unchanged pages aren't re-extracted.
  • Multilingual: vendor sites come in Chinese and English; the extraction layer is language-agnostic and the presentation layer does i18n (English first, Chinese reserved).

4.4 The real-test verification chain (the moat itself): programmatically handshaking an MCP

This is the layer others skip, yet it is fully automatable and directly determines trust. All probes are read-only, side-effect-free handshakes — no write tools are invoked and no target paid quota is consumed.

Four probe levels, scheduled progressively by cost (not every entity runs the heaviest):

  1. L0 endpoint liveness: for remotes[].url — HTTP reachability, TLS cert, redirects, /.well-known, response latency, timeout, status code. Lightweight; full coverage at high frequency (e.g. daily).
  2. L1 protocol handshake: send initialize per the MCP spec (declaring client name/version/protocol/capabilities); check it correctly returns server capabilities and a protocol version; then send ping. Decide "is it a real MCP endpoint, which protocol version, is it compatible".
  3. L2 capability inventory: send tools/list (and resources/prompts per advertised capability) to obtain the tool list and each tool's inputSchema; record tool count, naming, parameters, pagination, and whether auth is required (on an auth challenge/401, record the auth method and authorization URL — never attempt to bypass).
  4. L3 client compatibility matrix (differentiation highlight, can come later): repeat the handshake with multiple "client profiles" (different protocol versions / capability declarations: Claude Desktop, Cursor, various SDK versions), recording who connects, who degrades, who fails. This is the MCP counterpart of A-013's cross-model/cross-client test matrix. For local stdio types, spin up npm i/a container in a sandbox and do the same handshake, recording install success / build time / dependency issues.

Each probe yields an immutable ProbeResult: {server_id, probe_level, started_at, latency_ms, ok, protocol_version, auth_required, auth_method, tool_count, capabilities, errors[], client_profile, evidence}, and rolls forward both the "current status" and the "availability time series" (uptake / first-available / last-available / consecutive-failure days / supply-cut flag).

  • Security & compliance: read-only handshakes; for key-required services the mere fact "auth required" is enough — do not hoard user keys; enterprise users may optionally "bring their own key for private probes" (a future value-add). Local package installs are isolated in a sandbox (container + resource/network limits) against malicious/low-quality packages.
  • Politeness & robustness: rate limits, concurrency budgets, backoff-retry, respect robots/ToS, failure classification (down / blocked / protocol-mismatch / auth-required / timeout), to avoid misreading "rate-limited by us" as "it's down" (lesson from "a uniform wall of all-exceptions first means suspect the ruler": the probe itself needs health self-checks).

4.5 Data model (normalized core entities; field-level schema in Appendix B)

  • Server (canonical entity): id, slug, display_name, canonical_homepage, one-line description, taxonomy tags, official status, first-seen / last-verified timestamps, composite trust score, alias set.
  • ServerVersion (time dimension): server_id, version, published_at, is_latest, status, source — maps to the official Registry's versioned records and enables change history.
  • Endpoint: url, transport (stdio/streamable-http/sse), hosted/local, package coordinate (npm/pip/docker), owning version.
  • Vendor: organization/author, homepage, ownership of multiple servers, verification status.
  • Pricing / Auth / Compliance (mostly AI extraction + evidence): price model, currency/amount/unit, free tier, auth type, data-residency/compliance hints, evidence and confidence.
  • ProbeResult (immutable) and CompatibilityRow: handshake outcome at server_version × client_profile × time.
  • Snapshot/SourceRecord (immutable raw layer): source, url, fetched_at, hash, payload.
  • Review/UgcClaim: community feedback and vendor claims with independent credibility.
  • Extension by design: adding a dimension (SLA, region, SOC2, etc.) is a new table/JSON field + one new ingestor/probe, without changing existing pages or the API contract (version the API with @vN; only add fields, never break old ones).