Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 48 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,53 @@ export** — off licensed data, with no social-network account and no scraping.
reason the LLM chose it. Everything downstream of that file — the message, the mailbox, the
sequence, the opt-out — belongs to whatever tool the operator already sends with.

## Explicit paid source: CompanyEnrich

CompanyEnrich is a B2B company/people data API. Its integration is an optional,
provider-selected page source (`core/discovery_source.py`), separate from the default
free BetterContact search planner and the email-resolver seam. The CLI requires
`--companyenrich-query PATH`, `--companyenrich-request-id ID` and
`--discovery-credit-budget N` together. Only that invocation reads
`OPENOUTFIND_COMPANYENRICH_API_KEY` for discovery; readiness uses that key instead of
BetterContact's. Selection resets at the command boundary, even on errors.

`companyenrich.py` validates direct people-search filters against a bundled OpenAPI
subset (no `companyFilter`, expansions or unknown fields), defaults to page 1/size 10,
and caps size at 100. The reservation is `2 * pageSize` credits at the current published
rate. It sends one POST to the fixed HTTPS host with timeout/body bounds, no redirects,
no retries and redacted errors. A larger budget never causes more requests. `find 0`
never calls it; existing pool work may meet the goal before any purchase.

`core.CompanyEnrichPage` is a durable reservation keyed by the operator-supplied ID,
created in autocommit before HTTP. The same ID cannot submit again, including after a
crash or concurrent invocation. An unknown response blocks reuse until the operator
inspects usage and explicitly authorizes a new ID. A successful full response is saved
before local embedding, enabling no-spend recovery. Query, maximum credits and import
state persist; keys do not. It does not create free-walk Search/Page observations.
The paid-source branch returns the same boolean judge decision as the current planner,
before building the free search belief or fit-rate map.
The generated `0013` merge migration joins the existing CompanyEnrich and market/keyword
guess histories without rewriting either, retaining paid-request reservations on upgrade.

The adapter accepts a usable professional profile URL and current title, copies reported
name parts, country and current employment, and applies the existing company cap.
The free provider's headline heuristic is not applied. The qualifier consumes local
candidates before considering this one page; exhausted authorized data gives
`goal_unreached` and no implicit further spending. The full raw response keeps exact
IDs, nested fields and pagination; existing CSV/JSON lead columns remain unchanged.
`--json` run metadata gains a `discovery` object; existing error types cover refusal.

`crm.Lead.discovery_source` records CompanyEnrich provenance, including duplicate
identities whose existing profile stays unchanged. Later discovery cannot clear that
marker. Contacts-hub resolution and contribution skip those leads permanently; raw
CompanyEnrich profiles/companies never enter the hub sharing path. Qualification still
uses the operator's LLM, and exports still go wherever the operator directs stdout.
See [setup and the complete scope/cost contract](docs/companyenrich.md).
Later free-provider pages also filter identities already marked CompanyEnrich before
sharing raw profiles or employers; alternate host/trailing-slash/query spellings are
matched conservatively. A new source cannot undo the persisted local-only choice.


## Project Layout

All source lives in the single `openoutfind/` package; Django apps are nested inside it
Expand All @@ -28,7 +75,7 @@ openoutfind/
# config.py (the environment) + readiness.py, llm.py, conf.py,
# ML (qualifier/embeddings), discovery+qualify pipeline,
# the lead export, db/ helpers, geo, management commands
enrichment/ # the one paid step (not an app — no models) — the BetterContact
enrichment/ # paid email resolution (not an app — no models) — the BetterContact
# client and the two-step buy/check lookup
crm/ # app (label: outfind_crm) — Lead, Company, Deal
contacts/ # central contacts-store client (service.py only — no models, not an app)
Expand Down
35 changes: 33 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ OpenOutFind — a self-hosted **lead finder that qualifies for you**, with **zer

**define ICP → discover → qualify → rank → (optionally) resolve an address → export CSV.**

- **Discovery** is a licensed source — BetterContact **Lead Finder** (ICP search returns firmographic profiles, no emails, billed nothing). The walk is a best-first search: the LLM writes the first 4, and each later round fires the one search — a next page, or a search composed from the qualified profiles' keywords — that one draw of the belief puts first among the ten worth the most at the mean (Thompson sampling), on a belief per keyword and per fired search measured by the people searches brought, five of whom are judged at random per search.
- **Default discovery** is a licensed source — BetterContact **Lead Finder** (ICP search returns firmographic profiles, no emails, billed nothing). The walk is a best-first search: the LLM writes the first 4, and each later round fires the one search — a next page, or a search composed from the qualified profiles' keywords — that one draw of the belief puts first among the ten worth the most at the mean (Thompson sampling), on a belief per keyword and per fired search measured by the people searches brought, five of whom are judged at random per search.
- **Qualification** is the crown jewel — **GPR + BALD active learning** over 384-dim embeddings, decided by an LLM, which also **writes down why**. That reason is the product.
- **Enrichment** is the one paid step — a **finder** resolves a work email for the top-ranked leads only (one credit per verified hit), fronted by a free cross-operator cache (the hub). Optional: a lead with no address still exports. **The resolver sits behind a seam** (`enrichment/provider.py`), which picks a finder from whichever key is configured — BetterContact (async waterfall) is the only one upstream ships, and the seam's `Lookup` still spans a synchronous finder so a fork can add one without touching the pipeline. **Only the resolver leg is behind it** — discovery is BetterContact's Lead Finder and nothing else.
- **Email enrichment** is a separate paid step — a **finder** resolves a work email for the top-ranked leads only (one credit per verified hit), fronted by a free cross-operator cache (the hub). Optional: a lead with no address still exports. **The resolver sits behind a seam** (`enrichment/provider.py`), which picks a finder from whichever key is configured — BetterContact (async waterfall) is the only one upstream ships, and the seam's `Lookup` still spans a synchronous finder so a fork can add one without touching the pipeline. **Only the resolver leg is behind that seam.** Default discovery is BetterContact Lead Finder; an explicitly selected CompanyEnrich page uses the separate `core/discovery_source.py` seam.
- **The output is the command's own stdout.** `outfind find 10 emails > leads.csv` prints **every lead in the store** in the column names Instantly and Smartlead already read, so the newest file supersedes every earlier one. No export command, and no file the tool manages. See *The lead export* below.

Django + Django Admin own the CRM/ORM (models are this project's); pydantic-ai drives the LLM calls.
Expand All @@ -37,6 +37,37 @@ Three consequences worth knowing before changing anything:
2. **The opt-out duty belongs to whoever sends.** A finder that contacts nobody is not the sender under CAN-SPAM/GDPR/CASL. `Lead.disqualified` is the export's exclusion filter; nothing sets it automatically. The hub store's own suppression is separate and unaffected.
3. **The GP never trains on a market signal.** `get_labeled_arrays` reads the LLM's fit verdict alone — no reply or bounce data ever enters that loop, by construction, since OpenOutFind never sees a reply or a bounce.

## Optional paid CompanyEnrich discovery

`find --companyenrich-query PATH --companyenrich-request-id ID --discovery-credit-budget N`
selects one bounded people-search page. All three flags are required; a configured
`OPENOUTFIND_COMPANYENRICH_API_KEY` alone is inert. `N >= 2 * pageSize` reserves the
maximum at the current published 2-credit/profile rate, independently of `--emails`.
No automatic next page, compound company lookup, expansion or free-walk fallback.
The default bare `find N` retains its free BetterContact discovery contract.

`core/discovery_source.py` scopes selection to one command and restores it on failure.
`companyenrich.py` validates the query against the checked-in official schema subset,
makes one bounded/no-retry/no-redirect HTTPS request and adapts current professional
profiles. `core/pipeline/companyenrich.py` commits `CompanyEnrichPage` before HTTP,
stores the full response before embedding, and reuses it across resumed invocations.
An ambiguous outcome never automatically replays; a different query cannot reuse its
ID. The paid source does not fabricate free lattice searches or change their estimates.
The generated `0013` merge migration preserves both the paid-request reservation history
and upstream market/keyword guess history; never replay a reservation during an upgrade.

Persist `Lead.discovery_source=companyenrich` even for previously known URLs. Do not
clear it on later BetterContact discovery: hub `resolve` and `contribute` must stay
blocked, and raw CompanyEnrich pages/companies never go to `share_*`. Existing LLM
qualification and operator-directed exports still consume these profiles. CLI JSON
metadata adds `discovery`; export columns and error vocabulary remain unchanged.
See [the complete contract](docs/companyenrich.md).
Later free-provider pages also filter identities already marked CompanyEnrich before
sharing raw profiles or employers; alternate host/trailing-slash/query spellings are
matched conservatively. A new source cannot undo the persisted local-only choice.
Keep these flags and paid/local
boundaries synchronized in README, ARCHITECTURE and `skills/find-leads/SKILL.md`.

## Commands

```bash
Expand Down
3 changes: 2 additions & 1 deletion LEGAL_NOTICE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ OpenOutFind is a self-hosted, open-source **lead finder**. It discovers B2B lead
> This notice describes how the software behaves and is **not legal advice**. You are responsible for your own compliance; where the stakes warrant it, consult a lawyer. Material aspects of the data model below are still pending a formal legal review.

### 1. No Platform Scraping or Automation
OpenOutFind performs **no** automated access to any social or professional network — no login, no browser session, no bot, no scraping, no messaging on such a platform. Lead **discovery** comes from a licensed data provider (currently BetterContact **Lead Finder**), and **enrichment** (resolving a work email) comes from a paid email-finder — both third-party services **you** sign up for and configure with **your own** API key, used under **that provider's** terms.
OpenOutFind performs **no** automated access to any social or professional network — no login, no browser session, no bot, no scraping, no messaging on such a platform. Lead **discovery** comes from a licensed data provider (BetterContact **Lead Finder** by default, or explicitly selected paid CompanyEnrich discovery), and **enrichment** (resolving a work email) comes from a paid email-finder — both third-party services **you** sign up for and configure with **your own** API key, used under **that provider's** terms.

- **Profile URLs are identifiers, not fetch targets.** A discovered lead may carry a professional-network profile URL as an opaque identifier. OpenOutFind **stores it and never visits it** — it is a lookup/dedup key, nothing more.
- **You accept the third-party terms.** You are responsible for using the data provider and email-finder in line with each provider's terms of service and acceptable-use policy.
Expand Down Expand Up @@ -51,6 +51,7 @@ Address resolution runs through a **third-party email-finder** (e.g. BetterConta
### 6. Central Contacts Store (Contribution and Resolution)
OpenOutFind connects to a **central contacts store operated by the project maintainer** (`hub.openoutreach.app`). It pools three things across the OpenOutreach network: work email addresses, so a contact one operator has already paid to resolve can be served — for free — to another; the professional profiles discovery returns, so the pool of people outlives the free index they came from; and the company records those results carry, from which the maintainer builds a company index. By running the software you participate as described here.

- **CompanyEnrich exception:** CompanyEnrich discovery pages, employers and embeddings are not contributed to the central store. Persisted CompanyEnrich identities also bypass hub resolution, later email give-back and sharing from later free-provider pages. The operator's existing LLM qualification and directed exports still apply. See [the source contract](docs/companyenrich.md).
- **Resolved addresses — what is sent, and when**: **after a paid email-finder returns a verified work email**, OpenOutFind sends the person's **profile identifier** (the stored, never-fetched profile URL), the **country code** searched, and the **work email address(es)** resolved.
- **Discovery profiles — what is sent, and when**: **every page of results a Lead Finder search returns** is sent as one batch, including people your campaign discards and people you already hold. Each row carries the **profile identifier**, the **country code** searched, and the row's professional fields as the provider returned them — **name, headline, job title, seniority, industry, state/region and country, employer name and domain, employer industry** — plus a **384-dimension numeric vector** computed locally from those fields and the name of the model that produced it. The employer's free-text **description and keywords are never sent with the person**: the provider fuzzy-matches them and they often describe a different company. **No email address is part of this record, and it earns no store credit.**
- **Company records — what is sent, and when**: with every such page, OpenOutFind also sends **one record per distinct company** the page carried that has a LinkedIn company page: that **page's URL**, and the company's **name, domain, industry, keywords, description, LinkedIn size band and headquarters city and country** as the provider returned them, plus a **384-dimension numeric vector** computed locally from the company's name, industry, keywords and description, and the name of the model that produced it. **No person is attached to a company record** — not who was employed there, not the search that found it — and it earns no store credit. The same gates apply as for discovery profiles: nothing is sent from an EEA/UK/CH-based operator or for a page searched there. The maintainer keeps these records in a **private company index** for its own use; the store does not serve them to operators.
Expand Down
2 changes: 2 additions & 0 deletions PRIVACY_NOTICE.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ Data reaches the store from OpenOutreach operators, at two moments:
- **Email records** arrive **after an operator's paid email-finder lookup returns a verified work email.**
- **Profile records** arrive **whenever an operator's prospect search returns a page of results.** The source of those results is the operator's own account with the B2B data provider **BetterContact** (its Lead Finder search), which compiles publicly available professional information.

CompanyEnrich discovery is excluded from this shared store by the client, including later contributions for identities marked as originating from that source.

The maintainer does **not** scrape any website, buy data, or run searches of its own to populate the store; it is filled only by operators' own searches and lookups, subject to the geographic exclusion above.

## How data is used and disclosed
Expand Down
Loading