The pipeline is currently being slowed most clearly by memory throttling and a shared budget-file lock. The eight model slots are not the same as eight continuously busy model requests.

This read-only snapshot was collected on 4 October 2026 at 06:49:56–06:50:03 UTC, 08:49:56–08:50:03 Europe/Budapest. The saved dashboard state is from 05:47:20 UTC, 62 minutes 43 seconds earlier. It says 91 ready, 799 processed and 2,127 candidates, with $5.51666002 committed. Those are the dashboard's last persisted numbers, not a fresh throughput measurement. The service was active with PID 1275030. No production changes or provider service calls were made for this audit.

The main findings:

- The process spent only 0.03 CPU seconds during a 5-second sample. Six threads, including the progress-writing main thread, waited on file locks. One thread was explicitly waiting on the kernel's memory.high handler. Six queued lock waiters mapped to this run's budget.lock. Memory was approximately 803 MiB against a 768 MiB soft limit. Full memory-pressure stalls averaged 66.45% over the preceding minute. This is direct evidence of current resource and lock pressure, not proof of a permanent deadlock.
- The process was actually down for 1 hour 48 minutes 7 seconds after the OOM kill at 01:17:45 UTC until restart at 03:05:52 UTC. Three earlier stop/restart gaps total another 4 minutes 29 seconds. The running service had automatic abnormal restart protection at this snapshot, but its current working set was still above the memory.high threshold.
- Candidate processing runs in batches of at most eight. Verification, scraping, model requests and local persistence occupy those same workers. The next batch waits for the slowest one. History refresh and Maps acquisition happen outside that batch and block new model work. The persisted peak is seven model calls, with no live slot-occupancy history from which to calculate an idle percentage.
- Five complete history snapshots and one interrupted snapshot took 41 minutes 11 seconds of recorded fetch intervals in total. A complete pass takes about seven minutes. Each complete pass rereads 261 pages, including old sent/manual email history. This work has no model charge, but it stops the production loop while it runs.
- The ledger has 793 completed email checks: 163 ok, 276 catch-all, 178 invalid and 176 unknown, plus three uncertain attempts. Only ok can pass the current rule. The current wrapper still scrapes companies whose email already failed, purely to keep review evidence. Those optional scrapes compete with productive work.
- Six of 119 completed model responses spent all 12,000 completion tokens on reasoning and returned no JSON. The visible TypeError is a consequence of parsing null content. This is a real model-output problem, but its 5.04% frequency does not explain the multi-hour slowdown by itself.
- Six candidate rows are held after “database is locked.” These are SQLite failures, separate from the shared budget.lock contention seen in the process sample.

There is one combined paid AI request per candidate. The four script names are layers of the same Python process. They do not launch four separate AI agents.

| Script | Actual responsibility in this run |
|---|---|
| new2000.py | Owns the 2000 target, $100 ledger, source priority, email-first branch, eight-candidate batches and ready count |
| scale1200.py | Supplies history exclusion logic, prepared request construction, cross-process model slots, paid-result recovery and final export helpers |
| fresh100.py | Supplies the source scraper, current category schema and Unicode quote recovery. Its old main function and position-specific human overrides are not executed by this wrapper |
| endtoend100.py | Supplies the model HTTP call, pricing/balance reads, dispatch/accounting boundary and base result validation |
| pilot100.py | Packages page IDs and validates field types, fragment shape, names and legal quotes |
| run_trial.py | Supplies free bounded website fetching/cache, low-level API calls, file locks, atomic saves and reservation calculation |
| production_pool.py | Supplies paginated history reads, normalized identity exclusions and MillionVerifier calls |
| review_server.py | Stores review rows/evidence in SQLite and exposes the existing review UI. It does not create these model prompts |
| check2/resource_guard.py | Checks memory/disk admission before a service start and records exits. It is not ongoing AI supervision and does not add model workers |

1. Open the durable run and reserve one producer

Resume the existing run, take run/source producer locks, preserve the older paid ledger, load the separate USD100 ledger, and confirm provider balances. Previously completed rows and paid responses are reused. Only review preparation is authorized.

Cost/reuse: No generation purchase in this stage. Local work and provider account reads.

Concurrency: One main process. Shared whole-ledger reads and writes serialize on budget.lock.

Checks: Run identity, cap and ledger identity must agree. Required provider access must exist. Duplicate paid dispatch is not allowed.

Code: new2000.py:368-386, new2000.py:156-194, new2000.py:37-74. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

2. Rebuild the do-not-use history

Read Instantly membership, blocklist, sent email, manual email, campaigns and lists, then the Notion CRM. Add prior review companies, prior paid/reserved identities and local lead-bank suppression. Connect aliases so a different email or domain for the same known company also stays excluded.

Cost/reuse: No model or paid verification call. API reads and local processing.

Concurrency: Serial main-loop barrier. Runs initially and when the oldest proof is at least one hour old, and again at final acceptance.

Checks: Every paginated snapshot must reach its end. Previous exclusions remain merged. The main loop does not feed another worker batch until this finishes.

Code: production_pool.py:114-169, scale1200.py:152-198, new2000.py:76-101, scale1200.py:473-490. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

3. Select unused source companies

The first launch loaded unused old raw pools and Hungarian lead-bank companies. The current wrapper prefers newly acquired Maps companies when the fresh queue is empty, then returns to older unused candidates after the fresh plan is exhausted. Construction and evidenced team/project/location mentions sort first. Those phrases are priority signals, not a measured company-size qualification.

Cost/reuse: Old raw records are reused free.

Concurrency: Selection and full candidate/exclusion scans are serial. Work is chosen in batches of at most eight.

Checks: Email/domain/name/company duplicates, all historical prepared/paid identities, unproved email-to-company identity and invalid domains are excluded before append.

Code: new2000.py:134-154, new2000.py:176-183, scale1200.py:209-222, new2000.py:361-417. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

4. Acquire more Maps records when the fresh queue is empty

Run the installed Google Places actor for one Hungarian query/city combination. It requests up to 200 places with websites, disables paid contact enrichment and rejects social-only, closed, non-Hungarian or previously used companies. The deterministic plan contains 225 query/city combinations.

Cost/reuse: Paid Apify actor, from the same USD100 total. Per-actor reservation USD1.61. Existing actor receipts and completed query folders are reused.

Concurrency: One actor at a time, polled every 15 seconds. Candidate-generation work does not overlap this acquisition function.

Checks: No blind actor retry if a start outcome is ambiguous. Record charges and retain unresolved reservations. Maximum dataset retrieval is 1000 records, and reaching that bound is rejected.

Code: new2000.py:28-31, new2000.py:260-267, new2000.py:287-329. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

5. Find a real contact on each Maps company website

Fetch the company homepage and observed contact/about links. Take a visible email from a matching company page, prefer its business domain and common role mailbox, and keep the page quote/hash proving the connection. A free-mail address is accepted only when it was shown on the company-owned page.

Cost/reuse: Free website HTTP requests. The resulting scrape cache is reused downstream.

Concurrency: Up to eight free website crawls. The actor batch must finish crawling and filtering before it appends any candidates and returns to model processing.

Checks: Usable same-company page, a visible syntactically valid email on that page, and no identity/history conflict. No guessed email address.

Code: new2000.py:273-285, new2000.py:330-336, fresh100.py:133-154, run_trial.py:341-422. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

6. Check email deliverability before buying current personalization

Send the exact candidate email to MillionVerifier once, or reuse this run's saved verification. Only a completed ok result without a provider error proceeds to the model path. Catch-all, invalid, unknown or uncertain addresses are held. The current implementation still scrapes held companies for review evidence even though their paid personalization will not be bought.

Cost/reuse: Paid verification credits, conservatively accounted at USD0.0039 each. Repeat reads of the saved verification do not buy another check. This is accounting allocation, not proof of a new cash purchase.

Concurrency: Inside the same pool of up to eight candidate workers, with 20-second provider verification and 45-second HTTP timeout.

Checks: Return email must match and provider status must be recognized. Held candidates do not count toward 2000.

Code: new2000.py:206-231, production_pool.py:410-421. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

7. Load current website evidence and build the one combined request

Recheck current exclusions, take the company source lock and reuse the run-local scrape or fetch it. Save full evidence, then give the model only bounded exact text with the company name and domain. A saved prepared request is immutable on resume.

Cost/reuse: Free local/cache/HTTP work. No payment if pages are unusable or input bounds fail.

Concurrency: Runs in the candidate worker before its model call. It is not a separate permanent scrape worker pool.

Checks: Scraper seeks up to three usable pages, with up to eight requests, a nominal 35-second crawl deadline and up to a 12-second fallback. Source page text is capped to 18000 characters per page for the model. The assembly code permits at most four evidence pages, but all 127 actual payloads had one to three. Full request hard limit is 100000 bytes. Timeouts are per operation, not a verified hard total wall-time bound.

Code: scale1200.py:436-466, run_trial.py:341-422, fresh100.py:133-154, pilot100.py:363-374. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

8. Make one combined Luna request

In one paid call, classify the business, write the short Hungarian activity fragment, choose a generic or evidence-backed personal greeting, choose cég or vállalkozás, and return exact supporting page quotes. The other Python files are imported helpers, not separate model agents or separate paid classification/name/legal calls.

Cost/reuse: Paid OpenRouter request using openai/gpt-6-luna, reasoning max, Azure only. Saved paid results are reused instead of bought again.

Concurrency: At most eight model calls through both a bounded semaphore and cross-process slot files. Those slots share the eight candidate workers. Having eight configured slots does not mean eight requests stay busy.

Checks: Reserve worst-case spend before dispatch. Response model, actual provider and actual cost must match. HTTP timeout is 120 seconds. Uncertain dispatch is held without automatic paid retry.

Code: scale1200.py:253-267, scale1200.py:345-388, endtoend100.py:457-550. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

9. Validate and normalize the returned row in Python

Require exactly the schema fields. Check the activity quote is present in the saved page, check the short fragment shape, and verify person and legal-form quotes. Repair page IDs, whitespace and Unicode representation only by recovering actual source text. If person evidence is weak, use Szép napot. If legal-form evidence is weak, use vállalkozás.

Cost/reuse: No second model judge and no extra paid refinement.

Concurrency: Local work and budget-ledger/SQLite writes inside each candidate worker.

Checks: The validator enforces schema, quote presence, fragment length and certain name/legal rules. Category choice, business identity interpretation and Hungarian naturalness still rely on the single model plus human review. It is not a 100% semantic quality proof. Six saved responses hit the entire 12000-token output cap with reasoning only, returned null content and became model-held TypeErrors.

Code: fresh100.py:95-116, endtoend100.py:219-278, pilot100.py:377-429, scale1200.py:403-434. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

10. Decide whether the row counts and save progress

For supported construction or other-service personalization, reuse the email receipt, check the current exclusion snapshot and write a separate eligibility receipt. Count the row only if it has a generation ID, nonempty personalization, an accepted category, ok email, matching email and the ready flags/receipt.

Cost/reuse: Reuses existing paid receipts. No email send or campaign import.

Concurrency: Every completed future asks the main thread to reload rows and the ledger and write progress. The next batch waits for the slowest worker in the current batch.

Checks: INSTANTLY_READY is an internal review/import-eligibility label. Rows still await human review. It does not prove an import, send, reply or sale. The final write can lag behind the finished model response.

Code: new2000.py:196-204, new2000.py:233-255, scale1200.py:471, new2000.py:418-428. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

11. Final target acceptance, when 2000 is actually reached

Force a fresh history scan, recheck eligible rows, require exactly 2000 distinct source-backed eligible companies, then create the final CSV and an immutable hashed target receipt.

Cost/reuse: Local validation and refreshed read-only history. Existing email results are reused by email_attempt.

Concurrency: Serial final acceptance.

Checks: Exact target, no duplicate identities/history conflicts, source quote present, domain and email receipts match. This stage has not completed at the audited snapshot.

Code: new2000.py:342-349, new2000.py:393-399, scale1200.py:492-516. All code references point to /home/matt/.local/share/get-leads-review/scripts, except resource_guard.py in the run check2 directory.

Measured time, with the limits kept explicit:

| Measurement | Observed result | What it includes |
|---|---:|---|
| Wall time from saved run start to audit | 7 h 13 m 48 s | Includes all downtime and parallel work |
| OOM outage | 1 h 48 m 7 s | Journal kill to service start |
| Other fully stopped intervals | 4 m 29 s | Does not include the four-minute stop timeout while the process still existed |
| Saved history fetch intervals | 41 m 11 s | Five complete scans and one interrupted scan, 1,541 page receipts |
| 16 actor executions | 13 m 49 s total, 40.114 s median | Provider startedAt to finishedAt, one actor at a time |
| Actor finish to local candidate append | 9 m 17 s total, 26.160 s median | Poll delay, download, exclusion rebuild, free crawl and append together |
| 795 email attempts with both timestamps | 2.048 s median, 20.202 s p90 | Dispatch to saved verification receipt |
| 113 valid model results | 19.751 s median, 39.844 s p90 | Dispatch through response accounting and validation, not pure API latency |
| Valid model results before 03:05 restart | 101 results, 18.194 s median, 54.205 s maximum | Same combined prompt/settings |
| Valid model results after 03:05 restart | 12 results, 320.389 s median, 3,334.668 s maximum | Large local/resource delays may be included |

These intervals are not an additive total breakdown. Per-company calls overlap, and dispatch-to-result includes local waiting and settlement. Scrape start/end, exact response arrival, per-lock waiting and complete slot utilization were not logged. No ETA is justified from these measurements. The final model dispatch found in the saved ledger was at 05:12:46 UTC. The last valid result timestamp was 05:40:11 UTC. That is consistent with the later local backlog and stale progress file.

Repeated work is not all repeated paid work. The same company scrape path is called for Maps email discovery and later personalization, but a matching run-local cache is reused. Already paid model results and email receipts are reused. Completed Maps query folders are skipped. The expensive repeated local work is rebuilding exclusion objects, rereading and rewriting the whole budget JSON under a lock, reloading review rows for progress, and performing new full hourly history snapshots. The saved task ledger was 6,279,407 bytes at collection. Individual worker copies and their allocation sizes were not profiled, so the exact cause of the working-set growth remains an inference from code plus the measured resource pressure.

The full current prompt is in prompts-verbatim.txt. prompt-settings.json contains the exact static payload object, response schema and hashes. The active settings are openai/gpt-6-luna, Max reasoning with reasoning text excluded, a 12,000 completion-token limit, Azure only, no provider fallback, and no explicit temperature. All 127 saved prepared requests have the same static system/user instructions, schema and settings. Their 127 full request hashes differ because the company input changes. Every saved request hash verified. Of those requests, 121 have a ledger reservation, 120 were dispatched and 119 have completed receipts. All 119 identify Luna and Azure. Six prepared requests have no ledger reservation yet, and one reservation has no dispatch timestamp.

The single request asks the model to classify the business, write one short Hungarian activity phrase, select an evidence-backed greeting, choose the business noun and quote the source. It gets only candidate ID, company name, domain and bounded website pages. It does not receive a complete company dossier or a second model's review. The 127 payloads contain 9 one-page, 20 two-page and 98 three-page inputs. The largest actual payload was 58,580 canonical bytes. Pages were capped at 18,000 text characters each.

The retained earlier wrapper hash is 10b9e487e3fdf0c68b841bd611ecaf356814234ecf8938567df7986743658926. The current wrapper hash is 1c31b19311d85428db194f772783072536b04b59b4a54a4cea8e01e40aae3432 and was started at 00:44:23 UTC. Its material change was fresh-Maps priority ahead of the remaining legacy candidates. Seventy-five model dispatch timestamps fall before that start and 45 at or after it, with one later pending reservation. This is a time partition, not a per-attempt script-hash attestation. Prepared files do not embed their script version, so earlier launch-time live edits cannot all be reconstructed byte-for-byte from retained source alone. The prompt inventory itself is complete for every saved prepared payload. The actual check1 receipt is 48 ready from 688 processed. The code comment “47/678” is an older observation and must not be used as the check1 snapshot.

Narrow future improvements, in order:

1. Profile allocations in an isolated replay, share immutable exclusion snapshots across workers, and replace repeated whole-ledger parse/rewrites with bounded transaction records or one ledger owner. Set resource limits from measured working set and host headroom. Do not merely add workers.

2. Preserve the existing receipt-aware recovery while first reducing the working set and confirming sustained progress under the actual resource limit.

3. Use separately bounded source, verification, evidence and model queues with a rolling executor, retaining the same eight-call model ceiling and exact budget/reservation rules.

4. Preserve the same history clearance while using a durable cursor/change watermark where supported, one shared immutable snapshot, and a final fresh reconciliation. Keep incomplete snapshots unusable.

5. Prioritize source/city combinations using verified contact yield and defer optional held-row website evidence. Retain existing scrape caches and deduplication.

6. Handle this finish reason explicitly and expose it in the review UI. Any output-limit/prompt/model tradeoff should be evaluated with a small authorized benchmark while preserving the requested Luna Max setting unless changed by the user. Do not automatically rebuy uncertain calls.

7. Serialize review writes or add a narrow retry of known local transactions with preserved paid receipts. Instrument the write queue and keep a single owner for each row.

No improvement above was applied. The source/stage audit supplies the complete funnel denominators. Current code behavior, archived runtime receipts and configured intent are kept separate here. Review readiness remains preparation for human review, not an import or sent campaign.

Evidence files are compact and contain no bulk prospect inputs: pipeline-audit.json embeds the source hashes and numeric evidence, while the private runtime directory holds process-profile.json, lock-map.json, journal-events.json, history-timing.json, acquisition-timing.json, attempt-timing.json, model-response-metadata.json and the exact source copies. The remote audit directory is /home/matt/.local/state/get-leads-review/audits/funnel-20261004/runtime.

The saved provider generation_time field is retained as raw metadata because its units were not stated in the receipt or rendered public API reference. All displayed seconds above come from actual timestamps. Public reference checked: [OpenRouter generation metadata reference](https://openrouter.ai/docs/api/api-reference/generations/get-generation).