Dossier → OCR → analysis

End-to-end flow from "Nouveau dossier" to a polled AnalysisResult. Every arrow below is an actual HTTP boundary in the code; "SoR" = locus-backend.

Nouveau dossier → POST /api/locus/deals
Deal Room upload → SHA-256 → R2 Put → POST /api/documents
  → extractPages (native / harvest / Tesseract)
  → POST /api/documents/{id}/ocr-result → ExtractedPages + Citations
  → PersistFinancialExtract → FinancialStatements / ExtractedCells / SourceCitationId
Enqueue analysis → AnalysisOrchestrator → Python worker
  → Contradictions, Blockers, Alerts, RatioSnapshots
Deal GET → financial_periods + contradiction_alerts from SoR

1. Full sequence

sequenceDiagram
    autonumber
    actor A as Analyst
    participant UI as DealRoomExplorer (browser)
    participant BFF as locus-web BFF
    participant R2 as Cloudflare R2
    participant SoR as locus-backend
    participant W as locus-workers :8000
    participant Q as Job store (Redis)

    A->>UI: Nouveau dossier
    UI->>BFF: POST /api/locus/deals (CreateDealSchema)
    BFF->>SoR: POST /api/deals (Bearer session, dossier:create)
    SoR-->>BFF: 201 deal_* (status ORIGINATION)

    A->>UI: Upload PDF in Data Room
    UI->>BFF: POST /api/locus/documents/upload (multipart, ≤ 40 MB, dealroom:upload)
    BFF->>BFF: sha256Plaintext(bytes); AES-256-GCM pack if keyed
    BFF->>R2: PutObject tenants/{tenantId}/dossiers/{dossier}/{doc}/…
    BFF->>SoR: POST /api/documents {id, dossierId, s3Key, sha256Hash, …}
    alt same Sha256Hash on same dossier
        SoR-->>BFF: 409 DUPLICATE_DOCUMENT
        BFF-->>UI: 409
    else new
        SoR-->>BFF: 201 (OcrStatus=PENDING, audit DOCUMENT_UPLOAD)
    end
    BFF->>BFF: extractPagesWithEngine (native → harvest)
    opt harvest insufficient
        BFF->>W: POST /api/v1/ocr {filename, contentBase64}
        W-->>BFF: pages[] + words[] (x0,y0,x1,y1,confidence)
    end
    BFF->>SoR: POST /api/documents/{id}/ocr-result {pages[]}
    SoR->>SoR: replace ExtractedPages, Citations cite_{doc}_p{n}, SearchDocuments
    SoR->>W: POST /api/v1/extract-syscohada {text, periodLabel}
    SoR->>SoR: FinancialStatement + ExtractedCells + RatioSnapshot + Blockers
    SoR-->>BFF: {ocrStatus: COMPLETED, pageCount}
    BFF-->>UI: meta + downloadUrl /api/locus/documents/{id}/content

    A->>UI: Lancer l'analyse
    UI->>BFF: POST /api/locus/deals/{id}/analysis (dossier:write)
    BFF->>SoR: POST /api/deals/{id}/analysis
    SoR->>Q: save AnalysisJob (Queued, 5%)
    SoR-->>BFF: 202 + Location /api/analysisjobs/{jobId}
    BFF-->>UI: {jobId, status, pollUrl}
    loop poll
        UI->>BFF: GET /api/locus/analysisjobs/{jobId} (dossier:read)
        BFF->>SoR: GET /api/analysisjobs/{jobId}
        SoR-->>UI: {status, progressPercent, currentPhase, durationSeconds, result?}
    end
    SoR->>W: POST /api/v1/analyze (best-effort ping)
    SoR->>SoR: RunFullDossierAnalysisAsync → Postgres
    UI->>BFF: GET /api/locus/deals/{id}
    BFF->>SoR: GET /api/deals/{id}
    SoR-->>UI: financial_periods, contradiction_alerts

2. Endpoint ledger

Step	Layer	Endpoint	Permission	Result / error codes
Create dossier	BFF → SoR	POST /api/locus/deals → POST /api/deals	dossier:create	201; 400 VALIDATION_ERROR
Upload	BFF	POST /api/locus/documents/upload	dealroom:upload	201 meta; 503 R2_NOT_CONFIGURED / DOCUMENT_ENCRYPTION_REQUIRED; 400 > 40 MB; 409 duplicate
Register document	SoR	POST /api/documents	dealroom:upload	400 INVALID_SHA256, 400 INVALID_S3_KEY, 409 DUPLICATE_DOCUMENT
OCR escalation	Worker	POST /api/v1/ocr	internal	413 OCR_TOO_LARGE, 503 OCR_ENGINE_UNAVAILABLE
Persist OCR	SoR	POST /api/documents/{id}/ocr-result	dealroom:upload	400 OCR_UNREADABLE (OcrStatus=FAILED), else COMPLETED
Extract SYSCOHADA	Worker	POST /api/v1/extract-syscohada	internal	falls back to in-process ExtractCellsFromText when < 3 values or worker offline
Verify citation	BFF → SoR	POST /api/locus/documents/{id}/verify-citation	dealroom:read_approved	±5 px; 404 MISSING_DOCUMENT
Enqueue analysis	BFF → SoR	POST /api/locus/deals/{id}/analysis → POST /api/deals/{dealId}/analysis	dossier:write	202 + Location
Poll	BFF → SoR	GET /api/locus/analysisjobs/{jobId} → GET /api/analysisjobs/{jobId}	dossier:read	200 / 404
History	BFF → SoR	GET /api/locus/deals/{id}/jobs → GET /api/deals/{dealId}/jobs	dossier:read	list
Reconciliation	BFF → SoR	GET /api/locus/deals/{id}/reconciliation	financial:read	covenantCertificationBlocked
Cell override	BFF → SoR	POST /api/locus/deals/{id}/cells/{cellId}/override	financial:override	see OCR override SOP

3. Page extraction paths (ocr-ingest.ts)

Order	Path (ExtractResult.path)	Condition	isScanned	ocrConfidence
1	native	Non-binary / UTF-8 text without %PDF magic	false	0.97
2	scanned_harvest	Printable-string harvest from binary PDF ≥ 40 chars	true	0.85
3	tesseract	Harvest insufficient → runWorkerTesseractOcr → worker /api/v1/ocr (fra+eng)	true	worker mean confidence
—	OCR_UNREADABLE	all of the above fail → FAILED, request a clean copy	—	—

Upload enqueues DOCUMENT_OCR on the BullMQ task queue (when REDIS_URL set) so upload + hash stays inside the 2 s SLA; SoR persist runs in the background.

4. Analysis job lifecycle

JobStatus: Queued → Processing → Completed | Failed.

progressPercent	currentPhase
5	Queued in Redis Job Buffer
20	Loading OCR pages from PostgreSQL SoR
45	Dispatching HTTP Request to Python Analytical Worker
70	SYSCOHADA extract, ratios, contradictions, blockers, alerts → Postgres
100	Completed

RunFullDossierAnalysisAsync steps (engine locus-dossier-analytics 3.0.0):

1. A1 — for each financial-looking document: infer fiscal year → extract cells → PersistLineCitations → UpsertFinancialStatement → UpsertRatioSnapshot.
2. A2 — ScanAndPersistContradictionsAsync over OCR pages (see Contradictions).
3. B1 — UpsertBlockers against CompletenessRules.
4. B2 — EmitAlertsAsync from contradictions, covenant breaches, fatal blockers.
5. Term-sheet facility lines → workspace payload (review grids); RefreshWorkspacePayloadAsync.
6. Build the typed AnalysisResult (contract).

Failure modes: ANALYSIS_NO_OCR when the dossier has no OCR pages; TENANT_REQUIRED when the job has no org id; worker offline is not fatal.

Completeness rules (BLK-*):

Id	Missing document	Severity
BLK-FIN-01	SYSCOHADA financial statements	FATAL
BLK-LOAN-01	Loan / facility agreement	FATAL
BLK-TS-01	Syndication term sheet	MAJOR
BLK-LEGAL-01	Independent legal opinion	FATAL
BLK-CONC-01	Concession agreement (if applicable)	MAJOR

5. Persistence map

Table	Written by	Notes
Documents	POST /api/documents	S3Key, Sha256Hash (plaintext digest), OcrStatus
ExtractedPages	ocr-result	Id = pg_{docId}_{n}, WordsJson, OcrConfidence (default 0.9 if 0)
Citations	ocr-result + line match	page cite cite_{docId}_p{n}; line cite cite_{docId}_p{page}_{lineCode}
SearchDocuments	ocr-result	ocr_{docId}_p{n}, EntityType=document_page, target /workspace/deal-room?project=…
FinancialStatements	analytics	fs_{dossierId}_{year}, ActifPassifDelta, ReconciliationState
ExtractedCells	analytics	SourceCitationId null when unmatched
RatioSnapshots	analytics	ratio_{dossier}_{year}_{scenario}, FormulaId=STD_SYSCOHADA_V2
Contradictions	analytics	MARGIN_LEAKAGE / TENOR_MISMATCH
Blockers, alerts	analytics	BLK-*; alert categories CONTRADICTION, COVENANT, BLOCKER
AuditEvents	all	DOCUMENT_UPLOAD, DOCUMENT_OCR_PERSIST, CELL_OVERRIDE, …

6. Citations (R4)

• Page cites: cite_{docId}_p{n} from PersistOcr; BoundingBox is the first OCR word box, or [] when there are no words.
• Line cites when OCR words match taxonomy: cite_{docId}_p{page}_{lineCode} on cells.
• Contradiction alerts hydrate from Citation rows — no invented bboxes.
• UI: Voir source / Voir sur le PDF; highlight skipped if bbox unusable.

See Citations (R4).

7. Reading the result

GET /api/deals/{id} returns financial_periods (with source_citations) and contradiction_alerts hydrated from SoR rows. An empty tenant gets an empty deal — there is no demo overlay for authenticated tenants.

Override

Analyst cell override SOP: OCR override.