[Bug] Index all for search skips PDFs when Codex is the AI provider

[HercoGC]

HealthLog version

1.37.11 (build ca88d8c1a9cb)

Deployment type

Self-hosted (Docker)

What happened?

A mixed set of 18 documents was uploaded: 17 PDFs and one JPEG. With ChatGPT Account (Codex OAuth) connected and automatic AI document reading enabled, HealthLog successfully generated an individual AI summary for every document. This confirms that authentication, AI consent, image input and the server-side PDF rasterisation path were working.

Running Index all for search indexed the JPEG but repeatedly left all 17 PDFs without a content index.

The v1.37.11 batch code in src/lib/jobs/document-content-index-backfill.ts builds its candidate MIME list from pick.pdfSupported. The Codex provider supports image input rather than native PDF input, so PDFs are filtered out before prepareVisionInput() can use the existing PDF-to-image path.

Expected behavior

Index all for search should accept a PDF when either:

• the selected provider supports PDFs natively; or
• HealthLog can rasterise the PDF and the provider supports image input.

The batch result should separately report indexed, skipped and failed documents, and rerunning it should remain idempotent.

Steps to reproduce

1. Connect ChatGPT Account (Codex) as the AI provider.
2. Enable document AI consent.
3. Upload at least one JPEG and one PDF.
4. Confirm that Read with AI or automatic AI reading can summarise the PDF successfully.
5. Open the Documents page and run Index all for search.
6. Observe that the JPEG gains a content index while the PDF remains unindexed.
7. Run the batch again and observe the same result.

Workaround used

Temporarily disable Read documents automatically with AI, open each unindexed PDF, and run Read with AI manually. This successfully read the tested PDF and changed the action to Read again, but it is a repetitive document-by-document workaround and does not repair the batch operation.

The workaround is also constrained by HealthLog's shared six-per-hour document-AI limit described below. Manually reading 17 PDFs therefore requires at least three separate allowance windows, and using Suggest details, Summarise, Show extracted text, structured extraction or another manual indexing action can consume the same allowance.

Additional finding: the six-per-hour limit is imposed by HealthLog

Investigation after filing confirmed that this is an application-owned limit, not a Docker, Apple or OpenAI/Codex rate limit.

In v1.37.11:

• src/lib/documents/ai-route-support.ts hard-codes DOCUMENT_AI_LIMIT_PER_HOUR = 6 and a one-hour window.
• Index, suggest and summary routes use the shared documents-inbound:<userId> bucket.
• The structured document-extraction route uses the same bucket and the same six/hour values.
• The gate runs before the provider call, so it applies regardless of whether the selected provider is Codex, a BYO API key or another provider.
• The values are not exposed as an environment or admin setting.
• The API sends X-RateLimit-Remaining and X-RateLimit-Reset, but the user-facing message only says Too many AI requests — try again in a few minutes. Depending on when the first request opened the bucket, the actual wait can be much longer than a few minutes.
• Some routes check the bucket before input preparation, so a request that fails before provider dispatch can still consume an allowance slot.

The limit itself appears intentional as a cost and abuse safeguard. The usability problem is that it is shared across several visible actions, not explained before the user starts a multi-document workflow, not configurable for a personal self-hosted instance, and does not expose the reset time in the UI.

Possible solution

• Build the candidate MIME list from effective transform capability: native PDF support or server PDF rasterisation plus provider image support.
• Allow prepareVisionInput() to rasterise PDFs for Codex instead of excluding them before that function runs.
• Return per-document terminal outcomes and separate batch counts for indexed, skipped and failed.
• Preserve idempotent reruns so existing content-index rows are not duplicated.
• Add tests for a mixed JPEG, text-layer PDF and scanned PDF corpus with CHATGPT_OAUTH.
• Display the shared remaining allowance and exact reset time using the rate-limit headers already returned by the API.
• Explain which actions share the document-AI allowance before a bulk workflow starts.
• Consider an operator-configurable limit for self-hosted instances while retaining six/hour as the safe default.
• Avoid consuming a slot until a provider call is actually attempted, or provide equivalent handling for pre-provider failures.

Relevant logs / screenshots

No document contents or private screenshots are attached. The safe observed result was one JPEG indexed and 17 PDFs still unindexed after repeated batch runs. The UI also displayed the generic AI rate-limit message after the shared allowance was exhausted.

Browser / OS (if applicable)

HealthLog web app on macOS 27 arm64.

Pre-flight checks

• I confirmed this is not a duplicate of existing issue [Feature] Import pathology/laboratory PDF reports into structured HealthLog data with AI analysis #487, which introduced pathology/PDF functionality but does not cover this Codex batch-indexing path.
• I redacted any secrets, tokens, personal data and document contents.

[MBombeck]

Live in v1.37.13, in three parts. Read-everything now includes PDFs whenever the provider accepts images: the pages go through the same rasterisation a single manual read already used, so a Codex-style setup no longer skips every PDF in the vault. The hourly limit is yours to set now, DOCUMENT_AI_LIMIT_PER_HOUR in the compose environment, default unchanged at 6. And the accounting is honest: the limit message names the actual time the next read frees up instead of a vague few minutes, and a read that fails before anything reaches the provider gives its slot back. Please re-run your index-all and tell me whether the PDFs go through. Refs #776

[HercoGC]

Thank you for the very quick and complete response, and for turning v1.37.13 around so quickly. I tested the batch fix with Codex. It created content indexes for 15 additional PDFs, so the new PDF rasterisation path is working in this deployment.

One supported PDF still remains unindexed. I initially suspected that it might be blank, but verified the source directly: it is a valid one-page, image-only scanned PDF with one embedded page image and no text layer. HealthLog has the document stored and its summary is ready, but it has no content index and no stored error reason. That makes it a useful residual case for the scanned-PDF path rather than an invalid/blank input.

There is also a smaller status issue: the batch UI displayed Indexing true document(s) instead of a numeric queued count.

No document contents, filename, health values or private logs are included here. I can provide sanitised structural diagnostics if helpful.

[MBombeck]

Thank you for the batch retest, fifteen new indexes is exactly what the rasterisation path should have produced. Two things from your comment go on the list for the next patch, and this issue stays open to carry them. The scanned one-page PDF that ends up with neither a content index nor a stored error reason is the more important one: whatever the scanned-PDF path stumbled over, silence is the wrong outcome, so at minimum it must record why. And the batch status showing "Indexing true document(s)" is a boolean landing in the count placeholder, which I will fix alongside. Sanitised structural diagnostics for that one PDF would indeed help if it is no trouble: page dimensions, the image codec inside it, and the PDF version are the three most likely suspects. Refs #776

[HercoGC]

Thank you for keeping this open and for looking into both residuals. I inspected the exact one-page PDF and kept the diagnostics structural only:

• PDF version: 1.4 (%PDF-1.4)
• Page count: 1
• Page dimensions: MediaBox and CropBox are both 595.20 × 841.92 pt, portrait A4 (approximately 210 × 297 mm)
• Embedded images: one image XObject
• Image dimensions: 2480 × 3508 px, reported at 300 × 300 dpi
• Colour/bit depth: /DeviceGray, 1 bit per component
• Image filter/codec: /CCITTFaxDecode
• Decode parameters: /Columns 2480, /Rows 3508, /K -1 — this identifies CCITT Group 4 two-dimensional compression
• Other structure: no text layer; the PDF is not encrypted and is not tagged

pdfinfo, pdfimages -list and direct inspection of the image XObject agree on these values. I have deliberately omitted the filename, document image/content, health information, hash and private logs.

[MBombeck]

That is exactly the diagnostic set I hoped for, thank you — and it names the culprit precisely. The embedded page image is CCITT Group 4 fax compression (1-bit DeviceGray, /K -1 two-dimensional coding), which is a classic stumbling block for PDF rasterisers: everything about the file is valid, but the renderer can fail on precisely this codec, and until now that failure evaporated without a trace.

Two things follow, both already in the patch being built. Every index attempt now records its outcome on the document, so a raster failure like this one shows up in the detail view with a plain-words reason instead of leaving the document indistinguishable from never-tried. And the rasteriser reports its failure reason explicitly, so the fax-compression class is named rather than swallowed. Whether this particular codec can also be made to render is a separate question I will look at on top of the honesty fix; recording the truth comes first. Refs #776

[MBombeck]

The honesty half is live in v1.37.15. Every index attempt now records its outcome on the document, and the detail view says in plain words why an attempt failed, with scanned fax-compression formats named as a known cause. An empty provider transcription can also no longer slip through as a successful empty index, and the reindex message reports the honest number of documents.

On your CCITT Group 4 file specifically: I probed the rasteriser against a synthetic G4 PDF matching your diagnostics, and it renders the page blank rather than failing, which is why the failure had been so silent. Recording the truth was the right first step; making that codec actually render is a separate piece of work I have on the list, and your structural diagnostics are what will make it possible. After updating, reading that document again should at least tell you honestly what happened. Refs #776