Package version
3.19.1
Package
lexical-graph
Python version
3.12.13
Operating System
macOS
Description
A document that fails to upload during S3 staging either hangs the run or is reported as staged. Neither raises.
_upload_batch polls until count == target_count and treats queue.Empty as "keep waiting", without checking whether the producer thread is alive. Three except Exception: log blocks can put that count out of reach:
_get_callback_fn puts nothing on the queue when future.result() raises_submit_proxy returns without releasing the semaphore, while _doc_publisher still runs count += 1 for a document it never submitted_doc_publisher keeps queue.put(count) inside the try, so a failure above it skips the put
What keeps this from firing today is _upload_doc swallowing its own S3 exception and returning None. The callback puts the None, the count advances, and accept() yields it and counts it in the "Finished writing N source documents" total. So the failed document is reported as written, and a retry has no idea it needs to be redone.
That makes the two a pair: stop swallowing in _upload_doc on its own and the silent success becomes a hang.
Expected: an upload failure raises, and a document that failed to write is not counted as staged.
This is not #418. Same symptom and same path, but that one was a forked worker inheriting a held BufferedWriter lock, and it is fixed. This came in with 4b8743c0.
Steps to reproduce
# 1. Producer dies -> the consumer never returns.
with patch.object(S3DocUploader, '_doc_publisher', side_effect=RuntimeError('boom')):
list(uploader.upload([doc])) # blocks indefinitely, no exception
# 2. The coupling. Against a working S3 stub, patch _upload_doc two ways:
# returns None (today) -> finished=True yielded=[None] <- failure reported as success
# raises -> finished=False yielded=[] <- hangsOther information
A fix would be one invariant: every submitted document puts exactly one item on the queue.
_get_callback_fn always puts, a failure marker on exception_submit_proxy releases the semaphore in a finally and reports whether it submitted_doc_publisher puts the count in a finally, catching BaseException_upload_batch breaks on queue.Empty when the producer is dead, and re-raises after join()_upload_doc stops swallowingaccept() excludes failures from its staged count
No test fails a document mid-batch, which is the gap that hid this.
Package version
3.19.1
Package
lexical-graph
Python version
3.12.13
Operating System
macOS
Description
A document that fails to upload during S3 staging either hangs the run or is reported as staged. Neither raises.
_upload_batchpolls untilcount == target_countand treatsqueue.Emptyas "keep waiting", without checking whether the producer thread is alive. Threeexcept Exception: logblocks can put that count out of reach:_get_callback_fnputs nothing on the queue whenfuture.result()raises_submit_proxyreturns without releasing the semaphore, while_doc_publisherstill runscount += 1for a document it never submitted_doc_publisherkeepsqueue.put(count)inside thetry, so a failure above it skips the putWhat keeps this from firing today is
_upload_docswallowing its own S3 exception and returningNone. The callback puts theNone, the count advances, andaccept()yields it and counts it in the "Finished writing N source documents" total. So the failed document is reported as written, and a retry has no idea it needs to be redone.That makes the two a pair: stop swallowing in
_upload_docon its own and the silent success becomes a hang.Expected: an upload failure raises, and a document that failed to write is not counted as staged.
This is not #418. Same symptom and same path, but that one was a forked worker inheriting a held
BufferedWriterlock, and it is fixed. This came in with4b8743c0.Steps to reproduce
Other information
A fix would be one invariant: every submitted document puts exactly one item on the queue.
_get_callback_fnalways puts, a failure marker on exception_submit_proxyreleases the semaphore in afinallyand reports whether it submitted_doc_publisherputs the count in afinally, catchingBaseException_upload_batchbreaks onqueue.Emptywhen the producer is dead, and re-raises afterjoin()_upload_docstops swallowingaccept()excludes failures from its staged countNo test fails a document mid-batch, which is the gap that hid this.