Standardizing a format to stream results from a SPARQL query

[vemonet]

For large results sets some triplestores are streaming results (e.g. Qlever, MilleniumDB). For CONSTRUCT/DESCRIBE it is easy to handle you can just ask for ntriples.

But for SELECT they are streaming a huge application/sparql-results+json object. Parsing this object is not clean and requires hacks

Streaming results from an HTTP GET/POST request have became quite common lately, especially with LLM streaming responses. Which are all using a format, dubbed "OpenAI-compatible APIs", looking like Server-Sent Event (https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events), but with POST instead of GET because they are sending a complex JSON. The SSE specification is only enabling using GET, which is unfortunately disconnected from the reality of complex and long queries required on the modern web

We could describe a similar streaming format for SPARQL queries, where we stream bindings chunk by chunk, for example something along the lines:

event: sparql_results_head
data: {'vars': ['s', 'p', 'o']}

event: sparql_results_binding
data: {'s': {'type': 'uri', 'value': 'http://purl.uniprot.org/uniprot/O13227#SIP556AB4700293B270'}, 'p': {'type': 'uri', 'value': 'http://purl.uniprot.org/core/certain'}, 'o': {'datatype': 'http://www.w3.org/2001/XMLSchema#boolean', 'type': 'literal', 'value': 'false'}}

event: sparql_results_binding
data: {'s': {'type': 'uri', 'value': 'http://purl.uniprot.org/uniprot/O82081#SIPB8F70E02AB555160'}, 'p': {'type': 'uri', 'value': 'http://purl.uniprot.org/core/certain'}, 'o': {'datatype': 'http://www.w3.org/2001/XMLSchema#boolean', 'type': 'literal', 'value': 'false'}}

The following mime-type is used for streaming events: text/event-stream, we could reuse it, or have a special mime-type like application/sparql-results-stream+json (not sure what's the convention there)

Is there any discussions already around streaming standardization in SPARQL 1.2? I could not find anything

[rubensworks]

Streaming in general is certainly interesting (and deserves more attention IMO).
But no discussions are ongoing about this for SPARQL 1.2 (and with the WG finalizing things, I don't think it is feasible to include this for 1.2), but it could be for 1.3.

For reference, it is possible to parse SPARQL JSON in a streaming manner (e.g. https://github.com/rubensworks/sparqljson-parse.js, which is used in Comunica), but having something newline-delimited or similar would be easier indeed.

[JervenBolleman]

I also note that the RDF4j parsers for application/sparql-results+json (or the XML version) are streaming too.

Maybe something like automatic pagination could be an option?
e.g.

{ "head" : {"vars":["s","p","o"]},
{ "s": {"type": "uri", "value":"ex1" }, "p": {"type": "uri", "value":"ex1" }, "o": {"type": "uri", "value":"ex1" } },
{ "s": {"type": "uri", "value":"ex2" }, "p": {"type": "uri", "value":"ex2" }, "o": {"type": "uri", "value":"ex2" } },
{"tail":{"continue-at":"https://example.org/?query=...&offset=2&limit=2"}}

and then make a request for the url in continue-at

{ "head" : {"vars":["s","p","o"]},
{ "s": {"type": "uri", "value":"ex3" }, "p": {"type": "uri", "value":"ex3" }, "o": {"type": "uri", "value":"ex3" } },
{ "s": {"type": "uri", "value":"ex4" }, "p": {"type": "uri", "value":"ex4" }, "o": {"type": "uri", "value":"ex4" } },
{"tail":{"continue-at":"https://example.org/?query=...&offset=4&limit=2"}}

This would give bounded size json objects.

[afs]

Jena streams JSON results when the JSON comes in the right order and non-streaming otherwise. It appears that JSON/right order is the most common form; steam writers do this naturally, and "h" is before "r" so whole document writers usually do as well.

A simple step would be to advise stream writers to put "head" first.

newline-delimited

TSV is newline-delimited and streaming however this is also a problem. The receiver can't know if the results are truncated and that happened on a NL boundary.

application/sparql-results+xml is streaming and the XML schema means the head comes first.
So it also makes explicit if the connection is truncated -- having a trailer is useful.

As a general point, in HTTP/1.1 and HTTP/2 incomplete connections can cause blocking so both reader and writer need to be robust. (Less of a problem with HTTP/3.)

Using HTTP chunked transfer encoding is a possibility - the chunks are length-delimited and can have pagination header info.