Offer protoc's language targets alongside protobuf-net's

README.md

@@ -1,10 +1,15 @@
 # protobuf-net.dev
 
 The site behind **[protobuf-net.dev](https://protobuf-net.dev)**: generate C# and
-VB.NET from `.proto` schemas, and pull apart raw protobuf payloads without a schema.
+VB.NET from `.proto` schemas, reach `protoc` for the languages Google's compiler provides, and
+pull apart raw protobuf payloads without a schema.
 
-Everything runs in the browser. There is no server, no API and no upload — schemas and payloads
-never leave the machine they were pasted on. The site is a folder of static files.
+The site is a folder of static files. C#, VB.NET and payload decoding run in the browser through
+WebAssembly, so a schema or payload pasted into either of those never leaves the machine it was
+pasted on. The `protoc` targets are the exception and cannot be otherwise — `protoc` is a native
+executable — so they are compiled by
+[protoc.protobuf-net.dev](https://github.com/protobuf-net/protoc.protobuf-net.dev), and only when
+the user picks one of those languages and presses Generate.
 
 This replaces the older ASP.NET-hosted site that lived in the
 [protobuf-net](https://github.com/protobuf-net/protobuf-net) repo under `src/protogen.site`.
@@ -40,6 +45,11 @@ npm test         # vitest, unit tests for the payload parsing
 npm run build    # wasm + typecheck + tests + bundle -> web/dist
 ```
 
+The `protoc` targets point at the live service by default. To develop against a local one, set
+`VITE_PROTOC_ENDPOINT` — `VITE_PROTOC_ENDPOINT=http://localhost:8787 npm run dev` alongside
+`wrangler dev` in the
+[service repo](https://github.com/protobuf-net/protoc.protobuf-net.dev).
+
 `npm run wasm` alone re-publishes the .NET side into `web/public/_framework`. That folder is
 generated and git-ignored; it is copied verbatim rather than bundled, because the .NET boot process
 resolves its own content-hashed filenames.
@@ -152,20 +162,42 @@ are live.
 
 ## Scope
 
-**C# and VB.NET only.** The old site also offered C++, Java, JavaScript, Objective-C, PHP, Python
-and Ruby by shelling out to `protoc` on the server. `protoc` is a native executable and cannot run
-client-side, so those targets are gone rather than quietly broken.
-
-Compiling `protoc` itself to WebAssembly was investigated and parked. There is no maintained build:
-`kwonoj/protobuf-wasm` and `mjz20/protobuf_wasm` patch the protobuf *runtime* for Emscripten and
-explicitly do not build the compiler, and the one recent attempt at the compiler proper
+**C# and VB.NET locally; everything else through `protoc`.** protobuf-net generates the first two,
+in the browser. C++, Java, Kotlin, Objective-C, PHP, Python, Ruby and Google's own C# come from
+`protoc`, which is a native executable and cannot run client-side, so they are compiled by
+[protoc.protobuf-net.dev](https://github.com/protobuf-net/protoc.protobuf-net.dev) — a Cloudflare
+Worker in front of a container. That repository documents the service; what matters here is the
+contract:
+
+- `web/src/protoc.ts` holds the target list, the endpoint and the response reader. Adding a target
+  protoc already supports means one entry there and one in the service's own table.
+- The service answers with the same `GenerateResponse` shape the WebAssembly path returns, so both
+  render through `applyResponse` in `web/src/schema.ts` and there is one code path for output,
+  diagnostics and the editor gutter.
+- Nothing is sent while typing. Choosing a `protoc` target generates once, because choosing it is
+  the decision to use it; after that the output is marked stale on edit and re-sent only when the
+  user presses Generate.
+- The service carries the same `.proto` corpus that `protobuf-net.Reflection` embeds, so imports of
+  `google/**` and `protobuf-net/**` resolve identically on both paths. Nothing about imports is
+  sent from here.
+
+Compiling `protoc` itself to WebAssembly — which would have avoided the server altogether — was
+investigated and parked. There is no maintained build: `kwonoj/protobuf-wasm` and
+`mjz20/protobuf_wasm` patch the protobuf *runtime* for Emscripten and explicitly do not build the
+compiler, and the one recent attempt at the compiler proper
 ([protobuf#20819](https://github.com/protocolbuffers/protobuf/issues/20819)) produces a `protoc.js`
 that builds but fails code generation on path resolution. The real work is MEMFS plumbing for
 `--proto_path` and generated outputs, unmaintained upstream, on top of a C++ binary that links
-every language generator and would likely dwarf the .NET runtime.
+every language generator and would likely dwarf the .NET runtime. If that ever lands, the service
+becomes unnecessary rather than merely unfortunate.
+
+JavaScript and TypeScript are on neither list: `protoc` dropped its JavaScript generator in v21.
+`protobufjs` and `@bufbuild/protobuf` generate both in pure JS, so that one belongs back in the
+browser if it is ever worth having.
 
-If a target language is ever worth restoring, JavaScript/TypeScript is the cheap one: `protobufjs`
-and `@bufbuild/protobuf` generate it in pure JS with no `protoc` involved.
+PHP output is not syntax-highlighted. `@codemirror/legacy-modes` has no PHP mode, and
+`@codemirror/lang-php` brings the HTML, CSS and JavaScript parsers with it — too much weight for
+colouring output that is read once and copied.
 
 See [`docs/trimming.md`](docs/trimming.md) for why the build suppresses `IL2104`.
 

web/index.html

@@ -6,7 +6,7 @@
     <title>protobuf-net — protobuf schema codegen and payload analysis</title>
     <meta
       name="description"
-      content="Generate C# and VB.NET from .proto schemas, and pull apart raw protobuf payloads without a schema. Runs entirely in your browser."
+      content="Generate C# and VB.NET from .proto schemas in your browser, reach protoc for C++, Java, Python and the rest, and pull apart raw protobuf payloads without a schema."
     />
     <link rel="icon" href="/protobuf-net-mark.svg" type="image/svg+xml" />
   </head>
@@ -21,7 +21,8 @@
         <a href="#decode" data-view="decode">Decode payload</a>
       </nav>
       <div class="header-right">
-        <span class="privacy" title="Nothing you paste leaves your browser">runs locally</span>
+        <!-- says where the current target runs; schema.ts flips it when a protoc target is picked -->
+        <span class="privacy" id="privacy-badge" title="Nothing you paste leaves your browser">runs locally</span>
         <a href="https://docs.protobuf-net.dev/" rel="noopener">Docs</a>
         <a href="https://github.com/protobuf-net/protobuf-net" rel="noopener">GitHub</a>
       </div>
@@ -34,29 +35,39 @@
           <label>
             Language
             <select name="language" id="opt-language">
-              <option value="csharp">C#</option>
-              <option value="vb">VB.NET</option>
+              <optgroup label="protobuf-net — in your browser">
+                <option value="csharp">C#</option>
+                <option value="vb">VB.NET</option>
+              </optgroup>
+              <!-- filled from PROTOC_LANGUAGES, so the picker and the service cannot disagree -->
+              <optgroup label="protoc — on protoc.protobuf-net.dev" id="opt-language-protoc"></optgroup>
             </select>
           </label>
-          <label id="opt-langver-wrap">
+          <button type="button" class="ghost remote-only" id="run-protoc">Generate</button>
+          <span class="note remote-only">
+            compiled by <code>protoc</code> on
+            <a href="https://protoc.protobuf-net.dev" rel="noopener">protoc.protobuf-net.dev</a> —
+            your schema is sent there when you press Generate
+          </span>
+          <label id="opt-langver-wrap" class="net-only">
             Version
             <select name="languageVersion" id="opt-langver"></select>
           </label>
-          <label>
+          <label class="net-only">
             Naming
             <select name="namingConvention" id="opt-naming">
               <option value="auto">Auto (.NET conventions)</option>
               <option value="noplural">Auto, no pluralising</option>
               <option value="original">Original (as written)</option>
             </select>
           </label>
-          <label class="check"><input type="checkbox" name="services" checked /> services</label>
-          <label class="check"><input type="checkbox" name="oneOfEnum" /> <code>oneof</code> as enum</label>
-          <label class="check"><input type="checkbox" name="listSet" /> <code>repeated</code> with setters</label>
-          <label class="check"><input type="checkbox" name="repeatedAsList" /> <code>repeated</code> as list</label>
-          <label class="check"><input type="checkbox" name="nullableValueType" /> nullable value types</label>
-          <label class="check"><input type="checkbox" name="disableNullWrappers" /> no null wrappers</label>
-          <label class="check"><input type="checkbox" name="disableCompatLevel" /> no <code>CompatibilityLevel</code></label>
+          <label class="check net-only"><input type="checkbox" name="services" checked /> services</label>
+          <label class="check net-only"><input type="checkbox" name="oneOfEnum" /> <code>oneof</code> as enum</label>
+          <label class="check net-only"><input type="checkbox" name="listSet" /> <code>repeated</code> with setters</label>
+          <label class="check net-only"><input type="checkbox" name="repeatedAsList" /> <code>repeated</code> as list</label>
+          <label class="check net-only"><input type="checkbox" name="nullableValueType" /> nullable value types</label>
+          <label class="check net-only"><input type="checkbox" name="disableNullWrappers" /> no null wrappers</label>
+          <label class="check net-only"><input type="checkbox" name="disableCompatLevel" /> no <code>CompatibilityLevel</code></label>
         </form>
 
         <div class="split">
@@ -125,7 +136,9 @@
     <footer class="app-footer">
       <span id="engine-version">loading engine…</span>
       <span>
-        Everything runs in your browser via WebAssembly — no payload or schema is uploaded.
+        C#, VB.NET and payload decoding run in your browser via WebAssembly and are never uploaded.
+        protoc's own languages are compiled on
+        <a href="https://protoc.protobuf-net.dev" rel="noopener">protoc.protobuf-net.dev</a>.
       </span>
     </footer>
 

web/src/editor.ts

@@ -6,14 +6,27 @@ import { StreamLanguage, syntaxHighlighting, bracketMatching } from '@codemirror
 import { highlightStyle } from './highlight';
 import { lintGutter, setDiagnostics, type Diagnostic } from '@codemirror/lint';
 import { protobuf } from './protobufMode';
-import { csharp } from '@codemirror/legacy-modes/mode/clike';
+import { csharp, cpp, java, kotlin, objectiveC } from '@codemirror/legacy-modes/mode/clike';
 import { vb } from '@codemirror/legacy-modes/mode/vb';
+import { python } from '@codemirror/legacy-modes/mode/python';
+import { ruby } from '@codemirror/legacy-modes/mode/ruby';
 import type { SchemaError } from './types';
 
+// protobuf-net generates the first two; the rest arrive from protoc and are read-only output like
+// any other, so they need a tokenizer and nothing else.
 const languages = {
   protobuf: StreamLanguage.define(protobuf),
   csharp: StreamLanguage.define(csharp),
   vb: StreamLanguage.define(vb),
+  cpp: StreamLanguage.define(cpp),
+  java: StreamLanguage.define(java),
+  kotlin: StreamLanguage.define(kotlin),
+  objectiveC: StreamLanguage.define(objectiveC),
+  // deliberately unhighlighted: @codemirror/legacy-modes has no PHP mode, and @codemirror/lang-php
+  // brings the HTML, CSS and JavaScript parsers with it to colour output nobody edits here
+  php: [] as Extension,
+  python: StreamLanguage.define(python),
+  ruby: StreamLanguage.define(ruby),
 } as const;
 
 export type LanguageName = keyof typeof languages;

web/src/protoc.test.ts

@@ -0,0 +1,59 @@
+import { describe, expect, it } from 'vitest';
+import { PROTOC_LANGUAGES, protocLanguage, protocValue, readResponse } from './protoc';
+
+describe('language values', () => {
+  it('round-trips every target', () => {
+    for (const language of PROTOC_LANGUAGES) {
+      expect(protocLanguage(protocValue(language))).toBe(language);
+    }
+  });
+
+  it('does not claim the local generators', () => {
+    // protoc has a csharp generator too, which is exactly why the values are namespaced
+    expect(protocLanguage('csharp')).toBeUndefined();
+    expect(protocLanguage('vb')).toBeUndefined();
+  });
+
+  it('ignores a target it does not offer', () => {
+    expect(protocLanguage('protoc:fortran')).toBeUndefined();
+  });
+});
+
+describe('readResponse', () => {
+  it('reads a successful generation', () => {
+    const response = readResponse(
+      200,
+      JSON.stringify({ files: [{ name: 'my_pb2.py', text: 'x = 1' }], errors: [] }),
+    );
+    expect(response.files).toEqual([{ name: 'my_pb2.py', text: 'x = 1' }]);
+    expect(response.exception).toBeUndefined();
+  });
+
+  it('reads diagnostics, which are an answer rather than a failure', () => {
+    const response = readResponse(
+      200,
+      JSON.stringify({
+        files: [],
+        errors: [{ isError: true, lineNumber: 2, columnNumber: 3, message: 'Expected ";".' }],
+      }),
+    );
+    expect(response.errors).toHaveLength(1);
+    expect(response.exception).toBeUndefined();
+  });
+
+  it('keeps an exception the service reports', () => {
+    const response = readResponse(429, JSON.stringify({ files: [], errors: [], exception: 'too many requests' }));
+    expect(response.exception).toBe('too many requests');
+  });
+
+  it('survives a reply that is not the service at all', () => {
+    const response = readResponse(502, '<html>Bad gateway</html>');
+    expect(response.files).toEqual([]);
+    expect(response.exception).toContain('502');
+    expect(response.exception).toContain('Bad gateway');
+  });
+
+  it('survives an empty body', () => {
+    expect(readResponse(504, '').exception).toContain('504');
+  });
+});

web/src/protoc.ts

@@ -0,0 +1,93 @@
+import type { GenerateResponse } from './types';
+import type { LanguageName } from './editor';
+
+/**
+ * The language targets that come from Google's compiler rather than from protobuf-net.
+ *
+ * protoc is a native executable, so unlike everything else on this site these cannot run in the
+ * browser: the schema is posted to protoc.protobuf-net.dev, which runs protoc in a container and
+ * hands back what it wrote. That service carries the same .proto corpus protobuf-net.Reflection
+ * embeds, so an import that resolves locally resolves there too.
+ *
+ * See https://github.com/protobuf-net/protoc.protobuf-net.dev.
+ */
+const ENDPOINT: string = import.meta.env['VITE_PROTOC_ENDPOINT'] ?? 'https://protoc.protobuf-net.dev';
+
+/** The single virtual file the service compiles; it names the diagnostics that come back, too. */
+export const SCHEMA_FILE = 'my.proto';
+
+/** Namespaces the picker's values: protoc's `csharp` is not protobuf-net's `csharp`. */
+const PREFIX = 'protoc:';
+
+export interface ProtocLanguage {
+  /** what the service calls it */
+  id: string;
+  /** what the picker shows */
+  label: string;
+  /** how the output pane highlights it */
+  editor: LanguageName;
+}
+
+export const PROTOC_LANGUAGES: readonly ProtocLanguage[] = [
+  { id: 'cpp', label: 'C++', editor: 'cpp' },
+  // Google's C# generator, which emits Google.Protobuf types — a genuinely different answer to
+  // the same question as the C# above it, and worth being able to compare against
+  { id: 'csharp', label: 'C# (Google.Protobuf)', editor: 'csharp' },
+  { id: 'java', label: 'Java', editor: 'java' },
+  { id: 'kotlin', label: 'Kotlin', editor: 'kotlin' },
+  { id: 'objc', label: 'Objective-C', editor: 'objectiveC' },
+  { id: 'php', label: 'PHP', editor: 'php' },
+  { id: 'python', label: 'Python', editor: 'python' },
+  { id: 'ruby', label: 'Ruby', editor: 'ruby' },
+];
+
+/** The picker value for a target, e.g. `protoc:cpp`. */
+export function protocValue(language: ProtocLanguage): string {
+  return PREFIX + language.id;
+}
+
+/** The target a picker value names, or undefined when it is one of the local generators. */
+export function protocLanguage(value: string): ProtocLanguage | undefined {
+  if (!value.startsWith(PREFIX)) return undefined;
+  const id = value.slice(PREFIX.length);
+  return PROTOC_LANGUAGES.find((language) => language.id === id);
+}
+
+/**
+ * Reads the service's answer. It replies with a GenerateResponse whether the schema compiled or
+ * not — a schema that does not compile is an answer, not a failure — but anything between here
+ * and there (a cold start that times out, a proxy, an outage) will reply with something else
+ * entirely, so the body is not assumed to be JSON.
+ */
+export function readResponse(status: number, body: string): GenerateResponse {
+  let parsed: Partial<GenerateResponse> | undefined;
+  try {
+    parsed = JSON.parse(body) as Partial<GenerateResponse>;
+  } catch {
+    parsed = undefined;
+  }
+
+  if (!parsed || !Array.isArray(parsed.files) || !Array.isArray(parsed.errors)) {
+    const detail = body.trim().slice(0, 200) || '(no detail)';
+    return { files: [], errors: [], exception: `protoc.protobuf-net.dev replied ${status}: ${detail}` };
+  }
+  return { files: parsed.files, errors: parsed.errors, ...(parsed.exception ? { exception: parsed.exception } : {}) };
+}
+
+export async function generateWithProtoc(
+  language: ProtocLanguage,
+  schema: string,
+  signal: AbortSignal,
+): Promise<GenerateResponse> {
+  const response = await fetch(`${ENDPOINT}/generate`, {
+    method: 'POST',
+    headers: { 'content-type': 'application/json' },
+    body: JSON.stringify({
+      language: language.id,
+      files: { [SCHEMA_FILE]: schema },
+      entry: [SCHEMA_FILE],
+    }),
+    signal,
+  });
+  return readResponse(response.status, await response.text());
+}