xql-toolchain

Offline tooling for XQL — the real validation, not a heuristic. The ANTLR4 grammar and the semantic tree-walk listener are extracted from the query web app's shipped JavaScript and run locally, so an unknown column, a bad function name or a malformed stage is reported with the same wording the product uses.

Three things come out of it, over one shared core:

	what	where
CLI	`xql-lint` — a query in, errors and warnings out, exit 1 on error	`cli/xql-lint.mjs`
Monaco plugin	highlighting, completion, hover, worker-backed validation	`lib/xql-lang.js`
Library	`createXql()` → `validate()` / `suggest()`	`lib/xql-core.mjs`

⚠️ Full validation needs intellisense data

The grammar is built in, so syntax checking always works. Resolving names — is this a real column, dataset or function? — needs a schema (xql-intellisense.json). Without one, every tool here silently sees half the picture:
$ node xql-lint.mjs query.xql          # no schema present
xql-lint: no tenant schema found — validating syntax only. See --help.
query.xql:4:1  error  missing ')' at '<EOF>'

$ node xql-lint.mjs query.xql          # xql-intellisense.json alongside it
query.xql:2:10  error  bogus_zzz is not a valid field      <-- only found with a schema
query.xql:3:19  error  missing  ')'
The schema in the releases is community-sourced. It is one snapshot from one tenant, so it may not list the datasets yours has, and a dataset it does list may have a different shape — different columns, different types. Findings against it are indicative, not authoritative: a "not a valid field" on a column your tenant really does have is a gap in the schema, not in your query.

For results you can trust, generate your own against your tenant:
crtxtool xql schema        # writes config/xql-schema.json + output/xql-intellisense.json
Either file works — point --schema at it, or drop it where the CLI looks (see below). No rebuild is needed to swap schemas.

Using it

The parser bundles and the schema are 10 MB of extracted vendor code and schema data, so they are not in this repository — they ship as release assets.

asset	contents
`xql-lint.mjs`	the CLI as one self-contained file — Node 18+, nothing else
`xql-monaco.zip`	`xql-lang.js`, its data files, the validation worker, both example pages
`xql-lib.zip`	the ESM library (`xql-core.mjs` + the parser bundle)
`xql-intellisense.json.gz`	a community-sourced schema to resolve names against — see the note above

CLI

gunzip xql-intellisense.json.gz
node xql-lint.mjs --schema xql-intellisense.json query.xql

query.xql:4:14  error  field is invalid - does not exist
query.xql:1:1   warning  datamodel is planned for deprecation ...

1 error, 1 warning

Put xql-intellisense.json next to xql-lint.mjs (or in ~/.config/xql-toolchain/, or point $XQL_SCHEMA at it) and --schema becomes unnecessary. It also reads stdin, takes several files at once, and accepts -e '<query>'. --json for machine-readable findings, --strict to fail on warnings too, --help for the rest.

Without a schema it still works, reporting syntax errors only — field and dataset names just cannot be resolved, and it says so on stderr rather than passing a partial check off as clean.

Monaco

Unzip xql-monaco.zip, serve it over HTTP (not file://), and open one of the two examples.

// highlighting only — needs xql-highlight.json (~175 KB)
const xql = XQLLang.registerHighlighting(monaco, highlightData);
monaco.editor.create(el, { language: xql.languageId, theme: xql.themes.dark });

// the full language — needs xql-data.json (~920 KB) and, for real validation, the worker
const xql = XQLLang.register(monaco, data, { suggest });

examples/html/highlight.html is the first of those, self-contained in one file. examples/html/xql-editor.html is the second: worker-backed real validation, a problems panel, context-aware completion and a theme toggle. Both load Monaco from a CDN — point MONACO_VS at a local vs folder to run offline.

Library

import { loadSchema, createXql } from "./xql-core.mjs";

const xql = createXql(await loadSchema(schemaJsonText));
xql.validate("dataset = xdr_data | limit 10");        // -> [] when clean
xql.suggest(text, { lineNumber: 1, column: 12 });     // caret-aware completions

loadSchema accepts a parsed object, a JSON string, or an http(s) URL, and takes the schema either wrapped as { reply: … } or bare.

Building it yourself

Building needs the sibling crtxtool CLI, which fetches the app's JS chunks and the schema over an authenticated session. There is no CI path for this — the inputs cannot live in a repository.

npm install
npm run build:all      # crtxtool's chunks -> extract -> build lib/ artifacts + dist/xql-lint.mjs
npm test               # regression corpus: valid/invalid queries + silent-throw detection
npm run serve          # http.server at :8000

npm run build skips the extraction step and rebuilds from extractor/raw/ as it stands — that is the usual loop. Re-extract (npm run extract) only when the XQL grammar changes; schema changes just need a fresh lib/xql-intellisense.json.

Open http://localhost:8000/examples/html/xql-editor.html — serve from the project root so the examples' ../../lib/… references resolve.

The demo site

npm run site -- --zip           # -> dist/site/ (9 MB) and dist/xql-site.zip (1.7 MB)
cd dist/site && python3 -m http.server 8100

Four static pages for any file host — this README as the overview, a CLI page, the live editor, and a syntax-highlighting page. It needs the query library as well as the usual build output:

cd ../crtxtool && ./crtxtool xql library      # -> output/xql-library.json

Every example in that library is loadable in the editor, colourisable on the highlighting page, and linted during the build so each one carries a verdict. The CLI page's terminal output is produced by running the CLI at build time, so it cannot drift. --repo-url <url> points the README's release links somewhere real; --crtxtool <dir> overrides where the library is looked for.

site/ holds authored templates only — the build refuses to run if anything else appears there, and copies the schema and lib/ artifacts into dist/site/ instead.

Cutting a release

npm run release -- v1.2.0                # crtxtool -> extract -> build -> test -> package -> gh
npm run release -- v1.2.0 --dry-run      # everything except publishing
npm run release -- v1.2.0 --skip-fetch   # reuse the chunks/schema already on disk

scripts/release.mjs locates crtxtool at ../crtxtool (override with --crtxtool or $CRTXTOOL_DIR), refreshes both inputs, rebuilds, aborts if the regression corpus fails or if the extractor could not resolve a discovery anchor, packages the four assets into dist/, and calls gh release create.

Layout

lib/          the runtime library
  xql-core.mjs             loadSchema + createXql — what the CLI and the worker share
  xql-semantic.mjs         holder/config glue driving the extracted listener verbatim
  xql-lang.js              Monaco plugin: registerHighlighting() and register()
  xql-validator.worker.js  Web Worker source
  xql-antlr.mjs ·  xql-validator.js  ·  xql-data.json  ·  xql-highlight.json
  xql-intellisense.json                                        (all built — gitignored)
cli/          xql-lint.mjs — the CLI (bundled to dist/xql-lint.mjs for release)
examples/     the two example pages + tests
site/         demo-site templates and CSS (authored only — no tenant data)
extractor/    rebuilds the gitignored artifacts from the app's shipped chunks
scripts/      build-cli.mjs, build-site.mjs, release.mjs

extractor/ is the interesting half. See extractor/EXTRACTION.md for how the extraction works and how to redo it when the frontend changes, and extractor/CRTXTOOL-SPEC.md for how crtxtool fetches the chunks.

Before you publish or share the artifacts

A schema describes one specific tenant — every dataset, preset, data model and column name it exposes. That is internal information. The community-sourced one in the releases is already out there; the one you generate with crtxtool xql schema is yours, and publishing it publishes your tenant's shape.
The parser bundles are extracted from a vendor's proprietary web application. They are not authored here and this repository's license does not cover them. Whether you may redistribute them is your call to make.

Hosting this on a private repository keeps both points moot.