Dremio is now part of SAP
Dremio Blog

37 minute read · November 17, 2026

Testing Apache Ossie Semantic Models in CI

Testing Apache Ossie Semantic Models in CI
Copied to clipboard

Answer up front: Treat Apache Ossie models like code. A practical CI pipeline first validates the model document against the Ossie schema, then lints references and relationship keys, exercises metric expressions against representative fixtures and SQL dialects, checks converter round trips and backward compatibility, and gates publication on those results. This article provides an operational CI workflow, concrete jobs you can add, the failure modes you will hit, and a validation plan you can copy into your project.

TL;DR

  • Start with a schema-validation job using the official Ossie core-spec schema to reject malformed documents.
  • Add a reference and relationship linter to catch broken keys and invalid relationship directions.
  • Run fixture-based metric tests across the dialects you support; do not assume one dialect covers all SQL behavior.
  • Include backward-compatibility and round-trip conversion gates: spec version pins matter because Ossie is evolving.
  • Fail CI for any test that relies on vendor-specific SQL behavior; document where operator checks are required.

Why this matters: the Ossie specification defines the structure and intent of semantic models, but it does not and cannot guarantee engine behavior when expressions are evaluated. Validation prevents obvious publishing errors. Linting prevents broken model navigation. Tests with fixtures catch semantic regressions and dialect edge cases. Conversion and compatibility gates protect downstream consumers.

Background and assumptions

This article is grounded on the official project sources: the Apache Ossie website, the GitHub repository, and the core-spec document. See the Sources section for direct links to each. When I say "the Ossie spec" I mean the version your repository points to; the project is in active development and the spec can change between releases. Always pin the spec version you validate against.

Operational assumptions:

  • You have a repository with Ossie model files (YAML or JSON) and CI (GitHub Actions, GitLab CI, Jenkins, etc.).
  • Your CI environment can run a small container or script that uses the Ossie project validation utilities and a SQL engine or emulators for dialect testing.
  • You will separate source authorization from model authorization: Ossie models describe semantics, not privileges or data access controls.

High-level CI pipeline

Start with these stages in order. I recommend failing fast: if schema validation fails, stop and return a clear error that points to the spec sections involved.

  • Schema-validation job: validate against the core-spec schema referenced by your pinned version.
  • Static semantic checks: reference linter, relationship direction checks, duplicate key detection.
  • Metric fixture tests: execute metric SQL expressions against representative data fixtures across the dialects you support.
  • Converter round-trip and compatibility checks: convert model to target artifacts and back, ensure no semantic loss or version incompatibility.
  • Promotion gate: only allow publishing if all prior stages pass and human review signed off for changes that alter metrics or relationships.

Concrete CI jobs and implementation steps

Below are reproducible job outlines. I name example files, but these snippets are illustrative: adapt paths and commands to your CI and repository layout.

1) Schema-validation job

Purpose: reject malformed models before any later checks. Use the core-spec JSON Schema from the Ossie repo for the exact rules. The Ossie project includes a validation folder with utilities you can reuse. See the project validation directory for scripts and helpers.

Implementation steps

  1. Pin the spec version: add a file like ossie-spec-pin.txt with the commit hash or release tag that CI will fetch from the ossie GitHub repo.
  2. Checkout that version of the spec during CI, for example by cloning the Ossie repo and checking out the pinned ref. Confirm the presence of core-spec/spec.md and the validation schemas in validation/. See the project core spec for structure and required fields.
  3. Install a JSON Schema validator or use the Ossie validation utilities under validation/. If using a JSON Schema validator, feed the schema file and the model file(s) into the validator. Fail on any schema error.
  4. Output a machine-readable report (JSON) and a human-friendly summary that points to file names, JSON paths, and spec rules violated.

Illustrative command (adapt to your CI):

<!-- language: sh -->
# clone pinned spec
git clone https://github.com/apache/ossie.git /tmp/ossie
cd /tmp/ossie
git checkout $(cat /repo/ossie-spec-pin.txt)
# run a validator on model files in repository
ajv validate -s core-spec/schema.json -d /repo/models/*.json --errors=text -o /tmp/schema-report.json

Failure modes to expect

  • Missing required fields, such as model id, kind, or entity definitions. These should be surfaced with exact JSON path locations.
  • Type mismatches, e.g., a number supplied where a string was required.
  • Schema version drift: if the model declares a specVersion that CI is not validating against, fail and require the operator to update the spec pin.

2) Reference and relationship linter

Purpose: check internal integrity that the schema cannot statically guarantee, such as: references to other model files, relationship directions and keys, duplicate keys across the model surface, and name collisions that cause ambiguous navigation. The Ossie examples folder shows typical model layouts you can use as test cases.

Implementation steps

  1. Collect all model files that belong to the change set. For PRs, lint only changed files plus any referenced targets. For branch builds, lint the whole models directory.
  2. Build an index of entity ids, metric ids, dimension ids, and their file locations.
  3. For each reference in a model, confirm the referenced id exists and the target kind is compatible. For example, a metric referencing an entity must point to an entity id defined in the index.
  4. Validate relationship declarations: ensure foreign keys or relationship keys map to fields in the referenced entity, and verify the cardinality and direction fields make sense. If the model describes a join that depends on a composite key, ensure the key arrays match in length and type.
  5. Report warnings for questionable patterns: ambiguous names across namespaces, reuse of entity ids with slightly different structures, or relationships that reference non-indexed fields.

Illustrative pseudo-code for a linter routine

<!-- language: python -->
# Load all models
index = {}
for f in model_files:
    doc = load_yaml(f)
    for entity in doc.get('entities', []):
        if entity['id'] in index:
            report_error(f"duplicate entity id {entity['id']} in {f} and {index[entity['id']]}")
        index[entity['id']] = f
# Validate references
for f in model_files:
    doc = load_yaml(f)
    for ref in doc.get('references', []):
        if ref['target'] not in index:
            report_error(f"{f}: reference to unknown target {ref['target']}")

Failure modes to expect

  • References that point to entities removed in the same or a concurrent change.
  • Relationship keys that exist only in some environments because the source schema is not synchronized with the model.
  • Duplicate ids created by automated tooling renaming files without updating ids inside the documents.

3) Fixture-based metric tests and dialect coverage

OSSIE MODEL CI PIPELINE1Schema-validation job2Reference and relationship…3Fixture-based metric test4Backward-compatibility and…Each stage produces evidence that can stop the rollout before the next boundary.
Ossie model CI pipeline.

Purpose: execute metric expressions against small representative fixtures to validate expression semantics, null handling, grouping behavior, and dialect-specific SQL quirks. Important caution: schema validity does not prove metric correctness. Also, SQL expressions may need more than one dialect to catch differences in behavior. Tests should avoid asserting vendor-specific behavior as portable behavior.

Design goals for fixture tests

  • Keep fixtures small, deterministic, and representative of edge cases: null values, zero counts, duplicate keys, and time zone edge conditions if time-based metrics exist.
  • Define expected results explicitly in the test. The expected values are authoritative for that fixture and dialect combination.
  • Run the same expression across the set of dialects you support, compare results according to a normalization strategy (for example, numeric tolerance for floating point, canonical time zone for timestamps).

Implementation steps

  1. Create a fixtures directory with small CSVs or SQL DDL/INSERT scripts that populate temporary test tables. Include at least: base entity rows, a table for each join target, and tables that exercise nulls and duplicates.
  2. Choose a SQL engine or lightweight runner for each dialect. You can use SQLite for simple SQL, DuckDB for analytic SQL, and a containerized Postgres or MySQL for those dialects. The Ossie examples folder includes sample model files that can guide fixture shape.
  3. For each metric in the model, generate the SQL expression the model specifies. If your pipeline includes a converter that produces SQL for a dialect, use that converter artifact. Otherwise, map the metric expression to SQL within the test harness, documenting when the mapping is illustrative and requires operator verification.
  4. Execute the SQL against the fixture and compare the result to the expected value. Where dialect differences are expected, document the acceptable differences per-dialect. For example, integer division rounding differs across engines; do not assert a behavior as portable if it is engine-specific.
  5. Report failures with diffs that show the SQL run, expected value, actual value, and the fixture used.

Illustrative fixture structure

<!-- language: text -->
fixtures/
  customers.csv  # id, name, created_at
  orders.csv     # id, customer_id, amount, created_at
  metrics/
    order_count_fixture.sql
    revenue_sum_fixture.sql

Example failure mode: metric expression handles NULL differently in dialect A versus dialect B. If tests assume dialect A behavior, dialect B will fail CI and reveal that the model's expression is not portable. The correct action is either to document a required minimum dialect, change the expression to be portable, or add a conversion layer that adapts expressions per-dialect.

Converter round-trip and backward compatibility gate

Purpose: ensure converting a model into target artifacts and back does not lose semantics, and that changes remain compatible with prior consumers. The Ossie specification is evolving, so version-specific cautions apply: specify the converter version and the spec version you expect.

Implementation steps

  1. Define the conversion paths you need: for example, Ossie model -> SQL artifacts for engine X, or Ossie model -> Polaris API objects. Use the converter implementations you have. If you are using community converters, pin their versions. If you implement your own, document the mapping and include unit tests for the converter itself.
  2. Round-trip test: convert the model to the target format and then back to Ossie. Compare the original and the round-tripped model for semantic equivalence, not byte-for-byte equality. Define equivalence rules: id equality, same set of entities and metrics, and identical expression ASTs where possible.
  3. Backward compatibility test: for changes that claim to be backward compatible, apply the change and run downstream consumer tests against a saved snapshot of the older model to confirm no breakage. For incompatible changes, require a manual approval step and a migration plan in the PR description.
  4. Fail CI on any change that the round-trip test marks as losing semantics, or that breaks backward compatibility without an explicit approval marker in the PR.

Failure modes to expect

  • Converters that drop metadata fields not present in the target format. Preserve fields explicitly or surface them as warnings so operators can decide to retain them.
  • Expression normalization differences where one converter emits syntactically different but semantically equivalent SQL; equivalence tests must rely on AST or canonical forms, not raw text.
  • Spec drift: a model validates against the spec version used locally but the converter expects a different spec version. Enforce pinned versions across the pipeline to avoid subtle mismatches.

Decision table: when to fail CI, when to warn, when to require manual approval

Use this decision table as a quick policy to embed in your CI scripts or governance docs. These are operational recommendations, not specification mandates.

ConditionActionRationale
Schema validation errorFail CIModel is malformed and cannot be relied upon
Reference/relationship missing targetFail CINavigation and joins will break at runtime
Metric test fails on all dialectsFail CIMetric likely incorrect
Metric behaves differently across dialectsWarn, require explicit per-dialect expectation or portability noteSQL portability issues must be documented or fixed
Converter round-trip loses non-critical metadataWarnMetadata loss may be tolerable but must be documented
Change breaks backward compatibilityRequire manual approval and migration planDownstream consumers may break

Validation plan and operating metrics

Include these metrics in your CI dashboards so you can measure model quality over time.

  • Schema-pass rate: percent of PRs passing schema validation on first attempt.
  • Lint-fail rate: percent of PRs failing reference/relationship linter.
  • Metric regression rate: percent of PRs that introduce metric test failures versus baseline fixtures.
  • Dialect divergence count: number of metrics exhibiting dialect differences that are not documented as acceptable.
  • Round-trip delta count: number of fields or semantic constructs lost during conversion and round-trip.

Operational checks to run daily or weekly

  • Run a nightly job that validates the entire models directory against the pinned spec and reports new failures.
  • Run a weekly compatibility sweep that converts the canonical model set into downstream artifacts and executes smoke tests in dev clusters.
  • Maintain a changelog of spec pins and converter versions used by CI; update when you intentionally adopt a new spec version and run a compatibility campaign.

Version-specific cautions

The Ossie project is developing actively. The core-spec file in the ossie repository is the authoritative specification for any given commit or release. Always pin the spec in CI to a specific commit or release. If you see a new field in core-spec/spec.md that your current schema validator does not recognize, treat it as a spec version mismatch and update the pinned ref before accepting models that use the new field.

Related caution: converters and validation utilities in the project validation folder may change. When you upgrade a converter make sure to run a broad compatibility test and a conversion round-trip sweep. Do not assume newer is safe without explicit validation.

Failure modes and how to recover

Common failure modes, diagnosis steps, and remedies.

  • Schema errors: read the validator JSON paths, open the model in an editor that supports JSON Schema, fix missing/incorrect types, and rerun CI. If the schema seems outdated, verify the spec pin and update it in the repo with a justification.
  • Broken references: run the linter locally, search for the missing id, and either add the missing artifact or update the reference to a correct id. If the referenced model is intentionally removed, require a migration that updates consumers.
  • Dialect divergences: identify the SQL fragment that yields differing results. Create a minimized reproduction fixture and either adjust the metric expression for portability or add per-dialect adaptations in the converter. Document the decision in the PR.
  • Converter round-trip losses: run a field-level diff report. For dropped metadata, either modify the converter to preserve fields or change the model to avoid relying on unsupported fields. Flag the change for manual approval if downstream impact is unknown.
  • Backward compatibility breaks: revert or provide an upgrade path. Use the compatibility suite to identify which consumers break and generate migration scripts if required.

Validation checklist you can copy into CI

STATIC SEMANTIC VALIDATION LAYERSRisk 1Schema validity does not prove metric correctnessTest itRisk 2SQL expressions may need more than one dialectBound itRisk 3Source authorization remains separate from model authoriz…Monitor it
Static semantic validation layers.

Copy this as a checklist in your repo README or CI job descriptions so PR authors know what will run.

  1. Schema-validation job runs against pinned Ossie spec. Fail on any error.
  2. Reference and relationship linter runs. Fail on missing targets or duplicate ids.
  3. Fixture-based metric tests run on supported dialects with documented expectations. Fail if metric is incorrect across all dialects. Warn if dialect divergence is detected without documentation.
  4. Converter round-trip test runs and compares semantic equivalence. Fail on semantic loss.
  5. Backward compatibility check runs. Require manual approval for incompatible changes.
  6. Human review required for changes that modify metrics, relationships, or entity ids.

Designing Failure Injection and Negative Tests for Ossie Models

Answer up front: Injecting failure modes into CI gives you high confidence that schema validation and linting are meaningful, and it reveals brittle assumptions in converters, linters, and dialect tests. Below I list practical failure injections, how to implement them in each CI job, and what operational evidence to collect.

Why failure injection matters here

Schema validity does not prove metric correctness. A model that passes schema-validation but contains incorrect references, ambiguous relationships, or engine-specific SQL will slip into production. Purposeful negative tests exercise the precise weaknesses the mandatory cautions call out: expression portability, authorization boundaries, evolving specs, and vendor-specific assumptions.

Categories of failure injections

  • Schema regressions: missing required keys, wrong types, and extra properties that violate the spec in Ossie core-spec.
  • Reference breakage: metrics referencing non-existent dimensions or models, transitive reference chains that should resolve but do not.
  • Relationship contradictions: cycles where a DAG is required, or inconsistent cardinality declarations.
  • Dialect edge cases: SQL expressions that work in one engine but not another, use of vendor functions, or unsupported CAST semantics.
  • Converter and round-trip misbehavior: converters that drop options, rename keys, or reorder arrays in ways that break subsequent validation or downstream tooling.
  • Authorization and access simulation: tests that assume a source is authorized but the CI runner lacks rights; also tests that should fail when model-level auth is violated.

How to implement failure injections per CI job

Each existing CI job gets a few deterministic negative tests. Keep injections small and revertible, and tag them so humans can find the defect-inducing fixture quickly.

  • Schema-validation job: Add a fixture with a required property removed and a second fixture with the wrong value type. Assert the job fails with clear error codes. Use the spec in the Ossie core-spec repository as the authoritative validator rule set. If your validator accepts both fixtures, that is a false pass and should open a bug against the validator.
  • Reference and relationship linter: Include fixtures where a metric references a dimension with a typo, and a fixture that creates a cycle in relationships. The linter should fail deterministically and point to the model file and byte offset if possible.
  • Fixture-based metric tests: Provide a metric whose SQL uses a vendor function, plus a parallel fixture where the same metric is expressed with portable SQL where feasible. The dialect runner should fail on the vendor-specific variant for the dialect that does not support it, and pass the portable variant. Capture the SQL string emitted and the engine error to store as operational evidence.
  • Converter round-trip gate: Run a converter to translate from Ossie to an engine-specific format and back. Inject a fixture where an optional converter field is present. The round-trip should preserve the field; if not, the gate must fail and create a human-review ticket including the diff.

Make these negative fixtures part of the validation harness in the Ossie validation folder, which you can find under the project validation examples in the upstream repository. Use the structure and examples in the official Ossie validation and examples folders when possible, and pin to the spec version you are testing against. The Ossie organization on GitHub is the source of truth for validation behavior.

Concrete test fixtures and their intended signals

METRIC FIXTURE AND DIALECT TESTSDeclareversioned contractExecutecontrolled fixtureCompareexpected stateGateship or stopA failed result returns with evidence. It does not become an undocumented exception.
Metric fixture and dialect tests.

Answer up front: Build a minimal set of fixtures that exercise each decision boundary: schema shape, references, relationships, dialect features, and converter fidelity. Keep fixtures small, document their intent, and categorize them so CI can run subsets during quick pre-commit checks and full suites in nightly runs.

Fixture taxonomy

  • Minimal valid model: One model, one dimension, one metric that compiles across all target dialects. Purpose: baseline success signal.
  • Missing required field: Remove a required attribute from the minimal model. Purpose: validator rejects the model.
  • Wrong-type field: Provide a number where the spec requires a string. Purpose: type checking must fail.
  • Broken reference: Metric references a non-existent column. Purpose: linter detects unresolved references.
  • Relationship cycle: Two models reference each other as parents. Purpose: linter rejects cyclic relationships.
  • Vendor function example: Metric SQL uses a vendor-only function. Purpose: dialect runner shows which dialects reject it.
  • Converter-sensitive field: Include an optional field that some converters historically dropped. Purpose: round-trip detects loss.

Operational evidence to collect for each fixture

  • Validator output, including error codes and file/line where supported.
  • Linter diagnostics with a unique rule identifier and file context.
  • Dialect runner logs: emitted SQL, execution error text, stack traces when present, and engine version used.
  • Converter round-trip diff: before and after JSON/YAML canonicalized and a delta highlighting dropped or renamed keys.
  • Test runtime durations so you can detect performance regressions.

Store this evidence alongside the CI run artifact, and annotate failing runs with a human-readable explanation. Over time you will learn which errors are flaky versus deterministic; track that in your operating metrics (see the Validation plan and operating metrics section you already have).

Decision boundaries, gating timing, and CI job partitioning

Answer up front: Not everything needs to gate the mainline. Partition tests into pre-commit quick checks, merge-gate blockers, and nightly extended suites. Use the decision table in the existing content to map severity to action, then map test suites to pipeline stages.

Three-stage pipeline mapping

  • Pre-commit (fast): Schema-validation against the pinned spec version and linter checks for obvious reference typos. Target: under 10 seconds. These run locally or in fast CI and should only warn for non-fatal issues.
  • Merge-gate (medium): Full reference and relationship linter, converter round-trip checks for changed files, and dialect smoke tests against a single, canonical engine per dialect. Target: minutes. Fail the merge if any deterministic, spec-violating errors appear.
  • Nightly (slow): Exhaustive dialect matrix, extended fixtures, and flaky test re-runs. Target: hours. Failures create prioritized defects, but do not automatically revert merged changes; instead require manual rollback for high-severity uncovered regressions.

How to decide which tests fail CI vs warn

Use these rules when mapping a test to action. The decision table in your existing article gives a broad view. Operationalize it with the following boundaries:

  • Fail CI when a test indicates a violation of the pinned Ossie spec version, such as missing required keys per core-spec. The spec is your single source of truth for pass/fail decisions tied to structure.
  • Fail CI when a converter round-trip loses or renames fields that are marked persistent in the spec or in your integration contract.
  • Warn but do not fail when a dialect runner rejects vendor-specific SQL that you have explicitly marked as optional within the model metadata, provided there is a fallback portable expression. Record which dialects fail.
  • Use manual approval for changes that are intentionally vendor-specific but must be reviewed by a semantic model owner and a data platform operator, for example, when adding engine-only optimization hints.

Practical CI partitioning and caching strategies

MODEL PROMOTION AND ROLLBACKObservecapture evidenceCompareuse a baselineDiagnosefind the boundaryActchange one variablemeasured resultunexpected changesmallest safe responsenew baseline
Model promotion and rollback.

Runtime matters. Partition tests to exploit cache and parallelism.

  • Cache downloaded dialect images, engine binaries, and container layers between pipeline runs. Keep caches keyed by engine version and spec version.
  • Run schema validation and linters in parallel across changed files. Use path-based triggers to only validate modified models in merge requests.
  • Run converter round-trip only for models that touch converter-sensitive fields or when converters themselves change.
  • Use matrix builds for dialect tests but limit the matrix in merge gates to one representative engine per dialect family and run the full matrix in nightly builds.

These choices let you keep the merge-gate lean while still exercising full coverage overnight. Record which changes skipped which tests so you can audit risk after a regression.

Operational monitoring, alerting, and continuous validation

Answer up front: Treat CI test results as telemetry. Create dashboards for failure counts, flakiness, and time-to-fix. Combine that with converter diffs to prioritize engineering work. The following operational items are practical evidence you will need to keep validation useful.

Metrics to collect

  • Pass rate per test class (schema, linter, dialect, converter) per day and per week.
  • Flakiness rate: tests that fail intermittently across re-runs within a 24 hour window.
  • Median and 95th percentile durations for each job so you can identify regressions that slow developer feedback.
  • Time to remediation: mean time from first failure to merged fix or approved bypass.
  • Number of manual approvals triggered by model owners for vendor-specific behavior.

Alerting and escalation

Define alert thresholds and who to notify. For example:

  • Immediate pager for merge-gate failures of the full test matrix that block multiple teams for more than 30 minutes.
  • Daily digest for nightly-suite new failures with links to converter diffs and validator logs.
  • Slack channel alerts for flakiness spikes with an automated triage job that re-runs failed tests three times to separate flakes from deterministic failures.

Continuous validation and spec drift detection

Ossie is evolving, per the mandatory caution about evolving specs, so track spec drift.

Related Dremio resources (supporting guidance)

Practical recommendation

Start by adding the schema-validation job and the reference/relationship linter to your CI. Those two provide the largest early-value return: they stop the majority of accidental breaks at the source. Next add fixture-based metric tests for the small set of critical metrics your business relies on. Enforce manual review for any metric or relationship changes. Finally, schedule a periodic spec pin review so you intentionally adopt new Ossie spec releases with a compatibility campaign, not accidentally.

Keep learning

For a deeper treatment, download Apache Polaris: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.

Use Dremio's Apache Polaris guide as the evergreen reference for the wider architecture and terminology.

Try Dremio Cloud free for 30 days

Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.