Dremio is now part of SAP
Dremio Blog

37 minute read · November 17, 2026

Migrating Semantic Models with Apache Ossie Converters

Migrating Semantic Models with Apache Ossie Converters
Copied to clipboard

Apache Ossie converters create a hub-and-spoke migration path between semantic platforms. The hard part is not producing valid YAML. The hard part is proving that metric meaning, join behavior, SQL dialects, AI context, and unsupported vendor metadata survive the trip. This article gives a hands-on migration and interoperability guide for importing and exporting semantic models through Apache Ossie converters, with concrete steps, tests, and operational advice.

TL;DR

  • Use an inventory and portability classification before any converter run.
  • Convert to Ossie core only when you can map vendor features to the core spec; preserve vendor extras in extensions.
  • Run round-trip fidelity tests for metrics, joins, and SQL samples, not only YAML validation.
  • Do a parallel run and controlled cutover with measurements for numerical equivalence and query behavior.
  • Expect and plan for dialect fallback and unsupported metadata; do not assume lossless translation.

How Apache Ossie converters fit into a migration strategy

Answer, up front: Ossie converters let you import and export semantic models between supported vendors by converting vendor format into the Ossie interchange format, and vice versa. The supplied project readme and repository document the converters and the core spec. The practical work is proving that the converted model behaves the same when used for metrics, joins, and SQL generation in your environment.

Apache Ossie is a project with a core specification; converters are an implementation set that map vendor artifacts into that spec. The primary repository contains the converters code and the README that lists supported formats. The Ossie core specification defines the flat, single-document shape that the converters produce; that shape is one Ossie document per semantic model. See the project readme and the core spec for the authoritative descriptions.

Key practical cautions you must accept before starting: converters do not guarantee lossless translation for every vendor feature; one Ossie document represents one semantic model in the current flat shape; dialect fallback must be tested rather than assumed; unsupported metadata should be preserved in extensions instead of silently discarded; and migration must measure numerical equivalence as well as schema validity. These cautions come from the converter project documentation and the core spec, which I cite in the Sources section.

Inventory and portability classification

First step, inventory. You cannot migrate what you have not cataloged. Create a machine-readable inventory of:

  • Models and their identifiers (logical names, owners, tags)
  • Metrics, with definitions including aggregation type, filters, and expressions
  • Dimensions and hierarchies
  • Join definitions: left/right, cardinality assumptions, enforced keys
  • SQL-derived tables or custom SQL sources and their dialects
  • Vendor-specific metadata and extensions (annotations, access controls, UI hints)
  • Example queries and reports that must continue to produce the same numbers

Portability classification is a mapping from each inventory item to one of three buckets: Core portable, Portable with extension, Not portable. Be explicit about why an item is placed in each bucket.

  • Core portable: maps cleanly to Ossie core constructs as defined in the core spec. Example: a simple metric with sum of a numeric column and no vendor-only features.
  • Portable with extension: cannot be represented in the core fields but can be expressed as preserved metadata in a vendor extension section, or as an extension field as allowed by the spec. Example: vendor UI hints or security labels that Ossie core does not define but that you want to preserve for reimport into the original platform.
  • Not portable: cannot be expressed in Ossie without losing semantics, or is an implementation detail of a vendor engine that will have to be reimplemented. Example: proprietary in-memory aggregation hint that affects execution and not captured in model semantics.

Record the rationale. For each metric, state whether numerical equivalence is provable via expression translation, or whether differences must be expected because of dialect or function differences.

Import, validate, and normalize workflow

High level workflow: import vendor model into Ossie using the appropriate converter, run structural validation against the Ossie core spec, normalize the document (move vendor extras into an extension field rather than removing them), then run functional tests against a local copy of your query set. The converter repository README lists supported vendor converters and usage. The core spec is the authoritative source for validation rules.

Concrete implementation steps

These steps are explicitly illustrative. Exact CLI invocations depend on the converter toolchain you install from the repository. Replace example commands with the project CLI you have installed.

  1. Prepare environment: check out the ossie-converters repository and any vendor-specific adapter repositories. Verify versions of the converters and the core spec you will target. The converter README lists supported formats and instructions for building. Confirm the converter commit or release you will run, and record it in your migration plan.
  2. Export vendor model: use vendor export tools to produce the canonical vendor model artifact(s). Include all metadata exports available. For each export, preserve the exact artifact and timestamp for traceability.
  3. Run converter import: run the converter to produce an Ossie YAML document. Example, illustrative:
    <!-- language: text -->
    # illustrative only: replace with your converter CLI
    ossie-converter import --from vendorA --input vendorA-export.json --output model.ossie.yaml
    
  4. Validate against core spec: use the Ossie core spec validator or schema validation step documented in the core-spec repository. If the project provides a validator, run it. Otherwise, validate that required core keys exist per the spec document. Record all schema errors.
  5. Normalize: move vendor-only fields into an explicit extensions block. Do not discard unsupported metadata. The core spec allows an extensions area; use it to preserve vendor metadata for future reimport or audit.
  6. Annotate dialects and functions: for any SQL expressions or functions that depend on a dialect, annotate them with the vendor dialect and add a test case to your round-trip suite. The converter README and core spec describe where dialects should be recorded.
  7. Store artifact in version control and an artifact repository that supports immutable storage. Tag with converter version and core-spec version.

Validation should not stop at YAML. Run SQL samples and metric calculations against a test database that mirrors production semantics as closely as possible. The converter README and core spec do not guarantee functional equivalence; you must test it.

Round-trip comparison

Round-trip testing is where you prove, or find, differences. The round-trip process exports from source vendor to Ossie and back into a target vendor or back into the source. The converter repository documents import and export directions. The core spec documents what the Ossie document must contain after import. For each round-trip, build a comparison plan:

  • Structural comparison: compare model boundaries, lists of metrics and dimensions, join definitions, and extensions presence.
  • Semantic comparison: compare metric expressions and filter logic. If expressions use functions that are vendor-specific, note functional differences.
  • Numerical validation: run a representative set of queries and compare returned numbers to source results with tolerances for floating point differences.
  • Behavioral tests: run query plans that exercise join order, null semantics, and aggregate behavior.

Concrete comparison example, illustrative only: export metric "total_revenue" from Vendor A, convert to Ossie, import to Vendor B, run SELECT total_revenue FROM model LIMIT 1 in both systems and compare values across a curated dataset. If Vendor B computes totals using a different default null-handling rule, you will see divergences. That is expected; record it and map a fix: either adjust the Ossie expression to replicate original semantics, or accept a behavioral difference and document for reimplementation in the target.

Round-trip fidelity test checklist

  • YAML validates against core spec
  • All vendor metadata preserved in extensions
  • Metric expressions exist and have an annotated dialect
  • Sample queries return numerically equivalent results within a defined tolerance
  • Join semantics produce the same row counts for critical queries
  • Any not-portable items are flagged with mitigation plans

Failure modes to watch for: dropped vendor-only annotations, untranslated SQL functions, changed join cardinality due to different default join types, and mismatched aggregation null handling. In many cases, converters can preserve information but cannot reimplement engine-specific behavior; that is why you must test.

Parallel run and controlled cutover

OSSIE HUB-AND-SPOKE CONVERSION PATH1Inventory and portability…2Import, validate, and norm…3Round-trip comparison4Parallel run and controlle…Each stage produces evidence that can stop the rollout before the next boundary.
Ossie hub-and-spoke conversion path.

Once round-trip tests reach acceptable fidelity, move to a parallel run. Do not cut over purely on schema validation. A safe pattern is a canary set of reports or dashboards that are driven by both the original platform and the target (imported from Ossie). Compare numbers continuously for a defined period. After a successful parallel run, perform a controlled cutover with rollback plans.

Phased migration plan

Phasing example: pick low-risk subject areas first, then mid-risk with more complex metrics, then business-critical areas last. Use the phased plan to refine converters, extensions, and mapping rules.

Decision table for migration actions

Use this decision table to map inventory classification to actions.

PortabilityActionWhoMetrics
Core portableConvert to Ossie core; validate; include in parallel runData engineeringNumerical equivalence, schema validation pass
Portable with extensionConvert to Ossie; preserve vendor metadata in extensions; add reimplementation ticket for targetData engineering + Model ownerExtensions preserved, functional tests added
Not portableDo not convert automatically; create reimplementation or alternative planModel owner, ArchitectureMitigation plan, manual implementation tests

Failure modes and mitigation

  • Dropped metadata: Mitigation, move vendor fields to extensions and verify presence post-conversion.
  • Dialect mismatch: Mitigation, annotate dialect and provide translation adapter or rewrite expressions in target dialect.
  • Aggregation differences: Mitigation, add numerical tolerance checks and unit tests that compare pre- and post-migration values.
  • Join cardinality changes: Mitigation, add row-count tests and protect critical joins via explicit join conditions in Ossie expressions.
  • Operator or function missing in target: Mitigation, preserve original expression, add fallback UDF or emulation layer in the target engine, document behavioral difference.

Validation plan and operating metrics

Validation activities must include automated checks and human review. Monitoring should continue after cutover. Suggested validation plan items and operating metrics follow.

Validation plan

  1. Pre-conversion: inventory completeness, source query baseline capture for critical reports, and converter version locking.
  2. Post-conversion structural validation: Ossie YAML schema validation per core spec.
  3. Functional validation: run curated sample queries and compare results numerically with tolerances documented per metric.
  4. Behavioral validation: row-count and join tests for critical relationships.
  5. Security/privilege validation: ensure access control metadata is preserved in extensions and map to target privileges in your target platform. See governing guidance for using Ossie with a policy engine in the governance article link below for operational context.
  6. Cutover validation: parallel run for a defined window with continuous comparison and alerting on regressions.

Operating metrics to track

  • Conversion success rate: percent of models that pass schema validation on first run.
  • Round-trip fidelity rate: percent of metrics that match numerically within tolerance.
  • Extension preservation rate: percent of vendor metadata preserved into extension blocks.
  • Cutover regression rate: percent of reports failing comparisons during parallel run.
  • Time to troubleshoot: Mean time to resolve mismatches found during round-trip tests.

Version-specific cautions: the Ossie core spec and converters can change between releases. Always record the converter commit or release and the core-spec version you used. The Ossie and ossie-converters repositories are the authoritative sources; check them when you plan a migration. If you need governance guidance for using Ossie models with an external policy or authorization API, see the Dremio article that covers governing Ossie models with Apache Polaris APIs, privileges, and scope. Also see the Dremio announcement about Apache Ossie incubating for context on the project maturity and goals. For Polaris background, use the Dremio guide to Polaris.

Test Fixtures and Failure Injection for Converter Validation

PORTABLE CORE AND CUSTOM EXTENSIONSRisk 1Converters do not guarantee lossless translation for ever…Test itRisk 2One Ossie document represents one semantic model in the c…Bound itRisk 3Dialect fallback must be tested rather than assumedMonitor it
Portable core and custom extensions.

Answer first: start with synthetic fixtures that exercise edge cases, then inject targeted failures to confirm converters preserve semantics and produce actionable errors. This section gives reproducible failure injection patterns, what to assert, and how to interpret results.

Create test fixtures that mirror the three-layer structure you use in production: schema definitions, calculated measures, and presentation metadata. Use the Apache Ossie core spec document to craft fixtures that intentionally stress each required property and optional extension. The Ossie converters repository README provides the converter entry points you will call to perform translation; the Ossie core spec defines the expected document shape and required fields that validation must check.

Fixture categories

  • Happy path fixtures: minimal valid semantic models with simple measures and dimensions.
  • Complex calculation fixtures: nested expressions, window functions, time intelligence expressions, and vendor specific SQL snippets placed in expression text to test dialect handling.
  • Metadata-heavy fixtures: large sets of labels, descriptions, tags, and presentation groups, including vendor proprietary properties mapped into Ossie extensions.
  • Schema edge fixtures: nullable vs non-nullable mismatches, array and map types, nested record fields and deeply nested column paths.
  • Concurrent-change fixtures: two variants of the same model that differ only in ordering or whitespace to test idempotency and normalization stability.

Pick at least one fixture from each category. Keep filenames clear, for example: test-fixtures/complex-calc-2026-11-01.json.

Failure injection patterns

  • Drop required fields: remove the model identifier or top level semantics key and assert the converter returns a deterministic validation error rather than succeeding silently.
  • Introduce incompatible types: change a column type from integer to struct and confirm the converter either maps using an explicit rule or surfaces a type compatibility error.
  • Dialect escape sequences: insert vendor SQL with nonstandard quoting and confirm dialect fallback behavior. Do not assume automatic handling; capture the failure and document a remediation.
  • Large metadata payloads: add 10kB labels or repeated arrays to test extension preservation and performance impacts.
  • Unsupported feature flags: add vendor-only features that have no mapping; assert that the converter records them to extension metadata rather than discarding.

When an injected failure occurs, log the full Ossie converter input and output plus any intermediate normalized document. Keep these artifacts as part of the test report so engineers have a concrete starting point for fixes or for adding explicit translations to the converter codebase.

Operational Evidence and Acceptance Criteria for Migration Stages

Answer plainly: define pass/fail criteria per migration stage, require numerical equivalence for measures, and collect operational evidence during parallel runs. This reduces subjective "looks right" signoffs.

Acceptance criteria per stage

  • Import and Validation: converter must produce a syntactically valid Ossie document conforming to the Ossie core spec. Use the spec's required fields as pass conditions. See the Ossie core spec for the authoritative field list.
  • Normalization: the normalized document must preserve all unsupported vendor metadata within the Ossie extensions field. No supported property should be shifted to extensions without an explicit mapping rule.
  • Round-trip Comparison: after converting to Ossie and back to the target vendor format, key schema attributes (column names, types, nullability) and measure definitions must match the original or differ only in documented, acceptable ways. Numerics must compare equal within the chosen tolerance.
  • Parallel Run: queries produced by the original and migrated semantic models must return numerically equivalent results for a representative query set, and performance degradation must be within predefined limits per query class.

Record acceptance evidence as an artifact bundle: original model, converted Ossie JSON, normalized JSON, re-exported vendor model, executed SQL traces, query logs, and numeric comparison reports. Store the bundle with a timestamped identifier and metadata about the converter version used. The Ossie converters repository is the place to source the exact converter version you used for repeatability.

Representative query set and numerical equivalence

Build a query set that samples these categories: exact aggregations, filtered aggregations, joins with null-propagation edge cases, time-based windows, and expressions with casts. For each query, record three numbers: absolute difference, relative difference, and a boolean pass/fail against tolerance. Choose tolerances per metric type: money fields expect tighter relative tolerance than approximate floats used for scores.

Additionally, include sanity queries that exercise semantics like default filters and row-level security stamps. If the converter changes filter semantics, the numerical equivalence checks will expose the difference quickly.

Test Harness Implementation and CI Integration

Answer first: implement a test harness that wraps converter calls, executes queries in both systems, and produces machine-readable reports. Integrate this harness into CI so migrations are reproducible and converter changes are validated automatically.

Components of the harness

ROUND-TRIP FIDELITY TESTDeclareversioned contractExecutecontrolled fixtureCompareexpected stateGateship or stopA failed result returns with evidence. It does not become an undocumented exception.
Round-trip fidelity test.
  • Converter runner: a script or small program that loads a vendor model, invokes the appropriate converter in the Apache Ossie converters project, and writes the Ossie JSON output. The converters README lists available converters and invocation patterns.
  • Validator: a module that checks the output JSON against the Ossie core spec, returning a machine-readable pass/fail and a list of missing or extra fields.
  • Normalizer: deterministic normalization that sorts arrays in a documented order and collapses whitespace in expressions so round-trip comparisons are stable.
  • Round-trip exporter: invokes the reverse conversion back to the vendor model where supported. If a reverse converter does not exist, the harness marks this as a documented limitation and fails the round-trip stage but proceeds to downstream validation that operates directly on Ossie documents.
  • Query executor: runs the representative query set against both source and migrated semantic layers. Depending on the environment, this might execute SQL directly against the target systems or use the system APIs to run semantic queries. Capture execution plans when available.
  • Comparator and reporter: computes numerical and schema diffs, produces JSON reports, and emits HTML for human review.

Wrap these components with an orchestrator that executes a defined pipeline: import, validate, normalize, round-trip (if applicable), deploy to staging, run queries, compare results. Make each step idempotent and emit a stable artifact at the end.

CI practices and scheduling

Place the harness in a CI job that runs on pull requests to the converter codebase and on a nightly schedule for a small selection of canonical fixtures. For large migrations, establish a scheduled job that re-runs the full fixture suite weekly while teams prepare cutovers. Keep the CI runs lightweight by sampling from the full fixture set for pull requests and using full runs for nightly or pre-cutover validation.

Make the CI produce artifact bundles referenced in the migration ticket so reviewers can reproduce failures locally using the exact converter commit. Link the converter commit SHA in the acceptance evidence metadata to ensure traceability. The Ossie converters GitHub repository should be the single source for the converter code used in CI.

Decision Boundaries and When to Pause a Migration

Answer first: pause when numerical equivalence fails for critical metrics, when unsupported features are widespread, or when round-trip fidelity falls below your threshold. This section defines thresholds and actions.

Hard stop conditions

  • Critical metric breaches: any top five business KPIs show numerical differences beyond agreed tolerances in a parallel run. Treat this as a hard stop.
  • Unsupported vendor feature frequency: if more than X percent of semantic models use a vendor feature that the converter cannot map and the feature affects query semantics, pause and either extend the converter or postpone those models. Set X according to your portfolio; 10 to 20 percent is a reasonable starting point.
  • Round-trip failure rate: if more than Y percent of models fail round-trip fidelity checks due to schema mismatches, investigate the converter rules or normalization steps. Set Y conservatively, for example 5 percent, for an initial migration.

Soft pause conditions and mitigation actions

  • Performance regressions: if migrated queries are slower by more than Z percent on average for a query class, consider targeted optimization or isolation of those models. Z can vary by use case: 20 percent for interactive dashboards, 50 percent for aggregated batch reports.
  • Operator confusion: if support teams raise repeated issues around presentation grouping or label loss, produce a remediation plan that includes UI mapping adjustments and documentation updates.
  • Extension metadata piling up: if the extensions store grows such that core properties are continually placed into extensions, add an explicit mapping sprint to the converter backlog instead of proceeding with migration.

For each pause condition, create a structured action item with an owner, deadline, required artifacts, and a risk decision. Keep decisions recorded in the migration runbook for auditability.

Decision matrix for automation

Automate cutover gating with a simple rule engine that consumes CI reports. Use these inputs: percentage of models passing round-trip, percentage of critical queries passing numerical equivalence, number of unsupported features discovered, and performance delta percentiles. Map these to actions: automated proceed, manual review, or stop.

Example automated rule set, to be tuned by your organization:

{
  "proceed": {
    "round_trip_pass_pct": ">=95",
    "critical_queries_pass_pct": ">=98",
    "unsupported_feature_pct": "<=5",
    "median_perf_delta_pct": "<=10"
  },
  "manual_review": {
    "round_trip_pass_pct": "80-95",
    "critical_queries_pass_pct": "90-98",
    "unsupported_feature_pct": "5-15"
  },
  "stop": {
    "round_trip_pass_pct": "<80",
    "critical_queries_pass_pct": "<90",
    "unsupported_feature_pct": ">15",
    "critical_metric_breach": true
  }
}

These numbers are illustrative. Tune thresholds using a pilot migration and record the rationale for chosen thresholds in your runbook.

FAQ

PHASED SEMANTIC-MODEL MIGRATIONObservecapture evidenceCompareuse a baselineDiagnosefind the boundaryActchange one variablemeasured resultunexpected changesmallest safe responsenew baseline
Phased semantic-model migration.
  1. Q: Do converters guarantee lossless translation?
    A: No. The converters and the core spec do not guarantee lossless translation for every vendor feature. The project documentation explicitly cautions this. Use extensions to preserve metadata and plan reimplementation for engine-specific behavior.
  2. Q: How is a semantic model represented in Ossie?
    A: As of the current core spec, one Ossie document represents one semantic model in a flat shape. The core spec describes the required and optional keys; consult the spec for details.
  3. Q: What should I do with vendor-specific metadata?
    A: Preserve vendor-specific metadata in the permitted extensions block rather than discarding it. The converter README and core spec provide guidance that extensions exist for this purpose. This makes reimport or audit possible.
  4. Q: How do I trust metric equivalence?
    A: You do not trust equivalence by looking at YAML. Run numerical validation: curated queries, known aggregates, and tolerance checks. Include edge cases like nulls and distinct counts.
  5. Q: What about SQL dialect differences?
    A: Dialect fallback must be tested. Annotate every expression with a dialect tag where the converter cannot translate semantics. Implement translation adapters or adjust the expression in the target platform.
  6. Q: Are there automated validators I can use?
    A: The Ossie core-spec repository provides the normative specification. The converter repository README documents converter tooling. If the converters include a validator, use it; otherwise validate structural compliance programmatically against the spec. You must still perform runtime functional tests.

Concrete worked migration material

The following subsections give the mandatory worked material you must include in any operational migration plan: Inventory and portability classification, an import/validate/normalize workflow, round-trip comparison details, and a parallel run cutover plan. Each section includes explicit checks and expected artifacts.

Inventory and portability classification, worked example

Artifact: CSV or JSON of models with these fields: model_id, owner, metric_id, metric_expression, metric_type, join_id, join_left_key, join_right_key, dialect, vendor_metadata_keys, portability_bucket, portability_rationale.

Check: every model has at least one representative query recorded and a metric marked critical or non-critical. Portability rationale must cite the core spec clause when claiming core portability, or cite why an extension is needed.

Import, validate, and normalize workflow, worked example

  1. Store vendor export in an artifact store with checksum.
  2. Run converter CLI to produce ossie YAML, recording the converter version and commit.
  3. Run structural validator: check for required keys listed in the core spec. Produce a diff of missing keys.
  4. Move vendor-only fields into an extensions object; if you cannot programmatically identify them, produce a manual mapping document.
  5. Run the functional tests: for each critical metric, run the curated query in source and target and store results with timestamps and environment identifiers.

Round-trip comparison, worked example

Artifacts: original vendor export, ossie YAML, re-exported vendor model from Ossie import into target, query result sets for source and target.

  1. Structural diff of models: use JSON/YAML diff tools to check lists of metrics and dimensions.
  2. Expression diff: compare expressions token-by-token and flag dialect-specific functions.
  3. Numerical diff: for each metric, compute relative error. Define acceptable threshold; for currency sums, typical tolerance is 0.0 if integers or exact decimals are used, otherwise a small epsilon for floating arithmetic. Document the threshold per metric type.
  4. Behavioral diff: compare row counts on join-heavy test queries.

Parallel run and controlled cutover, worked example

  1. Pick 5 to 15 representative reports for the canary set, covering each critical model.
  2. Route these reports to both source and target; schedule automated comparisons daily for a 2 week window.
  3. Alert if any metric deviates beyond its defined threshold; investigate and freeze cutover if alerts exceed a configured limit.
  4. After successful window, migrate remaining reports in waves with the same pattern. Maintain the ability to roll back per report for at least one billing cycle.

Version-specific cautions

Ossie is an Apache project; the core spec and converters evolve. Always record the commit or release for both ossie and ossie-converters you used. The core spec is authoritative for what Ossie documents may contain; behavior of converters can differ. If a feature appears unsupported in the converters, verify whether the converter version intentionally excludes it or whether a mapping exists in a newer commit. The project repositories are the authoritative sources for these details.

Keep learning

For a deeper treatment, download Apache Polaris: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.

Use Dremio's Apache Polaris guide as the evergreen reference for the wider architecture and terminology.

Sources

Practical recommendation: do the inventory and portability classification first, and lock converter and core-spec versions before writing any migration automation. Preserve all vendor metadata in extensions; do not assume automatic reimplementation of engine-specific behavior. Use round-trip numerical tests and a phased parallel run to establish operational confidence before cutting over production reports. That single practical step avoids the majority of migration surprises and gives a traceable rollback path if you encounter untranslatable semantics.

Try Dremio Cloud free for 30 days

Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.