Home /Modules/churn/DATA-CATALOG.md

DATA-CATALOG.md

churn/DATA-CATALOG.md · 125 line(s) · markdown

Churn Data Catalog and Integration Plan

Max Rupplin - MEARVK LLC - 2026

Purpose

This catalog defines how the five Churn dimensions fit together and how new data should be added without creating duplicate, untraceable, or dimensionally incompatible records. It complements the set-specific READMEs and BASIS documents; it does not replace them.

Existing data inventory

SetDimensionExisting examplesIntegration role
1 — LongsTime serieslongs.csv, assumptions CSV and SLeeLa modelPeriod-based measurements and assumptions
2 — StructuresScalar/value with unit or evidence kindMath and physics constants, SI base units, chemical elements, solar-system and social-labor dataCanonical values and measurement definitions
3 — FacetsProperty of an identified objectMath/physics facets, moons, electronegativity, SI derived units, social-labor facetsObject-to-property relationships
4 — AreasTwo-dimensional extentGeometry areas, country areas, Earth surface, Great Lakes, continents, US densityArea values and spatial context
5 — LinesOne-dimensional length or countIntegers, geometry lengths, physical lengths, US lengths, common countsCount/length values and derived line measures

The repository has broad representative coverage. File presence alone does not prove correctness, completeness, licensing, or currentness; do not label a dataset authoritative without checking its publisher and source document.

Shared record contract

For newly normalized machine-readable datasets, use the following fields where applicable:

New normalized JSON datasets may opt into the validator's strict contract by setting top-level "record_contract": "churn-1" and supplying top-level source_uri and license. This is an opt-in migration path; legacy data is not silently rewritten.

Cross-dimensional relationships

Use explicit identifiers rather than relying on filenames or matching display labels. A relationship record should contain from_id, relationship, and to_id; derived relationships also need method or formula and explicit unit handling.

Formulas and dimensional compatibility

Store formula identity separately from observed values. A formula record should document its inputs, output unit, and assumptions. Validate unit compatibility before comparing or combining quantities. Examples of expected dimensional checks:

Inheritance and composition

Prefer composition for cross-dimensional records: a Long observation references a Structure definition, a Facet references an object and a property, and derived records reference their inputs. This avoids pretending that a time series, area, and line are interchangeable types.

If actual SLeeLa class inheritance is introduced later, keep a small shared evidence/provenance contract and make the five dimension types distinct descendants only if compiler semantics support that design. The Java constraint-report classes are build/test utilities, not the data-model inheritance hierarchy. The current sets should therefore be described as related by composition and shared validation, not as implemented subclasses.

Import policy

Before importing a dataset:

  1. Identify the publisher and canonical source URL.
  2. Record retrieval date, license/terms, version, and attribution requirements.
  3. Preserve the original source file or a reproducible retrieval procedure.
  4. Normalize units and document conversion formulas.
  5. Map records to stable IDs and check duplicates against canonical records.
  6. Label primary, observed, assumed, or derived data correctly.
  7. Add a source-specific note and a parsing/invariant test.
  8. Update the relevant set README and this catalog in the same change.

Do not import bulk data merely to increase file count. Prefer authoritative publisher sources, but import only when the data adds a clear dimension/domain, provenance is verifiable, and the data can be validated. Respect source terms and licenses.

Validation and test coverage

validate_data.py currently checks JSON/XML parseability, CSV header/row consistency, duplicate CSV IDs where an id column exists, duplicate IDs in record-array JSON, known evidence-kind values, and the opt-in churn-1 JSON contract. These structural checks do not establish scientific correctness or that a source remains current.

The validation roadmap is:

  1. Required fields and stable IDs for normalized datasets.
  2. Referential integrity for derived_from and cross-dimensional relationship records.
  3. Unit compatibility before arithmetic or comparison.
  4. Source-backed range/invariant checks.
  5. Provenance, license, and retrieval metadata for imports.
  6. Deterministic smoke tests for each SLeeLa model and Java constraint report.

Legacy files may remain outside the strict contract until migrated; do not turn missing optional legacy metadata into false validation failures. Invalid syntax and explicit contract violations fail the check.

Build entry points

From repository root:

make churn
make churn-data-check
make churn-test

From this directory:

make all
make data-check
make test
make clean

The root Makefile already dispatches these targets. Keep the local Makefile authoritative and avoid duplicating compiler commands at the repository root.

Immediate implementation status