insight-miner

A domain-blind statistical engine that mines tabular and JSON data for actual patterns and insights — outliers, trends, concentration (Pareto/ Gini), and correlations/associations between fields — ranked by effect size, not a per-cell validity checker. Point it at any CSV/JSON/JSONL file and it tells you what's actually going on in the data.

This project supersedes data-insight-profiler, a prior private repo that flagged whether individual cells look valid (junk tokens, format outliers). That's a data-quality classifier, not pattern discovery, and it was the wrong target — this repo starts over with the actual goal: find what's interesting in the data.

It found a real bug in a real SEC filing

Running insight-miner mine on flattened SEC EDGAR XBRL facts for 10 large US companies (2009–2026 filings) surfaces this as the #1 finding by z-score:

1.000 [outlier] 'val' has 15 extreme outlier(s); most extreme is 4.819e+15
      at row 231494 (z=+521.7, field mean=7.642e+10, std=9.237e+12)

That row is Oracle's EntityCommonStockSharesOutstanding for the period ending 2012-09-17: 4,819,056,000,000,000 shares. Every neighboring filing (e.g. the 10-K three months earlier) reports ~4.88 billion shares. 4,819,056,000,000,000 / 4,882,506,000 ≈ 1,000,000 — this is a real XBRL scale/decimals error in Oracle's actual 2012 10-Q filing, found by a domain-blind z-score check with no idea what "shares outstanding" means. Reproduce with make mine FILE=data/sec/facts_flat.csv after make fetch.

On a 3rd dataset from a different domain, it rediscovered real seismology — with zero seismology built in

make mine FILE=data/usgs/earthquakes.csv (USGS earthquake catalog, real 2026 data) surfaces roughly ten independent findings that all trace back to the same underlying physical mechanism, cross-validating each other:

• Every top horizontalError/depthError/dmin/magError outlier is a remote or sparsely-instrumented event — e.g. an M4.3 on the Mid-Indian Ridge with dmin=59° (≈6,550 km to the nearest seismic station) and only 9 stations reporting.
• Every top nst/magNst outlier is the opposite case — large earthquakes (M5.5+) detected by hundreds of stations across a wide network.
• gap decreases as nst increases (Pearson r=-0.52) — a correlation check independently rediscovering the general form of the outlier stories above: more reporting stations → better azimuthal coverage → smaller location uncertainty.
• magType='mww' has mean magnitude 1.40x the overall average (ω²=0.84) — moment magnitude (W-phase) is a scale seismologists only compute for larger earthquakes; a domain-blind group-effect check rediscovered that real methodological convention.

None of this was hand-picked — it's what a hand-verification pass over the top findings turned up. Full trace in NEXT-STEPS.md.

Why the benchmark design, not hand-picked examples

A single cool finding proves the engine can find something, not that it reliably does. So engine/inject_patterns.py plants known patterns (a target Pearson r, a target Gini concentration, a target trend slope, a set of engineered outliers) into real data as new synthetic columns, and a separate shuffle-based control destroys every real cross-column relationship to check for false discoveries. engine/miner/ never imports the injector and has no access to what was planted — it only sees the resulting file, same as any real dataset.

v15 result (results/v15/benchmark.json), across four real datasets (O*NET CSV, flattened SEC JSON, USGS earthquake catalog, NOAA storm events) plus five synthetic stress tests: 24/24 planted patterns detected, 0 cross-field false discoveries. That 0 was earned, not assumed. Every detector below started with a fixed threshold that seemed reasonable in isolation; every one turned out to have a real, measurable bias once tested against a synthetic null built specifically to stress it. Seven bugs found and fixed in seven cycles (v2-v7):

Detector	Bug	Found via	Fix
concentration (Gini)	Biased by group size — a field 89% one category "looked concentrated" from row count alone, even with the numeric field independent of it	Adding a 2nd real dataset made a synthetic-null false-discovery rate go 0→7	Population-weighted Gini over group means
concentration (weighted_gini)	Silently dropped groups with negative mean — a selection bias, already live on real data (SEC's val is 10.5% negative)	Broadening an unrelated stress test	Shift field to non-negative by its global min before grouping
group_effect (eta²)	Degrees-of-freedom bias — inflates with many small groups (4% false-positive at k=150/n=3000)	Testing a hypothesis (wrong guess: not the same bug as concentration's)	Omega-squared (bias-corrected)
association (Cramér's V)	Same DoF bias, worse — 100% false-positive at the same k=150/n=3000 scale	Testing a hypothesis (this time confirmed)	Bergsma-corrected Cramér's V
trend (R²)	Worst single-test bug by magnitude — regression's real n is bucket count, not row count; 43.6% false-positive at the default MIN_BUCKETS=4	Auditing the last untested small-sample case	Adjusted R² + MIN_BUCKETS 4→12 (stated residual risk, not closed)
outliers (z-score)	Opposite bias — large-sample, not small: fixed z>3 threshold has ~100% chance of firing on any big field (both real datasets' scale) with zero real anomaly	Auditing the last unaudited detector	Bonferroni-corrected, n-aware z-threshold (stdlib NormalDist)
trend (again)	The v5 residual risk showing up for real: 16.7% dataset-wide false-discovery rate on ONET across 60 seeds, 100% of it trend firing on ONET's real Date field (15 buckets, just inside the danger zone)	Multi-seed testing (not one fixed seed) after all 6 detectors were individually audited	MIN_BUCKETS 12→20 — 0/30 seeds after. Real cost, stated plainly: 12-19-bucket time series now get no trend analysis, not a safer one

Full trace of what was hypothesized vs. measured for each is in NEXT-STEPS.md. A follow-up validation pass (tag v0.7.1, no code change) closed the two gaps v7 left open: planted-pattern recall multi-seeded on O*NET (100%, 30/30, all 5 pattern kinds) and the dataset-wide false-discovery check multi-seeded on SEC (0/18 seeds) — both came back clean, real evidence the calibrations generalize rather than an assumption. v8 then added a new capability (grouped_outlier — z-scores within a categorical field's groups, not the whole numeric column), and v9 gave it planted ground truth to confirm the exact scenario it exists for (a value extreme only within its own group, invisible to the whole-column check) is actually detected: yes, ranked #3 and #4 among all findings on the two real datasets, not a marginal pass.

v10 is the real test of all of this: a third real dataset from a completely different domain. Every fix and calibration through v9 was validated on the same two datasets (O*NET, SEC) — the USGS earthquake catalog (public FDSN API, keyless, 22 columns, dense daily-timestamped event data, real Jan–Aug 2026 filings) is structurally unlike either: event-log shaped, not survey/filing shaped; 11-12 numeric fields including several independent measurement-uncertainty columns; daily rather than monthly time density. Result: every calibration held — 0 cross-field false discoveries, all 6 pattern types detected — with zero new bugs. One real calibration issue was caught and fixed before publishing (the concentration injector's tier gap needed raising for this dataset's category structure, verified not to regress the other two first), which is a different, better outcome than a silent pass: the check that would have caught a problem was exercised and worked. A hand-verification pass (tag v0.10.1) then confirmed USGS's real findings — outliers, a correlation, and a group effect — all trace to the same real seismological mechanism (see above).

v11 set out to validate looks_like_sparse_identifier (tested on exactly one case, CIK, until now) and found two real bugs instead of zero. First: USGS's actual time column — the whole reason a dense-time-series dataset was added — was invisible to trends.py, because try_parse_date had no pattern for ISO 8601 datetimes (2026-05-14T02:03:05.557Z), only bare dates. Second, found by the synthetic tests the cycle was actually about: a classic 1-5 Likert scale was misclassified as a sparse identifier and silently excluded from all numeric analysis — the relative gap formula blows up for any small- magnitude scale regardless of how densely packed it is. Fixed both; 0 regressions, still 18/18. A known gap remains, stated not hidden: synthetic zip codes still narrowly miss detection (0.17 vs the 0.2 cutoff) — see NEXT-STEPS.md.

v12: even fixed, USGS's real time field still showed zero trend findings — trends.py only ever aggregated to calendar month, and Jan–Aug 2026 is only ~8 months, despite 225 real days of density. Gave it adaptive granularity: try month first (unchanged everywhere it already worked — measured, not assumed, that O*NET/SEC's date fields never hit the new path), fall back to day only when month fails but day succeeds. Result: 6 new real trend findings on USGS — nst, magNst, mag, rms, dmin trending up and gap down, all at 172 daily buckets. Deliberately not claimed as understood the way the station-coverage story was: these trend over updated (when USGS last revised the record), not time (when the quake happened), so the same pattern could mean real network improvement over the year or just a processing artifact — flagged as the next thing to dig into, not asserted as a second Oracle-grade finding. Confirmed the new day-level boundary is as safe as the existing month-level one with a dedicated synthetic stress test, not assumed from shared regression math: 0 false discoveries either way. Still 18/18, 0 regressions.

Follow-up investigation resolved it: the trend is a data-pipeline artifact, not a real network improvement. Three checks: magnitude has essentially no correlation with revision lag (r=0.034, ruling out "bigger events take longer to finalize"); the trend vanishes entirely (6 → 0 findings) when bucketing by occurrence time instead of revision time; and updated shows real batch clustering — some days have 6-9x the average number of revisions, consistent with periodic reprocessing runs rather than a smooth trend in data quality. A more mundane answer than a second Oracle-grade bug, and the honest one — not every statistically real pattern this tool finds is a discovery about the underlying phenomenon; some are artifacts of how the data was produced, and telling those apart is exactly what this follow-up was for.

v13 finally tested the long-deferred wide-dataset question — and it mattered. Every dataset through v12 had ≤22 columns; the O(fields²) pairwise-check exposure to multiple comparisons had never actually been stressed. Fetched the NOAA Storm Events database (51 columns, 23,255 rows, keyless bulk CSV, real 2026 data): 24 numeric fields × 17 categorical, several conditionally-populated (tornado-only, flood-only columns). Running the noise-baseline check against it broke the 0-false-discovery streak held since v7: 7 cross-field false discoveries on pure noise. Diagnosed to a specific, new mechanism — not v4's degrees-of-freedom bias (Bergsma already handles many balanced categories on adequate n), but a high-cardinality field colliding with a mostly-null one: DAMAGE_PROPERTY (183 categories) × FLOOD_CAUSE (96.6% null) left only ~600 overlapping rows, an average expected cell count around 1 — far under the classical chi-square validity floor (Cochran's rule, minimum average expected count ≥5). Verified against a synthetic replica of the exact cardinality/overlap shape before fixing. Fix: cramers_v_corrected now requires an average expected cell count ≥10 (double the classical floor — the floor alone still let one borderline case through). Dropped NOAA's false discoveries 7→3, with zero regression on the existing 18/18 planted-pattern recall. The remaining 3 are a different, not-yet-fixed mechanism — group_effect on categorical fields that are >99% null, collapsing to 8-9 groups of 5-14 rows each, combined with a zero-inflated numeric field. Stated as the top open item, not hidden: see NEXT-STEPS.md.

v14 set out to fix that one remaining mechanism and found six more — not by rushing, but by refusing to stop at a result that looked clean too early. The group_effect fix (raising omega²'s minimum group size) looked clean at a single seed, but this project already knows one seed isn't enough (v7 learned that the hard way for a different detector) — a 10-seed re-check found the fix was incomplete and surfaced 9 false discoveries spanning three detector kinds, not one. Chasing each down individually turned up a real, distinct mechanism every time: an initial group-size floor that needed raising further after a proper sweep (5→20→50); a chi-square validity check that used the wrong statistic (an average instead of the classical minimum-cell/fraction rule — a single near-zero-expected cell from a singleton category could dominate chi2 even with a healthy average); a fix for that which then broke a real planted-pattern recall test (the injector relabels rare tail categories too, so pooling them out was needed to recover the genuine signal without reopening the noise problem); a correlation threshold with a measured 20% false-positive rate at the minimum sample size, fixed the same principled way outliers.py's z-threshold was in v6 (Fisher's z-transformation + Bonferroni correction, stdlib only); an analogous significance gate needed for association, because a bias correction fixes a statistic's mean but not its exposure to hitting significance by chance across ~136 tested pairs; the same missing per-group floor bug turning up independently in concentration.py's Gini calculation, discoverable only by bootstrapping real skewed NOAA data since a synthetic Gaussian null failed to reproduce it; and finally, pushing the check to 30-40 seeds surfaced one more — a correlation driven entirely by a single data point because one field was 98.8% a single repeated value, which no amount of n-aware correction fixes since it breaks Pearson r's continuous-data assumption regardless of n. Result: a full 40-seed NOAA noise-baseline sweep, 1 residual false discovery (2.5%) — checked and confirmed to be the expected behavior of a correctly-calibrated α=0.05 significance test, not a bug (both fields non-degenerate, sample size and effect size both clear the Bonferroni-corrected gate; roughly 1-in-20 independent runs should do this by design). The full benchmark separately maintained 18/18 planted recall, 0 false discoveries, zero net regression. Full technical detail for all seven fixes is in engine/miner/correlations.py and engine/miner/concentration.py's module docstrings; NEXT-STEPS.md has the accessible summary. The honest framing: this is one recurring statistical principle — a correctly bias-corrected effect-size estimator can still be unreliable at small n, small group count, or on near-degenerate data — showing up in four different detectors, found only by refusing to trust a result that looked clean until it had actually been checked hard enough to be sure.

v15 integrated NOAA into the full quantitative benchmark as a 4th real dataset — the natural next step once v14's fixes were confirmed stable. A qualitative first pass (make mine, same practice as USGS in v10) surfaced a genuine, non-error outlier before any calibration work began: BEGIN_LAT/END_LAT values near -14° stood out sharply against the field's ~37° mean. Traced to specific rows: real storm events in American Samoa (Tutuila, Manu'a), correctly geocoded — a real US territory, just far outside the CONUS-dominated distribution that makes up the vast majority of the dataset. Same "correctly unusual, not a bug" shape as USGS's Mid-Indian Ridge finding — a domain-blind check correctly finding something real and legitimately rare, with zero geography built into the tool.

Choosing which categorical field to build ground truth on was not trivial, and getting it wrong twice taught something real each time. STATE (63 categories) looked reasonable by every surface metric — 100% populated, comparable row counts to other datasets' fields — but planted association testing against it failed. Diagnosed precisely: 63 categories on both sides of the table pushed 49% of cells under the classical expected-5 floor, not a data pathology, just too many categories for even a strong signal to clear v14's now-strict validity check. This project had already written down the lesson (plant_concentration's docstring: "a reasonable safety margin... not a substitute for choosing a sane categorical field") — it just took a 4th dataset to need re-learning it for a different detector. Switching to CZ_TIMEZONE (11 categories) still failed association at first, but this time the cause was a real bug: plant_association's injector relabels every category of the source field, including rare ones — so even though the miner correctly pools small categories out of the real field before testing, the injected field kept all of them (noise draws still populate rare labels), building an asymmetric table that failed validity even for a genuinely strong planted signal. The same "injector needs to account for the miner's pooling behavior" lesson as v14's MIN_CATEGORY_N fix, independently rediscovered. Fixed with a min_category_n parameter on plant_association (default: a no-op, byte-identical on the first three datasets), wired through benchmark.py reading the live threshold from correlations.py — same "read live, don't hardcode a stale copy" pattern already used for trends.py's MIN_BUCKETS. Result: 24/24 planted patterns detected across all 4 real datasets, 0 false discoveries, 0 regression.

Quick start

make bench                                    # fetch data + run the full benchmark
make mine FILE=data/onet/skills.csv           # mine any file for patterns on its own
make mine FILE=data/sec/facts_flat.csv
make mine FILE=data/usgs/earthquakes.csv
make mine FILE=data/noaa/storm_events.csv

Stdlib-only in engine/miner/ (no numpy/scipy) — resource-friendly by construction, not by tuning. engine/fetch/ uses duckdb for convenience parsing O*NET's TSV dumps; falls back to plain python3 if ~/venvs/e1 isn't present.

Layout

engine/miner/	The domain-blind engine: stats.py (field typing + univariate stats), correlations.py, concentration.py, trends.py, outliers.py, grouped_outliers.py, findings.py
engine/inject_patterns.py	Plants known patterns + a shuffle-based noise control — kept separate from the miner on purpose
engine/benchmark.py	Runs the noise control + planted-pattern detection, scored by matching on field-set AND kind (a caught bug: matching by fields alone let a group_effect finding silently steal credit from a concentration finding on the same two columns)
engine/fetch/	Keyless downloaders: O*NET numeric rating tables, SEC EDGAR ticker list + company facts JSON, USGS earthquake catalog (FDSN event API)
engine/fetch/flatten_sec_facts.py	Flattens SEC's deeply-nested companyfacts JSON (cik → taxonomy → tag → units → unit → observations) into one flat table — the actual "JSON" test case, not just a flat array
results/vN/	Benchmark output per version (v1, v2, ...)
NEXT-STEPS.md	What's measured, what's known-weak, what to build next

What it finds (all domain-blind — no schema, no hints)

• Outliers: numeric values beyond a Bonferroni-corrected, n-aware z-threshold within their own column (a fixed z>3 cutoff has ~100% false-positive probability on a 60K+ row field by pure order statistics — see NEXT-STEPS.md v6)
• Grouped outliers: the same check, but within a categorical field's groups instead of the whole column — catches values that are extreme for their category even if unremarkable pooled with unrelated scales (e.g. a USD figure buried among share counts)
• Correlations: Pearson r between numeric field pairs (|r| ≥ 0.30, "moderate" by Cohen's convention)
• Group effects: does a categorical field's groups have a real mean difference on a numeric field — gated on omega² (bias-corrected for degrees of freedom, not raw eta²), ≥ 0.06
• Associations: Bergsma bias-corrected Cramér's V between categorical field pairs (≥ 0.20) — raw Cramér's V inflates with many categories relative to n
• Concentration: population-weighted Gini of a numeric total across a categorical field's groups (≥ 0.40, immune to group-size skew) — the "20% of X drives 80% of Y" shape
• Trends: adjusted-R² linear fit of a numeric field's monthly mean against any date-like field, only reported with ≥20 buckets (a line through a handful of noisy points looks deceptively clean)

Field typing (stats.classify_and_profile) is itself domain-blind: numeric vs categorical vs date vs high-cardinality id, decided from the data's own shape — including catching numeric-looking identifiers (SEC's cik) via a sparse-value-spacing heuristic so they don't get treated as summable quantities. See NEXT-STEPS.md for where that heuristic is known to be imperfect.

Known limitations (honest, as of v8)

• Thresholds are standard effect-size conventions (Cohen) or bias corrections (Bonferroni, Bergsma, omega²/adjusted-R²) validated against synthetic nulls and 2 real datasets — not p-value-exact, not tuned per-dataset, and not proven to generalize to structurally different data (a 3rd, differently-shaped dataset is the next open item).
• trends.py's MIN_BUCKETS=20 floor is a real, stated tradeoff, not a free fix: any dataset whose real time dimension has 12-19 buckets gets no trend analysis from this tool, not a worse-calibrated one — see NEXT-STEPS.md v7.
• grouped_outlier (v8) has no dedicated planted-pattern ground truth yet — it's confirmed to find real things on both datasets, but not yet benchmarked against a synthetic case built specifically for it.
• Statistically extreme ≠ wrong. Outlier detection (grouped or not) surfaces unusual values; distinguishing a data error (Oracle's XBRL scale bug) from a genuine extreme (JPMorgan's real multi-trillion- dollar derivatives notional) takes independent context this tool doesn't have — see the v8 section of NEXT-STEPS.md for both examples.