Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 5314071c6b | |||
| 86feec2e77 | |||
| 4f3fe64cad | |||
| 33ee191966 | |||
| 6f23f83bc8 | |||
| 39a5a4ec0c | |||
| 8510903771 | |||
| 3e52117c1a | |||
| 8153777951 | |||
| 9b3d52d89b |
@@ -0,0 +1,158 @@
|
|||||||
|
# tDCS reaching study — GLM methods
|
||||||
|
|
||||||
|
This document explains the statistics in `tdcs_glm.py`: the models, the exact
|
||||||
|
formulas, how each subject's progress is accounted for, and the caveats.
|
||||||
|
|
||||||
|
## The question
|
||||||
|
|
||||||
|
Two **known** conditions anchor the performance scale and differ from each other:
|
||||||
|
|
||||||
|
- **H2 (anchor check):** `Electrode-Box-B2` performs **better** than `Electrode-Box-A2`.
|
||||||
|
|
||||||
|
Two **unknown** conditions are then **classified** against those anchors — for
|
||||||
|
each of `Electrode-Box-A` and `Right-Electrode`, is it **A2-like** or **B2-like**?
|
||||||
|
We do *not* assume either belongs to B2; each unknown is compared to *both*
|
||||||
|
anchors, and the anchor it cannot be distinguished from is its likely class.
|
||||||
|
|
||||||
|
There is also a **merged-assumption** mode (`--merge`): assume the two unknowns
|
||||||
|
resolve as `Right-Electrode == Box-B2` and `Electrode-Box-A == Box-A2`, fold them
|
||||||
|
into the anchors, and re-estimate everything with 4 subjects per group.
|
||||||
|
|
||||||
|
## Data and outcome
|
||||||
|
|
||||||
|
- **Outcome:** `success` = successful reaches in a session
|
||||||
|
(`analysis_summary.counts.Success`), a non-negative **count**.
|
||||||
|
- **Attempts:** `total` = reach attempts in the session (`analysis_summary.total`);
|
||||||
|
used as the denominator for the rate model. `success_rate == success / total`.
|
||||||
|
- **Time:** `day` = the "# Days Reach" field (training day, 0..26). Analyses use
|
||||||
|
days ≥ 0 (pre-training negative days and zero-attempt sessions are excluded;
|
||||||
|
zero-attempt sessions are undefined for the rate model).
|
||||||
|
- **Groups (subjects):** Naive (4), Box-A2 (3), Box-B2 (3), Box-A (1),
|
||||||
|
Right-Electrode (1). Under `--merge`: Box-A2 (4), Box-B2 (4), Naive (4).
|
||||||
|
|
||||||
|
**Provenance:** `tdcs_reach_data.csv` was reconstructed from the experiment
|
||||||
|
database and verified cell-by-cell against the exported matrix (71/71
|
||||||
|
unambiguous cells on days 0–5 matched exactly).
|
||||||
|
|
||||||
|
## The three models
|
||||||
|
|
||||||
|
All models use `Electrode-Box-B2` as the **reference** group, so each group term
|
||||||
|
is that group's contrast *versus Box-B2*. `day_c` is the centered training day and
|
||||||
|
`day_c2 = day_c²` captures the rise-then-plateau of the learning curve.
|
||||||
|
|
||||||
|
### (A) Level — count (primary)
|
||||||
|
|
||||||
|
Poisson GEE on the success counts, clustered by subject:
|
||||||
|
|
||||||
|
```
|
||||||
|
success ~ C(group, Treatment('Electrode-Box-B2')) + day_c + day_c2
|
||||||
|
family = Poisson (log link)
|
||||||
|
groups = subject # repeated-measures cluster
|
||||||
|
cov_struct = Exchangeable # working within-subject correlation
|
||||||
|
SE = robust (sandwich)
|
||||||
|
```
|
||||||
|
|
||||||
|
`exp(coef)` for a group term is an **incidence-rate ratio (IRR)**: expected
|
||||||
|
successes relative to Box-B2.
|
||||||
|
|
||||||
|
### (B) Level — rate
|
||||||
|
|
||||||
|
Binomial GLM on successes-out-of-attempts, with cluster-robust SEs by subject:
|
||||||
|
|
||||||
|
```
|
||||||
|
cbind(success, total - success) ~ C(group, Treatment('Electrode-Box-B2')) + day_c + day_c2
|
||||||
|
family = Binomial (logit link)
|
||||||
|
cov_type = cluster (groups = subject)
|
||||||
|
```
|
||||||
|
|
||||||
|
`exp(coef)` is an **odds ratio** for a successful reach relative to Box-B2. This
|
||||||
|
controls for differing numbers of attempts, so it answers "who is more accurate
|
||||||
|
per attempt?" rather than "who attempts more?"
|
||||||
|
|
||||||
|
### (C) Learning rate
|
||||||
|
|
||||||
|
Poisson GEE with a **group × day** interaction, to ask whether groups improve at
|
||||||
|
different *rates* (not just different levels):
|
||||||
|
|
||||||
|
```
|
||||||
|
success ~ C(group, Treatment('Electrode-Box-B2')) * day_c + day_c2
|
||||||
|
```
|
||||||
|
|
||||||
|
Each `group[T.X]:day_c` term is the difference in log-slope versus Box-B2; a joint
|
||||||
|
Wald test asks whether *any* group's slope differs. Large p ⇒ parallel learning.
|
||||||
|
|
||||||
|
### Anchors-only model
|
||||||
|
|
||||||
|
The A2-vs-B2 comparison, refit on **just the two anchor groups** so nothing else
|
||||||
|
influences the shared day terms or the dispersion/correlation nuisance:
|
||||||
|
|
||||||
|
```
|
||||||
|
# subset to {Box-A2, Box-B2}
|
||||||
|
success ~ C(group, Treatment('Electrode-Box-B2')) + day_c + day_c2 # count
|
||||||
|
cbind(success, total-success) ~ C(group, ...) + day_c + day_c2 # rate
|
||||||
|
```
|
||||||
|
|
||||||
|
With B2 as reference the single group term is the A2-vs-B2 effect; H2 is a
|
||||||
|
one-sided test that this coefficient is below zero.
|
||||||
|
|
||||||
|
## How each subject's progress is accounted for
|
||||||
|
|
||||||
|
Two distinct pieces:
|
||||||
|
|
||||||
|
1. **Progress over training** — the `day_c + day_c2` fixed terms model the average
|
||||||
|
learning curve, so groups are compared at comparable points in training rather
|
||||||
|
than being confounded by *when* each was measured.
|
||||||
|
|
||||||
|
2. **Repeated measures / individual baselines** — each subject contributes many
|
||||||
|
correlated sessions and has its own baseline. Handled two ways:
|
||||||
|
|
||||||
|
- **Primary (GEE):** subject is the cluster; an exchangeable working
|
||||||
|
correlation plus robust (sandwich) SEs give *population-average* group
|
||||||
|
effects whose inference is valid under within-subject correlation and
|
||||||
|
Poisson overdispersion.
|
||||||
|
|
||||||
|
- **Sensitivity (mixed model):** a Poisson model with a **per-subject random
|
||||||
|
intercept** (`(1 | subject)`) so each animal gets its own baseline level;
|
||||||
|
the group effects are estimated after allowing for that individual variation.
|
||||||
|
statsmodels has no frequentist Poisson GLMM, so this is the **MAP/Laplace**
|
||||||
|
fit (`fit_vb` diverges on these large counts). Its p-values are approximate —
|
||||||
|
read it as a direction/magnitude check that should agree with GEE.
|
||||||
|
|
||||||
|
## Classification logic (unknowns)
|
||||||
|
|
||||||
|
For each unknown group, `diff_contrast` builds a linear contrast of that group
|
||||||
|
against **each anchor** (both coded vs the B2 reference) and tests it:
|
||||||
|
|
||||||
|
- indistinguishable from B2 **and** different from A2 → **B2-like**
|
||||||
|
- indistinguishable from A2 **and** different from B2 → **A2-like**
|
||||||
|
- indistinguishable from both → **ambiguous** (report the numerically nearer one)
|
||||||
|
- different from both → **unlike both**
|
||||||
|
|
||||||
|
## Caveats
|
||||||
|
|
||||||
|
- **Tiny groups.** Box-A2/B2 have 3 subjects, Naive 4, and each unknown has
|
||||||
|
**n = 1 subject**. Classifying a one-subject condition is weak: "matches
|
||||||
|
anchor X" means "not statistically distinguishable from X," **not** proof of
|
||||||
|
equivalence.
|
||||||
|
- **Single-cluster fragility.** Cluster-robust/GEE inference with one cluster in a
|
||||||
|
group can produce artificially small SEs — most visibly the learning-rate
|
||||||
|
interaction for the n=1 groups; do not read those p-values literally.
|
||||||
|
- **Mixed-model confounding.** With a per-subject random intercept, a group made
|
||||||
|
of one subject is partly confounded with that subject's random intercept, so its
|
||||||
|
fixed effect is shrunk.
|
||||||
|
- **Rate vs count.** "Success" alone is a count; the rate model (success/attempts)
|
||||||
|
is the fairer accuracy comparison when attempt counts differ.
|
||||||
|
|
||||||
|
## Files and usage
|
||||||
|
|
||||||
|
- `tdcs_glm.py` — the analysis (run it directly).
|
||||||
|
- `tdcs_reach_data.csv` — the verified long-format data (`subject, group, day, success, total`).
|
||||||
|
- `tdcs_learning_curves*.png` — per-group learning curves (count and rate panels).
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 analysis/tdcs_glm.py # all training days, 5 groups
|
||||||
|
python3 analysis/tdcs_glm.py --max-day 10 # restrict to days 0–10
|
||||||
|
python3 analysis/tdcs_glm.py --merge # assume Right==B2 and Box-A==A2 (3 groups)
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires: pandas, numpy, scipy, statsmodels, matplotlib.
|
||||||
Binary file not shown.
@@ -0,0 +1,440 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
tDCS reaching study — GLM hypothesis tests.
|
||||||
|
|
||||||
|
QUESTION
|
||||||
|
Two KNOWN conditions anchor the scale and differ from each other:
|
||||||
|
H2 (anchor check): Electrode-Box-B2 performs BETTER than Electrode-Box-A2.
|
||||||
|
Two UNKNOWN conditions then get CLASSIFIED against those anchors:
|
||||||
|
For "Electrode-Box-A" and for "Right-Electrode", is each one
|
||||||
|
A2-like or B2-like? (We do not assume either belongs to B2.)
|
||||||
|
Each unknown is compared to BOTH anchors; a group it cannot be
|
||||||
|
distinguished from is its likely class.
|
||||||
|
|
||||||
|
Each comparison is examined through THREE complementary GLMs:
|
||||||
|
|
||||||
|
(A) LEVEL / count Poisson GEE on `success` (successful reaches per
|
||||||
|
session), adjusting for training day. Answers
|
||||||
|
"who makes more successful reaches overall?"
|
||||||
|
|
||||||
|
(B) LEVEL / rate Binomial GLM on success / attempts
|
||||||
|
(`success` out of `total`), cluster-robust.
|
||||||
|
Controls for differing numbers of attempts —
|
||||||
|
"who is more accurate per attempt?"
|
||||||
|
|
||||||
|
(C) LEARNING RATE Poisson GEE with a group x day interaction.
|
||||||
|
Answers "do the groups improve at different
|
||||||
|
RATES?" (slope of the learning curve).
|
||||||
|
|
||||||
|
All models adjust for time with day (and day^2 for the plateau) and account
|
||||||
|
for repeated measures on each subject via GEE (subject = cluster,
|
||||||
|
exchangeable correlation, robust SE) or cluster-robust standard errors.
|
||||||
|
Group is coded with Electrode-Box-B2 as the REFERENCE, so H1 and H2 read
|
||||||
|
directly off the group terms:
|
||||||
|
Right-Electrode term -> H1 (expect ~0 / non-significant)
|
||||||
|
Electrode-Box-A2 term -> H2 (expect < 0: B2 above A2)
|
||||||
|
exp(coef) is an incidence-rate ratio (count model) or odds ratio (rate
|
||||||
|
model) relative to Box-B2.
|
||||||
|
|
||||||
|
CAVEATS
|
||||||
|
* Tiny groups: Box-A2 n=3, Box-B2 n=3, Naive n=4, and each UNKNOWN
|
||||||
|
(Electrode-Box-A, Right-Electrode) has only n=1 SUBJECT. Classifying a
|
||||||
|
1-subject condition is weak: "matches anchor X" means "not statistically
|
||||||
|
distinguishable from X", NOT proof of equivalence, and cluster-robust
|
||||||
|
inference with a single cluster in a group is fragile (can show
|
||||||
|
artificially small SEs, especially in the learning-rate interaction).
|
||||||
|
|
||||||
|
DATA PROVENANCE
|
||||||
|
analysis/tdcs_reach_data.csv was reconstructed from the experiment
|
||||||
|
database (experiment "tDCS", outcome = analysis_summary.counts.Success,
|
||||||
|
attempts = analysis_summary.total, x-axis = the "# Days Reach" field) and
|
||||||
|
verified cell-by-cell against the exported matrix (71/71 unambiguous
|
||||||
|
cells on days 0-5 matched exactly).
|
||||||
|
|
||||||
|
USAGE
|
||||||
|
python3 analysis/tdcs_glm.py
|
||||||
|
Requires: pandas, numpy, scipy, statsmodels, matplotlib.
|
||||||
|
Writes: analysis/tdcs_learning_curves.png
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import argparse
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
import patsy
|
||||||
|
import statsmodels.api as sm
|
||||||
|
import statsmodels.formula.api as smf
|
||||||
|
from scipy import stats
|
||||||
|
import matplotlib
|
||||||
|
matplotlib.use("Agg")
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
|
||||||
|
REF = "Electrode-Box-B2"
|
||||||
|
# Two KNOWN anchor conditions that differ (Box-B2 > Box-A2, see H2).
|
||||||
|
ANCHOR_LOW = "Electrode-Box-A2"
|
||||||
|
ANCHOR_HIGH = "Electrode-Box-B2"
|
||||||
|
# Two UNKNOWN conditions to classify: is each one A2-like or B2-like?
|
||||||
|
CANON_UNKNOWNS = ["Electrode-Box-A", "Right-Electrode"]
|
||||||
|
ALL_GROUPS = ["Naive", ANCHOR_LOW, ANCHOR_HIGH] + CANON_UNKNOWNS
|
||||||
|
# The active group config is set at runtime (after any --merge/--assign remap).
|
||||||
|
UNKNOWNS = list(CANON_UNKNOWNS)
|
||||||
|
MAIN = list(ALL_GROUPS)
|
||||||
|
OTHERS = [g for g in MAIN if g != REF]
|
||||||
|
# --merge preset: assume each unknown resolves into an anchor.
|
||||||
|
MERGE_MAP = {"Electrode-Box-A": ANCHOR_LOW, "Right-Electrode": ANCHOR_HIGH}
|
||||||
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
DATA = os.path.join(HERE, "tdcs_reach_data.csv")
|
||||||
|
PLOT = os.path.join(HERE, "tdcs_learning_curves.png")
|
||||||
|
RHS = f"C(group, Treatment('{REF}')) + day_c + day_c2"
|
||||||
|
|
||||||
|
|
||||||
|
def gterm(name):
|
||||||
|
"""statsmodels/patsy column name for a group level (vs the reference)."""
|
||||||
|
return f"C(group, Treatment('{REF}'))[T.{name}]"
|
||||||
|
|
||||||
|
|
||||||
|
def xterm(name):
|
||||||
|
"""Interaction column name (group level x day slope)."""
|
||||||
|
return f"C(group, Treatment('{REF}'))[T.{name}]:day_c"
|
||||||
|
|
||||||
|
|
||||||
|
def load(max_day=None, mapping=None):
|
||||||
|
df = pd.read_csv(DATA)
|
||||||
|
if mapping: # remap unknown -> target group
|
||||||
|
df["group"] = df["group"].map(lambda g: mapping.get(g, g))
|
||||||
|
df = df[df["group"].isin(ALL_GROUPS)].copy() # keep canonical groups
|
||||||
|
df = df[df["day"] >= 0].copy() # training days only (day 0..)
|
||||||
|
if max_day is not None:
|
||||||
|
df = df[df["day"] <= max_day].copy() # restrict the analysis window
|
||||||
|
df["day_c"] = df["day"] - df["day"].mean() # center day
|
||||||
|
df["day_c2"] = df["day_c"] ** 2
|
||||||
|
present = [g for g in ALL_GROUPS if g != REF and g in set(df["group"])]
|
||||||
|
df["group"] = pd.Categorical(df["group"], categories=[REF] + present)
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
def contrast(res, tname):
|
||||||
|
"""coef, robust SE, p, effect=exp(coef) and its 95% CI for one term."""
|
||||||
|
ci = res.conf_int()
|
||||||
|
lo, hi = ci.loc[tname]
|
||||||
|
return dict(coef=res.params[tname], se=res.bse[tname], p=res.pvalues[tname],
|
||||||
|
eff=np.exp(res.params[tname]), lo=np.exp(lo), hi=np.exp(hi))
|
||||||
|
|
||||||
|
|
||||||
|
def one_sided_p_below(c):
|
||||||
|
"""One-sided p for H: coef < 0 (i.e. Box-B2 above the compared group)."""
|
||||||
|
return stats.norm.cdf(c["coef"] / c["se"])
|
||||||
|
|
||||||
|
|
||||||
|
def diff_contrast(res, a, b):
|
||||||
|
"""Compare group `a` vs group `b` (both coded relative to REF) via a linear
|
||||||
|
contrast. Returns coef, robust SE, p, effect=exp(coef)=ratio a/b, and CI.
|
||||||
|
Works for any pair, including b == REF (then it is just the `a` term)."""
|
||||||
|
names = list(res.params.index)
|
||||||
|
v = np.zeros(len(names))
|
||||||
|
if a != REF:
|
||||||
|
v[names.index(gterm(a))] += 1.0
|
||||||
|
if b != REF:
|
||||||
|
v[names.index(gterm(b))] -= 1.0
|
||||||
|
tt = res.t_test(v)
|
||||||
|
coef = float(np.ravel(tt.effect)[0])
|
||||||
|
se = float(np.ravel(tt.sd)[0])
|
||||||
|
p = float(np.ravel(tt.pvalue)[0])
|
||||||
|
lo, hi = np.ravel(tt.conf_int())[:2]
|
||||||
|
return dict(coef=coef, se=se, p=p, eff=np.exp(coef),
|
||||||
|
lo=np.exp(lo), hi=np.exp(hi))
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------- models
|
||||||
|
def fit_count_level(df):
|
||||||
|
"""(A) Poisson GEE on success counts; group main effects vs Box-B2."""
|
||||||
|
return smf.gee(f"success ~ {RHS}", groups="subject", data=df,
|
||||||
|
family=sm.families.Poisson(),
|
||||||
|
cov_struct=sm.cov_struct.Exchangeable()).fit()
|
||||||
|
|
||||||
|
|
||||||
|
def fit_rate_level(df):
|
||||||
|
"""(B) Binomial GLM on success/attempts with cluster-robust SE.
|
||||||
|
Zero-attempt sessions (total==0) carry no rate information and are dropped."""
|
||||||
|
d = df[df["total"] > 0].copy()
|
||||||
|
dropped = len(df) - len(d)
|
||||||
|
if dropped:
|
||||||
|
print(f"[rate model] dropped {dropped} zero-attempt session(s) (undefined rate)")
|
||||||
|
X = patsy.dmatrix(RHS, d, return_type="dataframe")
|
||||||
|
endog = np.column_stack([d["success"].values,
|
||||||
|
(d["total"] - d["success"]).values])
|
||||||
|
return sm.GLM(endog, X, family=sm.families.Binomial()).fit(
|
||||||
|
cov_type="cluster", cov_kwds={"groups": d["subject"].values})
|
||||||
|
|
||||||
|
|
||||||
|
def fit_learning_rate(df):
|
||||||
|
"""(C) Poisson GEE with group x day interaction (slope differences)."""
|
||||||
|
return smf.gee(
|
||||||
|
f"success ~ C(group, Treatment('{REF}')) * day_c + day_c2",
|
||||||
|
groups="subject", data=df, family=sm.families.Poisson(),
|
||||||
|
cov_struct=sm.cov_struct.Exchangeable()).fit()
|
||||||
|
|
||||||
|
|
||||||
|
def fit_mixed_count(df):
|
||||||
|
"""Subject random-intercept Poisson mixed model (MAP / Laplace).
|
||||||
|
|
||||||
|
Adds a per-subject random intercept so each subject has its own baseline
|
||||||
|
level; the group fixed effects are then estimated after allowing for that
|
||||||
|
individual variation. A subject-specific complement to the population-
|
||||||
|
average GEE. statsmodels has no frequentist Poisson GLMM, so this is the
|
||||||
|
posterior-mode (MAP) fit with a Laplace covariance for the fixed effects
|
||||||
|
(fit_vb failed to converge on these large counts; MAP is stable)."""
|
||||||
|
import warnings
|
||||||
|
from statsmodels.genmod.bayes_mixed_glm import PoissonBayesMixedGLM
|
||||||
|
vc = {"subject": "0 + C(subject)"} # random intercept per subject
|
||||||
|
model = PoissonBayesMixedGLM.from_formula(f"success ~ {RHS}", vc, df)
|
||||||
|
with warnings.catch_warnings():
|
||||||
|
# MAP stops at |gradient|~5e-5 (effectively converged); silence the
|
||||||
|
# over-strict "did not converge" notice.
|
||||||
|
warnings.filterwarnings("ignore", message="Laplace fitting did not converge")
|
||||||
|
return model.fit_map()
|
||||||
|
|
||||||
|
|
||||||
|
def mixed_contrast(res, tname):
|
||||||
|
"""coef, SD, approx p and IRR + 95% interval from a BayesMixedGLM fit."""
|
||||||
|
names = list(res.model.exog_names)
|
||||||
|
i = names.index(tname)
|
||||||
|
mean, sd = float(res.fe_mean[i]), float(res.fe_sd[i])
|
||||||
|
p = 2.0 * stats.norm.sf(abs(mean / sd))
|
||||||
|
return dict(coef=mean, se=sd, p=p, eff=np.exp(mean),
|
||||||
|
lo=np.exp(mean - 1.96 * sd), hi=np.exp(mean + 1.96 * sd))
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------------------- reports
|
||||||
|
def describe(df):
|
||||||
|
print("=" * 78)
|
||||||
|
print("DESCRIPTIVES")
|
||||||
|
print("=" * 78)
|
||||||
|
grp = df.groupby("group", observed=True)
|
||||||
|
tbl = pd.DataFrame({
|
||||||
|
"n_subj": grp["subject"].nunique(),
|
||||||
|
"n_sessions": grp.size(),
|
||||||
|
"mean_success": grp["success"].mean().round(1),
|
||||||
|
"mean_rate": (grp["success"].sum() / grp["total"].sum()).round(3),
|
||||||
|
"max_day": grp["day"].max(),
|
||||||
|
})
|
||||||
|
print(tbl.to_string())
|
||||||
|
|
||||||
|
|
||||||
|
def report_level(title, res, effect_label):
|
||||||
|
print("\n" + "=" * 78)
|
||||||
|
print(title)
|
||||||
|
print("=" * 78)
|
||||||
|
print(f"Effect vs {REF} ({effect_label} relative to Box-B2):")
|
||||||
|
h = f"{'group':20s} {'effect':>7s} {'95% CI':>16s} {'coef':>8s} {'SE':>7s} {'p':>9s}"
|
||||||
|
print(h)
|
||||||
|
print("-" * len(h))
|
||||||
|
out = {}
|
||||||
|
for g in OTHERS:
|
||||||
|
c = contrast(res, gterm(g))
|
||||||
|
out[g] = c
|
||||||
|
ci = f"[{c['lo']:.2f}, {c['hi']:.2f}]"
|
||||||
|
print(f"{g:20s} {c['eff']:7.3f} {ci:>16s} {c['coef']:8.3f} {c['se']:7.3f} {c['p']:9.4f}")
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def report_learning(res):
|
||||||
|
print("\n" + "=" * 78)
|
||||||
|
print("(C) LEARNING RATE — Poisson GEE, group x day interaction")
|
||||||
|
print("=" * 78)
|
||||||
|
base = res.params["day_c"]
|
||||||
|
print(f"Box-B2 learning slope: {np.exp(base):.3f}x successes per training day "
|
||||||
|
f"(baseline).")
|
||||||
|
print("Slope DIFFERENCE vs Box-B2 (exp(coef) = per-day multiplier on the rate ratio):")
|
||||||
|
h = f"{'group':20s} {'slopeΔ/day':>10s} {'coef':>8s} {'SE':>7s} {'p':>9s}"
|
||||||
|
print(h)
|
||||||
|
print("-" * len(h))
|
||||||
|
for g in OTHERS:
|
||||||
|
c = contrast(res, xterm(g))
|
||||||
|
print(f"{g:20s} {c['eff']:10.3f} {c['coef']:8.3f} {c['se']:7.3f} {c['p']:9.4f}")
|
||||||
|
# Joint Wald test: do ANY groups differ from Box-B2 in learning rate?
|
||||||
|
names = list(res.params.index)
|
||||||
|
idx = [names.index(xterm(g)) for g in OTHERS]
|
||||||
|
R = np.zeros((len(idx), len(names)))
|
||||||
|
for i, j in enumerate(idx):
|
||||||
|
R[i, j] = 1.0
|
||||||
|
jt = res.wald_test(R, scalar=True)
|
||||||
|
print(f"\nJoint test (all group x day interactions = 0): "
|
||||||
|
f"chi2={float(jt.statistic):.2f}, df={len(idx)}, p={float(jt.pvalue):.4f}")
|
||||||
|
print(" -> small p = groups improve at DIFFERENT rates; large p = parallel learning.")
|
||||||
|
|
||||||
|
|
||||||
|
def report_anchor_only(df):
|
||||||
|
"""A2-vs-B2 ONLY: refit on just the two anchors so nothing else influences
|
||||||
|
the shared day terms / dispersion. B2 is the reference, so the single group
|
||||||
|
term is the A2-vs-B2 comparison."""
|
||||||
|
print("\n" + "=" * 78)
|
||||||
|
print("ANCHORS ONLY — Box-A2 vs Box-B2 (two-group models)")
|
||||||
|
print("=" * 78)
|
||||||
|
d = df[df["group"].isin([ANCHOR_LOW, ANCHOR_HIGH])].copy()
|
||||||
|
d["day_c"] = d["day"] - d["day"].mean()
|
||||||
|
d["day_c2"] = d["day_c"] ** 2
|
||||||
|
d["group"] = pd.Categorical(d["group"], categories=[ANCHOR_HIGH, ANCHOR_LOW])
|
||||||
|
n_sub = d.groupby("group", observed=True)["subject"].nunique().to_dict()
|
||||||
|
print(f"subjects: Box-B2={n_sub.get(ANCHOR_HIGH)}, Box-A2={n_sub.get(ANCHOR_LOW)}"
|
||||||
|
f" sessions: {len(d)}")
|
||||||
|
print(f"formula: success ~ C(group, Treatment('{REF}')) + day_c + day_c2")
|
||||||
|
for label, res in [("count/level (Poisson GEE)", fit_count_level(d)),
|
||||||
|
("rate/level (Binomial cluster-robust)", fit_rate_level(d))]:
|
||||||
|
c = contrast(res, gterm(ANCHOR_LOW))
|
||||||
|
p1 = one_sided_p_below(c)
|
||||||
|
tag = "SUPPORTED" if (c["coef"] < 0 and p1 < 0.05) else "not supported"
|
||||||
|
print(f" [{label}] A2/B2 = {c['eff']:.2f} [{c['lo']:.2f}, {c['hi']:.2f}] "
|
||||||
|
f"=> B2 = {1.0 / c['eff']:.2f}x A2, one-sided p={p1:.4f} -> H2 {tag}")
|
||||||
|
|
||||||
|
|
||||||
|
def report_mixed(df):
|
||||||
|
"""Subject random-intercept Poisson mixed model (sensitivity vs GEE)."""
|
||||||
|
print("\n" + "=" * 78)
|
||||||
|
print("SENSITIVITY — Poisson MIXED model, per-subject random intercept (MAP/Laplace)")
|
||||||
|
print("=" * 78)
|
||||||
|
try:
|
||||||
|
res = fit_mixed_count(df)
|
||||||
|
except Exception as exc: # VB can be finicky; degrade gracefully
|
||||||
|
print(f" (mixed model skipped: {type(exc).__name__}: {exc})")
|
||||||
|
return
|
||||||
|
print("Group effect vs Box-B2 (IRR; each subject given its own baseline):")
|
||||||
|
h = f"{'group':20s} {'IRR':>7s} {'~95% CI':>16s} {'mean':>8s} {'SD':>7s} {'~p':>9s}"
|
||||||
|
print(h)
|
||||||
|
print("-" * len(h))
|
||||||
|
for g in OTHERS:
|
||||||
|
c = mixed_contrast(res, gterm(g))
|
||||||
|
ci = f"[{c['lo']:.2f}, {c['hi']:.2f}]"
|
||||||
|
print(f"{g:20s} {c['eff']:7.3f} {ci:>16s} {c['coef']:8.3f} {c['se']:7.3f} {c['p']:9.4f}")
|
||||||
|
print(f" subject random-intercept SD = {float(np.exp(res.vcp_mean[0])):.3f} "
|
||||||
|
"(log scale); Laplace SEs, treat p-values as approximate.")
|
||||||
|
print(" Compare directions/magnitudes with the GEE table (they should agree).")
|
||||||
|
print(" Single-subject groups (Box-A, Right-Electrode) are partly confounded")
|
||||||
|
print(" with their own random intercept here, so their effects are shrunk.")
|
||||||
|
|
||||||
|
|
||||||
|
def plot_curves(df, plot_path):
|
||||||
|
fig, ax = plt.subplots(1, 2, figsize=(12, 4.8))
|
||||||
|
for g in MAIN:
|
||||||
|
s = df[df["group"] == g]
|
||||||
|
d = s.groupby("day").apply(
|
||||||
|
lambda x: pd.Series({"succ": x["success"].mean(),
|
||||||
|
"rate": x["success"].sum() / x["total"].sum()}),
|
||||||
|
include_groups=False)
|
||||||
|
ax[0].plot(d.index, d["succ"], marker="o", ms=3, label=g)
|
||||||
|
ax[1].plot(d.index, d["rate"], marker="o", ms=3, label=g)
|
||||||
|
ax[0].set(xlabel="# Days Reach (training day)", ylabel="mean successful reaches",
|
||||||
|
title="(A) Success count learning curves")
|
||||||
|
ax[1].set(xlabel="# Days Reach (training day)", ylabel="success rate (success/attempts)",
|
||||||
|
title="(B) Success-rate learning curves")
|
||||||
|
for a in ax:
|
||||||
|
a.legend(fontsize=8)
|
||||||
|
a.grid(alpha=0.3)
|
||||||
|
fig.tight_layout()
|
||||||
|
fig.savefig(plot_path, dpi=120)
|
||||||
|
print(f"\nSaved learning-curve plot -> {plot_path}")
|
||||||
|
|
||||||
|
|
||||||
|
def classify_one(res, unknown):
|
||||||
|
"""Compare an unknown group against BOTH anchors; return a label + detail."""
|
||||||
|
vlo = diff_contrast(res, unknown, ANCHOR_LOW) # unknown vs Box-A2
|
||||||
|
vhi = diff_contrast(res, unknown, ANCHOR_HIGH) # unknown vs Box-B2
|
||||||
|
like_lo = vlo["p"] >= 0.05 # indistinguishable from Box-A2
|
||||||
|
like_hi = vhi["p"] >= 0.05 # indistinguishable from Box-B2
|
||||||
|
if like_hi and not like_lo:
|
||||||
|
verdict = "B2-like (differs from A2, matches B2)"
|
||||||
|
elif like_lo and not like_hi:
|
||||||
|
verdict = "A2-like (differs from B2, matches A2)"
|
||||||
|
elif like_lo and like_hi:
|
||||||
|
# tie: point toward whichever ratio is closer to 1 on the log scale
|
||||||
|
nearer = ANCHOR_HIGH if abs(vhi["coef"]) < abs(vlo["coef"]) else ANCHOR_LOW
|
||||||
|
verdict = f"AMBIGUOUS (matches both; numerically nearer {nearer})"
|
||||||
|
else:
|
||||||
|
verdict = "UNLIKE BOTH (differs from A2 and B2)"
|
||||||
|
return vlo, vhi, verdict
|
||||||
|
|
||||||
|
|
||||||
|
def verdicts(count, rate):
|
||||||
|
print("\n" + "=" * 78)
|
||||||
|
print("VERDICTS")
|
||||||
|
print("=" * 78)
|
||||||
|
|
||||||
|
print("\nAnchor check — H2: Box-B2 BETTER than Box-A2 (the two anchors must differ)")
|
||||||
|
for label, res in [("count/level", count), ("rate/level ", rate)]:
|
||||||
|
c = contrast(res, gterm(ANCHOR_LOW))
|
||||||
|
p1 = one_sided_p_below(c)
|
||||||
|
tag = "SUPPORTED" if (c["coef"] < 0 and p1 < 0.05) else "not supported"
|
||||||
|
print(f" [{label}] Box-B2 = {1.0 / c['eff']:.2f}x Box-A2 "
|
||||||
|
f"(one-sided p={p1:.4f}) -> {tag}")
|
||||||
|
|
||||||
|
if not UNKNOWNS:
|
||||||
|
print("\n(No unknown groups — they were merged into the anchors; "
|
||||||
|
"only the H2 anchor contrast applies.)")
|
||||||
|
return
|
||||||
|
print("\nClassification — is each UNKNOWN condition A2-like or B2-like?")
|
||||||
|
print("(ratio >1 = above that anchor; p = differs from that anchor)")
|
||||||
|
for u in UNKNOWNS:
|
||||||
|
print(f"\n {u}:")
|
||||||
|
for label, res in [("count/level", count), ("rate/level ", rate)]:
|
||||||
|
vlo, vhi, verdict = classify_one(res, u)
|
||||||
|
print(f" [{label}] vs Box-A2: {vlo['eff']:.2f}x "
|
||||||
|
f"[{vlo['lo']:.2f},{vlo['hi']:.2f}] p={vlo['p']:.3f} "
|
||||||
|
f"vs Box-B2: {vhi['eff']:.2f}x [{vhi['lo']:.2f},{vhi['hi']:.2f}] "
|
||||||
|
f"p={vhi['p']:.3f}")
|
||||||
|
print(f" -> {verdict}")
|
||||||
|
print("\n NOTE: each unknown has ONLY 1 subject. 'Matches' means 'not statistically")
|
||||||
|
print(" distinguishable' — weak evidence at n=1, not proof of equivalence.")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description="tDCS GLM hypothesis tests")
|
||||||
|
ap.add_argument("--max-day", type=int, default=None,
|
||||||
|
help="restrict analysis to training days 0..MAX_DAY (default: all)")
|
||||||
|
ap.add_argument("--merge", action="store_true",
|
||||||
|
help="preset: Electrode-Box-A->Box-A2 and Right-Electrode->Box-B2")
|
||||||
|
ap.add_argument("--assign", type=str, default=None,
|
||||||
|
help="custom merges 'UNKNOWN=TARGET,...', e.g. "
|
||||||
|
"'Electrode-Box-A=Electrode-Box-B2,Right-Electrode=Electrode-Box-B2'")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
mapping = {}
|
||||||
|
if args.merge:
|
||||||
|
mapping.update(MERGE_MAP)
|
||||||
|
if args.assign:
|
||||||
|
for pair in args.assign.split(","):
|
||||||
|
k, v = pair.split("=")
|
||||||
|
mapping[k.strip()] = v.strip()
|
||||||
|
|
||||||
|
df = load(args.max_day, mapping)
|
||||||
|
|
||||||
|
global MAIN, OTHERS, UNKNOWNS
|
||||||
|
present = set(df["group"])
|
||||||
|
MAIN = [g for g in ALL_GROUPS if g in present]
|
||||||
|
OTHERS = [g for g in MAIN if g != REF]
|
||||||
|
UNKNOWNS = [g for g in CANON_UNKNOWNS if g in present]
|
||||||
|
|
||||||
|
window = f"days 0-{args.max_day}" if args.max_day is not None else "all training days (0+)"
|
||||||
|
tag = f"_d0-{args.max_day}" if args.max_day is not None else ""
|
||||||
|
tag += "_merged" if mapping else ""
|
||||||
|
plot_path = os.path.join(HERE, f"tdcs_learning_curves{tag}.png")
|
||||||
|
mode = ("assign " + ", ".join(f"{k.replace('Electrode-','')}->{v.replace('Electrode-','')}"
|
||||||
|
for k, v in mapping.items())) if mapping \
|
||||||
|
else "5 groups (anchors + unknowns)"
|
||||||
|
print(f"ANALYSIS WINDOW: {window} | MODE: {mode} | observations: {len(df)}")
|
||||||
|
describe(df)
|
||||||
|
count = fit_count_level(df)
|
||||||
|
rate = fit_rate_level(df)
|
||||||
|
learn = fit_learning_rate(df)
|
||||||
|
report_level("(A) LEVEL / COUNT — Poisson GEE on successful reaches",
|
||||||
|
count, "incidence-rate ratio")
|
||||||
|
report_level("(B) LEVEL / RATE — Binomial GLM on success/attempts (cluster-robust)",
|
||||||
|
rate, "odds ratio")
|
||||||
|
report_learning(learn)
|
||||||
|
report_anchor_only(df)
|
||||||
|
report_mixed(df)
|
||||||
|
verdicts(count, rate)
|
||||||
|
plot_curves(df, plot_path)
|
||||||
|
print("\nDone. See the module docstring for modeling choices and caveats.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 161 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 154 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 133 KiB |
@@ -0,0 +1,194 @@
|
|||||||
|
subject,group,day,success,total
|
||||||
|
Vu-vuong,Naive,0,18,61
|
||||||
|
Vu-vuong,Naive,1,35,91
|
||||||
|
Vu-vuong,Naive,2,61,127
|
||||||
|
Vu-vuong,Naive,3,34,146
|
||||||
|
Vu-vuong,Naive,4,61,141
|
||||||
|
Vu-vuong,Naive,5,37,151
|
||||||
|
Vu-vuong,Naive,6,49,131
|
||||||
|
Vu-vuong,Naive,7,65,149
|
||||||
|
Vu-vuong,Naive,8,67,141
|
||||||
|
Vu-vuong,Naive,9,73,140
|
||||||
|
Vu-vuong,Naive,10,72,131
|
||||||
|
Vu-vuong,Naive,11,89,158
|
||||||
|
Vu-vuong,Naive,12,97,154
|
||||||
|
Vu-vuong,Naive,13,93,145
|
||||||
|
Vu-vuong,Naive,14,75,147
|
||||||
|
Vu-vuong,Naive,15,68,142
|
||||||
|
Vu-vuong,Naive,16,96,162
|
||||||
|
Vu-vuong,Naive,17,68,144
|
||||||
|
Vu-vuong,Naive,18,81,134
|
||||||
|
Vu-vuong,Naive,19,66,144
|
||||||
|
Vu-vuong,Naive,20,84,127
|
||||||
|
Vu-vuong,Naive,21,66,133
|
||||||
|
Khoai-tay-1,Electrode-Box-A,-2,1,18
|
||||||
|
Khoai-tay-1,Electrode-Box-A,-1,3,24
|
||||||
|
Khoai-tay-1,Electrode-Box-A,0,14,62
|
||||||
|
Khoai-tay-1,Electrode-Box-A,1,22,83
|
||||||
|
Khoai-tay-1,Electrode-Box-A,2,6,79
|
||||||
|
Khoai-tay-1,Electrode-Box-A,3,28,97
|
||||||
|
Khoai-tay-1,Electrode-Box-A,4,60,134
|
||||||
|
Khoai-tay-1,Electrode-Box-A,5,75,138
|
||||||
|
Khoai-tay-1,Electrode-Box-A,6,81,137
|
||||||
|
Khoai-tay-1,Electrode-Box-A,7,78,147
|
||||||
|
Khoai-tay-1,Electrode-Box-A,8,91,132
|
||||||
|
Khoai-tay-1,Electrode-Box-A,9,99,146
|
||||||
|
Khoai-tay-1,Electrode-Box-A,10,94,143
|
||||||
|
Khoai-tay-1,Electrode-Box-A,11,110,156
|
||||||
|
Khoai-tay-1,Electrode-Box-A,12,105,143
|
||||||
|
Khoai-tay-1,Electrode-Box-A,13,106,153
|
||||||
|
Khoai-tay-1,Electrode-Box-A,14,91,152
|
||||||
|
Banh-mi-2,Electrode-Box-A2,-1,2,42
|
||||||
|
Banh-mi-2,Electrode-Box-A2,0,25,97
|
||||||
|
Banh-mi-2,Electrode-Box-A2,1,22,101
|
||||||
|
Banh-mi-2,Electrode-Box-A2,2,22,119
|
||||||
|
Banh-mi-2,Electrode-Box-A2,3,29,118
|
||||||
|
Banh-mi-2,Electrode-Box-A2,4,27,136
|
||||||
|
Banh-mi-2,Electrode-Box-A2,5,43,146
|
||||||
|
Banh-mi-2,Electrode-Box-A2,6,74,146
|
||||||
|
Banh-mi-2,Electrode-Box-A2,7,70,148
|
||||||
|
Banh-mi-2,Electrode-Box-A2,8,65,130
|
||||||
|
Banh-mi-2,Electrode-Box-A2,9,79,151
|
||||||
|
Banh-mi-2,Electrode-Box-A2,10,93,152
|
||||||
|
Egg-tart-2,Electrode-Box-A2,0,9,32
|
||||||
|
Egg-tart-2,Electrode-Box-A2,1,2,38
|
||||||
|
Egg-tart-2,Electrode-Box-A2,2,31,93
|
||||||
|
Egg-tart-2,Electrode-Box-A2,3,44,101
|
||||||
|
Egg-tart-2,Electrode-Box-A2,4,54,131
|
||||||
|
Egg-tart-2,Electrode-Box-A2,5,84,139
|
||||||
|
Egg-tart-2,Electrode-Box-A2,6,85,145
|
||||||
|
Egg-tart-2,Electrode-Box-A2,7,79,143
|
||||||
|
Egg-tart-2,Electrode-Box-A2,8,76,131
|
||||||
|
Egg-tart-2,Electrode-Box-A2,9,88,149
|
||||||
|
Egg-tart-2,Electrode-Box-A2,10,81,151
|
||||||
|
Egg-tart-2,Electrode-Box-A2,11,78,152
|
||||||
|
Egg-tart-2,Electrode-Box-A2,12,96,155
|
||||||
|
Egg-tart-2,Electrode-Box-A2,13,84,155
|
||||||
|
Root-beer-2,Electrode-Box-A2,0,22,74
|
||||||
|
Root-beer-2,Electrode-Box-A2,1,31,87
|
||||||
|
Root-beer-2,Electrode-Box-A2,2,49,134
|
||||||
|
Root-beer-2,Electrode-Box-A2,3,31,89
|
||||||
|
Root-beer-2,Electrode-Box-A2,4,60,140
|
||||||
|
Root-beer-2,Electrode-Box-A2,5,84,147
|
||||||
|
Banh-mi-1,Electrode-Box-B2,0,13,84
|
||||||
|
Banh-mi-1,Electrode-Box-B2,1,35,86
|
||||||
|
Banh-mi-1,Electrode-Box-B2,2,47,110
|
||||||
|
Banh-mi-1,Electrode-Box-B2,3,65,140
|
||||||
|
Banh-mi-1,Electrode-Box-B2,4,70,127
|
||||||
|
Banh-mi-1,Electrode-Box-B2,5,102,142
|
||||||
|
Banh-mi-1,Electrode-Box-B2,6,90,131
|
||||||
|
Banh-mi-1,Electrode-Box-B2,7,109,148
|
||||||
|
Banh-mi-1,Electrode-Box-B2,8,104,137
|
||||||
|
Banh-mi-1,Electrode-Box-B2,9,119,150
|
||||||
|
Banh-mi-1,Electrode-Box-B2,10,121,158
|
||||||
|
Banh-mi-1,Electrode-Box-B2,11,121,148
|
||||||
|
Banh-mi-1,Electrode-Box-B2,12,120,149
|
||||||
|
Banh-mi-1,Electrode-Box-B2,13,135,154
|
||||||
|
Egg-tart-1,Electrode-Box-B2,0,7,56
|
||||||
|
Egg-tart-1,Electrode-Box-B2,1,16,78
|
||||||
|
Egg-tart-1,Electrode-Box-B2,2,23,103
|
||||||
|
Egg-tart-1,Electrode-Box-B2,3,63,120
|
||||||
|
Egg-tart-1,Electrode-Box-B2,4,69,132
|
||||||
|
Egg-tart-1,Electrode-Box-B2,5,83,136
|
||||||
|
Egg-tart-1,Electrode-Box-B2,6,71,142
|
||||||
|
Egg-tart-1,Electrode-Box-B2,7,79,138
|
||||||
|
Egg-tart-1,Electrode-Box-B2,8,98,142
|
||||||
|
Egg-tart-1,Electrode-Box-B2,9,89,139
|
||||||
|
Egg-tart-1,Electrode-Box-B2,10,96,143
|
||||||
|
Egg-tart-1,Electrode-Box-B2,11,96,148
|
||||||
|
Egg-tart-1,Electrode-Box-B2,12,101,156
|
||||||
|
Egg-tart-1,Electrode-Box-B2,13,103,152
|
||||||
|
Egg-tart-1,Electrode-Box-B2,14,97,152
|
||||||
|
Root-beer-1,Electrode-Box-B2,0,11,85
|
||||||
|
Root-beer-1,Electrode-Box-B2,1,18,76
|
||||||
|
Root-beer-1,Electrode-Box-B2,2,40,105
|
||||||
|
Root-beer-1,Electrode-Box-B2,3,55,134
|
||||||
|
Root-beer-1,Electrode-Box-B2,4,75,136
|
||||||
|
Root-beer-1,Electrode-Box-B2,5,64,133
|
||||||
|
Root-beer-1,Electrode-Box-B2,6,104,139
|
||||||
|
Root-beer-1,Electrode-Box-B2,7,98,148
|
||||||
|
Root-beer-1,Electrode-Box-B2,8,81,145
|
||||||
|
Root-beer-1,Electrode-Box-B2,9,89,156
|
||||||
|
Root-beer-1,Electrode-Box-B2,10,105,158
|
||||||
|
Khoai-lang-2,Naive,0,0,0
|
||||||
|
Khoai-lang-2,Naive,1,0,0
|
||||||
|
Khoai-lang-2,Naive,2,10,47
|
||||||
|
Khoai-lang-2,Naive,3,11,52
|
||||||
|
Khoai-lang-2,Naive,4,9,56
|
||||||
|
Khoai-lang-2,Naive,5,34,95
|
||||||
|
Khoai-lang-2,Naive,6,21,72
|
||||||
|
Khoai-lang-2,Naive,7,23,99
|
||||||
|
Khoai-lang-2,Naive,8,64,136
|
||||||
|
Khoai-lang-2,Naive,9,75,131
|
||||||
|
Khoai-lang-2,Naive,10,63,134
|
||||||
|
Khoai-lang-2,Naive,11,59,139
|
||||||
|
Khoai-lang-2,Naive,12,51,129
|
||||||
|
Khoai-lang-2,Naive,13,73,143
|
||||||
|
Khoai-lang-2,Naive,14,82,136
|
||||||
|
Khoai-lang-2,Naive,15,70,145
|
||||||
|
Khoai-lang-2,Naive,16,76,135
|
||||||
|
Khoai-lang-2,Naive,17,76,150
|
||||||
|
Khoai-lang-2,Naive,18,63,122
|
||||||
|
Khoai-lang-2,Naive,19,48,116
|
||||||
|
Khoai-lang-2,Naive,20,65,134
|
||||||
|
Khoai-lang-2,Naive,21,75,131
|
||||||
|
Khoai-lang-2,Naive,22,98,146
|
||||||
|
Khoai-lang-2,Naive,23,88,139
|
||||||
|
Khoai-lang-2,Naive,24,94,148
|
||||||
|
Khoai-lang-2,Naive,25,56,102
|
||||||
|
Khoai-lang-2,Naive,26,75,143
|
||||||
|
Khoai-tay-2,Naive,1,21,68
|
||||||
|
Khoai-tay-2,Naive,2,6,79
|
||||||
|
Khoai-tay-2,Naive,3,0,83
|
||||||
|
Khoai-tay-2,Naive,4,28,87
|
||||||
|
Khoai-tay-2,Naive,5,31,125
|
||||||
|
Khoai-tay-2,Naive,6,62,144
|
||||||
|
Khoai-tay-2,Naive,7,87,141
|
||||||
|
Khoai-tay-2,Naive,8,105,152
|
||||||
|
Khoai-tay-2,Naive,9,75,148
|
||||||
|
Khoai-tay-2,Naive,10,101,149
|
||||||
|
Khoai-tay-2,Naive,11,102,154
|
||||||
|
Khoai-tay-2,Naive,12,95,144
|
||||||
|
Khoai-tay-2,Naive,13,77,149
|
||||||
|
Khoai-tay-2,Naive,14,91,153
|
||||||
|
OM-2,Naive,-2,0,0
|
||||||
|
OM-2,Naive,-1,5,18
|
||||||
|
OM-2,Naive,0,1,7
|
||||||
|
OM-2,Naive,1,11,33
|
||||||
|
OM-2,Naive,2,19,62
|
||||||
|
OM-2,Naive,3,29,117
|
||||||
|
OM-2,Naive,4,73,126
|
||||||
|
OM-2,Naive,5,53,135
|
||||||
|
OM-2,Naive,6,73,138
|
||||||
|
OM-2,Naive,7,80,131
|
||||||
|
OM-2,Naive,8,91,141
|
||||||
|
OM-2,Naive,9,90,135
|
||||||
|
OM-2,Naive,10,95,142
|
||||||
|
OM-2,Naive,11,60,133
|
||||||
|
OM-2,Naive,12,58,142
|
||||||
|
Khoai-lang-1,Right-Electrode,-4,0,0
|
||||||
|
Khoai-lang-1,Right-Electrode,-3,2,7
|
||||||
|
Khoai-lang-1,Right-Electrode,-1,0,0
|
||||||
|
Khoai-lang-1,Right-Electrode,0,3,69
|
||||||
|
Khoai-lang-1,Right-Electrode,1,18,97
|
||||||
|
Khoai-lang-1,Right-Electrode,2,27,84
|
||||||
|
Khoai-lang-1,Right-Electrode,3,46,121
|
||||||
|
Khoai-lang-1,Right-Electrode,4,65,143
|
||||||
|
Khoai-lang-1,Right-Electrode,5,76,141
|
||||||
|
Khoai-lang-1,Right-Electrode,6,79,146
|
||||||
|
Khoai-lang-1,Right-Electrode,7,81,153
|
||||||
|
Khoai-lang-1,Right-Electrode,8,98,144
|
||||||
|
Khoai-lang-1,Right-Electrode,9,101,149
|
||||||
|
Khoai-lang-1,Right-Electrode,10,103,150
|
||||||
|
Khoai-lang-1,Right-Electrode,11,113,154
|
||||||
|
Khoai-lang-1,Right-Electrode,12,117,148
|
||||||
|
Khoai-lang-1,Right-Electrode,13,114,146
|
||||||
|
Khoai-lang-1,Right-Electrode,14,119,132
|
||||||
|
Khoai-lang-1,Right-Electrode,15,111,156
|
||||||
|
Khoai-lang-1,Right-Electrode,16,74,156
|
||||||
|
Khoai-lang-1,Right-Electrode,17,109,139
|
||||||
|
Khoai-lang-1,Right-Electrode,18,88,138
|
||||||
|
Khoai-lang-1,Right-Electrode,19,98,140
|
||||||
|
Khoai-lang-1,Right-Electrode,20,118,153
|
||||||
|
Khoai-lang-1,Right-Electrode,21,109,146
|
||||||
|
Khoai-lang-1,Right-Electrode,22,106,141
|
||||||
|
@@ -0,0 +1,668 @@
|
|||||||
|
# Subject-Series Data Export Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Replace the export modal's assignable-axes UI with three pickers — X field (rows), Data value (cells), and Group by — where columns are always the subjects, matching the cross-subject metrics plot.
|
||||||
|
|
||||||
|
**Architecture:** Add X-coordinate helpers to `dataExport.js` (`numericFieldVal`, `listSampleMembers`, `buildSampleIndex`) and a focused `buildSubjectSeriesMatrix` that emits the existing `toCSV` shape. Rewrite `ExportDataModal.jsx` to the three pickers. Then delete the now-dead assignable-axes machinery.
|
||||||
|
|
||||||
|
**Tech Stack:** React 18, plain ES modules, Jest + `@testing-library/react` (jsdom). No new dependencies.
|
||||||
|
|
||||||
|
**Spec:** `docs/superpowers/specs/2026-07-19-subject-series-export-design.md`
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- `dataExport.js` stays pure: no React, no I/O.
|
||||||
|
- Cell value semantics: `0` and `''` are **real** values; only genuinely-missing lookups render blank (`''`), via the existing `hasValue` + `extractValue`.
|
||||||
|
- CSV: LF (`\n`) line endings; escape every cell via the existing `escapeCsvCell`.
|
||||||
|
- **X = `Date`** → rows are distinct `YYYY-MM-DD` dates, sorted.
|
||||||
|
- **X = `# Days Reach`** (`'__days__'`) → rows are `1, 2, 3, …`; the max over subjects of that subject's count of recorded statuses (a status must have `animal_id` and `date`). A cell for `(subject, N)` reads the subject's Nth recorded day, counting **all** records sorted ascending by date (per-status ordinal, Day 1 = earliest).
|
||||||
|
- **X = daily field** (its `fieldId`) → rows are the distinct **numeric** values (`numericFieldVal`), sorted ascending; non-numeric excluded. A cell for `(subject, v)` reads that subject's first status (by date) whose field equals `v`.
|
||||||
|
- Columns are **always** the subjects (`listSubjectMembers`), clustered by group then name. Entirely-blank sample rows are dropped; subject columns are always kept.
|
||||||
|
- Group by defaults to the saved field (`groupField` prop), validated against `__none__`/`__name__`/`__id__`/active `subjectTemplate` fields; coerced to `__none__` when unset or stale.
|
||||||
|
- CSV: line 1 is `Data,<metric label>`; blank line; a `Group,…` row only when grouping (missing group → `—`); header `= <X label>,<subject headers…>`; data rows `= <X value>,<values…>`. Filename via existing `csvFilename(title, dataLabel)`.
|
||||||
|
- Default X = `Date`; default Data = first parameter (`Total attempts`).
|
||||||
|
- Run tests from `frontend/`: `npx jest tests/dataExport.test.js`, `npx jest tests/ExportDataModal.test.jsx`, full `npm test`.
|
||||||
|
- Two suites (`tests/ExperimentCalendar.test.jsx`, `tests/ExperimentDayView.test.jsx`, 31 tests) fail for a PRE-EXISTING, unrelated reason and must be left as-is; changes must introduce no new failures.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: X-coordinate helpers
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `frontend/src/lib/dataExport.js` (add three exports; keep everything else for now)
|
||||||
|
- Test: `frontend/tests/dataExport.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: existing private `dateKey(date)`.
|
||||||
|
- Produces:
|
||||||
|
- `numericFieldVal(status, field): number|null` — `field.builtin ? status[field.key] : status.custom_fields[field.fieldId]`, `parseFloat`, non-numeric → null.
|
||||||
|
- `listSampleMembers(statuses, xField, dailyTemplate): Array<{ id, label, sampleValue }>` — row members for the chosen X (`xField` is `'__date__'`, `'__days__'`, or a daily `fieldId`).
|
||||||
|
- `buildSampleIndex(statuses, xField, dailyTemplate): Map<animalId, Map<sampleValue, status>>` — first status per `(subject, sampleValue)` wins, per-subject date-ordered.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing tests**
|
||||||
|
|
||||||
|
Append to `frontend/tests/dataExport.test.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
import { numericFieldVal, listSampleMembers, buildSampleIndex } from '../src/lib/dataExport';
|
||||||
|
|
||||||
|
const sDaily = [{ fieldId: 'c-w', key: 'w', label: 'Weight', builtin: false, active: true }];
|
||||||
|
const sStatuses = [
|
||||||
|
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', custom_fields: { 'c-w': '250' } },
|
||||||
|
{ animal_id: 'a1', date: '2026-07-03T00:00:00Z', custom_fields: { 'c-w': '260' } },
|
||||||
|
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', custom_fields: { 'c-w': '250' } },
|
||||||
|
];
|
||||||
|
|
||||||
|
describe('numericFieldVal', () => {
|
||||||
|
it('reads builtin via key and custom via fieldId, parsed to number', () => {
|
||||||
|
expect(numericFieldVal({ weight: '250' }, { builtin: true, key: 'weight' })).toBe(250);
|
||||||
|
expect(numericFieldVal({ custom_fields: { 'c-w': '12.5' } }, { builtin: false, fieldId: 'c-w' })).toBe(12.5);
|
||||||
|
});
|
||||||
|
it('returns null for non-numeric, missing, or no field', () => {
|
||||||
|
expect(numericFieldVal({ custom_fields: { 'c-w': 'abc' } }, { builtin: false, fieldId: 'c-w' })).toBeNull();
|
||||||
|
expect(numericFieldVal({}, { builtin: true, key: 'weight' })).toBeNull();
|
||||||
|
expect(numericFieldVal({ weight: 1 }, null)).toBeNull();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('listSampleMembers', () => {
|
||||||
|
it('date: distinct dates sorted', () => {
|
||||||
|
expect(listSampleMembers(sStatuses, '__date__', sDaily).map((m) => m.sampleValue)).toEqual(['2026-07-01', '2026-07-03']);
|
||||||
|
});
|
||||||
|
it('# days reach: 1..max records per subject', () => {
|
||||||
|
expect(listSampleMembers(sStatuses, '__days__', sDaily).map((m) => m.sampleValue)).toEqual([1, 2]);
|
||||||
|
});
|
||||||
|
it('daily field: distinct numeric values sorted ascending', () => {
|
||||||
|
expect(listSampleMembers(sStatuses, 'c-w', sDaily).map((m) => m.sampleValue)).toEqual([250, 260]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('buildSampleIndex', () => {
|
||||||
|
it('date coord maps animalId -> dateKey -> status', () => {
|
||||||
|
const idx = buildSampleIndex(sStatuses, '__date__', sDaily);
|
||||||
|
expect(idx.get('a1').get('2026-07-03').date).toContain('2026-07-03');
|
||||||
|
});
|
||||||
|
it('# days reach assigns per-subject ordinals in date order', () => {
|
||||||
|
const idx = buildSampleIndex(sStatuses, '__days__', sDaily);
|
||||||
|
expect(idx.get('a1').get(1).date).toContain('2026-07-01');
|
||||||
|
expect(idx.get('a1').get(2).date).toContain('2026-07-03');
|
||||||
|
expect(idx.get('a2').get(1).date).toContain('2026-07-01');
|
||||||
|
expect(idx.get('a2').has(2)).toBe(false);
|
||||||
|
});
|
||||||
|
it('daily field maps numeric value to first status by date', () => {
|
||||||
|
const idx = buildSampleIndex(sStatuses, 'c-w', sDaily);
|
||||||
|
expect(idx.get('a1').get(250).date).toContain('2026-07-01');
|
||||||
|
expect(idx.get('a1').get(260).date).toContain('2026-07-03');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run tests to verify they fail**
|
||||||
|
|
||||||
|
Run: `npx jest tests/dataExport.test.js -t "listSampleMembers"`
|
||||||
|
Expected: FAIL — `listSampleMembers is not defined`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Write minimal implementation**
|
||||||
|
|
||||||
|
Add to `frontend/src/lib/dataExport.js` (after `listExportableParams`, before the private `hasValue`/`dateKey` — note `dateKey` is defined lower in the file but hoisted as a function declaration, so referencing it here is fine):
|
||||||
|
|
||||||
|
```js
|
||||||
|
// Numeric value of a daily field on one status (mirrors the metrics plot's helper).
|
||||||
|
export function numericFieldVal(status, field) {
|
||||||
|
if (!field) return null;
|
||||||
|
const raw = field.builtin ? status?.[field.key] : status?.custom_fields?.[field.fieldId];
|
||||||
|
const n = parseFloat(raw);
|
||||||
|
return Number.isNaN(n) ? null : n;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ordered row members for the chosen X coordinate.
|
||||||
|
// xField: '__date__' | '__days__' | <daily fieldId>
|
||||||
|
export function listSampleMembers(statuses, xField, dailyTemplate) {
|
||||||
|
const list = statuses ?? [];
|
||||||
|
if (xField === '__date__') {
|
||||||
|
const keys = new Set();
|
||||||
|
for (const s of list) if (s?.date) keys.add(dateKey(s.date));
|
||||||
|
return [...keys].sort().map((k) => ({ id: k, label: k, sampleValue: k }));
|
||||||
|
}
|
||||||
|
if (xField === '__days__') {
|
||||||
|
const counts = new Map();
|
||||||
|
for (const s of list) {
|
||||||
|
if (s?.animal_id == null || !s?.date) continue;
|
||||||
|
counts.set(s.animal_id, (counts.get(s.animal_id) ?? 0) + 1);
|
||||||
|
}
|
||||||
|
const max = counts.size ? Math.max(...counts.values()) : 0;
|
||||||
|
return Array.from({ length: max }, (_, i) => ({ id: String(i + 1), label: String(i + 1), sampleValue: i + 1 }));
|
||||||
|
}
|
||||||
|
const field = (dailyTemplate ?? []).find((f) => f.fieldId === xField);
|
||||||
|
const vals = new Set();
|
||||||
|
for (const s of list) {
|
||||||
|
const v = numericFieldVal(s, field);
|
||||||
|
if (v !== null) vals.add(v);
|
||||||
|
}
|
||||||
|
return [...vals].sort((a, b) => a - b).map((v) => ({ id: String(v), label: String(v), sampleValue: v }));
|
||||||
|
}
|
||||||
|
|
||||||
|
// animalId -> Map(sampleValue -> status). First status per (subject, sampleValue) wins,
|
||||||
|
// scanning each subject's statuses in ascending date order.
|
||||||
|
export function buildSampleIndex(statuses, xField, dailyTemplate) {
|
||||||
|
const bySubject = new Map();
|
||||||
|
for (const s of statuses ?? []) {
|
||||||
|
if (s?.animal_id == null || !s?.date) continue;
|
||||||
|
if (!bySubject.has(s.animal_id)) bySubject.set(s.animal_id, []);
|
||||||
|
bySubject.get(s.animal_id).push(s);
|
||||||
|
}
|
||||||
|
const field = xField !== '__date__' && xField !== '__days__'
|
||||||
|
? (dailyTemplate ?? []).find((f) => f.fieldId === xField)
|
||||||
|
: null;
|
||||||
|
|
||||||
|
const index = new Map();
|
||||||
|
for (const [animalId, subjStatuses] of bySubject) {
|
||||||
|
const ordered = [...subjStatuses].sort((a, b) => dateKey(a.date).localeCompare(dateKey(b.date)));
|
||||||
|
const m = new Map();
|
||||||
|
ordered.forEach((s, i) => {
|
||||||
|
let key;
|
||||||
|
if (xField === '__date__') key = dateKey(s.date);
|
||||||
|
else if (xField === '__days__') key = i + 1;
|
||||||
|
else { const v = numericFieldVal(s, field); if (v === null) return; key = v; }
|
||||||
|
if (!m.has(key)) m.set(key, s);
|
||||||
|
});
|
||||||
|
index.set(animalId, m);
|
||||||
|
}
|
||||||
|
return index;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run tests to verify they pass**
|
||||||
|
|
||||||
|
Run: `npx jest tests/dataExport.test.js`
|
||||||
|
Expected: PASS (new suites green; all pre-existing suites still green).
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add frontend/src/lib/dataExport.js frontend/tests/dataExport.test.js
|
||||||
|
git commit -m "feat(export): X-coordinate helpers (date, # days reach, daily field)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: buildSubjectSeriesMatrix
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `frontend/src/lib/dataExport.js` (add `buildSubjectSeriesMatrix` + private `xFieldLabel`)
|
||||||
|
- Test: `frontend/tests/dataExport.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `listSubjectMembers`, `listSampleMembers`, `buildSampleIndex`, `extractValue`, private `hasValue`.
|
||||||
|
- Produces: `buildSubjectSeriesMatrix({ xField, dataParam, groupField }, { statuses, animals, dailyTemplate })` → `{ corner, context: { label: 'Data', value }, groupAxis: 'col'|null, columns: [{ id, label, animalId, group }], rows: [{ member: { id, label, sampleValue }, values: [value|''] }] }` — the same shape the existing `toCSV` consumes (`groupAxis: 'col'` triggers its group-row branch).
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing tests**
|
||||||
|
|
||||||
|
Append to `frontend/tests/dataExport.test.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
import { buildSubjectSeriesMatrix } from '../src/lib/dataExport';
|
||||||
|
|
||||||
|
const ssAnimals = [
|
||||||
|
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
||||||
|
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||||
|
];
|
||||||
|
const ssStatuses = [
|
||||||
|
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5 } },
|
||||||
|
{ animal_id: 'a1', date: '2026-07-02T00:00:00Z', analysis_summary: { total: 7 } },
|
||||||
|
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 9 } },
|
||||||
|
];
|
||||||
|
const totalParam = { id: '__total__', label: 'Total attempts', kind: 'total', group: 'metrics' };
|
||||||
|
|
||||||
|
describe('buildSubjectSeriesMatrix', () => {
|
||||||
|
it('date X: subjects as columns, date rows, blank where missing, context names the metric', () => {
|
||||||
|
const m = buildSubjectSeriesMatrix(
|
||||||
|
{ xField: '__date__', dataParam: totalParam, groupField: '__none__' },
|
||||||
|
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||||
|
);
|
||||||
|
expect(m.corner).toBe('Date');
|
||||||
|
expect(m.context).toEqual({ label: 'Data', value: 'Total attempts' });
|
||||||
|
expect(m.groupAxis).toBeNull();
|
||||||
|
expect(m.columns.map((c) => c.label)).toEqual(['Alpha', 'Beta']);
|
||||||
|
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01', '2026-07-02']);
|
||||||
|
expect(m.rows[0].values).toEqual([5, 9]);
|
||||||
|
expect(m.rows[1].values).toEqual([7, '']);
|
||||||
|
});
|
||||||
|
it('# days reach X: rows align each subject by day ordinal', () => {
|
||||||
|
const m = buildSubjectSeriesMatrix(
|
||||||
|
{ xField: '__days__', dataParam: totalParam, groupField: '__none__' },
|
||||||
|
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||||
|
);
|
||||||
|
expect(m.corner).toBe('# Days Reach');
|
||||||
|
expect(m.rows.map((r) => r.member.label)).toEqual(['1', '2']);
|
||||||
|
expect(m.rows[0].values).toEqual([5, 9]);
|
||||||
|
expect(m.rows[1].values).toEqual([7, '']);
|
||||||
|
});
|
||||||
|
it('grouping sets groupAxis col and clusters subjects', () => {
|
||||||
|
const m = buildSubjectSeriesMatrix(
|
||||||
|
{ xField: '__date__', dataParam: totalParam, groupField: 'grp' },
|
||||||
|
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||||
|
);
|
||||||
|
expect(m.groupAxis).toBe('col');
|
||||||
|
expect(m.columns.map((c) => [c.group, c.label])).toEqual([['Control', 'Alpha'], ['Drug', 'Beta']]);
|
||||||
|
});
|
||||||
|
it('drops entirely-blank rows', () => {
|
||||||
|
const statuses = [
|
||||||
|
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5 } },
|
||||||
|
{ animal_id: 'a1', date: '2026-07-05T00:00:00Z', analysis_summary: null },
|
||||||
|
];
|
||||||
|
const m = buildSubjectSeriesMatrix(
|
||||||
|
{ xField: '__date__', dataParam: totalParam, groupField: '__none__' },
|
||||||
|
{ statuses, animals: ssAnimals, dailyTemplate: [] },
|
||||||
|
);
|
||||||
|
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01']);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run tests to verify they fail**
|
||||||
|
|
||||||
|
Run: `npx jest tests/dataExport.test.js -t "buildSubjectSeriesMatrix"`
|
||||||
|
Expected: FAIL — `buildSubjectSeriesMatrix is not defined`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Write minimal implementation**
|
||||||
|
|
||||||
|
Add to `frontend/src/lib/dataExport.js` (after `buildSampleIndex`):
|
||||||
|
|
||||||
|
```js
|
||||||
|
function xFieldLabel(xField, dailyTemplate) {
|
||||||
|
if (xField === '__date__') return 'Date';
|
||||||
|
if (xField === '__days__') return '# Days Reach';
|
||||||
|
return (dailyTemplate ?? []).find((f) => f.fieldId === xField)?.label ?? '';
|
||||||
|
}
|
||||||
|
|
||||||
|
// Subjects as columns, X-field values as rows, one Data value per cell.
|
||||||
|
// Returns the { context, corner, groupAxis, columns, rows } shape toCSV consumes.
|
||||||
|
export function buildSubjectSeriesMatrix({ xField, dataParam, groupField }, { statuses, animals, dailyTemplate }) {
|
||||||
|
const columns = listSubjectMembers(animals, groupField);
|
||||||
|
const rowMembers = listSampleMembers(statuses, xField, dailyTemplate);
|
||||||
|
const index = buildSampleIndex(statuses, xField, dailyTemplate);
|
||||||
|
|
||||||
|
const cellValue = (sampleValue, animalId) => {
|
||||||
|
const status = index.get(animalId)?.get(sampleValue);
|
||||||
|
if (!status || !dataParam) return '';
|
||||||
|
const v = extractValue(status, dataParam);
|
||||||
|
return hasValue(v) ? v : '';
|
||||||
|
};
|
||||||
|
|
||||||
|
let rows = rowMembers.map((rm) => ({
|
||||||
|
member: rm,
|
||||||
|
values: columns.map((c) => cellValue(rm.sampleValue, c.animalId)),
|
||||||
|
}));
|
||||||
|
rows = rows.filter((r) => r.values.some((v) => v !== '')); // drop entirely-blank sample rows
|
||||||
|
|
||||||
|
const grouped = !!groupField && groupField !== '__none__';
|
||||||
|
return {
|
||||||
|
corner: xFieldLabel(xField, dailyTemplate),
|
||||||
|
context: { label: 'Data', value: dataParam?.label ?? '' },
|
||||||
|
groupAxis: grouped ? 'col' : null,
|
||||||
|
columns,
|
||||||
|
rows,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run tests to verify they pass**
|
||||||
|
|
||||||
|
Run: `npx jest tests/dataExport.test.js`
|
||||||
|
Expected: PASS (all suites green).
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add frontend/src/lib/dataExport.js frontend/tests/dataExport.test.js
|
||||||
|
git commit -m "feat(export): buildSubjectSeriesMatrix — subjects as columns over an X field"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Rewrite ExportDataModal to the three pickers
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `frontend/src/components/ExportDataModal.jsx` (full body rewrite; keep `triggerDownload`)
|
||||||
|
- Test: `frontend/tests/ExportDataModal.test.jsx` (rewrite)
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `listExportableParams`, `buildSubjectSeriesMatrix`, `toCSV`, `csvFilename` from `../lib/dataExport`; `experimentsApi.getDailyStatuses`.
|
||||||
|
- Props unchanged: `experimentTitle`, `experimentId`, `dailyTemplate`, `animals`, `subjectTemplate` (default `[]`), `groupField` (default `'__none__'`), `onClose`.
|
||||||
|
- Produces: modal with three selects — X (rows), Data (cells), Group by.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Rewrite the component test**
|
||||||
|
|
||||||
|
Replace the entire contents of `frontend/tests/ExportDataModal.test.jsx` with:
|
||||||
|
|
||||||
|
```jsx
|
||||||
|
import React from 'react';
|
||||||
|
import { render, screen, waitFor, fireEvent } from '@testing-library/react';
|
||||||
|
import ExportDataModal from '../src/components/ExportDataModal';
|
||||||
|
import * as client from '../src/api/client';
|
||||||
|
|
||||||
|
jest.mock('../src/api/client', () => ({
|
||||||
|
experimentsApi: { getDailyStatuses: jest.fn() },
|
||||||
|
}));
|
||||||
|
|
||||||
|
const animals = [
|
||||||
|
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
||||||
|
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||||
|
];
|
||||||
|
const statuses = [
|
||||||
|
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5, success_rate: 0.5 } },
|
||||||
|
{ animal_id: 'a1', date: '2026-07-02T00:00:00Z', analysis_summary: { total: 7, success_rate: 0.7 } },
|
||||||
|
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 9, success_rate: 0.9 } },
|
||||||
|
];
|
||||||
|
const subjectTemplate = [{ fieldId: 'grp', label: 'Group', active: true }];
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
jest.clearAllMocks();
|
||||||
|
client.experimentsApi.getDailyStatuses.mockResolvedValue(statuses);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Intercept the Blob handed to URL.createObjectURL so tests can read the real CSV.
|
||||||
|
// Also stub anchor.click() — jsdom treats a click on <a href="blob:..."> as an
|
||||||
|
// unimplemented navigation that throws async and can flake a later test.
|
||||||
|
function mockDownload() {
|
||||||
|
let blob = null;
|
||||||
|
const origCreate = global.URL.createObjectURL;
|
||||||
|
const origRevoke = global.URL.revokeObjectURL;
|
||||||
|
const origClick = window.HTMLAnchorElement.prototype.click;
|
||||||
|
global.URL.createObjectURL = jest.fn((b) => { blob = b; return 'blob:mock'; });
|
||||||
|
global.URL.revokeObjectURL = jest.fn();
|
||||||
|
window.HTMLAnchorElement.prototype.click = jest.fn();
|
||||||
|
return {
|
||||||
|
getBlob: () => blob,
|
||||||
|
restore: async () => {
|
||||||
|
await new Promise((r) => setTimeout(r, 0)); // flush handleExport's setTimeout revoke
|
||||||
|
global.URL.createObjectURL = origCreate;
|
||||||
|
global.URL.revokeObjectURL = origRevoke;
|
||||||
|
window.HTMLAnchorElement.prototype.click = origClick;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function readBlobText(blob) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const fr = new FileReader();
|
||||||
|
fr.onload = () => resolve(fr.result);
|
||||||
|
fr.onerror = reject;
|
||||||
|
fr.readAsText(blob);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderModal(props = {}) {
|
||||||
|
return render(
|
||||||
|
<ExportDataModal
|
||||||
|
experimentId="exp-1"
|
||||||
|
experimentTitle="My Study"
|
||||||
|
dailyTemplate={[]}
|
||||||
|
animals={animals}
|
||||||
|
subjectTemplate={subjectTemplate}
|
||||||
|
groupField="__none__"
|
||||||
|
onClose={jest.fn()}
|
||||||
|
{...props}
|
||||||
|
/>,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('ExportDataModal', () => {
|
||||||
|
it('renders X / Data / Group by pickers; X defaults to Date', async () => {
|
||||||
|
renderModal();
|
||||||
|
expect(await screen.findByLabelText('X axis (rows)')).toHaveValue('__date__');
|
||||||
|
expect(screen.getByLabelText('Data (cells)')).toBeInTheDocument();
|
||||||
|
expect(screen.getByLabelText('Group by')).toHaveValue('__none__');
|
||||||
|
await waitFor(() => expect(screen.getByText(/2 rows × 2 subjects/i)).toBeInTheDocument());
|
||||||
|
});
|
||||||
|
|
||||||
|
it('default export (X=Date, Data=Total attempts) has subject columns, date rows, blank where missing', async () => {
|
||||||
|
const dl = mockDownload();
|
||||||
|
renderModal();
|
||||||
|
await screen.findByLabelText('X axis (rows)');
|
||||||
|
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||||
|
fireEvent.click(screen.getByText('Export CSV'));
|
||||||
|
const text = await readBlobText(dl.getBlob());
|
||||||
|
expect(text).toBe('Data,Total attempts\n\nDate,Alpha,Beta\n2026-07-01,5,9\n2026-07-02,7,');
|
||||||
|
await dl.restore();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('switching X to # Days Reach makes rows the day ordinals', async () => {
|
||||||
|
const dl = mockDownload();
|
||||||
|
renderModal();
|
||||||
|
fireEvent.change(await screen.findByLabelText('X axis (rows)'), { target: { value: '__days__' } });
|
||||||
|
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||||
|
fireEvent.click(screen.getByText('Export CSV'));
|
||||||
|
const text = await readBlobText(dl.getBlob());
|
||||||
|
expect(text).toBe('Data,Total attempts\n\n# Days Reach,Alpha,Beta\n1,5,9\n2,7,');
|
||||||
|
await dl.restore();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('a valid group field adds a Group row and clusters subjects', async () => {
|
||||||
|
const dl = mockDownload();
|
||||||
|
renderModal({ groupField: 'grp' });
|
||||||
|
await screen.findByLabelText('X axis (rows)');
|
||||||
|
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||||
|
fireEvent.click(screen.getByText('Export CSV'));
|
||||||
|
const text = await readBlobText(dl.getBlob());
|
||||||
|
expect(text).toMatch(/^Group,Control,Drug$/m);
|
||||||
|
await dl.restore();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run the test to verify it fails**
|
||||||
|
|
||||||
|
Run: `npx jest tests/ExportDataModal.test.jsx`
|
||||||
|
Expected: FAIL — old modal has no `X axis (rows)` / `Data (cells)` labels.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Rewrite the component**
|
||||||
|
|
||||||
|
Replace the entire contents of `frontend/src/components/ExportDataModal.jsx` with:
|
||||||
|
|
||||||
|
```jsx
|
||||||
|
import React, { useEffect, useMemo, useState } from 'react';
|
||||||
|
import { experimentsApi } from '../api/client';
|
||||||
|
import Button from './ui/Button';
|
||||||
|
import Alert from './ui/Alert';
|
||||||
|
import { listExportableParams, buildSubjectSeriesMatrix, toCSV, csvFilename } from '../lib/dataExport';
|
||||||
|
|
||||||
|
const PARAM_GROUP_LABELS = { metrics: 'Session metrics', daily: 'Daily record fields' };
|
||||||
|
|
||||||
|
function triggerDownload(filename, text) {
|
||||||
|
const blob = new Blob([text], { type: 'text/csv;charset=utf-8;' });
|
||||||
|
const url = URL.createObjectURL(blob);
|
||||||
|
const a = document.createElement('a');
|
||||||
|
a.href = url;
|
||||||
|
a.download = filename;
|
||||||
|
document.body.appendChild(a);
|
||||||
|
a.click();
|
||||||
|
document.body.removeChild(a);
|
||||||
|
setTimeout(() => URL.revokeObjectURL(url), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function ExportDataModal({
|
||||||
|
experimentTitle, experimentId, dailyTemplate, animals, subjectTemplate = [], groupField = '__none__', onClose,
|
||||||
|
}) {
|
||||||
|
const [loading, setLoading] = useState(true);
|
||||||
|
const [error, setError] = useState(null);
|
||||||
|
const [statuses, setStatuses] = useState([]);
|
||||||
|
|
||||||
|
const [xField, setXField] = useState('__date__');
|
||||||
|
const [dataId, setDataId] = useState(null);
|
||||||
|
const [groupBy, setGroupBy] = useState(() => {
|
||||||
|
const valid = groupField === '__none__' || groupField === '__name__' || groupField === '__id__'
|
||||||
|
|| subjectTemplate.some((f) => f.active && f.fieldId === groupField);
|
||||||
|
return valid ? groupField : '__none__';
|
||||||
|
});
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
let alive = true;
|
||||||
|
setLoading(true);
|
||||||
|
setError(null);
|
||||||
|
experimentsApi.getDailyStatuses(experimentId, {})
|
||||||
|
.then((data) => { if (alive) setStatuses(data); })
|
||||||
|
.catch((err) => { if (alive) setError(err.message); })
|
||||||
|
.finally(() => { if (alive) setLoading(false); });
|
||||||
|
return () => { alive = false; };
|
||||||
|
}, [experimentId]);
|
||||||
|
|
||||||
|
const params = useMemo(() => listExportableParams(dailyTemplate, statuses), [dailyTemplate, statuses]);
|
||||||
|
const effDataId = dataId ?? params[0]?.id ?? null;
|
||||||
|
const dataParam = params.find((p) => p.id === effDataId) ?? params[0] ?? null;
|
||||||
|
|
||||||
|
const activeDaily = useMemo(() => (dailyTemplate ?? []).filter((f) => f.active), [dailyTemplate]);
|
||||||
|
const xOptions = useMemo(() => [
|
||||||
|
{ id: '__date__', label: 'Date' },
|
||||||
|
{ id: '__days__', label: '# Days Reach' },
|
||||||
|
...activeDaily.map((f) => ({ id: f.fieldId, label: f.label })),
|
||||||
|
], [activeDaily]);
|
||||||
|
|
||||||
|
const matrix = useMemo(
|
||||||
|
() => (dataParam
|
||||||
|
? buildSubjectSeriesMatrix({ xField, dataParam, groupField: groupBy }, { statuses, animals, dailyTemplate })
|
||||||
|
: { columns: [], rows: [], context: { label: 'Data', value: '' }, corner: '', groupAxis: null }),
|
||||||
|
[xField, dataParam, groupBy, statuses, animals, dailyTemplate],
|
||||||
|
);
|
||||||
|
const hasData = matrix.rows.length > 0;
|
||||||
|
|
||||||
|
const groupedParams = useMemo(() => {
|
||||||
|
const out = [];
|
||||||
|
for (const p of params) {
|
||||||
|
let g = out.find((x) => x.group === p.group);
|
||||||
|
if (!g) { g = { group: p.group, items: [] }; out.push(g); }
|
||||||
|
g.items.push(p);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}, [params]);
|
||||||
|
|
||||||
|
function handleExport() {
|
||||||
|
if (!hasData || !dataParam) return;
|
||||||
|
triggerDownload(csvFilename(experimentTitle, dataParam.label), toCSV(matrix));
|
||||||
|
onClose();
|
||||||
|
}
|
||||||
|
|
||||||
|
if (loading) return <p className="text-sm text-gray-400" aria-live="polite">Loading data…</p>;
|
||||||
|
if (error) return (
|
||||||
|
<div className="space-y-4">
|
||||||
|
<Alert type="error" message={error} />
|
||||||
|
<div className="flex justify-end"><Button variant="secondary" onClick={onClose}>Close</Button></div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
|
||||||
|
const selectCls = 'w-full border border-gray-200 rounded px-2 py-1.5 text-sm bg-white focus:outline-none focus:ring-1 focus:ring-indigo-400';
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="space-y-4">
|
||||||
|
<p className="text-sm text-gray-500">
|
||||||
|
One value per subject over an X axis — each column is a subject, each row an X value. Blank where a subject has no value.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div>
|
||||||
|
<label htmlFor="export-x" className="block text-xs font-medium text-gray-500 mb-1">X axis (rows)</label>
|
||||||
|
<select id="export-x" value={xField} onChange={(e) => setXField(e.target.value)} className={selectCls}>
|
||||||
|
{xOptions.map((o) => <option key={o.id} value={o.id}>{o.label}</option>)}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div>
|
||||||
|
<label htmlFor="export-data" className="block text-xs font-medium text-gray-500 mb-1">Data (cells)</label>
|
||||||
|
<select id="export-data" value={effDataId ?? ''} onChange={(e) => setDataId(e.target.value)} className={selectCls}>
|
||||||
|
{groupedParams.map((g) => (
|
||||||
|
<optgroup key={g.group} label={PARAM_GROUP_LABELS[g.group] ?? g.group}>
|
||||||
|
{g.items.map((p) => <option key={p.id} value={p.id}>{p.label}</option>)}
|
||||||
|
</optgroup>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div>
|
||||||
|
<label htmlFor="export-group" className="block text-xs font-medium text-gray-500 mb-1">Group by</label>
|
||||||
|
<select id="export-group" value={groupBy} onChange={(e) => setGroupBy(e.target.value)} className={selectCls}>
|
||||||
|
<option value="__none__">— None —</option>
|
||||||
|
<option value="__name__">Name</option>
|
||||||
|
<option value="__id__">Subject ID</option>
|
||||||
|
{subjectTemplate.filter((f) => f.active).map((f) => (
|
||||||
|
<option key={f.fieldId} value={f.fieldId}>{f.label}</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p className="text-xs text-gray-400">
|
||||||
|
{hasData
|
||||||
|
? `${matrix.rows.length} row${matrix.rows.length !== 1 ? 's' : ''} × ${matrix.columns.length} subject${matrix.columns.length !== 1 ? 's' : ''}`
|
||||||
|
: 'No data for this selection yet.'}
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div className="flex justify-end gap-2 pt-2">
|
||||||
|
<Button variant="secondary" onClick={onClose}>Cancel</Button>
|
||||||
|
<Button onClick={handleExport} disabled={!hasData}>Export CSV</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run the test to verify it passes**
|
||||||
|
|
||||||
|
Run: `npx jest tests/ExportDataModal.test.jsx`
|
||||||
|
Expected: PASS (all four cases).
|
||||||
|
|
||||||
|
- [ ] **Step 5: Run the full suite**
|
||||||
|
|
||||||
|
Run: `npm test`
|
||||||
|
Expected: `dataExport` + `ExportDataModal` green; only the two known pre-existing suites fail; no new failures.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add frontend/src/components/ExportDataModal.jsx frontend/tests/ExportDataModal.test.jsx
|
||||||
|
git commit -m "feat(export): modal becomes X / Data / Group by pickers, subjects as columns"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: Remove the dead assignable-axes code
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `frontend/src/lib/dataExport.js` (delete unused exports)
|
||||||
|
- Test: `frontend/tests/dataExport.test.js` (delete their describe blocks + imports)
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Removes (now unused after Task 3): `EXPORT_DIMENSIONS`, `dimLabel`, `otherDimension`, `listDateMembers`, `buildStatusIndex`, `listParameterMembers`, `listDimensionMembers`, and `buildMatrix`.
|
||||||
|
- Keeps: `extractValue`, `listExportableParams`, `subjectGroupValue`, `listSubjectMembers`, `numericFieldVal`, `listSampleMembers`, `buildSampleIndex`, `buildSubjectSeriesMatrix`, `toCSV`, `csvFilename`, and the private `hasValue`, `dateKey`, `escapeCsvCell`, `xFieldLabel`.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Delete the dead source exports**
|
||||||
|
|
||||||
|
In `frontend/src/lib/dataExport.js`, delete these eight exported definitions entirely (they are only used by each other and the old modal, now replaced): `EXPORT_DIMENSIONS`, `dimLabel`, `otherDimension`, `listDateMembers`, `buildStatusIndex`, `listParameterMembers`, `listDimensionMembers`, `buildMatrix`. Also update the top-of-file comment to describe the subject-series export (drop the "dimension-agnostic buildMatrix" wording). Keep `dateKey`, `hasValue`, `escapeCsvCell`, and everything listed under "Keeps" above.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Delete their tests**
|
||||||
|
|
||||||
|
In `frontend/tests/dataExport.test.js`, delete the describe blocks and their now-unused import identifiers for the removed functions: the `describe('dimension primitives', …)`, `describe('subjectGroupValue', …)` KEEP (subjectGroupValue is kept — do NOT delete it), `describe('listDateMembers', …)`, `describe('buildStatusIndex', …)`, `describe('listParameterMembers / listDimensionMembers', …)`, and `describe('buildMatrix', …)` blocks. Remove `EXPORT_DIMENSIONS`, `dimLabel`, `otherDimension`, `listDateMembers`, `buildStatusIndex`, `listDimensionMembers`, `listParameterMembers`, and `buildMatrix` from the file's `import` statements, leaving the imports for kept functions (`extractValue`, `listExportableParams`, `subjectGroupValue`, `listSubjectMembers`, `numericFieldVal`, `listSampleMembers`, `buildSampleIndex`, `buildSubjectSeriesMatrix`, `toCSV`, `csvFilename`).
|
||||||
|
|
||||||
|
- [ ] **Step 3: Verify nothing references the removed names**
|
||||||
|
|
||||||
|
Run: `grep -rnE "buildMatrix|EXPORT_DIMENSIONS|listDimensionMembers|buildStatusIndex|listDateMembers|listParameterMembers|otherDimension|dimLabel" frontend/src frontend/tests`
|
||||||
|
Expected: no output (all references gone).
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run the suites**
|
||||||
|
|
||||||
|
Run: `npx jest tests/dataExport.test.js tests/ExportDataModal.test.jsx`
|
||||||
|
Expected: PASS. Then `npm test` — only the two known pre-existing suites fail, no new failures.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add frontend/src/lib/dataExport.js frontend/tests/dataExport.test.js
|
||||||
|
git commit -m "refactor(export): remove dead assignable-axes helpers and their tests"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review Notes
|
||||||
|
|
||||||
|
- **Spec coverage:** three pickers (Task 3) · X = Date/# Days Reach/daily field with the exact semantics (Tasks 1–2) · subjects always columns, clustered by group, blank rows dropped (Task 2) · Group by validated/defaulted (Task 3) · CSV `Data,<metric>` + group row + X-corner (Tasks 2–3, reusing `toCSV`) · assignable-axes UI/machinery removed (Tasks 3–4). All covered.
|
||||||
|
- **Type consistency:** sample member `{ id, label, sampleValue }`, subject column `{ id, label, animalId, group }`, and matrix `{ context, corner, groupAxis, columns, rows }` are used identically across Tasks 1–3 and match the existing `toCSV`.
|
||||||
|
- **Out of scope (per spec):** assignable columns/pinned, subject subsetting, analyzed-only day counting, footer/repeat line.
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
# Subject-Series Data Export — Design
|
||||||
|
|
||||||
|
**Date:** 2026-07-19
|
||||||
|
**Status:** Approved (design)
|
||||||
|
**Area:** `frontend/src/lib/dataExport.js`, `frontend/src/components/ExportDataModal.jsx`, `frontend/tests/`
|
||||||
|
**Supersedes:** the assignable-axes UI from `2026-07-19-configurable-data-export-design.md` (that branch's `buildMatrix`/`toCSV`/subject-group work is reused; its rows/columns/pinned modal is replaced).
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The export modal currently offers assignable row/column/pinned axes. That is more than needed. The user wants the export to mirror the **cross-subject metrics plot**: pick an **X field** and a **data value**, with each **subject as a column series**, empty where a subject has no data point, plus each subject's **group** included. The one capability missing today is choosing the X (row) field — it is fixed to calendar Date, whereas the plot's X-axis offers **# Days Reach** and daily fields.
|
||||||
|
|
||||||
|
## Layout (fixed)
|
||||||
|
|
||||||
|
- **Columns = subjects**, one series per subject, ordered clustered-by-group then by name (existing `listSubjectMembers`).
|
||||||
|
- **Rows = the chosen X field's values.**
|
||||||
|
- **Cells = the chosen Data value** for that (subject, X); empty (`''`) when absent. `0` and `''` remain real values (existing `hasValue`/`extractValue`).
|
||||||
|
- **Group row** gives each subject's group value (existing group-row support).
|
||||||
|
|
||||||
|
There is no pinned dimension and no assignable columns.
|
||||||
|
|
||||||
|
## The three pickers
|
||||||
|
|
||||||
|
1. **X (rows).** Options: `Date`, `# Days Reach`, and each **active** daily-template field. Default `Date` (keeps the same date-rows × subject-columns grid as today; only the line-1 label changes to `Data`, see CSV output).
|
||||||
|
2. **Data (cells).** Options: `listExportableParams(dailyTemplate, statuses)` — total, success rate, each count category, each daily field. Default: first parameter (`Total attempts`).
|
||||||
|
3. **Group by.** Options: `— None —`, `Name`, `Subject ID`, and each active `subject_info` field. Defaults to the experiment's saved group field (`localStorage['exp-subject-group-<id>']`), validated against the available options and coerced to `__none__` when unset or stale. Always visible (subjects are always the columns).
|
||||||
|
|
||||||
|
Preview line: `N rows × M subjects`. Export disabled when there are no data rows.
|
||||||
|
|
||||||
|
## X-field semantics
|
||||||
|
|
||||||
|
Each X coordinate maps a `(subject, X-value)` pair to a single daily status, from which the Data value is read:
|
||||||
|
|
||||||
|
- **`Date`** → rows are the distinct calendar dates present (`YYYY-MM-DD`), sorted. Cell reads the subject's status on that date. (Today's behavior.)
|
||||||
|
- **`# Days Reach`** → rows are `1, 2, 3, …` up to the maximum, per subject, of the count of that subject's recorded daily statuses. A cell for `(subject, N)` reads the subject's **Nth recorded day**, counting **all** recorded statuses sorted ascending by date (Day 1 = earliest record; days without an analysis summary are still counted — this intentionally differs from the plot, which counts analyzed sessions only).
|
||||||
|
- **daily field** (e.g. `Weight (g)`) → rows are the distinct **numeric** values that field takes across the data (`numericFieldVal`: builtin via `status[field.key]`, custom via `status.custom_fields[field.fieldId]`, `parseFloat`, non-numeric → excluded), sorted ascending. A cell for `(subject, v)` reads that subject's status where the field equals `v`; on ties, first status by date wins.
|
||||||
|
|
||||||
|
Row members that are entirely blank across all subjects are dropped (matching Date's existing behavior); subject columns are always kept even when empty.
|
||||||
|
|
||||||
|
## CSV output
|
||||||
|
|
||||||
|
Structure (X = # Days Reach, Data = Success rate, Group by = Treatment):
|
||||||
|
|
||||||
|
```
|
||||||
|
Data,Success rate
|
||||||
|
Group,Control,Control,Drug
|
||||||
|
# Days Reach,Mouse-A,Mouse-B,Mouse-C
|
||||||
|
1,0.5,0.6,0.7
|
||||||
|
2,0.55,,0.72
|
||||||
|
3,0.6,0.65,
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Line 1** names the Data metric so the file is self-describing: two cells, `Data` and the metric label.
|
||||||
|
- **Blank line.**
|
||||||
|
- **Group row** (`Group` + each subject's group), only when Group by ≠ None; missing group value → `—`.
|
||||||
|
- **Header row:** the X-field label in the corner, then subject headers.
|
||||||
|
- **Data rows:** the X value, then one cell per subject.
|
||||||
|
|
||||||
|
LF line endings; every cell escaped via existing `escapeCsvCell`. Filename: existing `csvFilename(title, dataLabel)`.
|
||||||
|
|
||||||
|
## Implementation notes
|
||||||
|
|
||||||
|
Reuse from the current branch: `extractValue`, `listExportableParams`, `subjectGroupValue`, `listSubjectMembers`, `escapeCsvCell`, `toCSV` (its `{ context, corner, groupAxis, columns, rows }` shape is kept), and `csvFilename`.
|
||||||
|
|
||||||
|
Replace: the general assignable-axes `buildMatrix(config, ctx)` and its members/dispatch helpers (`otherDimension`, `dimLabel`, `EXPORT_DIMENSIONS`, `listDateMembers`, `listDimensionMembers`, `listParameterMembers`, `buildStatusIndex` in its current form) — remove what the fixed layout no longer uses rather than leaving dead code.
|
||||||
|
|
||||||
|
Add:
|
||||||
|
- `numericFieldVal(status, field)` — mirror of the plot's helper (or import/share it).
|
||||||
|
- `listSampleMembers(statuses, xField, dailyTemplate)` → the ordered row members for the chosen X coordinate (`{ id, label, ... }`).
|
||||||
|
- `buildSampleIndex(statuses, xField, dailyTemplate)` → `Map(animalId → Map(sampleValue → status))`, first-status-per-cell wins, encoding the date/#days/field resolution above.
|
||||||
|
- `buildSubjectSeriesMatrix({ xField, dataParam, groupField }, { statuses, animals, dailyTemplate })` → `{ context: { label: 'Data', value: dataParam.label }, corner: <X label>, groupAxis: grouping ? 'col' : null, columns: <subjects>, rows: [{ member, values }] }`, consumable by the existing `toCSV`.
|
||||||
|
|
||||||
|
Simplify `ExportDataModal.jsx` to the three pickers over these helpers; drop the Rows/Columns/Fixed controls.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- `listSampleMembers` / `buildSampleIndex` for each X coordinate: Date (unchanged), # Days Reach (all-records ordinal, per-subject alignment), daily field (distinct numeric values sorted, non-numeric dropped, tie first-by-date).
|
||||||
|
- `buildSubjectSeriesMatrix`: subjects as columns clustered by group; empty cells where a subject lacks a point; blank row dropped; `groupAxis` null when Group by None.
|
||||||
|
- `toCSV` reused; add/keep a snapshot for the new context line + group row.
|
||||||
|
- `ExportDataModal`: renders three pickers; default X = Date produces date rows × subject columns with the chosen metric in cells; switching X to # Days Reach changes rows to ordinals; group row present when a valid group field is chosen; stale/unset group field coerces to None.
|
||||||
|
|
||||||
|
## Out of scope (YAGNI)
|
||||||
|
|
||||||
|
- Assignable columns / pinned dimension (removed).
|
||||||
|
- Choosing a subset of subjects (all subjects always included).
|
||||||
|
- Matching the plot's analyzed-only day counting (we count all recorded days).
|
||||||
|
- A footer or repeated context line.
|
||||||
@@ -2,10 +2,7 @@ import React, { useEffect, useMemo, useState } from 'react';
|
|||||||
import { experimentsApi } from '../api/client';
|
import { experimentsApi } from '../api/client';
|
||||||
import Button from './ui/Button';
|
import Button from './ui/Button';
|
||||||
import Alert from './ui/Alert';
|
import Alert from './ui/Alert';
|
||||||
import {
|
import { listExportableParams, buildSubjectSeriesMatrix, toCSV, csvFilename } from '../lib/dataExport';
|
||||||
EXPORT_DIMENSIONS, dimLabel, otherDimension,
|
|
||||||
listDimensionMembers, buildMatrix, toCSV, csvFilename,
|
|
||||||
} from '../lib/dataExport';
|
|
||||||
|
|
||||||
const PARAM_GROUP_LABELS = { metrics: 'Session metrics', daily: 'Daily record fields' };
|
const PARAM_GROUP_LABELS = { metrics: 'Session metrics', daily: 'Daily record fields' };
|
||||||
|
|
||||||
@@ -28,10 +25,13 @@ export default function ExportDataModal({
|
|||||||
const [error, setError] = useState(null);
|
const [error, setError] = useState(null);
|
||||||
const [statuses, setStatuses] = useState([]);
|
const [statuses, setStatuses] = useState([]);
|
||||||
|
|
||||||
const [rowDim, setRowDim] = useState('date');
|
const [xField, setXField] = useState('__date__');
|
||||||
const [colDim, setColDim] = useState('subject');
|
const [dataId, setDataId] = useState(null);
|
||||||
const [pinnedId, setPinnedId] = useState(null);
|
const [groupBy, setGroupBy] = useState(() => {
|
||||||
const [groupBy, setGroupBy] = useState(groupField ?? '__none__');
|
const valid = groupField === '__none__' || groupField === '__name__' || groupField === '__id__'
|
||||||
|
|| subjectTemplate.some((f) => f.active && f.fieldId === groupField);
|
||||||
|
return valid ? groupField : '__none__';
|
||||||
|
});
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
let alive = true;
|
let alive = true;
|
||||||
@@ -44,42 +44,38 @@ export default function ExportDataModal({
|
|||||||
return () => { alive = false; };
|
return () => { alive = false; };
|
||||||
}, [experimentId]);
|
}, [experimentId]);
|
||||||
|
|
||||||
const pinnedDim = otherDimension(rowDim, colDim);
|
const params = useMemo(() => listExportableParams(dailyTemplate ?? [], statuses), [dailyTemplate, statuses]);
|
||||||
const subjectIsAxis = rowDim === 'subject' || colDim === 'subject';
|
const effDataId = dataId ?? params[0]?.id ?? null;
|
||||||
const effGroup = subjectIsAxis ? groupBy : '__none__';
|
const dataParam = params.find((p) => p.id === effDataId) ?? params[0] ?? null;
|
||||||
const ctx = useMemo(() => ({ statuses, animals, dailyTemplate }), [statuses, animals, dailyTemplate]);
|
|
||||||
|
|
||||||
// Members of the pinned dimension populate its value dropdown.
|
const activeDaily = useMemo(() => (dailyTemplate ?? []).filter((f) => f.active), [dailyTemplate]);
|
||||||
const pinnedMembers = useMemo(
|
const xOptions = useMemo(() => [
|
||||||
() => listDimensionMembers(pinnedDim, { ...ctx, groupField: effGroup }),
|
{ id: '__date__', label: 'Date' },
|
||||||
[pinnedDim, ctx, effGroup],
|
{ id: '__days__', label: '# Days Reach (ordinal)' },
|
||||||
);
|
...activeDaily.map((f) => ({ id: f.fieldId, label: f.label })),
|
||||||
const effPinnedId = pinnedId ?? pinnedMembers[0]?.id ?? null;
|
], [activeDaily]);
|
||||||
const pinnedMember = pinnedMembers.find((m) => m.id === effPinnedId) ?? pinnedMembers[0] ?? null;
|
|
||||||
|
|
||||||
const matrix = useMemo(
|
const matrix = useMemo(
|
||||||
() => (pinnedMember
|
() => (dataParam
|
||||||
? buildMatrix({ rowDim, colDim, pinnedMember, groupField: effGroup }, ctx)
|
? buildSubjectSeriesMatrix({ xField, dataParam, groupField: groupBy }, { statuses, animals, dailyTemplate })
|
||||||
: { columns: [], rows: [], context: { label: '', value: '' }, corner: '', groupAxis: null }),
|
: { columns: [], rows: [], context: { label: 'Data', value: '' }, corner: '', groupAxis: null }),
|
||||||
[rowDim, colDim, pinnedMember, effGroup, ctx],
|
[xField, dataParam, groupBy, statuses, animals, dailyTemplate],
|
||||||
);
|
);
|
||||||
const hasData = matrix.rows.length > 0;
|
const hasData = matrix.rows.length > 0;
|
||||||
|
|
||||||
// When the user changes an axis, keep the two distinct and reset the pinned pick.
|
const groupedParams = useMemo(() => {
|
||||||
function changeRowDim(next) {
|
const out = [];
|
||||||
setRowDim(next);
|
for (const p of params) {
|
||||||
if (colDim === next) setColDim(otherDimension(next, null) ?? EXPORT_DIMENSIONS.find((d) => d.dim !== next).dim);
|
let g = out.find((x) => x.group === p.group);
|
||||||
setPinnedId(null);
|
if (!g) { g = { group: p.group, items: [] }; out.push(g); }
|
||||||
}
|
g.items.push(p);
|
||||||
function changeColDim(next) {
|
|
||||||
setColDim(next);
|
|
||||||
if (rowDim === next) setRowDim(EXPORT_DIMENSIONS.find((d) => d.dim !== next).dim);
|
|
||||||
setPinnedId(null);
|
|
||||||
}
|
}
|
||||||
|
return out;
|
||||||
|
}, [params]);
|
||||||
|
|
||||||
function handleExport() {
|
function handleExport() {
|
||||||
if (!hasData || !pinnedMember) return;
|
if (!hasData || !dataParam) return;
|
||||||
triggerDownload(csvFilename(experimentTitle, pinnedMember.label), toCSV(matrix));
|
triggerDownload(csvFilename(experimentTitle, dataParam.label), toCSV(matrix));
|
||||||
onClose();
|
onClose();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -92,50 +88,31 @@ export default function ExportDataModal({
|
|||||||
);
|
);
|
||||||
|
|
||||||
const selectCls = 'w-full border border-gray-200 rounded px-2 py-1.5 text-sm bg-white focus:outline-none focus:ring-1 focus:ring-indigo-400';
|
const selectCls = 'w-full border border-gray-200 rounded px-2 py-1.5 text-sm bg-white focus:outline-none focus:ring-1 focus:ring-indigo-400';
|
||||||
const colOptions = EXPORT_DIMENSIONS.filter((d) => d.dim !== rowDim);
|
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div className="space-y-4">
|
<div className="space-y-4">
|
||||||
<p className="text-sm text-gray-500">
|
<p className="text-sm text-gray-500">
|
||||||
Choose what goes on the rows and columns. The remaining dimension is fixed to one value, shown at the top of the file.
|
One value per subject over an X axis — each column is a subject, each row an X value. Blank where a subject has no value.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<div className="grid grid-cols-2 gap-3">
|
|
||||||
<div>
|
<div>
|
||||||
<label htmlFor="export-rows" className="block text-xs font-medium text-gray-500 mb-1">Rows</label>
|
<label htmlFor="export-x" className="block text-xs font-medium text-gray-500 mb-1">X axis (rows)</label>
|
||||||
<select id="export-rows" value={rowDim} onChange={(e) => changeRowDim(e.target.value)} className={selectCls}>
|
<select id="export-x" value={xField} onChange={(e) => setXField(e.target.value)} className={selectCls}>
|
||||||
{EXPORT_DIMENSIONS.map((d) => <option key={d.dim} value={d.dim}>{d.label}</option>)}
|
{xOptions.map((o) => <option key={o.id} value={o.id}>{o.label}</option>)}
|
||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
<div>
|
|
||||||
<label htmlFor="export-cols" className="block text-xs font-medium text-gray-500 mb-1">Columns</label>
|
|
||||||
<select id="export-cols" value={colDim} onChange={(e) => changeColDim(e.target.value)} className={selectCls}>
|
|
||||||
{colOptions.map((d) => <option key={d.dim} value={d.dim}>{d.label}</option>)}
|
|
||||||
</select>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div>
|
<div>
|
||||||
<label htmlFor="export-pinned" className="block text-xs font-medium text-gray-500 mb-1">
|
<label htmlFor="export-data" className="block text-xs font-medium text-gray-500 mb-1">Data (cells)</label>
|
||||||
Fixed: {dimLabel(pinnedDim)}
|
<select id="export-data" value={effDataId ?? ''} onChange={(e) => setDataId(e.target.value)} className={selectCls}>
|
||||||
</label>
|
{groupedParams.map((g) => (
|
||||||
<select
|
|
||||||
id="export-pinned"
|
|
||||||
value={effPinnedId ?? ''}
|
|
||||||
onChange={(e) => setPinnedId(e.target.value)}
|
|
||||||
className={selectCls}
|
|
||||||
>
|
|
||||||
{pinnedDim === 'parameter'
|
|
||||||
? groupParamMembers(pinnedMembers).map((g) => (
|
|
||||||
<optgroup key={g.group} label={PARAM_GROUP_LABELS[g.group] ?? g.group}>
|
<optgroup key={g.group} label={PARAM_GROUP_LABELS[g.group] ?? g.group}>
|
||||||
{g.items.map((m) => <option key={m.id} value={m.id}>{m.label}</option>)}
|
{g.items.map((p) => <option key={p.id} value={p.id}>{p.label}</option>)}
|
||||||
</optgroup>
|
</optgroup>
|
||||||
))
|
))}
|
||||||
: pinnedMembers.map((m) => <option key={m.id} value={m.id}>{m.label}</option>)}
|
|
||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{subjectIsAxis && (
|
|
||||||
<div>
|
<div>
|
||||||
<label htmlFor="export-group" className="block text-xs font-medium text-gray-500 mb-1">Group by</label>
|
<label htmlFor="export-group" className="block text-xs font-medium text-gray-500 mb-1">Group by</label>
|
||||||
<select id="export-group" value={groupBy} onChange={(e) => setGroupBy(e.target.value)} className={selectCls}>
|
<select id="export-group" value={groupBy} onChange={(e) => setGroupBy(e.target.value)} className={selectCls}>
|
||||||
@@ -147,11 +124,10 @@ export default function ExportDataModal({
|
|||||||
))}
|
))}
|
||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
)}
|
|
||||||
|
|
||||||
<p className="text-xs text-gray-400">
|
<p className="text-xs text-gray-400">
|
||||||
{hasData
|
{hasData
|
||||||
? `${matrix.rows.length} ${matrix.corner.toLowerCase()}${matrix.rows.length !== 1 ? 's' : ''} × ${matrix.columns.length} ${dimLabel(colDim).toLowerCase()}${matrix.columns.length !== 1 ? 's' : ''}`
|
? `${matrix.rows.length} row${matrix.rows.length !== 1 ? 's' : ''} × ${matrix.columns.length} subject${matrix.columns.length !== 1 ? 's' : ''}`
|
||||||
: 'No data for this selection yet.'}
|
: 'No data for this selection yet.'}
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
@@ -162,14 +138,3 @@ export default function ExportDataModal({
|
|||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Group parameter members by their .group for <optgroup> rendering, preserving order.
|
|
||||||
function groupParamMembers(members) {
|
|
||||||
const out = [];
|
|
||||||
for (const m of members) {
|
|
||||||
let g = out.find((x) => x.group === m.group);
|
|
||||||
if (!g) { g = { group: m.group, items: [] }; out.push(g); }
|
|
||||||
g.items.push(m);
|
|
||||||
}
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|||||||
+104
-102
@@ -1,4 +1,7 @@
|
|||||||
// Pure helpers for exporting daily-parameter data as a date×subject CSV matrix.
|
// Pure helpers for exporting daily-parameter data as a CSV matrix. The export is a
|
||||||
|
// subject-series matrix: subjects are always columns, the chosen X coordinate
|
||||||
|
// (Date / # Days Reach / a daily field) provides the rows, and one Data
|
||||||
|
// parameter supplies the cell values.
|
||||||
// No React, no I/O — mirrors the crossSubjectChart.js pure-helper pattern.
|
// No React, no I/O — mirrors the crossSubjectChart.js pure-helper pattern.
|
||||||
|
|
||||||
// Read a single parameter's raw value from one daily status.
|
// Read a single parameter's raw value from one daily status.
|
||||||
@@ -54,18 +57,104 @@ export function listExportableParams(dailyTemplate, statuses) {
|
|||||||
return params;
|
return params;
|
||||||
}
|
}
|
||||||
|
|
||||||
export const EXPORT_DIMENSIONS = [
|
// Numeric value of a daily field on one status (mirrors the metrics plot's helper).
|
||||||
{ dim: 'date', label: 'Date' },
|
export function numericFieldVal(status, field) {
|
||||||
{ dim: 'subject', label: 'Subject' },
|
if (!field) return null;
|
||||||
{ dim: 'parameter', label: 'Parameter' },
|
const raw = field.builtin ? status?.[field.key] : status?.custom_fields?.[field.fieldId];
|
||||||
];
|
const n = parseFloat(raw);
|
||||||
|
return Number.isNaN(n) ? null : n;
|
||||||
export function dimLabel(dim) {
|
|
||||||
return EXPORT_DIMENSIONS.find((d) => d.dim === dim)?.label ?? '';
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export function otherDimension(rowDim, colDim) {
|
// Ordered row members for the chosen X coordinate.
|
||||||
return EXPORT_DIMENSIONS.map((d) => d.dim).find((d) => d !== rowDim && d !== colDim) ?? null;
|
// xField: '__date__' | '__days__' | <daily fieldId>
|
||||||
|
export function listSampleMembers(statuses, xField, dailyTemplate) {
|
||||||
|
const list = statuses ?? [];
|
||||||
|
if (xField === '__date__') {
|
||||||
|
const keys = new Set();
|
||||||
|
for (const s of list) if (s?.date) keys.add(dateKey(s.date));
|
||||||
|
return [...keys].sort().map((k) => ({ id: k, label: k, sampleValue: k }));
|
||||||
|
}
|
||||||
|
if (xField === '__days__') {
|
||||||
|
const counts = new Map();
|
||||||
|
for (const s of list) {
|
||||||
|
if (s?.animal_id == null || !s?.date) continue;
|
||||||
|
counts.set(s.animal_id, (counts.get(s.animal_id) ?? 0) + 1);
|
||||||
|
}
|
||||||
|
const max = counts.size ? Math.max(...counts.values()) : 0;
|
||||||
|
return Array.from({ length: max }, (_, i) => ({ id: String(i + 1), label: String(i + 1), sampleValue: i + 1 }));
|
||||||
|
}
|
||||||
|
const field = (dailyTemplate ?? []).find((f) => f.fieldId === xField);
|
||||||
|
const vals = new Set();
|
||||||
|
for (const s of list) {
|
||||||
|
const v = numericFieldVal(s, field);
|
||||||
|
if (v !== null) vals.add(v);
|
||||||
|
}
|
||||||
|
return [...vals].sort((a, b) => a - b).map((v) => ({ id: String(v), label: String(v), sampleValue: v }));
|
||||||
|
}
|
||||||
|
|
||||||
|
// animalId -> Map(sampleValue -> status). First status per (subject, sampleValue) wins,
|
||||||
|
// scanning each subject's statuses in ascending date order.
|
||||||
|
export function buildSampleIndex(statuses, xField, dailyTemplate) {
|
||||||
|
const bySubject = new Map();
|
||||||
|
for (const s of statuses ?? []) {
|
||||||
|
if (s?.animal_id == null || !s?.date) continue;
|
||||||
|
if (!bySubject.has(s.animal_id)) bySubject.set(s.animal_id, []);
|
||||||
|
bySubject.get(s.animal_id).push(s);
|
||||||
|
}
|
||||||
|
const field = xField !== '__date__' && xField !== '__days__'
|
||||||
|
? (dailyTemplate ?? []).find((f) => f.fieldId === xField)
|
||||||
|
: null;
|
||||||
|
|
||||||
|
const index = new Map();
|
||||||
|
for (const [animalId, subjStatuses] of bySubject) {
|
||||||
|
const ordered = [...subjStatuses].sort((a, b) => dateKey(a.date).localeCompare(dateKey(b.date)));
|
||||||
|
const m = new Map();
|
||||||
|
ordered.forEach((s, i) => {
|
||||||
|
let key;
|
||||||
|
if (xField === '__date__') key = dateKey(s.date);
|
||||||
|
else if (xField === '__days__') key = i + 1;
|
||||||
|
else { const v = numericFieldVal(s, field); if (v === null) return; key = v; }
|
||||||
|
if (!m.has(key)) m.set(key, s);
|
||||||
|
});
|
||||||
|
index.set(animalId, m);
|
||||||
|
}
|
||||||
|
return index;
|
||||||
|
}
|
||||||
|
|
||||||
|
function xFieldLabel(xField, dailyTemplate) {
|
||||||
|
if (xField === '__date__') return 'Date';
|
||||||
|
if (xField === '__days__') return '# Days Reach (ordinal)';
|
||||||
|
return (dailyTemplate ?? []).find((f) => f.fieldId === xField)?.label ?? '';
|
||||||
|
}
|
||||||
|
|
||||||
|
// Subjects as columns, X-field values as rows, one Data value per cell.
|
||||||
|
// Returns the { context, corner, groupAxis, columns, rows } shape toCSV consumes.
|
||||||
|
export function buildSubjectSeriesMatrix({ xField, dataParam, groupField }, { statuses, animals, dailyTemplate }) {
|
||||||
|
const columns = listSubjectMembers(animals, groupField);
|
||||||
|
const rowMembers = listSampleMembers(statuses, xField, dailyTemplate);
|
||||||
|
const index = buildSampleIndex(statuses, xField, dailyTemplate);
|
||||||
|
|
||||||
|
const cellValue = (sampleValue, animalId) => {
|
||||||
|
const status = index.get(animalId)?.get(sampleValue);
|
||||||
|
if (!status || !dataParam) return '';
|
||||||
|
const v = extractValue(status, dataParam);
|
||||||
|
return hasValue(v) ? v : '';
|
||||||
|
};
|
||||||
|
|
||||||
|
let rows = rowMembers.map((rm) => ({
|
||||||
|
member: rm,
|
||||||
|
values: columns.map((c) => cellValue(rm.sampleValue, c.animalId)),
|
||||||
|
}));
|
||||||
|
rows = rows.filter((r) => r.values.some((v) => v !== '')); // drop entirely-blank sample rows
|
||||||
|
|
||||||
|
const grouped = !!groupField && groupField !== '__none__';
|
||||||
|
return {
|
||||||
|
corner: xFieldLabel(xField, dailyTemplate),
|
||||||
|
context: { label: 'Data', value: dataParam?.label ?? '' },
|
||||||
|
groupAxis: grouped ? 'col' : null,
|
||||||
|
columns,
|
||||||
|
rows,
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Resolve a subject's group value for a given group field id.
|
// Resolve a subject's group value for a given group field id.
|
||||||
@@ -79,29 +168,6 @@ export function subjectGroupValue(animal, groupField) {
|
|||||||
return v === null || v === undefined || v === '' ? '—' : v;
|
return v === null || v === undefined || v === '' ? '—' : v;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function listDateMembers(statuses) {
|
|
||||||
const keys = new Set();
|
|
||||||
for (const s of statuses ?? []) {
|
|
||||||
if (s?.date) keys.add(dateKey(s.date));
|
|
||||||
}
|
|
||||||
return [...keys].sort().map((k) => ({ id: k, label: k, dateKey: k }));
|
|
||||||
}
|
|
||||||
|
|
||||||
// dateKey -> Map(animalId -> status); first status per (date, animal) wins,
|
|
||||||
// scanning in date order (matches the legacy buildMatrix tie-break).
|
|
||||||
export function buildStatusIndex(statuses) {
|
|
||||||
const index = new Map();
|
|
||||||
const ordered = [...(statuses ?? [])].sort((a, b) => dateKey(a?.date).localeCompare(dateKey(b?.date)));
|
|
||||||
for (const s of ordered) {
|
|
||||||
if (!s?.date || s.animal_id === null || s.animal_id === undefined) continue;
|
|
||||||
const dk = dateKey(s.date);
|
|
||||||
if (!index.has(dk)) index.set(dk, new Map());
|
|
||||||
const byAnimal = index.get(dk);
|
|
||||||
if (!byAnimal.has(s.animal_id)) byAnimal.set(s.animal_id, s);
|
|
||||||
}
|
|
||||||
return index;
|
|
||||||
}
|
|
||||||
|
|
||||||
export function listSubjectMembers(animals, groupField) {
|
export function listSubjectMembers(animals, groupField) {
|
||||||
const list = [...(animals ?? [])];
|
const list = [...(animals ?? [])];
|
||||||
const nameCounts = new Map();
|
const nameCounts = new Map();
|
||||||
@@ -112,7 +178,7 @@ export function listSubjectMembers(animals, groupField) {
|
|||||||
const members = list.map((a) => {
|
const members = list.map((a) => {
|
||||||
const name = a?.animal_name ?? '';
|
const name = a?.animal_name ?? '';
|
||||||
const label = nameCounts.get(name) > 1 ? `${name} (${a?.animal_id_string ?? ''})` : name;
|
const label = nameCounts.get(name) > 1 ? `${name} (${a?.animal_id_string ?? ''})` : name;
|
||||||
return { id: a.id, label, animalId: a.id, group: subjectGroupValue(a, groupField) };
|
return { id: a?.id, label, animalId: a?.id, group: subjectGroupValue(a, groupField) };
|
||||||
});
|
});
|
||||||
const grouped = !!groupField && groupField !== '__none__';
|
const grouped = !!groupField && groupField !== '__none__';
|
||||||
members.sort((x, y) => {
|
members.sort((x, y) => {
|
||||||
@@ -125,19 +191,6 @@ export function listSubjectMembers(animals, groupField) {
|
|||||||
return members;
|
return members;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function listParameterMembers(dailyTemplate, statuses) {
|
|
||||||
return listExportableParams(dailyTemplate, statuses).map((p) => ({
|
|
||||||
id: p.id, label: p.label, param: p, group: p.group,
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
|
|
||||||
export function listDimensionMembers(dim, { statuses, animals, dailyTemplate, groupField }) {
|
|
||||||
if (dim === 'date') return listDateMembers(statuses);
|
|
||||||
if (dim === 'subject') return listSubjectMembers(animals, groupField);
|
|
||||||
if (dim === 'parameter') return listParameterMembers(dailyTemplate, statuses);
|
|
||||||
return [];
|
|
||||||
}
|
|
||||||
|
|
||||||
// A cell has a value unless it is null, undefined, or the empty string.
|
// A cell has a value unless it is null, undefined, or the empty string.
|
||||||
// (0 and false are real values.)
|
// (0 and false are real values.)
|
||||||
function hasValue(v) {
|
function hasValue(v) {
|
||||||
@@ -148,66 +201,13 @@ function dateKey(date) {
|
|||||||
return String(date).slice(0, 10); // ISO datetime or date → YYYY-MM-DD
|
return String(date).slice(0, 10); // ISO datetime or date → YYYY-MM-DD
|
||||||
}
|
}
|
||||||
|
|
||||||
// Pivot statuses into a 2-D matrix per the chosen row/column dimensions.
|
|
||||||
// The leftover (pinned) dimension is fixed to pinnedMember. Every cell resolves
|
|
||||||
// a (dateKey, animalId, param) triple through the status index.
|
|
||||||
export function buildMatrix(config, ctx) {
|
|
||||||
const { rowDim, colDim, pinnedMember, groupField } = config;
|
|
||||||
const pinnedDim = otherDimension(rowDim, colDim);
|
|
||||||
const index = buildStatusIndex(ctx.statuses);
|
|
||||||
const full = { ...ctx, groupField };
|
|
||||||
const rowMembers = listDimensionMembers(rowDim, full);
|
|
||||||
const colMembers = listDimensionMembers(colDim, full);
|
|
||||||
|
|
||||||
const cellValue = (rowM, colM) => {
|
|
||||||
const byDim = { [rowDim]: rowM, [colDim]: colM, [pinnedDim]: pinnedMember };
|
|
||||||
const dk = byDim.date?.dateKey ?? null;
|
|
||||||
const animalId = byDim.subject?.animalId ?? null;
|
|
||||||
const param = byDim.parameter?.param ?? null;
|
|
||||||
if (dk === null || animalId === null || animalId === undefined || !param) return '';
|
|
||||||
const status = index.get(dk)?.get(animalId);
|
|
||||||
if (!status) return '';
|
|
||||||
const v = extractValue(status, param);
|
|
||||||
return hasValue(v) ? v : '';
|
|
||||||
};
|
|
||||||
|
|
||||||
let rows = rowMembers.map((rowM) => ({
|
|
||||||
member: rowM,
|
|
||||||
values: colMembers.map((colM) => cellValue(rowM, colM)),
|
|
||||||
}));
|
|
||||||
|
|
||||||
// Empty-member rule: keep all subjects; drop entirely-blank date/parameter rows & columns.
|
|
||||||
if (rowDim !== 'subject') rows = rows.filter((r) => r.values.some((v) => v !== ''));
|
|
||||||
|
|
||||||
let keptCols = colMembers.map((_, i) => i);
|
|
||||||
if (colDim !== 'subject') keptCols = keptCols.filter((i) => rows.some((r) => r.values[i] !== ''));
|
|
||||||
const columns = keptCols.map((i) => colMembers[i]);
|
|
||||||
rows = rows.map((r) => ({ member: r.member, values: keptCols.map((i) => r.values[i]) }));
|
|
||||||
|
|
||||||
const grouped = !!groupField && groupField !== '__none__';
|
|
||||||
const groupAxis = grouped
|
|
||||||
? (rowDim === 'subject' ? 'row' : colDim === 'subject' ? 'col' : null)
|
|
||||||
: null;
|
|
||||||
|
|
||||||
return {
|
|
||||||
rowDim,
|
|
||||||
colDim,
|
|
||||||
pinnedDim,
|
|
||||||
corner: dimLabel(rowDim),
|
|
||||||
context: { label: dimLabel(pinnedDim), value: pinnedMember?.label ?? '' },
|
|
||||||
groupAxis,
|
|
||||||
columns,
|
|
||||||
rows,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
function escapeCsvCell(value) {
|
function escapeCsvCell(value) {
|
||||||
const s = value === null || value === undefined ? '' : String(value);
|
const s = value === null || value === undefined ? '' : String(value);
|
||||||
return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s;
|
return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Serialize a matrix to an RFC-4180-ish CSV string (LF line endings):
|
// Serialize a matrix to an RFC-4180-ish CSV string (LF line endings):
|
||||||
// <pinned label>,<pinned value>
|
// Data,<metric>
|
||||||
// (blank line)
|
// (blank line)
|
||||||
// [Group,<group per subject column>] -- only when groupAxis === 'col'
|
// [Group,<group per subject column>] -- only when groupAxis === 'col'
|
||||||
// <corner>[,Group?],<column headers...>
|
// <corner>[,Group?],<column headers...>
|
||||||
@@ -222,6 +222,8 @@ export function toCSV(matrix) {
|
|||||||
lines.push(groupRow.map(escapeCsvCell).join(','));
|
lines.push(groupRow.map(escapeCsvCell).join(','));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// toCSV is a general serializer; the subject-series builder never emits groupAxis
|
||||||
|
// === 'row' (only 'col' or null), but the branch is retained for that generality.
|
||||||
const header = matrix.groupAxis === 'row'
|
const header = matrix.groupAxis === 'row'
|
||||||
? ['Group', matrix.corner, ...matrix.columns.map((c) => c.label)]
|
? ['Group', matrix.corner, ...matrix.columns.map((c) => c.label)]
|
||||||
: [matrix.corner, ...matrix.columns.map((c) => c.label)];
|
: [matrix.corner, ...matrix.columns.map((c) => c.label)];
|
||||||
|
|||||||
@@ -417,7 +417,7 @@ export default function ExperimentDetail() {
|
|||||||
dailyTemplate={dailyTemplate}
|
dailyTemplate={dailyTemplate}
|
||||||
animals={animals}
|
animals={animals}
|
||||||
subjectTemplate={subjectTemplate}
|
subjectTemplate={subjectTemplate}
|
||||||
groupField={groupField}
|
groupField={localStorage.getItem(`exp-subject-group-${id}`) ?? '__none__'}
|
||||||
onClose={() => setShowExport(false)}
|
onClose={() => setShowExport(false)}
|
||||||
/>
|
/>
|
||||||
</Modal>
|
</Modal>
|
||||||
|
|||||||
@@ -12,8 +12,9 @@ const animals = [
|
|||||||
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||||
];
|
];
|
||||||
const statuses = [
|
const statuses = [
|
||||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 5 } },
|
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5, success_rate: 0.5 } },
|
||||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 9 } },
|
{ animal_id: 'a1', date: '2026-07-02T00:00:00Z', analysis_summary: { total: 7, success_rate: 0.7 } },
|
||||||
|
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 9, success_rate: 0.9 } },
|
||||||
];
|
];
|
||||||
const subjectTemplate = [{ fieldId: 'grp', label: 'Group', active: true }];
|
const subjectTemplate = [{ fieldId: 'grp', label: 'Group', active: true }];
|
||||||
|
|
||||||
@@ -22,6 +23,37 @@ beforeEach(() => {
|
|||||||
client.experimentsApi.getDailyStatuses.mockResolvedValue(statuses);
|
client.experimentsApi.getDailyStatuses.mockResolvedValue(statuses);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Intercept the Blob handed to URL.createObjectURL so tests can read the real CSV.
|
||||||
|
// Also stub anchor.click() — jsdom treats a click on <a href="blob:..."> as an
|
||||||
|
// unimplemented navigation that throws async and can flake a later test.
|
||||||
|
function mockDownload() {
|
||||||
|
let blob = null;
|
||||||
|
const origCreate = global.URL.createObjectURL;
|
||||||
|
const origRevoke = global.URL.revokeObjectURL;
|
||||||
|
const origClick = window.HTMLAnchorElement.prototype.click;
|
||||||
|
global.URL.createObjectURL = jest.fn((b) => { blob = b; return 'blob:mock'; });
|
||||||
|
global.URL.revokeObjectURL = jest.fn();
|
||||||
|
window.HTMLAnchorElement.prototype.click = jest.fn();
|
||||||
|
return {
|
||||||
|
getBlob: () => blob,
|
||||||
|
restore: async () => {
|
||||||
|
await new Promise((r) => setTimeout(r, 0)); // flush handleExport's setTimeout revoke
|
||||||
|
global.URL.createObjectURL = origCreate;
|
||||||
|
global.URL.revokeObjectURL = origRevoke;
|
||||||
|
window.HTMLAnchorElement.prototype.click = origClick;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function readBlobText(blob) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
const fr = new FileReader();
|
||||||
|
fr.onload = () => resolve(fr.result);
|
||||||
|
fr.onerror = reject;
|
||||||
|
fr.readAsText(blob);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
function renderModal(props = {}) {
|
function renderModal(props = {}) {
|
||||||
return render(
|
return render(
|
||||||
<ExportDataModal
|
<ExportDataModal
|
||||||
@@ -38,27 +70,55 @@ function renderModal(props = {}) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
describe('ExportDataModal', () => {
|
describe('ExportDataModal', () => {
|
||||||
it('defaults to Rows=Date, Columns=Subject and shows the preview', async () => {
|
it('renders X / Data / Group by pickers; X defaults to Date', async () => {
|
||||||
renderModal();
|
renderModal();
|
||||||
expect(await screen.findByLabelText('Rows')).toHaveValue('date');
|
expect(await screen.findByLabelText('X axis (rows)')).toHaveValue('__date__');
|
||||||
expect(screen.getByLabelText('Columns')).toHaveValue('subject');
|
expect(screen.getByLabelText('Data (cells)')).toBeInTheDocument();
|
||||||
await waitFor(() => expect(screen.getByText(/1 date.*2 subjects/i)).toBeInTheDocument());
|
expect(screen.getByLabelText('Group by')).toHaveValue('__none__');
|
||||||
|
await waitFor(() => expect(screen.getByText(/2 rows × 2 subjects/i)).toBeInTheDocument());
|
||||||
});
|
});
|
||||||
|
|
||||||
it('hides the Group by control until Subject is an axis', async () => {
|
it('default export (X=Date, Data=Total attempts) has subject columns, date rows, blank where missing', async () => {
|
||||||
|
const dl = mockDownload();
|
||||||
renderModal();
|
renderModal();
|
||||||
await screen.findByLabelText('Rows');
|
await screen.findByLabelText('X axis (rows)');
|
||||||
// subject is a column by default -> Group by visible
|
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||||
expect(screen.getByLabelText('Group by')).toBeInTheDocument();
|
fireEvent.click(screen.getByText('Export CSV'));
|
||||||
// move subject off both axes: Rows=Date, Columns=Parameter -> subject pinned
|
const text = await readBlobText(dl.getBlob());
|
||||||
fireEvent.change(screen.getByLabelText('Columns'), { target: { value: 'parameter' } });
|
expect(text).toBe('Data,Total attempts\n\nDate,Alpha,Beta\n2026-07-01,5,9\n2026-07-02,7,');
|
||||||
await waitFor(() => expect(screen.queryByLabelText('Group by')).not.toBeInTheDocument());
|
await dl.restore();
|
||||||
});
|
});
|
||||||
|
|
||||||
it('keeps row and column dimensions distinct', async () => {
|
it('switching X to # Days Reach makes rows the day ordinals', async () => {
|
||||||
|
const dl = mockDownload();
|
||||||
renderModal();
|
renderModal();
|
||||||
const rows = await screen.findByLabelText('Rows');
|
fireEvent.change(await screen.findByLabelText('X axis (rows)'), { target: { value: '__days__' } });
|
||||||
fireEvent.change(rows, { target: { value: 'subject' } }); // collides with column=subject
|
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||||
await waitFor(() => expect(screen.getByLabelText('Columns')).not.toHaveValue('subject'));
|
fireEvent.click(screen.getByText('Export CSV'));
|
||||||
|
const text = await readBlobText(dl.getBlob());
|
||||||
|
expect(text).toBe('Data,Total attempts\n\n# Days Reach (ordinal),Alpha,Beta\n1,5,9\n2,7,');
|
||||||
|
await dl.restore();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('a valid group field adds a Group row and clusters subjects', async () => {
|
||||||
|
const dl = mockDownload();
|
||||||
|
renderModal({ groupField: 'grp' });
|
||||||
|
await screen.findByLabelText('X axis (rows)');
|
||||||
|
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||||
|
fireEvent.click(screen.getByText('Export CSV'));
|
||||||
|
const text = await readBlobText(dl.getBlob());
|
||||||
|
expect(text).toMatch(/^Group,Control,Drug$/m);
|
||||||
|
await dl.restore();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('a stale/unset group field coerces to None (no Group row in the export)', async () => {
|
||||||
|
const dl = mockDownload();
|
||||||
|
renderModal({ groupField: 'ghost-field' });
|
||||||
|
await screen.findByLabelText('X axis (rows)');
|
||||||
|
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||||
|
fireEvent.click(screen.getByText('Export CSV'));
|
||||||
|
const text = await readBlobText(dl.getBlob());
|
||||||
|
expect(text).not.toMatch(/^Group,/m);
|
||||||
|
await dl.restore();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
+118
-128
@@ -71,73 +71,6 @@ describe('listExportableParams', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
import { buildMatrix } from '../src/lib/dataExport';
|
|
||||||
|
|
||||||
const gAnimals = [
|
|
||||||
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
|
||||||
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
|
||||||
];
|
|
||||||
const gStatuses = [
|
|
||||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 5, success_rate: 0.5 } },
|
|
||||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 9, success_rate: 0.9 } },
|
|
||||||
{ animal_id: 'a1', date: '2026-07-02T00:00:00.000Z', analysis_summary: { total: 7, success_rate: 0.7 } },
|
|
||||||
];
|
|
||||||
const ctx = { statuses: gStatuses, animals: gAnimals, dailyTemplate: [] };
|
|
||||||
const totalMember = { id: '__total__', label: 'Total attempts', param: { kind: 'total' }, group: 'metrics' };
|
|
||||||
|
|
||||||
describe('buildMatrix', () => {
|
|
||||||
it('default preset: rows=date, cols=subject, pinned=parameter (legacy layout)', () => {
|
|
||||||
const m = buildMatrix({ rowDim: 'date', colDim: 'subject', pinnedMember: totalMember, groupField: '__none__' }, ctx);
|
|
||||||
expect(m.corner).toBe('Date');
|
|
||||||
expect(m.context).toEqual({ label: 'Parameter', value: 'Total attempts' });
|
|
||||||
expect(m.columns.map((c) => c.label)).toEqual(['Alpha', 'Beta']);
|
|
||||||
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01', '2026-07-02']);
|
|
||||||
expect(m.rows[0].values).toEqual([5, 9]);
|
|
||||||
expect(m.rows[1].values).toEqual([7, '']); // Beta blank on 07-02
|
|
||||||
expect(m.groupAxis).toBeNull();
|
|
||||||
});
|
|
||||||
|
|
||||||
it('drops blank date rows but keeps all subject columns', () => {
|
|
||||||
const statuses = [
|
|
||||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 0 } }, // 0 is a value
|
|
||||||
{ animal_id: 'a1', date: '2026-07-03T00:00:00.000Z', analysis_summary: null }, // no value
|
|
||||||
];
|
|
||||||
const m = buildMatrix({ rowDim: 'date', colDim: 'subject', pinnedMember: totalMember, groupField: '__none__' },
|
|
||||||
{ statuses, animals: gAnimals, dailyTemplate: [] });
|
|
||||||
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01']);
|
|
||||||
expect(m.columns.length).toBe(2);
|
|
||||||
expect(m.rows[0].values).toEqual([0, '']);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('subject as columns sets groupAxis=col and clusters subjects', () => {
|
|
||||||
const m = buildMatrix({ rowDim: 'date', colDim: 'subject', pinnedMember: totalMember, groupField: 'grp' }, ctx);
|
|
||||||
expect(m.groupAxis).toBe('col');
|
|
||||||
expect(m.columns.map((c) => [c.group, c.label])).toEqual([['Control', 'Alpha'], ['Drug', 'Beta']]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('subject as rows sets groupAxis=row; pinned date; cols=parameter', () => {
|
|
||||||
const dateMember = { id: '2026-07-01', label: '2026-07-01', dateKey: '2026-07-01' };
|
|
||||||
const m = buildMatrix({ rowDim: 'subject', colDim: 'parameter', pinnedMember: dateMember, groupField: 'grp' },
|
|
||||||
{ statuses: gStatuses, animals: gAnimals, dailyTemplate: [] });
|
|
||||||
expect(m.corner).toBe('Subject');
|
|
||||||
expect(m.context).toEqual({ label: 'Date', value: '2026-07-01' });
|
|
||||||
expect(m.groupAxis).toBe('row');
|
|
||||||
expect(m.rows.map((r) => [r.member.group, r.member.label])).toEqual([['Control', 'Alpha'], ['Drug', 'Beta']]);
|
|
||||||
// columns are parameters (Total attempts, Success rate); Alpha on 07-01: total 5, rate 0.5
|
|
||||||
const totalCol = m.columns.findIndex((c) => c.label === 'Total attempts');
|
|
||||||
expect(m.rows[0].values[totalCol]).toBe(5);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('drops blank parameter columns (params always dropped when empty)', () => {
|
|
||||||
// dailyTemplate adds a custom field with no data anywhere -> its column drops
|
|
||||||
const dailyTemplate = [{ fieldId: 'c-x', key: 'x', label: 'X', builtin: false, active: true }];
|
|
||||||
const dateMember = { id: '2026-07-01', label: '2026-07-01', dateKey: '2026-07-01' };
|
|
||||||
const m = buildMatrix({ rowDim: 'subject', colDim: 'parameter', pinnedMember: dateMember, groupField: '__none__' },
|
|
||||||
{ statuses: gStatuses, animals: gAnimals, dailyTemplate });
|
|
||||||
expect(m.columns.map((c) => c.label)).not.toContain('X');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
import { toCSV, csvFilename } from '../src/lib/dataExport';
|
import { toCSV, csvFilename } from '../src/lib/dataExport';
|
||||||
|
|
||||||
describe('toCSV', () => {
|
describe('toCSV', () => {
|
||||||
@@ -206,23 +139,7 @@ describe('csvFilename', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
import {
|
import { subjectGroupValue } from '../src/lib/dataExport';
|
||||||
EXPORT_DIMENSIONS, dimLabel, otherDimension, subjectGroupValue,
|
|
||||||
listDateMembers, buildStatusIndex,
|
|
||||||
} from '../src/lib/dataExport';
|
|
||||||
|
|
||||||
describe('dimension primitives', () => {
|
|
||||||
it('exposes the three dimensions with labels', () => {
|
|
||||||
expect(EXPORT_DIMENSIONS.map((d) => d.dim)).toEqual(['date', 'subject', 'parameter']);
|
|
||||||
expect(dimLabel('date')).toBe('Date');
|
|
||||||
expect(dimLabel('subject')).toBe('Subject');
|
|
||||||
expect(dimLabel('parameter')).toBe('Parameter');
|
|
||||||
});
|
|
||||||
it('otherDimension returns the leftover', () => {
|
|
||||||
expect(otherDimension('date', 'subject')).toBe('parameter');
|
|
||||||
expect(otherDimension('parameter', 'date')).toBe('subject');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('subjectGroupValue', () => {
|
describe('subjectGroupValue', () => {
|
||||||
const animal = { animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { sex: 'M', cohort: '' } };
|
const animal = { animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { sex: 'M', cohort: '' } };
|
||||||
@@ -241,37 +158,7 @@ describe('subjectGroupValue', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('listDateMembers', () => {
|
import { listSubjectMembers } from '../src/lib/dataExport';
|
||||||
it('returns sorted unique dateKeys with id/label', () => {
|
|
||||||
const statuses = [
|
|
||||||
{ date: '2026-07-02T00:00:00.000Z' },
|
|
||||||
{ date: '2026-07-01T12:00:00.000Z' },
|
|
||||||
{ date: '2026-07-02T09:00:00.000Z' },
|
|
||||||
];
|
|
||||||
expect(listDateMembers(statuses)).toEqual([
|
|
||||||
{ id: '2026-07-01', label: '2026-07-01', dateKey: '2026-07-01' },
|
|
||||||
{ id: '2026-07-02', label: '2026-07-02', dateKey: '2026-07-02' },
|
|
||||||
]);
|
|
||||||
});
|
|
||||||
it('tolerates empty input', () => {
|
|
||||||
expect(listDateMembers(undefined)).toEqual([]);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('buildStatusIndex', () => {
|
|
||||||
it('indexes by dateKey then animalId, first status wins', () => {
|
|
||||||
const statuses = [
|
|
||||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 7 } },
|
|
||||||
{ animal_id: 'a1', date: '2026-07-01T06:00:00.000Z', analysis_summary: { total: 99 } },
|
|
||||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 3 } },
|
|
||||||
];
|
|
||||||
const idx = buildStatusIndex(statuses);
|
|
||||||
expect(idx.get('2026-07-01').get('a1').analysis_summary.total).toBe(7);
|
|
||||||
expect(idx.get('2026-07-01').get('a2').analysis_summary.total).toBe(3);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
import { listSubjectMembers, listParameterMembers, listDimensionMembers } from '../src/lib/dataExport';
|
|
||||||
|
|
||||||
describe('listSubjectMembers', () => {
|
describe('listSubjectMembers', () => {
|
||||||
const animals = [
|
const animals = [
|
||||||
@@ -300,19 +187,122 @@ describe('listSubjectMembers', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('listParameterMembers / listDimensionMembers', () => {
|
import { numericFieldVal, listSampleMembers, buildSampleIndex } from '../src/lib/dataExport';
|
||||||
const dailyTemplate = [{ fieldId: 'b-vitals', key: 'vitals', label: 'Vitals', builtin: true, active: true }];
|
|
||||||
const statuses = [{ date: '2026-07-01T00:00:00.000Z', animal_id: 'a1', analysis_summary: { total: 1 } }];
|
const sDaily = [{ fieldId: 'c-w', key: 'w', label: 'Weight', builtin: false, active: true }];
|
||||||
const animals = [{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001' }];
|
const sStatuses = [
|
||||||
it('wraps params with id/label/param/group', () => {
|
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', custom_fields: { 'c-w': '250' } },
|
||||||
const m = listParameterMembers(dailyTemplate, statuses);
|
{ animal_id: 'a1', date: '2026-07-03T00:00:00Z', custom_fields: { 'c-w': '260' } },
|
||||||
expect(m[0]).toMatchObject({ id: '__total__', label: 'Total attempts', group: 'metrics' });
|
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', custom_fields: { 'c-w': '250' } },
|
||||||
expect(m[0].param).toMatchObject({ kind: 'total' });
|
];
|
||||||
|
|
||||||
|
describe('numericFieldVal', () => {
|
||||||
|
it('reads builtin via key and custom via fieldId, parsed to number', () => {
|
||||||
|
expect(numericFieldVal({ weight: '250' }, { builtin: true, key: 'weight' })).toBe(250);
|
||||||
|
expect(numericFieldVal({ custom_fields: { 'c-w': '12.5' } }, { builtin: false, fieldId: 'c-w' })).toBe(12.5);
|
||||||
});
|
});
|
||||||
it('dispatches by dimension', () => {
|
it('returns null for non-numeric, missing, or no field', () => {
|
||||||
const ctx = { statuses, animals, dailyTemplate, groupField: '__none__' };
|
expect(numericFieldVal({ custom_fields: { 'c-w': 'abc' } }, { builtin: false, fieldId: 'c-w' })).toBeNull();
|
||||||
expect(listDimensionMembers('date', ctx).map((d) => d.id)).toEqual(['2026-07-01']);
|
expect(numericFieldVal({}, { builtin: true, key: 'weight' })).toBeNull();
|
||||||
expect(listDimensionMembers('subject', ctx).map((d) => d.id)).toEqual(['a1']);
|
expect(numericFieldVal({ weight: 1 }, null)).toBeNull();
|
||||||
expect(listDimensionMembers('parameter', ctx)[0].id).toBe('__total__');
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('listSampleMembers', () => {
|
||||||
|
it('date: distinct dates sorted', () => {
|
||||||
|
expect(listSampleMembers(sStatuses, '__date__', sDaily).map((m) => m.sampleValue)).toEqual(['2026-07-01', '2026-07-03']);
|
||||||
|
});
|
||||||
|
it('# days reach: 1..max records per subject', () => {
|
||||||
|
expect(listSampleMembers(sStatuses, '__days__', sDaily).map((m) => m.sampleValue)).toEqual([1, 2]);
|
||||||
|
});
|
||||||
|
it('daily field: distinct numeric values sorted ascending', () => {
|
||||||
|
expect(listSampleMembers(sStatuses, 'c-w', sDaily).map((m) => m.sampleValue)).toEqual([250, 260]);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('buildSampleIndex', () => {
|
||||||
|
it('date coord maps animalId -> dateKey -> status', () => {
|
||||||
|
const idx = buildSampleIndex(sStatuses, '__date__', sDaily);
|
||||||
|
expect(idx.get('a1').get('2026-07-03').date).toContain('2026-07-03');
|
||||||
|
});
|
||||||
|
it('# days reach assigns per-subject ordinals in date order', () => {
|
||||||
|
const idx = buildSampleIndex(sStatuses, '__days__', sDaily);
|
||||||
|
expect(idx.get('a1').get(1).date).toContain('2026-07-01');
|
||||||
|
expect(idx.get('a1').get(2).date).toContain('2026-07-03');
|
||||||
|
expect(idx.get('a2').get(1).date).toContain('2026-07-01');
|
||||||
|
expect(idx.get('a2').has(2)).toBe(false);
|
||||||
|
});
|
||||||
|
it('daily field maps numeric value to first status by date', () => {
|
||||||
|
const idx = buildSampleIndex(sStatuses, 'c-w', sDaily);
|
||||||
|
expect(idx.get('a1').get(250).date).toContain('2026-07-01');
|
||||||
|
expect(idx.get('a1').get(260).date).toContain('2026-07-03');
|
||||||
|
});
|
||||||
|
it('# days reach counts per-status, not per-distinct-date (same-day statuses get separate ordinals)', () => {
|
||||||
|
const sameDay = [
|
||||||
|
{ animal_id: 'z1', date: '2026-07-01T08:00:00Z', analysis_summary: { total: 1 } },
|
||||||
|
{ animal_id: 'z1', date: '2026-07-01T20:00:00Z', analysis_summary: { total: 2 } },
|
||||||
|
];
|
||||||
|
const idx = buildSampleIndex(sameDay, '__days__', []);
|
||||||
|
expect(idx.get('z1').has(1)).toBe(true);
|
||||||
|
expect(idx.get('z1').has(2)).toBe(true);
|
||||||
|
expect(idx.get('z1').has(3)).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
import { buildSubjectSeriesMatrix } from '../src/lib/dataExport';
|
||||||
|
|
||||||
|
const ssAnimals = [
|
||||||
|
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
||||||
|
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||||
|
];
|
||||||
|
const ssStatuses = [
|
||||||
|
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5 } },
|
||||||
|
{ animal_id: 'a1', date: '2026-07-02T00:00:00Z', analysis_summary: { total: 7 } },
|
||||||
|
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 9 } },
|
||||||
|
];
|
||||||
|
const totalParam = { id: '__total__', label: 'Total attempts', kind: 'total', group: 'metrics' };
|
||||||
|
|
||||||
|
describe('buildSubjectSeriesMatrix', () => {
|
||||||
|
it('date X: subjects as columns, date rows, blank where missing, context names the metric', () => {
|
||||||
|
const m = buildSubjectSeriesMatrix(
|
||||||
|
{ xField: '__date__', dataParam: totalParam, groupField: '__none__' },
|
||||||
|
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||||
|
);
|
||||||
|
expect(m.corner).toBe('Date');
|
||||||
|
expect(m.context).toEqual({ label: 'Data', value: 'Total attempts' });
|
||||||
|
expect(m.groupAxis).toBeNull();
|
||||||
|
expect(m.columns.map((c) => c.label)).toEqual(['Alpha', 'Beta']);
|
||||||
|
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01', '2026-07-02']);
|
||||||
|
expect(m.rows[0].values).toEqual([5, 9]);
|
||||||
|
expect(m.rows[1].values).toEqual([7, '']);
|
||||||
|
});
|
||||||
|
it('# days reach X: rows align each subject by day ordinal', () => {
|
||||||
|
const m = buildSubjectSeriesMatrix(
|
||||||
|
{ xField: '__days__', dataParam: totalParam, groupField: '__none__' },
|
||||||
|
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||||
|
);
|
||||||
|
expect(m.corner).toBe('# Days Reach (ordinal)');
|
||||||
|
expect(m.rows.map((r) => r.member.label)).toEqual(['1', '2']);
|
||||||
|
expect(m.rows[0].values).toEqual([5, 9]);
|
||||||
|
expect(m.rows[1].values).toEqual([7, '']);
|
||||||
|
});
|
||||||
|
it('grouping sets groupAxis col and clusters subjects', () => {
|
||||||
|
const m = buildSubjectSeriesMatrix(
|
||||||
|
{ xField: '__date__', dataParam: totalParam, groupField: 'grp' },
|
||||||
|
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||||
|
);
|
||||||
|
expect(m.groupAxis).toBe('col');
|
||||||
|
expect(m.columns.map((c) => [c.group, c.label])).toEqual([['Control', 'Alpha'], ['Drug', 'Beta']]);
|
||||||
|
});
|
||||||
|
it('drops entirely-blank rows', () => {
|
||||||
|
const statuses = [
|
||||||
|
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5 } },
|
||||||
|
{ animal_id: 'a1', date: '2026-07-05T00:00:00Z', analysis_summary: null },
|
||||||
|
];
|
||||||
|
const m = buildSubjectSeriesMatrix(
|
||||||
|
{ xField: '__date__', dataParam: totalParam, groupField: '__none__' },
|
||||||
|
{ statuses, animals: ssAnimals, dailyTemplate: [] },
|
||||||
|
);
|
||||||
|
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01']);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
Reference in New Issue
Block a user