Compare commits
16 Commits
cd4e213d6f
...
5314071c6b
| Author | SHA1 | Date | |
|---|---|---|---|
| 5314071c6b | |||
| 86feec2e77 | |||
| 4f3fe64cad | |||
| 33ee191966 | |||
| 6f23f83bc8 | |||
| 39a5a4ec0c | |||
| 8510903771 | |||
| 3e52117c1a | |||
| 8153777951 | |||
| 9b3d52d89b | |||
| e65de8892a | |||
| 8ab270a9c9 | |||
| 958666580f | |||
| 600323f40a | |||
| 7d112de0d7 | |||
| a411d59305 |
@@ -0,0 +1,158 @@
|
||||
# tDCS reaching study — GLM methods
|
||||
|
||||
This document explains the statistics in `tdcs_glm.py`: the models, the exact
|
||||
formulas, how each subject's progress is accounted for, and the caveats.
|
||||
|
||||
## The question
|
||||
|
||||
Two **known** conditions anchor the performance scale and differ from each other:
|
||||
|
||||
- **H2 (anchor check):** `Electrode-Box-B2` performs **better** than `Electrode-Box-A2`.
|
||||
|
||||
Two **unknown** conditions are then **classified** against those anchors — for
|
||||
each of `Electrode-Box-A` and `Right-Electrode`, is it **A2-like** or **B2-like**?
|
||||
We do *not* assume either belongs to B2; each unknown is compared to *both*
|
||||
anchors, and the anchor it cannot be distinguished from is its likely class.
|
||||
|
||||
There is also a **merged-assumption** mode (`--merge`): assume the two unknowns
|
||||
resolve as `Right-Electrode == Box-B2` and `Electrode-Box-A == Box-A2`, fold them
|
||||
into the anchors, and re-estimate everything with 4 subjects per group.
|
||||
|
||||
## Data and outcome
|
||||
|
||||
- **Outcome:** `success` = successful reaches in a session
|
||||
(`analysis_summary.counts.Success`), a non-negative **count**.
|
||||
- **Attempts:** `total` = reach attempts in the session (`analysis_summary.total`);
|
||||
used as the denominator for the rate model. `success_rate == success / total`.
|
||||
- **Time:** `day` = the "# Days Reach" field (training day, 0..26). Analyses use
|
||||
days ≥ 0 (pre-training negative days and zero-attempt sessions are excluded;
|
||||
zero-attempt sessions are undefined for the rate model).
|
||||
- **Groups (subjects):** Naive (4), Box-A2 (3), Box-B2 (3), Box-A (1),
|
||||
Right-Electrode (1). Under `--merge`: Box-A2 (4), Box-B2 (4), Naive (4).
|
||||
|
||||
**Provenance:** `tdcs_reach_data.csv` was reconstructed from the experiment
|
||||
database and verified cell-by-cell against the exported matrix (71/71
|
||||
unambiguous cells on days 0–5 matched exactly).
|
||||
|
||||
## The three models
|
||||
|
||||
All models use `Electrode-Box-B2` as the **reference** group, so each group term
|
||||
is that group's contrast *versus Box-B2*. `day_c` is the centered training day and
|
||||
`day_c2 = day_c²` captures the rise-then-plateau of the learning curve.
|
||||
|
||||
### (A) Level — count (primary)
|
||||
|
||||
Poisson GEE on the success counts, clustered by subject:
|
||||
|
||||
```
|
||||
success ~ C(group, Treatment('Electrode-Box-B2')) + day_c + day_c2
|
||||
family = Poisson (log link)
|
||||
groups = subject # repeated-measures cluster
|
||||
cov_struct = Exchangeable # working within-subject correlation
|
||||
SE = robust (sandwich)
|
||||
```
|
||||
|
||||
`exp(coef)` for a group term is an **incidence-rate ratio (IRR)**: expected
|
||||
successes relative to Box-B2.
|
||||
|
||||
### (B) Level — rate
|
||||
|
||||
Binomial GLM on successes-out-of-attempts, with cluster-robust SEs by subject:
|
||||
|
||||
```
|
||||
cbind(success, total - success) ~ C(group, Treatment('Electrode-Box-B2')) + day_c + day_c2
|
||||
family = Binomial (logit link)
|
||||
cov_type = cluster (groups = subject)
|
||||
```
|
||||
|
||||
`exp(coef)` is an **odds ratio** for a successful reach relative to Box-B2. This
|
||||
controls for differing numbers of attempts, so it answers "who is more accurate
|
||||
per attempt?" rather than "who attempts more?"
|
||||
|
||||
### (C) Learning rate
|
||||
|
||||
Poisson GEE with a **group × day** interaction, to ask whether groups improve at
|
||||
different *rates* (not just different levels):
|
||||
|
||||
```
|
||||
success ~ C(group, Treatment('Electrode-Box-B2')) * day_c + day_c2
|
||||
```
|
||||
|
||||
Each `group[T.X]:day_c` term is the difference in log-slope versus Box-B2; a joint
|
||||
Wald test asks whether *any* group's slope differs. Large p ⇒ parallel learning.
|
||||
|
||||
### Anchors-only model
|
||||
|
||||
The A2-vs-B2 comparison, refit on **just the two anchor groups** so nothing else
|
||||
influences the shared day terms or the dispersion/correlation nuisance:
|
||||
|
||||
```
|
||||
# subset to {Box-A2, Box-B2}
|
||||
success ~ C(group, Treatment('Electrode-Box-B2')) + day_c + day_c2 # count
|
||||
cbind(success, total-success) ~ C(group, ...) + day_c + day_c2 # rate
|
||||
```
|
||||
|
||||
With B2 as reference the single group term is the A2-vs-B2 effect; H2 is a
|
||||
one-sided test that this coefficient is below zero.
|
||||
|
||||
## How each subject's progress is accounted for
|
||||
|
||||
Two distinct pieces:
|
||||
|
||||
1. **Progress over training** — the `day_c + day_c2` fixed terms model the average
|
||||
learning curve, so groups are compared at comparable points in training rather
|
||||
than being confounded by *when* each was measured.
|
||||
|
||||
2. **Repeated measures / individual baselines** — each subject contributes many
|
||||
correlated sessions and has its own baseline. Handled two ways:
|
||||
|
||||
- **Primary (GEE):** subject is the cluster; an exchangeable working
|
||||
correlation plus robust (sandwich) SEs give *population-average* group
|
||||
effects whose inference is valid under within-subject correlation and
|
||||
Poisson overdispersion.
|
||||
|
||||
- **Sensitivity (mixed model):** a Poisson model with a **per-subject random
|
||||
intercept** (`(1 | subject)`) so each animal gets its own baseline level;
|
||||
the group effects are estimated after allowing for that individual variation.
|
||||
statsmodels has no frequentist Poisson GLMM, so this is the **MAP/Laplace**
|
||||
fit (`fit_vb` diverges on these large counts). Its p-values are approximate —
|
||||
read it as a direction/magnitude check that should agree with GEE.
|
||||
|
||||
## Classification logic (unknowns)
|
||||
|
||||
For each unknown group, `diff_contrast` builds a linear contrast of that group
|
||||
against **each anchor** (both coded vs the B2 reference) and tests it:
|
||||
|
||||
- indistinguishable from B2 **and** different from A2 → **B2-like**
|
||||
- indistinguishable from A2 **and** different from B2 → **A2-like**
|
||||
- indistinguishable from both → **ambiguous** (report the numerically nearer one)
|
||||
- different from both → **unlike both**
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Tiny groups.** Box-A2/B2 have 3 subjects, Naive 4, and each unknown has
|
||||
**n = 1 subject**. Classifying a one-subject condition is weak: "matches
|
||||
anchor X" means "not statistically distinguishable from X," **not** proof of
|
||||
equivalence.
|
||||
- **Single-cluster fragility.** Cluster-robust/GEE inference with one cluster in a
|
||||
group can produce artificially small SEs — most visibly the learning-rate
|
||||
interaction for the n=1 groups; do not read those p-values literally.
|
||||
- **Mixed-model confounding.** With a per-subject random intercept, a group made
|
||||
of one subject is partly confounded with that subject's random intercept, so its
|
||||
fixed effect is shrunk.
|
||||
- **Rate vs count.** "Success" alone is a count; the rate model (success/attempts)
|
||||
is the fairer accuracy comparison when attempt counts differ.
|
||||
|
||||
## Files and usage
|
||||
|
||||
- `tdcs_glm.py` — the analysis (run it directly).
|
||||
- `tdcs_reach_data.csv` — the verified long-format data (`subject, group, day, success, total`).
|
||||
- `tdcs_learning_curves*.png` — per-group learning curves (count and rate panels).
|
||||
|
||||
```
|
||||
python3 analysis/tdcs_glm.py # all training days, 5 groups
|
||||
python3 analysis/tdcs_glm.py --max-day 10 # restrict to days 0–10
|
||||
python3 analysis/tdcs_glm.py --merge # assume Right==B2 and Box-A==A2 (3 groups)
|
||||
```
|
||||
|
||||
Requires: pandas, numpy, scipy, statsmodels, matplotlib.
|
||||
Binary file not shown.
@@ -0,0 +1,440 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
tDCS reaching study — GLM hypothesis tests.
|
||||
|
||||
QUESTION
|
||||
Two KNOWN conditions anchor the scale and differ from each other:
|
||||
H2 (anchor check): Electrode-Box-B2 performs BETTER than Electrode-Box-A2.
|
||||
Two UNKNOWN conditions then get CLASSIFIED against those anchors:
|
||||
For "Electrode-Box-A" and for "Right-Electrode", is each one
|
||||
A2-like or B2-like? (We do not assume either belongs to B2.)
|
||||
Each unknown is compared to BOTH anchors; a group it cannot be
|
||||
distinguished from is its likely class.
|
||||
|
||||
Each comparison is examined through THREE complementary GLMs:
|
||||
|
||||
(A) LEVEL / count Poisson GEE on `success` (successful reaches per
|
||||
session), adjusting for training day. Answers
|
||||
"who makes more successful reaches overall?"
|
||||
|
||||
(B) LEVEL / rate Binomial GLM on success / attempts
|
||||
(`success` out of `total`), cluster-robust.
|
||||
Controls for differing numbers of attempts —
|
||||
"who is more accurate per attempt?"
|
||||
|
||||
(C) LEARNING RATE Poisson GEE with a group x day interaction.
|
||||
Answers "do the groups improve at different
|
||||
RATES?" (slope of the learning curve).
|
||||
|
||||
All models adjust for time with day (and day^2 for the plateau) and account
|
||||
for repeated measures on each subject via GEE (subject = cluster,
|
||||
exchangeable correlation, robust SE) or cluster-robust standard errors.
|
||||
Group is coded with Electrode-Box-B2 as the REFERENCE, so H1 and H2 read
|
||||
directly off the group terms:
|
||||
Right-Electrode term -> H1 (expect ~0 / non-significant)
|
||||
Electrode-Box-A2 term -> H2 (expect < 0: B2 above A2)
|
||||
exp(coef) is an incidence-rate ratio (count model) or odds ratio (rate
|
||||
model) relative to Box-B2.
|
||||
|
||||
CAVEATS
|
||||
* Tiny groups: Box-A2 n=3, Box-B2 n=3, Naive n=4, and each UNKNOWN
|
||||
(Electrode-Box-A, Right-Electrode) has only n=1 SUBJECT. Classifying a
|
||||
1-subject condition is weak: "matches anchor X" means "not statistically
|
||||
distinguishable from X", NOT proof of equivalence, and cluster-robust
|
||||
inference with a single cluster in a group is fragile (can show
|
||||
artificially small SEs, especially in the learning-rate interaction).
|
||||
|
||||
DATA PROVENANCE
|
||||
analysis/tdcs_reach_data.csv was reconstructed from the experiment
|
||||
database (experiment "tDCS", outcome = analysis_summary.counts.Success,
|
||||
attempts = analysis_summary.total, x-axis = the "# Days Reach" field) and
|
||||
verified cell-by-cell against the exported matrix (71/71 unambiguous
|
||||
cells on days 0-5 matched exactly).
|
||||
|
||||
USAGE
|
||||
python3 analysis/tdcs_glm.py
|
||||
Requires: pandas, numpy, scipy, statsmodels, matplotlib.
|
||||
Writes: analysis/tdcs_learning_curves.png
|
||||
"""
|
||||
import os
|
||||
import argparse
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import patsy
|
||||
import statsmodels.api as sm
|
||||
import statsmodels.formula.api as smf
|
||||
from scipy import stats
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
REF = "Electrode-Box-B2"
|
||||
# Two KNOWN anchor conditions that differ (Box-B2 > Box-A2, see H2).
|
||||
ANCHOR_LOW = "Electrode-Box-A2"
|
||||
ANCHOR_HIGH = "Electrode-Box-B2"
|
||||
# Two UNKNOWN conditions to classify: is each one A2-like or B2-like?
|
||||
CANON_UNKNOWNS = ["Electrode-Box-A", "Right-Electrode"]
|
||||
ALL_GROUPS = ["Naive", ANCHOR_LOW, ANCHOR_HIGH] + CANON_UNKNOWNS
|
||||
# The active group config is set at runtime (after any --merge/--assign remap).
|
||||
UNKNOWNS = list(CANON_UNKNOWNS)
|
||||
MAIN = list(ALL_GROUPS)
|
||||
OTHERS = [g for g in MAIN if g != REF]
|
||||
# --merge preset: assume each unknown resolves into an anchor.
|
||||
MERGE_MAP = {"Electrode-Box-A": ANCHOR_LOW, "Right-Electrode": ANCHOR_HIGH}
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DATA = os.path.join(HERE, "tdcs_reach_data.csv")
|
||||
PLOT = os.path.join(HERE, "tdcs_learning_curves.png")
|
||||
RHS = f"C(group, Treatment('{REF}')) + day_c + day_c2"
|
||||
|
||||
|
||||
def gterm(name):
|
||||
"""statsmodels/patsy column name for a group level (vs the reference)."""
|
||||
return f"C(group, Treatment('{REF}'))[T.{name}]"
|
||||
|
||||
|
||||
def xterm(name):
|
||||
"""Interaction column name (group level x day slope)."""
|
||||
return f"C(group, Treatment('{REF}'))[T.{name}]:day_c"
|
||||
|
||||
|
||||
def load(max_day=None, mapping=None):
|
||||
df = pd.read_csv(DATA)
|
||||
if mapping: # remap unknown -> target group
|
||||
df["group"] = df["group"].map(lambda g: mapping.get(g, g))
|
||||
df = df[df["group"].isin(ALL_GROUPS)].copy() # keep canonical groups
|
||||
df = df[df["day"] >= 0].copy() # training days only (day 0..)
|
||||
if max_day is not None:
|
||||
df = df[df["day"] <= max_day].copy() # restrict the analysis window
|
||||
df["day_c"] = df["day"] - df["day"].mean() # center day
|
||||
df["day_c2"] = df["day_c"] ** 2
|
||||
present = [g for g in ALL_GROUPS if g != REF and g in set(df["group"])]
|
||||
df["group"] = pd.Categorical(df["group"], categories=[REF] + present)
|
||||
return df
|
||||
|
||||
|
||||
def contrast(res, tname):
|
||||
"""coef, robust SE, p, effect=exp(coef) and its 95% CI for one term."""
|
||||
ci = res.conf_int()
|
||||
lo, hi = ci.loc[tname]
|
||||
return dict(coef=res.params[tname], se=res.bse[tname], p=res.pvalues[tname],
|
||||
eff=np.exp(res.params[tname]), lo=np.exp(lo), hi=np.exp(hi))
|
||||
|
||||
|
||||
def one_sided_p_below(c):
|
||||
"""One-sided p for H: coef < 0 (i.e. Box-B2 above the compared group)."""
|
||||
return stats.norm.cdf(c["coef"] / c["se"])
|
||||
|
||||
|
||||
def diff_contrast(res, a, b):
|
||||
"""Compare group `a` vs group `b` (both coded relative to REF) via a linear
|
||||
contrast. Returns coef, robust SE, p, effect=exp(coef)=ratio a/b, and CI.
|
||||
Works for any pair, including b == REF (then it is just the `a` term)."""
|
||||
names = list(res.params.index)
|
||||
v = np.zeros(len(names))
|
||||
if a != REF:
|
||||
v[names.index(gterm(a))] += 1.0
|
||||
if b != REF:
|
||||
v[names.index(gterm(b))] -= 1.0
|
||||
tt = res.t_test(v)
|
||||
coef = float(np.ravel(tt.effect)[0])
|
||||
se = float(np.ravel(tt.sd)[0])
|
||||
p = float(np.ravel(tt.pvalue)[0])
|
||||
lo, hi = np.ravel(tt.conf_int())[:2]
|
||||
return dict(coef=coef, se=se, p=p, eff=np.exp(coef),
|
||||
lo=np.exp(lo), hi=np.exp(hi))
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------- models
|
||||
def fit_count_level(df):
|
||||
"""(A) Poisson GEE on success counts; group main effects vs Box-B2."""
|
||||
return smf.gee(f"success ~ {RHS}", groups="subject", data=df,
|
||||
family=sm.families.Poisson(),
|
||||
cov_struct=sm.cov_struct.Exchangeable()).fit()
|
||||
|
||||
|
||||
def fit_rate_level(df):
|
||||
"""(B) Binomial GLM on success/attempts with cluster-robust SE.
|
||||
Zero-attempt sessions (total==0) carry no rate information and are dropped."""
|
||||
d = df[df["total"] > 0].copy()
|
||||
dropped = len(df) - len(d)
|
||||
if dropped:
|
||||
print(f"[rate model] dropped {dropped} zero-attempt session(s) (undefined rate)")
|
||||
X = patsy.dmatrix(RHS, d, return_type="dataframe")
|
||||
endog = np.column_stack([d["success"].values,
|
||||
(d["total"] - d["success"]).values])
|
||||
return sm.GLM(endog, X, family=sm.families.Binomial()).fit(
|
||||
cov_type="cluster", cov_kwds={"groups": d["subject"].values})
|
||||
|
||||
|
||||
def fit_learning_rate(df):
|
||||
"""(C) Poisson GEE with group x day interaction (slope differences)."""
|
||||
return smf.gee(
|
||||
f"success ~ C(group, Treatment('{REF}')) * day_c + day_c2",
|
||||
groups="subject", data=df, family=sm.families.Poisson(),
|
||||
cov_struct=sm.cov_struct.Exchangeable()).fit()
|
||||
|
||||
|
||||
def fit_mixed_count(df):
|
||||
"""Subject random-intercept Poisson mixed model (MAP / Laplace).
|
||||
|
||||
Adds a per-subject random intercept so each subject has its own baseline
|
||||
level; the group fixed effects are then estimated after allowing for that
|
||||
individual variation. A subject-specific complement to the population-
|
||||
average GEE. statsmodels has no frequentist Poisson GLMM, so this is the
|
||||
posterior-mode (MAP) fit with a Laplace covariance for the fixed effects
|
||||
(fit_vb failed to converge on these large counts; MAP is stable)."""
|
||||
import warnings
|
||||
from statsmodels.genmod.bayes_mixed_glm import PoissonBayesMixedGLM
|
||||
vc = {"subject": "0 + C(subject)"} # random intercept per subject
|
||||
model = PoissonBayesMixedGLM.from_formula(f"success ~ {RHS}", vc, df)
|
||||
with warnings.catch_warnings():
|
||||
# MAP stops at |gradient|~5e-5 (effectively converged); silence the
|
||||
# over-strict "did not converge" notice.
|
||||
warnings.filterwarnings("ignore", message="Laplace fitting did not converge")
|
||||
return model.fit_map()
|
||||
|
||||
|
||||
def mixed_contrast(res, tname):
|
||||
"""coef, SD, approx p and IRR + 95% interval from a BayesMixedGLM fit."""
|
||||
names = list(res.model.exog_names)
|
||||
i = names.index(tname)
|
||||
mean, sd = float(res.fe_mean[i]), float(res.fe_sd[i])
|
||||
p = 2.0 * stats.norm.sf(abs(mean / sd))
|
||||
return dict(coef=mean, se=sd, p=p, eff=np.exp(mean),
|
||||
lo=np.exp(mean - 1.96 * sd), hi=np.exp(mean + 1.96 * sd))
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------- reports
|
||||
def describe(df):
|
||||
print("=" * 78)
|
||||
print("DESCRIPTIVES")
|
||||
print("=" * 78)
|
||||
grp = df.groupby("group", observed=True)
|
||||
tbl = pd.DataFrame({
|
||||
"n_subj": grp["subject"].nunique(),
|
||||
"n_sessions": grp.size(),
|
||||
"mean_success": grp["success"].mean().round(1),
|
||||
"mean_rate": (grp["success"].sum() / grp["total"].sum()).round(3),
|
||||
"max_day": grp["day"].max(),
|
||||
})
|
||||
print(tbl.to_string())
|
||||
|
||||
|
||||
def report_level(title, res, effect_label):
|
||||
print("\n" + "=" * 78)
|
||||
print(title)
|
||||
print("=" * 78)
|
||||
print(f"Effect vs {REF} ({effect_label} relative to Box-B2):")
|
||||
h = f"{'group':20s} {'effect':>7s} {'95% CI':>16s} {'coef':>8s} {'SE':>7s} {'p':>9s}"
|
||||
print(h)
|
||||
print("-" * len(h))
|
||||
out = {}
|
||||
for g in OTHERS:
|
||||
c = contrast(res, gterm(g))
|
||||
out[g] = c
|
||||
ci = f"[{c['lo']:.2f}, {c['hi']:.2f}]"
|
||||
print(f"{g:20s} {c['eff']:7.3f} {ci:>16s} {c['coef']:8.3f} {c['se']:7.3f} {c['p']:9.4f}")
|
||||
return out
|
||||
|
||||
|
||||
def report_learning(res):
|
||||
print("\n" + "=" * 78)
|
||||
print("(C) LEARNING RATE — Poisson GEE, group x day interaction")
|
||||
print("=" * 78)
|
||||
base = res.params["day_c"]
|
||||
print(f"Box-B2 learning slope: {np.exp(base):.3f}x successes per training day "
|
||||
f"(baseline).")
|
||||
print("Slope DIFFERENCE vs Box-B2 (exp(coef) = per-day multiplier on the rate ratio):")
|
||||
h = f"{'group':20s} {'slopeΔ/day':>10s} {'coef':>8s} {'SE':>7s} {'p':>9s}"
|
||||
print(h)
|
||||
print("-" * len(h))
|
||||
for g in OTHERS:
|
||||
c = contrast(res, xterm(g))
|
||||
print(f"{g:20s} {c['eff']:10.3f} {c['coef']:8.3f} {c['se']:7.3f} {c['p']:9.4f}")
|
||||
# Joint Wald test: do ANY groups differ from Box-B2 in learning rate?
|
||||
names = list(res.params.index)
|
||||
idx = [names.index(xterm(g)) for g in OTHERS]
|
||||
R = np.zeros((len(idx), len(names)))
|
||||
for i, j in enumerate(idx):
|
||||
R[i, j] = 1.0
|
||||
jt = res.wald_test(R, scalar=True)
|
||||
print(f"\nJoint test (all group x day interactions = 0): "
|
||||
f"chi2={float(jt.statistic):.2f}, df={len(idx)}, p={float(jt.pvalue):.4f}")
|
||||
print(" -> small p = groups improve at DIFFERENT rates; large p = parallel learning.")
|
||||
|
||||
|
||||
def report_anchor_only(df):
|
||||
"""A2-vs-B2 ONLY: refit on just the two anchors so nothing else influences
|
||||
the shared day terms / dispersion. B2 is the reference, so the single group
|
||||
term is the A2-vs-B2 comparison."""
|
||||
print("\n" + "=" * 78)
|
||||
print("ANCHORS ONLY — Box-A2 vs Box-B2 (two-group models)")
|
||||
print("=" * 78)
|
||||
d = df[df["group"].isin([ANCHOR_LOW, ANCHOR_HIGH])].copy()
|
||||
d["day_c"] = d["day"] - d["day"].mean()
|
||||
d["day_c2"] = d["day_c"] ** 2
|
||||
d["group"] = pd.Categorical(d["group"], categories=[ANCHOR_HIGH, ANCHOR_LOW])
|
||||
n_sub = d.groupby("group", observed=True)["subject"].nunique().to_dict()
|
||||
print(f"subjects: Box-B2={n_sub.get(ANCHOR_HIGH)}, Box-A2={n_sub.get(ANCHOR_LOW)}"
|
||||
f" sessions: {len(d)}")
|
||||
print(f"formula: success ~ C(group, Treatment('{REF}')) + day_c + day_c2")
|
||||
for label, res in [("count/level (Poisson GEE)", fit_count_level(d)),
|
||||
("rate/level (Binomial cluster-robust)", fit_rate_level(d))]:
|
||||
c = contrast(res, gterm(ANCHOR_LOW))
|
||||
p1 = one_sided_p_below(c)
|
||||
tag = "SUPPORTED" if (c["coef"] < 0 and p1 < 0.05) else "not supported"
|
||||
print(f" [{label}] A2/B2 = {c['eff']:.2f} [{c['lo']:.2f}, {c['hi']:.2f}] "
|
||||
f"=> B2 = {1.0 / c['eff']:.2f}x A2, one-sided p={p1:.4f} -> H2 {tag}")
|
||||
|
||||
|
||||
def report_mixed(df):
|
||||
"""Subject random-intercept Poisson mixed model (sensitivity vs GEE)."""
|
||||
print("\n" + "=" * 78)
|
||||
print("SENSITIVITY — Poisson MIXED model, per-subject random intercept (MAP/Laplace)")
|
||||
print("=" * 78)
|
||||
try:
|
||||
res = fit_mixed_count(df)
|
||||
except Exception as exc: # VB can be finicky; degrade gracefully
|
||||
print(f" (mixed model skipped: {type(exc).__name__}: {exc})")
|
||||
return
|
||||
print("Group effect vs Box-B2 (IRR; each subject given its own baseline):")
|
||||
h = f"{'group':20s} {'IRR':>7s} {'~95% CI':>16s} {'mean':>8s} {'SD':>7s} {'~p':>9s}"
|
||||
print(h)
|
||||
print("-" * len(h))
|
||||
for g in OTHERS:
|
||||
c = mixed_contrast(res, gterm(g))
|
||||
ci = f"[{c['lo']:.2f}, {c['hi']:.2f}]"
|
||||
print(f"{g:20s} {c['eff']:7.3f} {ci:>16s} {c['coef']:8.3f} {c['se']:7.3f} {c['p']:9.4f}")
|
||||
print(f" subject random-intercept SD = {float(np.exp(res.vcp_mean[0])):.3f} "
|
||||
"(log scale); Laplace SEs, treat p-values as approximate.")
|
||||
print(" Compare directions/magnitudes with the GEE table (they should agree).")
|
||||
print(" Single-subject groups (Box-A, Right-Electrode) are partly confounded")
|
||||
print(" with their own random intercept here, so their effects are shrunk.")
|
||||
|
||||
|
||||
def plot_curves(df, plot_path):
|
||||
fig, ax = plt.subplots(1, 2, figsize=(12, 4.8))
|
||||
for g in MAIN:
|
||||
s = df[df["group"] == g]
|
||||
d = s.groupby("day").apply(
|
||||
lambda x: pd.Series({"succ": x["success"].mean(),
|
||||
"rate": x["success"].sum() / x["total"].sum()}),
|
||||
include_groups=False)
|
||||
ax[0].plot(d.index, d["succ"], marker="o", ms=3, label=g)
|
||||
ax[1].plot(d.index, d["rate"], marker="o", ms=3, label=g)
|
||||
ax[0].set(xlabel="# Days Reach (training day)", ylabel="mean successful reaches",
|
||||
title="(A) Success count learning curves")
|
||||
ax[1].set(xlabel="# Days Reach (training day)", ylabel="success rate (success/attempts)",
|
||||
title="(B) Success-rate learning curves")
|
||||
for a in ax:
|
||||
a.legend(fontsize=8)
|
||||
a.grid(alpha=0.3)
|
||||
fig.tight_layout()
|
||||
fig.savefig(plot_path, dpi=120)
|
||||
print(f"\nSaved learning-curve plot -> {plot_path}")
|
||||
|
||||
|
||||
def classify_one(res, unknown):
|
||||
"""Compare an unknown group against BOTH anchors; return a label + detail."""
|
||||
vlo = diff_contrast(res, unknown, ANCHOR_LOW) # unknown vs Box-A2
|
||||
vhi = diff_contrast(res, unknown, ANCHOR_HIGH) # unknown vs Box-B2
|
||||
like_lo = vlo["p"] >= 0.05 # indistinguishable from Box-A2
|
||||
like_hi = vhi["p"] >= 0.05 # indistinguishable from Box-B2
|
||||
if like_hi and not like_lo:
|
||||
verdict = "B2-like (differs from A2, matches B2)"
|
||||
elif like_lo and not like_hi:
|
||||
verdict = "A2-like (differs from B2, matches A2)"
|
||||
elif like_lo and like_hi:
|
||||
# tie: point toward whichever ratio is closer to 1 on the log scale
|
||||
nearer = ANCHOR_HIGH if abs(vhi["coef"]) < abs(vlo["coef"]) else ANCHOR_LOW
|
||||
verdict = f"AMBIGUOUS (matches both; numerically nearer {nearer})"
|
||||
else:
|
||||
verdict = "UNLIKE BOTH (differs from A2 and B2)"
|
||||
return vlo, vhi, verdict
|
||||
|
||||
|
||||
def verdicts(count, rate):
|
||||
print("\n" + "=" * 78)
|
||||
print("VERDICTS")
|
||||
print("=" * 78)
|
||||
|
||||
print("\nAnchor check — H2: Box-B2 BETTER than Box-A2 (the two anchors must differ)")
|
||||
for label, res in [("count/level", count), ("rate/level ", rate)]:
|
||||
c = contrast(res, gterm(ANCHOR_LOW))
|
||||
p1 = one_sided_p_below(c)
|
||||
tag = "SUPPORTED" if (c["coef"] < 0 and p1 < 0.05) else "not supported"
|
||||
print(f" [{label}] Box-B2 = {1.0 / c['eff']:.2f}x Box-A2 "
|
||||
f"(one-sided p={p1:.4f}) -> {tag}")
|
||||
|
||||
if not UNKNOWNS:
|
||||
print("\n(No unknown groups — they were merged into the anchors; "
|
||||
"only the H2 anchor contrast applies.)")
|
||||
return
|
||||
print("\nClassification — is each UNKNOWN condition A2-like or B2-like?")
|
||||
print("(ratio >1 = above that anchor; p = differs from that anchor)")
|
||||
for u in UNKNOWNS:
|
||||
print(f"\n {u}:")
|
||||
for label, res in [("count/level", count), ("rate/level ", rate)]:
|
||||
vlo, vhi, verdict = classify_one(res, u)
|
||||
print(f" [{label}] vs Box-A2: {vlo['eff']:.2f}x "
|
||||
f"[{vlo['lo']:.2f},{vlo['hi']:.2f}] p={vlo['p']:.3f} "
|
||||
f"vs Box-B2: {vhi['eff']:.2f}x [{vhi['lo']:.2f},{vhi['hi']:.2f}] "
|
||||
f"p={vhi['p']:.3f}")
|
||||
print(f" -> {verdict}")
|
||||
print("\n NOTE: each unknown has ONLY 1 subject. 'Matches' means 'not statistically")
|
||||
print(" distinguishable' — weak evidence at n=1, not proof of equivalence.")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="tDCS GLM hypothesis tests")
|
||||
ap.add_argument("--max-day", type=int, default=None,
|
||||
help="restrict analysis to training days 0..MAX_DAY (default: all)")
|
||||
ap.add_argument("--merge", action="store_true",
|
||||
help="preset: Electrode-Box-A->Box-A2 and Right-Electrode->Box-B2")
|
||||
ap.add_argument("--assign", type=str, default=None,
|
||||
help="custom merges 'UNKNOWN=TARGET,...', e.g. "
|
||||
"'Electrode-Box-A=Electrode-Box-B2,Right-Electrode=Electrode-Box-B2'")
|
||||
args = ap.parse_args()
|
||||
|
||||
mapping = {}
|
||||
if args.merge:
|
||||
mapping.update(MERGE_MAP)
|
||||
if args.assign:
|
||||
for pair in args.assign.split(","):
|
||||
k, v = pair.split("=")
|
||||
mapping[k.strip()] = v.strip()
|
||||
|
||||
df = load(args.max_day, mapping)
|
||||
|
||||
global MAIN, OTHERS, UNKNOWNS
|
||||
present = set(df["group"])
|
||||
MAIN = [g for g in ALL_GROUPS if g in present]
|
||||
OTHERS = [g for g in MAIN if g != REF]
|
||||
UNKNOWNS = [g for g in CANON_UNKNOWNS if g in present]
|
||||
|
||||
window = f"days 0-{args.max_day}" if args.max_day is not None else "all training days (0+)"
|
||||
tag = f"_d0-{args.max_day}" if args.max_day is not None else ""
|
||||
tag += "_merged" if mapping else ""
|
||||
plot_path = os.path.join(HERE, f"tdcs_learning_curves{tag}.png")
|
||||
mode = ("assign " + ", ".join(f"{k.replace('Electrode-','')}->{v.replace('Electrode-','')}"
|
||||
for k, v in mapping.items())) if mapping \
|
||||
else "5 groups (anchors + unknowns)"
|
||||
print(f"ANALYSIS WINDOW: {window} | MODE: {mode} | observations: {len(df)}")
|
||||
describe(df)
|
||||
count = fit_count_level(df)
|
||||
rate = fit_rate_level(df)
|
||||
learn = fit_learning_rate(df)
|
||||
report_level("(A) LEVEL / COUNT — Poisson GEE on successful reaches",
|
||||
count, "incidence-rate ratio")
|
||||
report_level("(B) LEVEL / RATE — Binomial GLM on success/attempts (cluster-robust)",
|
||||
rate, "odds ratio")
|
||||
report_learning(learn)
|
||||
report_anchor_only(df)
|
||||
report_mixed(df)
|
||||
verdicts(count, rate)
|
||||
plot_curves(df, plot_path)
|
||||
print("\nDone. See the module docstring for modeling choices and caveats.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 161 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 154 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 133 KiB |
@@ -0,0 +1,194 @@
|
||||
subject,group,day,success,total
|
||||
Vu-vuong,Naive,0,18,61
|
||||
Vu-vuong,Naive,1,35,91
|
||||
Vu-vuong,Naive,2,61,127
|
||||
Vu-vuong,Naive,3,34,146
|
||||
Vu-vuong,Naive,4,61,141
|
||||
Vu-vuong,Naive,5,37,151
|
||||
Vu-vuong,Naive,6,49,131
|
||||
Vu-vuong,Naive,7,65,149
|
||||
Vu-vuong,Naive,8,67,141
|
||||
Vu-vuong,Naive,9,73,140
|
||||
Vu-vuong,Naive,10,72,131
|
||||
Vu-vuong,Naive,11,89,158
|
||||
Vu-vuong,Naive,12,97,154
|
||||
Vu-vuong,Naive,13,93,145
|
||||
Vu-vuong,Naive,14,75,147
|
||||
Vu-vuong,Naive,15,68,142
|
||||
Vu-vuong,Naive,16,96,162
|
||||
Vu-vuong,Naive,17,68,144
|
||||
Vu-vuong,Naive,18,81,134
|
||||
Vu-vuong,Naive,19,66,144
|
||||
Vu-vuong,Naive,20,84,127
|
||||
Vu-vuong,Naive,21,66,133
|
||||
Khoai-tay-1,Electrode-Box-A,-2,1,18
|
||||
Khoai-tay-1,Electrode-Box-A,-1,3,24
|
||||
Khoai-tay-1,Electrode-Box-A,0,14,62
|
||||
Khoai-tay-1,Electrode-Box-A,1,22,83
|
||||
Khoai-tay-1,Electrode-Box-A,2,6,79
|
||||
Khoai-tay-1,Electrode-Box-A,3,28,97
|
||||
Khoai-tay-1,Electrode-Box-A,4,60,134
|
||||
Khoai-tay-1,Electrode-Box-A,5,75,138
|
||||
Khoai-tay-1,Electrode-Box-A,6,81,137
|
||||
Khoai-tay-1,Electrode-Box-A,7,78,147
|
||||
Khoai-tay-1,Electrode-Box-A,8,91,132
|
||||
Khoai-tay-1,Electrode-Box-A,9,99,146
|
||||
Khoai-tay-1,Electrode-Box-A,10,94,143
|
||||
Khoai-tay-1,Electrode-Box-A,11,110,156
|
||||
Khoai-tay-1,Electrode-Box-A,12,105,143
|
||||
Khoai-tay-1,Electrode-Box-A,13,106,153
|
||||
Khoai-tay-1,Electrode-Box-A,14,91,152
|
||||
Banh-mi-2,Electrode-Box-A2,-1,2,42
|
||||
Banh-mi-2,Electrode-Box-A2,0,25,97
|
||||
Banh-mi-2,Electrode-Box-A2,1,22,101
|
||||
Banh-mi-2,Electrode-Box-A2,2,22,119
|
||||
Banh-mi-2,Electrode-Box-A2,3,29,118
|
||||
Banh-mi-2,Electrode-Box-A2,4,27,136
|
||||
Banh-mi-2,Electrode-Box-A2,5,43,146
|
||||
Banh-mi-2,Electrode-Box-A2,6,74,146
|
||||
Banh-mi-2,Electrode-Box-A2,7,70,148
|
||||
Banh-mi-2,Electrode-Box-A2,8,65,130
|
||||
Banh-mi-2,Electrode-Box-A2,9,79,151
|
||||
Banh-mi-2,Electrode-Box-A2,10,93,152
|
||||
Egg-tart-2,Electrode-Box-A2,0,9,32
|
||||
Egg-tart-2,Electrode-Box-A2,1,2,38
|
||||
Egg-tart-2,Electrode-Box-A2,2,31,93
|
||||
Egg-tart-2,Electrode-Box-A2,3,44,101
|
||||
Egg-tart-2,Electrode-Box-A2,4,54,131
|
||||
Egg-tart-2,Electrode-Box-A2,5,84,139
|
||||
Egg-tart-2,Electrode-Box-A2,6,85,145
|
||||
Egg-tart-2,Electrode-Box-A2,7,79,143
|
||||
Egg-tart-2,Electrode-Box-A2,8,76,131
|
||||
Egg-tart-2,Electrode-Box-A2,9,88,149
|
||||
Egg-tart-2,Electrode-Box-A2,10,81,151
|
||||
Egg-tart-2,Electrode-Box-A2,11,78,152
|
||||
Egg-tart-2,Electrode-Box-A2,12,96,155
|
||||
Egg-tart-2,Electrode-Box-A2,13,84,155
|
||||
Root-beer-2,Electrode-Box-A2,0,22,74
|
||||
Root-beer-2,Electrode-Box-A2,1,31,87
|
||||
Root-beer-2,Electrode-Box-A2,2,49,134
|
||||
Root-beer-2,Electrode-Box-A2,3,31,89
|
||||
Root-beer-2,Electrode-Box-A2,4,60,140
|
||||
Root-beer-2,Electrode-Box-A2,5,84,147
|
||||
Banh-mi-1,Electrode-Box-B2,0,13,84
|
||||
Banh-mi-1,Electrode-Box-B2,1,35,86
|
||||
Banh-mi-1,Electrode-Box-B2,2,47,110
|
||||
Banh-mi-1,Electrode-Box-B2,3,65,140
|
||||
Banh-mi-1,Electrode-Box-B2,4,70,127
|
||||
Banh-mi-1,Electrode-Box-B2,5,102,142
|
||||
Banh-mi-1,Electrode-Box-B2,6,90,131
|
||||
Banh-mi-1,Electrode-Box-B2,7,109,148
|
||||
Banh-mi-1,Electrode-Box-B2,8,104,137
|
||||
Banh-mi-1,Electrode-Box-B2,9,119,150
|
||||
Banh-mi-1,Electrode-Box-B2,10,121,158
|
||||
Banh-mi-1,Electrode-Box-B2,11,121,148
|
||||
Banh-mi-1,Electrode-Box-B2,12,120,149
|
||||
Banh-mi-1,Electrode-Box-B2,13,135,154
|
||||
Egg-tart-1,Electrode-Box-B2,0,7,56
|
||||
Egg-tart-1,Electrode-Box-B2,1,16,78
|
||||
Egg-tart-1,Electrode-Box-B2,2,23,103
|
||||
Egg-tart-1,Electrode-Box-B2,3,63,120
|
||||
Egg-tart-1,Electrode-Box-B2,4,69,132
|
||||
Egg-tart-1,Electrode-Box-B2,5,83,136
|
||||
Egg-tart-1,Electrode-Box-B2,6,71,142
|
||||
Egg-tart-1,Electrode-Box-B2,7,79,138
|
||||
Egg-tart-1,Electrode-Box-B2,8,98,142
|
||||
Egg-tart-1,Electrode-Box-B2,9,89,139
|
||||
Egg-tart-1,Electrode-Box-B2,10,96,143
|
||||
Egg-tart-1,Electrode-Box-B2,11,96,148
|
||||
Egg-tart-1,Electrode-Box-B2,12,101,156
|
||||
Egg-tart-1,Electrode-Box-B2,13,103,152
|
||||
Egg-tart-1,Electrode-Box-B2,14,97,152
|
||||
Root-beer-1,Electrode-Box-B2,0,11,85
|
||||
Root-beer-1,Electrode-Box-B2,1,18,76
|
||||
Root-beer-1,Electrode-Box-B2,2,40,105
|
||||
Root-beer-1,Electrode-Box-B2,3,55,134
|
||||
Root-beer-1,Electrode-Box-B2,4,75,136
|
||||
Root-beer-1,Electrode-Box-B2,5,64,133
|
||||
Root-beer-1,Electrode-Box-B2,6,104,139
|
||||
Root-beer-1,Electrode-Box-B2,7,98,148
|
||||
Root-beer-1,Electrode-Box-B2,8,81,145
|
||||
Root-beer-1,Electrode-Box-B2,9,89,156
|
||||
Root-beer-1,Electrode-Box-B2,10,105,158
|
||||
Khoai-lang-2,Naive,0,0,0
|
||||
Khoai-lang-2,Naive,1,0,0
|
||||
Khoai-lang-2,Naive,2,10,47
|
||||
Khoai-lang-2,Naive,3,11,52
|
||||
Khoai-lang-2,Naive,4,9,56
|
||||
Khoai-lang-2,Naive,5,34,95
|
||||
Khoai-lang-2,Naive,6,21,72
|
||||
Khoai-lang-2,Naive,7,23,99
|
||||
Khoai-lang-2,Naive,8,64,136
|
||||
Khoai-lang-2,Naive,9,75,131
|
||||
Khoai-lang-2,Naive,10,63,134
|
||||
Khoai-lang-2,Naive,11,59,139
|
||||
Khoai-lang-2,Naive,12,51,129
|
||||
Khoai-lang-2,Naive,13,73,143
|
||||
Khoai-lang-2,Naive,14,82,136
|
||||
Khoai-lang-2,Naive,15,70,145
|
||||
Khoai-lang-2,Naive,16,76,135
|
||||
Khoai-lang-2,Naive,17,76,150
|
||||
Khoai-lang-2,Naive,18,63,122
|
||||
Khoai-lang-2,Naive,19,48,116
|
||||
Khoai-lang-2,Naive,20,65,134
|
||||
Khoai-lang-2,Naive,21,75,131
|
||||
Khoai-lang-2,Naive,22,98,146
|
||||
Khoai-lang-2,Naive,23,88,139
|
||||
Khoai-lang-2,Naive,24,94,148
|
||||
Khoai-lang-2,Naive,25,56,102
|
||||
Khoai-lang-2,Naive,26,75,143
|
||||
Khoai-tay-2,Naive,1,21,68
|
||||
Khoai-tay-2,Naive,2,6,79
|
||||
Khoai-tay-2,Naive,3,0,83
|
||||
Khoai-tay-2,Naive,4,28,87
|
||||
Khoai-tay-2,Naive,5,31,125
|
||||
Khoai-tay-2,Naive,6,62,144
|
||||
Khoai-tay-2,Naive,7,87,141
|
||||
Khoai-tay-2,Naive,8,105,152
|
||||
Khoai-tay-2,Naive,9,75,148
|
||||
Khoai-tay-2,Naive,10,101,149
|
||||
Khoai-tay-2,Naive,11,102,154
|
||||
Khoai-tay-2,Naive,12,95,144
|
||||
Khoai-tay-2,Naive,13,77,149
|
||||
Khoai-tay-2,Naive,14,91,153
|
||||
OM-2,Naive,-2,0,0
|
||||
OM-2,Naive,-1,5,18
|
||||
OM-2,Naive,0,1,7
|
||||
OM-2,Naive,1,11,33
|
||||
OM-2,Naive,2,19,62
|
||||
OM-2,Naive,3,29,117
|
||||
OM-2,Naive,4,73,126
|
||||
OM-2,Naive,5,53,135
|
||||
OM-2,Naive,6,73,138
|
||||
OM-2,Naive,7,80,131
|
||||
OM-2,Naive,8,91,141
|
||||
OM-2,Naive,9,90,135
|
||||
OM-2,Naive,10,95,142
|
||||
OM-2,Naive,11,60,133
|
||||
OM-2,Naive,12,58,142
|
||||
Khoai-lang-1,Right-Electrode,-4,0,0
|
||||
Khoai-lang-1,Right-Electrode,-3,2,7
|
||||
Khoai-lang-1,Right-Electrode,-1,0,0
|
||||
Khoai-lang-1,Right-Electrode,0,3,69
|
||||
Khoai-lang-1,Right-Electrode,1,18,97
|
||||
Khoai-lang-1,Right-Electrode,2,27,84
|
||||
Khoai-lang-1,Right-Electrode,3,46,121
|
||||
Khoai-lang-1,Right-Electrode,4,65,143
|
||||
Khoai-lang-1,Right-Electrode,5,76,141
|
||||
Khoai-lang-1,Right-Electrode,6,79,146
|
||||
Khoai-lang-1,Right-Electrode,7,81,153
|
||||
Khoai-lang-1,Right-Electrode,8,98,144
|
||||
Khoai-lang-1,Right-Electrode,9,101,149
|
||||
Khoai-lang-1,Right-Electrode,10,103,150
|
||||
Khoai-lang-1,Right-Electrode,11,113,154
|
||||
Khoai-lang-1,Right-Electrode,12,117,148
|
||||
Khoai-lang-1,Right-Electrode,13,114,146
|
||||
Khoai-lang-1,Right-Electrode,14,119,132
|
||||
Khoai-lang-1,Right-Electrode,15,111,156
|
||||
Khoai-lang-1,Right-Electrode,16,74,156
|
||||
Khoai-lang-1,Right-Electrode,17,109,139
|
||||
Khoai-lang-1,Right-Electrode,18,88,138
|
||||
Khoai-lang-1,Right-Electrode,19,98,140
|
||||
Khoai-lang-1,Right-Electrode,20,118,153
|
||||
Khoai-lang-1,Right-Electrode,21,109,146
|
||||
Khoai-lang-1,Right-Electrode,22,106,141
|
||||
|
@@ -0,0 +1,668 @@
|
||||
# Subject-Series Data Export Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the export modal's assignable-axes UI with three pickers — X field (rows), Data value (cells), and Group by — where columns are always the subjects, matching the cross-subject metrics plot.
|
||||
|
||||
**Architecture:** Add X-coordinate helpers to `dataExport.js` (`numericFieldVal`, `listSampleMembers`, `buildSampleIndex`) and a focused `buildSubjectSeriesMatrix` that emits the existing `toCSV` shape. Rewrite `ExportDataModal.jsx` to the three pickers. Then delete the now-dead assignable-axes machinery.
|
||||
|
||||
**Tech Stack:** React 18, plain ES modules, Jest + `@testing-library/react` (jsdom). No new dependencies.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-07-19-subject-series-export-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- `dataExport.js` stays pure: no React, no I/O.
|
||||
- Cell value semantics: `0` and `''` are **real** values; only genuinely-missing lookups render blank (`''`), via the existing `hasValue` + `extractValue`.
|
||||
- CSV: LF (`\n`) line endings; escape every cell via the existing `escapeCsvCell`.
|
||||
- **X = `Date`** → rows are distinct `YYYY-MM-DD` dates, sorted.
|
||||
- **X = `# Days Reach`** (`'__days__'`) → rows are `1, 2, 3, …`; the max over subjects of that subject's count of recorded statuses (a status must have `animal_id` and `date`). A cell for `(subject, N)` reads the subject's Nth recorded day, counting **all** records sorted ascending by date (per-status ordinal, Day 1 = earliest).
|
||||
- **X = daily field** (its `fieldId`) → rows are the distinct **numeric** values (`numericFieldVal`), sorted ascending; non-numeric excluded. A cell for `(subject, v)` reads that subject's first status (by date) whose field equals `v`.
|
||||
- Columns are **always** the subjects (`listSubjectMembers`), clustered by group then name. Entirely-blank sample rows are dropped; subject columns are always kept.
|
||||
- Group by defaults to the saved field (`groupField` prop), validated against `__none__`/`__name__`/`__id__`/active `subjectTemplate` fields; coerced to `__none__` when unset or stale.
|
||||
- CSV: line 1 is `Data,<metric label>`; blank line; a `Group,…` row only when grouping (missing group → `—`); header `= <X label>,<subject headers…>`; data rows `= <X value>,<values…>`. Filename via existing `csvFilename(title, dataLabel)`.
|
||||
- Default X = `Date`; default Data = first parameter (`Total attempts`).
|
||||
- Run tests from `frontend/`: `npx jest tests/dataExport.test.js`, `npx jest tests/ExportDataModal.test.jsx`, full `npm test`.
|
||||
- Two suites (`tests/ExperimentCalendar.test.jsx`, `tests/ExperimentDayView.test.jsx`, 31 tests) fail for a PRE-EXISTING, unrelated reason and must be left as-is; changes must introduce no new failures.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: X-coordinate helpers
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/lib/dataExport.js` (add three exports; keep everything else for now)
|
||||
- Test: `frontend/tests/dataExport.test.js`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: existing private `dateKey(date)`.
|
||||
- Produces:
|
||||
- `numericFieldVal(status, field): number|null` — `field.builtin ? status[field.key] : status.custom_fields[field.fieldId]`, `parseFloat`, non-numeric → null.
|
||||
- `listSampleMembers(statuses, xField, dailyTemplate): Array<{ id, label, sampleValue }>` — row members for the chosen X (`xField` is `'__date__'`, `'__days__'`, or a daily `fieldId`).
|
||||
- `buildSampleIndex(statuses, xField, dailyTemplate): Map<animalId, Map<sampleValue, status>>` — first status per `(subject, sampleValue)` wins, per-subject date-ordered.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Append to `frontend/tests/dataExport.test.js`:
|
||||
|
||||
```js
|
||||
import { numericFieldVal, listSampleMembers, buildSampleIndex } from '../src/lib/dataExport';
|
||||
|
||||
const sDaily = [{ fieldId: 'c-w', key: 'w', label: 'Weight', builtin: false, active: true }];
|
||||
const sStatuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', custom_fields: { 'c-w': '250' } },
|
||||
{ animal_id: 'a1', date: '2026-07-03T00:00:00Z', custom_fields: { 'c-w': '260' } },
|
||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', custom_fields: { 'c-w': '250' } },
|
||||
];
|
||||
|
||||
describe('numericFieldVal', () => {
|
||||
it('reads builtin via key and custom via fieldId, parsed to number', () => {
|
||||
expect(numericFieldVal({ weight: '250' }, { builtin: true, key: 'weight' })).toBe(250);
|
||||
expect(numericFieldVal({ custom_fields: { 'c-w': '12.5' } }, { builtin: false, fieldId: 'c-w' })).toBe(12.5);
|
||||
});
|
||||
it('returns null for non-numeric, missing, or no field', () => {
|
||||
expect(numericFieldVal({ custom_fields: { 'c-w': 'abc' } }, { builtin: false, fieldId: 'c-w' })).toBeNull();
|
||||
expect(numericFieldVal({}, { builtin: true, key: 'weight' })).toBeNull();
|
||||
expect(numericFieldVal({ weight: 1 }, null)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('listSampleMembers', () => {
|
||||
it('date: distinct dates sorted', () => {
|
||||
expect(listSampleMembers(sStatuses, '__date__', sDaily).map((m) => m.sampleValue)).toEqual(['2026-07-01', '2026-07-03']);
|
||||
});
|
||||
it('# days reach: 1..max records per subject', () => {
|
||||
expect(listSampleMembers(sStatuses, '__days__', sDaily).map((m) => m.sampleValue)).toEqual([1, 2]);
|
||||
});
|
||||
it('daily field: distinct numeric values sorted ascending', () => {
|
||||
expect(listSampleMembers(sStatuses, 'c-w', sDaily).map((m) => m.sampleValue)).toEqual([250, 260]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildSampleIndex', () => {
|
||||
it('date coord maps animalId -> dateKey -> status', () => {
|
||||
const idx = buildSampleIndex(sStatuses, '__date__', sDaily);
|
||||
expect(idx.get('a1').get('2026-07-03').date).toContain('2026-07-03');
|
||||
});
|
||||
it('# days reach assigns per-subject ordinals in date order', () => {
|
||||
const idx = buildSampleIndex(sStatuses, '__days__', sDaily);
|
||||
expect(idx.get('a1').get(1).date).toContain('2026-07-01');
|
||||
expect(idx.get('a1').get(2).date).toContain('2026-07-03');
|
||||
expect(idx.get('a2').get(1).date).toContain('2026-07-01');
|
||||
expect(idx.get('a2').has(2)).toBe(false);
|
||||
});
|
||||
it('daily field maps numeric value to first status by date', () => {
|
||||
const idx = buildSampleIndex(sStatuses, 'c-w', sDaily);
|
||||
expect(idx.get('a1').get(250).date).toContain('2026-07-01');
|
||||
expect(idx.get('a1').get(260).date).toContain('2026-07-03');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `npx jest tests/dataExport.test.js -t "listSampleMembers"`
|
||||
Expected: FAIL — `listSampleMembers is not defined`.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Add to `frontend/src/lib/dataExport.js` (after `listExportableParams`, before the private `hasValue`/`dateKey` — note `dateKey` is defined lower in the file but hoisted as a function declaration, so referencing it here is fine):
|
||||
|
||||
```js
|
||||
// Numeric value of a daily field on one status (mirrors the metrics plot's helper).
|
||||
export function numericFieldVal(status, field) {
|
||||
if (!field) return null;
|
||||
const raw = field.builtin ? status?.[field.key] : status?.custom_fields?.[field.fieldId];
|
||||
const n = parseFloat(raw);
|
||||
return Number.isNaN(n) ? null : n;
|
||||
}
|
||||
|
||||
// Ordered row members for the chosen X coordinate.
|
||||
// xField: '__date__' | '__days__' | <daily fieldId>
|
||||
export function listSampleMembers(statuses, xField, dailyTemplate) {
|
||||
const list = statuses ?? [];
|
||||
if (xField === '__date__') {
|
||||
const keys = new Set();
|
||||
for (const s of list) if (s?.date) keys.add(dateKey(s.date));
|
||||
return [...keys].sort().map((k) => ({ id: k, label: k, sampleValue: k }));
|
||||
}
|
||||
if (xField === '__days__') {
|
||||
const counts = new Map();
|
||||
for (const s of list) {
|
||||
if (s?.animal_id == null || !s?.date) continue;
|
||||
counts.set(s.animal_id, (counts.get(s.animal_id) ?? 0) + 1);
|
||||
}
|
||||
const max = counts.size ? Math.max(...counts.values()) : 0;
|
||||
return Array.from({ length: max }, (_, i) => ({ id: String(i + 1), label: String(i + 1), sampleValue: i + 1 }));
|
||||
}
|
||||
const field = (dailyTemplate ?? []).find((f) => f.fieldId === xField);
|
||||
const vals = new Set();
|
||||
for (const s of list) {
|
||||
const v = numericFieldVal(s, field);
|
||||
if (v !== null) vals.add(v);
|
||||
}
|
||||
return [...vals].sort((a, b) => a - b).map((v) => ({ id: String(v), label: String(v), sampleValue: v }));
|
||||
}
|
||||
|
||||
// animalId -> Map(sampleValue -> status). First status per (subject, sampleValue) wins,
|
||||
// scanning each subject's statuses in ascending date order.
|
||||
export function buildSampleIndex(statuses, xField, dailyTemplate) {
|
||||
const bySubject = new Map();
|
||||
for (const s of statuses ?? []) {
|
||||
if (s?.animal_id == null || !s?.date) continue;
|
||||
if (!bySubject.has(s.animal_id)) bySubject.set(s.animal_id, []);
|
||||
bySubject.get(s.animal_id).push(s);
|
||||
}
|
||||
const field = xField !== '__date__' && xField !== '__days__'
|
||||
? (dailyTemplate ?? []).find((f) => f.fieldId === xField)
|
||||
: null;
|
||||
|
||||
const index = new Map();
|
||||
for (const [animalId, subjStatuses] of bySubject) {
|
||||
const ordered = [...subjStatuses].sort((a, b) => dateKey(a.date).localeCompare(dateKey(b.date)));
|
||||
const m = new Map();
|
||||
ordered.forEach((s, i) => {
|
||||
let key;
|
||||
if (xField === '__date__') key = dateKey(s.date);
|
||||
else if (xField === '__days__') key = i + 1;
|
||||
else { const v = numericFieldVal(s, field); if (v === null) return; key = v; }
|
||||
if (!m.has(key)) m.set(key, s);
|
||||
});
|
||||
index.set(animalId, m);
|
||||
}
|
||||
return index;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `npx jest tests/dataExport.test.js`
|
||||
Expected: PASS (new suites green; all pre-existing suites still green).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add frontend/src/lib/dataExport.js frontend/tests/dataExport.test.js
|
||||
git commit -m "feat(export): X-coordinate helpers (date, # days reach, daily field)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: buildSubjectSeriesMatrix
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/lib/dataExport.js` (add `buildSubjectSeriesMatrix` + private `xFieldLabel`)
|
||||
- Test: `frontend/tests/dataExport.test.js`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `listSubjectMembers`, `listSampleMembers`, `buildSampleIndex`, `extractValue`, private `hasValue`.
|
||||
- Produces: `buildSubjectSeriesMatrix({ xField, dataParam, groupField }, { statuses, animals, dailyTemplate })` → `{ corner, context: { label: 'Data', value }, groupAxis: 'col'|null, columns: [{ id, label, animalId, group }], rows: [{ member: { id, label, sampleValue }, values: [value|''] }] }` — the same shape the existing `toCSV` consumes (`groupAxis: 'col'` triggers its group-row branch).
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Append to `frontend/tests/dataExport.test.js`:
|
||||
|
||||
```js
|
||||
import { buildSubjectSeriesMatrix } from '../src/lib/dataExport';
|
||||
|
||||
const ssAnimals = [
|
||||
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
||||
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||
];
|
||||
const ssStatuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5 } },
|
||||
{ animal_id: 'a1', date: '2026-07-02T00:00:00Z', analysis_summary: { total: 7 } },
|
||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 9 } },
|
||||
];
|
||||
const totalParam = { id: '__total__', label: 'Total attempts', kind: 'total', group: 'metrics' };
|
||||
|
||||
describe('buildSubjectSeriesMatrix', () => {
|
||||
it('date X: subjects as columns, date rows, blank where missing, context names the metric', () => {
|
||||
const m = buildSubjectSeriesMatrix(
|
||||
{ xField: '__date__', dataParam: totalParam, groupField: '__none__' },
|
||||
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||
);
|
||||
expect(m.corner).toBe('Date');
|
||||
expect(m.context).toEqual({ label: 'Data', value: 'Total attempts' });
|
||||
expect(m.groupAxis).toBeNull();
|
||||
expect(m.columns.map((c) => c.label)).toEqual(['Alpha', 'Beta']);
|
||||
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01', '2026-07-02']);
|
||||
expect(m.rows[0].values).toEqual([5, 9]);
|
||||
expect(m.rows[1].values).toEqual([7, '']);
|
||||
});
|
||||
it('# days reach X: rows align each subject by day ordinal', () => {
|
||||
const m = buildSubjectSeriesMatrix(
|
||||
{ xField: '__days__', dataParam: totalParam, groupField: '__none__' },
|
||||
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||
);
|
||||
expect(m.corner).toBe('# Days Reach');
|
||||
expect(m.rows.map((r) => r.member.label)).toEqual(['1', '2']);
|
||||
expect(m.rows[0].values).toEqual([5, 9]);
|
||||
expect(m.rows[1].values).toEqual([7, '']);
|
||||
});
|
||||
it('grouping sets groupAxis col and clusters subjects', () => {
|
||||
const m = buildSubjectSeriesMatrix(
|
||||
{ xField: '__date__', dataParam: totalParam, groupField: 'grp' },
|
||||
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||
);
|
||||
expect(m.groupAxis).toBe('col');
|
||||
expect(m.columns.map((c) => [c.group, c.label])).toEqual([['Control', 'Alpha'], ['Drug', 'Beta']]);
|
||||
});
|
||||
it('drops entirely-blank rows', () => {
|
||||
const statuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5 } },
|
||||
{ animal_id: 'a1', date: '2026-07-05T00:00:00Z', analysis_summary: null },
|
||||
];
|
||||
const m = buildSubjectSeriesMatrix(
|
||||
{ xField: '__date__', dataParam: totalParam, groupField: '__none__' },
|
||||
{ statuses, animals: ssAnimals, dailyTemplate: [] },
|
||||
);
|
||||
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01']);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `npx jest tests/dataExport.test.js -t "buildSubjectSeriesMatrix"`
|
||||
Expected: FAIL — `buildSubjectSeriesMatrix is not defined`.
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
Add to `frontend/src/lib/dataExport.js` (after `buildSampleIndex`):
|
||||
|
||||
```js
|
||||
function xFieldLabel(xField, dailyTemplate) {
|
||||
if (xField === '__date__') return 'Date';
|
||||
if (xField === '__days__') return '# Days Reach';
|
||||
return (dailyTemplate ?? []).find((f) => f.fieldId === xField)?.label ?? '';
|
||||
}
|
||||
|
||||
// Subjects as columns, X-field values as rows, one Data value per cell.
|
||||
// Returns the { context, corner, groupAxis, columns, rows } shape toCSV consumes.
|
||||
export function buildSubjectSeriesMatrix({ xField, dataParam, groupField }, { statuses, animals, dailyTemplate }) {
|
||||
const columns = listSubjectMembers(animals, groupField);
|
||||
const rowMembers = listSampleMembers(statuses, xField, dailyTemplate);
|
||||
const index = buildSampleIndex(statuses, xField, dailyTemplate);
|
||||
|
||||
const cellValue = (sampleValue, animalId) => {
|
||||
const status = index.get(animalId)?.get(sampleValue);
|
||||
if (!status || !dataParam) return '';
|
||||
const v = extractValue(status, dataParam);
|
||||
return hasValue(v) ? v : '';
|
||||
};
|
||||
|
||||
let rows = rowMembers.map((rm) => ({
|
||||
member: rm,
|
||||
values: columns.map((c) => cellValue(rm.sampleValue, c.animalId)),
|
||||
}));
|
||||
rows = rows.filter((r) => r.values.some((v) => v !== '')); // drop entirely-blank sample rows
|
||||
|
||||
const grouped = !!groupField && groupField !== '__none__';
|
||||
return {
|
||||
corner: xFieldLabel(xField, dailyTemplate),
|
||||
context: { label: 'Data', value: dataParam?.label ?? '' },
|
||||
groupAxis: grouped ? 'col' : null,
|
||||
columns,
|
||||
rows,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `npx jest tests/dataExport.test.js`
|
||||
Expected: PASS (all suites green).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add frontend/src/lib/dataExport.js frontend/tests/dataExport.test.js
|
||||
git commit -m "feat(export): buildSubjectSeriesMatrix — subjects as columns over an X field"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Rewrite ExportDataModal to the three pickers
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/components/ExportDataModal.jsx` (full body rewrite; keep `triggerDownload`)
|
||||
- Test: `frontend/tests/ExportDataModal.test.jsx` (rewrite)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `listExportableParams`, `buildSubjectSeriesMatrix`, `toCSV`, `csvFilename` from `../lib/dataExport`; `experimentsApi.getDailyStatuses`.
|
||||
- Props unchanged: `experimentTitle`, `experimentId`, `dailyTemplate`, `animals`, `subjectTemplate` (default `[]`), `groupField` (default `'__none__'`), `onClose`.
|
||||
- Produces: modal with three selects — X (rows), Data (cells), Group by.
|
||||
|
||||
- [ ] **Step 1: Rewrite the component test**
|
||||
|
||||
Replace the entire contents of `frontend/tests/ExportDataModal.test.jsx` with:
|
||||
|
||||
```jsx
|
||||
import React from 'react';
|
||||
import { render, screen, waitFor, fireEvent } from '@testing-library/react';
|
||||
import ExportDataModal from '../src/components/ExportDataModal';
|
||||
import * as client from '../src/api/client';
|
||||
|
||||
jest.mock('../src/api/client', () => ({
|
||||
experimentsApi: { getDailyStatuses: jest.fn() },
|
||||
}));
|
||||
|
||||
const animals = [
|
||||
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
||||
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||
];
|
||||
const statuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5, success_rate: 0.5 } },
|
||||
{ animal_id: 'a1', date: '2026-07-02T00:00:00Z', analysis_summary: { total: 7, success_rate: 0.7 } },
|
||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 9, success_rate: 0.9 } },
|
||||
];
|
||||
const subjectTemplate = [{ fieldId: 'grp', label: 'Group', active: true }];
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
client.experimentsApi.getDailyStatuses.mockResolvedValue(statuses);
|
||||
});
|
||||
|
||||
// Intercept the Blob handed to URL.createObjectURL so tests can read the real CSV.
|
||||
// Also stub anchor.click() — jsdom treats a click on <a href="blob:..."> as an
|
||||
// unimplemented navigation that throws async and can flake a later test.
|
||||
function mockDownload() {
|
||||
let blob = null;
|
||||
const origCreate = global.URL.createObjectURL;
|
||||
const origRevoke = global.URL.revokeObjectURL;
|
||||
const origClick = window.HTMLAnchorElement.prototype.click;
|
||||
global.URL.createObjectURL = jest.fn((b) => { blob = b; return 'blob:mock'; });
|
||||
global.URL.revokeObjectURL = jest.fn();
|
||||
window.HTMLAnchorElement.prototype.click = jest.fn();
|
||||
return {
|
||||
getBlob: () => blob,
|
||||
restore: async () => {
|
||||
await new Promise((r) => setTimeout(r, 0)); // flush handleExport's setTimeout revoke
|
||||
global.URL.createObjectURL = origCreate;
|
||||
global.URL.revokeObjectURL = origRevoke;
|
||||
window.HTMLAnchorElement.prototype.click = origClick;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function readBlobText(blob) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const fr = new FileReader();
|
||||
fr.onload = () => resolve(fr.result);
|
||||
fr.onerror = reject;
|
||||
fr.readAsText(blob);
|
||||
});
|
||||
}
|
||||
|
||||
function renderModal(props = {}) {
|
||||
return render(
|
||||
<ExportDataModal
|
||||
experimentId="exp-1"
|
||||
experimentTitle="My Study"
|
||||
dailyTemplate={[]}
|
||||
animals={animals}
|
||||
subjectTemplate={subjectTemplate}
|
||||
groupField="__none__"
|
||||
onClose={jest.fn()}
|
||||
{...props}
|
||||
/>,
|
||||
);
|
||||
}
|
||||
|
||||
describe('ExportDataModal', () => {
|
||||
it('renders X / Data / Group by pickers; X defaults to Date', async () => {
|
||||
renderModal();
|
||||
expect(await screen.findByLabelText('X axis (rows)')).toHaveValue('__date__');
|
||||
expect(screen.getByLabelText('Data (cells)')).toBeInTheDocument();
|
||||
expect(screen.getByLabelText('Group by')).toHaveValue('__none__');
|
||||
await waitFor(() => expect(screen.getByText(/2 rows × 2 subjects/i)).toBeInTheDocument());
|
||||
});
|
||||
|
||||
it('default export (X=Date, Data=Total attempts) has subject columns, date rows, blank where missing', async () => {
|
||||
const dl = mockDownload();
|
||||
renderModal();
|
||||
await screen.findByLabelText('X axis (rows)');
|
||||
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||
fireEvent.click(screen.getByText('Export CSV'));
|
||||
const text = await readBlobText(dl.getBlob());
|
||||
expect(text).toBe('Data,Total attempts\n\nDate,Alpha,Beta\n2026-07-01,5,9\n2026-07-02,7,');
|
||||
await dl.restore();
|
||||
});
|
||||
|
||||
it('switching X to # Days Reach makes rows the day ordinals', async () => {
|
||||
const dl = mockDownload();
|
||||
renderModal();
|
||||
fireEvent.change(await screen.findByLabelText('X axis (rows)'), { target: { value: '__days__' } });
|
||||
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||
fireEvent.click(screen.getByText('Export CSV'));
|
||||
const text = await readBlobText(dl.getBlob());
|
||||
expect(text).toBe('Data,Total attempts\n\n# Days Reach,Alpha,Beta\n1,5,9\n2,7,');
|
||||
await dl.restore();
|
||||
});
|
||||
|
||||
it('a valid group field adds a Group row and clusters subjects', async () => {
|
||||
const dl = mockDownload();
|
||||
renderModal({ groupField: 'grp' });
|
||||
await screen.findByLabelText('X axis (rows)');
|
||||
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||
fireEvent.click(screen.getByText('Export CSV'));
|
||||
const text = await readBlobText(dl.getBlob());
|
||||
expect(text).toMatch(/^Group,Control,Drug$/m);
|
||||
await dl.restore();
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test to verify it fails**
|
||||
|
||||
Run: `npx jest tests/ExportDataModal.test.jsx`
|
||||
Expected: FAIL — old modal has no `X axis (rows)` / `Data (cells)` labels.
|
||||
|
||||
- [ ] **Step 3: Rewrite the component**
|
||||
|
||||
Replace the entire contents of `frontend/src/components/ExportDataModal.jsx` with:
|
||||
|
||||
```jsx
|
||||
import React, { useEffect, useMemo, useState } from 'react';
|
||||
import { experimentsApi } from '../api/client';
|
||||
import Button from './ui/Button';
|
||||
import Alert from './ui/Alert';
|
||||
import { listExportableParams, buildSubjectSeriesMatrix, toCSV, csvFilename } from '../lib/dataExport';
|
||||
|
||||
const PARAM_GROUP_LABELS = { metrics: 'Session metrics', daily: 'Daily record fields' };
|
||||
|
||||
function triggerDownload(filename, text) {
|
||||
const blob = new Blob([text], { type: 'text/csv;charset=utf-8;' });
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = filename;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
setTimeout(() => URL.revokeObjectURL(url), 0);
|
||||
}
|
||||
|
||||
export default function ExportDataModal({
|
||||
experimentTitle, experimentId, dailyTemplate, animals, subjectTemplate = [], groupField = '__none__', onClose,
|
||||
}) {
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [error, setError] = useState(null);
|
||||
const [statuses, setStatuses] = useState([]);
|
||||
|
||||
const [xField, setXField] = useState('__date__');
|
||||
const [dataId, setDataId] = useState(null);
|
||||
const [groupBy, setGroupBy] = useState(() => {
|
||||
const valid = groupField === '__none__' || groupField === '__name__' || groupField === '__id__'
|
||||
|| subjectTemplate.some((f) => f.active && f.fieldId === groupField);
|
||||
return valid ? groupField : '__none__';
|
||||
});
|
||||
|
||||
useEffect(() => {
|
||||
let alive = true;
|
||||
setLoading(true);
|
||||
setError(null);
|
||||
experimentsApi.getDailyStatuses(experimentId, {})
|
||||
.then((data) => { if (alive) setStatuses(data); })
|
||||
.catch((err) => { if (alive) setError(err.message); })
|
||||
.finally(() => { if (alive) setLoading(false); });
|
||||
return () => { alive = false; };
|
||||
}, [experimentId]);
|
||||
|
||||
const params = useMemo(() => listExportableParams(dailyTemplate, statuses), [dailyTemplate, statuses]);
|
||||
const effDataId = dataId ?? params[0]?.id ?? null;
|
||||
const dataParam = params.find((p) => p.id === effDataId) ?? params[0] ?? null;
|
||||
|
||||
const activeDaily = useMemo(() => (dailyTemplate ?? []).filter((f) => f.active), [dailyTemplate]);
|
||||
const xOptions = useMemo(() => [
|
||||
{ id: '__date__', label: 'Date' },
|
||||
{ id: '__days__', label: '# Days Reach' },
|
||||
...activeDaily.map((f) => ({ id: f.fieldId, label: f.label })),
|
||||
], [activeDaily]);
|
||||
|
||||
const matrix = useMemo(
|
||||
() => (dataParam
|
||||
? buildSubjectSeriesMatrix({ xField, dataParam, groupField: groupBy }, { statuses, animals, dailyTemplate })
|
||||
: { columns: [], rows: [], context: { label: 'Data', value: '' }, corner: '', groupAxis: null }),
|
||||
[xField, dataParam, groupBy, statuses, animals, dailyTemplate],
|
||||
);
|
||||
const hasData = matrix.rows.length > 0;
|
||||
|
||||
const groupedParams = useMemo(() => {
|
||||
const out = [];
|
||||
for (const p of params) {
|
||||
let g = out.find((x) => x.group === p.group);
|
||||
if (!g) { g = { group: p.group, items: [] }; out.push(g); }
|
||||
g.items.push(p);
|
||||
}
|
||||
return out;
|
||||
}, [params]);
|
||||
|
||||
function handleExport() {
|
||||
if (!hasData || !dataParam) return;
|
||||
triggerDownload(csvFilename(experimentTitle, dataParam.label), toCSV(matrix));
|
||||
onClose();
|
||||
}
|
||||
|
||||
if (loading) return <p className="text-sm text-gray-400" aria-live="polite">Loading data…</p>;
|
||||
if (error) return (
|
||||
<div className="space-y-4">
|
||||
<Alert type="error" message={error} />
|
||||
<div className="flex justify-end"><Button variant="secondary" onClick={onClose}>Close</Button></div>
|
||||
</div>
|
||||
);
|
||||
|
||||
const selectCls = 'w-full border border-gray-200 rounded px-2 py-1.5 text-sm bg-white focus:outline-none focus:ring-1 focus:ring-indigo-400';
|
||||
|
||||
return (
|
||||
<div className="space-y-4">
|
||||
<p className="text-sm text-gray-500">
|
||||
One value per subject over an X axis — each column is a subject, each row an X value. Blank where a subject has no value.
|
||||
</p>
|
||||
|
||||
<div>
|
||||
<label htmlFor="export-x" className="block text-xs font-medium text-gray-500 mb-1">X axis (rows)</label>
|
||||
<select id="export-x" value={xField} onChange={(e) => setXField(e.target.value)} className={selectCls}>
|
||||
{xOptions.map((o) => <option key={o.id} value={o.id}>{o.label}</option>)}
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label htmlFor="export-data" className="block text-xs font-medium text-gray-500 mb-1">Data (cells)</label>
|
||||
<select id="export-data" value={effDataId ?? ''} onChange={(e) => setDataId(e.target.value)} className={selectCls}>
|
||||
{groupedParams.map((g) => (
|
||||
<optgroup key={g.group} label={PARAM_GROUP_LABELS[g.group] ?? g.group}>
|
||||
{g.items.map((p) => <option key={p.id} value={p.id}>{p.label}</option>)}
|
||||
</optgroup>
|
||||
))}
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label htmlFor="export-group" className="block text-xs font-medium text-gray-500 mb-1">Group by</label>
|
||||
<select id="export-group" value={groupBy} onChange={(e) => setGroupBy(e.target.value)} className={selectCls}>
|
||||
<option value="__none__">— None —</option>
|
||||
<option value="__name__">Name</option>
|
||||
<option value="__id__">Subject ID</option>
|
||||
{subjectTemplate.filter((f) => f.active).map((f) => (
|
||||
<option key={f.fieldId} value={f.fieldId}>{f.label}</option>
|
||||
))}
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<p className="text-xs text-gray-400">
|
||||
{hasData
|
||||
? `${matrix.rows.length} row${matrix.rows.length !== 1 ? 's' : ''} × ${matrix.columns.length} subject${matrix.columns.length !== 1 ? 's' : ''}`
|
||||
: 'No data for this selection yet.'}
|
||||
</p>
|
||||
|
||||
<div className="flex justify-end gap-2 pt-2">
|
||||
<Button variant="secondary" onClick={onClose}>Cancel</Button>
|
||||
<Button onClick={handleExport} disabled={!hasData}>Export CSV</Button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the test to verify it passes**
|
||||
|
||||
Run: `npx jest tests/ExportDataModal.test.jsx`
|
||||
Expected: PASS (all four cases).
|
||||
|
||||
- [ ] **Step 5: Run the full suite**
|
||||
|
||||
Run: `npm test`
|
||||
Expected: `dataExport` + `ExportDataModal` green; only the two known pre-existing suites fail; no new failures.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add frontend/src/components/ExportDataModal.jsx frontend/tests/ExportDataModal.test.jsx
|
||||
git commit -m "feat(export): modal becomes X / Data / Group by pickers, subjects as columns"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Remove the dead assignable-axes code
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/lib/dataExport.js` (delete unused exports)
|
||||
- Test: `frontend/tests/dataExport.test.js` (delete their describe blocks + imports)
|
||||
|
||||
**Interfaces:**
|
||||
- Removes (now unused after Task 3): `EXPORT_DIMENSIONS`, `dimLabel`, `otherDimension`, `listDateMembers`, `buildStatusIndex`, `listParameterMembers`, `listDimensionMembers`, and `buildMatrix`.
|
||||
- Keeps: `extractValue`, `listExportableParams`, `subjectGroupValue`, `listSubjectMembers`, `numericFieldVal`, `listSampleMembers`, `buildSampleIndex`, `buildSubjectSeriesMatrix`, `toCSV`, `csvFilename`, and the private `hasValue`, `dateKey`, `escapeCsvCell`, `xFieldLabel`.
|
||||
|
||||
- [ ] **Step 1: Delete the dead source exports**
|
||||
|
||||
In `frontend/src/lib/dataExport.js`, delete these eight exported definitions entirely (they are only used by each other and the old modal, now replaced): `EXPORT_DIMENSIONS`, `dimLabel`, `otherDimension`, `listDateMembers`, `buildStatusIndex`, `listParameterMembers`, `listDimensionMembers`, `buildMatrix`. Also update the top-of-file comment to describe the subject-series export (drop the "dimension-agnostic buildMatrix" wording). Keep `dateKey`, `hasValue`, `escapeCsvCell`, and everything listed under "Keeps" above.
|
||||
|
||||
- [ ] **Step 2: Delete their tests**
|
||||
|
||||
In `frontend/tests/dataExport.test.js`, delete the describe blocks and their now-unused import identifiers for the removed functions: the `describe('dimension primitives', …)`, `describe('subjectGroupValue', …)` KEEP (subjectGroupValue is kept — do NOT delete it), `describe('listDateMembers', …)`, `describe('buildStatusIndex', …)`, `describe('listParameterMembers / listDimensionMembers', …)`, and `describe('buildMatrix', …)` blocks. Remove `EXPORT_DIMENSIONS`, `dimLabel`, `otherDimension`, `listDateMembers`, `buildStatusIndex`, `listDimensionMembers`, `listParameterMembers`, and `buildMatrix` from the file's `import` statements, leaving the imports for kept functions (`extractValue`, `listExportableParams`, `subjectGroupValue`, `listSubjectMembers`, `numericFieldVal`, `listSampleMembers`, `buildSampleIndex`, `buildSubjectSeriesMatrix`, `toCSV`, `csvFilename`).
|
||||
|
||||
- [ ] **Step 3: Verify nothing references the removed names**
|
||||
|
||||
Run: `grep -rnE "buildMatrix|EXPORT_DIMENSIONS|listDimensionMembers|buildStatusIndex|listDateMembers|listParameterMembers|otherDimension|dimLabel" frontend/src frontend/tests`
|
||||
Expected: no output (all references gone).
|
||||
|
||||
- [ ] **Step 4: Run the suites**
|
||||
|
||||
Run: `npx jest tests/dataExport.test.js tests/ExportDataModal.test.jsx`
|
||||
Expected: PASS. Then `npm test` — only the two known pre-existing suites fail, no new failures.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add frontend/src/lib/dataExport.js frontend/tests/dataExport.test.js
|
||||
git commit -m "refactor(export): remove dead assignable-axes helpers and their tests"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
- **Spec coverage:** three pickers (Task 3) · X = Date/# Days Reach/daily field with the exact semantics (Tasks 1–2) · subjects always columns, clustered by group, blank rows dropped (Task 2) · Group by validated/defaulted (Task 3) · CSV `Data,<metric>` + group row + X-corner (Tasks 2–3, reusing `toCSV`) · assignable-axes UI/machinery removed (Tasks 3–4). All covered.
|
||||
- **Type consistency:** sample member `{ id, label, sampleValue }`, subject column `{ id, label, animalId, group }`, and matrix `{ context, corner, groupAxis, columns, rows }` are used identically across Tasks 1–3 and match the existing `toCSV`.
|
||||
- **Out of scope (per spec):** assignable columns/pinned, subject subsetting, analyzed-only day counting, footer/repeat line.
|
||||
@@ -0,0 +1,86 @@
|
||||
# Subject-Series Data Export — Design
|
||||
|
||||
**Date:** 2026-07-19
|
||||
**Status:** Approved (design)
|
||||
**Area:** `frontend/src/lib/dataExport.js`, `frontend/src/components/ExportDataModal.jsx`, `frontend/tests/`
|
||||
**Supersedes:** the assignable-axes UI from `2026-07-19-configurable-data-export-design.md` (that branch's `buildMatrix`/`toCSV`/subject-group work is reused; its rows/columns/pinned modal is replaced).
|
||||
|
||||
## Problem
|
||||
|
||||
The export modal currently offers assignable row/column/pinned axes. That is more than needed. The user wants the export to mirror the **cross-subject metrics plot**: pick an **X field** and a **data value**, with each **subject as a column series**, empty where a subject has no data point, plus each subject's **group** included. The one capability missing today is choosing the X (row) field — it is fixed to calendar Date, whereas the plot's X-axis offers **# Days Reach** and daily fields.
|
||||
|
||||
## Layout (fixed)
|
||||
|
||||
- **Columns = subjects**, one series per subject, ordered clustered-by-group then by name (existing `listSubjectMembers`).
|
||||
- **Rows = the chosen X field's values.**
|
||||
- **Cells = the chosen Data value** for that (subject, X); empty (`''`) when absent. `0` and `''` remain real values (existing `hasValue`/`extractValue`).
|
||||
- **Group row** gives each subject's group value (existing group-row support).
|
||||
|
||||
There is no pinned dimension and no assignable columns.
|
||||
|
||||
## The three pickers
|
||||
|
||||
1. **X (rows).** Options: `Date`, `# Days Reach`, and each **active** daily-template field. Default `Date` (keeps the same date-rows × subject-columns grid as today; only the line-1 label changes to `Data`, see CSV output).
|
||||
2. **Data (cells).** Options: `listExportableParams(dailyTemplate, statuses)` — total, success rate, each count category, each daily field. Default: first parameter (`Total attempts`).
|
||||
3. **Group by.** Options: `— None —`, `Name`, `Subject ID`, and each active `subject_info` field. Defaults to the experiment's saved group field (`localStorage['exp-subject-group-<id>']`), validated against the available options and coerced to `__none__` when unset or stale. Always visible (subjects are always the columns).
|
||||
|
||||
Preview line: `N rows × M subjects`. Export disabled when there are no data rows.
|
||||
|
||||
## X-field semantics
|
||||
|
||||
Each X coordinate maps a `(subject, X-value)` pair to a single daily status, from which the Data value is read:
|
||||
|
||||
- **`Date`** → rows are the distinct calendar dates present (`YYYY-MM-DD`), sorted. Cell reads the subject's status on that date. (Today's behavior.)
|
||||
- **`# Days Reach`** → rows are `1, 2, 3, …` up to the maximum, per subject, of the count of that subject's recorded daily statuses. A cell for `(subject, N)` reads the subject's **Nth recorded day**, counting **all** recorded statuses sorted ascending by date (Day 1 = earliest record; days without an analysis summary are still counted — this intentionally differs from the plot, which counts analyzed sessions only).
|
||||
- **daily field** (e.g. `Weight (g)`) → rows are the distinct **numeric** values that field takes across the data (`numericFieldVal`: builtin via `status[field.key]`, custom via `status.custom_fields[field.fieldId]`, `parseFloat`, non-numeric → excluded), sorted ascending. A cell for `(subject, v)` reads that subject's status where the field equals `v`; on ties, first status by date wins.
|
||||
|
||||
Row members that are entirely blank across all subjects are dropped (matching Date's existing behavior); subject columns are always kept even when empty.
|
||||
|
||||
## CSV output
|
||||
|
||||
Structure (X = # Days Reach, Data = Success rate, Group by = Treatment):
|
||||
|
||||
```
|
||||
Data,Success rate
|
||||
Group,Control,Control,Drug
|
||||
# Days Reach,Mouse-A,Mouse-B,Mouse-C
|
||||
1,0.5,0.6,0.7
|
||||
2,0.55,,0.72
|
||||
3,0.6,0.65,
|
||||
```
|
||||
|
||||
- **Line 1** names the Data metric so the file is self-describing: two cells, `Data` and the metric label.
|
||||
- **Blank line.**
|
||||
- **Group row** (`Group` + each subject's group), only when Group by ≠ None; missing group value → `—`.
|
||||
- **Header row:** the X-field label in the corner, then subject headers.
|
||||
- **Data rows:** the X value, then one cell per subject.
|
||||
|
||||
LF line endings; every cell escaped via existing `escapeCsvCell`. Filename: existing `csvFilename(title, dataLabel)`.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
Reuse from the current branch: `extractValue`, `listExportableParams`, `subjectGroupValue`, `listSubjectMembers`, `escapeCsvCell`, `toCSV` (its `{ context, corner, groupAxis, columns, rows }` shape is kept), and `csvFilename`.
|
||||
|
||||
Replace: the general assignable-axes `buildMatrix(config, ctx)` and its members/dispatch helpers (`otherDimension`, `dimLabel`, `EXPORT_DIMENSIONS`, `listDateMembers`, `listDimensionMembers`, `listParameterMembers`, `buildStatusIndex` in its current form) — remove what the fixed layout no longer uses rather than leaving dead code.
|
||||
|
||||
Add:
|
||||
- `numericFieldVal(status, field)` — mirror of the plot's helper (or import/share it).
|
||||
- `listSampleMembers(statuses, xField, dailyTemplate)` → the ordered row members for the chosen X coordinate (`{ id, label, ... }`).
|
||||
- `buildSampleIndex(statuses, xField, dailyTemplate)` → `Map(animalId → Map(sampleValue → status))`, first-status-per-cell wins, encoding the date/#days/field resolution above.
|
||||
- `buildSubjectSeriesMatrix({ xField, dataParam, groupField }, { statuses, animals, dailyTemplate })` → `{ context: { label: 'Data', value: dataParam.label }, corner: <X label>, groupAxis: grouping ? 'col' : null, columns: <subjects>, rows: [{ member, values }] }`, consumable by the existing `toCSV`.
|
||||
|
||||
Simplify `ExportDataModal.jsx` to the three pickers over these helpers; drop the Rows/Columns/Fixed controls.
|
||||
|
||||
## Testing
|
||||
|
||||
- `listSampleMembers` / `buildSampleIndex` for each X coordinate: Date (unchanged), # Days Reach (all-records ordinal, per-subject alignment), daily field (distinct numeric values sorted, non-numeric dropped, tie first-by-date).
|
||||
- `buildSubjectSeriesMatrix`: subjects as columns clustered by group; empty cells where a subject lacks a point; blank row dropped; `groupAxis` null when Group by None.
|
||||
- `toCSV` reused; add/keep a snapshot for the new context line + group row.
|
||||
- `ExportDataModal`: renders three pickers; default X = Date produces date rows × subject columns with the chosen metric in cells; switching X to # Days Reach changes rows to ordinals; group row present when a valid group field is chosen; stale/unset group field coerces to None.
|
||||
|
||||
## Out of scope (YAGNI)
|
||||
|
||||
- Assignable columns / pinned dimension (removed).
|
||||
- Choosing a subset of subjects (all subjects always included).
|
||||
- Matching the plot's analyzed-only day counting (we count all recorded days).
|
||||
- A footer or repeated context line.
|
||||
@@ -2,9 +2,9 @@ import React, { useEffect, useMemo, useState } from 'react';
|
||||
import { experimentsApi } from '../api/client';
|
||||
import Button from './ui/Button';
|
||||
import Alert from './ui/Alert';
|
||||
import { listExportableParams, buildMatrix, toCSV, csvFilename } from '../lib/dataExport';
|
||||
import { listExportableParams, buildSubjectSeriesMatrix, toCSV, csvFilename } from '../lib/dataExport';
|
||||
|
||||
const GROUP_LABELS = { metrics: 'Session metrics', daily: 'Daily record fields' };
|
||||
const PARAM_GROUP_LABELS = { metrics: 'Session metrics', daily: 'Daily record fields' };
|
||||
|
||||
function triggerDownload(filename, text) {
|
||||
const blob = new Blob([text], { type: 'text/csv;charset=utf-8;' });
|
||||
@@ -18,11 +18,20 @@ function triggerDownload(filename, text) {
|
||||
setTimeout(() => URL.revokeObjectURL(url), 0);
|
||||
}
|
||||
|
||||
export default function ExportDataModal({ experimentTitle, experimentId, dailyTemplate, animals, onClose }) {
|
||||
export default function ExportDataModal({
|
||||
experimentTitle, experimentId, dailyTemplate, animals, subjectTemplate = [], groupField = '__none__', onClose,
|
||||
}) {
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [error, setError] = useState(null);
|
||||
const [statuses, setStatuses] = useState([]);
|
||||
const [selectedId, setSelectedId] = useState(null);
|
||||
|
||||
const [xField, setXField] = useState('__date__');
|
||||
const [dataId, setDataId] = useState(null);
|
||||
const [groupBy, setGroupBy] = useState(() => {
|
||||
const valid = groupField === '__none__' || groupField === '__name__' || groupField === '__id__'
|
||||
|| subjectTemplate.some((f) => f.active && f.fieldId === groupField);
|
||||
return valid ? groupField : '__none__';
|
||||
});
|
||||
|
||||
useEffect(() => {
|
||||
let alive = true;
|
||||
@@ -35,21 +44,26 @@ export default function ExportDataModal({ experimentTitle, experimentId, dailyTe
|
||||
return () => { alive = false; };
|
||||
}, [experimentId]);
|
||||
|
||||
const params = useMemo(() => listExportableParams(dailyTemplate, statuses), [dailyTemplate, statuses]);
|
||||
const params = useMemo(() => listExportableParams(dailyTemplate ?? [], statuses), [dailyTemplate, statuses]);
|
||||
const effDataId = dataId ?? params[0]?.id ?? null;
|
||||
const dataParam = params.find((p) => p.id === effDataId) ?? params[0] ?? null;
|
||||
|
||||
// Derive the effective selection: the user's pick, else the first parameter.
|
||||
const effectiveId = selectedId ?? params[0]?.id ?? null;
|
||||
const selectedParam = params.find((p) => p.id === effectiveId) ?? null;
|
||||
const activeDaily = useMemo(() => (dailyTemplate ?? []).filter((f) => f.active), [dailyTemplate]);
|
||||
const xOptions = useMemo(() => [
|
||||
{ id: '__date__', label: 'Date' },
|
||||
{ id: '__days__', label: '# Days Reach (ordinal)' },
|
||||
...activeDaily.map((f) => ({ id: f.fieldId, label: f.label })),
|
||||
], [activeDaily]);
|
||||
|
||||
const matrix = useMemo(
|
||||
() => (selectedParam ? buildMatrix(statuses, animals, selectedParam) : { columns: [], rows: [] }),
|
||||
[statuses, animals, selectedParam],
|
||||
() => (dataParam
|
||||
? buildSubjectSeriesMatrix({ xField, dataParam, groupField: groupBy }, { statuses, animals, dailyTemplate })
|
||||
: { columns: [], rows: [], context: { label: 'Data', value: '' }, corner: '', groupAxis: null }),
|
||||
[xField, dataParam, groupBy, statuses, animals, dailyTemplate],
|
||||
);
|
||||
|
||||
const hasData = matrix.rows.length > 0;
|
||||
|
||||
// Group params for <optgroup> rendering, preserving order.
|
||||
const grouped = useMemo(() => {
|
||||
const groupedParams = useMemo(() => {
|
||||
const out = [];
|
||||
for (const p of params) {
|
||||
let g = out.find((x) => x.group === p.group);
|
||||
@@ -60,8 +74,8 @@ export default function ExportDataModal({ experimentTitle, experimentId, dailyTe
|
||||
}, [params]);
|
||||
|
||||
function handleExport() {
|
||||
if (!hasData || !selectedParam) return;
|
||||
triggerDownload(csvFilename(experimentTitle, selectedParam.label), toCSV(matrix));
|
||||
if (!hasData || !dataParam) return;
|
||||
triggerDownload(csvFilename(experimentTitle, dataParam.label), toCSV(matrix));
|
||||
onClose();
|
||||
}
|
||||
|
||||
@@ -69,38 +83,52 @@ export default function ExportDataModal({ experimentTitle, experimentId, dailyTe
|
||||
if (error) return (
|
||||
<div className="space-y-4">
|
||||
<Alert type="error" message={error} />
|
||||
<div className="flex justify-end">
|
||||
<Button variant="secondary" onClick={onClose}>Close</Button>
|
||||
</div>
|
||||
<div className="flex justify-end"><Button variant="secondary" onClick={onClose}>Close</Button></div>
|
||||
</div>
|
||||
);
|
||||
|
||||
const selectCls = 'w-full border border-gray-200 rounded px-2 py-1.5 text-sm bg-white focus:outline-none focus:ring-1 focus:ring-indigo-400';
|
||||
|
||||
return (
|
||||
<div className="space-y-4">
|
||||
<p className="text-sm text-gray-500">
|
||||
Export one parameter as a CSV file — each row is a date, each column is a subject.
|
||||
One value per subject over an X axis — each column is a subject, each row an X value. Blank where a subject has no value.
|
||||
</p>
|
||||
|
||||
<div>
|
||||
<label htmlFor="export-param" className="block text-xs font-medium text-gray-500 mb-1">Parameter</label>
|
||||
<select
|
||||
id="export-param"
|
||||
value={effectiveId ?? ''}
|
||||
onChange={(e) => setSelectedId(e.target.value)}
|
||||
className="w-full border border-gray-200 rounded px-2 py-1.5 text-sm bg-white focus:outline-none focus:ring-1 focus:ring-indigo-400"
|
||||
>
|
||||
{grouped.map((g) => (
|
||||
<optgroup key={g.group} label={GROUP_LABELS[g.group] ?? g.group}>
|
||||
<label htmlFor="export-x" className="block text-xs font-medium text-gray-500 mb-1">X axis (rows)</label>
|
||||
<select id="export-x" value={xField} onChange={(e) => setXField(e.target.value)} className={selectCls}>
|
||||
{xOptions.map((o) => <option key={o.id} value={o.id}>{o.label}</option>)}
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label htmlFor="export-data" className="block text-xs font-medium text-gray-500 mb-1">Data (cells)</label>
|
||||
<select id="export-data" value={effDataId ?? ''} onChange={(e) => setDataId(e.target.value)} className={selectCls}>
|
||||
{groupedParams.map((g) => (
|
||||
<optgroup key={g.group} label={PARAM_GROUP_LABELS[g.group] ?? g.group}>
|
||||
{g.items.map((p) => <option key={p.id} value={p.id}>{p.label}</option>)}
|
||||
</optgroup>
|
||||
))}
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label htmlFor="export-group" className="block text-xs font-medium text-gray-500 mb-1">Group by</label>
|
||||
<select id="export-group" value={groupBy} onChange={(e) => setGroupBy(e.target.value)} className={selectCls}>
|
||||
<option value="__none__">— None —</option>
|
||||
<option value="__name__">Name</option>
|
||||
<option value="__id__">Subject ID</option>
|
||||
{subjectTemplate.filter((f) => f.active).map((f) => (
|
||||
<option key={f.fieldId} value={f.fieldId}>{f.label}</option>
|
||||
))}
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<p className="text-xs text-gray-400">
|
||||
{hasData
|
||||
? `${matrix.rows.length} date${matrix.rows.length !== 1 ? 's' : ''} × ${matrix.columns.length} subject${matrix.columns.length !== 1 ? 's' : ''}`
|
||||
: 'No data for this parameter yet.'}
|
||||
? `${matrix.rows.length} row${matrix.rows.length !== 1 ? 's' : ''} × ${matrix.columns.length} subject${matrix.columns.length !== 1 ? 's' : ''}`
|
||||
: 'No data for this selection yet.'}
|
||||
</p>
|
||||
|
||||
<div className="flex justify-end gap-2 pt-2">
|
||||
|
||||
+168
-55
@@ -1,4 +1,7 @@
|
||||
// Pure helpers for exporting daily-parameter data as a date×subject CSV matrix.
|
||||
// Pure helpers for exporting daily-parameter data as a CSV matrix. The export is a
|
||||
// subject-series matrix: subjects are always columns, the chosen X coordinate
|
||||
// (Date / # Days Reach / a daily field) provides the rows, and one Data
|
||||
// parameter supplies the cell values.
|
||||
// No React, no I/O — mirrors the crossSubjectChart.js pure-helper pattern.
|
||||
|
||||
// Read a single parameter's raw value from one daily status.
|
||||
@@ -54,6 +57,140 @@ export function listExportableParams(dailyTemplate, statuses) {
|
||||
return params;
|
||||
}
|
||||
|
||||
// Numeric value of a daily field on one status (mirrors the metrics plot's helper).
|
||||
export function numericFieldVal(status, field) {
|
||||
if (!field) return null;
|
||||
const raw = field.builtin ? status?.[field.key] : status?.custom_fields?.[field.fieldId];
|
||||
const n = parseFloat(raw);
|
||||
return Number.isNaN(n) ? null : n;
|
||||
}
|
||||
|
||||
// Ordered row members for the chosen X coordinate.
|
||||
// xField: '__date__' | '__days__' | <daily fieldId>
|
||||
export function listSampleMembers(statuses, xField, dailyTemplate) {
|
||||
const list = statuses ?? [];
|
||||
if (xField === '__date__') {
|
||||
const keys = new Set();
|
||||
for (const s of list) if (s?.date) keys.add(dateKey(s.date));
|
||||
return [...keys].sort().map((k) => ({ id: k, label: k, sampleValue: k }));
|
||||
}
|
||||
if (xField === '__days__') {
|
||||
const counts = new Map();
|
||||
for (const s of list) {
|
||||
if (s?.animal_id == null || !s?.date) continue;
|
||||
counts.set(s.animal_id, (counts.get(s.animal_id) ?? 0) + 1);
|
||||
}
|
||||
const max = counts.size ? Math.max(...counts.values()) : 0;
|
||||
return Array.from({ length: max }, (_, i) => ({ id: String(i + 1), label: String(i + 1), sampleValue: i + 1 }));
|
||||
}
|
||||
const field = (dailyTemplate ?? []).find((f) => f.fieldId === xField);
|
||||
const vals = new Set();
|
||||
for (const s of list) {
|
||||
const v = numericFieldVal(s, field);
|
||||
if (v !== null) vals.add(v);
|
||||
}
|
||||
return [...vals].sort((a, b) => a - b).map((v) => ({ id: String(v), label: String(v), sampleValue: v }));
|
||||
}
|
||||
|
||||
// animalId -> Map(sampleValue -> status). First status per (subject, sampleValue) wins,
|
||||
// scanning each subject's statuses in ascending date order.
|
||||
export function buildSampleIndex(statuses, xField, dailyTemplate) {
|
||||
const bySubject = new Map();
|
||||
for (const s of statuses ?? []) {
|
||||
if (s?.animal_id == null || !s?.date) continue;
|
||||
if (!bySubject.has(s.animal_id)) bySubject.set(s.animal_id, []);
|
||||
bySubject.get(s.animal_id).push(s);
|
||||
}
|
||||
const field = xField !== '__date__' && xField !== '__days__'
|
||||
? (dailyTemplate ?? []).find((f) => f.fieldId === xField)
|
||||
: null;
|
||||
|
||||
const index = new Map();
|
||||
for (const [animalId, subjStatuses] of bySubject) {
|
||||
const ordered = [...subjStatuses].sort((a, b) => dateKey(a.date).localeCompare(dateKey(b.date)));
|
||||
const m = new Map();
|
||||
ordered.forEach((s, i) => {
|
||||
let key;
|
||||
if (xField === '__date__') key = dateKey(s.date);
|
||||
else if (xField === '__days__') key = i + 1;
|
||||
else { const v = numericFieldVal(s, field); if (v === null) return; key = v; }
|
||||
if (!m.has(key)) m.set(key, s);
|
||||
});
|
||||
index.set(animalId, m);
|
||||
}
|
||||
return index;
|
||||
}
|
||||
|
||||
function xFieldLabel(xField, dailyTemplate) {
|
||||
if (xField === '__date__') return 'Date';
|
||||
if (xField === '__days__') return '# Days Reach (ordinal)';
|
||||
return (dailyTemplate ?? []).find((f) => f.fieldId === xField)?.label ?? '';
|
||||
}
|
||||
|
||||
// Subjects as columns, X-field values as rows, one Data value per cell.
|
||||
// Returns the { context, corner, groupAxis, columns, rows } shape toCSV consumes.
|
||||
export function buildSubjectSeriesMatrix({ xField, dataParam, groupField }, { statuses, animals, dailyTemplate }) {
|
||||
const columns = listSubjectMembers(animals, groupField);
|
||||
const rowMembers = listSampleMembers(statuses, xField, dailyTemplate);
|
||||
const index = buildSampleIndex(statuses, xField, dailyTemplate);
|
||||
|
||||
const cellValue = (sampleValue, animalId) => {
|
||||
const status = index.get(animalId)?.get(sampleValue);
|
||||
if (!status || !dataParam) return '';
|
||||
const v = extractValue(status, dataParam);
|
||||
return hasValue(v) ? v : '';
|
||||
};
|
||||
|
||||
let rows = rowMembers.map((rm) => ({
|
||||
member: rm,
|
||||
values: columns.map((c) => cellValue(rm.sampleValue, c.animalId)),
|
||||
}));
|
||||
rows = rows.filter((r) => r.values.some((v) => v !== '')); // drop entirely-blank sample rows
|
||||
|
||||
const grouped = !!groupField && groupField !== '__none__';
|
||||
return {
|
||||
corner: xFieldLabel(xField, dailyTemplate),
|
||||
context: { label: 'Data', value: dataParam?.label ?? '' },
|
||||
groupAxis: grouped ? 'col' : null,
|
||||
columns,
|
||||
rows,
|
||||
};
|
||||
}
|
||||
|
||||
// Resolve a subject's group value for a given group field id.
|
||||
// null -> no grouping (field is falsy or '__none__')
|
||||
// '—' -> grouping is on but this subject has no value
|
||||
export function subjectGroupValue(animal, groupField) {
|
||||
if (!groupField || groupField === '__none__') return null;
|
||||
if (groupField === '__name__') return animal?.animal_name ?? '—';
|
||||
if (groupField === '__id__') return animal?.animal_id_string ?? '—';
|
||||
const v = animal?.subject_info?.[groupField];
|
||||
return v === null || v === undefined || v === '' ? '—' : v;
|
||||
}
|
||||
|
||||
export function listSubjectMembers(animals, groupField) {
|
||||
const list = [...(animals ?? [])];
|
||||
const nameCounts = new Map();
|
||||
for (const a of list) {
|
||||
const n = a?.animal_name ?? '';
|
||||
nameCounts.set(n, (nameCounts.get(n) ?? 0) + 1);
|
||||
}
|
||||
const members = list.map((a) => {
|
||||
const name = a?.animal_name ?? '';
|
||||
const label = nameCounts.get(name) > 1 ? `${name} (${a?.animal_id_string ?? ''})` : name;
|
||||
return { id: a?.id, label, animalId: a?.id, group: subjectGroupValue(a, groupField) };
|
||||
});
|
||||
const grouped = !!groupField && groupField !== '__none__';
|
||||
members.sort((x, y) => {
|
||||
if (grouped) {
|
||||
const g = String(x.group ?? '').localeCompare(String(y.group ?? ''));
|
||||
if (g !== 0) return g;
|
||||
}
|
||||
return String(x.label).localeCompare(String(y.label));
|
||||
});
|
||||
return members;
|
||||
}
|
||||
|
||||
// A cell has a value unless it is null, undefined, or the empty string.
|
||||
// (0 and false are real values.)
|
||||
function hasValue(v) {
|
||||
@@ -64,66 +201,42 @@ function dateKey(date) {
|
||||
return String(date).slice(0, 10); // ISO datetime or date → YYYY-MM-DD
|
||||
}
|
||||
|
||||
// Pivot statuses into { columns, rows }.
|
||||
// columns: [{ id, header }] — all animals, ordered by name, headers
|
||||
// disambiguated with animal_id_string on duplicate names.
|
||||
// rows: [{ date, values }] — one per date that has ≥1 value; values are
|
||||
// aligned to columns, blank cells are '' (empty string).
|
||||
export function buildMatrix(statuses, animals, param) {
|
||||
const sortedAnimals = [...(animals ?? [])].sort((a, b) =>
|
||||
String(a.animal_name ?? '').localeCompare(String(b.animal_name ?? '')),
|
||||
);
|
||||
|
||||
const nameCounts = new Map();
|
||||
for (const a of sortedAnimals) {
|
||||
const n = a.animal_name ?? '';
|
||||
nameCounts.set(n, (nameCounts.get(n) ?? 0) + 1);
|
||||
}
|
||||
const columns = sortedAnimals.map((a) => {
|
||||
const name = a.animal_name ?? '';
|
||||
const header = nameCounts.get(name) > 1 ? `${name} (${a.animal_id_string ?? ''})` : name;
|
||||
return { id: a.id, header };
|
||||
});
|
||||
const colIndex = new Map(columns.map((c, i) => [c.id, i]));
|
||||
|
||||
// cells: dateKey -> Map(animalId -> value); first status per (date, animal) wins.
|
||||
const cells = new Map();
|
||||
const datesWithData = new Set();
|
||||
const ordered = [...(statuses ?? [])].sort((a, b) => dateKey(a.date).localeCompare(dateKey(b.date)));
|
||||
for (const s of ordered) {
|
||||
if (!colIndex.has(s.animal_id)) continue;
|
||||
const dk = dateKey(s.date);
|
||||
if (!cells.has(dk)) cells.set(dk, new Map());
|
||||
const byAnimal = cells.get(dk);
|
||||
if (byAnimal.has(s.animal_id)) continue; // first wins
|
||||
const v = extractValue(s, param);
|
||||
byAnimal.set(s.animal_id, v);
|
||||
if (hasValue(v)) datesWithData.add(dk);
|
||||
}
|
||||
|
||||
const rows = [...datesWithData].sort().map((dk) => {
|
||||
const byAnimal = cells.get(dk);
|
||||
const values = columns.map((c) => {
|
||||
const v = byAnimal.has(c.id) ? byAnimal.get(c.id) : null;
|
||||
return hasValue(v) ? v : '';
|
||||
});
|
||||
return { date: dk, values };
|
||||
});
|
||||
|
||||
return { columns, rows };
|
||||
}
|
||||
|
||||
function escapeCsvCell(value) {
|
||||
const s = value === null || value === undefined ? '' : String(value);
|
||||
return /[",\n]/.test(s) ? `"${s.replace(/"/g, '""')}"` : s;
|
||||
}
|
||||
|
||||
// Serialize a matrix to an RFC-4180-ish CSV string (LF line endings).
|
||||
// First column is the date; remaining columns follow matrix.columns order.
|
||||
// Serialize a matrix to an RFC-4180-ish CSV string (LF line endings):
|
||||
// Data,<metric>
|
||||
// (blank line)
|
||||
// [Group,<group per subject column>] -- only when groupAxis === 'col'
|
||||
// <corner>[,Group?],<column headers...>
|
||||
// <row label>[,group?],<values...>
|
||||
export function toCSV(matrix) {
|
||||
const header = ['Date', ...matrix.columns.map((c) => c.header)].map(escapeCsvCell).join(',');
|
||||
const lines = matrix.rows.map((r) => [r.date, ...r.values].map(escapeCsvCell).join(','));
|
||||
return [header, ...lines].join('\n');
|
||||
const lines = [];
|
||||
lines.push([matrix.context.label, matrix.context.value].map(escapeCsvCell).join(','));
|
||||
lines.push('');
|
||||
|
||||
if (matrix.groupAxis === 'col') {
|
||||
const groupRow = ['Group', ...matrix.columns.map((c) => c.group ?? '—')];
|
||||
lines.push(groupRow.map(escapeCsvCell).join(','));
|
||||
}
|
||||
|
||||
// toCSV is a general serializer; the subject-series builder never emits groupAxis
|
||||
// === 'row' (only 'col' or null), but the branch is retained for that generality.
|
||||
const header = matrix.groupAxis === 'row'
|
||||
? ['Group', matrix.corner, ...matrix.columns.map((c) => c.label)]
|
||||
: [matrix.corner, ...matrix.columns.map((c) => c.label)];
|
||||
lines.push(header.map(escapeCsvCell).join(','));
|
||||
|
||||
for (const r of matrix.rows) {
|
||||
const lead = matrix.groupAxis === 'row'
|
||||
? [r.member.group ?? '—', r.member.label]
|
||||
: [r.member.label];
|
||||
lines.push([...lead, ...r.values].map(escapeCsvCell).join(','));
|
||||
}
|
||||
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
// Build a download filename from the experiment title and parameter label.
|
||||
|
||||
@@ -416,6 +416,8 @@ export default function ExperimentDetail() {
|
||||
experimentTitle={experiment.title}
|
||||
dailyTemplate={dailyTemplate}
|
||||
animals={animals}
|
||||
subjectTemplate={subjectTemplate}
|
||||
groupField={localStorage.getItem(`exp-subject-group-${id}`) ?? '__none__'}
|
||||
onClose={() => setShowExport(false)}
|
||||
/>
|
||||
</Modal>
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
import React from 'react';
|
||||
import { render, screen, waitFor, fireEvent } from '@testing-library/react';
|
||||
import ExportDataModal from '../src/components/ExportDataModal';
|
||||
import * as client from '../src/api/client';
|
||||
|
||||
jest.mock('../src/api/client', () => ({
|
||||
experimentsApi: { getDailyStatuses: jest.fn() },
|
||||
}));
|
||||
|
||||
const animals = [
|
||||
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
||||
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||
];
|
||||
const statuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5, success_rate: 0.5 } },
|
||||
{ animal_id: 'a1', date: '2026-07-02T00:00:00Z', analysis_summary: { total: 7, success_rate: 0.7 } },
|
||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 9, success_rate: 0.9 } },
|
||||
];
|
||||
const subjectTemplate = [{ fieldId: 'grp', label: 'Group', active: true }];
|
||||
|
||||
beforeEach(() => {
|
||||
jest.clearAllMocks();
|
||||
client.experimentsApi.getDailyStatuses.mockResolvedValue(statuses);
|
||||
});
|
||||
|
||||
// Intercept the Blob handed to URL.createObjectURL so tests can read the real CSV.
|
||||
// Also stub anchor.click() — jsdom treats a click on <a href="blob:..."> as an
|
||||
// unimplemented navigation that throws async and can flake a later test.
|
||||
function mockDownload() {
|
||||
let blob = null;
|
||||
const origCreate = global.URL.createObjectURL;
|
||||
const origRevoke = global.URL.revokeObjectURL;
|
||||
const origClick = window.HTMLAnchorElement.prototype.click;
|
||||
global.URL.createObjectURL = jest.fn((b) => { blob = b; return 'blob:mock'; });
|
||||
global.URL.revokeObjectURL = jest.fn();
|
||||
window.HTMLAnchorElement.prototype.click = jest.fn();
|
||||
return {
|
||||
getBlob: () => blob,
|
||||
restore: async () => {
|
||||
await new Promise((r) => setTimeout(r, 0)); // flush handleExport's setTimeout revoke
|
||||
global.URL.createObjectURL = origCreate;
|
||||
global.URL.revokeObjectURL = origRevoke;
|
||||
window.HTMLAnchorElement.prototype.click = origClick;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function readBlobText(blob) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const fr = new FileReader();
|
||||
fr.onload = () => resolve(fr.result);
|
||||
fr.onerror = reject;
|
||||
fr.readAsText(blob);
|
||||
});
|
||||
}
|
||||
|
||||
function renderModal(props = {}) {
|
||||
return render(
|
||||
<ExportDataModal
|
||||
experimentId="exp-1"
|
||||
experimentTitle="My Study"
|
||||
dailyTemplate={[]}
|
||||
animals={animals}
|
||||
subjectTemplate={subjectTemplate}
|
||||
groupField="__none__"
|
||||
onClose={jest.fn()}
|
||||
{...props}
|
||||
/>,
|
||||
);
|
||||
}
|
||||
|
||||
describe('ExportDataModal', () => {
|
||||
it('renders X / Data / Group by pickers; X defaults to Date', async () => {
|
||||
renderModal();
|
||||
expect(await screen.findByLabelText('X axis (rows)')).toHaveValue('__date__');
|
||||
expect(screen.getByLabelText('Data (cells)')).toBeInTheDocument();
|
||||
expect(screen.getByLabelText('Group by')).toHaveValue('__none__');
|
||||
await waitFor(() => expect(screen.getByText(/2 rows × 2 subjects/i)).toBeInTheDocument());
|
||||
});
|
||||
|
||||
it('default export (X=Date, Data=Total attempts) has subject columns, date rows, blank where missing', async () => {
|
||||
const dl = mockDownload();
|
||||
renderModal();
|
||||
await screen.findByLabelText('X axis (rows)');
|
||||
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||
fireEvent.click(screen.getByText('Export CSV'));
|
||||
const text = await readBlobText(dl.getBlob());
|
||||
expect(text).toBe('Data,Total attempts\n\nDate,Alpha,Beta\n2026-07-01,5,9\n2026-07-02,7,');
|
||||
await dl.restore();
|
||||
});
|
||||
|
||||
it('switching X to # Days Reach makes rows the day ordinals', async () => {
|
||||
const dl = mockDownload();
|
||||
renderModal();
|
||||
fireEvent.change(await screen.findByLabelText('X axis (rows)'), { target: { value: '__days__' } });
|
||||
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||
fireEvent.click(screen.getByText('Export CSV'));
|
||||
const text = await readBlobText(dl.getBlob());
|
||||
expect(text).toBe('Data,Total attempts\n\n# Days Reach (ordinal),Alpha,Beta\n1,5,9\n2,7,');
|
||||
await dl.restore();
|
||||
});
|
||||
|
||||
it('a valid group field adds a Group row and clusters subjects', async () => {
|
||||
const dl = mockDownload();
|
||||
renderModal({ groupField: 'grp' });
|
||||
await screen.findByLabelText('X axis (rows)');
|
||||
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||
fireEvent.click(screen.getByText('Export CSV'));
|
||||
const text = await readBlobText(dl.getBlob());
|
||||
expect(text).toMatch(/^Group,Control,Drug$/m);
|
||||
await dl.restore();
|
||||
});
|
||||
|
||||
it('a stale/unset group field coerces to None (no Group row in the export)', async () => {
|
||||
const dl = mockDownload();
|
||||
renderModal({ groupField: 'ghost-field' });
|
||||
await screen.findByLabelText('X axis (rows)');
|
||||
await waitFor(() => expect(screen.getByText(/2 rows/i)).toBeInTheDocument());
|
||||
fireEvent.click(screen.getByText('Export CSV'));
|
||||
const text = await readBlobText(dl.getBlob());
|
||||
expect(text).not.toMatch(/^Group,/m);
|
||||
await dl.restore();
|
||||
});
|
||||
});
|
||||
@@ -71,78 +71,61 @@ describe('listExportableParams', () => {
|
||||
});
|
||||
});
|
||||
|
||||
import { buildMatrix } from '../src/lib/dataExport';
|
||||
|
||||
const animals = [
|
||||
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002' },
|
||||
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001' },
|
||||
];
|
||||
const totalParam = { kind: 'total' };
|
||||
|
||||
describe('buildMatrix', () => {
|
||||
it('orders columns by animal_name and rows by date ascending', () => {
|
||||
const statuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-02T00:00:00.000Z', analysis_summary: { total: 5 } },
|
||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 9 } },
|
||||
];
|
||||
const m = buildMatrix(statuses, animals, totalParam);
|
||||
expect(m.columns.map((c) => c.header)).toEqual(['Alpha', 'Beta']);
|
||||
expect(m.rows.map((r) => r.date)).toEqual(['2026-07-01', '2026-07-02']);
|
||||
// Row for 07-01: Alpha blank, Beta 9
|
||||
expect(m.rows[0].values).toEqual(['', 9]);
|
||||
// Row for 07-02: Alpha 5, Beta blank
|
||||
expect(m.rows[1].values).toEqual([5, '']);
|
||||
});
|
||||
it('includes only dates that have at least one value (0 counts as a value)', () => {
|
||||
const statuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 0 } },
|
||||
{ animal_id: 'a1', date: '2026-07-03T00:00:00.000Z', analysis_summary: null }, // no value
|
||||
];
|
||||
const m = buildMatrix(statuses, animals, totalParam);
|
||||
expect(m.rows.map((r) => r.date)).toEqual(['2026-07-01']);
|
||||
expect(m.rows[0].values).toEqual([0, '']);
|
||||
});
|
||||
it('disambiguates duplicate animal names with the id string', () => {
|
||||
const dupAnimals = [
|
||||
{ id: 'a1', animal_name: 'Rat', animal_id_string: 'R-001' },
|
||||
{ id: 'a2', animal_name: 'Rat', animal_id_string: 'R-002' },
|
||||
];
|
||||
const statuses = [{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 1 } }];
|
||||
const m = buildMatrix(statuses, dupAnimals, totalParam);
|
||||
expect(m.columns.map((c) => c.header)).toEqual(['Rat (R-001)', 'Rat (R-002)']);
|
||||
});
|
||||
it('keeps the first status when a subject has duplicates on one date', () => {
|
||||
const statuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 7 } },
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: { total: 99 } },
|
||||
];
|
||||
const m = buildMatrix(statuses, animals, totalParam);
|
||||
expect(m.rows[0].values).toEqual([7, '']);
|
||||
});
|
||||
it('returns empty rows when no status has a value', () => {
|
||||
const m = buildMatrix([{ animal_id: 'a1', date: '2026-07-01T00:00:00.000Z', analysis_summary: null }], animals, totalParam);
|
||||
expect(m.rows).toEqual([]);
|
||||
expect(m.columns.length).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
import { toCSV, csvFilename } from '../src/lib/dataExport';
|
||||
|
||||
describe('toCSV', () => {
|
||||
it('writes a header row of Date + column headers, then data rows', () => {
|
||||
it('emits context line, blank line, header, data (no group)', () => {
|
||||
const matrix = {
|
||||
columns: [{ id: 'a1', header: 'Alpha' }, { id: 'a2', header: 'Beta' }],
|
||||
rows: [{ date: '2026-07-01', values: [5, ''] }, { date: '2026-07-02', values: ['', 9] }],
|
||||
};
|
||||
expect(toCSV(matrix)).toBe('Date,Alpha,Beta\n2026-07-01,5,\n2026-07-02,,9');
|
||||
});
|
||||
it('escapes commas, quotes, and newlines per RFC 4180', () => {
|
||||
const matrix = {
|
||||
columns: [{ id: 'a1', header: 'Note, field' }],
|
||||
rows: [{ date: '2026-07-01', values: ['he said "hi"'] }, { date: '2026-07-02', values: ['line1\nline2'] }],
|
||||
corner: 'Date',
|
||||
context: { label: 'Parameter', value: 'Total attempts' },
|
||||
groupAxis: null,
|
||||
columns: [{ id: 'a1', label: 'Alpha', group: null }, { id: 'a2', label: 'Beta', group: null }],
|
||||
rows: [{ member: { label: '2026-07-01' }, values: [5, ''] }, { member: { label: '2026-07-02' }, values: ['', 9] }],
|
||||
};
|
||||
expect(toCSV(matrix)).toBe(
|
||||
'Date,"Note, field"\n2026-07-01,"he said ""hi"""\n2026-07-02,"line1\nline2"',
|
||||
'Parameter,Total attempts\n\nDate,Alpha,Beta\n2026-07-01,5,\n2026-07-02,,9',
|
||||
);
|
||||
});
|
||||
|
||||
it('adds a Group header row when subjects are columns', () => {
|
||||
const matrix = {
|
||||
corner: 'Date',
|
||||
context: { label: 'Parameter', value: 'Total attempts' },
|
||||
groupAxis: 'col',
|
||||
columns: [{ id: 'a1', label: 'Alpha', group: 'Control' }, { id: 'a2', label: 'Beta', group: 'Drug' }],
|
||||
rows: [{ member: { label: '2026-07-01' }, values: [5, 9] }],
|
||||
};
|
||||
expect(toCSV(matrix)).toBe(
|
||||
'Parameter,Total attempts\n\nGroup,Control,Drug\nDate,Alpha,Beta\n2026-07-01,5,9',
|
||||
);
|
||||
});
|
||||
|
||||
it('adds a leading Group column when subjects are rows', () => {
|
||||
const matrix = {
|
||||
corner: 'Subject',
|
||||
context: { label: 'Date', value: '2026-07-01' },
|
||||
groupAxis: 'row',
|
||||
columns: [{ id: 'p1', label: 'Total attempts', group: 'metrics' }],
|
||||
rows: [
|
||||
{ member: { label: 'Alpha', group: 'Control' }, values: [5] },
|
||||
{ member: { label: 'Beta', group: 'Drug' }, values: [9] },
|
||||
],
|
||||
};
|
||||
expect(toCSV(matrix)).toBe(
|
||||
'Date,2026-07-01\n\nGroup,Subject,Total attempts\nControl,Alpha,5\nDrug,Beta,9',
|
||||
);
|
||||
});
|
||||
|
||||
it('escapes commas, quotes, and newlines per RFC 4180', () => {
|
||||
const matrix = {
|
||||
corner: 'Date',
|
||||
context: { label: 'Parameter', value: 'Note, field' },
|
||||
groupAxis: null,
|
||||
columns: [{ id: 'a1', label: 'he said "hi"', group: null }],
|
||||
rows: [{ member: { label: '2026-07-01' }, values: ['line1\nline2'] }],
|
||||
};
|
||||
expect(toCSV(matrix)).toBe(
|
||||
'Parameter,"Note, field"\n\nDate,"he said ""hi"""\n2026-07-01,"line1\nline2"',
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -155,3 +138,171 @@ describe('csvFilename', () => {
|
||||
expect(csvFilename('', '')).toBe('export.csv');
|
||||
});
|
||||
});
|
||||
|
||||
import { subjectGroupValue } from '../src/lib/dataExport';
|
||||
|
||||
describe('subjectGroupValue', () => {
|
||||
const animal = { animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { sex: 'M', cohort: '' } };
|
||||
it('returns null when no group field', () => {
|
||||
expect(subjectGroupValue(animal, '__none__')).toBeNull();
|
||||
expect(subjectGroupValue(animal, '')).toBeNull();
|
||||
});
|
||||
it('resolves name / id / subject_info fields', () => {
|
||||
expect(subjectGroupValue(animal, '__name__')).toBe('Alpha');
|
||||
expect(subjectGroupValue(animal, '__id__')).toBe('R-001');
|
||||
expect(subjectGroupValue(animal, 'sex')).toBe('M');
|
||||
});
|
||||
it('renders missing or empty values as an em dash', () => {
|
||||
expect(subjectGroupValue(animal, 'cohort')).toBe('—');
|
||||
expect(subjectGroupValue({}, 'sex')).toBe('—');
|
||||
});
|
||||
});
|
||||
|
||||
import { listSubjectMembers } from '../src/lib/dataExport';
|
||||
|
||||
describe('listSubjectMembers', () => {
|
||||
const animals = [
|
||||
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
||||
{ id: 'a3', animal_name: 'Gamma', animal_id_string: 'R-003', subject_info: { grp: 'Control' } },
|
||||
];
|
||||
it('orders by name and carries group=null when no group field', () => {
|
||||
const m = listSubjectMembers(animals, '__none__');
|
||||
expect(m.map((s) => s.label)).toEqual(['Alpha', 'Beta', 'Gamma']);
|
||||
expect(m.every((s) => s.group === null)).toBe(true);
|
||||
expect(m[0]).toMatchObject({ id: 'a1', animalId: 'a1' });
|
||||
});
|
||||
it('clusters by group value then name when grouping is on', () => {
|
||||
const m = listSubjectMembers(animals, 'grp');
|
||||
expect(m.map((s) => [s.group, s.label])).toEqual([
|
||||
['Control', 'Alpha'], ['Control', 'Gamma'], ['Drug', 'Beta'],
|
||||
]);
|
||||
});
|
||||
it('disambiguates duplicate names with the id string', () => {
|
||||
const dup = [
|
||||
{ id: 'a1', animal_name: 'Rat', animal_id_string: 'R-001' },
|
||||
{ id: 'a2', animal_name: 'Rat', animal_id_string: 'R-002' },
|
||||
];
|
||||
expect(listSubjectMembers(dup, '__none__').map((s) => s.label)).toEqual(['Rat (R-001)', 'Rat (R-002)']);
|
||||
});
|
||||
});
|
||||
|
||||
import { numericFieldVal, listSampleMembers, buildSampleIndex } from '../src/lib/dataExport';
|
||||
|
||||
const sDaily = [{ fieldId: 'c-w', key: 'w', label: 'Weight', builtin: false, active: true }];
|
||||
const sStatuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', custom_fields: { 'c-w': '250' } },
|
||||
{ animal_id: 'a1', date: '2026-07-03T00:00:00Z', custom_fields: { 'c-w': '260' } },
|
||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', custom_fields: { 'c-w': '250' } },
|
||||
];
|
||||
|
||||
describe('numericFieldVal', () => {
|
||||
it('reads builtin via key and custom via fieldId, parsed to number', () => {
|
||||
expect(numericFieldVal({ weight: '250' }, { builtin: true, key: 'weight' })).toBe(250);
|
||||
expect(numericFieldVal({ custom_fields: { 'c-w': '12.5' } }, { builtin: false, fieldId: 'c-w' })).toBe(12.5);
|
||||
});
|
||||
it('returns null for non-numeric, missing, or no field', () => {
|
||||
expect(numericFieldVal({ custom_fields: { 'c-w': 'abc' } }, { builtin: false, fieldId: 'c-w' })).toBeNull();
|
||||
expect(numericFieldVal({}, { builtin: true, key: 'weight' })).toBeNull();
|
||||
expect(numericFieldVal({ weight: 1 }, null)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('listSampleMembers', () => {
|
||||
it('date: distinct dates sorted', () => {
|
||||
expect(listSampleMembers(sStatuses, '__date__', sDaily).map((m) => m.sampleValue)).toEqual(['2026-07-01', '2026-07-03']);
|
||||
});
|
||||
it('# days reach: 1..max records per subject', () => {
|
||||
expect(listSampleMembers(sStatuses, '__days__', sDaily).map((m) => m.sampleValue)).toEqual([1, 2]);
|
||||
});
|
||||
it('daily field: distinct numeric values sorted ascending', () => {
|
||||
expect(listSampleMembers(sStatuses, 'c-w', sDaily).map((m) => m.sampleValue)).toEqual([250, 260]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildSampleIndex', () => {
|
||||
it('date coord maps animalId -> dateKey -> status', () => {
|
||||
const idx = buildSampleIndex(sStatuses, '__date__', sDaily);
|
||||
expect(idx.get('a1').get('2026-07-03').date).toContain('2026-07-03');
|
||||
});
|
||||
it('# days reach assigns per-subject ordinals in date order', () => {
|
||||
const idx = buildSampleIndex(sStatuses, '__days__', sDaily);
|
||||
expect(idx.get('a1').get(1).date).toContain('2026-07-01');
|
||||
expect(idx.get('a1').get(2).date).toContain('2026-07-03');
|
||||
expect(idx.get('a2').get(1).date).toContain('2026-07-01');
|
||||
expect(idx.get('a2').has(2)).toBe(false);
|
||||
});
|
||||
it('daily field maps numeric value to first status by date', () => {
|
||||
const idx = buildSampleIndex(sStatuses, 'c-w', sDaily);
|
||||
expect(idx.get('a1').get(250).date).toContain('2026-07-01');
|
||||
expect(idx.get('a1').get(260).date).toContain('2026-07-03');
|
||||
});
|
||||
it('# days reach counts per-status, not per-distinct-date (same-day statuses get separate ordinals)', () => {
|
||||
const sameDay = [
|
||||
{ animal_id: 'z1', date: '2026-07-01T08:00:00Z', analysis_summary: { total: 1 } },
|
||||
{ animal_id: 'z1', date: '2026-07-01T20:00:00Z', analysis_summary: { total: 2 } },
|
||||
];
|
||||
const idx = buildSampleIndex(sameDay, '__days__', []);
|
||||
expect(idx.get('z1').has(1)).toBe(true);
|
||||
expect(idx.get('z1').has(2)).toBe(true);
|
||||
expect(idx.get('z1').has(3)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
import { buildSubjectSeriesMatrix } from '../src/lib/dataExport';
|
||||
|
||||
const ssAnimals = [
|
||||
{ id: 'a1', animal_name: 'Alpha', animal_id_string: 'R-001', subject_info: { grp: 'Control' } },
|
||||
{ id: 'a2', animal_name: 'Beta', animal_id_string: 'R-002', subject_info: { grp: 'Drug' } },
|
||||
];
|
||||
const ssStatuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5 } },
|
||||
{ animal_id: 'a1', date: '2026-07-02T00:00:00Z', analysis_summary: { total: 7 } },
|
||||
{ animal_id: 'a2', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 9 } },
|
||||
];
|
||||
const totalParam = { id: '__total__', label: 'Total attempts', kind: 'total', group: 'metrics' };
|
||||
|
||||
describe('buildSubjectSeriesMatrix', () => {
|
||||
it('date X: subjects as columns, date rows, blank where missing, context names the metric', () => {
|
||||
const m = buildSubjectSeriesMatrix(
|
||||
{ xField: '__date__', dataParam: totalParam, groupField: '__none__' },
|
||||
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||
);
|
||||
expect(m.corner).toBe('Date');
|
||||
expect(m.context).toEqual({ label: 'Data', value: 'Total attempts' });
|
||||
expect(m.groupAxis).toBeNull();
|
||||
expect(m.columns.map((c) => c.label)).toEqual(['Alpha', 'Beta']);
|
||||
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01', '2026-07-02']);
|
||||
expect(m.rows[0].values).toEqual([5, 9]);
|
||||
expect(m.rows[1].values).toEqual([7, '']);
|
||||
});
|
||||
it('# days reach X: rows align each subject by day ordinal', () => {
|
||||
const m = buildSubjectSeriesMatrix(
|
||||
{ xField: '__days__', dataParam: totalParam, groupField: '__none__' },
|
||||
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||
);
|
||||
expect(m.corner).toBe('# Days Reach (ordinal)');
|
||||
expect(m.rows.map((r) => r.member.label)).toEqual(['1', '2']);
|
||||
expect(m.rows[0].values).toEqual([5, 9]);
|
||||
expect(m.rows[1].values).toEqual([7, '']);
|
||||
});
|
||||
it('grouping sets groupAxis col and clusters subjects', () => {
|
||||
const m = buildSubjectSeriesMatrix(
|
||||
{ xField: '__date__', dataParam: totalParam, groupField: 'grp' },
|
||||
{ statuses: ssStatuses, animals: ssAnimals, dailyTemplate: [] },
|
||||
);
|
||||
expect(m.groupAxis).toBe('col');
|
||||
expect(m.columns.map((c) => [c.group, c.label])).toEqual([['Control', 'Alpha'], ['Drug', 'Beta']]);
|
||||
});
|
||||
it('drops entirely-blank rows', () => {
|
||||
const statuses = [
|
||||
{ animal_id: 'a1', date: '2026-07-01T00:00:00Z', analysis_summary: { total: 5 } },
|
||||
{ animal_id: 'a1', date: '2026-07-05T00:00:00Z', analysis_summary: null },
|
||||
];
|
||||
const m = buildSubjectSeriesMatrix(
|
||||
{ xField: '__date__', dataParam: totalParam, groupField: '__none__' },
|
||||
{ statuses, animals: ssAnimals, dailyTemplate: [] },
|
||||
);
|
||||
expect(m.rows.map((r) => r.member.label)).toEqual(['2026-07-01']);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user