Experiments DB Dev
102 lines
6.1 KiB
Markdown
102 lines
6.1 KiB
Markdown
# Configurable Data Export — Design
|
||
|
||
**Date:** 2026-07-19
|
||
**Status:** Approved (design)
|
||
**Area:** `frontend/src/lib/dataExport.js`, `frontend/src/components/ExportDataModal.jsx`, `frontend/tests/dataExport.test.js`
|
||
|
||
## Problem
|
||
|
||
Today's data export has a fixed layout: each **row is a date**, each **column is a subject**, and the user picks **one parameter** whose value fills every cell (`buildMatrix` in `dataExport.js`). Users want to:
|
||
|
||
1. Choose which dimension goes on rows and which on columns (fully assignable axes).
|
||
2. See the pinned/third dimension labeled at the top of the file ("group information at the beginning").
|
||
3. When Subject is an axis, see each subject's **group** in a separate cell, similar to the cross-subject metrics view.
|
||
|
||
## Model: three interchangeable dimensions
|
||
|
||
Export data spans three dimensions: **Date**, **Subject**, **Parameter**. A 2-D CSV grid can place two of them on the axes; the third must be **pinned** to a single value (a cell holds exactly one value).
|
||
|
||
Each dimension yields an ordered list of **members**:
|
||
|
||
- **Date** → sorted unique `YYYY-MM-DD` keys present in the data.
|
||
- **Subject** → animals, headers disambiguated by `animal_id_string` on duplicate names (existing logic). When a group field is chosen, subjects are **clustered by group value, then sorted by name** within each group.
|
||
- **Parameter** → `listExportableParams(dailyTemplate, statuses)` (session metrics, then daily fields — unchanged).
|
||
|
||
Every cell is identified by a full `(dateKey, animalId, param)` triple: two coordinates come from the row/column members, the third from the pinned value. Lookup goes through a status index `Map(dateKey → Map(animalId → status))` (first-status-per-cell wins, as today), then `extractValue(status, param)`.
|
||
|
||
Because Subject is always exactly one of the three dimensions (row, column, or pinned), the group annotation attaches to **at most one axis** — there is no 2-D corner-block complexity.
|
||
|
||
## Modal UI (`ExportDataModal.jsx`)
|
||
|
||
Replaces today's single "Parameter" dropdown with four controls:
|
||
|
||
- **Rows** — Date / Subject / Parameter.
|
||
- **Columns** — the two dimensions not chosen for Rows (auto-excludes the row pick; if a change collides, the other axis auto-shifts to a still-valid dimension).
|
||
- **Fixed: `<leftover dim>`** — value dropdown for the auto-pinned third dimension. Its options are that dimension's members (a date, a subject, or a parameter, grouped like today for parameters).
|
||
- **Group by** — subject_info fields plus "— None —". **Shown only when Subject is a row or column axis.** Defaults to the experiment's saved group field (`localStorage['exp-subject-group-<id>']`); falls back to None when unset or the field no longer exists.
|
||
|
||
Live preview line stays: `N rows × M columns` (or "No data …" when empty). **Default state reproduces today's export exactly**: Rows = Date, Columns = Subject, Fixed = Parameter (first parameter). Export button disabled when the grid has no data rows.
|
||
|
||
## CSV output format
|
||
|
||
Structure: pinned-dimension context line, blank line, optional group row, header row, data rows.
|
||
|
||
- **Line 1 (context):** two cells — the pinned dimension's label and its value, e.g. `Parameter,Total attempts` or `Subject,Mouse-01`. Beginning only (no footer).
|
||
- **Blank line.**
|
||
- **Grid**, with the group cell riding on whichever axis holds Subject.
|
||
|
||
**Subject = columns** (e.g. Rows = Date, Fixed = Parameter) → extra **header row** labeled `Group`:
|
||
|
||
```
|
||
Parameter,Total attempts
|
||
|
||
Group,Control,Control,Drug,Drug
|
||
Date,Mouse-01,Mouse-02,Mouse-03,Mouse-04
|
||
2026-01-01,12,10,8,9
|
||
```
|
||
|
||
**Subject = rows** (e.g. Cols = Parameter, Fixed = Date) → extra **leading column** labeled `Group`:
|
||
|
||
```
|
||
Date,2026-01-01
|
||
|
||
Group,Subject,Total attempts,Success rate
|
||
Control,Mouse-01,12,0.83
|
||
Drug,Mouse-03,8,0.71
|
||
```
|
||
|
||
Rules:
|
||
|
||
- The grid **corner label** is the row dimension's name (`Date`, `Subject`, or `Parameter`) — no longer hardcoded `Date`.
|
||
- Missing group value → `—`. When **Group by = None**, the group row/column is omitted entirely.
|
||
- **Empty-member rule:** Subject members are **always kept** (matches today's "keep all subject columns"). Date and Parameter members are **dropped when entirely blank** in the current grid.
|
||
- Cell values use existing semantics: `0` / `''` are real; genuinely-missing lookups render as blank. CSV escaping via existing `escapeCsvCell`.
|
||
|
||
## Refactor of `dataExport.js`
|
||
|
||
Keep `extractValue` and `listExportableParams` unchanged. Replace single-purpose `buildMatrix` with dimension-agnostic pieces:
|
||
|
||
- `listDateMembers(statuses)` → sorted unique dateKeys.
|
||
- `listSubjectMembers(animals, groupField)` → `[{ id, header, group }]`, clustered by group then name; header disambiguation preserved.
|
||
- `buildStatusIndex(statuses)` → `Map(dateKey → Map(animalId → status))`, first-wins.
|
||
- `buildMatrix({ rowDim, colDim, pinnedDim, pinnedMember, groupField }, { statuses, animals, dailyTemplate })` → `{ context, groupAxis, corner, columns, rows }`. Resolves each cell's `(dateKey, animalId, param)` triple through the index + `extractValue`; applies the keep-subjects / drop-blank-date-and-param rule.
|
||
- `toCSV(matrix)` → emits context line, blank line, optional group row, header row, data rows, with the dynamic corner label.
|
||
- `csvFilename(title, pinnedMember.label)` → signature unchanged.
|
||
|
||
## Testing (`tests/dataExport.test.js`)
|
||
|
||
Existing `extractValue` / `listExportableParams` tests stay green. Add:
|
||
|
||
- `listDateMembers`, `listSubjectMembers` (including group clustering + name tie-break + header disambiguation), `buildStatusIndex` (first-wins).
|
||
- `buildMatrix` for all three axis pairings in both orientations.
|
||
- Group row (Subject = columns) vs. group column (Subject = rows) placement; `—` fallback; Group = None omits it.
|
||
- Empty-member rule: blank date/parameter members dropped, subjects retained.
|
||
- `toCSV` snapshot including the context line and a group row.
|
||
|
||
## Out of scope (YAGNI)
|
||
|
||
- Stacking multiple values of the pinned dimension into one file (chose "pick one value").
|
||
- Footer/repeat of the context line (beginning only).
|
||
- Experiment-level metadata block (pinned dimension only).
|
||
- Persisting axis choices across sessions (default reproduces current behavior; group field reuses existing saved value only).
|