docs: design for experiment data export (CSV)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Experiments DB Dev
@@ -0,0 +1,148 @@
|
||||
# Experiment Data Export (CSV) — Design
|
||||
|
||||
**Date:** 2026-07-06
|
||||
**Status:** Approved, ready for implementation plan
|
||||
|
||||
## Goal
|
||||
|
||||
Let a user on the experiment page export a chosen daily parameter as a CSV
|
||||
file shaped as a matrix: **each row is a date, each column is a subject**, each
|
||||
cell is that subject's value for the parameter on that date. Every daily
|
||||
parameter must be exportable, including the session metrics derived from
|
||||
`analysis_summary`.
|
||||
|
||||
## Scope
|
||||
|
||||
- **In:** CSV export, one parameter per file, client-side generation, a picker
|
||||
UI on the experiment page.
|
||||
- **Out (deferred):** true `.xlsx` output (would add a client-side library such
|
||||
as SheetJS later; the pure helpers stay format-agnostic so the workbook
|
||||
builder can be added without reshaping the data), multi-parameter / multi-sheet
|
||||
export, a backend export endpoint.
|
||||
|
||||
## Approach
|
||||
|
||||
Fully client-side. The existing `GET /api/experiments/:id/daily-statuses`
|
||||
endpoint already returns every field required per status — builtin fields
|
||||
(`experiment_description`, `vitals`, `treatment`, `notes`), `custom_fields`, and
|
||||
`analysis_summary`. The experiment page currently fetches only the
|
||||
`analysisOnly=true` subset (for charts); the export modal fetches the **full**
|
||||
set on open, pivots it into a date×subject matrix in the browser, serializes to
|
||||
CSV, and triggers a download. No backend change.
|
||||
|
||||
Rejected alternative: a backend `/export` route streaming CSV. It only pays off
|
||||
for very large datasets or server-side xlsx, at the cost of a new route, tests,
|
||||
and duplicating the template/parameter logic on the server. Research datasets
|
||||
here are small (tens of subjects, hundreds of dates), so client-side is simpler
|
||||
and unit-testable.
|
||||
|
||||
## Exportable parameters
|
||||
|
||||
The picker is grouped into two sections.
|
||||
|
||||
### Session metrics (from `analysis_summary`)
|
||||
|
||||
- **Total attempts** → `analysis_summary.total`
|
||||
- **Success rate** → `analysis_summary.success_rate`, exported as the **raw
|
||||
0–1 fraction** (not multiplied to a percentage)
|
||||
- One entry per **dynamic count category** found in the data →
|
||||
`analysis_summary.counts[category]`. Categories are enumerated by scanning the
|
||||
fetched statuses (e.g. `Success`, `Failure`, `Other`, and any others present).
|
||||
|
||||
### Daily record fields (from the daily template)
|
||||
|
||||
- **Every** field defined in the daily template, including ones currently
|
||||
hidden/inactive, since historical data may exist for them.
|
||||
- Builtin fields read from `status[field.key]`; custom fields read from
|
||||
`status.custom_fields[field.fieldId]` (same accessor logic as
|
||||
`AnalysisCharts.numericFieldVal`).
|
||||
- Values export as-is: text fields (e.g. `notes`, `vitals`) as strings, numeric
|
||||
fields as numbers. No numeric coercion is required for export.
|
||||
|
||||
## Matrix shape
|
||||
|
||||
- **Rows** = every date that has at least one value for the chosen parameter
|
||||
across any subject, sorted ascending, formatted `YYYY-MM-DD`. The first column
|
||||
header is `Date`.
|
||||
- **Columns** = all enrolled animals, ordered by `animal_name`. A subject is a
|
||||
column even if it has no value for the parameter (blank cells). On duplicate
|
||||
`animal_name`, disambiguate the header with the subject's `animal_id_string`
|
||||
(e.g. `Rat A (R-001)`).
|
||||
- **Cell** = the subject's value for the parameter on that date. Assumes one
|
||||
daily status per (subject, date); if duplicates are ever present, the first in
|
||||
date-ascending order wins.
|
||||
- Empty result (no dates have any value) is surfaced in the UI as a disabled
|
||||
export / "no data for this parameter" note rather than producing an empty file.
|
||||
|
||||
## Components
|
||||
|
||||
### `frontend/src/lib/dataExport.js` (new, pure — no React, no I/O)
|
||||
|
||||
Mirrors the existing `frontend/src/lib/crossSubjectChart.js` pure-helper pattern.
|
||||
|
||||
- `listExportableParams(dailyTemplate, statuses)` → grouped, ordered parameter
|
||||
descriptors. Each descriptor carries an id, label, group (`'metrics'` |
|
||||
`'daily'`), and enough info to extract a value from a status. Session-metric
|
||||
categories are derived from `statuses`; template params from `dailyTemplate`.
|
||||
- `extractValue(status, param)` → the raw cell value for one status/param (or
|
||||
`null`/blank when absent). Encapsulates the builtin-vs-custom and
|
||||
`analysis_summary` accessor logic.
|
||||
- `buildMatrix(statuses, animals, param)` → `{ columns, rows }` where `columns`
|
||||
are the ordered subject headers and `rows` are `{ date, values: [...] }`
|
||||
aligned to `columns`, including blank cells; rows limited to dates with data.
|
||||
- `toCSV(matrix)` → CSV string. Escapes any field containing a comma, double
|
||||
quote, or newline per RFC 4180 (wrap in quotes, double embedded quotes).
|
||||
|
||||
### `frontend/src/components/ExportDataModal.jsx` (new)
|
||||
|
||||
- On open, fetches the full daily-statuses set via
|
||||
`experimentsApi.getDailyStatuses(id, {})` (no `analysisOnly`).
|
||||
- Renders a grouped `<select>` of exportable parameters (Session metrics /
|
||||
Daily record fields).
|
||||
- Shows a small summary line: "N dates × M subjects".
|
||||
- **Export CSV** button builds a `Blob`, creates an object URL, clicks a
|
||||
temporary anchor, and revokes the URL. Filename:
|
||||
`<sanitized-experiment-title>-<sanitized-param-label>.csv`.
|
||||
- Handles loading and error states for the fetch.
|
||||
|
||||
### `frontend/src/pages/ExperimentDetail.jsx` (edit)
|
||||
|
||||
- Add an **Export data** button in the header area (near **+ Add Animal**) that
|
||||
opens `ExportDataModal`, passing the experiment id, title, `dailyTemplate`,
|
||||
and `animals`.
|
||||
|
||||
## Data flow
|
||||
|
||||
1. User clicks **Export data** → modal opens.
|
||||
2. Modal fetches full statuses (all fields, all subjects).
|
||||
3. `listExportableParams(dailyTemplate, statuses)` builds the picker options.
|
||||
4. User selects a parameter → `buildMatrix(statuses, animals, param)` computes
|
||||
the date×subject matrix; summary line updates.
|
||||
5. User clicks **Export CSV** → `toCSV(matrix)` → Blob download.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Fetch failure: show an inline error in the modal; no download.
|
||||
- Parameter with no data across all subjects/dates: disable export and show a
|
||||
brief note.
|
||||
- CSV correctness relies on `toCSV` escaping; covered by tests.
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests for `dataExport.js`, following `frontend/tests/crossSubjectChart.test.js`:
|
||||
|
||||
- `listExportableParams`: includes all template fields (incl. inactive), the two
|
||||
fixed session metrics, and each dynamic count category discovered in the data;
|
||||
correct grouping/ordering.
|
||||
- `extractValue`: builtin vs custom field access; `total`, `success_rate` (raw
|
||||
0–1), and `counts[category]`; missing values → blank/null.
|
||||
- `buildMatrix`: rows only for dates with data, ascending; all subjects as
|
||||
columns with blanks; duplicate-name disambiguation; one-status-per-cell.
|
||||
- `toCSV`: escaping of commas, quotes, and newlines; header row; blank cells.
|
||||
|
||||
## Out of scope / future
|
||||
|
||||
- `.xlsx` output via a client-side workbook library (helpers already
|
||||
format-agnostic).
|
||||
- Selecting multiple parameters (multi-sheet xlsx or multi-file/zip CSV).
|
||||
- Server-side export for very large datasets.
|
||||
Reference in New Issue
Block a user