experiments-database/docs/superpowers/specs/2026-07-06-experiment-data-export-design.md

# Experiment Data Export (CSV) — Design

**Date:** 2026-07-06
**Status:** Approved, ready for implementation plan

## Goal

Let a user on the experiment page export a chosen daily parameter as a CSV
file shaped as a matrix: **each row is a date, each column is a subject**, each
cell is that subject's value for the parameter on that date. Every daily
parameter must be exportable, including the session metrics derived from
`analysis_summary`.

## Scope

- **In:** CSV export, one parameter per file, client-side generation, a picker
  UI on the experiment page.
- **Out (deferred):** true `.xlsx` output (would add a client-side library such
  as SheetJS later; the pure helpers stay format-agnostic so the workbook
  builder can be added without reshaping the data), multi-parameter / multi-sheet
  export, a backend export endpoint.

## Approach

Fully client-side. The existing `GET /api/experiments/:id/daily-statuses`
endpoint already returns every field required per status — builtin fields
(`experiment_description`, `vitals`, `treatment`, `notes`), `custom_fields`, and
`analysis_summary`. The experiment page currently fetches only the
`analysisOnly=true` subset (for charts); the export modal fetches the **full**
set on open, pivots it into a date×subject matrix in the browser, serializes to
CSV, and triggers a download. No backend change.

Rejected alternative: a backend `/export` route streaming CSV. It only pays off
for very large datasets or server-side xlsx, at the cost of a new route, tests,
and duplicating the template/parameter logic on the server. Research datasets
here are small (tens of subjects, hundreds of dates), so client-side is simpler
and unit-testable.

## Exportable parameters

The picker is grouped into two sections.

### Session metrics (from `analysis_summary`)

- **Total attempts** → `analysis_summary.total`
- **Success rate** → `analysis_summary.success_rate`, exported as the **raw
  0–1 fraction** (not multiplied to a percentage)
- One entry per **dynamic count category** found in the data →
  `analysis_summary.counts[category]`. Categories are enumerated by scanning the
  fetched statuses (e.g. `Success`, `Failure`, `Other`, and any others present).

### Daily record fields (from the daily template)

- **Every** field defined in the daily template, including ones currently
  hidden/inactive, since historical data may exist for them.
- Builtin fields read from `status[field.key]`; custom fields read from
  `status.custom_fields[field.fieldId]` (same accessor logic as
  `AnalysisCharts.numericFieldVal`).
- Values export as-is: text fields (e.g. `notes`, `vitals`) as strings, numeric
  fields as numbers. No numeric coercion is required for export.

## Matrix shape

- **Rows** = every date that has at least one value for the chosen parameter
  across any subject, sorted ascending, formatted `YYYY-MM-DD`. The first column
  header is `Date`.
- **Columns** = all enrolled animals, ordered by `animal_name`. A subject is a
  column even if it has no value for the parameter (blank cells). On duplicate
  `animal_name`, disambiguate the header with the subject's `animal_id_string`
  (e.g. `Rat A (R-001)`).
- **Cell** = the subject's value for the parameter on that date. Assumes one
  daily status per (subject, date); if duplicates are ever present, the first in
  date-ascending order wins.
- Empty result (no dates have any value) is surfaced in the UI as a disabled
  export / "no data for this parameter" note rather than producing an empty file.

## Components

### `frontend/src/lib/dataExport.js` (new, pure — no React, no I/O)

Mirrors the existing `frontend/src/lib/crossSubjectChart.js` pure-helper pattern.

- `listExportableParams(dailyTemplate, statuses)` → grouped, ordered parameter
  descriptors. Each descriptor carries an id, label, group (`'metrics'` |
  `'daily'`), and enough info to extract a value from a status. Session-metric
  categories are derived from `statuses`; template params from `dailyTemplate`.
- `extractValue(status, param)` → the raw cell value for one status/param (or
  `null`/blank when absent). Encapsulates the builtin-vs-custom and
  `analysis_summary` accessor logic.
- `buildMatrix(statuses, animals, param)` → `{ columns, rows }` where `columns`
  are the ordered subject headers and `rows` are `{ date, values: [...] }`
  aligned to `columns`, including blank cells; rows limited to dates with data.
- `toCSV(matrix)` → CSV string. Escapes any field containing a comma, double
  quote, or newline per RFC 4180 (wrap in quotes, double embedded quotes).

### `frontend/src/components/ExportDataModal.jsx` (new)

- On open, fetches the full daily-statuses set via
  `experimentsApi.getDailyStatuses(id, {})` (no `analysisOnly`).
- Renders a grouped `<select>` of exportable parameters (Session metrics /
  Daily record fields).
- Shows a small summary line: "N dates × M subjects".
- **Export CSV** button builds a `Blob`, creates an object URL, clicks a
  temporary anchor, and revokes the URL. Filename:
  `<sanitized-experiment-title>-<sanitized-param-label>.csv`.
- Handles loading and error states for the fetch.

### `frontend/src/pages/ExperimentDetail.jsx` (edit)

- Add an **Export data** button in the header area (near **+ Add Animal**) that
  opens `ExportDataModal`, passing the experiment id, title, `dailyTemplate`,
  and `animals`.

## Data flow

1. User clicks **Export data** → modal opens.
2. Modal fetches full statuses (all fields, all subjects).
3. `listExportableParams(dailyTemplate, statuses)` builds the picker options.
4. User selects a parameter → `buildMatrix(statuses, animals, param)` computes
   the date×subject matrix; summary line updates.
5. User clicks **Export CSV** → `toCSV(matrix)` → Blob download.

## Error handling

- Fetch failure: show an inline error in the modal; no download.
- Parameter with no data across all subjects/dates: disable export and show a
  brief note.
- CSV correctness relies on `toCSV` escaping; covered by tests.

## Testing

Unit tests for `dataExport.js`, following `frontend/tests/crossSubjectChart.test.js`:

- `listExportableParams`: includes all template fields (incl. inactive), the two
  fixed session metrics, and each dynamic count category discovered in the data;
  correct grouping/ordering.
- `extractValue`: builtin vs custom field access; `total`, `success_rate` (raw
  0–1), and `counts[category]`; missing values → blank/null.
- `buildMatrix`: rows only for dates with data, ascending; all subjects as
  columns with blanks; duplicate-name disambiguation; one-status-per-cell.
- `toCSV`: escaping of commas, quotes, and newlines; header row; blank cells.

## Out of scope / future

- `.xlsx` output via a client-side workbook library (helpers already
  format-agnostic).
- Selecting multiple parameters (multi-sheet xlsx or multi-file/zip CSV).
- Server-side export for very large datasets.