From 2d3b7c4607e416612d4aa7ad9c6d788dd8e8095d Mon Sep 17 00:00:00 2001
From: Experiments DB Dev <dev@experiments-db.local>
Date: Mon, 6 Jul 2026 15:25:25 -0400
Subject: [PATCH] docs: design for experiment data export (CSV)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 ...026-07-06-experiment-data-export-design.md | 148 ++++++++++++++++++
 1 file changed, 148 insertions(+)
 create mode 100644 docs/superpowers/specs/2026-07-06-experiment-data-export-design.md

diff --git a/docs/superpowers/specs/2026-07-06-experiment-data-export-design.md b/docs/superpowers/specs/2026-07-06-experiment-data-export-design.md
new file mode 100644
index 0000000..d03b087
--- /dev/null
+++ b/docs/superpowers/specs/2026-07-06-experiment-data-export-design.md
@@ -0,0 +1,148 @@
+# Experiment Data Export (CSV) — Design
+
+**Date:** 2026-07-06
+**Status:** Approved, ready for implementation plan
+
+## Goal
+
+Let a user on the experiment page export a chosen daily parameter as a CSV
+file shaped as a matrix: **each row is a date, each column is a subject**, each
+cell is that subject's value for the parameter on that date. Every daily
+parameter must be exportable, including the session metrics derived from
+`analysis_summary`.
+
+## Scope
+
+- **In:** CSV export, one parameter per file, client-side generation, a picker
+  UI on the experiment page.
+- **Out (deferred):** true `.xlsx` output (would add a client-side library such
+  as SheetJS later; the pure helpers stay format-agnostic so the workbook
+  builder can be added without reshaping the data), multi-parameter / multi-sheet
+  export, a backend export endpoint.
+
+## Approach
+
+Fully client-side. The existing `GET /api/experiments/:id/daily-statuses`
+endpoint already returns every field required per status — builtin fields
+(`experiment_description`, `vitals`, `treatment`, `notes`), `custom_fields`, and
+`analysis_summary`. The experiment page currently fetches only the
+`analysisOnly=true` subset (for charts); the export modal fetches the **full**
+set on open, pivots it into a date×subject matrix in the browser, serializes to
+CSV, and triggers a download. No backend change.
+
+Rejected alternative: a backend `/export` route streaming CSV. It only pays off
+for very large datasets or server-side xlsx, at the cost of a new route, tests,
+and duplicating the template/parameter logic on the server. Research datasets
+here are small (tens of subjects, hundreds of dates), so client-side is simpler
+and unit-testable.
+
+## Exportable parameters
+
+The picker is grouped into two sections.
+
+### Session metrics (from `analysis_summary`)
+
+- **Total attempts** → `analysis_summary.total`
+- **Success rate** → `analysis_summary.success_rate`, exported as the **raw
+  0–1 fraction** (not multiplied to a percentage)
+- One entry per **dynamic count category** found in the data →
+  `analysis_summary.counts[category]`. Categories are enumerated by scanning the
+  fetched statuses (e.g. `Success`, `Failure`, `Other`, and any others present).
+
+### Daily record fields (from the daily template)
+
+- **Every** field defined in the daily template, including ones currently
+  hidden/inactive, since historical data may exist for them.
+- Builtin fields read from `status[field.key]`; custom fields read from
+  `status.custom_fields[field.fieldId]` (same accessor logic as
+  `AnalysisCharts.numericFieldVal`).
+- Values export as-is: text fields (e.g. `notes`, `vitals`) as strings, numeric
+  fields as numbers. No numeric coercion is required for export.
+
+## Matrix shape
+
+- **Rows** = every date that has at least one value for the chosen parameter
+  across any subject, sorted ascending, formatted `YYYY-MM-DD`. The first column
+  header is `Date`.
+- **Columns** = all enrolled animals, ordered by `animal_name`. A subject is a
+  column even if it has no value for the parameter (blank cells). On duplicate
+  `animal_name`, disambiguate the header with the subject's `animal_id_string`
+  (e.g. `Rat A (R-001)`).
+- **Cell** = the subject's value for the parameter on that date. Assumes one
+  daily status per (subject, date); if duplicates are ever present, the first in
+  date-ascending order wins.
+- Empty result (no dates have any value) is surfaced in the UI as a disabled
+  export / "no data for this parameter" note rather than producing an empty file.
+
+## Components
+
+### `frontend/src/lib/dataExport.js` (new, pure — no React, no I/O)
+
+Mirrors the existing `frontend/src/lib/crossSubjectChart.js` pure-helper pattern.
+
+- `listExportableParams(dailyTemplate, statuses)` → grouped, ordered parameter
+  descriptors. Each descriptor carries an id, label, group (`'metrics'` |
+  `'daily'`), and enough info to extract a value from a status. Session-metric
+  categories are derived from `statuses`; template params from `dailyTemplate`.
+- `extractValue(status, param)` → the raw cell value for one status/param (or
+  `null`/blank when absent). Encapsulates the builtin-vs-custom and
+  `analysis_summary` accessor logic.
+- `buildMatrix(statuses, animals, param)` → `{ columns, rows }` where `columns`
+  are the ordered subject headers and `rows` are `{ date, values: [...] }`
+  aligned to `columns`, including blank cells; rows limited to dates with data.
+- `toCSV(matrix)` → CSV string. Escapes any field containing a comma, double
+  quote, or newline per RFC 4180 (wrap in quotes, double embedded quotes).
+
+### `frontend/src/components/ExportDataModal.jsx` (new)
+
+- On open, fetches the full daily-statuses set via
+  `experimentsApi.getDailyStatuses(id, {})` (no `analysisOnly`).
+- Renders a grouped `<select>` of exportable parameters (Session metrics /
+  Daily record fields).
+- Shows a small summary line: "N dates × M subjects".
+- **Export CSV** button builds a `Blob`, creates an object URL, clicks a
+  temporary anchor, and revokes the URL. Filename:
+  `<sanitized-experiment-title>-<sanitized-param-label>.csv`.
+- Handles loading and error states for the fetch.
+
+### `frontend/src/pages/ExperimentDetail.jsx` (edit)
+
+- Add an **Export data** button in the header area (near **+ Add Animal**) that
+  opens `ExportDataModal`, passing the experiment id, title, `dailyTemplate`,
+  and `animals`.
+
+## Data flow
+
+1. User clicks **Export data** → modal opens.
+2. Modal fetches full statuses (all fields, all subjects).
+3. `listExportableParams(dailyTemplate, statuses)` builds the picker options.
+4. User selects a parameter → `buildMatrix(statuses, animals, param)` computes
+   the date×subject matrix; summary line updates.
+5. User clicks **Export CSV** → `toCSV(matrix)` → Blob download.
+
+## Error handling
+
+- Fetch failure: show an inline error in the modal; no download.
+- Parameter with no data across all subjects/dates: disable export and show a
+  brief note.
+- CSV correctness relies on `toCSV` escaping; covered by tests.
+
+## Testing
+
+Unit tests for `dataExport.js`, following `frontend/tests/crossSubjectChart.test.js`:
+
+- `listExportableParams`: includes all template fields (incl. inactive), the two
+  fixed session metrics, and each dynamic count category discovered in the data;
+  correct grouping/ordering.
+- `extractValue`: builtin vs custom field access; `total`, `success_rate` (raw
+  0–1), and `counts[category]`; missing values → blank/null.
+- `buildMatrix`: rows only for dates with data, ascending; all subjects as
+  columns with blanks; duplicate-name disambiguation; one-status-per-cell.
+- `toCSV`: escaping of commas, quotes, and newlines; header row; blank cells.
+
+## Out of scope / future
+
+- `.xlsx` output via a client-side workbook library (helpers already
+  format-agnostic).
+- Selecting multiple parameters (multi-sheet xlsx or multi-file/zip CSV).
+- Server-side export for very large datasets.