experiments-database/docs/superpowers/specs/2026-07-06-experiment-data-export-design.md

Experiment Data Export (CSV) — Design

Date: 2026-07-06 Status: Approved, ready for implementation plan

Goal

Let a user on the experiment page export a chosen daily parameter as a CSV file shaped as a matrix: each row is a date, each column is a subject, each cell is that subject's value for the parameter on that date. Every daily parameter must be exportable, including the session metrics derived from analysis_summary.

Scope

• In: CSV export, one parameter per file, client-side generation, a picker UI on the experiment page.
• Out (deferred): true .xlsx output (would add a client-side library such as SheetJS later; the pure helpers stay format-agnostic so the workbook builder can be added without reshaping the data), multi-parameter / multi-sheet export, a backend export endpoint.

Approach

Fully client-side. The existing GET /api/experiments/:id/daily-statuses endpoint already returns every field required per status — builtin fields (experiment_description, vitals, treatment, notes), custom_fields, and analysis_summary. The experiment page currently fetches only the analysisOnly=true subset (for charts); the export modal fetches the full set on open, pivots it into a date×subject matrix in the browser, serializes to CSV, and triggers a download. No backend change.

Rejected alternative: a backend /export route streaming CSV. It only pays off for very large datasets or server-side xlsx, at the cost of a new route, tests, and duplicating the template/parameter logic on the server. Research datasets here are small (tens of subjects, hundreds of dates), so client-side is simpler and unit-testable.

Exportable parameters

The picker is grouped into two sections.

Session metrics (from analysis_summary)

• Total attempts → analysis_summary.total
• Success rate → analysis_summary.success_rate, exported as the raw 0–1 fraction (not multiplied to a percentage)
• One entry per dynamic count category found in the data → analysis_summary.counts[category]. Categories are enumerated by scanning the fetched statuses (e.g. Success, Failure, Other, and any others present).

Daily record fields (from the daily template)

• Every field defined in the daily template, including ones currently hidden/inactive, since historical data may exist for them.
• Builtin fields read from status[field.key]; custom fields read from status.custom_fields[field.fieldId] (same accessor logic as AnalysisCharts.numericFieldVal).
• Values export as-is: text fields (e.g. notes, vitals) as strings, numeric fields as numbers. No numeric coercion is required for export.

Matrix shape

• Rows = every date that has at least one value for the chosen parameter across any subject, sorted ascending, formatted YYYY-MM-DD. The first column header is Date.
• Columns = all enrolled animals, ordered by animal_name. A subject is a column even if it has no value for the parameter (blank cells). On duplicate animal_name, disambiguate the header with the subject's animal_id_string (e.g. Rat A (R-001)).
• Cell = the subject's value for the parameter on that date. Assumes one daily status per (subject, date); if duplicates are ever present, the first in date-ascending order wins.
• Empty result (no dates have any value) is surfaced in the UI as a disabled export / "no data for this parameter" note rather than producing an empty file.

Components

frontend/src/lib/dataExport.js (new, pure — no React, no I/O)

Mirrors the existing frontend/src/lib/crossSubjectChart.js pure-helper pattern.

• listExportableParams(dailyTemplate, statuses) → grouped, ordered parameter descriptors. Each descriptor carries an id, label, group ('metrics' | 'daily'), and enough info to extract a value from a status. Session-metric categories are derived from statuses; template params from dailyTemplate.
• extractValue(status, param) → the raw cell value for one status/param (or null/blank when absent). Encapsulates the builtin-vs-custom and analysis_summary accessor logic.
• buildMatrix(statuses, animals, param) → { columns, rows } where columns are the ordered subject headers and rows are { date, values: [...] } aligned to columns, including blank cells; rows limited to dates with data.
• toCSV(matrix) → CSV string. Escapes any field containing a comma, double quote, or newline per RFC 4180 (wrap in quotes, double embedded quotes).

frontend/src/components/ExportDataModal.jsx (new)

• On open, fetches the full daily-statuses set via experimentsApi.getDailyStatuses(id, {}) (no analysisOnly).
• Renders a grouped <select> of exportable parameters (Session metrics / Daily record fields).
• Shows a small summary line: "N dates × M subjects".
• Export CSV button builds a Blob, creates an object URL, clicks a temporary anchor, and revokes the URL. Filename: <sanitized-experiment-title>-<sanitized-param-label>.csv.
• Handles loading and error states for the fetch.

frontend/src/pages/ExperimentDetail.jsx (edit)

• Add an Export data button in the header area (near + Add Animal) that opens ExportDataModal, passing the experiment id, title, dailyTemplate, and animals.

Data flow

1. User clicks Export data → modal opens.
2. Modal fetches full statuses (all fields, all subjects).
3. listExportableParams(dailyTemplate, statuses) builds the picker options.
4. User selects a parameter → buildMatrix(statuses, animals, param) computes the date×subject matrix; summary line updates.
5. User clicks Export CSV → toCSV(matrix) → Blob download.

Error handling

• Fetch failure: show an inline error in the modal; no download.
• Parameter with no data across all subjects/dates: disable export and show a brief note.
• CSV correctness relies on toCSV escaping; covered by tests.

Testing

Unit tests for dataExport.js, following frontend/tests/crossSubjectChart.test.js:

• listExportableParams: includes all template fields (incl. inactive), the two fixed session metrics, and each dynamic count category discovered in the data; correct grouping/ordering.
• extractValue: builtin vs custom field access; total, success_rate (raw 0–1), and counts[category]; missing values → blank/null.
• buildMatrix: rows only for dates with data, ascending; all subjects as columns with blanks; duplicate-name disambiguation; one-status-per-cell.
• toCSV: escaping of commas, quotes, and newlines; header row; blank cells.

Out of scope / future

• .xlsx output via a client-side workbook library (helpers already format-agnostic).
• Selecting multiple parameters (multi-sheet xlsx or multi-file/zip CSV).
• Server-side export for very large datasets.