From 2ed69aa6d6a37b7ff56c25110828e9a03df40a9b Mon Sep 17 00:00:00 2001 From: Experiments DB Dev Date: Fri, 1 May 2026 07:30:18 -0400 Subject: [PATCH] docs: add design spec for numerical CSV-analysis buckets per animal Spec for adding a numerical bucket type to CSV analysis that plots note values (numeric portion) against frame/timestamp, persisted on the Animal model so subsequent uploads reuse the configuration. Co-Authored-By: Claude Opus 4.7 --- ...2026-05-01-numerical-csv-buckets-design.md | 235 ++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-01-numerical-csv-buckets-design.md diff --git a/docs/superpowers/specs/2026-05-01-numerical-csv-buckets-design.md b/docs/superpowers/specs/2026-05-01-numerical-csv-buckets-design.md new file mode 100644 index 0000000..1164f00 --- /dev/null +++ b/docs/superpowers/specs/2026-05-01-numerical-csv-buckets-design.md @@ -0,0 +1,235 @@ +# Numerical CSV-analysis buckets, persisted per animal + +**Date:** 2026-05-01 +**Scope:** `frontend/src/components/CsvAnalysis.jsx`, `frontend/src/lib/csvAnalysis.js`, `frontend/src/api/client.js`, `frontend/src/pages/DailyStatusDetail.jsx`, `backend/prisma/schema.prisma`, backend animal route. + +## Goal + +In the daily-status CSV-analysis flow, let the user define an extra type of bucket — a **numerical** bucket — alongside the existing note-classification buckets (Success / Failure / Other / custom). A numerical bucket plots its assigned notes' numeric portion against the CSV frame column (or timestamp as fallback) and reports an average. The bucket structure and per-bucket note assignments persist on the `Animal` so subsequent CSV uploads for the same subject don't need to be re-classified from scratch. + +## Section 0 — Preconditions + +Before applying the schema migration, back up the live Postgres database to a sibling `archived/` folder. + +```bash +mkdir -p /home/sam/docker-images/archived +docker exec experiments_postgres pg_dump -U expuser -d experiments_db -F c \ + > /home/sam/docker-images/archived/experiments_db_$(date +%Y%m%d_%H%M%S).dump +``` + +- Verify the dump file size is greater than 0 bytes before proceeding. +- Restore command (if needed): + ```bash + docker exec -i experiments_postgres pg_restore -U expuser -d experiments_db -c \ + < /home/sam/docker-images/archived/ + ``` + +## Section 1 — Data model + +### Schema change + +Add one nullable JSON column to `Animal`: + +```prisma +model Animal { + id String @id @default(uuid()) + experiment_id String + animal_id_string String + animal_name String + subject_info Json? + csv_analysis_config Json? // <— new + experiment Experiment @relation(fields: [experiment_id], references: [id], onDelete: Cascade) + daily_statuses DailyStatus[] + + @@map("animals") +} +``` + +### Stored shape + +```jsonc +{ + "buckets": [ + { "name": "Success", "type": "note", "notes": ["S", "S1"] }, + { "name": "Failure", "type": "note", "notes": ["F"] }, + { "name": "Other", "type": "note", "notes": [] }, + { "name": "Lick latency", "type": "numerical", "notes": ["L25", "L30"] } + ] +} +``` + +### Rules + +- `type: "note"` — existing behavior (counted, run-length-encoded, contributes to daily-status summary). +- `type: "numerical"` — plotted vs. frame (or timestamp); reports average; **excluded** from category counts, consecutive runs, and the "Save metrics to daily status" calculation. +- The three default note buckets (Success, Failure, Other) are always present: seeded into `buckets` on first save and undeletable in the UI. +- Custom buckets (both note and numerical) can be removed via an `✕` control on the bucket. Removal puts any notes that were assigned to that bucket back into Unassigned for the current session and drops the bucket from `buckets[]` on next save. +- `notes` arrays persist as raw trimmed strings. A new CSV upload looks up each unique note in the persisted `notes` arrays to auto-classify; any leftover notes appear as **Unassigned**. +- A single note value belongs to at most one bucket. Assigning a note to bucket B removes it from any other bucket's `notes[]` first. +- Bucket names are unique within `buckets`. Renaming is out of scope for v1. +- `csv_analysis_config` may be `null` (animal never had CSV analysis run); UI falls back to `DEFAULT_BUCKETS` in that case. + +### Backend API + +- `GET /api/animals/:id` — already returns the full animal row. `csv_analysis_config` is included automatically. +- `PATCH /api/animals/:id/csv-config` — body `{ buckets: [...] }`. Validates: + - `buckets` is an array. + - Each entry has non-empty trimmed `name`, `type ∈ {"note","numerical"}`, `notes` is `string[]`. + - `name` values are unique within the array. + - The three default note buckets (Success/Failure/Other) are present. + Replaces the column atomically. Returns updated animal. Audit-logged via existing middleware. + +## Section 2 — Frontend changes + +### `CsvAnalysis.jsx` (the main refactor) + +1. **New prop `animal`** — passed from `DailyStatusDetail`. Used to read & write `csv_analysis_config`. + +2. **State refactor**: replace the parallel `categories: string[]` + `customCategories: string[]` + `classifications: { [note]: category }` with a single source of truth: + ```js + const [buckets, setBuckets] = useState([]); + // bucket: { name: string, type: 'note' | 'numerical', notes: string[] } + ``` + Derive `classifications`, `categories`, `customCategories` from `buckets` for compatibility with downstream code (`runAnalysis`, color helpers). + +3. **Seeding on entering classify phase**: + - If `animal.csv_analysis_config?.buckets` exists, use it. + - Otherwise initialize from `DEFAULT_BUCKETS` (Success/Failure/Other, all `type: "note"`, empty `notes`). + - Auto-classify: any `uniqueNote` whose value appears in some bucket's `notes[]` is considered already-assigned and does not appear in Unassigned. + +4. **"+ Add field" button** (replaces today's "+ New category"): + - Inline form: `` + radio toggle `( ) Note ( ) Numerical`. + - On submit, appends `{ name, type, notes: [] }` to `buckets`. + +5. **`CategoryBucket` extension**: + - Accepts `type` prop. + - Color allocation: `getCategoryColors` is updated to take the full bucket list and assign `EXTRA_COLORS` round-robin across all custom buckets (note + numerical combined), so adding numerical buckets does not shift the colors of existing note buckets. + - Numerical buckets render with a "numerical" header badge to distinguish them visually. + - Numerical buckets show an "**Add all unassigned**" button that bulk-moves every currently-unassigned unique note into this bucket's `notes[]`. + - Custom buckets (note or numerical) show an `✕` remove control on hover. + - In results phase, numerical buckets show a footer line: `Avg: 24.3 · 47/50 rows plottable`. + +6. **`compute()`**: + - Note-type buckets: pass through `runAnalysis(rows, timestampCol, noteCol, classifications)` as today, but `classifications` is built only from `note`-type buckets (numerical-bucket assignments are excluded so they don't pollute counts/runs). + - Numerical-type buckets: for each, call new helper `computeNumericSeries(rows, frameCol || timestampCol, noteCol, bucket)` returning `{ points: [{x, value, raw}], avg, skipped, total, xLabel }`. + - After successful analysis, call `animalsApi.updateCsvConfig(animal.id, { buckets })`. Failure surfaces a non-blocking toast/inline error; analysis results still render. + +7. **Results phase additions**: + - Existing tables (Category Counts, Consecutive Runs) iterate `buckets.filter(b => b.type === 'note')` — numerical buckets are excluded. + - New "Numerical Plots" section after `RunSequenceView`. One panel per numerical bucket, containing: + - Bucket name, average, skipped-count. + - Recharts `` with `XAxis dataKey="x"` (label = `xLabel`) and a single ``. Color from `getCategoryColors(bucket.name, customCategories)`. + - If a numerical bucket has zero plottable rows, render `"No plottable values — N of N note rows had no extractable number."` instead of a chart. + +### `frontend/src/lib/csvAnalysis.js` + +```js +// Extract first signed decimal from a string. Returns null if none found. +export function extractNumber(str) { + if (str == null) return null; + const m = String(str).match(/-?\d+(\.\d+)?/); + return m ? parseFloat(m[0]) : null; +} + +// Build a numerical series for one bucket. +// xColName: frameCol if set, else timestampCol. xLabel reflects which. +export function computeNumericSeries(rows, xColName, noteCol, bucket) { + const noteSet = new Set(bucket.notes); + const points = []; + let skipped = 0; + let total = 0; + + for (const r of rows) { + const note = String(r[noteCol] ?? '').trim(); + if (!noteSet.has(note)) continue; + total++; + const value = extractNumber(note); + const x = parseFloat(r[xColName]); + if (value == null || Number.isNaN(x)) { skipped++; continue; } + points.push({ x, value, raw: note }); + } + + points.sort((a, b) => a.x - b.x); + const avg = points.length === 0 + ? null + : points.reduce((s, p) => s + p.value, 0) / points.length; + + return { points, avg, skipped, total, xLabel: xColName }; +} +``` + +### `frontend/src/api/client.js` + +Add to `animalsApi`: +```js +updateCsvConfig: (animalId, config) => + request(`/animals/${animalId}/csv-config`, { method: 'PATCH', body: JSON.stringify(config) }), +``` + +### `frontend/src/pages/DailyStatusDetail.jsx` + +Single change: pass `animal={status.animal}` to ``. + +## Section 3 — Behavior details + +- **Frame column** stays optional. If unset when running analysis with ≥1 numerical bucket, x-axis falls back to `timestampCol`. Chart axis label and the panel header reflect which axis is in use. +- **Letter stripping** uses `/-?\d+(\.\d+)?/` (first signed decimal in the string). Notes with no extractable number are counted as `skipped`. +- **Persistence trigger**: config is saved after `compute()` succeeds. Editing buckets in classify phase without running analysis does not persist. +- **Multiple numerical buckets**: fully supported. Each gets its own chart and its own average. +- **Same note in multiple buckets**: prevented at the UI level — assigning a note to a bucket removes it from any other bucket. Backend does not enforce this (UI invariant). + +## Section 4 — Error handling + +| Failure | Behavior | +|-----------------------------------------------|-------------------------------------------------------------------------------------------| +| Save config (PATCH) fails | Local state unchanged; non-blocking inline error in classify/results UI; analysis still renders. | +| Numerical bucket has zero plottable rows | Panel shows "No plottable values — N of N rows had no extractable number." No crash. | +| Frame column absent + numerical bucket exists | Falls back to timestamp; chart axis label reflects this. | +| Migration | Run the Section 0 backup first; abort if dump file size is 0 bytes. | +| Validation rejection on PATCH | UI shows server error message; user can re-edit and retry. | + +## Section 5 — Testing + +### Frontend unit (Vitest, `frontend/src/lib/`) + +- `extractNumber()`: + - `"L25.5"` → 25.5 + - `"R-12"` → -12 + - `"trial_007"` → 7 + - `"3.2e2"` → 3.2 (loses scientific notation — accepted limitation) + - `"42 ms"` → 42 + - `"abc"` → null + - `""` → null + - `null` → null +- `computeNumericSeries()`: + - All rows valid → correct points + average + skipped=0 + - Some unparseable → correct skipped count, points exclude bad rows + - All unparseable → empty points, avg=null, skipped=total + - `xColName` missing from rows → all skipped (NaN x) + - Bucket with empty `notes[]` → empty points, total=0 + - Output sorted ascending by x + +### Backend unit + +- `PATCH /animals/:id/csv-config`: + - Valid body → 200, persisted, audit log row written + - Empty bucket name → 400 + - Duplicate names → 400 + - Invalid `type` value → 400 + - `notes` not an array → 400 + - Missing default bucket (e.g. no "Success") → 400 + +### Integration / manual + +- Upload a CSV → create one numerical bucket → assign notes → run analysis → reload page → upload same CSV → confirm bucket is pre-populated and assignments are restored. +- Same flow with a CSV containing genuinely-new note values → confirm new notes show as Unassigned, old ones do not. +- Numerical bucket with frame col present → chart x-axis labeled with frame col name. +- Numerical bucket with frame col cleared → chart x-axis labeled with timestamp col name. + +## Section 6 — Out of scope (v1) + +- Bucket renaming after creation. +- Y-axis range customization, multi-line overlays. +- Showing the chart for past `DailyAnalysis` records (frame data isn't stored; user re-drops the CSV to plot). +- Per-experiment defaults (config is per-animal only). +- Editing config from a settings page; config is only edited via the CSV-analysis flow.