docs: add design spec for numerical CSV-analysis buckets per animal
Spec for adding a numerical bucket type to CSV analysis that plots note values (numeric portion) against frame/timestamp, persisted on the Animal model so subsequent uploads reuse the configuration. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Experiments DB Dev
@@ -0,0 +1,235 @@
|
||||
# Numerical CSV-analysis buckets, persisted per animal
|
||||
|
||||
**Date:** 2026-05-01
|
||||
**Scope:** `frontend/src/components/CsvAnalysis.jsx`, `frontend/src/lib/csvAnalysis.js`, `frontend/src/api/client.js`, `frontend/src/pages/DailyStatusDetail.jsx`, `backend/prisma/schema.prisma`, backend animal route.
|
||||
|
||||
## Goal
|
||||
|
||||
In the daily-status CSV-analysis flow, let the user define an extra type of bucket — a **numerical** bucket — alongside the existing note-classification buckets (Success / Failure / Other / custom). A numerical bucket plots its assigned notes' numeric portion against the CSV frame column (or timestamp as fallback) and reports an average. The bucket structure and per-bucket note assignments persist on the `Animal` so subsequent CSV uploads for the same subject don't need to be re-classified from scratch.
|
||||
|
||||
## Section 0 — Preconditions
|
||||
|
||||
Before applying the schema migration, back up the live Postgres database to a sibling `archived/` folder.
|
||||
|
||||
```bash
|
||||
mkdir -p /home/sam/docker-images/archived
|
||||
docker exec experiments_postgres pg_dump -U expuser -d experiments_db -F c \
|
||||
> /home/sam/docker-images/archived/experiments_db_$(date +%Y%m%d_%H%M%S).dump
|
||||
```
|
||||
|
||||
- Verify the dump file size is greater than 0 bytes before proceeding.
|
||||
- Restore command (if needed):
|
||||
```bash
|
||||
docker exec -i experiments_postgres pg_restore -U expuser -d experiments_db -c \
|
||||
< /home/sam/docker-images/archived/<dumpfile>
|
||||
```
|
||||
|
||||
## Section 1 — Data model
|
||||
|
||||
### Schema change
|
||||
|
||||
Add one nullable JSON column to `Animal`:
|
||||
|
||||
```prisma
|
||||
model Animal {
|
||||
id String @id @default(uuid())
|
||||
experiment_id String
|
||||
animal_id_string String
|
||||
animal_name String
|
||||
subject_info Json?
|
||||
csv_analysis_config Json? // <— new
|
||||
experiment Experiment @relation(fields: [experiment_id], references: [id], onDelete: Cascade)
|
||||
daily_statuses DailyStatus[]
|
||||
|
||||
@@map("animals")
|
||||
}
|
||||
```
|
||||
|
||||
### Stored shape
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"buckets": [
|
||||
{ "name": "Success", "type": "note", "notes": ["S", "S1"] },
|
||||
{ "name": "Failure", "type": "note", "notes": ["F"] },
|
||||
{ "name": "Other", "type": "note", "notes": [] },
|
||||
{ "name": "Lick latency", "type": "numerical", "notes": ["L25", "L30"] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Rules
|
||||
|
||||
- `type: "note"` — existing behavior (counted, run-length-encoded, contributes to daily-status summary).
|
||||
- `type: "numerical"` — plotted vs. frame (or timestamp); reports average; **excluded** from category counts, consecutive runs, and the "Save metrics to daily status" calculation.
|
||||
- The three default note buckets (Success, Failure, Other) are always present: seeded into `buckets` on first save and undeletable in the UI.
|
||||
- Custom buckets (both note and numerical) can be removed via an `✕` control on the bucket. Removal puts any notes that were assigned to that bucket back into Unassigned for the current session and drops the bucket from `buckets[]` on next save.
|
||||
- `notes` arrays persist as raw trimmed strings. A new CSV upload looks up each unique note in the persisted `notes` arrays to auto-classify; any leftover notes appear as **Unassigned**.
|
||||
- A single note value belongs to at most one bucket. Assigning a note to bucket B removes it from any other bucket's `notes[]` first.
|
||||
- Bucket names are unique within `buckets`. Renaming is out of scope for v1.
|
||||
- `csv_analysis_config` may be `null` (animal never had CSV analysis run); UI falls back to `DEFAULT_BUCKETS` in that case.
|
||||
|
||||
### Backend API
|
||||
|
||||
- `GET /api/animals/:id` — already returns the full animal row. `csv_analysis_config` is included automatically.
|
||||
- `PATCH /api/animals/:id/csv-config` — body `{ buckets: [...] }`. Validates:
|
||||
- `buckets` is an array.
|
||||
- Each entry has non-empty trimmed `name`, `type ∈ {"note","numerical"}`, `notes` is `string[]`.
|
||||
- `name` values are unique within the array.
|
||||
- The three default note buckets (Success/Failure/Other) are present.
|
||||
Replaces the column atomically. Returns updated animal. Audit-logged via existing middleware.
|
||||
|
||||
## Section 2 — Frontend changes
|
||||
|
||||
### `CsvAnalysis.jsx` (the main refactor)
|
||||
|
||||
1. **New prop `animal`** — passed from `DailyStatusDetail`. Used to read & write `csv_analysis_config`.
|
||||
|
||||
2. **State refactor**: replace the parallel `categories: string[]` + `customCategories: string[]` + `classifications: { [note]: category }` with a single source of truth:
|
||||
```js
|
||||
const [buckets, setBuckets] = useState([]);
|
||||
// bucket: { name: string, type: 'note' | 'numerical', notes: string[] }
|
||||
```
|
||||
Derive `classifications`, `categories`, `customCategories` from `buckets` for compatibility with downstream code (`runAnalysis`, color helpers).
|
||||
|
||||
3. **Seeding on entering classify phase**:
|
||||
- If `animal.csv_analysis_config?.buckets` exists, use it.
|
||||
- Otherwise initialize from `DEFAULT_BUCKETS` (Success/Failure/Other, all `type: "note"`, empty `notes`).
|
||||
- Auto-classify: any `uniqueNote` whose value appears in some bucket's `notes[]` is considered already-assigned and does not appear in Unassigned.
|
||||
|
||||
4. **"+ Add field" button** (replaces today's "+ New category"):
|
||||
- Inline form: `<input name>` + radio toggle `( ) Note ( ) Numerical`.
|
||||
- On submit, appends `{ name, type, notes: [] }` to `buckets`.
|
||||
|
||||
5. **`CategoryBucket` extension**:
|
||||
- Accepts `type` prop.
|
||||
- Color allocation: `getCategoryColors` is updated to take the full bucket list and assign `EXTRA_COLORS` round-robin across all custom buckets (note + numerical combined), so adding numerical buckets does not shift the colors of existing note buckets.
|
||||
- Numerical buckets render with a "numerical" header badge to distinguish them visually.
|
||||
- Numerical buckets show an "**Add all unassigned**" button that bulk-moves every currently-unassigned unique note into this bucket's `notes[]`.
|
||||
- Custom buckets (note or numerical) show an `✕` remove control on hover.
|
||||
- In results phase, numerical buckets show a footer line: `Avg: 24.3 · 47/50 rows plottable`.
|
||||
|
||||
6. **`compute()`**:
|
||||
- Note-type buckets: pass through `runAnalysis(rows, timestampCol, noteCol, classifications)` as today, but `classifications` is built only from `note`-type buckets (numerical-bucket assignments are excluded so they don't pollute counts/runs).
|
||||
- Numerical-type buckets: for each, call new helper `computeNumericSeries(rows, frameCol || timestampCol, noteCol, bucket)` returning `{ points: [{x, value, raw}], avg, skipped, total, xLabel }`.
|
||||
- After successful analysis, call `animalsApi.updateCsvConfig(animal.id, { buckets })`. Failure surfaces a non-blocking toast/inline error; analysis results still render.
|
||||
|
||||
7. **Results phase additions**:
|
||||
- Existing tables (Category Counts, Consecutive Runs) iterate `buckets.filter(b => b.type === 'note')` — numerical buckets are excluded.
|
||||
- New "Numerical Plots" section after `RunSequenceView`. One panel per numerical bucket, containing:
|
||||
- Bucket name, average, skipped-count.
|
||||
- Recharts `<LineChart>` with `XAxis dataKey="x"` (label = `xLabel`) and a single `<Line dataKey="value">`. Color from `getCategoryColors(bucket.name, customCategories)`.
|
||||
- If a numerical bucket has zero plottable rows, render `"No plottable values — N of N note rows had no extractable number."` instead of a chart.
|
||||
|
||||
### `frontend/src/lib/csvAnalysis.js`
|
||||
|
||||
```js
|
||||
// Extract first signed decimal from a string. Returns null if none found.
|
||||
export function extractNumber(str) {
|
||||
if (str == null) return null;
|
||||
const m = String(str).match(/-?\d+(\.\d+)?/);
|
||||
return m ? parseFloat(m[0]) : null;
|
||||
}
|
||||
|
||||
// Build a numerical series for one bucket.
|
||||
// xColName: frameCol if set, else timestampCol. xLabel reflects which.
|
||||
export function computeNumericSeries(rows, xColName, noteCol, bucket) {
|
||||
const noteSet = new Set(bucket.notes);
|
||||
const points = [];
|
||||
let skipped = 0;
|
||||
let total = 0;
|
||||
|
||||
for (const r of rows) {
|
||||
const note = String(r[noteCol] ?? '').trim();
|
||||
if (!noteSet.has(note)) continue;
|
||||
total++;
|
||||
const value = extractNumber(note);
|
||||
const x = parseFloat(r[xColName]);
|
||||
if (value == null || Number.isNaN(x)) { skipped++; continue; }
|
||||
points.push({ x, value, raw: note });
|
||||
}
|
||||
|
||||
points.sort((a, b) => a.x - b.x);
|
||||
const avg = points.length === 0
|
||||
? null
|
||||
: points.reduce((s, p) => s + p.value, 0) / points.length;
|
||||
|
||||
return { points, avg, skipped, total, xLabel: xColName };
|
||||
}
|
||||
```
|
||||
|
||||
### `frontend/src/api/client.js`
|
||||
|
||||
Add to `animalsApi`:
|
||||
```js
|
||||
updateCsvConfig: (animalId, config) =>
|
||||
request(`/animals/${animalId}/csv-config`, { method: 'PATCH', body: JSON.stringify(config) }),
|
||||
```
|
||||
|
||||
### `frontend/src/pages/DailyStatusDetail.jsx`
|
||||
|
||||
Single change: pass `animal={status.animal}` to `<CsvAnalysis>`.
|
||||
|
||||
## Section 3 — Behavior details
|
||||
|
||||
- **Frame column** stays optional. If unset when running analysis with ≥1 numerical bucket, x-axis falls back to `timestampCol`. Chart axis label and the panel header reflect which axis is in use.
|
||||
- **Letter stripping** uses `/-?\d+(\.\d+)?/` (first signed decimal in the string). Notes with no extractable number are counted as `skipped`.
|
||||
- **Persistence trigger**: config is saved after `compute()` succeeds. Editing buckets in classify phase without running analysis does not persist.
|
||||
- **Multiple numerical buckets**: fully supported. Each gets its own chart and its own average.
|
||||
- **Same note in multiple buckets**: prevented at the UI level — assigning a note to a bucket removes it from any other bucket. Backend does not enforce this (UI invariant).
|
||||
|
||||
## Section 4 — Error handling
|
||||
|
||||
| Failure | Behavior |
|
||||
|-----------------------------------------------|-------------------------------------------------------------------------------------------|
|
||||
| Save config (PATCH) fails | Local state unchanged; non-blocking inline error in classify/results UI; analysis still renders. |
|
||||
| Numerical bucket has zero plottable rows | Panel shows "No plottable values — N of N rows had no extractable number." No crash. |
|
||||
| Frame column absent + numerical bucket exists | Falls back to timestamp; chart axis label reflects this. |
|
||||
| Migration | Run the Section 0 backup first; abort if dump file size is 0 bytes. |
|
||||
| Validation rejection on PATCH | UI shows server error message; user can re-edit and retry. |
|
||||
|
||||
## Section 5 — Testing
|
||||
|
||||
### Frontend unit (Vitest, `frontend/src/lib/`)
|
||||
|
||||
- `extractNumber()`:
|
||||
- `"L25.5"` → 25.5
|
||||
- `"R-12"` → -12
|
||||
- `"trial_007"` → 7
|
||||
- `"3.2e2"` → 3.2 (loses scientific notation — accepted limitation)
|
||||
- `"42 ms"` → 42
|
||||
- `"abc"` → null
|
||||
- `""` → null
|
||||
- `null` → null
|
||||
- `computeNumericSeries()`:
|
||||
- All rows valid → correct points + average + skipped=0
|
||||
- Some unparseable → correct skipped count, points exclude bad rows
|
||||
- All unparseable → empty points, avg=null, skipped=total
|
||||
- `xColName` missing from rows → all skipped (NaN x)
|
||||
- Bucket with empty `notes[]` → empty points, total=0
|
||||
- Output sorted ascending by x
|
||||
|
||||
### Backend unit
|
||||
|
||||
- `PATCH /animals/:id/csv-config`:
|
||||
- Valid body → 200, persisted, audit log row written
|
||||
- Empty bucket name → 400
|
||||
- Duplicate names → 400
|
||||
- Invalid `type` value → 400
|
||||
- `notes` not an array → 400
|
||||
- Missing default bucket (e.g. no "Success") → 400
|
||||
|
||||
### Integration / manual
|
||||
|
||||
- Upload a CSV → create one numerical bucket → assign notes → run analysis → reload page → upload same CSV → confirm bucket is pre-populated and assignments are restored.
|
||||
- Same flow with a CSV containing genuinely-new note values → confirm new notes show as Unassigned, old ones do not.
|
||||
- Numerical bucket with frame col present → chart x-axis labeled with frame col name.
|
||||
- Numerical bucket with frame col cleared → chart x-axis labeled with timestamp col name.
|
||||
|
||||
## Section 6 — Out of scope (v1)
|
||||
|
||||
- Bucket renaming after creation.
|
||||
- Y-axis range customization, multi-line overlays.
|
||||
- Showing the chart for past `DailyAnalysis` records (frame data isn't stored; user re-drops the CSV to plot).
|
||||
- Per-experiment defaults (config is per-animal only).
|
||||
- Editing config from a settings page; config is only edited via the CSV-analysis flow.
|
||||
Reference in New Issue
Block a user