Interaction Layer – Multimodal Memory (nadine_phd)¶

This page documents the observation-scene memory of the Hybrid Cloud version (nadine_phd). It extends the visual memory of the local platform described in Interaction Layer – Memory & RAG and Perception Layer – Selective Memory; read those first. Everything below describes nadine_phd unless stated otherwise.

Overview¶

A scene is one captured moment of a conversation with a known user. It consists of:

the camera frame at capture time (images/<scene_id>.jpg),
a CLIP image embedding (embeddings/<scene_id>_embedding.npy, omitted for observation scenes),
a short caption generated by Moondream2 (description),
the user's utterance in that moment (user_message), which is the text the scene is retrieved by,
a scene_type (moment, first_meeting, or observation), the emotion signal that triggered the store, and a timestamp.

The local platform stores scenes on a perception-side timer and retrieves them by matching the user's question against the caption and the image. The Hybrid Cloud version anchors every scene to what the user said when it was captured, stores exactly one scene per qualifying dialogue turn, retrieves on the utterance alone, and only surfaces scenes when the user is asking to recall the past.

Differences from nadine_local¶

Aspect	nadine_local	nadine_phd
Storage trigger	Perception polls every `memorability_check_interval` s with a 15 s cooldown	One evaluation per dialogue turn, on each new `nadine/affect/state` message (`NADINE_STORAGE_TRIGGER=event`)
Storage signal	Nadine's own appraised emotion, `base(label) × intensity` (`pad_arousal`)	The user's expressed-emotion intensity from the dialogue, no per-emotion weights (`intensity` policy)
Retrieval anchor	Caption and image	The user's utterance at capture time; caption only as fallback for legacy scenes
Description embeddings	CLIP text encoder	OpenAI `text-embedding-3-large`, CLIP (mean-centered) as fallback
Image channel	`retrieval_alpha: 0.3`	`retrieval_alpha: 0.0` (CLIP image embeddings are stored but not used for retrieval)
Thresholds	One `similarity_threshold`	Strict vs. recall thresholds, plus a stricter bar for first-meeting scenes
Recall detection	Regex on the message	`is_recall` judged by the turn-understanding LLM, regex kept as fallback
Scene types	`moment` only	`moment`, `first_meeting` (stored at first face registration), `observation` (user-directed vision moments)
Text memory injection	Always	Only on recall turns
Study support	None	`NADINE_ENABLE_VISUAL_MEMORY` A/B switch, `StudyLogger`, FER sampler

Storage¶

Who stores what¶

Scenes are written by two processes into the same per-user folder, interaction/db/memory/user_profiles/<user_id>/memorable_scenes/:

Scene type	Written by	Trigger
`moment`	Perception (`IntensityMemoryModule.store_memorable_scene`)	The user's expressed-emotion intensity for the current turn reaches `intensity_threshold`
`first_meeting`	Perception (`store_first_meeting_scene`)	The first face registration for a new user
`observation`	Interaction (`observation_memory.store_observation_scene`)	The vision agent answered a question about something specific the user pointed at

All storage requires an identified user. Unknown users are never stored, in either process.

Moment scenes: the per-turn evaluation¶

The merged understand node (see Cloud Configuration) returns, in the same structured call as intent and plan, the user's expressed emotion (user_emotion.label, user_emotion.intensity in 0–1) and an is_recall flag.
The interaction layer publishes nadine/affect/state with Nadine's appraised emotion and, in addition to what nadine_local sends, user_label, user_intensity, user_message (the current utterance, up to 500 characters), and suppress_storage.
Perception marks a new affect event. On the next frame in which the user's face is tracked, it evaluates storage once and consumes the event, so there is at most one scene per turn and no time-based cooldown.
memorability = intensity of the selected signal. NADINE_STORAGE_EMOTION_SOURCE=user (default) uses the user's expressed emotion; robot reverts to Nadine's own appraised intensity.
The scene is stored when memorability ≥ intensity_threshold (perception config.yaml: 0.5; code fallback: 0.5, reading the legacy key arousal_threshold if present) and suppress_storage is false.
Perception generates the Moondream2 caption, writes image, embedding, and metadata, and publishes nadine/memory/scene_stored.

suppress_storage is true on recall turns (remembering is not a new moment) and on end_conversation turns (a polite "thanks, bye" can be appraised as mildly positive and otherwise cross the threshold). Greeting turns are not suppressed, because a participant may bundle a greeting with a real moment in one utterance; a plain greeting carries low intensity and fails the threshold on its own.

The legacy frame-polled behavior remains available with NADINE_STORAGE_TRIGGER=interval, which restores the check interval and the 15 s cooldown for A/B comparison. The interval trigger always applies to the vision policy, which is unchanged from nadine_local.

First-meeting scenes¶

When perception stores the first face image for a newly registered user, it also captures the current frame as a first_meeting scene with memorability: 1.0, a neutral emotion, and a canonical anchor instead of the user's utterance:

The first time we met — <user_name> introduced themselves to me.

The canonical anchor replaced the greeting utterance because a greeting anchor matched every later "hi". First-meeting scenes are only retrievable on recall turns and have their own, stricter threshold (below).

Observation scenes¶

When the plan routes to the vision agent and the vision model answers, graph._maybe_store_observation files the snapshot as an observation scene if all of the following hold:

visual memory is enabled and the user is identified,
the turn is not a recall turn,
the utterance points at something specific: it matches my, mine, me, this, that, these, those, look at, or watch me/this,
the vision answer is non-empty and does not start with "Sorry".

Generic "what do you see?" descriptions are deliberately not stored. The description of an observation scene is the vision model's actual answer rather than a Moondream2 caption, user_message is the utterance, and no CLIP embedding is written because retrieval does not use the image channel.

On-disk layout¶

interaction/db/memory/user_profiles/<user_id>/memorable_scenes/
├── images/<scene_id>.jpg
├── embeddings/<scene_id>_embedding.npy      # moment and first_meeting only
└── metadata/<scene_id>_metadata.json

scene_id is scene_<YYYYMMDD_HHMMSS>_<6 hex chars>; the random suffix avoids collisions within one second. Metadata fields: scene_id, timestamp (ISO), image_path, memorability, emotion_label, intensity, description, user_message, scene_type, reason.

The user profiles directory is tracked in the nadine_phd repository.

Retrieval and Recall¶

Retrieval runs in the memory_retrieve_agent node for a known user, after understand and before routing, so a retrieved scene is available to every downstream agent. It is skipped entirely when NADINE_ENABLE_VISUAL_MEMORY=0; textual memory is unaffected.

Query and candidates¶

The query is the latest user message.
Every scene with a non-empty description is a candidate, except that first_meeting scenes are only considered on recall turns.
The retrieval text of a scene is its user_message; episode_context is accepted for legacy scenes, and the caption is used only when neither exists. The caption was dropped from the retrieval text because Moondream2 captions are near-identical across the homogeneous lab scenes and diluted the utterance signal.

Similarity¶

combined = alpha × sim_image + (1 − alpha) × sim_desc

sim_desc: cosine similarity between the query and the retrieval text, embedded with OpenAI text-embedding-3-large (NADINE_DESC_EMBED_MODEL). If the OpenAI call fails, both texts are embedded with CLIP's text encoder after subtracting the mean CLIP embedding of a fixed generic corpus; raw CLIP text cosine sits in a narrow band for any pair of texts and cannot be thresholded.
sim_image: cosine similarity between the CLIP text embedding of the query and the stored CLIP image embedding. It is only computed when retrieval_alpha > 0.
retrieval_alpha: config.yaml: 0.0 (code fallback: 0.3). With 0.0 the image channel is off and CLIP is not loaded in the interaction process.

Thresholds¶

Which threshold applies depends on whether the turn is a recall turn.

Situation	Key	config.yaml	Code fallback
Ordinary turn	`similarity_threshold`	0.35	0.30
Recall turn	`recall_threshold`	0.13	0.18
Recall turn, `first_meeting` scene	`first_meeting_recall_threshold`	0.32	0.32

On ordinary turns the strict threshold means a scene is only injected on a strong content match, so scenes are never volunteered into small talk. The config comment records why it moved from 0.30 to 0.35: a mood-adjacent turn ("I'm having a great day today!") matched a "sad because it's too hot" scene at 0.325, while genuine content matches measure 0.38 and above with the OpenAI embeddings.
On recall turns the permissive threshold lets a real recall request surface the best scene even when the absolute score is modest.
First-meeting scenes use max(recall_threshold, first_meeting_recall_threshold). Their canonical anchor scores 0.12–0.28 against almost any recall phrasing, so under the permissive threshold it attached to nearly every recall turn; genuine first-meeting questions score 0.38–0.56, and 0.32 sits in the gap.

The best candidate above its threshold is returned as state["visual_memory"] with image_path, scene_id, similarity, description, user_message, episode_context, and timestamp.

The recall flag¶

state["is_recall"] is set by the understand node: the LLM judges whether the user is asking Nadine to remember anything from the past, in any phrasing, and the _VISUAL_RECALL_CUE regex from nadine_local is kept as a fallback for misses. The flag gates three things:

the permissive visual threshold and eligibility of first-meeting scenes,
injection of episodic and conversation memory into the response prompt,
suppression of scene storage on that turn.

Episodes and scenes¶

Episodic memory is unchanged in structure (a summary saved to ChromaDB at end_conversation or on a user switch), but the episodic save now summarizes the full session transcript kept in dm.py rather than the trailing four-message window. Scenes are no longer paired with episodes at save time: an earlier version stamped each scene with the episode's observation as episode_context, but that depended on the conversation ending cleanly and was replaced by the immediate utterance anchor. episode_context is still read for scenes created before that change.

How the response agent surfaces a scene¶

When visual_memory carries an image and a description, the response agent adds one line to the background block of the prompt:

A scene you remember from a past meeting with this person (on <timestamp>): <description>
(what was happening then: <user_message>) — this is a remembered PAST moment, not what
you currently see; never describe it as the person's present appearance.

and attaches the image as a high-detail image_url content item. The explicit past-tense instruction was added because the model sometimes narrated the stored photo in present tense. The system prompt tells the model to recall an attached scene naturally as a shared moment, never to say "I see an image", and never to invent visual details when no scene is attached.

On recall turns the episode summary and the retrieved conversation transcript are injected as well. When both Nadine's own knowledge and shared memories are present, an extra directive asks for her own facts first and the shared memory only if it relates.

Runtime, Flags, and MQTT¶

Topics changed relative to nadine_local¶

Topic	Direction	Change
`nadine/affect/state`	interaction → perception	Adds `user_label`, `user_intensity`, `user_message`, `suppress_storage` to `label`, `arousal`, `intensity`
`nadine/memory/scene_stored`	perception → (no subscriber)	Adds `user_message`; first-meeting stores add `scene_type: "first_meeting"`

Observation scenes are stored in-process by the interaction layer and publish nothing.

Environment variables¶

Set by start_nadine.sh for the perception and interaction processes.

Variable	Default	Effect
`NADINE_ENABLE_VISUAL_MEMORY`	`1`	`0` disables capture (the selective memory module is not loaded), retrieval, and the visual part of the response prompt
`NADINE_STORAGE_TRIGGER`	`event`	`interval` restores frame-polled storage with the cooldown
`NADINE_STORAGE_EMOTION_SOURCE`	`user`	`robot` stores on Nadine's appraised intensity instead of the user's
`NADINE_DESC_EMBED_MODEL`	`text-embedding-3-large`	OpenAI embedding model for the description channel
`NADINE_STUDY_LOG`	`0`	`1` enables `StudyLogger`
`NADINE_STUDY_CONDITION`	unset	`A` or `B`; defaults to `A` when visual memory is on, `B` otherwise
`NADINE_PARTICIPANT_ID`	`anon`	Participant label in study logs
`NADINE_STUDY_AUDIO_TIMEOUT`	`10`	Seconds to wait for audio onset before writing a turn record without it
`NADINE_ENABLE_FER_SAMPLING`	follows `--study`	Continuous facial-emotion sampling in perception
`NADINE_FER_INTERVAL`	`1.5`	Seconds between FER samples per user
`NADINE_FER_NOVELTY`	follows FER sampling	Also log CLIP novelty per sample
`NADINE_FER_LOG`	`interaction/db/study_logs/fer_samples.jsonl`	FER output file

Start script flags¶

--no-visual-memory sets NADINE_ENABLE_VISUAL_MEMORY=0 for both processes (study condition B).
--study sets NADINE_STUDY_LOG=1 and, unless overridden, turns FER sampling and FER novelty on.

Perception's config.yaml selects the policy with selective_memory.policy: "intensity"; "pad_arousal" is accepted as a legacy alias for the same module.

Evaluation Setup¶

The code contains the infrastructure for a two-condition study; no results are described here.

Conditions¶

Condition A: visual memory on. Scenes are captured, retrieved, and attached to the response prompt.
Condition B: visual memory off. No capture, no retrieval. Textual memory (profile, episodes, conversation) is identical in both conditions.

The two conditions differ in exactly one block of the response system prompt. In A the block explains how to recall an attached scene; in B it states that Nadine has no visual memories of past meetings and must not invent visual details, while she can still recall what was talked about. Everything else in the prompt is the same.

Study logging¶

With NADINE_STUDY_LOG=1, StudyLogger (common/study_logger.py) writes one JSONL file per session under interaction/db/study_logs/, named <condition>_<participant>_<start time>.jsonl, plus a matching .log copy of the human-readable interaction log and per-user transcripts under by_user/. Each turn record contains the user and robot text, intent, language, user ID, the user's expressed emotion and intensity, Nadine's appraised emotion and intensity, a summary of what visual retrieval returned (injected, scene_id, similarity, description), a heuristic flag for whether the reply referenced the scene, and three latency fields:

Field	Measured from	Measured to
`latency_s`	Transcript received by the dialogue manager	Final response text ready
`latency_first_sentence_s`	Transcript received	First sentence handed to TTS
`full_latency_s`	End of user speech	Robot audio onset, reported by control on `start_speak`

A turn record is written at the later of graph completion and audio onset, with a timeout fallback so a turn is never lost if audio never starts.

Facial-emotion sampling¶

perception/fer_sampler.py runs DeepFace's emotion model on the perception frames at a fixed interval per user and appends one JSON line per sample (dominant_emotion, confidence, per-emotion scores, optional CLIP novelty). It is decoupled from the live pipeline and only writes a file; the samples are aligned to study-log turns offline by timestamp. It exists to give the facial storage signal a measured baseline against the dialogue-based signal used in deployment. DeepFace runs on CPU because the rig's TensorFlow GPU path does not work.