Skip to content

Interaction Layer – Runtime & MQTT

This page explains how the interaction layer runs at runtime: from microphone input, through the dialogue manager and multi-agent graph, to MQTT messages that drive speech and animations.

Runtime Entrypoint (nadine/__main__.py)

python -m nadine launches the interaction stack.

Initialization

Nadine.__init__(nomqtt: bool):

  • Loads environment variables from interaction/.env.
  • Reads the current language from language_config (default: French).
  • Creates:
  • logger via LoggersFactory.getLogger().
  • Translation for multi-language translations.
  • UI window (status, user text, agent reply).
  • STTManager for microphone + Google STT.
  • MQTTCommunication for all interaction-layer MQTT I/O.
  • DialogueManager for multi-agent conversation logic.

Modes

__main__.py supports:

  • Default mode: voice + UI + MQTT.
  • nadine = Nadine(nomqtt=False)
  • nadine.start_all()
  • --nomqtt:
  • Runs without MQTT (e.g. for offline experimentation).
  • --chatmode:
  • Text-only mode:
    • Creates DialogueManager() directly.
    • Simple REPL in the console:
    • User types a message, DM returns text.

Voice Interaction Flow

In normal mode each user utterance passes through three phases: listen, process, respond.

Listen

STT runs continuously and hands finished utterances to Nadine.

  1. Nadine.start_all():
  2. Starts STT listening.
  3. Sets initial UI status (microphone, user availability, etc.).

  4. Starts the UI event loop.

  5. User speaks:

  6. STTManager converts audio to text and calls Nadine.user_speech_detected(text).

Process

user_speech_detected gates the microphone and hands the text to the dialogue manager.

  1. user_speech_detected:
  2. Suspends STT while Nadine is speaking (when MQTT is enabled).
  3. Translates user input to English if current language is not English.
  4. Calls DialogueManager.processInput(text_en).

Respond

The reply is shown, spoken, and the microphone is re-armed.

  1. Updates the UI (user input + agent output).
  2. Calls mqtt_comm.speak(reply_en, self.language) to trigger speech and animation.
  3. After response:
  4. STT is re-activated.
  5. UI status is updated again to reflect current listening/speaking state.

This loop repeats for each user utterance.

DialogueManager Runtime (dm.py)

DialogueManager.processInput(user_input: str) orchestrates the main interaction logic.

Initialization (__init__)

  • Creates:
  • mqtt_comm – shared MQTTCommunication instance.
  • logger – shared logger.
  • chat_history – list of LangChain HumanMessage/AIMessage.
  • user_id – unique ID via generate_unique_user_id().
  • user_info – default profile dict via user_info_init(user_id).
  • c_state – current graph state (custom state dict).
  • multi_agent_graph – compiled LangGraph from build_agent_graph().
  • conversation_limit – how many recent messages to keep in chat_history (4, i.e. two user–robot exchanges).
  • Does not warm up models itself: the Ollama models are pre-warmed by start_nadine.sh before the interaction process starts (warmup_llms in utils.py is an unused helper).

Prepare the turn

Before the graph runs, DM syncs the user, answers trivial inputs directly, and resolves any pending name confirmation.

State refresh & name confirmation

Before invoking the graph, DM:

  • Checks face-recognition info via mqtt_comm.get_detected_user_info().
  • _refresh_state(detected_user_id):
  • If a different user was detected:
    • Saves an episodic memory for the outgoing user (_save_episodic_on_switch, skipped for unknown users).
    • Loads the new user's user_info.json from the interaction DB.
    • Clears chat_history and resets c_state, so greeting flags, affect, and memories are per user.
    • Resets the language to English through set_language_callback.
  • Updates c_state via default_custom_state(c_state, chat_history, user_info).

Pre-graph short-circuits

Before the graph runs, _short_circuit_reply answers two kinds of input directly and appends them to the history:

  • A request for a joke returns a random entry from french_kids_jokes.json.
  • "stop talking" and similar phrases return "Okay. I will stop talking."

Independently, should_trigger_handshake matches phrases such as "shake hands" and calls mqtt_comm.give_handshake() before the graph runs.

If the memory agents previously requested name confirmation, name_confirmation(user_input):

  • Handles:
  • Confirming a suggested existing user.
  • Creating a new user from a name explicitly mentioned by the user.
  • Rotating through remaining candidate names if needed.
  • Synchronizes user info back to face recognition via MQTT.

Run the graph

One invoke call runs the whole multi-agent graph on the prepared state.

Graph invocation

After state prep:

  • results = multi_agent_graph.invoke(self.c_state)

The result includes:

  • Updated user_info
  • Updated affect state
  • Updated memories
  • final_message – raw response text/JSON
  • Optional name_confirmation payload
  • intent – classified intent (e.g., first_greeting, end_conversation)

Respond

DM adopts the new state, extracts the reply, and triggers any motion tied to the intent.

Result handling

The DM:

  • Optionally calls set_language_callback if results["language"] changed.
  • Adopts self.c_state = results.
  • Ensures user_info and user_id in DM align with graph output.
  • Extracts:
  • Final text + emotion via _extract_robot_response(results).
  • Updates chat_history with the new AI message.
  • Resets chat history on end_conversation and trims to conversation_limit messages. After the reply is spoken, __main__.py also resets the language to French on end_conversation.

Motion side-effects

For certain intents (first_greeting, end_conversation), DM asks the control layer to wave:

  • self.mqtt_comm.give_wave() → publishes a nadine/agent/control/animation command.

MQTT Topics (Interaction Perspective)

The interaction layer uses MQTTCommunication as its main MQTT client.

Subscribed

  • nadine/graph/user_detected
  • From perception layer.
  • Payload: {"user_name": str, "user_id": str, "confidence": float}.

  • Used to track which user is currently in front of Nadine.

  • nadine/graph/face_stored

  • From perception layer.
  • Payload: {"user_name": str, "user_id": str, "status": "face_stored" | "face_stored_new_user"}.

  • Used to update interaction/db/user_ids.json when new users are added.

  • nadine/agent/feedback/start_speak, nadine/agent/feedback/end_speak

  • From control layer.
  • Indicate when Nadine starts/finishes speaking.

Published

  • nadine/face_recognition/user_info
  • To perception layer.
  • Payload: {"user_name": ..., "user_id": ...}.

  • Used when memory agents or DM finalize a user’s name → triggers face storage.

  • nadine/perception/capture_current_view

  • To perception layer.
  • Payload: image path string.

  • Used to request a snapshot for vision/visual memory.

  • nadine/agent/control/speak

  • To control layer.
  • Payload: final response text (in the appropriate language).

  • Triggers TTS + lip-sync + associated animations.

  • nadine/agent/control/animation

  • To control layer.
  • Payload: animation name.

  • Used by helper methods like give_wave, give_greeting, give_smile, give_handshake, etc.

  • nadine/agent/control/look_at_target

  • To control layer.
  • Payload: posture name (Posture_LookAtInterviewer, Posture_LookAtZoom, or LOOKUPPostureDefault to clear the target).

  • Sent by the UI's "Gaze Direction" radio buttons via mqtt_comm.look_at_target.

  • nadine/affect/state

  • To perception layer.
  • Payload: {"label": str, "arousal": float, "intensity": float}.
  • Published by the affective appraisal node after every appraisal; perception's selective memory uses it to decide whether the current scene is memorable.

Quick Dev Tips

  • To debug the LangGraph flow in isolation, run graph.py directly; its async main() REPL (started with asyncio.run) drives the graph with its own simplified name-confirmation handling.
  • To test the dialogue manager without STT/UI, you can either:

  • Run the built‑in chat mode:

    cd interaction
conda activate nadine
python -m nadine --chatmode

  • Or call the DialogueManager directly from a small script:

    from nadine.agents.dm import DialogueManager

dm = DialogueManager()
while True:
    text = input("You: ")
    if text.lower() == "exit":
        break
    reply = dm.processInput(text)
    print("Nadine:", reply)

  • Use the Agents & Graph and Memory & RAG pages to understand how state fields like user_info, conversation_memory, episode_memory, and visual_memory are set and used.