notebooks/interpreter_prompting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding HybridAGI prompting\n",
    "\n",
    "In this notebook we are going to show you how HybridAGI build the prompts for the LLM, that way you can understand how exactly the Agent use tools. To do so, we are going to make the simplest agent possible and inspect the LLM calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/yoan/.cache/pypoetry/virtualenvs/hybridagi-B1GoJrSC-py3.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Given the fields `objective`, `context`, `purpose`, `prompt`, produce the fields `message`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Objective: The long-term objective (what you are doing)\n",
      "\n",
      "Context: The previous actions (what you have done)\n",
      "\n",
      "Purpose: The purpose of the action (what you have to do now)\n",
      "\n",
      "Prompt: The action specific instructions (How to do it)\n",
      "\n",
      "Message: The message to send to the user\n",
      "\n",
      "---\n",
      "\n",
      "Objective: What is the city of the spatial and robotics industry in France?\n",
      "\n",
      "Context:\n",
      "--- Step 0 ---\n",
      "Call Program: main\n",
      "Program Purpose: What is the city of the spatial and robotics industry in France?\n",
      "--- END OF TRACE ---\n",
      "\n",
      "Purpose: Answer the user question\n",
      "\n",
      "Prompt: Please answer to the Objective's question\n",
      "\n",
      "Message:\n"
     ]
    }
   ],
   "source": [
    "import dspy\n",
    "import hybridagi.core.graph_program as gp\n",
    "from hybridagi.memory.integration.local import LocalProgramMemory\n",
    "from hybridagi.core.datatypes import AgentState, Query\n",
    "from hybridagi.modules.agents import GraphInterpreterAgent\n",
    "from hybridagi.modules.agents.tools import SpeakTool, PredictTool\n",
    "\n",
    "main = gp.GraphProgram(\n",
    "\tname = \"main\",\n",
    "\tdescription = \"The main program\",\n",
    ")\n",
    "\n",
    "main.add(gp.Action(\n",
    "    id = \"answer\",\n",
    "\ttool = \"Speak\",\n",
    "\tpurpose = \"Answer the user question\",\n",
    "\tprompt = \"Please answer to the Objective's question\",\n",
    "))\n",
    "\n",
    "main.connect(\"start\", \"answer\")\n",
    "main.connect(\"answer\", \"end\")\n",
    "\n",
    "# We build the program to verify its structure\n",
    "main.build()\n",
    "\n",
    "program_memory = LocalProgramMemory(index_name=\"prompting\")\n",
    "\n",
    "program_memory.update(main)\n",
    "\n",
    "agent_state = AgentState()\n",
    "\n",
    "tools = [\n",
    "    SpeakTool(\n",
    "        agent_state = agent_state\n",
    "    ),\n",
    "    PredictTool(),\n",
    "]\n",
    "\n",
    "agent = GraphInterpreterAgent(\n",
    "    program_memory = program_memory,\n",
    "    agent_state = agent_state,\n",
    "    tools = tools,\n",
    "    entrypoint = \"main\", # The interpreter entrypoint (default 'main')\n",
    "    num_history = 5, # The number of steps to include in the agent's context (default 5)\n",
    "    verbose = False, # Weither or not to display the colorful traces (default True)\n",
    ")\n",
    "\n",
    "# We can now setup the LLM using Ollama client from DSPy\n",
    "\n",
    "lm = dspy.OllamaLocal(model='mistral', max_tokens=1024, stop=[\"\\n\\n\\n\"])\n",
    "dspy.configure(lm=lm)\n",
    "\n",
    "# And call our agent\n",
    "\n",
    "result = agent(Query(query=\"What is the city of the spatial and robotics industry in France?\"))\n",
    "\n",
    "print(lm.history[-1][\"prompt\"])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The city of the spatial and robotics industry in France is Toulouse. This is because Toulouse hosts several major aerospace companies such as Airbus, Thales Alenia Space, and the European Space Agency (ESA). Additionally, it has a strong robotics community with institutions like the Institute for Research in Computer Science, Robotics, and Automation of Grenoble (INRIA) and the National Institute of Applied Sciences (INSA) Toulouse.\n"
     ]
    }
   ],
   "source": [
    "print(result.final_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Step 0 ---\n",
      "Call Program: main\n",
      "Program Purpose: What is the city of the spatial and robotics industry in France?\n",
      "--- Step 1 ---\n",
      "Action Purpose: Answer the user question\n",
      "Action: {\n",
      "  \"message\": \"The city of the spatial and robotics industry in France is Toulouse. This is because Toulouse hosts several major aerospace companies such as Airbus, Thales Alenia Space, and the European Space Agency (ESA). Additionally, it has a strong robotics community with institutions like the Institute for Research in Computer Science, Robotics, and Automation of Grenoble (INRIA) and the National Institute of Applied Sciences (INSA) Toulouse.\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# This is the last (and final) answer of the Agent we can also inspect the resulting trace like that\n",
    "print(\"\\n\".join(result.program_trace.steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Agent Context\n",
    "\n",
    "The context always contains the last `N` steps (the param `num_history` of the agent) of the program trace. At each step, the long-term objective of the program and the short-term purpose of the step to help the system behave, note that the Action's prompt is not present in the resulting step of the trace, this decision have been made to isolate the prompts and give the possibility to use multiple \"roles\" in the same program seamlessly thus avoiding prompt contamination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Given the fields `objective`, `context`, `purpose`, `prompt`, produce the fields `message`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Objective: The long-term objective (what you are doing)\n",
      "\n",
      "Context: The previous actions (what you have done)\n",
      "\n",
      "Purpose: The purpose of the action (what you have to do now)\n",
      "\n",
      "Prompt: The action specific instructions (How to do it)\n",
      "\n",
      "Message: The message to send to the user\n",
      "\n",
      "---\n",
      "\n",
      "Objective: What is the city of the spatial and robotics industry in France?\n",
      "\n",
      "Context:\n",
      "--- Step 0 ---\n",
      "Call Program: main\n",
      "Program Purpose: What is the city of the spatial and robotics industry in France?\n",
      "--- Step 1 ---\n",
      "Action Purpose: Elaborate on the Objective question\n",
      "Action: {\n",
      "  \"answer\": \"The city with a significant presence in the spatial (aerospace) and robotics industry in France is Toulouse. This city is often referred to as \\\"La Ville Rose\\\" or \\\"The Pink City.\\\" It is home to several major aerospace manufacturers, including Airbus, ATR, and the European Space Agency's (ESA) largest establishment outside of its headquarters in the Netherlands. The robotics industry also has a strong presence in Toulouse, with numerous research centers and companies focusing on robotics and automation.\"\n",
      "}\n",
      "--- END OF TRACE ---\n",
      "\n",
      "Purpose: Answer the user question\n",
      "\n",
      "Prompt: Please answer to the Objective's question\n",
      "\n",
      "Message:\n"
     ]
    }
   ],
   "source": [
    "# Now a program that involve two actions to better understand the trace objective\n",
    "\n",
    "main = gp.GraphProgram(\n",
    "\tname = \"main\",\n",
    "\tdescription = \"The main program\",\n",
    ")\n",
    "\n",
    "main.add(gp.Action(\n",
    "    id = \"elaborate\",\n",
    "\ttool = \"Predict\",\n",
    "\tpurpose = \"Elaborate on the Objective question\",\n",
    "\tprompt = \"Please elaborate on the Objective's question\",\n",
    "))\n",
    "\n",
    "main.add(gp.Action(\n",
    "    id = \"answer\",\n",
    "\ttool = \"Speak\",\n",
    "\tpurpose = \"Answer the user question\",\n",
    "\tprompt = \"Please answer to the Objective's question\",\n",
    "))\n",
    "\n",
    "main.connect(\"start\", \"elaborate\")\n",
    "main.connect(\"elaborate\", \"answer\")\n",
    "main.connect(\"answer\", \"end\")\n",
    "\n",
    "# Verify its structure\n",
    "main.build()\n",
    "\n",
    "# Update the program in memory\n",
    "program_memory.update(main)\n",
    "\n",
    "result = agent(Query(query=\"What is the city of the spatial and robotics industry in France?\"))\n",
    "\n",
    "print(lm.history[-1][\"prompt\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The city of France with a significant presence in both the spatial (aerospace) and robotics industry is Toulouse, often referred to as \"La Ville Rose\" or \"The Pink City.\" It houses major aerospace manufacturers like Airbus, ATR, and the European Space Agency's largest establishment outside its headquarters in the Netherlands. Additionally, it has a robust robotics industry with numerous research centers and companies focusing on robotics and automation.\n"
     ]
    }
   ],
   "source": [
    "print(result.final_answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Step 0 ---\n",
      "Call Program: main\n",
      "Program Purpose: What is the city of the spatial and robotics industry in France?\n",
      "--- Step 1 ---\n",
      "Action Purpose: Elaborate on the Objective question\n",
      "Action: {\n",
      "  \"answer\": \"The city with a significant presence in the spatial (aerospace) and robotics industry in France is Toulouse. This city is often referred to as \\\"La Ville Rose\\\" or \\\"The Pink City.\\\" It is home to several major aerospace manufacturers, including Airbus, ATR, and the European Space Agency's (ESA) largest establishment outside of its headquarters in the Netherlands. The robotics industry also has a strong presence in Toulouse, with numerous research centers and companies focusing on robotics and automation.\"\n",
      "}\n",
      "--- Step 2 ---\n",
      "Action Purpose: Answer the user question\n",
      "Action: {\n",
      "  \"message\": \"The city of France with a significant presence in both the spatial (aerospace) and robotics industry is Toulouse, often referred to as \\\"La Ville Rose\\\" or \\\"The Pink City.\\\" It houses major aerospace manufacturers like Airbus, ATR, and the European Space Agency's largest establishment outside its headquarters in the Netherlands. Additionally, it has a robust robotics industry with numerous research centers and companies focusing on robotics and automation.\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\".join(result.program_trace.steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Making Steps\n",
    "\n",
    "Decision making steps are central in HybridAGI, they allow the system to branch over the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "// @desc: The main program\n",
      "CREATE\n",
      "// Nodes declaration\n",
      "(start:Control {id: \"start\"}),\n",
      "(end:Control {id: \"end\"}),\n",
      "(answer:Action {\n",
      "  id: \"answer\",\n",
      "  purpose: \"Answer the user question\",\n",
      "  tool: \"Speak\",\n",
      "  prompt: \"Please answer to the Objective's question\"\n",
      "}),\n",
      "(is_answer_correct:Decision {\n",
      "  id: \"is_answer_correct\",\n",
      "  purpose: \"Check if the answer is correct\",\n",
      "  question: \"Is the answer to the Objectve's question correct?\"\n",
      "}),\n",
      "(critique_answer:Action {\n",
      "  id: \"critique_answer\",\n",
      "  purpose: \"Critique your previous answer to better answer the Objective's question\",\n",
      "  tool: \"Speak\",\n",
      "  prompt: \"Please critique your previous answer to better answer\"\n",
      "}),\n",
      "// Structure declaration\n",
      "(start)-[:NEXT]->(answer),\n",
      "(answer)-[:NEXT]->(is_answer_correct),\n",
      "(is_answer_correct)-[:YES]->(critique_answer),\n",
      "(is_answer_correct)-[:NO]->(answer),\n",
      "(critique_answer)-[:NEXT]->(end)\n"
     ]
    }
   ],
   "source": [
    "main = gp.GraphProgram(\n",
    "\tname = \"main\",\n",
    "\tdescription = \"The main program\",\n",
    ")\n",
    "\n",
    "main.add(gp.Action(\n",
    "    id = \"answer\",\n",
    "\ttool = \"Speak\",\n",
    "\tpurpose = \"Answer the user question\",\n",
    "\tprompt = \"Please answer to the Objective's question\",\n",
    "))\n",
    "\n",
    "main.add(gp.Decision(\n",
    "    id = \"is_answer_correct\",\n",
    "    purpose = \"Check if the answer is correct\",\n",
    "    question = \"Is the answer to the Objectve's question correct?\",\n",
    "))\n",
    "\n",
    "main.add(gp.Action(\n",
    "    id = \"critique_answer\",\n",
    "\ttool = \"Speak\",\n",
    "\tpurpose = \"Critique your previous answer to better answer the Objective's question\",\n",
    "\tprompt = \"Please critique your previous answer to better answer\",\n",
    "))\n",
    "\n",
    "main.connect(\"start\", \"answer\")\n",
    "main.connect(\"answer\", \"is_answer_correct\")\n",
    "main.connect(\"is_answer_correct\", \"critique_answer\", label=\"Yes\")\n",
    "main.connect(\"is_answer_correct\", \"answer\", label=\"No\")\n",
    "main.connect(\"critique_answer\", \"end\")\n",
    "\n",
    "# We build the program to verify its structure\n",
    "main.build()\n",
    "\n",
    "print(main)\n",
    "\n",
    "program_memory.update(main)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Given the fields `objective`, `context`, `purpose`, `question`, `options`, produce the fields `choice`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Objective: The long-term objective (what you are doing)\n",
      "\n",
      "Context: The previous actions (what you have done)\n",
      "\n",
      "Purpose: The purpose of the decision (what you have to do now)\n",
      "\n",
      "Question: The decision-making question\n",
      "\n",
      "Options: The available options for the decision-making question\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the choice}. We ...\n",
      "\n",
      "Choice: The best choice to the decision-making question\n",
      "\n",
      "---\n",
      "\n",
      "Objective: What is the city of the spatial and robotics industry in France?\n",
      "\n",
      "Context:\n",
      "--- Step 0 ---\n",
      "Call Program: main\n",
      "Program Purpose: What is the city of the spatial and robotics industry in France?\n",
      "--- Step 1 ---\n",
      "Action Purpose: Answer the user question\n",
      "Action: {\n",
      "  \"message\": \"The city of the spatial and robotics industry in France is Toulouse. This is because Toulouse hosts several major aerospace companies such as Airbus, Thales Alenia Space, and the European Space Agency (ESA). Additionally, it has a strong robotics community with institutions like the Institute for Research in Computer Science, Robotics, and Automation of Grenoble (INRIA) and the National Institute of Applied Sciences (INSA) Toulouse.\"\n",
      "}\n",
      "--- END OF TRACE ---\n",
      "\n",
      "Purpose: Check if the answer is correct\n",
      "\n",
      "Question: Is the answer to the Objectve's question correct?\n",
      "\n",
      "Options: YES or NO\n",
      "\n",
      "Reasoning: Let's think step by step in order to Choose the option: YES\n",
      "\n",
      "Choice:\n"
     ]
    }
   ],
   "source": [
    "result = agent(Query(query=\"What is the city of the spatial and robotics industry in France?\"))\n",
    "\n",
    "# Let's inspect the decision prompt\n",
    "print(lm.history[-2][\"prompt\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Decision Prompt\n",
    "\n",
    "The decisions uses the ChainOfThought technique to enhance the decision making process, but sometimes it is not enough in that case, you can populate the context with a pre-made answers fetched using the memory or infered using The `Predict` or `ChainOfThought` tools. The decision making process can be viewed as the classification of the program state including the last `N` steps of the trace and the long-term objective. Note also that the Decision step output edges give the possible answers to the system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'role': 'assistant', 'content': ' YES'}\n"
     ]
    }
   ],
   "source": [
    "# Here we can inspect the LLM response\n",
    "\n",
    "print(lm.history[-2][\"response\"][\"choices\"][0][\"message\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Step 0 ---\n",
      "Call Program: main\n",
      "Program Purpose: What is the city of the spatial and robotics industry in France?\n",
      "--- Step 1 ---\n",
      "Action Purpose: Answer the user question\n",
      "Action: {\n",
      "  \"message\": \"The city of the spatial and robotics industry in France is Toulouse. This is because Toulouse hosts several major aerospace companies such as Airbus, Thales Alenia Space, and the European Space Agency (ESA). Additionally, it has a strong robotics community with institutions like the Institute for Research in Computer Science, Robotics, and Automation of Grenoble (INRIA) and the National Institute of Applied Sciences (INSA) Toulouse.\"\n",
      "}\n",
      "--- Step 2 ---\n",
      "Decision Purpose: Check if the answer is correct\n",
      "Decision Question: Is the answer to the Objectve's question correct?\n",
      "Decision: YES\n",
      "--- Step 3 ---\n",
      "Action Purpose: Critique your previous answer to better answer the Objective's question\n",
      "Action: {\n",
      "  \"message\": \"The city of the spatial and robotics industry in France is indeed Toulouse. However, I could provide more detailed information about the significance of Toulouse in these industries. For instance, Toulouse hosts several major aerospace companies such as Airbus, Thales Alenia Space, and the European Space Agency (ESA). Additionally, it has a strong robotics community with institutions like the Institute for Research in Computer Science, Robotics, and Automation of Grenoble (INRIA) and the National Institute of Applied Sciences (INSA) Toulouse. This information will help to emphasize the importance of Toulouse as a hub for spatial and robotics industries in France.\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# And the full trace\n",
    "\n",
    "print(\"\\n\".join(result.program_trace.steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sub-Program Calls\n",
    "\n",
    "When the system encounter a Program step, it jump to another program and commit a ProgramCall step in the trace. The step contains the purpose of the subprogram call to help the system understand what it needs to do. When the agent starts, it commit a first ProgramCall step (the call of the main program, or entrypoint) with the objective as purpose. When a program finish, it commit a ProgramEnd step with the name of the program finished, that way, after a call the Agent can verify the execution of the program and know its name (usefull for further inspection)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hybridagi-B1GoJrSC-py3.10",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}