This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
FAQ
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Introducing Assistant Tool Calling Pipelines

Luna
Employee
Employee
Wednesday

Introduction

OpenAI and Azure OpenAI assistants can invoke models and utilize tools to accomplish tasks. This article primarily focuses on constructing pipelines to leverage the tool-calling capabilities of an existing assistant.

Given the substantial similarity in assistant tool calling between OpenAI and Azure versions, the examples provided in this article are applicable to both platforms.

In part 1, we'll provide a simple introduction to creating an assistant in OpenAI Dashboard and adding user-defined tools for subsequent pipeline use. We'll provide all the necessary data and files.

In part 2, we'll demonstrate two questions and their corresponding assistant responses to illustrate the types of tools the assistant can call, or requires users to call, upon to answer queries.

In part 3, we’ll introduce two new snaps: tool call router  and submit tool outputs , along with upgrades to the existing two snaps: run thread  and create and run thread .

In part 4, we'll delve into the pipeline workflow and the specific configurations required for setting up snaps.

Part 1: Prerequisite - Set Up An Assistant in OpenAI Dashboard

OpenAI and Azure OpenAI assistants manage the system prompt, the model used to generate response, tools (including file search, code interpreter, and other user-defined tools), and model configuration such as temperature and response format.

Here we will only introduce the most basic settings, and you can adjust them according to your needs.

Please refer to OpenAI and Azure OpenAI documentations for more information.

  1. Navigate to the OpenAI Dashboard: Go to the OpenAI dashboard - assistants and click the "Create" button in the top right corner to initiate the process of creating a new assistant.

  2. Name Your Assistant: Provide a name for your new assistant. You can choose any name you prefer, such as "Test Assistant".

  3. System Instruction (Optional): You can optionally provide a system instruction to guide the assistant's behavior. For now, let's skip this step.

  4. Select a Model: Choose the model you want to use for your assistant. In this case, we'll select "gpt-4o-mini".

  5. Enable Tools:

    1. Enable the "file search"

      1. File search is an OpenAI-provided managed RAG service. Using this tool allows the model to retrieve information relevant to the query from the vector store and use it to answer.

      2. In this case, please create a new vector store, upload the wildfire_stats.pdf file to the vector store, and add the vector store to the assistant.

    2. Enable the"code interpreter" tools

      1. The code interpreter is also a built-in tool within the OpenAI assistant. It can run the code produced by the model directly and provide the output.

    3. Create three custom functions with the following schema:

      1. By providing these definitions, we are enabling the model to identify which user-defined functions it can call. While the model can suggest the necessary function, the responsibility of executing the function lies with the user.

Function definition: get_weather

 

Spoiler
{
  "name": "get_weather",
  "description": "Determine weather in my location",
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string",
        "description": "The city and state e.g. San Francisco, CA"
      }
    },
    "additionalProperties": false,
    "required": [
      "city"
    ]
  }
}

 

Function definition: get_wiki_url

 

Spoiler
{
  "name": "get_wiki_url",
  "description": "Get the wiki url of a given location",
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string",
        "description": "The city and state e.g. San Francisco, CA"
      }
    },
    "additionalProperties": false,
    "required": [
      "city"
    ]
  }
}

 

Function definition: get_webpage

 

Spoiler
{
  "name": "get_webpage",
  "description": "A simple web scraper that is capable of extracting information out of a url.",
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "The url of the webpage to get"
      }
    },
    "additionalProperties": false,
    "required": [
      "url"
    ]
  }
}

 

In this way, we've successfully created the assistant we'll be using. It should look similar to the image below. Now you can directly go to the playground and ask some questions to see how the assistant responds.

image-20241031-185949.png

Up to this point, you should have created an assistant with three user-defined functions. The file search tool should have access to a vector store that contains a file.

Part 2: Two Examples of Assistant Tool Calling

To help you understand how the assistant works, we will use the following pipeline to ask the newly created assistants two questions in this section and examine their responses.

You can find the construction details for this pipeline in part 4. For now, let's focus on the pipeline's execution results.

Pipeline Overview

The Driver Pipeline

image-20241105-054212.png

The Worker Pipeline

image-20241105-055448.png

Prompt One

Our first question to the assistant is:

"What is the weather and the wiki url of San Francisco? And what is the content of the wiki page?"

Through this query, we're evaluating the assistant's capability to:

1) identify the necessary tools for a task - in this case, all three: get_weather, get_wiki_url, and get_webpage should be called;

2) understand the sequential dependencies between tools. For example, the assistant should recognize that get_wiki_url must be called before get_webpage to acquire the necessary URL.

As shown below, the model's response is both reasonable and correct.

image-20241105-041219.png

Prompt Two 

Our second question to the assistant is:

What is the number of federal fires from 2018 to 2022, and can you write a Python code to sort the years based on the number of fires in ascending order and tell me the weather in San Francisco? 

The question might seem a bit odd on its own, but our goal is to evaluate how the assistant handles built-in tools such as file search and code interpreter. Specifically, we want to determine if it can effectively combine these built-in tools with user-defined functions in providing an answer.

To answer this question, the model needs to first invoke the file search tool to retrieve the first row of data from the first table on the first page of the Wildfire PDF. Then, it generates a Python code snippet for sorting and calls the second tool, the code interpreter, to execute this code. Finally, it calls the third user-defined tool, get_weather, to obtain the weather in San Francisco.

Expected Data in Wildfire PDF:

image-20241105-071222.png

As shown below, the model responses as expected.

image-20241105-070606.png

Up to this point, you should understand that the assistant could utilize three different categories of tools to answer user questions.

Part 3: Introduction of New Snaps

We'll start by focusing on the new elements of the pipeline: two newly introduced snaps and the added attributes to the existing ones, before delving into the overall pipeline details.

1. Tool Call Router (new)

tool_call_router.png

  • The tool call router snap simplifies the assistant's response (the run object) for easier downstream processing.

  • It combines the functionalities of copy, mapper, and JSON splitter.

  • The first output view contains:

    • the original assistant's response

    • an empty list named tool_outputs to collect the results of all function executions in the subsequent message appender snap.

  • The second output view provides a list of tools to call, extracted from the required actions section of the assistant's response

2. Submit Tool Outputs (new)

submit_tool_outputs.png

This snap submits a list of function execution results to the assistant. The assistant will then generate the final response or request further tool calls.

3. Create and Run Thread (upgraded)

create_and_run_thread.png

  • We've added a new section to the Create And Run Thread configuration to specify detailed parameters for tool calls.

  • The Tool choice option allows you to instruct the assistant to:

    • automatically select tools (AUTO)

    • use no tools (NONE)

    • require at least one tool (REQUIRED)

    • use a specific user-defined tool (SPECIFY A FUNCTION, providing the function name).

  • The Parallel tool call option determines whether the assistant can call multiple tools simultaneously.

4. Run Thread (upgraded)

run_thread.png

Same configuration is added to the Run Thread snap as well.

Part 4: Hands-on Pipeline Construction

Pipeline workflow overview

There are a total of 5 pipelines.

  1. Driver pipeline : 

    • Sends the initial prompt to the assistant.

    • Receives a response containing tool call requests.

    • Passes the response to the "pipeloop" snap to trigger the worker pipelines to execute the tools.

  2. Worker pipeline: 

    • Executes the function calls specified in the tool call requests.

    • Collects the results of the function calls.

    • Sends the results back to the assistant.

    • This pipeline is executed repeatedly until there are no more tools to call.

  3. get_weather pipeline: 

    • Takes a city name as input.

    • Queries a weather API to get the current weather for the specified city.

    • Outputs the retrieved weather information.

  4. get_wiki_url pipeline: 

    • Takes a city name as input.

    • Searches for the Wikipedia page URL for the specified city.

    • Outputs the found URL.

  5. get_webpage pipeline: 

    • Takes a URL as input

    • Fetch the webpage by visiting the URL

    • Use a model to summarize the content of the webpage

    • Outputs the summary

The Driver Pipeline

The driver pipeline can be constructed in two ways: either using a combined "create and run" operation or by performing the creation and running steps sequentially. Both methods achieve the same result in this scenario.

image-20241105-190038.png

The Worker Pipeline

image-20241105-190200.png

The get_weather Pipeline

image-20241105-200324.png

 

The get_wiki_url Pipeline

image-20241105-200017.png

The get_webpage Pipeline

image-20241105-200612.png

  • Get Client: Access the webpage pointed to by the URL and retrieve the HTML content.

  • HTML Parser: Parse the HTML content into text format.

  • Summarize: Generate a user prompt and concatenate it with the webpage text.

  • OpenAI Summarize: Use the model to generate a summary of the webpage content.

Input and output of key snaps

We'll illustrate the essential inputs and outputs of the intermediate process through a single tool call interaction.

image-20241105-202651.png

1. Create and Run Thread

This snap forwards the user's initial prompt to the assistant and returns a run object. The highlight of this run object is the required action, which outlines the necessary tool calls.

Output of Create and Run Thread - a run object

Spoiler
[
  {
    "run": {
      "id": "run_0JYmmzkQlIGRe4E0cNK8daKm",
      "object": "thread.run",
      "created_at": 1730842197,
      "assistant_id": "asst_nwIrRaBwD6E6xa7EmnDOy2fx",
      "thread_id": "thread_L1mEVGH40bHfwTf2gsDXliMi",
      "status": "requires_action",
      "started_at": 1730842197,
      "expires_at": 1730842797,
      "cancelled_at": null,
      "failed_at": null,
      "completed_at": null,
      "required_action": {
        "type": "submit_tool_outputs",
        "submit_tool_outputs": {
          "tool_calls": [
            {
              "id": "call_uOIx8piJAUty3WpFpVCcxFMc",
              "type": "function",
              "function": {
                "name": "get_weather",
                "arguments": "{\"city\": \"San Francisco, CA\"}"
              }
            },
            {
              "id": "call_d14hJfMMOvcyxsvWpXbgi9uF",
              "type": "function",
              "function": {
                "name": "get_wiki_url",
                "arguments": "{\"city\": \"San Francisco, CA\"}"
              }
            }
          ]
        }
      },
      "last_error": null,
      "model": "gpt-4o",
      "instructions": "",
      "tools": [
        {
          "type": "code_interpreter"
        },
        {
          "type": "file_search",
          "file_search": {
            "ranking_options": {
              "ranker": "default_2024_08_21",
              "score_threshold": 0
            }
          }
        },
        {
          "type": "function",
          "function": {
            "name": "get_weather",
            "description": "Determine weather in my location",
            "parameters": {
              "type": "object",
              "properties": {
                "city": {
                  "type": "string",
                  "description": "The city and state e.g. San Francisco, CA"
                }
              },
              "additionalProperties": false,
              "required": [
                "city"
              ]
            },
            "strict": true
          }
        },
        {
          "type": "function",
          "function": {
            "name": "get_wiki_url",
            "description": "Get the wiki url of a given location",
            "parameters": {
              "type": "object",
              "properties": {
                "city": {
                  "type": "string",
                  "description": "The city and state e.g. San Francisco, CA"
                }
              },
              "additionalProperties": false,
              "required": [
                "city"
              ]
            },
            "strict": true
          }
        },
        {
          "type": "function",
          "function": {
            "name": "get_webpage",
            "description": "A simple web scraper that is capable of extracting information out of a url.",
            "parameters": {
              "type": "object",
              "properties": {
                "url": {
                  "type": "string",
                  "description": "The url of the webpage to get"
                }
              },
              "additionalProperties": false,
              "required": [
                "url"
              ]
            },
            "strict": true
          }
        }
      ],
      "tool_resources": {},
      "metadata": {},
      "temperature": 1,
      "top_p": 1,
      "max_completion_tokens": null,
      "max_prompt_tokens": null,
      "truncation_strategy": {
        "type": "auto",
        "last_messages": null
      },
      "incomplete_details": null,
      "usage": null,
      "response_format": {
        "type": "text"
      },
      "tool_choice": "auto",
      "parallel_tool_calls": true
    }
  }
]

2. Tool Call Router

 It's important to note that the first output view not only holds the assistant's response but also an empty "tool_outputs" list. This list serves as a container for storing function results as they are gathered in subsequent message appenders.

Tool Call Router - 1st output view

Spoiler
[
  {
    "run":{...},
    "tool_outputs":[]
  }
]

The second output view extracts the tool calls from the required actions and converts the argument values into JSON format, storing them in json_arguments. This eliminates the need for subsequent argument conversion by each tool.

Tool Call Router - 2nd output view

Spoiler
[
  {
    "id": "call_uOIx8piJAUty3WpFpVCcxFMc",
    "type": "function",
    "function": {
      "name": "get_weather",
      "arguments": "{\"city\": \"San Francisco, CA\"}",
      "json_arguments": {
        "city": "San Francisco, CA"
      }
    }
  },
  {
    "id": "call_d14hJfMMOvcyxsvWpXbgi9uF",
    "type": "function",
    "function": {
      "name": "get_wiki_url",
      "arguments": "{\"city\": \"San Francisco, CA\"}",
      "json_arguments": {
        "city": "San Francisco, CA"
      }
    }
  }
]

3. Pipeline Execute Snap - Get Weather Function

 Get Wiki URL Function - Output

Spoiler
[
  {
    "statusLine": {
      "httpVersion": "HTTP/1.1",
      "statusCode": 200,
      "statusText": "OK"
    },
    "headers": {...},
    "entity": [
      "San Francisco, CA",
      [
        "San Francisco, CA"
      ],
      [
        ""
      ],
      [
        "https://en.wikipedia.org/wiki/San_Francisco,_CA"
      ]
    ],
    "_debug": {...},
    "original": {
      "id": "call_d14hJfMMOvcyxsvWpXbgi9uF",
      "type": "function",
      "function": {
        "name": "get_wiki_url",
        "arguments": "{\"city\": \"San Francisco, CA\"}",
        "json_arguments": {
          "city": "San Francisco, CA"
        }
      }
    }
  }
]

The tool's output provides a full HTTP response, however, we're solely interested in the "entity" content which will serve as the tool's output. This extraction will occur in the subsequent snap, "Function Result Generator".

5. Message Appender

The Message Appender’s output contains a run object from upstream, however, we're solely interested in the tool_outputs field which is a list of function results. Thus in the subsequent snap, "Submit Tool Outputs", we will only use the tool_outputs field.

Message Appender - Output

Spoiler
[
  {
    "run": {...},
    "tool_outputs": [
      {
        "sl_role": "TOOL",
        "function_id": "call_d14hJfMMOvcyxsvWpXbgi9uF",
        "content": [
          "San Francisco, CA",
          [
            "San Francisco, CA"
          ],
          [
            ""
          ],
          [
            "https://en.wikipedia.org/wiki/San_Francisco,_CA"
          ]
        ]
      },
      {
        "sl_role": "TOOL",
        "function_id": "call_uOIx8piJAUty3WpFpVCcxFMc",
        "content": {
          "last_updated_epoch": 1730841300,
          "last_updated": "2024-11-05 13:15",
          "temp_c": 16.2,
          "temp_f": 61.2,
          "is_day": 1,
          "condition": {
            "text": "Mist",
            "icon": "//cdn.weatherapi.com/weather/64x64/day/143.png",
            "code": 1030
          },
          "wind_mph": 4.9,
          "wind_kph": 7.9,
          "wind_degree": 330,
          "wind_dir": "NNW",
          "pressure_mb": 1018,
          "pressure_in": 30.07,
          "precip_mm": 0,
          "precip_in": 0,
          "humidity": 77,
          "cloud": 0,
          "feelslike_c": 16.2,
          "feelslike_f": 61.2,
          "windchill_c": 16.2,
          "windchill_f": 61.1,
          "heatindex_c": 16,
          "heatindex_f": 60.9,
          "dewpoint_c": 11.9,
          "dewpoint_f": 53.4,
          "vis_km": 8,
          "vis_miles": 4,
          "uv": 2.7,
          "gust_mph": 7.2,
          "gust_kph": 11.6
        }
      }
    ]
  }
]

6. Submit Tool Outputs

This snap forwards function results to the assistant and receives a run object as a response. This object can either provide the final answer or dictate subsequent tool calls.

In this example, the assistant's output specifies the next tool to be called, as indicated by the "required action".

Submit Tool Outputs - Output - subsequent tool calls example

Spoiler
[
  {
    "run": {
      "id": "run_0JYmmzkQlIGRe4E0cNK8daKm",
      "object": "thread.run",
      "created_at": 1730842197,
      "assistant_id": "asst_nwIrRaBwD6E6xa7EmnDOy2fx",
      "thread_id": "thread_L1mEVGH40bHfwTf2gsDXliMi",
      "status": "requires_action",
      "started_at": 1730842201,
      "expires_at": 1730842797,
      "cancelled_at": null,
      "failed_at": null,
      "completed_at": null,
      "required_action": {
        "type": "submit_tool_outputs",
        "submit_tool_outputs": {
          "tool_calls": [
            {
              "id": "call_lsZpmQ4SPZ6QJ6Dsuk8tvzvy",
              "type": "function",
              "function": {
                "name": "get_webpage",
                "arguments": "{\"url\":\"https://en.wikipedia.org/wiki/San_Francisco,_CA\"}"
              }
            }
          ]
        }
      },
      "last_error": null,
      "model": "gpt-4o",
      "instructions": "",
      "tools": [...],
      "tool_resources": {},
      "metadata": {},
      "temperature": 1,
      "top_p": 1,
      "max_completion_tokens": null,
      "max_prompt_tokens": null,
      "truncation_strategy": {
        "type": "auto",
        "last_messages": null
      },
      "incomplete_details": null,
      "usage": null,
      "response_format": {
        "type": "text"
      },
      "tool_choice": "auto",
      "parallel_tool_calls": true
    },
    "original": {
      "run": {...},
      "tool_outputs": [...]
    }
  }
]

In the following example, the assistant outputs the final result. There's an extra message list in the output which contains the result itself as well as the original user prompt.

Submit Tool Outputs - Output - final answer example

Spoiler
[
  {
    "messages": [
      {
        "id": "msg_ETlH4jGXltDdopImtZFBhI7a",
        "object": "thread.message",
        "created_at": 1730845440,
        "assistant_id": "asst_nwIrRaBwD6E6xa7EmnDOy2fx",
        "thread_id": "thread_4CrAatFTqHktHhareDTkHPNc",
        "run_id": "run_lunPUVK7tmVV7NTiL72NKRju",
        "role": "assistant",
        "content": [
          {
            "type": "text",
            "text": {
              "value": "The current weather in San Francisco, CA is sunny with a temperature of 17.3°C (63.1°F). There's a light breeze coming from the northwest at about 10.8 kph (6.7 mph), and the humidity level is at 63%. There is no precipitation at the moment, and the visibility is 11 kilometers (6 miles) .\n\nYou can find more detailed information about San Francisco on its [Wikipedia page](https://en.wikipedia.org/wiki/San_Francisco,_CA) . Here's a summary of the content from the page:\n\n- **Location**: San Francisco is at the north end of the San Francisco Peninsula, embracing parts of the Pacific Ocean and San Francisco Bay.\n- **Population**: As of 2023, it has an estimated population of 808,988, making it the fourth-most populous city in California.\n- **Area**: It covers 46.9 square miles (121 square kilometers).\n- **Diversity**: The city is majority-minority with diverse demographics.\n- **History**: The indigenous Yelamu of the Ramaytush Ohlone inhabited the area before European settlement in 1776.\n- **Economic Importance**: Known globally for technology, finance, and tourism.\n- **Cultural Landmarks**: Home to iconic sites like the Golden Gate Bridge and Alcatraz Island.\n- **Climate**: It has a warm-summer Mediterranean climate with frequent fog in summer.\n- **Education**: Hosts the University of California, San Francisco, and San Francisco State University.\n- **Neighborhoods & Demographics**: Known for diverse neighborhoods with unique character.\n- **Economy & Culture**: A robust economy with significant tech and finance sectors, along with a vibrant arts scene.\n- **Government**: Operates under a Mayor-Council system.\n\nThis encapsulates San Francisco's historical, cultural, and economic significance .",
              "annotations": []
            }
          }
        ],
        "attachments": [],
        "metadata": {}
      },
      {
        "id": "msg_1VUwhJdB9k2zOl9g0NH2rGQU",
        "object": "thread.message",
        "created_at": 1730845275,
        "assistant_id": null,
        "thread_id": "thread_4CrAatFTqHktHhareDTkHPNc",
        "run_id": null,
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": {
              "value": "What is the weather and the wiki url of San Francisco? And what is the content of the wiki page?",
              "annotations": []
            }
          }
        ],
        "attachments": [],
        "metadata": {}
      }
    ],
    "run": {
      "id": "run_lunPUVK7tmVV7NTiL72NKRju",
      "object": "thread.run",
      "created_at": 1730845275,
      "assistant_id": "asst_nwIrRaBwD6E6xa7EmnDOy2fx",
      "thread_id": "thread_4CrAatFTqHktHhareDTkHPNc",
      "status": "completed",
      "started_at": 1730845440,
      "expires_at": null,
      "cancelled_at": null,
      "failed_at": null,
      "completed_at": 1730845445,
      "required_action": null,
      ...
    },
    "original": {}  
  }
]

Snap settings

This article particularly emphasizes the loop condition settings in the pipeloop.

image-20241105-223355.png

 We've configured the loop to terminate when the assistant's response indicates no further tool calls are required (i.e., "required_action" is null). This is because if there's no need for additional tool calls, there's no reason to continue executing the worker using Pipeloop.

Edge Case - When no tool call is needed

The previous driver pipeline had a limitation: it couldn't handle cases where the model could directly answer the user's query without calling any user-defined functions.

This was because the output of Create and Run Thread wouldn't contain the required_action field. Since the pipeloop snap follows a do-while logic, it would always run at least once before checking the stop condition. Consequently, when the assistant didn't require a tool call, submitting the tool call output to the assistant in the worker pipeline would result in an error.

The following driver pipeline offers a simple solution to this problem by using a router to bypass the pipeloop for requests that can be answered directly.

image-20241105-224423.png

Labels:
Preview file
426 KB
Preview file
3018 KB
Popular Articles