ChatGPT function calls for multiple inputs in a single request
OpenAI recently released support for function calls in their chat completions API. The example they give shows the ability to define a function such as get_current_weather(location: string) so that when a user asks “What’s the weather like in Boston right now?”, the chat completions API intelligently extracts parameters from the user’s input, responding with JSON to your server describing what function to call, eg: name: get_current_weather, arguments: {"location": "Boston"}. This allows you to extract the function name and argument values from the response, call an external weather API to get the temperature in Boston, then respond with data such as {“temperature": “22”, “unit": “celsius", “description": “Sunny”} which ChatGPT intelligently formats back as a response such as “The weather in Boston is currently sunny with a temperature of 22 degrees Celsius.”
Streak has a number of projects on our AI roadmap and this appeared useful for one in particular. But first, a bit of background.
Solving the “blank slate” problem
A challenge for most SaaS products is that when users sign up, they are left staring at a blank slate and are unsure where to start. A popular way to handle this is via your onboarding process by providing tutorial videos, helpful walkthroughs, context-sensitive help, and so on. These are often necessary but not sufficient as nothing beats having the user’s own data available to use within the SaaS product. We tackled this a while ago in our onboarding flow: when the user creates their first pipeline, our system looks at the user’s Gmail sent items and displays a “Quick Add” screen, offering to create boxes for the contacts they’ve sent emails to.
This has been incredibly useful to customers as the blank slate is now replaced with their own data where they can see a number of magic columns including when they last emailed the contact, when the contact last emailed them, and so on. It’s then easy to see who they should follow up with, review a history of all emails from everyone on their team, and take appropriate action.
Experimenting with AI and forcing JSON output
We thought a good upgrade from this functionality is to intelligently categorize all their emails. Rather than just looking at emails the user had recently sent, what if we looked at all recent email threads and used AI to categorize them based on our pipeline templates into Sales, Hiring, Job Search, Investor Relations, Fundraising, Real Estate, and so on? Someone in HR could use this to identify those who had applied as well as identify candidates the HR team had reached out to directly.
I tackled this as an R&D project, using emails in my own account to see how well ChatGPT 3.5 could classify emails as either HIRING or NOT_HIRING. I used the chat completions API and divided the input into the system message, which provided some initial prompt engineering and some background. Part of the system message primed ChatGPT with few-shot prompting as well as the expected output as structured JSON:
For example, given the inputs:
###
[0] Subject: Resume
Snippet: Hello, I've attached my resume and am applying for the content writer position
[1] Subject: Lunch on Tuesday
Snippet: Hi James, it's been a while! We should catch up this Tuesday as I wanted to go over the sales numbers
[2] Subject: Ad for Widgets
Snippet: Hi Doug, I'd like to be considered for the online ad posted on Facebook. Please find attached my job history.
###
You should output a JSON array with one object per email:
```
[
{
"index": 0,
"category": "HIRING",
"explanation": "Email mentions attaching a resume and that they are applying for a position"
},
{
"index": 1,
"category": "NOT_HIRING",
"explanation": "Email appears unrelated to hiring"
},
{
"index": 2,
"category": "HIRING",
"explanation": "Email mentions an online ad and they have included their job history"
}
]Asking for an explanation is helpful to understand the rationale behind the category. The user message is then similar to the few-shot prompts but includes a trailing output hint:
[0] Subject: Resume Snippet: Hello, I've attached my resume and am applying for the content writer position
[1] Subject: Lunch on Tuesday Snippet: Hi James, it's been a while! We should catch up this Tuesday as I wanted to go over the sales numbers
[2] Subject: Ad for Widgets Snippet: Hi Doug, I'd like to be considered for the online ad posted on Facebook. Please find attached my job history.
Output:
```The three backticks indicates that the JSON output is to follow, and ChatGPT reliably supplies the JSON array with objects as specified in the system message. This makes it easy to use a JSON parser to deserialize the text into our internal classes in Kotlin.
Migrating to function calls
To use function calls, you specify the function definitions using JSON schema. For our example, I want ChatGPT to call a classify_email function for a given message and provide the category and an explanation. The request JSON looks like the following:
{
"model": "gpt-3.5-turbo-0613",
"messages": [
{
"role": "system",
"content": "You are an assistant designed to analyze information from emails and assign the category that best matches the content."
},
{
"role": "user",
"content": "Subject: Resume\\nSnippet: Hello, I've attached my resume and am applying for the content writer position"
}
],
"functions": [
{
"name": "classify_email",
"description": "Classify an email into a category",
"parameters": {
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": [
"SALES",
"PROJECTS",
"BUSINESS_DEV",
"HIRING",
"JOB_SEARCH",
"INVESTOR",
"FUNDRAISING",
"ORDERS",
"REAL_ESTATE",
"SUPPORT",
"UNKNOWN"
],
"description": "The category to classify the email into"
},
"explanation": {
"type": "string",
"description": "An explanation of why the email belongs to the specified category"
}
},
"required": ["category", "explanation"]
}
}
],
"function_call": { "name": "classify_email" }
}
Here you can see there’s a system message which provides some background. Note: I’ve omitted some additional information from the system message describing each category. The user message supplies the subject and snippet for the email in question.
There are two important parts to making function calls work with ChatGPT. The first is supplying the functions parameter with the definition of your function. Because this uses JSON schema, I can supply a category property as an enum type and specify the allowable values. The second is the function_call property which, when you specify a name of the function to call, tells ChatGPT that it must call the function.
The response from the chat completions API indicates a function call with the information we’re looking for:
{
"id": "chatcmpl-<redacted>",
"object": "chat.completion",
"created": 1687490210,
"model": "gpt-3.5-turbo-0613",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"function_call": {
"name": "classify_email",
"arguments": "{\\n \\"category\\": \\"HIRING\\", \\"explanation\\": \\"Email mentions attaching a resume and that they are applying for a position.\\"\\n}"
}
},
"finish_reason": "stop"
}
]
}
Unlike OpenAI’s weather example, we don’t need to call the API again to provide a response that it can intelligently respond with to the user; the goal here is to get back structured data and avoid having to coerce ChatGPT into transforming the output into JSON. I can trust that the arguments property is JSON already and it’s trivial to parse and extract the category and explanation.
However, this is only a single message. When processing hundreds or thousands of items, I’ve found there’s a sweet spot in optimizing your queries to minimize response time and maximize token utilization combined with making parallel API requests. Too many parallel requests means not only are you burning through tokens faster, but you end up getting rate limited with 429 errors. In my case, both the system message and function definition are identical across all emails. So I can cut down on token usage by supplying the system message, the function definition, and then multiple user messages (one for each email).
I had hoped ChatGPT would respond with one function call per user message but it only responded with a single function call, classifying only one single email. The trick turned out to be have the function accept an array of messages, plus specifying the object definition for each message separately.
Here’s what the complete request looks like:
{
"model": "gpt-3.5-turbo-0613",
"messages": [
{
"role": "system",
"content": "You are an assistant designed to analyze information from emails and assign the category that best matches the content."
},
{
"role": "user",
"content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
},
{
"role": "user",
"content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
},
{
"role": "user",
"content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
},
{
"role": "user",
"content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
},
{
"role": "user",
"content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
},
{
"role": "user",
"content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
},
{
"role": "user",
"content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
},
{
"role": "user",
"content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
}
],
"functions": [
{
"name": "classify_email",
"description": "Called to classify emails",
"parameters": {
"type": "object",
"properties": {
"messages": {
"type": "array",
"items": { "$ref": "#/$defs/message" },
"description": "Array of emails to classify"
}
},
"$defs": {
"message": {
"type": "object",
"properties": {
"threadId": {
"type": "string",
"description": "The ThreadID of the email"
},
"category": {
"type": "string",
"enum": [
"SALES",
"PROJECTS",
"BUSINESS_DEV",
"HIRING",
"JOB_SEARCH",
"INVESTOR",
"FUNDRAISING",
"ORDERS",
"REAL_ESTATE",
"SUPPORT",
"UNKNOWN"
],
"description": "The category to classify the email into"
},
"explanation": {
"type": "string",
"description": "An explanation of why the email belongs to the specified category"
}
},
"required": ["threadId", "category", "explanation"],
"description": "The message to classify"
}
},
"required": ["messages"]
}
}
],
"function_call": { "name": "classify_email" }
}
This turns the function from classify_email(category: string, explanation: string) into classify_email(messages: array) and further, this specifies that the items in the array are of type messages defined at #/$defs/message. The $defs property then specifies the JSON schema for the message object with the same parameters as before.
The response now calls my classify_email function and passes an array of messages, where each message contains the arguments needed to categorize each one:
{
"id": "chatcmpl-<redacted>",
"object": "chat.completion",
"created": 1687490210,
"model": "gpt-3.5-turbo-0613",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"function_call": {
"name": "classify_email",
"arguments": "{\\n \\"messages\\": [\\n {\\n \\"threadId\\": \\"<redacted>\\",\\n \\"category\\": \\"UNKNOWN\\",\\n \\"explanation\\": \\"The email does not fit into any specific category.\\"\\n }\\n ]\\n}"
}
},
"finish_reason": "stop"
}
]
}
I’ve truncated the arguments value above, but the JSON for this contains an array of message objects with argument values as defined in the schema:
[
{
"threadId": "<redacted>",
"category": "UNKNOWN",
"explanation": "The email does not fit into any of the predefined categories."
},
{
"threadId": "<redacted>",
"category": "PROJECTS",
"explanation": "The email is related to a project meeting invitation."
},
{
"threadId": "<redacted>",
"category": "PROJECTS",
"explanation": "The email is related to a person going on call for a specific project."
},
{
"threadId": "<redacted>",
"category": "SUPPORT",
"explanation": "The email is related to providing feedback and information about an Android app."
},
{
"threadId": "<redacted>",
"category": "UNKNOWN",
"explanation": "The email does not fit into any of the predefined categories."
},
{
"threadId": "<redacted>",
"category": "UNKNOWN",
"explanation": "The email does not fit into any of the predefined categories."
},
{
"threadId": "<redacted>",
"category": "UNKNOWN",
"explanation": "The email does not fit into any of the predefined categories."
}
]
And there it is! Beautifully structured JSON without having to burn through tokens coercing ChatGPT via few-shot prompting, and I can trust the the category property uses only the enum values I specified.
Further improvements
This is a good MVP for proving the concept. but there is still a ways to go to improve the categorization accuracy. The system can provide more of the email content rather than just the Gmail snippet, plus additional filtering up front can reduce the number of emails we need to classify by excluding things like automated emails.
Conclusion
ChatGPT still calls the function only once, but by using an array argument, ChatGPT transforms the result for each user message into an item in the array. This cuts down on the number of requests, avoiding getting rate limited with 429 responses, as well as optimizing the number of tokens used by sharing the system message and function definition across multiple items.
Engineering at Streak
We work on many interesting challenges affecting millions of users and many terabytes of data. For more information, visit https://www.streak.com/careers