A similar problem exists when streaming responses generated by AI that I call “Flash of Incomplete Markdown” (FOIM). I’ve reproduced this within OpenAI’s playground by throttling my connection speed to dialup internet speeds:

While this is greatly exaggerated due to the slow speed, you can see the incomplete markdown appear in chunks. This occurs because OpenAI’s streaming API returns an event stream where it builds up a response message. These chunks are provided by what are called output text deltas:

You can read more about output text deltas in their API docs.

I’ve seen this same behavior occur in a number of commercially launched products, so this isn’t some obscure effect. We experienced this in our product as well. One of the AI features Streak offers is the ability to ask a question about one of your deals, where the answer may be in any related email threads, various comments, meeting notes, and so on.

You could ask “When did the customer sign the contract?” and getting back an answer of “Last Thursday” is useful, but even more useful is providing a link to the email thread from Thursday where they sent an email saying “Signed contract attached”. The Streak user can click on that and be confident that the answer is correct. You can see that here as I was implementing the functionality in my dev environment:

What should have been output as trivial [source] links ended up rendering the incomplete markdown using a lengthy link format of https://streak.com/a/boxes/{KEY}/itemtype/{KEY} and only when the browser received the closing ) of the markdown link did the content collapse to show just [source].

Hallucinations

Additionally, when we launched this internally to Streak employees someone reported that one of the links was incorrect. It turns out that OpenAI was hallucinating one of the URLs we provided as the citation by combining the first part of the box key with the last part of the comment key. The two keys had a common prefix and OpenAI returned a mangled URL. This resulted in a 404 error since the hallucinated link was invalid.

While the incomplete markdown issue is somewhat annoying, providing the user with incorrect links is unacceptable. So I set out to solve the hallucination issue.

Really short links

What if instead of https://streak.com/a/boxes/{KEY}/itemtype/{KEY} where the keys are lengthy (the URL can be over 200 characters long), the link was simply “#REF3”? Due to the tiny amount of tokens that takes, it’d be significantly less likely that OpenAI would hallucinate by combining unrelated parts of the output due to some shared common prefix. We decided to change the link text to output Wikipedia style numbers, where the links would get output as:

[1](#REF3)

and so on for each reference1. This would work great, and we could simply keep track of which short references map to which original links and replace them on the fly. But where would we perform this substitution? The server streams the output text deltas from OpenAI’s API to the client and each delta is completely arbitrary, where one delta could be “#”, the next delta might be “REF”, with another delta of just “3”. Would we need to provide the client with the complete mapping ahead of time so it could replace the assembled markdown as it’s received? This gets complicated as we may have dozens of citations and the answer may only need to cite one of them.

Bring on the state machine

It turns out there’s an easier, far less complicated solution where we can dynamically detect the start of a markdown link and begin buffering the output on the server, not sending anything to the client until the link has completed. This is simple to do via a state machine. The four states are TEXT, LINK_TEXT, EXIT_LINK_TEXT, and LINK_URL with the following behaviors:

1. Regardless of state, the \ character acts as an escape character, so we output the next character without any additional consideration.

2. Start out in TEXT state, outputting normal text that’s not a link, streaming tokens to the client.

3. Beginning of markdown link via [ character: transition to LINK_TEXT state and still stream tokens to the client.

4. If in LINK_TEXT state and there’s a matching end of markdown link via ] character2: transition to EXIT_LINK_TEXT state and still stream tokens to the client.

5. If in EXIT_LINK_TEXT state and next character is (, transition to LINK_URL state and begin buffering the URL without streaming anything to the client. Otherwise, if the next character is anything else transition back to the TEXT state and resume streaming.

6. If in LINK_URL state and there’s a matching end of the link via ) character3, see if the buffered URL exists as a key in our URL map and, if it is, replace the URL with the full URL. Then output the URL to the client and transition back to the default TEXT state.

Since we are asking OpenAI to provide citations using a specific format, this simple processing is sufficient for our needs. The markdown spec is surprisingly robust4, allowing additional formats for links such as:

• [link](/uri "title")

• [link](</my uri>)

• [a](<b)c>)

I haven’t implemented support for the full spec since we have well known formats for our link text and URLs, but should the need arise the state machine can easily be extended. Until then, YAGNI.

Improved output

You can see the result of implementing the above:

No more raw link URLs being flashed to the user. Just nice, clean links that appear as the full URL has been processed. You can see in the event stream that the server streams each output text delta as it’s received from OpenAI, except if there’s a link where the server buffers the link, replaces the short URL with the full URL, and sends the complete link to the client in a single chunk:

Just plain better

There are multiple benefits from taking this approach:

1. Instead of sending what ended up being links which consume approximately 50 tokens, our short links are 3 tokens. Fewer tokens means a smaller context for OpenAI and a lower usage bill.

2. Link hallucinations appear to be a thing of the past, which was the catalyst for making this change.

3. The streaming response from OpenAI is now much faster since it’s providing the response with our short URLs of only 3 tokens and the speed difference is noticeable to the user.

4. Since we buffer the URLs server-side and only provide the completed markdown link once we reach the end of the markdown URL, we are preventing flashes of incomplete markdown even if the link isn’t our short reference link. All links benefit from this buffering approach.

5. While our citation URLs don’t reveal sensitive information and are secured by the user’s Gmail account, there’s an additional privacy bonus in that this substitution never transmits the URL to OpenAI in the first place. For some organizations, this alone could be a win.

Faster, better, and cheaper — can’t go wrong with that!

Engineering at Streak

We work on many interesting challenges affecting millions of users and many terabytes of data. For more information, visit https://www.streak.com/careers

The average latency was a lot less but still spiked into the tens of seconds. After doing numerous optimizations and discovering a nice algorithm to avoid expensive read-time queries, the same API endpoints were vastly improved:

The graph almost looks like things stopped working halfway through the X axis, but zooming in on the data:

You can see that p95 latency is around the 500ms mark, and average latency about half that. While there are still occasional spikes, they are on the order of seconds, not minutes and return the expected data rather than timing out.

In this post I’ll dive into the problem, the old way of doing things, various infrastructure changes, and how the “threadweaver” algorithm works in detail.

Quick background

Streak’s CRM keeps track of boxes in a pipeline. A box is something like a sales deal or a project. Users can keep track of emails inside these boxes — we call that a “boxed thread” or “boxing an email”. Boxed email threads allow users to share certain emails with all users on their team. It looks like this:

The timeline on the left shows changes to the box including two email threads consisting of an example conversation from our own demo account.

We also support “autoboxing” — in the above example, you can specify that you want Streak to include all emails from the customer’s domain in the box. When enabled, any email sent to or from the customer from any user on your team (subject to each user’s sharing permission) will also be included in the box, automatically.

Unifying with TiDB and JGraph

Let’s say an email thread has 2 messages sent to 10 members of your team. These messages get delivered into everyone’s inboxes:

When you’re viewing the box timeline, you just want to see the one email thread, not the 10 copies from each person’s inbox.

So we unify all these individual threads together and only show the unique messages across all members of your team who have shared their emails. Since each email has a Message-ID header uniquely1 identifying it, we can deduplicate the emails regardless of whose inbox they are in.

This was done via a breadth first search using a simple SQL query for the set of users identified by gid (their Google-assigned ID) on the team:

SELECT DISTINCT m2.gid, m2.thread_id FROM messages m1
JOIN messages m2 USING (md5_rfc_message_id) 
WHERE (m1.gid, m1.thread_id) IN (:initialSet) AND m2.gid IN (:gidList)

The query runs on our TiDB cluster and the data is all nicely indexed so, for most Streak users, this was quite fast. When this query returns threads that weren’t in the initial set, we iterate on those until we get no new threads.

You can try it here. The query starts out with the initial set consisting of gid 1 and thread 1, and running it also finds gid 2 and thread 2. Plugging (2, 2) back into the query then also finds gid 3 and thread 3. Obviously a greatly simplified example, but it demonstrates how the query searches based on matching IDs.

The results are fed into JGraph where we construct a graph representation of all the emails we found and the relationships between them all. This is also where we filtered messages by the user’s custom permissions.

Must be dynamic

One of the reasons we used the above SQL query is that when we unify, the result needs to be dynamic. The unification should reflect changes in their team membership without having to perform lengthy recalculations to reunify the emails. Here’s a simple example where this comes into play:

In the image above, Alice has a thread of emails with message ids of 1, 2, 3 and Charlie has a different thread of emails with message ids 5, 6, 7. Because none match we can’t unify them into a single thread. However, let’s see what happens when Bob joins the team:

In the image above, Bob has a thread where message id 3 matches an email that Alice has. Similarly, message id 5 matches an email that Charlie has. These previously separate threads can now be unified together into a common thread.

You can try it here. Note that once we insert Bob’s message metadata, we can now connect Alice to Charlie when searching.

Gmail quirks

Gmail has a few interesting quirks that makes the volume of emails increase substantially in a few cases:

• messages with the same subject sent within a few days of each other get threaded together, even if there is no relationship between the emails within the thread. We’ve seen this a lot ourselves for automated emails and it’s great for cutting down the number of threads in a person’s inbox.

• when a thread has 100 messages, Gmail automatically creates a new thread. For long-running discussions, this ends up creating a bunch of new threads. We informally refer to this as the “multi-centi-thread problem” since each person receiving the emails (could potentially be everyone in the company) will have numerous 100 email threads all tied together.

Performance impact

Small teams saw results very quickly when they went to view a shared email thread. We could quickly identify and unify all threads across their team. But when there were a large number of emails across large teams, the Gmail quirks started causing huge performance issues as shown earlier:

Why is there a flat line at around the 180 second mark? That’s the limit we imposed on the length of a database connection, after which the database server gives up and returns an error.

Additionally, for large volumes of email when the database did return results, on occasion putting that data into JGraph could result in an Out of Memory error.

Practically speaking, for larger teams with a huge volume of emails they would click into a box that had a number of autoboxed threads, go and grab a coffee while they waited for the email data to load, then come back just to see that it returned nothing because the database timed out. Not a great experience!

Results were slightly incorrect

We also took a number of shortcuts in our implementation. In particular, keeping track of stats for these boxed threads could occasionally be inaccurate. By stats I mean things like the timestamp of the last sent or received email. Users use this to create views on their data like “show me all boxes where the last email was more than 3 days ago” or “show me boxes with a reply in the last week”.

Whenever we process a new email for a user, we see if that thread is part of an existing boxed thread. Doing this with full unification using the breadth-first database query was a non-starter due to the potential for heavy load. So we cheated: when processing emails, we ignored processing the email thread unless it was for the one user who boxed the thread.

Let’s say Alice, Bob, and Charlie are all participating on one email thread that was boxed by Charlie. Alice sends a reply to everyone, so there’s now an email in Alice’s sent folder plus emails in Bob and Charlie’s inbox, all with the same Message-Id. When we process Alice’s email, she didn’t box the thread so we ignore it. When we process Bob’s email, he didn’t box the thread either so we ignore that too. Finally we process Charlie’s email and since he was the one who boxed the thread, we update the stats and the last email timestamp accurately reflects the latest email:

This works for almost all emails, but where it goes wrong is if the person who boxed the thread no longer participates in the emails. Since we are ignoring everyone who didn’t box the thread, the stats never update and this impacted users’ workflows:

Because Charlie was removed as a recipient, and he was the one who boxed the thread, we never updated the stats since the remaining emails we process didn’t correspond with Charlie’s boxed thread and were ignored.

Finding a better algorithm

When you distill all this down from gids, thread ids, and message ids, there are a few simple theorems we can derive:

1. since we are unifying entire threads, all messages in a single thread are part of the same unified thread

2. when two messages in two different threads have the same Message-ID, they are part of the same unified thread

So there are really only two inputs we need to concern ourselves with: the thread id and the message id. If we can turn these into a unified thread id and know which individual user threads make up that unified thread, then it becomes a trivial lookup.

We can represent this as the following relationships, where “ut” is a unified thread:

Each threadId and messageId maps to a single unified thread, but a unified thread maps to the set of threadIds that reference it. Let’s model this using a simple scenario where rfc2 means the Message-ID value:

This scenario shows that user A sends a message to B, who then forwards it to C, who then forwards it to D, who then forwards it to E. If we process this in timeline order, one message at a time, we have:

1. Timeline 1

   1. Does thread “111” map to a unified thread? ❌
      Does rfc “1” map to a unified thread? ❌
      Assign each to a new unified thread value “ut_a”, and map “ut_a” to the set of {“111”}

   2. Does thread “222” map to a unified thread? ❌
      Does rfc “1” map to a unified thread? ✅ = “ut_a”
      Assign each to the existing unified thread value “ut_a” and add “222” to the “ut_a” set, which now contains {“111”, “222”}

2. Timeline 2

   1. Does thread “222” map to a unified thread? ✅ = “ut_a”
      Does rfc “2” map to a unified thread? ❌
      Assign rfc “2” to the existing unified thread value “ut_a”

   2. Does thread “333” map to a unified thread? ❌
      Does rfc “2” map to a unified thread? ✅ = “ut_a”
      Assign thread “333” to the existing unified thread value “ut_a” and add “333” to the “ut_a” set, which now contains {“111”, “222”, “333”}

3. Timeline 3

   1. Does thread “333” map to a unified thread? ✅ = “ut_a”
      Does rfc “3” map to a unified thread? ❌
      Assign rfc “3” to the existing unified thread value “ut_a”

   2. Does thread “444” map to a unified thread? ❌
      Does rfc “3” map to a unified thread? ✅ = “ut_a”
      Assign thread “444” to the existing unified thread value “ut_a” and add “444” to the “ut_a” set, which now contains {“111”, “222”, “333”, “444”}

4. Timeline 4

   1. Does thread “444” map to a unified thread? ✅ = “ut_a”
      Does rfc “4” map to a unified thread? ❌
      Assign rfc “4” to the existing unified thread value “ut_a”

   2. Does thread “555” map to a unified thread? ❌
      Does rfc “4” map to a unified thread? ✅ = “ut_a”
      Assign thread “555” to the existing unified thread value “ut_a” and add “555” to the “ut_a” set, which now contains {“111”, “222”, “333”, “444”, “555”}

Let’s compare this to how things would’ve worked with the breadth-first SQL query. We would start with thread “111” and find thread “222”. We repeat with thread “222” and find “333”, and so on until we eventually find “555” and a subsequent query doesn’t return any new threads.

But once we’ve written the data in the new format, let’s consider how we lookup the information. We start with thread “111” and lookup the corresponding unified thread, finding “ut_a”. We then lookup “ut_a” and find the set {“111”, “222”, “333”, “444”, “555”}. And that’s it. We’ve turned a recursive breadth-first SQL query into two trivial lookups.

Handling all cases

Remember the earlier example when we processed Alice’s email and Charlie’s email and each thread is distinct, but once Bob joins the team his emails unify the two?

If we processed this using the above timeline example, Alice’s emails would be “ut_a” and Charlie’s emails would be “ut_b” since they have no emails in common. But when Bob joins, we now have an exception: message 3 in Bob’s thread maps to “ut_a” and message 5 in Bob’s thread maps to “ut_b”. How do we handle this?

It’s simple: we store unified thread merges as well. This introduces a new relationship:

When the unified thread for a user’s email thread and the unified thread for one of the messageIds in that thread are different, we record a merge with the unordered pair of unified threads being merged. In the Alice, Bob, and Charlie example that would result in merges containing { (“ut_a”, “ut_b”) }.

This adds one more step to finding all threads given a starting thread:

1. lookup starting threadId to find unified thread

2. lookup any merges for the unified thread — if any, recurse finding more merges

3. from the set of all unified threads identified, lookup the threadIds

While step 2 is recursive, in practice merging threads happens rarely and it’s even rarer that there are multiple merges. To put this in numbers, the storage needed to represent the mapping of threadId → ut, messageId → ut, and ut → {threadId} takes up 9.6 TB. The storage needed to represent all unified thread merges is only 1.4 GB, almost 4 orders of magnitude less data.

Implementing the data storage layer

One requirement for making this algorithm work is that when looking up and storing data, we need strong consistency guarantees. My first thought was to use Redis given that it’s single threaded and implementing this using Lua scripting provides for atomic changes. And, on a sufficiently large system, Redis can support 100K+ queries per second which is within our target of needing to process peak rates of 30K/second. And should we need more capacity, Dragonfly claims an order of magnitude improvement over that.

However, what makes Redis fast is that the entire dataset is loaded in memory, and some back of the envelope math3 showed that the storage requirements would range from an estimate of 9.2 TB typical to over 37 TB in the worst case scenario, and would be impractical to make this work.

Scylla

The next contender was Scylla, a NoSQL system touted as being an improvement over Cassandra. While its storage model is eventually consistent, it also offers lightweight transactions (LWT) which provide for the strong consistency we require.

Here’s what the schema looked like:

-- 1:1 mapping of (gid, thread_id) -> unified thread id
create table if not exists gid_thread_to_ut (
    gid             ascii,
    thread_id       ascii,
    ut_id           ascii,		-- some "$gid:$thread_id"
    primary key ((gid, thread_id))	-- composite partition key with both gid and thread_id
);

-- reverse lookup of (gid, thread_id) by ut_id
create index if not exists on gid_thread_to_ut (ut_id);

-- 1:1 mapping of rfc message id -> unified thread id
create table if not exists rfc_to_ut (
    md5_rfc_message_id  ascii,
    ut_id               ascii,      -- some "$gid:$thread_id"
    primary key (md5_rfc_message_id)
);

-- reverse lookup of rfc message id by ut_id
create index if not exists on rfc_to_ut (ut_id);

-- n:n mapping, merging source unified thread -> target
create table if not exists ut_merge (
    source_ut_id    ascii,      -- source unified thread merged from
    target_ut_id    ascii,      -- target unified thread merged to
    primary key (source_ut_id, target_ut_id)
);

-- reverse lookup of source unified thread by target unified thread
create index if not exists on ut_merge (target_ut_id);

The data insertion from Kotlin was super simple:

fun setUTForGidThread(gid: String, threadId: String, defaultUT: String): String {
        val query =
            SimpleStatement.builder("INSERT INTO gid_thread_to_ut (gid, thread_id, ut_id) VALUES (?, ?, ?) IF NOT EXISTS")
                .addPositionalValues(gid, threadId, defaultUT)
                .build()

        val resultSet = scyllaSession.execute(query)
        val row = resultSet.one()
        val result =
            if (row!!.getBoolean("[applied]")) {
                defaultUT
            } else {
                row.getString("ut_id")!!
            }

        return result
    }

The nice thing about Scylla’s LWT inserts is that the “if not exists” will insert the value if and only if a value doesn’t already exist and, in any case, the result of the operation either indicates that the insert was applied, in which case we know that our defaultUT value was inserted, or it returns the existing record in which case we fetch the ut_id value.

Similar for messageId:

fun setUTForMessageId(rfcMessageId: String, defaultUT: String): String {
        val md5RfcMessageId = Utils.md5Hash(rfcMessageId)

        val query =
            SimpleStatement.builder("INSERT INTO rfc_to_ut (md5_rfc_message_id, ut_id) VALUES (?, ?) IF NOT EXISTS")
                .addPositionalValues(md5RfcMessageId, defaultUT)
                .build()

        val resultSet = scyllaSession.execute(query)
        val row = resultSet.one()
        val result =
            if (row!!.getBoolean("[applied]")) {
                defaultUT
            } else {
                row.getString("ut_id")!!
            }

        return result
    }

And finally to record a merge:

fun setUTMerge(sourceUT: String, targetUT: String) {
        val querySource =
            SimpleStatement.builder("INSERT INTO ut_merge (source_ut_id, target_ut_id) VALUES (?, ?) IF NOT EXISTS")
                .addPositionalValues(sourceUT, targetUT)
                .build()
        scyllaSession.execute(querySource)
    }

Whenever we process an email for a user, we know the user’s gid as well as the threadId. From this, we assume that if a unified thread doesn’t already exist, we’ll use the current thread, which we construct using a string template:

val defaultUT = "$gid:$threadId"

So the core algorithm, which I dubbed “threadweaver”, is super simple:

val existingGidThreadUt = getUnifiedThreadForGidThreadId(gid, threadId)

when (existingGidThreadUt) {
    null -> {
        // thread isn't yet associated with a unified thread
        val defaultUT = "$gid:$threadId"
        val utFromMessageId = setUTForMessageId(rfcMessageId, defaultUT)
        val utFromGidThread = setUTForGidThread(gid, thread, utFromMessageId)
        if (utFromGidThread != utFromMessageId) {
            setUTMerge(utFromGidThread, utFromMessageId)
        }
    }
    else -> {
        // thread already associated with a unified thread
        val utFromMessageId = setUTForMessageId(rfcMessageId, existingGidThreadUt)
        if (existingGidThreadUt != utFromMessageId) {
            setUTMerge(existingGidThreadUt, utFromMessageId)
        }        
    }
}

This is trivial to perform on every incoming email we process. We either assign a unified thread, or retrieve the one that already exists for both threadId and messageId. If the two differ, record a merge.

This worked great and comprehensive tests showed that it was functioning correctly, but in practice we ran into a lot of problems with Scylla operationally. Since the only way to guarantee strong consistency is the use of LWT, this made maintenance problematic. While we never modified any data once inserted, part of maintaining a Scylla cluster involves compaction and repair. However, our exclusive use of LWT to the tune of thousands of writes per second appeared to hinder any attempts to perform maintenance. Compaction and repairs never appeared to complete successfully. What should have been simple and routine turned out to be impossible. There’s a comment4 on Hacker News which, unfortunately, echoes my experience.

Given these operational issues, we couldn’t trust our data to Scylla. A shame as the simplicity of “insert … if not exists” made for incredibly easy to understand writes that matched the algorithm beautifully.

Bigtable

We had used Google’s Bigtable extensively in the past. We moved off of it as the workload was more suited to a relational database, but when used for its strengths, it performed exceptionally well. So much so that it earned an unofficial nickname of “Honey Badger” because, like that famous YouTube video5, it handled data effortlessly. Need to fetch thousands of rows? “Honey badger don’t care!” How about millions of rows? “Honey badger don’t care!” Do your rows have thousands of columns? You guessed it: “Honey badger don’t care!” Bigtable’s throughput is insanely fast.

Key to making this work is figuring out how to take advantage of strong consistency. Fortunately, by default a single Bigtable cluster provides strong consistency6 on single row operations which is sufficient. To implement the equivalent of “insert … if not exists” for Bigtable, we can take advantage of conditional writes (or “mutations” in Bigtable’s parlance).

Here’s what the Kotlin code looks like:

val filter =
    FILTERS.chain()
        .filter(FILTERS.family().exactMatch("gt"))
        .filter(FILTERS.qualifier().regex(".+"))
        .filter(FILTERS.value().regex(".+"))

// only perform the mutation if the filter doesn't match
// (ie: no matching qualifiers/values)
val mutation = Mutation.create()
    .setCell("gt", defaultUT, timestamp, "_")
val conditionalMutation = ConditionalRowMutation
    .create("gid-thread-to-ut", rowKey)
    .condition(filter)
    .otherwise(mutation)

client.checkAndMutateRow(conditionalMutation)

I’ll translate what this is doing:

• a conditional mutation evaluates the filter provided by .condition which, if true, runs the mutation provided by .then (not used here) or, if false, runs the mutation provided by .otherwise

• the filter used by the condition checks to see if the “gt” column family (a distinct group of columns) contains a qualifier (column name) that contains at least one character having a value that contains at least one character.

• the .otherwise mutation operates on the table “gt-thread-to-ut” for the provided rowKey, setting the cell value to the provided defaultUT value.

The result is that nothing happens if there’s any value already, since an existing value means that this thread is already mapped to a unified thread and, once mapped, we never change it. But if there’s no value, then the mutation executes and we’ve atomically set the mapping from that thread to the unified thread. Because single row operations have strong consistency, there’s no possibility of a race condition here.

The same type of operations are performed for conditionally setting the messageId to unified thread mapping as well as any unified thread merges.

The one difference from Scylla is that this operation doesn’t return whether the mutation took place or if there was already an existing value. So we simply query for the rowKey again to find the unified thread, whether it was the one we just inserted (mutated) or whether it was already there. But those are implementation details; the core threadweaver algorithm remains the same:

val existingGidThreadUt = getUnifiedThreadForGidThreadId(gid, threadId)

when (existingGidThreadUt) {
    null -> {
        // thread isn't yet associated with a unified thread
        val defaultUT = "$gid:$threadId"
        val utFromMessageId = setUTForMessageId(rfcMessageId, defaultUT)
        val utFromGidThread = setUTForGidThread(gid, thread, utFromMessageId)
        if (utFromGidThread != utFromMessageId) {
            setUTMerge(utFromGidThread, utFromMessageId)
        }
    }
    else -> {
        // thread already associated with a unified thread
        val utFromMessageId = setUTForMessageId(rfcMessageId, existingGidThreadUt)
        if (existingGidThreadUt != utFromMessageId) {
            setUTMerge(existingGidThreadUt, utFromMessageId)
        }        
    }
}

Performance

The honey badger of databases doesn’t disappoint. We routinely read anywhere from 50,000 rows per second to almost 250,000 rows per second representing roughly 64 MB/s to 256 MB/s of data transmitted.

Similarly, Bigtable easily handles writes:

Bigtable also simplifies management. While Scylla would’ve required continuous maintenance with compacting and repairing data plus working around things like VM outages, Bigtable is a managed service. We set the backup schedule, specify how much storage we want per node, set the target CPU% to trigger autoscaling, and Google handles everything. It’s a completely hands-off experience.

Every email, everywhere, all at once

Astute readers may have noticed that this unification algorithm doesn’t have any concept of what team the email we are processing is for. The only inputs under consideration are what thread a message is in and its Message-ID header value. It will happily unify threads for anyone who uses Streak if they were sent the same email having the same Message-ID header value. The result is that every email we process is unified across Streak’s entire userbase. And the simple write-time algorithm has constant O(1) performance.

We restrict emails by team trivially by filtering the results by the list of gid values for our team members. We resolve a starting thread to a set of one or more unified threads, each of which is a self-contained row. It’s then a simple matter to filter the cells7 in the row to those that match our list of gid values.

Once retrieved, we then need to apply individual user permissions based on what their sharing settings are. To do this, we no longer use JGraph as that was overkill, but rather apply the permissions directly on top of the efficiently retrieved data.

User reactions

This log scale graph shows overall performance when fetching the timeline for a box along with all unified threads, showing the same performance improvement as the graphs at the top of the post. The very right side of the graph is from when the change was rolled out to all users:

As mentioned at the beginning, this went from “I’m gonna grab a coffee” levels of performance to “I blinked and my data loaded”. Some happy expletives were shared. In addition to instant timeline retrieval, multiple other areas of Streak saw performance gains:

• when viewing 50 or 100 threads in your Gmail inbox, identifying which of those are associated with boxed threads is much, much faster

• processing autoboxed emails is not only faster but also fully correct, even for the edge cases where the user who boxed a thread no longer participates

• quicker retrieval of email threads/envelopes your team has shared with you

Infrastructure changes

Because of the performance gains from the threadweaver algorithm, we no longer needed as many VMs for our TiDB cluster, so we scaled that down as both our storage requirements and query volume dropped dramatically. Additionally, our API servers were no longer tied up waiting for the results of unification, so we ended up needing fewer pods for our API deployment in Kubernetes8. In fact, we used to maintain a separate “threads” API deployment because of the slow performance so that a backlog of thread-related requests wouldn’t impact our regular API traffic. But with threadweaver, performance is little different from any other endpoint, so those API endpoints are now handled by our main deployment.

Engineering at Streak

We work on many interesting challenges affecting billions of requests daily involving many terabytes of data. For more information, visit https://www.streak.com/careers

Leave a comment

Subscribe now

Deployment was fairly simple: we push our main branch to GitHub, CircleCI picks up the change, builds and tests everything, then pushes the docker image to GCR (Google Container Registry). Our deployment.yaml file had some lines like this:

      containers:
        - image: gcr.io/ourproject/imagename:abcd123

Here, abcd123 is the short commit hash for the new image.

The old way

To deploy to production, a developer would edit this file, change the commit hash to the latest, and run

kubectl apply -f deployment.yaml

This triggered a process that was largely hands off. Kubernetes picks up the change and starts transforming the deployment into the new desired state. At the time, the deployment ran at a fixed size of 150 pods. By modifying the commit hash, Kubernetes applies what’s known as a “rolling update”, which is governed by a few strategies. At the time we had:

  strategy:
    rollingUpdate:
      maxSurge: 10
      maxUnavailable: 10

Since we’re not increasing the size of the deployment, but rather replacing pods in the fixed size deployment, the maxUnavailable setting is most relevant. 10 pods begin the termination process. This isn’t instant; our deployment handles both API requests that complete as quickly as possible as well as processing asynchronous tasks. The vast majority of those tasks are handled as quickly as API requests, but a small percentage are longer running when they need to process large volumes of data. We want to give those time to complete, but still force a shutdown if it takes longer than necessary. That is handled by another setting:

      terminationGracePeriodSeconds: 700

This allows Jetty to gracefully shut down if it’s able, otherwise Kubernetes will forcefully terminate the pod after 700 seconds (just under 12 minutes) which is more than enough for even the longest tasks1. Our API sits behind Google Cloud Load Balancer which performs SSL termination and routes traffic to the service associated with the deployment. The service keeps track of which pods are ready via the readiness probe, so live traffic is only routed to pods which are online and serving.

The end result was 150 pods shutting down 10 at a time which, in the absolute worst case scenario, takes the full 700 seconds each before being replaced with 10 new pods, resulting in a new deploy taking up to 3 hours to go out. In practice, the worst case never occurred and most deploys were in the 20 to 40 minute range.

Typically this wasn’t an issue. We could still deploy many times throughout the day. The downside is that if you had a critical bug that you needed to fix, it would take much longer for it to reach all pods unless you resorted to force terminating the pods, which should be a last resort as you are also terminating active user requests and background tasks.

Blue-green deployments

The concept behind the blue-green deployment strategy is that you have your existing deployment (the “blue” one) and you want to release a new deployment (the “green” one). Rather than doing a replacement of your running code, you can temporarily run each one in parallel. This allows you to:

• quickly scale up the new deployment without being dependent on simultaneously scaling down the old deployment

• keep both around for a while in case there’s a critical issue in the new deployment so you can swap back

• increase flexibility in what you deploy since once you have the infrastructure for splitting traffic between two deployments, you can choose to split traffic in additional ways

The last point on splitting traffic is where you can do some interesting things.

Design goals

We decided to go with Envoy sitting between Google Cloud Load Balancer and our service. Envoy is a proxy originally created by Lyft and counts among its users Amazon AWS, Google, Netflix, Stripe, and many others. It’s what allows us to control routing of traffic at a more granular level than what we had before. We had a number of goals in the new design:

• Speed up the deploy process from approximately 30 minutes to as fast as possible

• Have the ability to quickly rollback a bad deployment

• Support the use of canary deployments. Like the proverbial canary in the coalmine, a canary deployment allows you to test changes to a small subset of users before rolling out to everyone. There were a few types we wanted:

  • Deploy to some % of our traffic. For example, 1% or 10%. This lets us do things like make performance tweaks and measure how the new code performs in production.2

  • Deploy to specific customers by their domain name.

  • Deploy to specific users by email address.

  • Deploy standalone canaries for independent testing.

• All the above must work correctly with no dropped requests

Additionally, it was a good opportunity to explore autoscaling. Rather than a fixed number of nodes having a fixed cost, we’d like the deployment to scale automatically based on load. During peak hours, the deployment should scale to as many pods as is required to handle traffic. For off-peak hours, there’s no sense paying for pods which are sitting idle, so the deployment should scale down to a reasonable minimum.

The scaling targets are based on two metrics:

1. CPU utilization. Most of our workload is I/O intensive, involving fetching from some cache, datastore, or remote API, then doing permission checks, performing minimal transformations of the data, then responding with the data. Our servers don’t deal with HTTPS encryption and decryption (that’s handled for us by Google Cloud Load Balancer), but for the occasional request that does consume significant CPU resources we want to ensure we have some headroom.

2. Thread utilization. We’ve configured each pod in our deployment with 40 threads for handling requests. When all 40 threads are busy, additional requests get added to a queue until a thread becomes free. Each request may spawn its own threads to do parallel I/O for example, but those are independent.

We set a target of 50% for each. In practice, we almost never see high CPU utilization for the reasons mentioned above. The goal with thread utilization is that by running at 50% capacity, we are able to quickly handle spikes up to double our current request volume before the new capacity goes online.3

For some quick back of the envelope math, if we’re running a fixed 150 pods with each pod having 40 threads for responding to requests, that’s 6000 simultaneous threads. If a typical API request takes 100ms to complete, we can handle 60,000 requests per second before we would need to scale up.

Horizontal Pod Autoscaler

As with non-cloud server architecture, you can scale any given service both horizontally and vertically. Vertical scaling means that if you’ve hit capacity while running on a fixed number of servers with 2 CPU and 8GB of RAM, you can upgrade the specs of those same number of servers to 4 CPU and 16GB of RAM. If that’s not enough, you move to 8 CPU / 32 GB and so on.

Horizontal scaling is where you hit capacity and you just keep adding more servers with the same specs as before. Kubernetes does this via the Horizontal Pod Autoscaler (HPA). The way it works is via a YAML file describing how you want Kubernetes to scale a given resource, such as a deployment. The important bits are:

• minReplicas — for a system serving up production traffic, you never want to scale down to a single replica, so this is where you set your reasonable minimum that is capable of handling the transition from off-peak to regular traffic.

• maxReplicas — on the opposite end, you want to control your costs and be aware of any issues that are causing unexpectedly high utilization. Setting the maximum number ensures that you don’t scale beyond this number.4

• scaleTargetRef — this is where you specify what thing the HPA needs to scale up or down. In our case, this is kind: Deployment and name: <deployment name>

• metrics — here you specify your metric(s) that Kubernetes monitors as well as the target type and value.

For our thread metric, we set the target at 500m, which can be read as “500 milli” and is equivalent to a value of 0.5 or 50%. The official Kubernetes site has a great writeup of how it works, the scaling algorithm, and more.

Setting up Envoy

The simple description of how Envoy works is that it operates on clusters and routes. Think of a cluster as essentially where you want to send requests to. A route then describes how to specify which request goes to which cluster. In our case, the cluster identifies that we want to send API traffic to our Kubernetes deployment and the route identifies which paths on the API to send. The simplest route is to simply send everything to one single cluster:

    routes:
    - match:
        prefix: "/"
      route:
        cluster: api

However, for blue/green deployments we have the concept of a primary API and a secondary API. The primary represents the current, active (blue) deployment and the secondary is then the new (green) deployment. Envoy has a concept of a weighted cluster which looks like this:

    routes:
    - match:
        prefix: "/"
      route:
        weighted_clusters:
          clusters:
            - name: secondaryapi
              weight: 10
            - name: primaryapi
              weight: 90

This will direct 10% of traffic to the new deployment while directing 90% of traffic to the old deployment. As the new deployment comes online and more and more pods are ready, we can gradually increase the weight of the secondary cluster and reduce the weight of the primary cluster.

But how do we know when the new deployment is ready to serve traffic? And how do we correctly update Envoy’s configuration?

Weighing traffic

Kubernetes provides a nice way to get the current status of a deployment. Let’s say we just deployed our API with a target of 80 replicas. Running:

kubectl get deployment api-deployment -o yaml

Returns output which includes some status information:

status:
  availableReplicas: 19

Given that our example target is 80 replicas, we know that 19/80 or 23.75% are ready to serve traffic. We round this down to the nearest integer, 23, and use that as the weight for the secondary API. Since Envoy’s weighted clusters must add to 100, the weight of the primary API is then 77. We then update our Envoy configuration with these values and the route looks like:

    routes:
    - match:
        prefix: "/"
      route:
        weighted_clusters:
          clusters:
            - name: secondaryapi
              weight: 23
            - name: primaryapi
              weight: 77

Envoy supports updating its runtime via a symbolic link swap, which is the only way to atomically modify a file. To support this, we store all our Envoy configuration files within a single Kubernetes configmap file and it’s trivial to overwrite the configmap with the new contents. To convert this into a link swap in the Envoy deployment, we make use of a sidecar container running Crossover which watches for changes in the configmap and, when it detects changes, it reads the contents of the configmap and outputs the new file contents to the filesystem, then performs a link swap on each. Envoy listens for link swaps and then applies the changes to its runtime environment.

By checking the status of the new deployment in a loop every few seconds, our deploy script keeps updating the deployment percentage as new pods become ready to serve traffic, and Kubernetes and Envoy handle the rest. Once we reach 100%, we promote the secondary to primary, and the deploy is done.5

This allows us to quickly roll out a new full deployment where the end goal is the new deployment having 100% of the traffic as well as deploying to some developer-specified percent of traffic, where we update Envoy until we reach the target percent.

Routing by email or domain

In addition, we can route requests based on the user’s email address or domain name. This is accomplished via changing how routes match. For normal requests, our UI appends ?email=user@example.com to API queries. Additionally, requests that end up creating asynchronous tasks append a header value X-Streak-Request-Email with the email address. We can easily match both of these with the following match definitions:

    - match:
        prefix: "/"
        query_parameters:
          name: "email"
          string_match:
            safe_regex:
              regex: {{DOMAIN_OR_EMAIL_REGEX}}
      route:
        # (route definition omitted)
    - match:
        prefix: "/"
        headers:
          name: "X-Streak-Request-Email"
          safe_regex_match:
            regex: {{DOMAIN_OR_EMAIL_REGEX}}
      route:
        # (route definition omitted)

This is a template to which we perform variable substitution, replacing {{DOMAIN_OR_EMAIL_REGEX}} with the appropriate email address or domain regex.

Rolling back a bad deploy

Another of our design goals was to quickly roll back a bad deployment. Our deploy script supports a swap command. Implementing this is easy. We first check that the old deployment hasn’t scaled down too far and, if it hasn’t, we simply update our Envoy cluster definition and switch the endpoint for the primaryapi and secondaryapi. Each cluster’s definition looks something like this simplified example:

resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: primaryapi
  type: STRICT_DNS
  dns_lookup_family: V4_ONLY
  lb_policy: LEAST_REQUEST
  load_assignment:
    cluster_name: primaryapi
    endpoints:
    - lb_endpoints:
        endpoint:
          address:
            socket_address:
              address: {{PRIMARY_CLUSTERIP_SERVICE}}.default.svc.cluster.local

Again, we do templated variable substitution here and update the Kubernetes configmap, which gets picked up by Crossover, which performs a link swap, which gets picked up by Envoy and we’ve swapped back in seconds.

Deploy script

We can perform deploys using a simple command line script. Here are the current suite of deploy options the deploy command supports:

In addition, we can also cancel, check the status of a deployment, or swap back to the previous deploy via other commands.

Testing

While this all sounds good in theory, with all these moving parts how can we verify that Google Cloud Load Balancer’s interaction with Envoy and its routing of traffic to a Kubernetes service, which then hits our API deployment is functioning correctly given that we’re also introducing Horizontal Pod Autoscaling into the mix?

Enter Tsung

Tsung is Erlang-based software designed for load testing. To use it, you provide a configuration file describing the traffic you want to send. In my testing, I used a simple config:

<tsung loglevel="info" dumptraffic="protocol" >
  <clients>
    <client host="localhost" use_controller_vm="true" maxusers="1000000" />
  </clients>
  <servers>
    <server host="api.streak.com" port="443" type="ssl" weight="100" />
  </servers>
  <load>
    <arrivalphase phase="1" duration="1" unit="minute">
      <users arrivalrate="5" unit="second" />
    </arrivalphase>
    <arrivalphase phase="2" duration="1" unit="minute">
      <users arrivalrate="10" unit="second" />
    </arrivalphase>
    <arrivalphase phase="3" duration="1" unit="minute">
      <users arrivalrate="15" unit="second" />
    </arrivalphase>
    <arrivalphase phase="4" duration="1" unit="minute">
      <users arrivalrate="20" unit="second" />
    </arrivalphase>
    <arrivalphase phase="5" duration="1" unit="minute">
      <users arrivalrate="25" unit="second" />
    </arrivalphase>
    <arrivalphase phase="6" duration="30" unit="minute">
      <users arrivalrate="30" unit="second" />
    </arrivalphase>
  </load>
  <sessions>
    <session name="consume-thread" probability="80" type="ts_http">
      <request>
        <http url="/path/to/threadSleep" method="GET" version="1.1">
          <www_authenticate userid="YOUR_API_KEY_HERE" passwd="" />
        </http>
      </request>
    </session>
    <session name="hello-world" probability="20" type="ts_http">
      <request>
        <http url="/path/to/hello" method="GET" version="1.1">
          <www_authenticate userid="YOUR_API_KEY_HERE" passwd="" />
        </http>
      </request>
    </session>
  </sessions>
</tsung>

The “load” section describes a timeline of traffic to be sent:

• initial ramp-up phase sending 5 new users per second for 1 minute

• increasing to 10 per second for 1 minute, then 15 per second, 20 per second, 25 per second

• then sustaining at 30 new requests per second for 30 minutes

The “sessions” section defines what kind of traffic to send:

• 80% of requests are to a “threadSleep” endpoint which ties up the request for 5 seconds

• 20% of requests are to a “hello” endpoint which returns a simple “hello world” response

As the majority of requests consume the thread for 5 seconds and we are sending many new users per second, this nicely simulates a gradual increase in traffic which lets the HPA scale up the deployment so that, over time, we are serving hundreds of simultaneous users.

A successful test of Envoy, the new deployment system, and Horizontal Pod Autoscaling will mean that out of all the requests we send, 100% of the requests will result in a successful response with zero timeouts and zero dropped requests.

Results

I’ll quote from the report I produced after implementing this:

Highlights:

• 20,865 requests sent (80% taking 5 seconds, 20% returning “hello world” immediately)

• 100% of requests succeeded with a 200 OK HTTP status code. Zero server failures scaling up, during cutover, or after cutover.

• The only problems were errors connecting to the API which was limited to 10 connections (out of 20K+, representing 0.05% of requests attempted) and was likely due to being on a wifi connection from my laptop.

Some Tsung-generated graphs in the report shows that as it scaled up the number of user requests over time, the API responses kept pace when scaling up and during cutover:

Conclusion

The change in how we deploy our API has been very successful as demonstrated by comprehensive testing as well as the thousands of deployments we’ve done since this system has been introduced. Deployment times were reduced from an average of 30 minutes to now taking roughly 3 minutes. Additionally, we now support a number of new blue/green strategies that give us additional flexibility when we want to test out changes in production for a single user, entire team, or for some chosen percentage of all traffic.

Lastly, because our API now scales up and down dynamically via the use of Horizontal Pod Autoscaler, our GCP bill has benefited from the lower utilization during off-peak hours:

Engineering at Streak

We work on many interesting challenges affecting billions of requests daily involving many terabytes of data. For more information, visit https://www.streak.com/careers

OpenAI recently released support for function calls in their chat completions API. The example they give shows the ability to define a function such as get_current_weather(location: string) so that when a user asks “What’s the weather like in Boston right now?”, the chat completions API intelligently extracts parameters from the user’s input, responding with JSON to your server describing what function to call, eg: name: get_current_weather, arguments: {"location": "Boston"}. This allows you to extract the function name and argument values from the response, call an external weather API to get the temperature in Boston, then respond with data such as {“temperature": “22”, “unit": “celsius", “description": “Sunny”} which ChatGPT intelligently formats back as a response such as “The weather in Boston is currently sunny with a temperature of 22 degrees Celsius.”

Streak has a number of projects on our AI roadmap and this appeared useful for one in particular. But first, a bit of background.

Solving the “blank slate” problem

A challenge for most SaaS products is that when users sign up, they are left staring at a blank slate and are unsure where to start. A popular way to handle this is via your onboarding process by providing tutorial videos, helpful walkthroughs, context-sensitive help, and so on. These are often necessary but not sufficient as nothing beats having the user’s own data available to use within the SaaS product. We tackled this a while ago in our onboarding flow: when the user creates their first pipeline, our system looks at the user’s Gmail sent items and displays a “Quick Add” screen, offering to create boxes for the contacts they’ve sent emails to.

This has been incredibly useful to customers as the blank slate is now replaced with their own data where they can see a number of magic columns including when they last emailed the contact, when the contact last emailed them, and so on. It’s then easy to see who they should follow up with, review a history of all emails from everyone on their team, and take appropriate action.

Experimenting with AI and forcing JSON output

We thought a good upgrade from this functionality is to intelligently categorize all their emails. Rather than just looking at emails the user had recently sent, what if we looked at all recent email threads and used AI to categorize them based on our pipeline templates into Sales, Hiring, Job Search, Investor Relations, Fundraising, Real Estate, and so on? Someone in HR could use this to identify those who had applied as well as identify candidates the HR team had reached out to directly.

I tackled this as an R&D project, using emails in my own account to see how well ChatGPT 3.5 could classify emails as either HIRING or NOT_HIRING. I used the chat completions API and divided the input into the system message, which provided some initial prompt engineering and some background. Part of the system message primed ChatGPT with few-shot prompting as well as the expected output as structured JSON:

For example, given the inputs:

###

[0] Subject: Resume
Snippet: Hello, I've attached my resume and am applying for the content writer position

[1] Subject: Lunch on Tuesday
Snippet: Hi James, it's been a while! We should catch up this Tuesday as I wanted to go over the sales numbers

[2] Subject: Ad for Widgets
Snippet: Hi Doug, I'd like to be considered for the online ad posted on Facebook. Please find attached my job history.

###

You should output a JSON array with one object per email:
```
[ 
  {
    "index": 0, 
    "category": "HIRING", 
    "explanation": "Email mentions attaching a resume and that they are applying for a position" 
  }, 
  { 
    "index": 1, 
    "category": "NOT_HIRING", 
    "explanation": "Email appears unrelated to hiring"
  },
  {
    "index": 2,
    "category": "HIRING",
    "explanation": "Email mentions an online ad and they have included their job history"
  }
]

Asking for an explanation is helpful to understand the rationale behind the category. The user message is then similar to the few-shot prompts but includes a trailing output hint:

[0] Subject: Resume Snippet: Hello, I've attached my resume and am applying for the content writer position

[1] Subject: Lunch on Tuesday Snippet: Hi James, it's been a while! We should catch up this Tuesday as I wanted to go over the sales numbers

[2] Subject: Ad for Widgets Snippet: Hi Doug, I'd like to be considered for the online ad posted on Facebook. Please find attached my job history.

Output:
```

The three backticks indicates that the JSON output is to follow, and ChatGPT reliably supplies the JSON array with objects as specified in the system message. This makes it easy to use a JSON parser to deserialize the text into our internal classes in Kotlin.

Migrating to function calls

To use function calls, you specify the function definitions using JSON schema. For our example, I want ChatGPT to call a classify_email function for a given message and provide the category and an explanation. The request JSON looks like the following:

{
  "model": "gpt-3.5-turbo-0613",
  "messages": [
    {
      "role": "system",
      "content": "You are an assistant designed to analyze information from emails and assign the category that best matches the content."
    },
    {
      "role": "user",
      "content": "Subject: Resume\\nSnippet: Hello, I've attached my resume and am applying for the content writer position"
    }
  ],
  "functions": [
    {
      "name": "classify_email",
      "description": "Classify an email into a category",
      "parameters": {
        "type": "object",
        "properties": {
          "category": {
            "type": "string",
            "enum": [
              "SALES",
              "PROJECTS",
              "BUSINESS_DEV",
              "HIRING",
              "JOB_SEARCH",
              "INVESTOR",
              "FUNDRAISING",
              "ORDERS",
              "REAL_ESTATE",
              "SUPPORT",
              "UNKNOWN"
            ],
            "description": "The category to classify the email into"
          },
          "explanation": {
            "type": "string",
            "description": "An explanation of why the email belongs to the specified category"
          }
        },
        "required": ["category", "explanation"]
      }
    }
  ],
  "function_call": { "name": "classify_email" }
}

Here you can see there’s a system message which provides some background. Note: I’ve omitted some additional information from the system message describing each category. The user message supplies the subject and snippet for the email in question.

There are two important parts to making function calls work with ChatGPT. The first is supplying the functions parameter with the definition of your function. Because this uses JSON schema, I can supply a category property as an enum type and specify the allowable values. The second is the function_call property which, when you specify a name of the function to call, tells ChatGPT that it must call the function.

The response from the chat completions API indicates a function call with the information we’re looking for:

{
  "id": "chatcmpl-<redacted>",
  "object": "chat.completion",
  "created": 1687490210,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "function_call": {
          "name": "classify_email",
          "arguments": "{\\n  \\"category\\": \\"HIRING\\",  \\"explanation\\": \\"Email mentions attaching a resume and that they are applying for a position.\\"\\n}"
        }
      },
      "finish_reason": "stop"
    }
  ]
}

Unlike OpenAI’s weather example, we don’t need to call the API again to provide a response that it can intelligently respond with to the user; the goal here is to get back structured data and avoid having to coerce ChatGPT into transforming the output into JSON. I can trust that the arguments property is JSON already and it’s trivial to parse and extract the category and explanation.

However, this is only a single message. When processing hundreds or thousands of items, I’ve found there’s a sweet spot in optimizing your queries to minimize response time and maximize token utilization combined with making parallel API requests. Too many parallel requests means not only are you burning through tokens faster, but you end up getting rate limited with 429 errors. In my case, both the system message and function definition are identical across all emails. So I can cut down on token usage by supplying the system message, the function definition, and then multiple user messages (one for each email).

I had hoped ChatGPT would respond with one function call per user message but it only responded with a single function call, classifying only one single email. The trick turned out to be have the function accept an array of messages, plus specifying the object definition for each message separately.

Here’s what the complete request looks like:

{
  "model": "gpt-3.5-turbo-0613",
  "messages": [
    {
      "role": "system",
      "content": "You are an assistant designed to analyze information from emails and assign the category that best matches the content."
    },
    {
      "role": "user",
      "content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
    },
    {
      "role": "user",
      "content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
    },
    {
      "role": "user",
      "content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
    },
    {
      "role": "user",
      "content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
    },
    {
      "role": "user",
      "content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
    },
    {
      "role": "user",
      "content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
    },
    {
      "role": "user",
      "content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
    },
    {
      "role": "user",
      "content": "ThreadID: <redacted>\\nSubject: <redacted>\\nSnippet: <redacted>"
    }
  ],
  "functions": [
    {
      "name": "classify_email",
      "description": "Called to classify emails",
      "parameters": {
        "type": "object",
        "properties": {
          "messages": {
            "type": "array",
            "items": { "$ref": "#/$defs/message" },
            "description": "Array of emails to classify"
          }
        },
        "$defs": {
          "message": {
            "type": "object",
            "properties": {
              "threadId": {
                "type": "string",
                "description": "The ThreadID of the email"
              },
              "category": {
                "type": "string",
                "enum": [
                  "SALES",
                  "PROJECTS",
                  "BUSINESS_DEV",
                  "HIRING",
                  "JOB_SEARCH",
                  "INVESTOR",
                  "FUNDRAISING",
                  "ORDERS",
                  "REAL_ESTATE",
                  "SUPPORT",
                  "UNKNOWN"
                ],
                "description": "The category to classify the email into"
              },
              "explanation": {
                "type": "string",
                "description": "An explanation of why the email belongs to the specified category"
              }
            },
            "required": ["threadId", "category", "explanation"],
            "description": "The message to classify"
          }
        },
        "required": ["messages"]
      }
    }
  ],
  "function_call": { "name": "classify_email" }
}

This turns the function from classify_email(category: string, explanation: string) into classify_email(messages: array) and further, this specifies that the items in the array are of type messages defined at #/$defs/message. The $defs property then specifies the JSON schema for the message object with the same parameters as before.

The response now calls my classify_email function and passes an array of messages, where each message contains the arguments needed to categorize each one:

{
  "id": "chatcmpl-<redacted>",
  "object": "chat.completion",
  "created": 1687490210,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "function_call": {
          "name": "classify_email",
          "arguments": "{\\n  \\"messages\\": [\\n    {\\n      \\"threadId\\": \\"<redacted>\\",\\n      \\"category\\": \\"UNKNOWN\\",\\n      \\"explanation\\": \\"The email does not fit into any specific category.\\"\\n    }\\n  ]\\n}"
        }
      },
      "finish_reason": "stop"
    }
  ]
}

I’ve truncated the arguments value above, but the JSON for this contains an array of message objects with argument values as defined in the schema:

[
  {
    "threadId": "<redacted>",
    "category": "UNKNOWN",
    "explanation": "The email does not fit into any of the predefined categories."
  },
  {
    "threadId": "<redacted>",
    "category": "PROJECTS",
    "explanation": "The email is related to a project meeting invitation."
  },
  {
    "threadId": "<redacted>",
    "category": "PROJECTS",
    "explanation": "The email is related to a person going on call for a specific project."
  },
  {
    "threadId": "<redacted>",
    "category": "SUPPORT",
    "explanation": "The email is related to providing feedback and information about an Android app."
  },
  {
    "threadId": "<redacted>",
    "category": "UNKNOWN",
    "explanation": "The email does not fit into any of the predefined categories."
  },
  {
    "threadId": "<redacted>",
    "category": "UNKNOWN",
    "explanation": "The email does not fit into any of the predefined categories."
  },
  {
    "threadId": "<redacted>",
    "category": "UNKNOWN",
    "explanation": "The email does not fit into any of the predefined categories."
  }
]

And there it is! Beautifully structured JSON without having to burn through tokens coercing ChatGPT via few-shot prompting, and I can trust the the category property uses only the enum values I specified.

Further improvements

This is a good MVP for proving the concept. but there is still a ways to go to improve the categorization accuracy. The system can provide more of the email content rather than just the Gmail snippet, plus additional filtering up front can reduce the number of emails we need to classify by excluding things like automated emails.

Conclusion

ChatGPT still calls the function only once, but by using an array argument, ChatGPT transforms the result for each user message into an item in the array. This cuts down on the number of requests, avoiding getting rate limited with 429 responses, as well as optimizing the number of tokens used by sharing the system message and function definition across multiple items.

Engineering at Streak

We work on many interesting challenges affecting millions of users and many terabytes of data. For more information, visit https://www.streak.com/careers

Design goals

In building this, we had a number of goals for the link tracking functionality:

1. Must work all the time. We need to ensure that any system we implement for link tracking is highly reliable. It’s one thing for our code to have to retry a request to the API if there’s a transient issue, but our users will be sending emails to their contacts where we don’t have such control, and errors would reflect not only poorly on Streak but also on our users.

2. Must be fast. If you’ve ever received a marketing email that has links in it, you’ve likely clicked a link and it takes forever to go to the destination because some slow CRM system is processing the click. We don’t like building slow functionality.

3. Links must always work. Because our users are sending emails to their contacts, any tracked links in those emails should work. Forever. This means we shouldn’t tie the implementation to any one specific technology or require any database of valid links.

4. Revocable. When a Streak users sends out any email, the email goes out through their Gmail account, which has the benefit of Google’s extensive anti-spam measures, quota enforcement, best in class deliverability, and keeping all their email in one place. However, should we need to revoke a link (for example, due to malicious use), we want to be able to prevent those specific links from working. Note that this is at odds with links always working.

5. Spoof proof. Links must only work if they were generated by Streak. When the system generates a link, any modification of the link should render the link invalid. This prevents a malicious attacker from taking a valid link and modifying the destination to go somewhere else, or otherwise circumventing checks added during link generation.

Technology

Since Streak runs directly within Gmail and we are a Google Technology Partner, our infrastructure runs within GCP (Google Cloud Platform).

We considered Google Cloud Run as it seemed like a cool technology where you deploy a containerized microservice and leave it up to Google to manage. Also under consideration was Google Cloud Functions which is essentially a containerless version of Cloud Run and similar to AWS Lambda.

We eventually chose Cloudflare Workers to power our link tracking. Cloudflare boasts that their network reaches 95% of the world’s population within 50ms. Additionally, as they power some of the largest properties on the web we were comfortable that their infrastructure’s reliability meets or exceeds our own.

Code written for Workers runs directly in each of their edge locations, meaning we get fast distribution across more than 200 points of presence. Additionally, since the code is isolated from our existing infrastructure it’s not dependent on whichever technology stack we happen to use.

Constructing a link specification

When users send emails out through Streak, our system records which links are within which emails. Then, when a recipient clicks on a link, our system associates the click with the correct email. And because Streak has millions of users sending, cumulatively, millions of emails, maintaining a copy of all that data on Cloudflare or anywhere else was a non-starter. So we needed a much simpler system.

The format of the URL that we settled on looks like this:

https://somestreakdomain.ext/uniqueidentifier/destinationurl

This consists of the following parts:

1. The domain that we host the link on. For our links, we wanted to be completely transparent about the functionality and chose streaklinks.com for the domain.

2. The destination URL. Because we want the links to work regardless of platform or infrastructure, we chose a simple URL encoded version of the destination. So if you are linking to https://example.com then the encoded version is https%3A%2F%2fexample.com. While simple URL encoding does make a link easily decodable by a recipient should Streak later block the URL, the main goal is that Streak’s infrastructure isn’t directly used to link to an unwanted site.

3. This leaves the unique identifier. As mentioned, we use this to later associate which link in which email the recipient clicked on. At the same time, we don’t want to perform an expensive database lookup nor do we want a malicious attacker to be able to spoof arbitrary URLs in our system. The solution is that the identifier combines an identifier within Streak’s CRM with a cryptographic checksum of the data.

Because the complete URL contains all the data necessary to both validate the URL and redirect the user to the destination, we don’t need to maintain any storage.

Validating a link

When a recipient clicks on a link, we extract the unique identifier and destination URL from the request. Validation is then a simple matter of performing an HMAC SHA-256 calculation of the unique identifier and the destination URL, which we use to ensure the integrity and authenticity of the link.

Cloudflare has some docs on Web Crypto at https://developers.cloudflare.com/workers/runtime-apis/web-crypto/ which outlines how to calculate a message digest. In our case, this looks something like this:

const encoder = new TextEncoder();
const secretKeyData = encoder.encode(SECRET_KEY);
const key = await crypto.subtle.importKey(
  'raw',
  secretKeyData,
  {name: 'HMAC', hash: 'SHA-256'},
  false,
  ['sign']
);
const mac = await crypto.subtle.sign('HMAC', key, encoder.encode(message));

In the above code, SECRET_KEY is a secret that has been set via wrangler:

wrangler secret put SECRET_KEY

The value of this secret key is shared between Cloudflare and our API code which generated the link in the first place. The mac (Message Authentication Code) value contains an ArrayBuffer with the bytes, which we extract and use to verify that the link was generated by our API without having to contact the API or lookup any data.

Additionally, we store a blocklist of undesirable links in Cloudflare Workers KV (a key/value store) and we verify that the link has not been blocklisted. Workers makes this quite simple. In our case, we created a new KV namespace like blocklisted-links and then bound it to the worker using the name BLOCKLIST. To check if a destination URL is on the blocklist, it’s simply:

const MAX_KEY_LENGTH = 512;
const blocklistKey = ('url:' + destinationUrl).substring(0, MAX_KEY_LENGTH);
const blocklistEntry = await BLOCKLIST.get(blocklistKey);

If blocklistEntry is anything other than null, that means we’ve blocklisted the URL and we can reject the request. Note the use of the url: prefix here for checking the full URL. This allows us to use various prefixes to expand the blocklist to support any kind of blocking, whether via hostname, ip address, user-agent, and so on.

If everything checks out, the Cloudflare Worker code immediately redirects the user to the destination:

return new Response('', {
  status: 302,
  headers: {
    location: destinationUrl
  }
});

Overall request flow

Fast performance

The Cloudflare Worker code redirects requests to the destination URL in single digit milliseconds. Real world performance testing shows that link redirection can occur in approximately 20ms including making the request, processing the request inside the Cloudflare Worker code, and sending the redirection response. Most requests are so quick that the browser does not have time to display the streaklinks.com domain before the request to the destination occurs.

Behind the scenes, we take advantage of the fact that inside a Worker, fetch events (see https://developers.cloudflare.com/workers/runtime-apis/fetch-event/) run asynchronously and live beyond the lifecycle of the user request. While the email recipient’s click has already been redirected to their destination, the Worker code pushes the click data to our system where we update the Streak user’s CRM data:

event.waitUntil(postToStreak(data));

We run this code just prior to returning the redirection response. Here, data is an object that contains information about the request including unique ID, destination, timestamp, and so on. When processed, our API associates this data with the original sent email.

Launching link tracking

Soon after we launched, we saw some incredible success with the system. Cloudflare’s performance excelled at our goal of having very fast redirection and handled a massive number of requests:

Geeking out on logs

Cloudflare lets you easily live stream requests using wrangler. A simple wrangler tail -f pretty from the terminal and requests flood in, including any console logging performed within the worker.

console.error('Something went wrong!');
console.debug('Everything is good');

See https://developers.cloudflare.com/workers/learning/logging-workers/ for details. During development, this was invaluable to quickly verify details of the request and response. And once rolled out to production, streaming only errors caught a few issues where we weren’t gracefully handling invalid requests. And, when we want to see everything, we stream the debug logs as well.

In examining the logs, there were a few interesting things to note:

1. Browsers, of course, attempt to fetch a favicon.ico file. We didn’t plan to serve one initially, but it was an easy matter of serving up Streak’s favicon directly from the Worker. Similarly for people who are curious about what streaklinks.com is all about, when they visit https://streaklinks.com/ we redirect them to https://www.streak.com/

2. Anyone who has run any internet-facing system is familiar with automated exploit attempts. Soon after the domain went live, we saw all manner of malicious traffic. Fortunately, Cloudflare doesn’t break a sweat with all those requests and the Worker code trivially rejects invalid requests without effort.

As part of our commitment to overall security, we allow ethical hackers to try and discover vulnerabilities our systems via Hacker One and link tracking is fair game. More information at https://www.streak.com/security

Wrapping it up

Exploring the benefits that Cloudflare has to offer has been fun. With the new feature successfully launched to everyone, Streak users now get additional valuable insights into how recipients are interacting with their emails without having to use third party tools or leave Gmail.

Cloudflare Workers has proven to be extremely reliable and fast and our link tracking functionality has operated flawlessly. Having proven the benefits of Cloudflare Workers as part of our engineering toolkit, we now have a reference on how to build out fast, reliable functionality that integrates with our existing infrastructure.

Engineering at Streak

We work on many interesting challenges affecting millions of users and many terabytes of data. For more information, visit https://www.streak.com/careers