Quality work you don’t see

Dovetail’s promise is to “build with facts, not vibes.” Every Chat and Agent response comes with citations to back that up—every answer traces back to something a customer actually said.

The moment that breaks, we fail to deliver on that promise, so we built the habit of continually monitoring it ourselves, early, before it’s anyone else’s problem.

How we catch it before you do

We dogfood hard. Every piece of feedback goes straight into a Channel in our own workspace that the team opens every morning—we use the product to build the product. A Dovetail Agent also runs weekly, synthesizing the whole Channel into a summary posted to our team Slack, so nothing sits unread for long. Finally, every piece of feedback is reviewed by our AI Center of Excellence on a weekly basis to keep an eye on quality and make sure we’re reacting to any new issues as they pop up.

We already hold every AI feature against our four-pillar bar: groundedness, task success, correct attribution, and reliability. In Chat and Agents, “groundedness” is the metric we use to measure citation quality. To help improve citation quality, we created a new metric to measure how often we drop citations that don’t make sense. When this metric was reviewed, we realized something was wrong.

What we found was that while Chat and Agents were reliably returning citations after the first question, follow-up questions would frequently drop them.

Multi-turn grounding

In July, we received a large amount of feedback that Chat was often “hallucinating.” When we looked into these reports, the majority of the cases weren’t real hallucination, but Chat would still tell the user that it was when they asked. This happened because between the first message and the second, we cleaned up the message history so that Chat didn’t see the search results it had used to derive an answer—so when the user asked for sources or evidence, Chat assumed it had made it all up.

We fixed this, but that introduced a different issue—laziness. Now that Chat had full access to its own history, it preferred to cite results from its original response rather than running fresh searches. This confused our citations system, which couldn’t find the search results Chat was talking about.

This was causing the majority of citation failures, and we quickly solved this issue by allowing Chat to cite historical search results.

Structured citations

Under the hood, we use AWS to serve our models. We aim to keep as close to the frontier as possible so that you can take advantage of the best available technology to query your data. Unfortunately, the release of Claude Sonnet 5 on AWS dropped support for enforcing formatting.

This was an issue, as frontier LLM models often have trouble following instructions the more information you give them, especially when it comes to specific formatting. We were seeing this happen regularly in longer conversations, where Claude would improperly format citations, causing the system to break.

We’ve temporarily paused our testing of Sonnet 5 while we ensure citations can properly be resolved in all cases on the model.

AI Agents cite outbound

AI Agents write to Slack and email, where citations weren’t rendering. We wired structured citations into send_email and send_slack, resolved to deep links before delivery. Slack renders them as clickable links. Email uses the full markdown-it rule set, so links and citations survive.

Triggered and scheduled runs now get an explicit “include citations” reminder in prompt guidance. Without a human in the loop asking for sources, the model relaxes.

We also unified AI Agents onto the same execution path as Chat, as a modifier rather than a separate strategy. That means every fix above applies uniformly across product Chat and AI Agents.

What you get: AI Agent messages in Slack and email cite the same way product Chat does, with links that go somewhere.

Citation resolution and UX

• Follow-up citations that reference an earlier turn now show a fallback badge with the tooltip “Source provided earlier in the conversation,” instead of a silent drop
• DatumSummary citations resolve like Datum. They were previously returning empty source_items and rendering as dead markers
• Doc citations resolve to canonical /docs/ paths and no longer show “Untitled” as the source label
• Video citations have thumbnail and clip fallbacks
• create_doc returns a deeplink; comment tools deep-link to the specific comment

Finishing the turn

The tool loop has a step budget. Citations consume a step, so complex multi-search turns were hitting the ceiling and returning empty.

• Raised the ceiling from 10 to 20
• When the budget is exhausted, we force a final written answer instead of returning nothing (with a follow-up fix so that path doesn’t hard-error under toolChoice: none)
• Fixed replay bugs on thinking blocks and errored tool calls that were causing Bedrock 400s
• Better diagnostics when prompt schema validation fails. This was the top undiagnosable failure class before

What you get: fewer “Something went wrong” turns, especially on hard multi-search questions.

Metrics

We already watched groundedness broadly across every AI feature. Citations now get their own dedicated signal on top of that: resolve rate tracked across Chat, Slack, and AI Agent email, sliced by conversation depth, failure reason, and object type. A contextual_chat_multi_turn eval runs against the same CitationResolver the product uses, and the results feed into the same weekly summary that caught the original regression.

What you get: when quality regresses, we see it before you do.

Going forward

The Channel’s still there. Standup still opens with it. The Agent still posts its summary every week. Most weeks there’s nothing dramatic in it, and that’s exactly the point—we’d rather catch the bugs ourselves at standup than hear about it from you.