Build an AI Meeting Notes Tool (Your Own Otter)

Otter, Fireflies, and Granola are good products. They are also the wrong answer for any team that handles client confidentiality, runs in a regulated industry, or has a workflow more specific than "dump a summary into Slack." The good news is that in 2026, building your own AI meeting notes tool is a 2 to 3 week project for one developer, and the cost-per-hour ends up an order of magnitude below the SaaS per-seat pricing. This post walks the full architecture, the code, and the numbers - using the same stack I have shipped for a legal client who could not send recordings to a third party.

Why build versus use Otter or Fireflies

Three reasons consistently push teams to build their own meeting notes stack instead of paying for Otter or Fireflies.

Privacy. Otter, Fireflies, and Granola all process and store your recordings on their infrastructure. For law firms, M&A teams, healthcare providers, regulated finance, and most enterprise procurement reviews, that is a hard no. The compliance answer is usually "we cannot send meeting audio to a third-party SaaS" and the conversation ends. Building your own stack - on your cloud, with your encryption keys, with a retention policy you control - turns that no into a yes.

Cost. Otter Business is $20 per seat per month. Fireflies Pro is $18. For a 50-person team that is $12K per year for what is, at the infrastructure level, $0.30 per hour of meeting in underlying provider costs. The crossover where building beats buying lands at roughly 100 hours of recorded meetings per month - about 5 active users at typical usage.

Custom workflow. Off-the-shelf tools push notes into a fixed list of destinations with a fixed summary format. Building your own lets you extract custom structured fields - deal stage, MEDDPICC scorecard, clinical SOAP notes, sprint blockers, hiring scorecard signals - and route them into your CRM, EHR, ATS, or internal tooling exactly the way your team already works. That is the part nobody talks about and the part that actually drives adoption.

The stack

Every meeting notes tool decomposes into the same six layers. The provider you pick at each layer changes cost, accuracy, and which corner cases you will hit - but the shape of the system is fixed.

The default stack I reach for in 2026: Recall.ai for capture (the meeting bot is a 3-week build I would rather buy), Deepgram Nova-3 for transcription with diarization, Claude Sonnet 4.6 for structured summary, Postgres on Supabase for storage, and direct API integrations for destinations. Total monthly cost at 200 hours of meetings: about $130, versus $1,000+ for Otter at the same volume.

Architecture - how a meeting flows through the system

The pipeline is a sequence of stages with a queue between each. Decoupling lets you retry failed stages, swap providers, and scale bottlenecks independently. The shape:

• Capture. A meeting bot joins the call (or a user uploads a recording). Raw audio lands in object storage (S3, R2) with a meeting ID and metadata (participants, calendar event, workspace ID).
• Job enqueue. A new audio file fires a webhook that enqueues a transcription job. Use a real queue (Inngest, Trigger.dev, BullMQ on Redis) - meetings can be 2 hours long and you do not want a serverless function timing out at 10 minutes.
• Diarize. Run pyannote 3.1 (or rely on Deepgram's built-in diarization) to segment the audio by speaker.
• Transcribe. Run Whisper or Deepgram with word-level timestamps. Align with the diarization segments. Output: a list of turns, each with speaker label, start/end timestamps, and text.
• Summarize. Single LLM call with a structured-output schema. Returns title, TL;DR, decisions, action items, follow-ups, topics, sentiment.
• Extract action items. Second LLM pass dedicated to action items - separate prompt because action item extraction has its own failure modes (owner inference, dedup, due date parsing).
• Push to destinations. Per-workspace routing config decides which action items go to Linear, which decisions go to Notion, and which summary lines go to a Slack channel.
• Notify. Email or Slack ping to the meeting owner with the summary, with an "edit before sending" button if you want human-in-the-loop review before destinations fire.

Two things are non-negotiable. First - every stage is async and retryable. A failed Deepgram call should not lose the audio. Second - you store the raw transcript and the structured summary separately, so users can re-run the summary with a new prompt or schema without re-transcribing (which is the expensive step).

Capture options - bot vs Chrome extension vs upload

Capture is the layer most teams get wrong. The right choice depends on which platforms your team uses and how much you want to build.

My recommendation for an MVP: start with manual upload + Recall.ai bot. Upload covers in-person meetings and historical recordings. Recall covers Zoom/Meet/Teams without you writing a single bot integration. You can replace Recall with native SDKs later if cost demands it - but only after you have 50+ active users and the per-hour math actually justifies the engineering.

Whisper deep dive - local vs hosted vs Deepgram

Transcription is the single biggest accuracy lever. Get this wrong and every downstream stage degrades. The 2026 landscape:

The honest defaults: Deepgram Nova-3 if you need real-time partials or speaker labels out of the box. OpenAI hosted Whisper if you want zero ops and only need batch transcription post-meeting. Groq if cost is the dominant constraint and batch latency is fine - $0.04 per hour is an order of magnitude below everyone else. Local whisper.cpp on a Mac Studio or a single A10G GPU if privacy is non-negotiable and you have someone who will own the ops.

For a multi-language workload (Spanish, Portuguese, German calls mixed with English), Deepgram's language detection plus per-language models hits 5 to 7% WER. Whisper large-v3 is the most robust to accented English of any model I have tested - drop the temperature to 0 and use condition_on_previous_text=false to avoid hallucination on silence.

Speaker diarization with pyannote

Diarization is the "who said what" step. Without it, your transcript is a wall of text and action item attribution is impossible. pyannote-audio 3.1 is the open-source state of the art. WhisperX wraps pyannote + Whisper + word-level alignment into one pipeline and is what I use when I'm self-hosting.

# diarize.py
from pyannote.audio import Pipeline
import torch
import whisperx

device = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Diarization
diarization = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token=HF_TOKEN,
).to(torch.device(device))

# 2. Transcription with WhisperX (Whisper + alignment)
model = whisperx.load_model("large-v3", device, compute_type="float16")

def process_meeting(audio_path: str):
    # Transcribe
    audio = whisperx.load_audio(audio_path)
    result = model.transcribe(audio, batch_size=16)

    # Align word timestamps
    align_model, metadata = whisperx.load_align_model(
        language_code=result["language"], device=device
    )
    aligned = whisperx.align(
        result["segments"], align_model, metadata, audio, device
    )

    # Diarize
    diarize_result = diarization(audio_path, num_speakers=None)

    # Assign speakers to words
    final = whisperx.assign_word_speakers(diarize_result, aligned)
    return final  # list of {speaker, start, end, text}

Accuracy numbers I have measured on real client audio: 6 to 9% DER on clean two-person Zoom calls, 10 to 14% DER on 4 to 6 person meetings with overlapping speech, 18 to 25% DER on noisy in-person rooms with a single mic. Two-mic recordings (a per-participant track) drop DER to near zero - if you can get Zoom or Meet to record per-participant audio, take it.

For naming speakers (not just "Speaker 1, Speaker 2"), the practical trick is to pull the participant list from the calendar event and match by speaking time - the host usually speaks the most, the meeting owner second-most, and so on. For higher accuracy, enroll voiceprints (pyannote supports this) once per user and match on every future meeting.

Structured summary with Claude

This is where the build pays back. Otter gives you a free-form paragraph and a bullet list. Your build can return a Zod-validated object with exactly the fields your downstream destinations need. The schema I use for a general-purpose meeting notes tool:

// schema.ts
import { z } from "zod";

export const MeetingSummarySchema = z.object({
  title: z.string().describe("Short descriptive title, 6-10 words"),
  tldr: z.string().describe("2-3 sentence executive summary"),
  topics: z.array(
    z.object({
      topic: z.string(),
      summary: z.string(),
      speakers: z.array(z.string()),
    })
  ),
  decisions: z.array(
    z.object({
      decision: z.string(),
      context: z.string(),
      decidedBy: z.string().optional(),
    })
  ),
  actionItems: z.array(
    z.object({
      action: z.string(),
      owner: z.string().describe("Participant name or 'unassigned'"),
      dueDate: z.string().nullable().describe("ISO date or null"),
      priority: z.enum(["high", "medium", "low"]),
      sourceQuote: z.string().describe("Verbatim quote that implied this"),
    })
  ),
  followUps: z.array(
    z.object({
      question: z.string(),
      raisedBy: z.string(),
    })
  ),
  sentiment: z.enum(["positive", "neutral", "tense", "blocked"]),
  redFlags: z.array(z.string()),
});

export type MeetingSummary = z.infer<typeof MeetingSummarySchema>;

The Claude call against this schema with structured outputs (the pattern is identical across Anthropic and OpenAI in 2026):

// summarize.ts
import { generateObject } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
import { MeetingSummarySchema } from "./schema";

const SYSTEM = `You are a meeting analyst. Given a transcript with speaker
labels, return a structured summary. Be precise. Use verbatim quotes for
action items. If an owner is unclear, mark "unassigned". Never invent
decisions, dates, or commitments that are not explicit in the transcript.`;

export async function summarize(
  transcript: { speaker: string; text: string; start: number }[],
  participants: string[]
) {
  const formatted = transcript
    .map((t) => `[${t.speaker}] ${t.text}`)
    .join("\n");

  const { object } = await generateObject({
    model: anthropic("claude-sonnet-4-6"),
    schema: MeetingSummarySchema,
    system: SYSTEM,
    prompt: `Participants: ${participants.join(", ")}

Transcript:
${formatted}`,
    maxTokens: 4000,
  });

  return object;
}

Two prompt details that matter. The system prompt explicitly says "never invent" - this single line cuts hallucinated action items by ~70% in my evals. And requiring a sourceQuote field for every action item forces the model to ground its output - you can then surface that quote in the UI for the user to verify.

Action item extraction - a separate pass

Action items are the highest-value output of a meeting notes tool and the easiest to get wrong. Three problems show up consistently: ambiguous owners ("someone should look into that"), duplicate items across consecutive meetings, and missed items the summary glosses over. A dedicated second pass solves all three:

// extract-actions.ts
import { generateObject } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
import { z } from "zod";

const ActionItemsSchema = z.object({
  items: z.array(
    z.object({
      action: z.string(),
      ownerCandidate: z.string(),
      ownerConfidence: z.enum(["high", "medium", "low"]),
      dueDate: z.string().nullable(),
      sourceQuote: z.string(),
      sourceTimestamp: z.number(),
    })
  ),
});

const EXTRACT_PROMPT = `Extract EVERY action item from the transcript.
Rules:
- An action item is any explicit or implicit commitment to do something.
- For ownerCandidate, prefer the speaker who took the commitment. If unclear,
  use the meeting owner. Mark ownerConfidence low if ambiguous.
- For dueDate, parse "next Friday", "EOD", "by Q3" into ISO dates relative to
  the meeting date. Return null if no date is implied.
- Always include the verbatim sourceQuote that implies the action.
- Do not invent action items. If the meeting had none, return an empty array.`;

export async function extractActions(
  transcript: { speaker: string; text: string; start: number }[],
  meetingDate: string,
  meetingOwner: string,
  openActionItems: { action: string; owner: string }[] // for dedup
) {
  const formatted = transcript
    .map((t) => `[${Math.round(t.start)}s][${t.speaker}] ${t.text}`)
    .join("\n");

  const { object } = await generateObject({
    model: anthropic("claude-sonnet-4-6"),
    schema: ActionItemsSchema,
    system: EXTRACT_PROMPT,
    prompt: `Meeting date: ${meetingDate}
Meeting owner: ${meetingOwner}

Already-open action items (do NOT re-extract these):
${openActionItems.map((a) => `- ${a.action} (${a.owner})`).join("\n")}

Transcript:
${formatted}`,
  });

  return object.items;
}

Pass the workspace's currently-open action items into the prompt as a deduplication hint. This is the cheapest dedup that works - the LLM sees what already exists and avoids creating "ship the landing page" for the fourth week in a row. For higher precision, run a semantic similarity check against an embedding of each new candidate vs all open items, and drop items with cosine similarity above 0.85 to an existing one.

Destinations - push to where work happens

A summary sitting in your meeting notes app is a summary nobody reads. The value lands when action items show up in Linear, decisions log themselves to Notion, and the TL;DR pings the channel. Each destination has a clean API; the pattern is a per-workspace routing config that maps schema fields to destination payloads.

// destinations/linear.ts
import { LinearClient } from "@linear/sdk";

const linear = new LinearClient({ apiKey: process.env.LINEAR_API_KEY! });

export async function pushActionItemsToLinear(
  items: ActionItem[],
  config: { teamId: string; defaultAssigneeId: string }
) {
  for (const item of items) {
    const assignee = await resolveAssignee(item.ownerCandidate)
      ?? config.defaultAssigneeId;

    await linear.createIssue({
      teamId: config.teamId,
      title: item.action,
      description: `From meeting on ${item.meetingDate}\n\n> ${item.sourceQuote}`,
      assigneeId: assignee,
      dueDate: item.dueDate ?? undefined,
      priority: item.priority === "high" ? 1 : item.priority === "medium" ? 2 : 3,
      labelIds: [MEETING_NOTES_LABEL_ID],
    });
  }
}

Mirror the same pattern for Notion (REST API, append rows to a database), Slack (Block Kit message in a channel), Salesforce/HubSpot (Note object via webhook), Jira (REST API), and Asana. The per-workspace config decides which destinations fire - most teams want action items in Linear or Jira, decisions in Notion or Confluence, and the TL;DR + link in a #meetings Slack channel.

The human-in-the-loop variant - and the one I recommend for any team that has been burned by AI agents firing prematurely - is a Slack notification with the structured summary and an "Approve and push" button. The destinations only fire when the meeting owner clicks approve. See my AI email automation post for the same pattern applied to inbox triage.

Privacy by default

This is the differentiator. If your build leaks PII, ships audio to third parties without consent, or has no retention policy, you have rebuilt Otter with worse UX. The non-negotiable list:

• Encryption at rest with per-workspace keys. Use envelope encryption - a workspace-scoped data encryption key (DEK) encrypts the transcript and summary; the DEK is itself encrypted by a key encryption key (KEK) held in AWS KMS or GCP KMS. When a workspace is deleted, you delete the KEK and the data becomes unrecoverable.
• Retention policy users can configure. Default to 90 days for raw audio, 30 days for transcripts, indefinite for the structured summary (which is what users actually need). Let workspace admins shorten any of these. GDPR erasure requests get honored inside 30 days, end-to-end.
• No training on customer data. Use Anthropic and OpenAI's zero-retention enterprise endpoints. Document it in your privacy policy. For higher-stakes deployments, route to an on-prem Whisper plus a local Llama 3.3 70B for summarization - yes, you give up some quality, and yes, certain clients will only sign if you do.
• PII redaction in long-term transcripts. Strip credit card numbers, SSNs, account numbers, and dates of birth at ingest using a regex pass plus an LLM redaction pass for context-dependent identifiers. Store the redacted version long-term; keep the un-redacted version only for the short audio retention window.
• Consent and disclosure. The meeting bot announces itself audibly when joining ("This meeting is being recorded and transcribed"). Two-party consent jurisdictions require it. Allow any participant to opt out, which suppresses their audio from the transcript.

Cost math at three scales

The unit economics are the build's strongest argument. Below is the fully-loaded per-hour-of-meeting cost using the default 2026 stack - Recall.ai capture + Deepgram Nova-3 + Claude Sonnet 4.6.

Now compare against the build-it-yourself-without-Recall path, which is the right move at scale: drop Recall ($0.40), add 30 minutes of engineering ops per meeting amortized ($0.05), and use Groq Whisper large-v3-turbo instead of Deepgram ($0.04). Total: $0.21 per hour of meeting. At 1,000 hours per month, that is $210 in infrastructure for what Otter would bill at roughly $830 (assuming a 50-seat team). Anthropic prompt caching on the summary system prompt cuts another 30 to 40% off the LLM line - see my OpenAI API cost breakdown for the caching tricks across providers.

Comparison: Otter vs Fireflies vs Granola vs your build

Realistic timeline to ship an MVP

Two to three weeks of focused work for one developer. This is the sequence I would run, based on having shipped this exact stack twice:

• Week 1 - capture and pipeline. Day 1-2: Postgres schema, S3 bucket, upload endpoint. Day 3-4: Recall.ai bot integration for Zoom/Meet/Teams. Day 5-7: Deepgram streaming + diarization, full transcript object in DB.
• Week 2 - intelligence and UI. Day 8-10: Zod schema, summary pipeline, action item extraction pipeline, eval set against 5-10 real meeting recordings you trust. Day 11-14: a basic Next.js UI for browsing meetings, viewing transcripts, editing extracted action items.
• Week 3 - destinations and polish. Day 15-17: Linear + Notion + Slack integrations with per-workspace routing config. Day 18-19: privacy controls (retention policy, encryption keys, PII redaction). Day 20-21: human-in-the-loop approval flow, email notifications, internal launch.

Cut anything you don't need on day one. A solo founder using it for their own meetings can ship in 4 to 5 days by skipping the bot (manual upload only), skipping diarization (single-speaker okay), and skipping destinations (read the summary in a basic Next.js page). That gets you to value fast, and you add capabilities as the workflow proves out.

What to do next

If you are scoping this build for your team or a client and want a senior engineer who has actually shipped it, my AI integration and MVP development practices cover this scope exactly. I work with teams worldwide and you can also hire an AI developer in Kosovo directly. Same person who built Caldra AI and the meeting intelligence stack behind two private client deployments.

For neighboring builds in this series - many of these reuse the same Whisper + Claude + structured-output pattern with a different output schema - see:

• AI email automation: build your own triage agent - the same approve-before-push pattern applied to inbox.
• AI scheduling assistant: 9 tools tested in 2026 - the build-vs-buy framework applied to calendar AI.
• AI document extraction: from PDF chaos to clean JSON - the structured-output pattern applied to documents.

Frequently asked questions