Gemini Audio Transcription for Long, Multilingual Recordings: A Production Pipeline on Vertex AI

Gemini audio transcription works well for long, mixed-language recordings if you treat it as a pipeline rather than a single API call. Probe the file, extract audio, split recordings longer than about 20 minutes at natural silences, send each chunk to Gemini on Vertex AI with a JSON schema for timestamped, speaker-labeled segments, then merge the chunks into one timeline by adding each chunk’s start offset. Route chunks to the cheapest model tier that meets your quality bar, retry with backoff when you hit rate limits, and generate SRT or WebVTT subtitles in code from the merged JSON.

That is the design OSKI used for an archive of educational lectures in which Hebrew, Aramaic and English are often mixed within a single recording. This article walks through each stage, the current Vertex AI limits that shape it, and the trade-offs we would point out to any team about to build something similar.

What Gemini on Vertex AI supports for audio in 2026

Before designing the pipeline, check the hard limits. As of October 2026, the Vertex AI audio understanding docs list:

The Gemini API docs describe audio as 32 tokens per second, so one minute is about 1,920 input tokens, and note that audio is downsampled and multi-channel audio is combined into one channel. If speakers are on separate channels in your source, that separation is lost unless you transcribe channels separately.

Which models to consider

Model names change quickly, so pin exact IDs and watch the lifecycle page. As of October 2026 on Vertex AI:

• Gemini 3.8 Flash (gemini-3.8-flash): generally available since September 2, 2026, 1,048,576-token context window, 65,536 maximum output tokens, structured output and batch inference supported.
• Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite): generally available since July 21, 2026, same context and output limits, structured output and batch supported.
• Gemini 3.1 Pro: listed for audio understanding as a preview model.
• Gemini 4 Argon (gemini-4-argon): released October 13, 2026 and listed for speech understanding. Check its model page for limits and pricing before routing audio to it.
• Gemini 2.5 Pro, Flash and Flash-Lite: reached their retirement date on October 20, 2026. Google lists Gemini 3.8 Flash, 3.5 Flash or 3.5 Flash-Lite as replacements, so any pipeline still calling 2.5 models needs to move now.
• Gemini 3.5 Transcribe (gemini-3.5-transcribe): a dedicated speech-to-text model, GA in October 2026. It supports diarization of up to 8 speakers and language auto-detection with code-mixing, but file processing is limited to 15 minutes per file, and the same limit applies with diarization or timestamps enabled, and it does not support system instructions, structured output or batch inference.

The lecture pipeline described below was built in 2025 on the then-current Gemini Pro, Flash and Flash-Lite models. The architecture does not depend on a specific version, which is the point: model routing should be configuration, not code.

Why split long recordings when Gemini accepts hours of audio?

If one request can take 8 hours, why split at all? Four practical reasons:

1. Output limits. A verbatim transcript of a dense lecture, wrapped in JSON with timestamps and speaker labels, is much larger than plain text. A single response is capped at 65,536 output tokens on current Flash models, and a truncated JSON response is worse than useless.
2. Parallelism. Ten 6-minute chunks finish far sooner than one hour-long request.
3. Cheaper failures. If one chunk fails or comes back malformed, you retry 6 minutes of audio, not 60.
4. Model routing. Different chunks can go to different tiers.

The trade-off is the merge step. Splitting in the wrong place cuts words in half, and getting offsets wrong breaks every subtitle after the first chunk.

Step 1: Probe and normalize every input

Uploads arrive as audio files and video containers with different codecs, sample rates and stream layouts. On the lecture transcription pipeline, FFprobe checks each upload first, and video files have their audio extracted automatically, so one path serves every format.

A normalization step usually looks like this (illustrative command):

# Inspect streams and duration
ffprobe -v error -show_entries format=duration:stream=index,codec_type,codec_name,channels \
  -of json input.mp4

# Extract the audio track as mono 16 kHz FLAC for transcription
ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -c:a flac audio.flac

Record the original duration from FFprobe. You will use it later to check that the merged transcript covers the whole recording.

Step 2: Split at silences, not at fixed intervals

Fixed 10-minute cuts land mid-word. FFmpeg’s silencedetect filter logs where silence starts and ends, which gives you safe cut points:

# Illustrative: log silences quieter than -35 dB lasting at least 0.6 s
ffmpeg -i audio.flac -af silencedetect=noise=-35dB:d=0.6 -f null - 2> silences.log

Then choose cut points from the log: for each target boundary (for example every 8 to 10 minutes), pick the silence closest to it and cut in the middle of that silence. Tune the noise threshold per source; lecture halls with background hum need a higher threshold than studio recordings.

In the lecture pipeline, recordings over 20 minutes are split at natural silences, never mid-sentence. Shorter files go through whole.

Keep a manifest per recording:

{
  "recordingId": "lec-0142",
  "durationSec": 3912.48,
  "chunks": [
    { "index": 0, "startSec": 0.0,    "endSec": 547.31,  "uri": "gs://bucket/lec-0142/000.flac" },
    { "index": 1, "startSec": 547.31, "endSec": 1103.92, "uri": "gs://bucket/lec-0142/001.flac" }
  ]
}

The manifest is the source of truth for offsets. Never infer offsets from file names or chunk lengths after the fact.

Step 3: Ask for structured JSON with timestamps and speakers

Do not ask the model for an SRT file. Ask for data. Vertex AI structured output lets you set responseMimeType to application/json and pass a responseSchema; Google’s docs note that setting only the MIME type is a strong hint, not a guarantee, so use both.

An illustrative request with the Google Gen AI Python SDK:

# Illustrative: one chunk, structured output, audio timestamps on
from google import genai
from google.genai import types

client = genai.Client(vertexai=True, project="my-project", location="global")

segment = types.Schema(
    type="OBJECT",
    properties={
        "start": types.Schema(type="STRING", description="MM:SS.mmm from chunk start"),
        "end": types.Schema(type="STRING", description="MM:SS.mmm from chunk start"),
        "speaker": types.Schema(type="STRING"),
        "language": types.Schema(type="STRING", enum=["he", "arc", "en"]),
        "text": types.Schema(type="STRING"),
    },
    required=["start", "end", "speaker", "language", "text"],
)

response = client.models.generate_content(
    model="gemini-3.8-flash",
    contents=[
        types.Part.from_uri(file_uri=chunk_uri, mime_type="audio/flac"),
        TRANSCRIPTION_PROMPT,
    ],
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=types.Schema(type="ARRAY", items=segment),
        audio_timestamp=True,
        temperature=0.0,
    ),
)

Prompting for mixed-language speech

Mixed-language speech is where generic prompts fail. In the lecture archive, speakers move between Hebrew, Aramaic and English in the same sentence, and the text needs diacritics and correct references to religious sources. That required special prompting and several formatting passes. Rules that help:

• Name the languages to expect and say explicitly: transcribe, do not translate.
• Say which script to use for each language and how to handle quotations from source texts.
• Give a short glossary of names and terms that recur in the archive.
• Ask for a language tag per segment, so cleanup can be language-aware.

Post-process by language

Some fixes are better done after transcription. In the lecture pipeline, post-processing converts Hebrew script, adds diacritics, tidies formatting and verifies citations of religious sources. Keep each pass small and testable, and store the raw model output so you can rerun cleanup without paying for transcription again.

Step 4: Merge chunks with correct offsets

Each chunk’s timestamps start at zero. Merging means adding the chunk’s startSec from the manifest to every segment, then validating the result.

# Illustrative merge with validation
def merge(manifest, chunk_results):
    merged = []
    for chunk in sorted(manifest["chunks"], key=lambda c: c["index"]):
        for seg in chunk_results[chunk["index"]]:
            start = chunk["startSec"] + parse_ts(seg["start"])
            end = chunk["startSec"] + parse_ts(seg["end"])
            end = min(end, chunk["endSec"])          # clamp model drift
            if merged and start < merged[-1]["end"]:
                start = merged[-1]["end"]            # remove small overlaps
            if end > start:
                merged.append({**seg, "start": start, "end": end})
    assert merged[-1]["end"] <= manifest["durationSec"] + 0.5
    return merged

Checks worth automating on every recording:

• Timestamps never go backwards.
• No segment crosses its chunk boundary by more than a small tolerance.
• No silent gap longer than you expect for lecture speech.
• The last segment ends near the recording’s real duration.
• Speaker labels are consistent across chunks. Models label speakers per request, so “Speaker 1” in chunk 3 may not be “Speaker 1” in chunk 4. Reconcile with a short pass, or ask for role-based labels such as “Lecturer” and “Audience”.

On the lecture pipeline, long recordings are split and joined with no gaps or overlaps in the timeline, and subtitle timestamp accuracy is 99% even across split recordings.

Step 5: Route models across tiers to cut cost

Not every minute of audio needs the most capable model. Routing rules can use chunk length, audio quality, language mix or a first-pass confidence check. In the lecture pipeline, Gemini Pro or Flash is chosen by length, with Flash-Lite also available. That dynamic model selection cut AI and API costs by up to 60%, while transcription precision on Hebrew and Aramaic content reached 99%.

A practical way to set up routing:

1. Build a test set of real chunks with human-checked transcripts, including the hardest audio you have.
2. Run every tier on the full set and score each one.
3. Set routing rules so each class of chunk goes to the cheapest tier that meets your threshold.
4. Escalate automatically: if a chunk fails schema validation or a quality check, retry it on the next tier up.
5. Re-run the test set whenever a model version changes.

Google’s current prices differ by model, region and consumption option, so check the Vertex AI pricing page for your setup. Current Flash and Flash-Lite models also support batch inference and Flex pay-as-you-go, which are worth considering for archive backlogs that are not time-sensitive.

Step 6: Handle rate limits and failures at scale

At hundreds of parallel jobs, you will see 429 RESOURCE_EXHAUSTED errors. In a March 2026 post, Google Cloud recommends not retrying 429 or 503 errors immediately, using exponential backoff with jitter, using the global endpoint where possible, and smoothing traffic, since sudden bursts are a primary cause of 429s. The Gen AI SDK has configurable retries through HttpRetryOptions.

What that means for a transcription pipeline:

• Queue chunks, not files. A worker pool with a global concurrency limit smooths bursts better than launching every chunk of every file at once.
• Back off adaptively. When the 429 rate rises, lower concurrency for a while, then raise it slowly.
• Make every step idempotent. Store each chunk result as it arrives. A restarted job should skip chunks that already succeeded.
• Validate before you store. A response that does not parse or fails timestamp checks counts as a failure and goes back to the queue.

The lecture pipeline runs more than 300 transcription jobs concurrently under queue management with retries and adaptive backoff, and no data is lost when connections drop: work re-syncs automatically.

Step 7: Generate subtitles and HLS video

Subtitles from the merged JSON

Once you have a validated timeline, subtitle files are a formatting job. Split long segments into cues of readable length, then write:

• SRT: numbered cues, timestamps as 00:12:05,340 --> 00:12:08,900.
• WebVTT: a WEBVTT header, timestamps with a period (00:12:05.340), and optional cue settings. WebVTT is the format browsers and HLS players expect.

Right-to-left languages such as Hebrew need testing in the actual players you support, especially lines that mix Hebrew with English words or numbers.

HLS video

For video recordings, the lecture pipeline has a parallel module that builds multi-bitrate HLS streams, thumbnails and previews, and stores results in cloud storage behind secure links. HLS (RFC 8216) splits media into segments listed in playlists, with a master playlist pointing to each quality variant, and subtitle tracks can be added as WebVTT renditions.

OSKI built a similar conversion service for a media business on AWS: the cloud media pipeline produces five video tiers from 240p at 300 kbps to 1080p at 5000 kbps in 4-second segments, encodes each audio track into five bitrate tiers between 32 and 192 kbps (two for commentary tracks), and links all variants in a master m3u8 so players can switch quality and language.

Step 8: Orchestrate on Kubernetes

Transcription and transcoding are bursty, heavy jobs, which suits a queue plus on-demand workers:

• One job per unit of work. A Kubernetes Job (or a pod created from a template) per recording or per chunk, with resource limits, ephemeral storage and a deadline (activeDeadlineSeconds).
• Retries with limits. Use backoffLimit for infrastructure failures, but keep API-level retries inside the worker where you control backoff.
• Cleanup. ttlSecondsAfterFinished removes finished Jobs so the cluster does not fill with completed pods.
• Scale to zero. Let the cluster autoscaler add nodes as the queue grows and remove them when it empties.

In the cloud media project, a job manager long-polls SQS, gives each job a dedicated pod with resource limits and node affinity, and the autoscaler removes all worker nodes when the queue is empty, so compute cost is $0 while idle. The lecture pipeline also runs on Kubernetes with Helm, scaling from one file to hundreds.

Production checklist

• Every input probed; audio extracted from video; original duration recorded.
• Recordings above your threshold split at silences, with a manifest of exact offsets.
• Structured output with a schema; raw responses stored.
• Merge validation: monotonic timestamps, no overlaps, coverage of full duration, consistent speakers.
• Model routing configured, with automatic escalation on failure.
• Concurrency limits, backoff with jitter and idempotent chunk storage.
• Subtitles generated in code and tested in real players, including right-to-left text.
• Model IDs pinned and retirement dates on the team calendar.

If you are planning a transcription or media pipeline and want a second opinion on chunking, model routing or cluster design, our AI development and cloud and DevOps teams have built both halves of this system.