Top Builders

Explore the top contributors showcasing the highest number of app submissions within our community.

OpenAI Whisper

The Whisper models are trained for speech recognition and translation tasks, capable of transcribing speech audio into the text in the language it is spoken (ASR) as well as translated into English (speech translation). Whisper has been trained on 680,000 hours of multilingual and multitask supervised data collected from the web. Whisper is Encoder-Decoder model. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and to-English speech translation.

General
Relese dateSeptember, 2020
AuthorOpenAI
Repositoryhttps://github.com/openai/whisper
Typegeneral-purpose speech recognition model

Start building with Whisper

We have collected the best Whisper libraries and resources to help you get started to build with Whisper today. To see what others are building with Whisper, check out the community built Whisper Use Cases and Applications.

Tutorials

Boilerplates

Kickstart your development with a GPT-3 based boilerplate. Boilerplates is a great way to headstart when building your next project with GPT-3.

Libraries

Whisper API libraries and connectors.

Edit on GitHub

OpenAI Whisper AI technology Hackathon projects

Discover innovative solutions crafted with OpenAI Whisper AI technology, developed by our community members during our engaging hackathons.

FourTakes

FourTakes

FourTakes is a video captioning agent for the AMD Developer Hackathon's Video Captioning track: given a short clip, it produces captions in four distinct voices — formal, sarcastic, humorous-tech, and humorous-non-tech — scored by an LLM judge on exact style match. Models used (both via Fireworks AI, one API key): Qwen 3.7 Plus (accounts/fireworks/models/qwen3p7-plus) for vision-language captioning, and Whisper v3 for optional audio transcription. Architecture: one Qwen 3.7 Plus vision call produces a neutral factual base caption per clip, reused across four parallel text-transform calls, one per style — keeping all four outputs factually consistent and cutting vision calls from four to one per video. What makes this different from a straightforward implementation: Qwen 3.7 Plus is a reasoning model; without disabling that, it burns its token budget on visible chain-of-thought instead of returning a caption — found and fixed against the live API, not a theoretical risk. Most Fireworks vision-language models, including several other Qwen3 variants, are on-demand-only and would 404 at judging time — the model choice was verified against the live serverless catalog, not assumed from documentation. Credentials are injected as runtime environment variables, never baked into the image, matching the judging harness's contract — verified by running the published image with credentials injected via -e, exactly as judging will. A fallback ladder guarantees every requested style always gets a caption, since an empty or missing style scores zero with the judge. Frames are downscaled for the 4K source clips to control payload size and cost. Validation: 42 offline unit tests (mocked API/ffmpeg/network) plus real validation — live Fireworks API calls succeeded across all three official example clips in every style with zero errors, reproduced through the published Docker image with runtime-injected credentials.

Raccoon Vision Translator

Raccoon Vision Translator

This is a video captioning agent for the Video Captioning track. Give it a clip and it writes four captions in four styles: formal, sarcastic, humorous_tech, and humorous_non_tech. All four come from the same source description, so the same real thing that happened in the clip gets described four different ways, not four different guesses. Before any of that, there's a small local step. ffmpeg pulls a handful of downscaled frames and the audio track, and a voice activity check decides if it's even worth running Whisper on that clip. A lot of video content is silent, so there's no reason to transcribe it every time. The real work happens through one model: Gemma 4 (31B), called via Hugging Face's Inference Providers. It looks at the sampled frames and writes a description of what's happening. Then it looks again, checks its own draft against the frames a second time, and fixes anything that was wrong or too vague. That gets cached once per clip, and all four styled captions are written from it. Each style also gets to see what the earlier styles already wrote, so they don't all collapse into the same sentence with different adjectives swapped in. Before anything gets written to results.json, each caption is checked against the word count rules the task requires, and it gets one retry if it fails that check. A bad caption shouldn't be able to take down the whole submission. One thing worth mentioning. An earlier version of this tried splitting the work across several different vision models feeding into each other. It didn't go well. The models disagreed with each other and the captions got less accurate, not more. Going back to one model doing both the looking and the writing turned out faster and a lot more reliable.

Video Captioning

Video Captioning

This pipeline accepts short video clips and generates four stylistically distinct captions per clip: formal, sarcastic, humorous-tech, and humorous-non-tech. The architecture is deliberately decoupled into four stages. First, OpenCV extracts keyframes and Whisper transcribes audio. Second, Minimax M3 via Fireworks AI analyzes frames and transcript together, producing a structured JSON scene description covering setting, subjects, actions, notable details, and audio context. Third, this scene JSON — not the raw video — is passed to four tone-specific prompts to generate captions, ensuring all four are factually grounded in identical scene facts while differing only in expression. Fourth, a self-judge loop powered by the same model scores each caption on accuracy and tone fit, mirroring the hackathon's own LLM-judge evaluation criterion and surfacing weak captions before submission. The Streamlit demo UI supports three modes: a results dashboard showing all processed clips with self-judge scores, a live video upload tab, and a URL/Google Drive input tab for downloading and processing clips on demand. The entire pipeline is containerized via Docker, with the pre-built image publicly available on Docker Hub for one-command deployment without any local setup.

Ensemble video-caption agent tuned for style-match

Ensemble video-caption agent tuned for style-match

Track2 Captioner is a Dockerized video-captioning agent: it reads /input/tasks.json, downloads each clip, and writes four styled English captions (formal, sarcastic, humorous_tech, humorous_non_tech) to /output/results.json. The engine is a vision-model ENSEMBLE. FFmpeg samples keyframes (scene-change detection + uniform fill), then three frontier vision models — GPT-5.5, Gemini 3.1 Pro and Claude Opus 4.5 — observe the frames independently, each returning an exhaustive list of concrete visual details. A writer model cross-references the three lists: details confirmed by 2+ models are trusted; single-model specific claims (exact colors, counts, sign text) are dropped unless corroborated. Detection comes from the UNION of what the models see; precision comes from cross-model AGREEMENT. Measured on the 15 official AMD clips via an adversarial vision audit: 0.942 caption accuracy, ~14.8 verified visual details per caption — about 3x any single model — and it reads street signs correctly that every individual model misreads alone. Style match is engineered separately: each style has a dedicated system prompt with few-shot examples and explicit bans on the other styles' traits (sarcastic and humorous_non_tech ban all technology words; formal bans jokes and first person). Whole-word style filters plus a repair-over-reject normalizer guarantee no caption is ever missing, malformed, or off-style. The runtime never fails: ensemble -> single-model pipeline -> free-tier degradation, inline 429 backoff honoring Retry-After, per-task timeouts, multi-provider failover, and Pydantic validation of the final JSON. All 15 official clips verified end-to-end even on the zero-credit fallback path: 60 captions, zero failures. Public image, linux/amd64, 0.25 GB: ghcr.io/theskygold/track2-captioner:latest

CineScribe : Style-Conditioned Captioning Agent

CineScribe : Style-Conditioned Captioning Agent

CineScribe is a next-generation video captioning agent built to solve the challenges of context-aware, stylistically diverse video description. By leveraging the latest serverless models on Fireworks AI, CineScribe breaks down video streams into rich chronological visual-audio event sequences and translates them into customized tones (Formal, Sarcastic, Humorous Tech, and Humorous Non-Tech). Key Innovation: 2-Stage Visual-Speech Fusion Stage 1 (VLM Scene Analysis & Speech Transcription): The agent dynamically segments video into scenes. Using the Kimi K2.6 Vision-Language Model on Fireworks, it extracts granular details (Subject, Action, Setting, OCR, and Objects) at key representative frame midpoints. Concurrently, it runs Whisper transcription to align any spoken content. Stage 2 (Stylistic Tone Adaptation): Instead of writing boring template-based captions, a high-performance LLM (DeepSeek V4 Flash) ingests the factual visual-audio timeline. Running at 0.6 temperature, it synthesizes creative, natural descriptions in the requested styles without compromising on accuracy. Core Features Zero Truncation: Configured with a 2000-token limit to allow deep reasoning-based visual descriptors. 100% Zero-Hardcoded: Fully generalizes across animal, urban, technology, cooking, and nature clips.

SellerKavach

SellerKavach

Social commerce is exploding in India, with millions of MSMEs and micro-entrepreneurs selling directly through WhatsApp, Instagram, and Telegram. However, unlike large e-commerce brands that use advanced Machine Learning to score checkout risk, these chat-based sellers operate completely blind. Relying heavily on Cash on Delivery (COD), they face catastrophic Return to Origin (RTO) rates—often exceeding 50%. Every undelivered package costs the seller ₹150–₹300 in wasted shipping and packaging, silently killing their margins. Enter SellerKavach, an AI-powered order intelligence layer built explicitly for India’s unorganized chat-sellers. Without requiring a website or any change in workflow, SellerKavach plugs directly into a seller's social channels. When a buyer sends a messy, Hinglish message with a vague address (e.g., "blue kurti bhej do, address: pink house mandir ke paas"), our AI instantly takes action. First, an LLM-powered extraction agent structures the chaotic chat into clean JSON. Next, an Address Intelligence pipeline resolves vague landmarks into actionable pin codes. A robust Risk Engine then scores the order's delivery likelihood. Finally, a LangGraph-powered Action Decision agent autonomously handles the situation: auto-confirming safe orders, nudging medium-risk buyers to verify their intent, or warning the seller to demand a prepaid advance for high-risk orders. The ultimate moat of SellerKavach is its Buyer Trust Network—a privacy-preserving, cross-seller database that aggregates hashed trust signals. If a buyer defaults on a shoe seller today, a clothing seller is protected tomorrow. Built for Industry 4.0 & 5.0, SellerKavach democratizes enterprise-grade AI and fraud prevention, transforming social commerce from chaos to intelligence.