OpenAI Whisper AI technology Top Builders

Explore the top contributors showcasing the highest number of OpenAI Whisper AI technology app submissions within our community.

OpenAI Whisper

The Whisper models are trained for speech recognition and translation tasks, capable of transcribing speech audio into the text in the language it is spoken (ASR) as well as translated into English (speech translation). Whisper has been trained on 680,000 hours of multilingual and multitask supervised data collected from the web. Whisper is Encoder-Decoder model. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and to-English speech translation.

General
Relese date	September, 2020
Author	OpenAI
Repository	https://github.com/openai/whisper
Type	general-purpose speech recognition model

Start building with Whisper

We have collected the best Whisper libraries and resources to help you get started to build with Whisper today. To see what others are building with Whisper, check out the community built Whisper Use Cases and Applications.

Tutorials

👉 Discover more Whisper Tutorials on lablab.ai

Boilerplates

Kickstart your development with a GPT-3 based boilerplate. Boilerplates is a great way to headstart when building your next project with GPT-3.

Python Whisper Boilerplate Whisper gpt3 email generator
Streamlit Whisper, GPT-3 Boilerplate Whisper gpt3 sentiment analysis
React app to upload audio file boilerplate Uploading audio file to an API endpoint
Whisper Flask API Boilerplate Whisper API with Flask
Whisper Flask API with GPT-3 Boilerplate Whisper API with Flask and GPT-3
Whisper Streamlit Boilerplate Automatic Speech Recognition using OpenAI's Whisper

Libraries

Whisper API libraries and connectors.

Whisper API Whisper API reference
OpenAI Node.js API library for Whisper The OpenAI Node.js library provides convenient access to the OpenAI API from Node.js applications
OpenAI Python Library for Whisper The OpenAI Python library provides convenient access to the OpenAI API from applications written in the Python language

Edit on Github

OpenAI Whisper AI technology Hackathon projects

Discover innovative solutions crafted with OpenAI Whisper AI technology, developed by our community members during our engaging hackathons.

ePluribusUnum

ePluribusUnum is a comprehensive translation platform that enables seamless speech-to-speech and document-to-document translation, with a specific focus on Ukrainian to Dutch (and vice versa) translation. The platform combines traditional translation methods with advanced LLM capabilities to provide accurate and context-aware translations.By leveraging the power of LLM, ePluribusUnum goes beyond standard translation, ensuring clarity, context relevance, and an intuitive experience across both spoken and written language translations. Its responsive design and advanced text-to-speech (TTS) capabilities make it suitable for various devices and environments, aiming to bridge language barriers and foster better understanding between users worldwide.

911 Agent

The 911 emergency response system is essential for public safety in the United States, yet it faces significant challenges, particularly in meeting diverse language needs and reducing wait times. Many emergency calls experience delays due to dispatcher shortages, creating queues that can hinder timely assistance. Additionally, existing services struggle to fully support non-English speakers and individuals with disabilities, creating critical gaps in communication. Our project aims to solve these issues by deploying AI agents to supplement the current 911 infrastructure. These agents work alongside human operators, analyzing caller tone, and providing multilingual support to streamline communication. By enhancing threat identification and response prioritization, they help reduce the time it takes to connect individuals with the appropriate emergency services. Designed for nationwide implementation, this solution enables users to access 911 services via video, audio, or text, addressing accessibility needs. Ultimately, our AI-enhanced approach aims to reduce latency, improve response times, and increase the efficiency of emergency response systems across the country.

Cheap and accurate video translation for masses

VideoLangua: Bridging the Language Divide through AI-Powered Video Translation English dominates the world of online content, with more than 60% of digital content in English. This bias also reflects in large language model training, where English remains the primary language. Yet, 75% of the world's population does not speak English, creating a significant information divide. For non-English speakers, the cost of translation is prohibitive—for instance, professional video subtitling can range from $0.18 to $0.25 per word, according to Marshub. VideoLangua addresses this challenge by offering cheap and accurate AI translation services to bridge language barriers, making information more accessible to non-English speakers. Our solution leverages a suite of large language models and specialized agents for efficient, culturally sensitive, and accurate video translation: 1. Speech-to-Text Conversion: We use Whisper for accurate transcription, including timestamps. 2. Translation: For translation, we call Llama 3.1-70B-Chinese and Llama 3.1 70B to translate text. To ensure translation accuracy, Llama 3.2 3B, which supports eight languages, reviews the translation. 3. Subtitle Generation: Translated text is synced back to the video’s timeline as subtitles. 4. Optional Dubbing: For users who need voiceover in their native language, we offer an option for dubbing, ideal for video content files. With the global translation service market valued at $40.95 billion in 2023 and projected to reach $41.78 billion in 2024, VideoLangua is well-positioned to capture a share by offering a high-quality, affordable alternative to traditional video translation services. Unlike typical text-based translation tools, VideoLangua focuses on video and aduio translation, combining affordability with quality. Our solution uses Llama fine-tuned for different languages, allowing users to easily access translation and dubbing features in multiple languages.

Smarter AI for Offline Car Communication

In-Car AI Agents is an innovative project designed to make in-car artificial intelligence smarter and more responsive in offline scenarios. Currently, most AI-powered car assistants rely on internet connectivity to process commands, leaving drivers without key functionalities in areas with poor or no network coverage. Our solution aims to address this gap by deploying edge-based AI models that can function without Wi-Fi, allowing drivers to interact with their cars using voice commands for basic operations such as controlling the air conditioning, navigation prompts, and music playback. The system uses pre-trained voice recognition models and offline edge computing to deliver real-time responses to the driver’s requests, all processed locally. Additionally, the project explores simple computer vision tasks like lane detection, integrated to run without network dependency. Our vision is to make cars smarter, safer, and more autonomous without requiring constant internet access. By providing an MVP, we aim to showcase how AI in cars can be optimized for offline use, thus improving the overall driving experience in areas where internet connectivity is unreliable.

LinguaPolis - Bridging Languages and Uniting Teams

Today we have workforces spread across multiple countries, cultures, and languages. Online meetings have become a key part of how businesses operate. But there's a significant challenge: not everyone speaks the same language or communicates effectively in a common one like English. Accents, dialects, or simply varying levels of proficiency often make it difficult for teams to fully understand each other. This problem results in inefficiencies—important points are missed, and post-meeting emails or clarifications are needed, which undermines the very purpose of the meeting. Ultimately, this wastes time, reduces productivity, and slows down decision-making. Consider a typical online meeting scenario. For many, English may not be their first language. Some struggle to express their ideas clearly, while others have difficulty understanding certain accents or pronunciations. Often, by the end of the meeting, there’s a lack of clarity, and participants need to follow up with additional communications, re-explaining points or providing more instructions. Now imagine a scenario where each participant could simply speak in their native language, and everyone else could understand them perfectly—in real time, without the need for human translators or extensive subtitles. This is where our idea comes in: an AI-powered online meeting platform that provides real-time dubbing and multilingual transcription. Each participant can speak comfortably in their native language, and the platform instantly dubs their speech into the local languages of all other participants. Additionally, real-time text transcripts of the conversation are generated in each participant’s language, providing both auditory and visual clarity. At the end of the meeting, the platform even summarizes the discussion, producing meeting minutes in every participant's native language, ensuring that everyone walks away with a clear understanding of what was discussed and the decisions that were made.

SpeakEasy

SpeakEasy is a cutting-edge platform that transforms how users engage with video content. It uses advanced voice cloning technology to seamlessly translate and clone voices in videos across multiple languages, while preserving the unique tone and characteristics of the original speaker. Whether for business, education, or entertainment, SpeakEasy allows content to be globally accessible without losing authenticity. Users can personalize their videos by modifying the tone and style of the cloned voice to suit formal, casual, or neutral settings, offering flexibility for different audiences. SpeakEasy also includes powerful video summarization technology, condensing long videos into concise, 3-4 minute highlight clips. This makes it perfect for social media sharing, quick reviews, and educational snippets, helping users focus on the most important content. With an intuitive interface and real-time progress tracking, SpeakEasy ensures a smooth, seamless experience. Users simply upload their video, track the processing progress, and download the final result—all in just a few clicks. Whether it's a long meeting or a casual vlog, SpeakEasy simplifies video processing while maintaining high quality. Designed for creators, businesses, and educators, SpeakEasy makes video content more accessible and impactful. By combining state-of-the-art technology with an easy-to-use interface, it empowers users to communicate effortlessly across languages and platforms.