Building Your AI Podcast Assistant

As a podcast enthusiast, especially ones on health or business, I often find myself overwhelmed with too many episodes and not enough time. Many podcasts, like the Huberman Lab, can stretch to three or four hours. It’s not just about finding the time; often, I’m only interested in certain parts of the conversation. That’s why I created a straightforward app using OpenAI models, LangChain, and Streamlit. It lets me paste a YouTube podcast link (like Huberman Lab or The Diary of a CEO), and quickly provides a summary of the conversation. Then, I can ask specific questions and get instant answers. This way, I get straight to the information I need without watching the entire episode.

Generated with DALL·E 3.

Understanding RAG

Before diving into the code, let’s briefly discuss RAG. It’s a method that combines the text-generation ability of models like GPT-4 with information retrieval to provide precise, context-relevant information. Traditional Language Learning Models (LLMs) might not always have access to specific data we need. RAG lets us input relevant data (like our podcast transcripts), so the model can deliver pertinent answers.

Here’s the simple process: We load our data (like podcast transcripts), break it into smaller segments, embed these, and store them in a vector database.

image from LangChain

When we query, the model compares our question to these data segments. The most relevant texts are retrieved and fed to the LLM along with our question, generating an accurate answer. In essence, RAG retrieves relevant information and uses it to answer our queries.

image from LangChain

That’s it, at a high level, this is what RAG does. It retrieves the pieces of information that are relevant to your query, and then uses both to generate an answer.

How the App Works

The app is user-friendly: paste a YouTube podcast URL into the Streamlit interface, enter your OpenAI key, and voilà! In moments, you’ll have a summary of the podcast. You can then ask specific questions like, “What books were recommended?” or “What are the top sleep tips mentioned?”.

The code is split into three main parts: fetching data from YouTube, generating summaries and answers, and the Streamlit frontend. There’s an optional fourth part for advanced RAG usage, but the basic version works quite well. Let’s dive into the code!

Part 1: Interacting with YouTube

To start, our app connects to YouTube, retrieving video titles and captions — key to understanding the podcast’s content.

Function 1: Fetching Video Titles

This function grabs a YouTube video’s title using its URL. It sends a web request, parses the HTML to find the title tag, and returns the title. It’s our first step to knowing what each video is about.

import requests
from bs4 import BeautifulSoup

def get_youtube_video_title(video_url):
 response = requests.get(video_url)
 soup = BeautifulSoup(response.content, 'html.parser')
 title = soup.find('meta', property='og:title')
 return title['content'] if title else "Title not found"

Function 2: Gathering Captions

Next, we fetch the video’s captions. We use the title-fetching function and employ the YoutubeLoader from LangChain to load captions. These transcripts will feed into our RAG system for generating relevant responses.

from langchain.document_loaders import YoutubeLoader
from langchain.schema import Document

def fetch_youtube_captions(video_url):
 title = get_youtube_video_title(video_url)
 loader = YoutubeLoader.from_youtube_url(video_url)
 docs = loader.load()
 if docs and len(docs) > 0:
 intro_sentence = "This is the title of the video/transcription/conversation: "
 title_content = intro_sentence + title
 docs[0] = Document(page_content=title_content + "\n\n" + docs[0].page_content)
 return docs

Part 2: Data Processing and AI Integration

Core App Logic: Turning Conversations into Data

This is where the magic happens. We’ll break down the podcast into digestible pieces, transform them for our AI, and store them for easy retrieval and summary generation.

Environment Setup and Global Variables

We begin by setting up our coding environment and defining global variables. This includes initializing a database and a memory system for our conversations.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from sklearn.cluster import KMeans
from langchain.chat_models import ChatOpenAI
from langchain.chains.summarize import load_summarize_chain
from langchain.schema import Document
import numpy as np
from langchain.chains import ConversationalRetrievalChain
import logging
from langchain.prompts import PromptTemplate
from langchain.memory import ConversationBufferMemory


logging.basicConfig(level=logging.INFO) 
# Initialize global variables
global_chromadb = None
global_documents = None
global_short_documents = None

Functions for Data Management

These functions are crucial for managing your application’s data. reset_globals resets the global variables, init_chromadb initializes a database with the processed data.

# Initialize the memory outside the function so it persists across different calls
conversation_memory = ConversationBufferMemory(
 memory_key="chat_history",
 max_len=50,
 input_key="question",
 output_key="answer",
 return_messages=True,
)

# Function to reset global variables
def reset_globals():
 global global_chromadb, global_documents, global_short_documents
 global_chromadb = None
 global_documents = None
 global_short_documents = None
 # Reset the conversation memory
 if conversation_memory:
 conversation_memory.clear()

def init_chromadb(openai_api_key):
 global global_chromadb, global_short_documents
 if global_chromadb is None and global_short_documents is not None:
 global_chromadb = Chroma.from_documents(documents=global_short_documents, embedding=OpenAIEmbeddings(openai_api_key=openai_api_key))

Processing Captions and Generating Summaries

process_and_cluster_captions: Here, we process the YouTube captions, preparing them for analysis and response generation. Here’s the process:

1. Initial Data Check: We begin by verifying the format of the captions, ensuring they’re suitable for processing.
2. Dividing Captions: The captions are divided into smaller segments for two purposes: one set for summarization and another for question-answering. This division is crucial for handling specific tasks efficiently.
3. Clustering for Relevance: We use the KMeans algorithm to cluster the summary segments. This is a strategic move to filter out redundant content and focus only on the most representative parts of the podcast. By selecting one segment from each cluster, we ensure that the AI receives diverse yet concise information, perfect for creating a meaningful summary.
4. Global Accessibility: The segmented captions are stored globally, allowing easy access for both summarization and answering queries.

def process_and_cluster_captions(captions, openai_api_key, num_clusters=12):
 global global_documents, global_short_documents 
 logging.info("Processing and clustering captions")

 # Log the first 500 characters of the captions to check their format
 logging.info(f"Captions received (first 500 characters): {captions[0].page_content[:500]}")
 caption_content = captions[0].page_content

 # Ensure captions is a string before processing
 if not isinstance(caption_content, str):
 logging.error("Captions are not in the expected string format")
 return []

 # Create longer chunks for summary
 summary_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0, separators=["\n\n", "\n", " ", ""])
 summary_docs = summary_splitter.create_documents([caption_content])

 # Create shorter chunks for QA
 qa_splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=0, separators=["\n\n", "\n", " ", ""])
 qa_docs = qa_splitter.create_documents([caption_content])

 # Process for summary
 summary_embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key).embed_documents([x.page_content for x in summary_docs])
 kmeans = KMeans(n_clusters=num_clusters, random_state=42).fit(summary_embeddings)
 closest_indices = [np.argmin(np.linalg.norm(summary_embeddings - center, axis=1)) for center in kmeans.cluster_centers_]
 representative_docs = [summary_docs[i] for i in closest_indices]

 # Store documents globally
 global_documents = summary_docs # For summary
 global_short_documents = qa_docs # For QA

 init_chromadb(openai_api_key) # Initialize database with longer chunks
 return representative_docs

generate_summary: This function takes the processed captions and generates a clear, concise summary of the video, leveraging an AI model from OpenAI. Here’s how:

1. Combining Key Segments: We amalgamate the chosen segments from the clustering process to form a continuous narrative. This ensures that the summary covers different aspects of the podcast.
2. Guided AI Summary: Using a custom prompt, we instruct the AI to focus on creating a concise and informative summary. This step is crucial for directing the AI’s attention to the overarching themes and important points, avoiding unnecessary details.
3. Executing Summary Chain: The AI then processes this combined text and generates a summary, offering users a quick understanding of the podcast’s core content.

This method ensures the generation of a summary that is both comprehensive and to the point, capturing the essence of the podcast.

def generate_summary(representative_docs, openai_api_key, model_name):
 logging.info("Generating summary")
 llm4 = ChatOpenAI(model_name=model_name, temperature=0.2, openai_api_key=openai_api_key)

 # Concatenate texts for summary
 summary_text = "\n".join([doc.page_content for doc in representative_docs])

 summary_prompt_template = PromptTemplate(
 template=(
 "Create a concise summary of a podcast conversation based on the text provided below. The text consists of selected, representative sections from different parts of the conversation. "
 "Your task is to synthesize these sections into a single cohesive and concise summary. Focus on the overarching themes and main points discussed throughout the podcast. "
 "The summary should give a clear and complete understanding of the conversation's key topics and insights, while omitting any extraneous details. It should be engaging and easy to read, ideally in one or two paragraphs. Keep it short where possible"
 "\n\nSelected Podcast Sections:\n{text}\n\nSummary:"
 ),
 input_variables=["text"]
 )
 # Load summarizer chain
 summarize_chain = load_summarize_chain(llm=llm4, chain_type="stuff", prompt=summary_prompt_template)

 # Run the summarizer chain
 summary = summarize_chain.run([Document(page_content=summary_text)])

 logging.info("Summary generation completed")
 return summarypythonCopy code

Answering User Questions

answer_question: The heart of our RAG system. This function uses the processed data to answer specific user queries, showcasing the power of retrieval-augmented generation.

1. Database Initialization: It first checks if the database, containing the processed segments, is ready. If not, it initializes the database for efficient retrieval.
2. Retrieval and Response Generation: The function then combines the user’s query with the relevant data retrieved from our database. This retrieval is tailored to find the most pertinent context for the question, ensuring that the AI’s response is accurate and relevant.
3. Generating Context-Aware Answers: The AI, equipped with the question and the best-matched context, crafts an answer that is succinct and directly addresses the user’s query.

Through this process, users receive targeted, informed responses, enhancing their interaction with the podcast content.

def answer_question(question, openai_api_key, model_name):
 llm4 = ChatOpenAI(model_name=model_name, temperature=0, openai_api_key=openai_api_key)
 global global_chromadb, global_short_documents

 if global_chromadb is None and global_short_documents is not None:
 init_chromadb(openai_api_key, documents=global_short_documents)

 logging.info(f"Answering question: {question}")
 chatTemplate = """
 You are an AI assistant tasked with answering questions based on context from a podcast conversation. Use the provided context and relevant chat messages to answer. If unsure, say so. Keep your answer to three sentences or less, focusing on the most relevant information.
 Chat Messages (if relevant): {chat_history}
 Question: {question} 
 Context from Podcast: {context} 
 Answer:"""
 QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question", "chat_history"],template=chatTemplate)
 qa_chain = ConversationalRetrievalChain.from_llm(
 llm=llm4, 
 chain_type="stuff", 
 retriever=global_chromadb.as_retriever(search_type="mmr", search_kwargs={"k":12}),
 memory=conversation_memory,
 #return_source_documents=True,
 combine_docs_chain_kwargs={'prompt': QA_CHAIN_PROMPT},
 )
 # Log the current chat history
 current_chat_history = conversation_memory.load_memory_variables({})
 logging.info(f"Current Chat History: {current_chat_history}")
 response = qa_chain({"question": question}) 
 logging.info(f"this is the result: {response}")
 output = response['answer'] 

 return output

Part 3: Streamlit User Interface

Bringing the App to Life

Finally, we use Streamlit to create an interactive web app to bring it all together. Users can input YouTube links, ask questions, and get AI-generated summaries and answers. The code below is my own version of the interface, feel free to use mine or customize it to your preference.

import streamlit as st
from youtuber import fetch_youtube_captions
from agent import process_and_cluster_captions, generate_summary, answer_question, reset_globals

# Set Streamlit page configuration with custom tab title
st.set_page_config(page_title="GPTpod", page_icon="", layout="wide")

def user_query(question, openai_api_key, model_name):
 """Process and display the query response."""
 # Add the user's question to the conversation
 st.session_state.conversation.append((f"{question}", "user-message"))

 # Check if this query has been processed before
 if question not in st.session_state.processed_questions:
 # Process the query
 answer = answer_question(question, openai_api_key, model_name)
 if isinstance(answer, str):
 st.session_state.conversation.append((f"{answer}", "grimoire-message"))
 else:
 st.session_state.conversation.append(("Could not find a proper answer.", "grimoire-message"))

 st.rerun()

 # Mark this question as processed
 st.session_state.processed_questions.add(question)


# Initialize session state
if 'conversation' not in st.session_state:
 st.session_state.conversation = []
 st.session_state.asked_questions = set()
 st.session_state.processed_questions = set()

# Sidebar for input and operations
with st.sidebar:
 st.title("GPT Podcast Surfer")
 st.image("img.png") 

 # Expandable Instructions
 with st.expander(" How to use:", expanded=False):
 st.markdown("""
 -  **Enter your OpenAI API Key.**
 -  **Paste a YouTube URL.**
 - ‍♂️ **Click 'Run it' to process.**
 - ️‍♂️ **Ask questions in the chat.**
 """)

 # Model selection in the sidebar
 model_choice = st.sidebar.selectbox("Choose Model:", 
 ("GPT-4 Turbo", "GPT-3.5 Turbo"), 
 index=0) # Default to GPT-4 Turbo

 # Map friendly names to actual model names
 model_name_mapping = {
 "GPT-4 Turbo": "gpt-4-1106-preview",
 "GPT-3.5 Turbo": "gpt-3.5-turbo"
 }

 selected_model = model_name_mapping[model_choice]
 st.session_state['selected_model'] = model_name_mapping[model_choice]


 # Input for OpenAI API Key
 openai_api_key = st.text_input("Enter your OpenAI API Key:", type="password")

 # Save the API key in session state if it's entered
 if openai_api_key:
 st.session_state['openai_api_key'] = openai_api_key

 youtube_url = st.text_input("Enter YouTube URL:")

 # Button to trigger processing
 if st.button("Run it"):
 if openai_api_key:
 if youtube_url and 'processed_data' not in st.session_state:
 reset_globals()
 with st.spinner('‍ GPT is cooking up your podcast... hang tight for a few secs'):
 captions = fetch_youtube_captions(youtube_url)
 if captions:
 representative_docs = process_and_cluster_captions(captions, st.session_state['openai_api_key'])
 summary = generate_summary(representative_docs, st.session_state['openai_api_key'], selected_model)
 st.session_state.processed_data = (representative_docs, summary)
 if 'summary_displayed' not in st.session_state:
 st.session_state.conversation.append((f"Here's a rundown of the conversation: {summary}", "summary-message"))
 guiding_message = "Feel free to ask me anything else about it! :)"
 st.session_state.conversation.append((guiding_message, "grimoire-message"))
 st.session_state['summary_displayed'] = True
 else:
 st.error("Failed to fetch captions.")
 else:
 st.warning("Please add the OpenAI API key first.")


# Main app logic
for message, css_class in st.session_state.conversation:
 role = "assistant" if css_class in ["grimoire-message", "summary-message", "suggestion-message"] else "user"
 with st.chat_message(role):
 st.markdown(message)


# Chat input field
if prompt := st.chat_input("Ask me anything about the podcast..."):
 user_query(prompt, st.session_state.get('openai_api_key', ''), st.session_state.get('selected_model', 'gpt-4-1106-preview'))

Here’s what the frontend looks like:

image from the author

Part 4: Optional RAG-Fusion for Enhanced Results

The Concept of RAG-Fusion

RAG-Fusion is an experimental feature I added for a bit of an upgrade. While it’s not necessary, it offers improved results at the cost of slightly more code and a bit of speed. The idea is to refine the AI’s understanding and response accuracy.

Why RAG-Fusion?

• Filling the Gaps: It overcomes some limitations of standard RAG by generating and reevaluating multiple versions of a user’s query. This ensures a broader and more accurate search.
• Better Search Results: It combines Reciprocal Rank Fusion and custom vector scoring, leading to more comprehensive and precise answers.

RAG-Fusion aims to decode not just what users ask but also what they mean to ask, digging deeper to uncover insights that are often missed.

RAG-Fusion’s Workflow

1. Transforming Queries: We start by translating the original user query into several similar yet distinct questions using a language model. This multi-perspective approach is crucial for a thorough search.
2. Enhanced Vector Searches: Each of these new queries undergoes a vector search, gathering diverse results.
3. Smart Reranking: Using reciprocal rank fusion, we reorganize all these results, prioritizing the most relevant ones.
4. Crafted Final Output: The top results, along with the new queries, guide the language model to generate a response that’s informed by a wider context.

Key Functions in RAG-Fusion

• reciprocal_rank_fusion: This function reshuffles the search results based on their relevance scores, ensuring that the best answers are considered first.
• generate_multiple_queries: It creates several variations of the initial query, expanding the search scope.
• answer_question: This is where everything comes together. The function first generates multiple queries, retrieves documents for each, and applies reciprocal rank fusion. It then uses a custom database with these refined results to guide the AI in crafting a more informed and accurate response.

def reciprocal_rank_fusion(results: list[list], k=60):
 fused_scores = {}
 for docs in results:
 # the docs are returned in sorted order of relevance
 for rank, doc in enumerate(docs):
 doc_str = dumps(doc)
 logging.info(f"Serialized Document: {doc_str}")
 if doc_str not in fused_scores:
 fused_scores[doc_str] = 0
 previous_score = fused_scores[doc_str]
 fused_scores[doc_str] += 1 / (rank + k)

 reranked_results = [
 loads(doc)
 for doc, score in sorted(fused_scores.items(), key=lambda x: x[1], reverse=True)
 ]
 logging.info(f"Reciprocal Rank Fusion applied. Reranked Results: {reranked_results[:10]}") # Log top 10 results
 return reranked_results


def generate_multiple_queries(question, llm):
 prompt = PromptTemplate(
 input_variables=["question"],
 template="""You are an AI language model assistant. Your task is to OUTPUT 4 
 different versions of the given user question to retrieve relevant documents from a vector 
 database. By generating multiple perspectives on the user question, your goal is to help
 the user overcome some of the limitations of the distance-based similarity search. 
 Provide these alternative questions separated by newlines.
 Original question: {question}""",
)
 # Create a chain with the language model and the prompt
 llm_chain = LLMChain(llm=llm, prompt=prompt, output_parser=StrOutputParser())

 # Run the chain
 response = llm_chain.run({"question": question})
 queries = response.split("\n")

 logging.info(f"Generated Queries: {queries}")
 return queries


def answer_question(question, openai_api_key, model_name):
 llm4 = ChatOpenAI(model_name=model_name, temperature=0.1, openai_api_key=openai_api_key)
 global global_chromadb, global_short_documents

 if global_chromadb is None and global_short_documents is not None:
 init_chromadb(openai_api_key, documents=global_short_documents)

 logging.info(f"Answering question: {question}")
 # Generate multiple queries
 queries = generate_multiple_queries(question, llm4)

 # Retrieve documents for each query
 results = []
 for query in queries:
 retrieved_docs_with_scores = global_chromadb.similarity_search_with_score(query, k=8)
 # Log the number of documents retrieved for each query and the first 3 docs
 logging.info(f"Retrieved {len(retrieved_docs_with_scores)} documents for query '{query}': {retrieved_docs_with_scores[:3]}")
 results.append(retrieved_docs_with_scores)

 # Apply reciprocal rank fusion
 reranked_results = reciprocal_rank_fusion(results)
 logging.info(f"Number of reranked documents: {len(reranked_results)}")

 #extract the Document object only
 reranked_documents = [doc for doc, _ in reranked_results]

 # Create a new Chroma instance with reranked results
 custom_chromadb = Chroma.from_documents(documents=reranked_documents, embedding=OpenAIEmbeddings(openai_api_key=openai_api_key))

 chatTemplate = """
 You are an AI assistant tasked with answering questions based on context from a podcast conversation.
 Use the provided context and relevant chat messages to answer. If unsure, say so. Keep your answer to four sentences or less, focusing on the most relevant information.
 Chat Messages (if relevant): {chat_history}
 Question: {question} 
 Context from Podcast: {context} 
 Answer:"""
 QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question", "chat_history"],template=chatTemplate)
 qa_chain = ConversationalRetrievalChain.from_llm(
 llm=llm4, 
 chain_type="stuff", 
 retriever=custom_chromadb.as_retriever(search_type="similarity", search_kwargs={"k":10}),
 memory=conversation_memory,
 return_source_documents=True,
 combine_docs_chain_kwargs={'prompt': QA_CHAIN_PROMPT},
 )
 # Log the current chat history
 current_chat_history = conversation_memory.load_memory_variables({})
 logging.info(f"Current Chat History: {current_chat_history}")
 response = qa_chain({"question": question}) 
 logging.info(f"Final response: {response}")
 output = response['answer'] 

 return output

Wrapping up

And there you have it! A step-by-step guide to creating an interactive, AI-powered podcast app. Give it a try here and feel free to leave any feedback.

Thanks for following along and happy coding! 🙂