How to Summarize PDF Files with Python and LLMs

# PDF Executive Summary Generator

## Overview
This workflow processes a folder of research PDFs, extracts text (including tables) using `pdfplumber`, chunks long documents to respect LLM context limits, and uses the OpenAI API to generate ~200-word executive summaries via a map-reduce strategy (summarize chunks, then combine). Results are saved as JSON for easy downstream use.

## Step-by-Step Guide
1. **Install dependencies**: `pdfplumber`, `openai`, and `tiktoken` for token counting.
2. **Set your API key** as an environment variable (`OPENAI_API_KEY`) — never hardcode it.
3. **Extract text page-by-page** with `pdfplumber`, including table rows converted to readable strings.
4. **Chunk the text** by token count (e.g., 3,000 tokens/chunk) to fit comfortably within model limits.
5. **Map step**: summarize each chunk concisely.
6. **Reduce step**: combine chunk summaries into a single 200-word executive summary.
7. **Iterate over the folder** and save outputs to a JSON file.

## Complete Python Solution

```python
import os
import json
from pathlib import Path
import pdfplumber
import tiktoken
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
MODEL = "gpt-4o-mini"
enc = tiktoken.get_encoding("cl100k_base")

def extract_pdf_text(pdf_path: Path) -> str:
    text_parts = []
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            text_parts.append(page.extract_text() or "")
            for table in page.extract_tables() or []:
                for row in table:
                    text_parts.append(" | ".join(c or "" for c in row))
    return "\n".join(text_parts)

def chunk_text(text: str, max_tokens: int = 3000) -> list[str]:
    tokens = enc.encode(text)
    return [enc.decode(tokens[i:i+max_tokens]) for i in range(0, len(tokens), max_tokens)]

def summarize(prompt: str, text: str, max_tokens: int = 400) -> str:
    resp = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "system", "content": prompt},
                  {"role": "user", "content": text}],
        temperature=0.3,
        max_tokens=max_tokens,
    )
    return resp.choices[0].message.content.strip()

def summarize_pdf(pdf_path: Path) -> str:
    text = extract_pdf_text(pdf_path)
    chunks = chunk_text(text)
    partials = [summarize("Summarize the key findings, methods, and conclusions concisely.", c)
                for c in chunks]
    combined = "\n\n".join(partials)
    final = summarize(
        "Write a polished ~200-word executive summary covering objective, methods, findings, and implications.",
        combined, max_tokens=350)
    return final

def process_folder(folder: str, out_file: str = "summaries.json"):
    results = {}
    for pdf in Path(folder).glob("*.pdf"):
        print(f"Processing {pdf.name}...")
        try:
            results[pdf.name] = summarize_pdf(pdf)
        except Exception as e:
            results[pdf.name] = f"ERROR: {e}"
    Path(out_file).write_text(json.dumps(results, indent=2))
    print(f"Saved to {out_file}")

if __name__ == "__main__":
    process_folder("./pdfs")
```

## Expected Output Structure
```json
{
  "paper_01.pdf": "This study investigates ... Methods included ... Results show ... Implications suggest ...",
  "paper_02.pdf": "The authors examine ..."
}
```

## Notes on Large Files & API Keys
- **Large PDFs**: 50-page papers typically produce 15k–30k tokens; the map-reduce chunking handles this safely. Increase `max_tokens` per chunk only if using a larger-context model.
- **API keys**: Store in `~/.zshrc` as `export OPENAI_API_KEY="sk-..."`. Consider adding retry logic (`tenacity`) and rate-limit backoff for batch jobs.

# PDF Summarization Workflow for Research Documents

## Overview
This workflow extracts text from multi-page research PDFs using **pdfplumber** (ideal for table handling), chunks the content to respect OpenAI API token limits, and generates concise 200-word executive summaries via the GPT-4 API. The solution processes entire folders efficiently with error handling.

## Step-by-Step Guide

1. **Install dependencies** – pdfplumber for extraction, openai for API calls
2. **Set up OpenAI API key** – store securely as environment variable
3. **Extract text from PDF** – use pdfplumber to preserve structure
4. **Chunk long content** – split text to stay within token limits (~3,000 tokens per chunk)
5. **Call OpenAI API** – send concatenated chunks with summarization prompt
6. **Save results** – write summaries to CSV or text files
7. **Process folder** – loop through all PDFs with error handling

## Complete Python Code

```python
import os
import pdfplumber
from openai import OpenAI
import csv
from pathlib import Path

# Initialize OpenAI client (uses OPENAI_API_KEY environment variable)
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def extract_text_from_pdf(pdf_path):
    """Extract text from PDF using pdfplumber."""
    text = ""
    try:
        with pdfplumber.open(pdf_path) as pdf:
            for page in pdf.pages:
                text += page.extract_text() or ""
        return text
    except Exception as e:
        print(f"Error extracting {pdf_path}: {e}")
        return None

def chunk_text(text, chunk_size=3000):
    """Split text into chunks by approximate token count."""
    words = text.split()
    chunks = []
    current_chunk = []
    
    for word in words:
        current_chunk.append(word)
        if len(" ".join(current_chunk)) > chunk_size:
            chunks.append(" ".join(current_chunk))
            current_chunk = []
    
    if current_chunk:
        chunks.append(" ".join(current_chunk))
    return chunks

def generate_summary(text, pdf_name):
    """Generate 200-word summary using OpenAI API."""
    chunks = chunk_text(text)
    combined_text = " ".join(chunks[:5])  # Use first ~15k tokens
    
    try:
        response = client.chat.completions.create(
            model="gpt-4",
            messages=[
                {
                    "role": "system",
                    "content": "You are an expert research analyst. Write clear, concise executive summaries."
                },
                {
                    "role": "user",
                    "content": f"Summarize this research paper in exactly 200 words, focusing on key findings, methodology, and conclusions:\n\n{combined_text}"
                }
            ],
            max_tokens=250,
            temperature=0.5
        )
        return response.choices[0].message.content
    except Exception as e:
        print(f"API error for {pdf_name}: {e}")
        return None

def process_pdf_folder(folder_path, output_csv="summaries.csv"):
    """Process all PDFs in folder and save summaries."""
    pdf_files = list(Path(folder_path).glob("*.pdf"))
    results = []
    
    for i, pdf_path in enumerate(pdf_files, 1):
        print(f"Processing {i}/{len(pdf_files)}: {pdf_path.name}")
        
        text = extract_text_from_pdf(str(pdf_path))
        if not text:
            continue
        
        summary = generate_summary(text, pdf_path.name)
        if summary:
            results.append({"filename": pdf_path.name, "summary": summary})
    
    # Save to CSV
    with open(output_csv, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["filename", "summary"])
        writer.writeheader()
        writer.writerows(results)
    
    print(f"\n✓ Processed {len(results)} PDFs. Results saved to {output_csv}")

# Usage
if __name__ == "__main__":
    process_pdf_folder("./research_pdfs")
```

## Key Configuration Notes

**API Key Setup (macOS):**
```bash
export OPENAI_API_KEY="sk-your-key-here"
python script.py
```

**Large File Handling:** The code chunks text intelligently and samples the first ~15k tokens to avoid excessive API costs. For 50-page PDFs, budget ~$0.05–0.10 per summary (GPT-4 pricing). Monitor usage via OpenAI dashboard.

**Expected Output:** CSV with filename and 200-word summary per row.

## Overview

Use `pdfplumber` to extract text page by page from each PDF, split the text into manageable chunks, summarize each chunk with the OpenAI API, then combine those chunk summaries into a final ~200-word executive summary. This works well for 50-page research PDFs and is easy to run on macOS with Python 3.11.

## Step-by-Step Guide

1. **Install dependencies**
   ```bash
   pip install pdfplumber openai python-dotenv
   ```

2. **Set your API key**
   Create a `.env` file:
   ```env
   OPENAI_API_KEY=your_api_key_here
   ```

3. **Prepare your PDF folder**
   Put all PDFs in a folder like `./pdfs`.

4. **Extract text with pdfplumber**
   `pdfplumber` handles many research PDFs better than basic extractors and can preserve table text more reliably.

5. **Chunk long documents**
   Split extracted text into chunks so each API request stays within model limits.

6. **Summarize in two passes**
   First summarize each chunk, then summarize the combined chunk summaries into one 200-word executive summary.

7. **Save results**
   Write each summary to a `.txt` file or print it to the console.

## Python Code

```python
import os
from pathlib import Path
from dotenv import load_dotenv
import pdfplumber
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

PDF_FOLDER = Path("./pdfs")
OUTPUT_FOLDER = Path("./summaries")
OUTPUT_FOLDER.mkdir(exist_ok=True)

MODEL = "gpt-4o-mini"

def extract_text_from_pdf(pdf_path):
    pages = []
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            text = page.extract_text() or ""
            tables = page.extract_tables()
            table_text = ""
            for table in tables:
                for row in table:
                    row = [cell if cell else "" for cell in row]
                    table_text += " | ".join(row) + "\n"
            pages.append(text + "\n" + table_text)
    return "\n".join(pages).strip()

def chunk_text(text, max_chars=12000):
    chunks, current = [], ""
    for para in text.split("\n\n"):
        if len(current) + len(para) < max_chars:
            current += para + "\n\n"
        else:
            if current.strip():
                chunks.append(current.strip())
            current = para + "\n\n"
    if current.strip():
        chunks.append(current.strip())
    return chunks

def chat_summary(prompt):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": "You summarize research papers clearly for executives."},
            {"role": "user", "content": prompt}
        ],
        temperature=0.2
    )
    return response.choices[0].message.content.strip()

def summarize_pdf(pdf_path):
    text = extract_text_from_pdf(pdf_path)
    if not text:
        return "No extractable text found."

    chunks = chunk_text(text)
    chunk_summaries = []

    for i, chunk in enumerate(chunks, 1):
        prompt = f"Summarize this research PDF section in 120 words or less, focusing on purpose, methods, findings, and implications:\n\n{chunk}"
        chunk_summaries.append(chat_summary(prompt))

    combined = "\n".join(chunk_summaries)
    final_prompt = (
        "Create a concise executive summary of about 200 words from these section summaries. "
        "Include: objective, methodology, key findings, and business/scientific implications.\n\n"
        f"{combined}"
    )
    return chat_summary(final_prompt)

def main():
    for pdf_file in PDF_FOLDER.glob("*.pdf"):
        print(f"Processing: {pdf_file.name}")
        summary = summarize_pdf(pdf_file)
        out_file = OUTPUT_FOLDER / f"{pdf_file.stem}_summary.txt"
        out_file.write_text(summary, encoding="utf-8")
        print(f"Saved: {out_file}")

if __name__ == "__main__":
    main()
```

## Expected Output Structure

```text
Title: [Optional paper title]

Executive Summary:
This paper examines...
The authors used...
Key findings show...
The main implication is...
```

## Notes

For large files, chunking is essential to avoid token limits and reduce API errors. Keep your API key in `.env` and never hardcode or commit it to version control.

Of course. Here is a complete, actionable workflow to summarize your research PDFs using Python, `pdfplumber`, and the OpenAI API.

### Overview

This solution automates the summarization of PDF documents located in a specified folder. The script will iterate through each PDF, extract its text content using `pdfplumber`, and handle long documents by breaking the text into smaller chunks. It then calls the OpenAI API to summarize these chunks and combines them into a final, concise 200-word executive summary for each file.

### Step-by-Step Guide

1.  **Install Libraries:** Open your terminal and install the necessary Python libraries.
    ```bash
    pip install pdfplumber openai python-dotenv
    ```
2.  **Set Up OpenAI API Key:** Create a file named `.env` in the same directory as your script. Add your API key to this file to keep it secure:
    ```
    OPENAI_API_KEY="your-api-key-here"
    ```
3.  **Organize Files:** Create a folder named `pdfs` in the same directory and place all the PDF files you want to summarize inside it.
4.  **Run the Script:** Save the code below as a Python file (e.g., `summarize_pdfs.py`) and run it from your terminal:
    ```bash
    python summarize_pdfs.py
    ```

### Python Summarization Script

```python
import os
import pdfplumber
import openai
from dotenv import load_dotenv

# --- Configuration ---
# Load environment variables from .env file (for OPENAI_API_KEY)
load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")

# Folder containing your PDF files
PDF_FOLDER = "pdfs"
MAX_TOKENS_PER_CHUNK = 2000 # Max tokens for the model (GPT-3.5 is ~4096)
SUMMARY_WORD_COUNT = 200

def extract_text_from_pdf(pdf_path):
    """Extracts text from a PDF file using pdfplumber."""
    full_text = ""
    try:
        with pdfplumber.open(pdf_path) as pdf:
            for page in pdf.pages:
                page_text = page.extract_text()
                if page_text:
                    full_text += page_text + "\n"
    except Exception as e:
        print(f"Error reading {pdf_path}: {e}")
        return None
    return full_text

def get_summary_from_openai(text, word_count):
    """Generates a summary for the given text using the OpenAI API."""
    try:
        response = openai.chat.completions.create(
            model="gpt-3

### Overview
This Python script processes a folder of research PDFs using pdfplumber for robust text extraction (including tables as text), chunks long docs to fit OpenAI limits, generates chunk summaries, then creates a final 200-word executive summary per PDF via GPT-4o-mini. Outputs JSON with summaries. Install: `pip install pdfplumber openai`.

### Step-by-Step Guide
1. **Set up**: Add your OpenAI API key as env var `OPENAI_API_KEY`. Place PDFs in a folder (e.g., `./pdfs/`).
2. **Run script**: Update `folder_path`, execute `python summarize_pdfs.py`. Handles 50-page PDFs by chunking ~4k chars each.
3. **Process**: Extracts text/tables per page, chunks text, summarizes chunks (100 words each), meta-summarizes to 200-word executive summary.
4. **Output**: JSON file `summaries.json` with `{ "filename.pdf": "200-word summary..." }`. Sample: `{"research1.pdf": "Key findings: ... (198 words)"}`.

### Python Code
```python
import pdfplumber
import openai
import os
import json
from dotenv import load_dotenv  # Optional: pip install python-dotenv

load_dotenv()  # Loads .env with OPENAI_API_KEY
client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def extract_text(pdf_path):
    text = ""
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            page_text = page.extract_text()
            if page_text:
                text += page_text + "\n"
            tables = page.extract_tables()
            for table in tables:
                text += "\n".join([" | ".join(row) for row in table if row]) + "\n"
    return text

def chunk_text(text, chunk_size=4000):
    return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]

def summarize_chunk(chunk):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Summarize this PDF chunk in 100 words:\n{chunk}"}]
    )
    return response.choices[0].message.content

def executive_summary(chunks):
    chunk_summaries = [summarize_chunk(chunk) for chunk in chunks]
    full_summary = " ".join(chunk_summaries)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Create a 200-word executive summary from these chunk summaries:\n{full_summary}"}]
    )
    return response.choices[0].message.content.strip()

folder_path = "./pdfs/"  # Update this
summaries = {}
for filename in os.listdir(folder_path):
    if filename.lower().endswith('.pdf'):
        pdf_path = os.path.join(folder_path, filename)
        text = extract_text(pdf_path)
        chunks = chunk_text(text)
        summary = executive_summary(chunks)
        summaries[filename] = summary
        print(f"Summarized {filename}")

with open("summaries.json", "w") as f:
    json.dump(summaries, f, indent=2)
```

### Notes
For large files (>50 pages), chunking prevents token limits (add error handling if needed). Secure API key in `.env`. Costs ~$0.01/PDF. Test on macOS Python 3.11. (347 words)