Disruptive RainDisruptive RainCookbook
AboutTopicsGuidesAPI docsSource
All guides

Summarizing long documents

Chunk, summarize, and merge outputs with quality checks.

Beginner11 min readApr 24, 2024
CompletionsTextSummarization
Text workflowsRetrieval + memory

Actions

Open in GitHubView as Markdown
Key takeaways
  • Use map-reduce for long inputs.
  • Dedupe facts across chunks.
  • Validate the final summary with spot checks.

Map-reduce summaries

Summarize each chunk separately, then merge into a final summary.

Dedupe facts

Remove repeated facts before producing the final summary.

Quality checks

Run a final verification pass to confirm key facts and numbers.