Doing research? Use Defuddle to save your token

I keep seeing tokenmaxxing used as shorthand for the idea that more tokens mean more productive AI work. I don’t buy it. High token usage is not the same as a useful result.

I’d rather optimize for value: complete the task with the smallest amount of context that still gives the model what it needs.

Web research is a good example. Raw web pages include navigation, headers, cookie banners, tracking scripts, styles, and framework data. For notes or summaries, I usually want the article itself. That led me to Defuddle, a tool that accepts a URL or HTML, extracts the main content, and returns cleaned HTML or Markdown for an LLM.

I ran a small experiment to see how much context it could remove.

The experiment

The task was to summarize LaunchDarkly’s article on LLM observability in five concise bullets for a backend engineer, covering its purpose, signals, operational practices, and rollout implications.

I counted tokens with OpenAI’s o200k_base tokenizer in two cases: passing the raw page response directly to the model, and passing Defuddle’s output instead.

Both inputs produced the same five-bullet summary. The input size was dramatically different:

Variant	Input tokens	Summary output tokens
Without Defuddle	318,587	149
With Defuddle	6,350	149
Difference	312,237 fewer	0

Defuddle cut the source context by 312,237 tokens, or 98.01%. The summary did not get shorter because the task and the useful source material stayed the same.

What Defuddle keeps and removes

For the LaunchDarkly page, Defuddle retained the article material relevant to the prompt:

• The introduction to observing model behaviour in development, testing, and production.
• Metrics, logs, spans, traces, prompt and context tracking, user feedback, system metrics, and automated evaluations.
• Guidance on data and prompt monitoring, performance, risk and compliance, token tracking, sampling, drift detection, and controlled rollouts.
• LaunchDarkly’s AI Configs workflow and related practices.

It removed the page chrome and infrastructure around that material: navigation, cookie-consent text, headers and footers, CSS, analytics scripts, JavaScript bundles, hydration state, and other framework-generated HTML.

The resulting article document was 33.4 KB and 4,529 words, compared with a 1.106 MB raw HTML response full of browser and application data.

How Defuddle works

Defuddle uses a DOM-based content extraction pipeline:

URL or HTML
  -> fetch and parse into a DOM
  -> extract metadata
  -> identify the main-content container
  -> remove boilerplate and low-value elements
  -> normalize the retained HTML
  -> return cleaned HTML, Markdown, or JSON

Fetch and parse the page

The CLI accepts a URL, a local HTML file, or HTML through standard input.

npx defuddle parse "https://example.com/article"

Extract metadata before cleanup

Before removing scripts, Defuddle reads metadata from page meta tags and structured data such as Schema.org. This can provide a title, author, site, publication date, canonical URL, language, description, and word count.

That ordering matters because JSON-LD metadata often lives inside <script> tags, which are removed from the article output later.

Find the main content

Defuddle uses site-specific extractors for sites including GitHub, Reddit, Medium, Substack, Hacker News, LinkedIn, YouTube, and some AI chat platforms. When there is no dedicated extractor, it uses a generic scoring path that looks for article-like regions.

Paragraph density and document structure count in an element’s favour; navigation menus, sidebars, related-post cards, and repeated site links usually do not.

Remove boilerplate and retry weak extractions

The cleanup pipeline can remove known non-content elements, hidden elements, low-scoring regions, decorative images, repeated metadata, boilerplate patterns, and tags such as <script>, <style>, and <noscript>.

If an extraction returns too little content, Defuddle can retry with less aggressive settings. That fallback matters for unusual layouts and index pages, where aggressive cleanup can accidentally remove useful content.

Normalize and return the result

Defuddle normalizes heading levels, duplicate titles, code blocks, images, tables, footnotes, math, callouts, empty nodes, unnecessary attributes, comments, and trailing headings. The result is more consistent even when the source markup is messy.

# Clean HTML
npx defuddle parse "https://example.com/article"

# Markdown, usually the best format for LLM context
npx defuddle parse --markdown "https://example.com/article"

# Markdown with source metadata
npx defuddle parse --markdown --frontmatter "https://example.com/article"

# Structured data
npx defuddle parse --json "https://example.com/article"

Why the page shrank so much

The raw response was mostly JavaScript and framework payload, styles, analytics, navigation, consent UI, and page chrome. Defuddle kept the article body, its useful headings, and its tables, then dropped the surrounding infrastructure.

It also keeps document structure rather than flattening everything into plain text. That makes it a good fit for summaries, retrieval, research, and fact extraction.

Verdict

For public, article-like pages, I would run Defuddle before sending content to an LLM. It does not change the requested summary length, but it can substantially reduce input cost and keep irrelevant HTML out of the model’s context.