Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Tutorial 3: reading a page

What you will have at the end. The main content of a web page as Markdown, read in two parts, and a clear picture of what rypsok refuses to read and why.

What you need. Tutorial 1 done.

1. Set the budgets

web_fetch returns at most max_chars characters of a page’s main content. The defaults are 3000 per call and a ceiling of 20000; this file raises the default:

# Tutorial 3: web_fetch with a larger default budget and cross-host
# redirects still refused (the default), so that a page that moved is
# reported, not followed.
config_version = 1

[fetch]
max_chars = 5000
max_chars_ceiling = 20000
follow_cross_host_redirects = false

[engines.wikipedia]
type = "wikipedia"
categories = ["general", "science"]

2. Read a page

{
  "request": {
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
      "name": "web_fetch",
      "arguments": {
        "url": "https://example.org/rust-ownership"
      },
      "_meta": {
        "io.modelcontextprotocol/protocolVersion": "2026-07-28",
        "io.modelcontextprotocol/clientInfo": {
          "name": "tutorial",
          "version": "1"
        },
        "io.modelcontextprotocol/clientCapabilities": {}
      }
    }
  },
  "response": {
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
      "resultType": "complete",
      "content": [
        {
          "type": "text",
          "text": "{\"content\":\"# Rust ownership, in short\\n\\nEvery value in Rust has one owner. When the owner goes out of scope, the value is dropped. A value can be moved to a new owner, or borrowed for a while by a reference.\\n\\n## Borrowing\\n\\nA shared reference lets many readers look at a value at once. A mutable reference lets one writer change it, and nobody else may look while it exists. The compiler checks these rules before the program runs.\\n\\n## Why it matters\\n\\nOwnership replaces a garbage collector with rules the compiler enforces, so memory is freed exactly once and data races are impossible in safe code. The rules take time to learn, and the compiler's messages name the conflicting borrow.\\n\\n- One owner at a time.\\n- Many readers or one writer.\\n- Checked at compile time.\",\"notices\":[],\"outcome\":\"ok\",\"title\":\"Rust ownership, in short\",\"truncated\":false,\"url\":\"https://example.org/rust-ownership\"}"
        }
      ],
      "structuredContent": {
        "content": "# Rust ownership, in short\n\nEvery value in Rust has one owner. When the owner goes out of scope, the value is dropped. A value can be moved to a new owner, or borrowed for a while by a reference.\n\n## Borrowing\n\nA shared reference lets many readers look at a value at once. A mutable reference lets one writer change it, and nobody else may look while it exists. The compiler checks these rules before the program runs.\n\n## Why it matters\n\nOwnership replaces a garbage collector with rules the compiler enforces, so memory is freed exactly once and data races are impossible in safe code. The rules take time to learn, and the compiler's messages name the conflicting borrow.\n\n- One owner at a time.\n- Many readers or one writer.\n- Checked at compile time.",
        "notices": [],
        "outcome": "ok",
        "title": "Rust ownership, in short",
        "truncated": false,
        "url": "https://example.org/rust-ownership"
      },
      "isError": false
    }
  }
}
  • url is the page that was actually read, after a redirect on the same host or an upgrade from http to https.
  • title and content are the page’s main content as Markdown: navigation, headers, footers, sidebars, scripts and anything a browser would not render are left out. Links keep their absolute targets.
  • truncated: false: the whole content fit in max_chars.

3. Read a long page in parts

Ask for 500 characters:

{
  "request": {
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/call",
    "params": {
      "name": "web_fetch",
      "arguments": {
        "url": "https://example.org/rust-ownership",
        "max_chars": 500
      },
      "_meta": {
        "io.modelcontextprotocol/protocolVersion": "2026-07-28",
        "io.modelcontextprotocol/clientInfo": {
          "name": "tutorial",
          "version": "1"
        },
        "io.modelcontextprotocol/clientCapabilities": {}
      }
    }
  },
  "response": {
    "jsonrpc": "2.0",
    "id": 2,
    "result": {
      "resultType": "complete",
      "content": [
        {
          "type": "text",
          "text": "{\"content\":\"# Rust ownership, in short\\n\\nEvery value in Rust has one owner. When the owner goes out of scope, the value is dropped. A value can be moved to a new owner, or borrowed for a while by a reference.\\n\\n## Borrowing\\n\\nA shared reference lets many readers look at a value at once. A mutable reference lets one writer change it, and nobody else may look while it exists. The compiler checks these rules before the program runs.\\n\\n## Why it matters\\n\\nOwnership replaces a garbage collector with rules the compile\",\"next_start_char\":500,\"notices\":[],\"outcome\":\"ok\",\"title\":\"Rust ownership, in short\",\"truncated\":true,\"url\":\"https://example.org/rust-ownership\"}"
        }
      ],
      "structuredContent": {
        "content": "# Rust ownership, in short\n\nEvery value in Rust has one owner. When the owner goes out of scope, the value is dropped. A value can be moved to a new owner, or borrowed for a while by a reference.\n\n## Borrowing\n\nA shared reference lets many readers look at a value at once. A mutable reference lets one writer change it, and nobody else may look while it exists. The compiler checks these rules before the program runs.\n\n## Why it matters\n\nOwnership replaces a garbage collector with rules the compile",
        "next_start_char": 500,
        "notices": [],
        "outcome": "ok",
        "title": "Rust ownership, in short",
        "truncated": true,
        "url": "https://example.org/rust-ownership"
      },
      "isError": false
    }
  },
  "_note": "the second call of the tutorial continues at next_start_char; its answer is in web-fetch-continuation-2.json"
}

truncated: true and next_start_char: 500 say where to continue. The next call passes that number as start_char:

{
  "request": {
    "jsonrpc": "2.0",
    "id": 3,
    "method": "tools/call",
    "params": {
      "name": "web_fetch",
      "arguments": {
        "url": "https://example.org/rust-ownership",
        "max_chars": 500,
        "start_char": 500
      },
      "_meta": {
        "io.modelcontextprotocol/protocolVersion": "2026-07-28",
        "io.modelcontextprotocol/clientInfo": {
          "name": "tutorial",
          "version": "1"
        },
        "io.modelcontextprotocol/clientCapabilities": {}
      }
    }
  },
  "response": {
    "jsonrpc": "2.0",
    "id": 3,
    "result": {
      "resultType": "complete",
      "content": [
        {
          "type": "text",
          "text": "{\"content\":\"r enforces, so memory is freed exactly once and data races are impossible in safe code. The rules take time to learn, and the compiler's messages name the conflicting borrow.\\n\\n- One owner at a time.\\n- Many readers or one writer.\\n- Checked at compile time.\",\"notices\":[],\"outcome\":\"ok\",\"title\":\"Rust ownership, in short\",\"truncated\":false,\"url\":\"https://example.org/rust-ownership\"}"
        }
      ],
      "structuredContent": {
        "content": "r enforces, so memory is freed exactly once and data races are impossible in safe code. The rules take time to learn, and the compiler's messages name the conflicting borrow.\n\n- One owner at a time.\n- Many readers or one writer.\n- Checked at compile time.",
        "notices": [],
        "outcome": "ok",
        "title": "Rust ownership, in short",
        "truncated": false,
        "url": "https://example.org/rust-ownership"
      },
      "isError": false
    }
  }
}

Each call fetches the page again, so a page that changed between the calls may not continue seamlessly. Budgets count characters, not tokens: Japanese text costs several times more tokens per character than English.

4. What a page can answer with

A page that moved to another host is not followed; rypsok reports where it went and lets the caller decide:

{
  "request": {
    "jsonrpc": "2.0",
    "id": 3,
    "method": "tools/call",
    "params": {
      "name": "web_fetch",
      "arguments": {
        "url": "https://example.org/moved"
      },
      "_meta": {
        "io.modelcontextprotocol/protocolVersion": "2026-07-28",
        "io.modelcontextprotocol/clientInfo": {
          "name": "tutorial",
          "version": "1"
        },
        "io.modelcontextprotocol/clientCapabilities": {}
      }
    }
  },
  "response": {
    "jsonrpc": "2.0",
    "id": 3,
    "result": {
      "resultType": "complete",
      "content": [
        {
          "type": "text",
          "text": "{\"error\":{\"detail\":{\"location\":\"https://example.org/new\"},\"kind\":\"redirected\",\"message\":\"the page moved to another host; see detail.location\"}}"
        }
      ],
      "structuredContent": {
        "error": {
          "detail": {
            "location": "https://example.org/new"
          },
          "kind": "redirected",
          "message": "the page moved to another host; see detail.location"
        }
      },
      "isError": true
    }
  }
}

A private or internal address is refused before any request leaves the machine:

{
  "request": {
    "jsonrpc": "2.0",
    "id": 4,
    "method": "tools/call",
    "params": {
      "name": "web_fetch",
      "arguments": {
        "url": "http://192.168.1.10/admin"
      },
      "_meta": {
        "io.modelcontextprotocol/protocolVersion": "2026-07-28",
        "io.modelcontextprotocol/clientInfo": {
          "name": "tutorial",
          "version": "1"
        },
        "io.modelcontextprotocol/clientCapabilities": {}
      }
    }
  },
  "response": {
    "jsonrpc": "2.0",
    "id": 4,
    "result": {
      "resultType": "complete",
      "content": [
        {
          "type": "text",
          "text": "{\"error\":{\"detail\":{\"reason\":\"egress\"},\"kind\":\"denied\",\"message\":\"the destination is not allowed by the egress policy\"}}"
        }
      ],
      "structuredContent": {
        "error": {
          "detail": {
            "reason": "egress"
          },
          "kind": "denied",
          "message": "the destination is not allowed by the egress policy"
        }
      },
      "isError": true
    }
  }
}

Other answers you will meet: denied with reason robots_txt (the site asks not to be read, at the address you gave or at a page it redirected to), unavailable with http_status and the status (the page answered 4xx or 5xx; its body is never returned), unsupported_content (not HTML, XHTML, plain text or Markdown), and too_large (above the byte limit).

5. Strict mode

An operator who wants web_fetch to read only what web_search found turns on strict mode and lists the hosts that may be read anyway:

# Strict mode: web_fetch reads only URLs that web_search returned in this
# process or that the operator allows below. Everything else is denied
# with reason strict_mode before any network request.
config_version = 1

[fetch]
strict_mode = true

[egress]
allow = ["docs.rs", "doc.rust-lang.org"]

[engines.wikipedia]
type = "wikipedia"
categories = ["general", "science"]

Any other URL is denied with reason strict_mode, before any request.

Next: Tutorial 4: network mode.