URL parsing versus permission to make a request

How-to Last tested 25 October 2026

A URL parser identifies components; it does not grant permission to contact the destination. A fetching application needs a separate policy for the scheme, hostname, port, resolved addresses and every redirect, and its connection must use an address that policy checked. A result that passes those checks still is not a verdict that the destination or its content is trustworthy.

Define the request the application is allowed to make

Start with the business requirement, such as retrieving a particular kind of resource from a small set of approved services. An arbitrary URL box gives the caller control over more than a name: protocol, port, credentials, path and query can all change the request. Where the job permits it, accepting an internal resource identifier and constructing the destination from a maintained configuration is easier to constrain than accepting a complete address.

This example studies the complete-address boundary because that is where parser and connection decisions often become separated. Its allowlist contains only reserved example names and explicit scheme/port pairs. It sends a fixed GET without caller-supplied authorization, cookies or headers. In a real application, allowed paths and operations also need review; allowing a hostname does not authorize every action exposed by that service.

OWASP distinguishes constrained destinations from unrestricted fetching and calls for both application and network controls. Its current guidance specifically addresses connecting to the validated address and checking retries and fallback connections¹. The example below has no automatic fallback: it checks the entire returned address set, selects its first address and attempts that selection once. A connection failure is a failure, not permission to try an unchecked destination.

Parse one candidate, then use its components

parse_target makes one urlsplit call for each candidate URL. It uses the resulting scheme, hostname and port for the allowlist, then builds the HTTP request from those same components. It rejects credentials, every raw ? or # delimiter, backslashes, control characters, unsupported authority forms and malformed percent escapes. Both empty and non-empty queries or fragments are therefore refused. This is intentionally narrower than the set of strings a general URL parser might accept.

Python documents that its parsing functions do not perform comprehensive input validation². Consequently, the code does not treat a returned object as a successful security decision. It separately compares the parsed tuple with ALLOWED. A substring match for example.test would not establish that the parsed destination was that exact hostname; matching a company name somewhere in a URL is an even weaker test.

The existing parser behavior table supplies a concrete disagreement. In its backslash case, the Node API reads the hostname as example.test, Python retains the backslash-bearing suffix in its hostname field, and Ruby rejects the input. Those are the dataset’s declared API versions, not new parser runs for this article. Passing bytes between independently interpreted validators and clients can therefore change the meaning being checked. This example refuses that syntax before either resolution or a connection.

The complete policy example

Save this original file as fetch_policy.py. It has no command-line target option and does nothing merely by being imported. Its reproduction instructions explain how the package’s controlled tests execute it. The reserved names in the file are examples, not addresses to submit to a scanner or open in a browser.

# SPDX-License-Identifier: MIT
"""A deliberately narrow GET policy; no CLI and no automatic target fetch."""
import asyncio
from dataclasses import dataclass
import ipaddress
import re
import socket
import ssl
from urllib.parse import urlsplit

ALLOWED = frozenset({
    ("http", "example.test", 80),
    ("http", "next.example.test", 80),
    ("https", "example.test", 443),
})
REDIRECTS = {301, 302, 303, 307, 308}
open_stream = asyncio.open_connection


class Refused(ValueError):
    pass


@dataclass(frozen=True)
class Target:
    scheme: str
    host: str
    port: int
    request_path: str


def parse_target(text):
    if not isinstance(text, str) or len(text) > 2048:
        raise Refused("URL type or length")
    if any(ord(c) < 33 or ord(c) > 126 for c in text) or "\\" in text:
        raise Refused("URL contains unsupported characters")
    if "?" in text or "#" in text:
        raise Refused("Query and fragment delimiters prohibited")
    if re.search(r"%(?![0-9A-Fa-f]{2})", text):
        raise Refused("Malformed percent escape")
    try:
        parsed = urlsplit(text)
        host = parsed.hostname
        port = parsed.port or {"http": 80, "https": 443}.get(parsed.scheme)
    except ValueError as error:
        raise Refused("Malformed authority") from error
    if parsed.username is not None or parsed.password is not None:
        raise Refused("Credentials prohibited")
    if (parsed.scheme, host, port) not in ALLOWED:
        raise Refused("Scheme, host or port prohibited")
    if parsed.netloc.lower() not in {host, f"{host}:{port}"}:
        raise Refused("Noncanonical authority")
    path = parsed.path or "/"
    if not path.startswith("/") or path.startswith("//"):
        raise Refused("Unsupported request path")
    return Target(parsed.scheme, host, port, path)


def check_address(text):
    if "%" in text:
        raise Refused("Scoped address prohibited")
    try:
        address = ipaddress.ip_address(text)
    except ValueError as error:
        raise Refused("Invalid resolved address") from error
    mapped = getattr(address, "ipv4_mapped", None)
    transition = (getattr(address, "sixtofour", None)
                  or getattr(address, "teredo", None))
    if (not address.is_global or address.is_private or address.is_loopback
            or address.is_link_local or address.is_multicast
            or address.is_reserved or address.is_unspecified
            or getattr(address, "is_site_local", False)
            or mapped is not None or transition is not None):
        raise Refused("Resolved address prohibited")
    return address


async def resolve_once(host, port):
    rows = await asyncio.get_running_loop().getaddrinfo(
        host, port, family=socket.AF_UNSPEC, type=socket.SOCK_STREAM)
    return list(dict.fromkeys(row[4][0] for row in rows))


async def pinned_exchange(target, address, remaining_bytes):
    tls = ssl.create_default_context() if target.scheme == "https" else None
    reader, writer = await open_stream(
        host=str(address), port=target.port,
        family=socket.AF_INET if address.version == 4 else socket.AF_INET6,
        flags=socket.AI_NUMERICHOST, ssl=tls,
        server_hostname=target.host if tls else None, limit=8192)
    try:
        authority = target.host
        if target.port != {"http": 80, "https": 443}[target.scheme]:
            authority += f":{target.port}"
        request = (f"GET {target.request_path} HTTP/1.1\r\n"
                   f"Host: {authority}\r\nConnection: close\r\n"
                   "Accept-Encoding: identity\r\n\r\n")
        writer.write(request.encode("ascii"))
        await writer.drain()
        header = await reader.readuntil(b"\r\n\r\n")
        if len(header) > min(8192, remaining_bytes):
            raise Refused("Response header or total size limit")
        lines = header[:-4].decode("iso-8859-1").split("\r\n")
        status_match = re.fullmatch(r"HTTP/1\.[01] ([0-9]{3})(?: .*)?", lines[0])
        if not status_match:
            raise Refused("Unsupported status line")
        status = int(status_match.group(1))
        fields = {}
        for line in lines[1:]:
            if ":" not in line or line[:1].isspace():
                raise Refused("Malformed or folded header")
            name, value = line.split(":", 1)
            if not re.fullmatch(r"[!#$%&'*+.^_`|~0-9A-Za-z-]+", name):
                raise Refused("Invalid header name")
            name = name.lower()
            if name in fields:
                raise Refused("Duplicate response header")
            fields[name] = value.strip()
        if "transfer-encoding" in fields:
            raise Refused("Transfer coding unsupported")
        if fields.get("content-encoding", "identity") != "identity":
            raise Refused("Content coding unsupported")
        length = fields.get("content-length")
        if length is None or not re.fullmatch(r"[0-9]+", length):
            raise Refused("A single Content-Length is required")
        length = int(length)
        if length + len(header) > remaining_bytes:
            raise Refused("Total response size limit")
        body = await reader.readexactly(length)
        return status, fields.get("location"), body, len(header) + length
    finally:
        writer.close()


async def fetch_allowed(text, *, resolver=resolve_once, max_hops=3,
                        max_bytes=65536, total_seconds=2.0, events=None):
    if max_hops < 0 or max_bytes < 1 or total_seconds <= 0:
        raise ValueError("Limits must be positive; max_hops may be zero")
    events = events if events is not None else []
    remaining = max_bytes
    async with asyncio.timeout(total_seconds):
        for hop in range(max_hops + 1):
            target = parse_target(text)
            answers = await resolver(target.host, target.port)
            if not answers:
                raise Refused("Empty address answer")
            addresses = [check_address(value) for value in dict.fromkeys(answers)]
            selected = addresses[0]
            event = {"hop": hop, "scheme": target.scheme, "host": target.host,
                     "port": target.port, "path": target.request_path,
                     "checked_address": str(selected), "resolved_count": len(addresses)}
            events.append(event)
            status, location, body, used = await pinned_exchange(
                target, selected, remaining)
            remaining -= used
            event.update(status=status, location=location, response_bytes=used)
            if status not in REDIRECTS:
                return body
            if hop == max_hops:
                raise Refused("Redirect hop limit")
            if not location:
                raise Refused("Redirect without Location")
            if target.scheme == "https" and not location.startswith("https://"):
                raise Refused("HTTPS downgrade or relative redirect prohibited")
            text = location
    raise AssertionError("Unreachable")

Check every answer before choosing one

resolve_once makes one asynchronous resolver call per hop. fetch_allowed checks every distinct returned address before choosing the first one. An answer containing both an allowed fixture address and a prohibited address is refused before any connection. Empty answers are refused too. Checking only the first response while a client later chooses another would leave a gap between validation and use.

Eleven separate syntax-policy cases stopped before the resolver was called: an unlisted scheme, hostname or port, credentials, a backslash, a control character, a malformed escape, and empty and non-empty query and fragment delimiters. Their recorded request counts are all zero. That distinction helps explain a refusal without suggesting that the application inspected a destination: these inputs never reached the address or connection stages. Keep those stages separate in error reporting, and avoid recording sensitive paths or query values merely to explain an allowlist failure. The synthetic examples contain no real tokens; a production log needs its own redaction and retention policy.

The strict address gate rejects non-global, private, loopback, link-local, multicast, reserved, unspecified and deprecated IPv6 site-local addresses. It also explicitly refuses IPv4-mapped IPv6 forms and the transition forms it recognizes. Python’s ipaddress API defines these classifications, including distinctions that make checking only is_private inadequate³. The runtime version matters because address classification is maintained software, not a timeless hand-written list. For reuse, install the latest CPython 3.12 patch release and rerun the examples; the recorded 3.12.14 pin supports comparison with the earlier measurements, not a recommendation to remain on it⁴.

The local suite exercised 18 refusal categories, including shared-address space, documentation ranges, deprecated site-local IPv6 and the well-known NAT64 prefix. It did not contact those addresses. Each result records the category and refusal reason; the published runner contains fixed category representatives, not addresses discovered from a network. This finite suite demonstrates the chosen cases, not exhaustive coverage of every special-use allocation or every network’s routing configuration.

NAT64 is a specific limit. In CPython 3.12.14, 64:ff9b::/96 falls inside the reserved ::/8 classification, so this policy refuses it through is_reserved, not by decoding its embedded IPv4 address⁵. On a DNS64/NAT64 network using that well-known prefix, this also refuses every synthesized IPv4-only destination, including public ones. Network-specific NAT64 prefixes are also permitted by the standard⁶; this code does not identify those prefixes or inspect their translated IPv4 destinations. An operator-assigned prefix can therefore wrap an internal IPv4 address without this classifier recognizing it. Network-specific egress controls and policy are still required; no NAT64 network was exercised here.

Pin the connection, not just a variable name

After selection, the connector receives the numeric address and the parsed target separately. The connection arguments use the selected address with AI_NUMERICHOST⁷; the HTTP Host field continues to identify the approved hostname. For the HTTPS branch, the hostname also goes to certificate verification and SNI through a default TLS context. Those separate roles are visible in the code: the connection destination must not silently become a fresh hostname lookup.

Python’s event-loop interfaces document the address-resolution and connection parameters used here⁸. A resolver check followed by a convenient client call on the original URL would not preserve this design. The second call could resolve again, follow redirects automatically or use a proxy with different destination rules. This implementation supplies neither a proxy nor an automatic redirect client.

The successful local requests need a precise testing qualification. The strict gate correctly refuses the documentation addresses used in the examples. The published local runner therefore grants exactly two documentation addresses after testing the unmodified gate, then maps the connector’s selected numeric address to a temporary local server. The adapter asserts that it received that exact address and the numeric-only flag. It does not establish that a real public network connection, DNS service or TLS handshake worked.

Reapply the policy at the next hop

A redirect is another candidate destination. The function receives a response without automatic following, then repeats parsing, allowlisting, resolution and address checks before the next request. It recognizes only the listed HTTP redirect statuses. An HTTP response’s Location is a destination reference, not evidence that the application has permission to follow it⁹.

This implementation requires absolute redirect locations. Relative locations fail its next parse; HTTPS-to-HTTP changes are refused explicitly. Supporting relative references is possible, but needs a deliberate base-resolution rule and corresponding tests. It is not added here by letting another client reinterpret the string. Cookies and authentication are not accumulated or carried into the next request.

Two resolver experiments separate the important cases. In the single-request case, the script offered a documentation answer followed by a loopback answer if asked again. There was one resolver call and one local connection, so the second answer was never consulted. In the same-host redirect case, the policy intentionally resolved anew for the new hop, received the prohibited second answer and stopped before making a second connection.

Bound the whole chain

The normal test limit is three redirects, meaning at most four requested responses. A single 2-second deadline surrounds resolution, connection, headers and bodies across the chain. The byte budget is 65,536 response bytes in total, counting accepted headers and bodies from each hop. These are example budgets; an application should choose limits from its own workload rather than copy them as universal thresholds.

The suite also configured a two-redirect limit for a loop, a 128-byte budget for an oversized response and a 0.05-second deadline for a delayed local response. Each stopped with the expected refusal or timeout. A failed limit does not produce a partial “safe” result. The run record retains the configured limits, resolver calls, connection selections and per-hop outcomes.

The small HTTP reader deliberately accepts only a restricted response shape: a single Content-Length, no transfer coding, no compressed content, and bounded headers. Duplicate headers and the tested transfer-coding response are refused. It is an instructional transport, not a general web client. Broadening protocol support requires preserving the same destination and resource-budget rules; quietly delegating to another redirecting client would change what was tested.

Treat the result as one part of an application boundary

The process-level test guard measured 14 local connections, zero system-DNS attempts and zero non-fixture connection attempts. The runner copies each counter and asserts the latter two are zero; it does not replace measured values. Earlier records are retained for history, but their overwritten zero fields are not measurements. That observation covers these test processes and their supplied adapters; it is not a machine-wide traffic capture. The production resolver and HTTPS branch were not exercised against an external service. Their presence in the source must not be mistaken for a completed deployment review.

Before integrating a fetching service, its security review still needs the real egress controls, resolver configuration, proxy behavior, TLS policy, concurrent resource limits and application-specific allowed operations. A globally routed address can still identify infrastructure the application should never reach. This example neither discovers those organization-specific destinations nor classifies returned content as benign.

For readers who only need to understand an address, the local URL inspector stops before requesting it. The lookup-versus-submission guide explains the separate disclosure decision involved in sending a URL to a provider. Parsing locally, authorizing a fetch and trusting a page answer different questions; keep their results separate in the interface and in the records.

Not covered on this page

• Public DNS, HTTPS execution or a completed security review for deployment.
• Relative redirects, compressed bodies, transfer coding or browser navigation.
• Organization-specific routing, proxy, NAT64 or application authorization policies.
• A verdict about destination ownership, honesty or returned content.