Skip to main content

Download a web page to a file

This program downloads the Nixpkgs manual and writes it to a file as it arrives, without holding the whole page in memory. It is an example from Hypershell, a shell-scripting language whose programs are Rust types, built with Context-Generic Programming (CGP), a Rust library that derives a program's behavior from its wiring. The page shows a pipeline whose last stage produces nothing, and a file path read from the context, the type the program runs on.

tip

New to CGP?​

This page builds on http_checksum_client, which introduces the HTTP stage and its input dispatch. For CGP itself, the Hello World tutorial is the quickest start, and Handlers explains the computations whose output types a pipeline chains.

The problem​

The task is to download a web page into a file, writing it as it arrives rather than holding the whole page in memory, with the URL and the file path both chosen when the program runs.

Without CGP​

curl -o manual.html <url> does this in a shell, and it is the simpler tool for a download run by hand. In Rust, streaming a response to disk means a loop that reads the body chunk by chunk and writes each chunk to a file, with the URL and the path passed in as arguments. This page expresses the download as a two-stage pipeline whose last stage writes to a file and produces nothing, so a download is one more program in the language rather than a loop written for the occasion.

Run it​

From the root of the Hypershell repository:

cargo run --example save_webpage

It needs the network. It writes the manual page to nix_manual.html in the directory the command is run from, and then prints:

Webpage saved to nix_manual.html

Run from the repository root, it writes the file into the checkout.

The program​

From save_webpage.rs:

pub type Program = hypershell! {
StreamingHttpRequest<
GetMethod,
FieldArg<"url">,
WithHeaders[ ],
>
| WriteFile<FieldArg<"file_path">>
};

#[derive(HasField)]
pub struct MyApp {
pub http_client: Client,
pub url: String,
pub file_path: String,
}

The context joins HypershellNamespace, and main sets file_path to the relative path nix_manual.html.

A stage that ends the pipeline​

WriteFile takes the path to write as its parameter, and the stage's input as the content. Its path is a FieldArg, so the program reads it from the context like any other runtime value. The path position asks for a file path rather than a string, so the same argument syntax that produced a URL in nix_manual produces a path here.

The request's output is a futures reader, and WriteFile accepts it because its wiring starts with the same input dispatcher as StreamingExec, described in http_checksum_client. The file is written as the response streams in.

WriteFile produces (), so the program's output is () and handle returns Result<(), Error>. The compiler computes that from the last stage; main only prints its message once the call returns.

The pattern​

This program shows a pipeline whose output type follows from its last stage. A caller of handle chooses only the input; what comes back is decided by the stages the program names, through their providers. A program ending in WriteFile returns nothing, and one ending in a decoding stage returns the decoded value, as github_issues shows.

The cost is that the signature is implicit. Nothing in Program names its output type; a reader works it out from the last stage's provider, and a caller who expects the wrong type learns so from a compile error at the call rather than from a declaration.

Where to go next​


An AI agent wrote this page using the CGP knowledge base. Its content was verified against the project's source. See How AI is used in this project.