Pipe three commands together, streaming
This program is the shell pipeline curl $url | sha256sum | cut -d ' ' -f 1: it downloads a web
page and prints its SHA-256 digest. It is an example from Hypershell, a
shell-scripting language whose programs are Rust types, built with
Context-Generic Programming (CGP), a Rust library that resolves each stage's
implementation when the program compiles. The page shows how streaming stages run concurrently,
and why each stage's input type decides what may come before it.
New to CGP?
Read hello and hello_name first, which introduce programs as
types and contexts with fields. For CGP itself, the Hello World tutorial is
the quickest start, and Handlers explains the family of computations each
stage belongs to.
The problem
The task is to download a web page and print its SHA-256 digest by piping three commands together,
curl, sha256sum, and cut, with all three running at once so the page streams through them
rather than being held in memory.
Without CGP
In a shell, this is one line, and for a pipeline run by hand the shell is the right tool. In Rust,
the same pipeline with std::process::Command or Tokio's process API means spawning each command,
connecting each one's output to the next one's input by hand, and waiting on all of them and
checking each exit status, so the plumbing outweighs the pipeline. This page declares the pipeline
as a type and leaves the plumbing to the providers that interpret it, which also check that the
stages fit.
Run it
From the root of the Hypershell repository:
cargo run --example http_checksum_cli
It needs the network and the curl, sha256sum, and cut commands. It prints the SHA-256 digest
of the Nixpkgs manual page as a line of hexadecimal, the same digest the next two examples print by
other means.
Three streaming stages
The program is three StreamingExec stages and a final StreamToStdout, from
http_checksum_cli.rs:
pub type Program = hypershell! {
StreamingExec<
StaticArg<"curl">,
WithArgs [
FieldArg<"url">,
],
>
| StreamingExec<
StaticArg<"sha256sum">,
WithStaticArgs [],
>
| StreamingExec<
StaticArg<"cut">,
WithStaticArgs [
"-d",
" ",
"-f",
"1",
],
>
| StreamToStdout
};
#[derive(HasField)]
pub struct MyApp {
pub url: String,
}
delegate_components! {
MyApp {
namespace HypershellNamespace;
}
}
curl reads its URL from the url field of the context, the type the program runs on, as
hello_name read its name. WithStaticArgs [] is an empty argument list. main
builds MyApp with the manual's URL and calls handle with an empty Vec<u8> as the input.
Each stage streams
StreamingExec differs from the SimpleExec the earlier examples used in when it returns.
SimpleExec waits for its command to finish and produces the whole output as bytes. StreamingExec
spawns its command and returns the command's standard output as a stream at once, while a background
task copies the stage's input into the command's standard input. So the three commands run at the
same time, each reading the previous one's output as it arrives, as they would in a shell.
That changes when a failure is reported. A streaming stage has already returned by the time its command fails, so a non-zero exit is reported when its output stream ends, and the stage reading that stream fails the program. The command's standard error becomes part of the message.
The stages must line up
Every stage's output is the next stage's input, and the compiler checks that each pair agrees. A
StreamingExec stage produces a Tokio reader, wrapped in Hypershell's TokioAsyncReadStream type,
and the next StreamingExec accepts it. The first stage accepts the Vec<u8> that main passes,
and StreamToStdout at the end accepts the stream.
A stage accepts several input types because its wiring puts a small input dispatcher in front of
the provider that does the work. Under HypershellNamespace, StreamingExec is interpreted by
three providers run in sequence:
@HandlerComponent.<Path, Args> StreamingExec<Path, Args>:
PipeHandlers<Product![
HandleToTokioAsyncRead,
HandleStreamingExec,
WrapTokioAsyncRead,
]>,
The first converts whatever input arrives, bytes or either kind of stream, into a Tokio reader. The
second runs the command. The third wraps its output in TokioAsyncReadStream, so that the next
stage can tell what kind of stream it received.
PipeHandlers is CGP's combinator for running
providers in sequence, each one's output feeding the next.
The simpler stages have no dispatcher. SimpleExec accepts only bytes, so a StreamingExec stage
cannot feed it directly: the compiler rejects the pair until a StreamToBytes stage collects the
stream in between. The streams and input dispatch
page has the table of what every stage accepts and produces.
The pattern
This program shows a typed pipeline: stages connected in order, with the compiler checking that
each stage can accept what the one before it produces. The check happens when the program is
compiled, so a pipeline whose stages do not fit never runs. The next example,
http_checksum_client, shows how a stage accepts more than one kind of
input.
The cost is the strictness that types bring. A shell passes bytes between any two commands, while a Hypershell pipeline rejects a pair of stages that do not fit, and the program has to add an adapter stage between them.
Where to go next
http_checksum_client: the next example, withcurlreplaced by a native HTTP request.- Streams and input dispatch: the stream types and the dispatchers.
- Handlers: CGP's family of computation components, including the combinators that compose them.
- Hypershell: a type-level DSL for shell-scripting in Rust: the post that announced Hypershell, which introduced this pipeline. Its code predates the current design.
An AI agent wrote this page using the CGP knowledge base. Its content was verified against the project's source. See How AI is used in this project.