Skip to main content

How Hypershell works

A Hypershell program looks like a shell pipeline, but it is a Rust type, and the compiler turns it into ordinary function calls. Hypershell is a small shell-scripting language built with Context-Generic Programming (CGP), a Rust library whose wiring decides which implementation a program runs. This page explains, in outline, how a type can be a program, what decides what that program does, and how the language is put together. Each idea has a page of its own with the details.

The program only says what to do​

In a shell script, a pipeline is text. The shell reads it, works out which programs to start, and connects them. A Hypershell pipeline is written in Rust instead, as a type:

pub type Program = hypershell! {
SimpleExec<StaticArg<"echo">, WithStaticArgs["hello", "world!"]>
| StreamToStdout
};

Each name in it, such as SimpleExec or StreamToStdout, is an empty struct. It records what the program asks for, "run echo" and "print the result", and contains no code for doing either. The hypershell! macro only turns the shell-like notation into that type. The abstract syntax page describes every kind of piece a program can contain.

The context decides how it is done​

Because the program contains no code, something else must supply it. That is the context: the type the program is run on, as in HypershellCli.handle(PhantomData::<Program>, input). For each piece of the program, the context's wiring picks an implementation, called a provider. One provider knows how to run a command, another how to print a stream, another how to send an HTTP request.

The compiler makes these choices while it type-checks the program. By the time the program runs, each piece has become a direct call into its provider, with no interpreter and no lookup at run time. This is what CGP is for: letting a type choose among implementations of the same interface, and resolving the choice at compile time.

The context makes one more choice: the error type. Providers report failures without naming an error type of their own, and the context decides what those failures become. Hypershell's standard wiring turns them all into anyhow::Error, using CGP's modular error handling.

Two consequences follow, and they are why the design is interesting. The same program can behave differently on two contexts that choose different providers. And a new piece of syntax, with a provider for it, can be added from any crate, without touching the crates that define the language.

The language is assembled from reusable parts​

A context does not list a provider for every piece of the language itself. Hypershell supplies its choices in layers, so that a user's context needs one line:

pub struct HypershellCli;

delegate_components! {
HypershellCli {
namespace HypershellNamespace;
}
}

Each backend, such as the one that runs processes with Tokio or the one that sends HTTP requests, publishes its providers as a bundle. HypershellNamespace gathers the bundles into the whole language, and a context that joins it gets everything in it. The assembly page explains the layers, and how a context can add to them.

Stages are typed, and fit together like functions​

Each stage of a pipeline receives the previous stage's output, as in a shell, but here the output has a Rust type. One stage produces bytes, another a stream, another a decoded JSON value. The compiler checks that every stage can accept what the one before it produces, and it chooses a stage's provider by that type as well as by the syntax. Most stages accept both bytes and streams, so they chain freely. The streams and input dispatch page explains how, and where they do not.

Each backend lives in its own crate​

The syntax lives in a crate that depends only on CGP, and each backend (Tokio, reqwest, serde_json, and the rest) lives in a crate of its own. A program's crate compiles only the backends it uses, and no provider names the context or the error type it will run with; those are chosen last. The crate layout page lists the crates and what the workspace needs to build.

What the design costs​

The flexibility is paid for in indirection. To find out what a piece of syntax does, a reader follows it from the program to the context, to the namespace, to a bundle, and finally to a provider. The compiler does that in an instant, but a person reading the code, or reading a long compile error, has to do it too. The debugging guide shows how to read such an error.

It also moves work to compile time. A program is fixed when it is compiled, so it cannot be loaded or changed while an application runs, and large programs make the compiler work hard: the biggest examples need nightly Rust's newer trait solver to type-check in reasonable memory.

For a pipeline written once and run by hand, a shell script is simpler, and for a single command a few lines of std::process::Command are. The design earns its cost when a pipeline's parts should be reusable, checked by the compiler, and replaceable without editing the program.

Where to go next​

  • Abstract syntax: what a program is made of.
  • hello: the design at its smallest, in a running program.
  • Type-level DSLs: the CGP technique behind Hypershell, explained with a smaller language.

An AI agent wrote this page using the CGP knowledge base. Its content was verified against the project's source. See How AI is used in this project.