Rama 101.1: HTTPS clients and layers of abstraction

In each article I will try to explain one or more concepts by means of a kind of example that I hope most of you are familiar with. For this first part we will use HTTPS clients as an example. But before that, and given that this is the start of the series, let us quickly answer another question first.

What is Rama

Rama is a modular service framework for the Rust language. That's how the Rama project README introduces it, and many of our other resources do so in a similar vein, be it perhaps more focused on proxies.

But if you boil it all down to its essentials, then it is this: Rama is meant to be the modular foundation that we can all share and build network services with. Meaning some kind of process that interacts with and within a computer network as a client, server, or a combination of both.

We call Rama a framework instead of a library to try to make it clear that it really provides you with an entire cohesive foundation. Rama consists of many crates (and modules), each focused on another feature or protocol. It is modular, as the framework is designed in such a way that it allows you to compose it all together in whatever way suits your business use case. You will see plenty of examples of that in this series.

Yet, because all these modules and crates are in fact part of a bigger framework (Rama), you do not need to worry about whether something is compatible (e.g. version-, API-, or protocol-wise). This important property is guaranteed not only by its design, but also by its many layers and different kinds of tests, right from the smallest unit test up to verifications of entire network services, their behavior and correctness. Not just today, but through its entire evolution, starting 5 years ago and hopefully continuing for many decades still to come.

And whenever any of us makes any kind of improvement, we all benefit from it. This is important as many pitfalls in network programming are not due to mistakes in any specific unit but are instead only made by means of composition. Those are the kind of hard lessons that the developers of projects made from duct-taped libraries have to learn the hard way, over and over again. But not you, as a Rama user.

An HTTPS Request

With the preface behind us, let us begin with something that I am hoping pretty much all of you are familiar with. Let's make an HTTPS request. The fact that you are reading this most likely means you used software that made an HTTPS request, be it via a web browser or via whatever application you use to read it through our RSS Feed.

1let my_ip: Ipv4Addr = client2    .get("http://ipv4.ramaproxy.org")3    .typed_header(UserAgent::rama())4    .send_with_timeout(Duration::from_secs(30))5    .await6    .expect("send http(s) request")7    .try_into_string_with(8        CollectOptions::new()9            .with_timeout(Duration::from_secs(30))10            .with_max_size(3 + 3 * 4),11    )12    .await13    .expect("collect response payload")14    .parse()15    .expect("parse as IPv4 Address");1617println!("my ip: {my_ip}");

In this example we make use of a constructed HTTPS client to "make" an HTTP request (that will in fact be redirected to an HTTPS request), wait (with limits) for it to be sent and for a response to come back, collect the body as a String, and parse it into an Ipv4Addr. That's a lot, yet at the same time hopefully familiar to most of you.

If not in code it might perhaps look more familiar to you as a CLI command:

1rama -L http://ipv4.ramaproxy.org

Let's bring our focus back on the code, because usually when we refer to Rama, we mean the (Rust) framework instead. I will now zoom in on the type of client used in the example above. The easiest way to create an HTTPS client is by using the EasyHttpWebClient:

1let client = EasyHttpWebClient::default();

It is a prime example of the highest kind of abstraction provided in Rama and excellent for things that many people need for all kinds of reasons. Even within the most advanced proxy project you might need to interact with a remote server to sync information in one direction or the other.

However, even a convenient high-level abstraction like EasyHttpWebClient does not yet provide everything you would want within a production environment setting. This is because we do not want to make choices for you based on assumptions that we cannot guarantee to be true. Decompression, retrying failed HTTP requests, setting HTTP headers, or following redirects are all examples of things this client doesn't do out of the box. For that we need layers.

1let client = EasyHttpWebClient::default();23(4    MapResponseBodyLayer::new_boxed_streaming_body(),5    DecompressionLayer::new(),6    FollowRedirectLayer::default(),7)8    .into_layer(client)

Each HTTP request will go through this entire stack ensuring among other things that the Accept-Encoding header is injected, such that the server has the info it needs to be able to compress the content given the provided algorithms, and that we follow redirects (e.g. upgrading to HTTPS or a different resource altogether).

Everything is a Service

The stack of Services does not stop here, however. Let's explore this further by building one layer by layer instead of creating the default EasyHttpWebClient:

1EasyHttpWebClient::connector_builder()2    // happy-eye balls IPv4/IPv6 TCP connection3    .with_default_transport_connector()4    // resolve A/AAAA records for (domain) name (e.g. `ipv4.ramaproxy.org`)5    // to IPv4 / IPv6 address(es)6    .with_default_dns_connector()7    // no (proxy) TLS tunnel desired (e.g. for HTTP proxis "over" TLS)8    .without_tls_proxy_support()9    // no HTTP(S) / SOCKS5(H) proxy support10    .without_proxy_support()11    // TLS support (the `S` in `HTTPS`) using BoringSSL12    // (Rama also has Rustls support, or bring your own TLS stack )13    .with_tls_support_using_boringssl_and_default_http_version(14        TlsClientConfig::default_http(),15        // (falls back to HTTP/1.1 in case no version was negotiated during ALPN)16        Version::HTTP_11,17    )18    // HTTP1 / H2 connector19    .with_default_http_connector(Executor::default())20    // Ability to reuse existing HTTP(S) connections21    .try_with_default_connection_pool()22    .expect("create default pool")23    // Wrap the connector within a `Service`,24    // acting like a client, establishing a connection25    // for each request.26    //27    // In case you use a "pool" connector you would28    // reuse existing HTTP(S) connections in case one is available.29    .build_client()

This shows you that connectors are also in fact Services, which, for a kind of input such as an HTTP Request, establish a connection and return that together with the linked input.

For a typical HTTPS handshake this results in this connector stack being responsible for:

• If an HTTPS connection is available for reuse, we do so and skip the next steps.
• Otherwise, if the target is the domain name of the destination Host, we resolve it into a stream of IP addresses (DNS).
• After this, we try to establish a transport connection using the Happy Eyeballs algorithm (TCP).
• We complete the TLS handshake with the origin server over the previously established TCP connection.
• We establish the HTTP handshake over the now established TLS connection (if h2; for HTTP/1.1 there is no handshake at all).
• We return an HTTP Service which wraps around the now established HTTP-over-TLS (HTTPS) connection.

That's a lot, but at the same time, still just layers of Services. It can take a while to get your head around the concept of services and learning to reason over multiple layers of abstraction. Even for Brecht, now also a maintainer of Rama, it took a while to fully understand these concepts and what they imply. Everything is a service.

Connectors

Let us now leave EasyHttpWebClient behind us, and go an abstraction layer lower to create a similar connector service stack ourselves:

1// 1. Create connector (once)2let https_connector = (3    ArcLayer::default(),4    HttpConnectorLayer::new(Executor::default()),5    TlsConnectorLayer::auto(),6    DnsConnectorLayer::new(),7)8    .into_layer(TcpConnector::default());910service_fn(move |req| {11    let https_connector = https_connector.clone();1213    async move {14        // 2. Establish a HTTP(S) connection15        let EstablishedClientConnection {16            input: req,17            conn: client,18        } = https_connector19            .connect(req)20            .await21            .context("establish HTTP(S) connection")?;2223        // 3. Round-trip the actual HTTP(S) request24        client25            .serve(req)26            .await27            .context("round-trip http(s) request")28    }29})

In this example the connector service stack is created once, and shared through an Arc cloned for each incoming HTTP Request. Using it we try to establish an HTTPS connection as described earlier. If all is well, we can at that point use an established or reused HTTPS client to do the actual roundtrip of sending the HTTP Request (finally) and awaiting either a response or an error.

You can also observe here that we package this entire concept in a Service of itself (using service_fn to convert a function or closure into a Service). This is the final contract, and just like the EasyHttpWebClient it allows us to conceptualise (abstract) it as an "HTTPS client" which for a given HTTP request gives us back a response or error.

You, however, know better now. One does not simply make an HTTP Request.

Extension Traits

Within Rust there is the concept of Extension Traits and it plays a crucial role here. It is what allows any Service within Rama with the right signature to be used as if it were a high-level HTTP(S) client.

1impl<S, Body> HttpClientExt for S2where3    S: Service<Request, Output = Response<Body>, Error: Into<BoxError>>,4    type ExecuteResponse = Response<Body>;5    type ExecuteError = S::Error;

As you can read here, we blanket implement this trait for any Service which takes as input an HTTP Request, and returns an HTTP response or error.

And it is because of this trait, and only once you import it in your own module, that you can start using the high-level HTTPS client API that you might already be familiar with, as shown earlier:

1let my_ip: Ipv4Addr = client2    .get("http://ipv4.ramaproxy.org")3    .typed_header(UserAgent::rama())4    .send_with_timeout(Duration::from_secs(30))5    .await6    .expect("send http(s) request")7    .try_into_string_with(8        CollectOptions::new()9            .with_timeout(Duration::from_secs(30))10            .with_max_size(3 + 3 * 4),11    )12    .await13    .expect("collect response payload")14    .parse()15    .expect("parse as IPv4 Address");1617println!("my ip: {my_ip}");

Remember though... everything is a service.

1service_fn(async || Ok::<_, Infallible>("0.0.0.0".into_response()))

For those coming from libraries such as Tower this will be the most familiar kind of Service. A server. In this case one that always returns the same text/plain HTTP response with 0.0.0.0 as the payload, regardless of the incoming HTTP request. Note, however, that the input of a Service does not have to be an HTTP request. For any other protocol this will be the case, but even for HTTP servers the story already gets blurrier. In the second part of this series we will take a look at an HTTPS server where we will see our first examples of the fact that Rama services take whatever input you want them to take.

But to conclude our story here, it does mean that also this Service can be used as if it were... an HTTPS client, as it also respects the required trait bound. And this comes in very handy to easily test web services developed with Rama without having to make your own test abstraction or intermediate client.

Conclusion

This concludes the first part in our Rama 101 series. In case you want to see a full example of an HTTPS client and be able to easily play and experiment with it yourself, please have a look at the http_high_level_client.rs example. This example also demonstrates how to use any HTTP client layer stack in a high-level manner using the HttpClientExt.