Build Your Own curl

We are going to build curl from scratch by accepting the coding challenge posted on Coding Challenges FYI.

Before moving ahead, you must need to know how tcp client server connections works.

You can read more about GeeksforGeeks.

On Server Side : -

Socket:- Socket object to expose our endpoints.
setsockopt:- This functions set the extra options for the sockets if needed.
Bind:- It binds the socket with the ip and port.
Listen:- Socket is now in listening state, and listen at the specified port for the incoming connection request. Here it queue incoming client request to connect. The second argument specifies the maximum number of request it can queue.
Accept:- In this phase, server calls accept function, it initiates the 3-way handshaking. The client sends SYN packet to the server, server responds back with the SYN-ACK packet and blocks(wait) the connection, until it finally get the ACK packet.
Send/Recv:- Once ACK received from client, communication can proceed to and fro.

On Client Side : -

Socket initialization:- In this setup, socket is defined with all the configuration needed to connect.
Connect:- In this phase, client call the connect function, which sends the SYN packet to the server with the intent to connect.
Send/Recv: Once connection is established, when client can send and receive the data.

We are doing to acheive in few steps: -

Getting the cli arguments
Creation of socket connection
Sending the request
Parsing the response

1. Getting the cli arguments.

We will be using library for Clap - A simple-to-use, efficient, and full-featured library for parsing command line arguments and subcommands.

Tha clap library provides two different ways to build parse object. First is the Builder pattern(creational design pattern to create complex things by step by step process) and second Derive pattern in which library automatically generate code based on the macros.

We are using Builder pattern for our cli tool.

But you can implement Derive pattern at doc.rs/clap/_derive

'cli.rs'

use clap::{Arg, ArgMatches, Command};

pub fn get_arguments()-> ArgMatches{
    Command::new("Ccurl - custom curl")
        .about("It helps to make http methods")
        .version("1.0")
        .author("Praveen Chaudhary <[email protected]>")
        .arg(Arg::new("url").index(1).required(true))
        .arg(
            Arg::new("x-method")
                .help("Http method which you want to use")
                .long("x-method")
                .short('X'),
        )
        .arg(
            Arg::new("data")
                .help("Payload you want to send with the request")
                .long("data")
                .short('d'),
        )
        .arg(
            Arg::new("headers")
                .help("Request header")
                .long("header")
                .short('H')
                .action(ArgAction::Append),
        )
        .arg(
            Arg::new("verbose")
                .help("verbose mode")
                .long("verbose")
                .short('v')
                .action(clap::ArgAction::SetTrue),
        )
        .get_matches()
}

Firstly, we have define the basic info like about, author and version.

We have defined all the arguments need for our own curl. We have made one positional required argument url.

Argument matching or parsing

Clap makes it easier to match arguments.
For verbose, we have used action method .action(clap::ArgAction::SetTrue) because it will not contain any subsequent value. For headers, similarly we have used action method .action(ArgAction::Append), Append will append new values to the previous value if any value have already encountered. For others, we have simply used get_one method to get the value.

let verbose_enabled = matches.contains_id("verbose") && matches.get_flag("verbose");
    let url = matches.get_one::<String>("url").unwrap();
    let data = matches.get_one::<String>("data");
    let method = matches.get_one::<String>("x-method");
    let headers: Vec<&str> = matches
        .get_many::<String>("headers")
        .unwrap_or_default()
        .map(|s| s.as_str())
        .collect();

2. Drafting request with input information [ HTTP 1.1 - RFC9110.]

We will be using the RFC9110 for HTTP 1.1 client.

we will start with empty string, and append all the information needed for the Request according to RFC.

fn populate_get_request(
    protocol: &str,
    host: &str,
    path: &str,
    data: Option<&String>,
    method: Option<&String>,
    headers: Vec<&str>,
) -> String {
    let default_method = String::from("GET");
    let method = method.unwrap_or(&default_method);
    let mut res = String::new();
    res += &format!("{} /{} {}\r\n", method, path, protocol);
    res += &format!("Host: {}\r\n", host);
    res += "Accept: */*\r\n";
    res += "Connection: close\r\n";
    ....
    ....
    ....
    res += "\r\n";
    res
}

For PUT and POST, we need to add headers and data.

fn populate_get_request(
    protocol: &str,
    host: &str,
    path: &str,
    data: Option<&String>,
    method: Option<&String>,
    headers: Vec<&str>,
) -> String {
    ....
    ....

    if method == "POST" || method == "PUT" {
        if headers.len() > 0 {
            for head in headers {
                res += head;
            }
            res += "\r\n"
        } else {
            res += "Content-Type: application/json\r\n";
        }
        if let Some(data_str) = data {
            let data_bytes = data_str.as_bytes();
            res += &format!("Content-Length: {}\r\n\r\n", data_bytes.len());
            res += data_str;
            res += "\r\n";
        }
    }

    ....
    res
}

According to RFC, for post or post we need to provide Content-Length and Content-Type header.

So now we have complete request string. Let's move to socket connection, sending this request string to the server.

fn populate_get_request(
    protocol: &str,
    host: &str,
    path: &str,
    data: Option<&String>,
    method: Option<&String>,
    headers: Vec<&str>,
) -> String {
    let default_method = String::from("GET");
    let method = method.unwrap_or(&default_method);
    let mut res = String::new();
    res += &format!("{} /{} {}\r\n", method, path, protocol);
    res += &format!("Host: {}\r\n", host);
    res += "Accept: */*\r\n";
    res += "Connection: close\r\n";

    if method == "POST" || method == "PUT" {
        if headers.len() > 0 {
            for head in headers {
                res += head;
            }
            res += "\r\n"
        } else {
            res += "Content-Type: application/json\r\n";
        }
        if let Some(data_str) = data {
            let data_bytes = data_str.as_bytes();
            res += &format!("Content-Length: {}\r\n\r\n", data_bytes.len());
            res += data_str;
            res += "\r\n";
        }
    }

    res += "\r\n";
    res
}

3. Creation of socket connection

We will using the standard rust network library for socket connection with the host server.

fn main() {
    ....
    ....

    let tcp_socket = TcpStream::connect(socket_addr);

    match tcp_socket {
        Ok(mut stream) => {
            ....
            ....
        }
        Err(e) => {
            eprintln!("Failed to establish connection: {}", e);
        }
    }
    ....
    ....
}

Once we are successfully connected, we can listen and send your own request to server.

4. Sending the request

First we have check if verbose mode is enabled, then we print out the request.
We have used the write_all to ensure our that our whole buffer is added to the stream.
Create a new empty buffer, and provide this buffer to the stream, to read the response data from the host.
Converts that bytes string to UTF-8 string using the from_utf8_lossy.
Print the response header and body.

fn main() {
    ....
    ....

    match tcp_socket {
        Ok(mut stream) => {
            if verbose_enabled {
                let lines = buffer_str.lines();
                for line in lines {
                    println!("> {}", line)
                }
            }
            stream
                .write_all(buffer_str.as_bytes())
                .expect("Failed to write data to stream");

            // initialising the buffer, reads data from the stream and stores it in the buffer.
            let mut buffer = [0; 1024];
            stream
                .read(&mut buffer)
                .expect("Failed to read from response from host!");

            // converts buffer data into a UTF-8 enccoded string (lossy ensures invalid data can be truncated).
            let response = String::from_utf8_lossy(&buffer[..]);

            // dividing the response headers and body
            let (response_header, response_data) = parse_resp(&response);
            if verbose_enabled {
                let lines = response_header.split("\r\n");
                for line in lines {
                    println!("< {}", line)
                }
            }
            println!("{}", response_data);
        }
        Err(e) => {
            eprintln!("Failed to establish connection: {}", e);
        }
    }
    ....
    ....
}

5. Time for Testing

cli - cargo run -- http://eu.httpbin.org:80/get response -

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "eu.httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-65fec214-25771a3e732101c433ce67a7"
  }, 
  "origin": "49.36.177.79", 
  "url": "http://eu.httpbin.org/get"
}

Similarly, you can test others.

Hurray!! We have able to make our own curl.

Rust implementation of custom numeric base conversion.

base_custom Use any characters as your own numeric base and convert to and from decimal. This can be taken advantage of in various ways: Mathematics:

5 Dec 28, 2021

Reference implementation of a decentralized exchange for custom instruments, risk, and fees

Dexterity What is Dexterity At a high level, Dexterity is a smart contract (or collection of smart contracts) that allow for the creation of a decentr

43 Dec 15, 2022

Use Thunk to build your Rust program that runs on old Windows platforms, support Windows XP and more!

Use Thunk to build your Rust program that runs on old platforms. Thunk uses VC-LTL5 and YY-Thunks to build programs that support old platforms. So, ho

6 May 21, 2023

A very simple third-party cargo subcommand to execute a custom command

cargo-x A very simple third-party cargo subcommand to execute a custom command Usage install cargo-x cargo install cargo-x or upgrade cargo install -

9 Dec 26, 2022

git-cliff can generate changelog files from the Git history by utilizing conventional commits as well as regex-powered custom parsers.⛰️

git-cliff can generate changelog files from the Git history by utilizing conventional commits as well as regex-powered custom parsers. The changelog template can be customized with a configuration file to match the desired format.