About a year ago I wrote a user_agent crate (not published to crates.io) for an internal project, which provides the concept of a Session wrapping a Client and does cookie handling per RFC6265. At the time, the only client I targeted was hyper::Client. I'd be interested in updating/adapting the code to reqwest, but wanted to touch base on any existing plans/work here.

First off, the library suffers some warts:

• Perhaps not the cleanest rust, one of the first libraries I developed when first learning rust.
• It implements its own Cookie class, which wraps/extends the existing cookie crate (providing a From implementation to handle conversion). This was to introduce some more domain objects (CookiePath, CookieDomain, etc.) to enforce/aid the implementation of some RFC6265 guidelines.
• At the time, hyper was on a different (0.1?) version of cookie-rs, whereas I was developing against a more recent version (0.2+?), meaning the library goes through some gyrations to "convert" between the two versions by stringifying-then-parsing cookies.
• While I wrote a fair set of tests, usage and testing was primarly geared towards my particular use case at the time, so implementation re: RFC6265 may not be complete or necessarily fully correct.

Some of these issues can be cleaned up over time, but in terms of bigger picture I have some questions/concerns:

• The re-implementation/extension of Cookie seems un-necessary; submitting PRs against cookie-rs to bring them in line would make sense...
• cookie-rs already provides a CookieJar, implementing hierarchical jars and jar signing. In my crate, user_agent::CookieStore is more concerned with storing/expiring/updating cookies per RFC6265, and handles storing cookies and setting outgoing cookie headers based on URLs visited during the Session, as well as serializing the current session (to .json currently, but really to anything serde will serialize to) so they can be saved/restored. So, they are in some sense orthogonal currently, but feels as though they could be merged as well.
• As mentioned, the crate currently tries to generalize the concept of Session, although the only concrete impl is HyperSession. This was done with the vain hope of making a library for "all the webs" (curl clients, etc.), but the realities of needing to get my project done means this was (IMO) sub-optimally implemented. I'm not sure if it makes more sense to ditch the currently not very ergonomic abstraction and focus on implementing specifically towards reqwest, or to fix/maintain the abstraction and publish the crate separately as user_agent and just implement a ReqwestSession here. At a high level, the Session implementation is mainly concerned with defining how to get/set cookies into/out of Request/Response objects.

Obviously some questions would be more easily answered with some code to review; I just wanted to get a feel for the lay of the land before updating code to be suitable for a PR.

I'm making an authenticated GET request that succeeds, but any attempt to read_to_string or read_to_end just hangs indefinitely, but only under all the following criteria:

• OSX (works fine on linux)
• responses in the range of 3835-16120 bytes (~260 bytes short of the 4K/16K boundaries).
• over SSL (works fine if I make the same request through a local http nginx proxy)

This is the debug output of an example Response object I see before it hangs during reading:

Response { inner: Response { status: Ok, headers: Headers { Date: Wed, 23 Nov 2016 03:06:35 GMT, X-Frame-Options: DENY, Strict-Transport-Security: max-age=0; includeSubDomains; preload, X-Data-Type: directory, Connection: keep-alive, Content-Length: 7452, Content-Type: application/json; charset=utf-8, }, version: Http11, url: "https://api.algorithmia.com/v1/connector/data/anowell/foo", status_raw: RawStatus(200, "OK"), message: Http11Message { is_proxied: false, method: None, stream: Wrapper { obj: Some(Reading(SizedReader(remaining=7452))) } } } }

And yet, I haven't managed to repro this with a simple test script to just reqwest::get some public URL in that size range, so I'm still missing some contributing factor. I've also tried an old build of my client which used hyper+openssl, and it had no issue.

Things on my mind to try still:

• Read in smaller chunks to see if perhaps read_to_end is waiting on the wrong number of bytes
• Put together the minimal repro using an API key for a shareable test account
• Try a bit harder to find a repro outside of our API

Port to Futures 0.3

What?

The point of this exercise is twofold-

1. refactor the external API to use futures 0.3 such that this crate is compatible with the async/await syntax
2. use the async/await syntax internally to refactor this crate

Why?

I currently depend on this crate for my crate chesterfield and i've had to use the compat layer to get futures 0.3. This means that there's about 4 places in my codebase where i've had to use the word compat(). For just a few dozen hours of work refactoring reqwest, i can remove these 4 calls. It really is that easy. In all seriousness it should massively contribute to the maintainability of this crate, and make the learning curve a little more shallow for people hoping to contribute features or fix bugs.

How?

I've done all the easy parts swapping the old Future and Stream implementations to the new style. I've also moved the crate to the 2018 edition (see #589) to allow to use async/await syntax.

I'm at a point now where I'd really appreciate some input, both from someone more familiar with this crate, and someone more familiar with the new futures crate.

• [x] port all the obvious bits from futures 0.1 to futures 0.3

• [ ] rewrite the 'wait' module with the spawning API (Some help here would be appreciated!)

• [ ] refactor the tests as required

• [ ] rewrite the examples using async/await syntax

• [ ] look for opportunities to refactor internally using async/await syntax. Some obvious candidates include-

  • ClientHandle::new
  • ClientHandle::execute_request
  • "impl Connect for Connector"

Notes

I humbly suggest that the best way to start using tokio and hyper 0.1 futures is to use the futures compat layer to convert them to futures 0.3, and vice versa to run 0.3 futures on tokio executors. When tokio and hyper futures 0.3 stabilises, we can switch to the new releases and simply remove the .compat() methods. This way we skip the whole hell of depending on the master version of every dependency.

ANY INPUT GREATLY APPRECIATED!!

Inspired by voider1's implementation in #143 . Fix #4

The most important feature of this version is that the request body is implemented as Read so that it does not need to be stored in memory when files are sent.

Additionally the API is different. I just went with what felt right to me because I had to change most things from voider1 anyways: In this version there is no additional MultipartRequestBuilder struct, instead there is a multipart method (like json or form) which sets the body of the request appropriately.

There are still many things marked as TODO in comments in the code where I wasnt sure what the best way to proceed was.

I also need to add tests, so far I have only written but never ran it.

We can probably cherry pick the best ideas from both pull requests. I also looked into using existing multipart crates but to me it seems like they cant easily be used none of them implement Read.

The author of a client library/tool isn't always able to stop a server from using self-signed or otherwise-invalid certificates. Is there a way to ask reqwest to ignore validation for a specific request?

(In my specific scenario, the server will only respond over HTTPS, so asking over bare HTTP isn't an option either)

For #161

I figured I'd open this up for visibility, and because it's easier to talk about code in PRs than issues or gitter. It's an attempt to support decoding in the async implementation, but there may be better approaches. This does introduce a bit of indirection and complexity.

I'm seeing something that has me very confused. I'm making this request:

            match self
                .client    // A reqwest::blocking::Client
                .post(&self.url)
                .header(CONTENT_TYPE, "application/json")
                .body(request_json.to_string())
                .send()

The server receives the request, and replies as seen in the server logs:

[2021-03-11T01:19:34.370440100Z TRACE jsonrpc_core::io] Request: {"id":1,"jsonrpc":"2.0","method":"getSlot","params":[{"commitment":"confirmed"}]}.
[2021-03-11T01:19:34.370516100Z DEBUG jsonrpc_core::io] Response: {"jsonrpc":"2.0","result":73,"id":1}.

This response gets registered in the client's logs for the request:

2021-03-11T01:19:34.369761900+00:00 [DEBUG] (12) hyper::client::pool: reuse idle connection for ("http", 172.23.0.12:8899)
2021-03-11T01:19:34.370162600+00:00 [DEBUG] (12) hyper::proto::h1::io: flushed 189 bytes
2021-03-11T01:19:34.371146500+00:00 [DEBUG] (12) hyper::proto::h1::io: parsed 3 headers
2021-03-11T01:19:34.371245700+00:00 [DEBUG] (12) hyper::proto::h1::conn: incoming body is content-length (37 bytes)
2021-03-11T01:19:34.371366300+00:00 [DEBUG] (12) hyper::proto::h1::conn: incoming body completed
2021-03-11T01:19:34.371491600+00:00 [DEBUG] (12) hyper::client::pool: pooling idle connection for ("http", 172.23.0.12:8899)
2021-03-11T01:19:34.371842600+00:00 [DEBUG] (12) reqwest::async_impl::client: response '200 OK' for http://172.23.0.12:8899/

and yet, that code returns an Err:

    ....
    8: error sending request for url (http://172.23.0.12:8899/): operation timed out
    9: operation timed out   // Bottom of stacktrace

How can this be?

I'm currently hosting my sites behind the reverse proxy Caddy and I'm using a software called Vigil to periodically poll the status of these sites. Now, the problem is that whenever Vigil polls a site, the following error message shows up in Caddy's logs:

2018/12/19 23:14:51 Unsolicited response received on idle HTTP channel starting with "0\r\n\r\n"; err=<nil>

The error message appears to be harmless in the sense that both Vigil and Caddy work fine in spite of it, but I still think that it's something that should be looked into because something seems to be wrong with the HTTP request that Vigil sends out when it polls a site.

I already opened an issue about this problem on Vigil's bug tracker and the author (@valeriansaliou) asked me to also report it on this project's bug tracker because Vigil is using the reqwest library to do HTTP checks,.

I've been developing the tooling for an apt replacement with the async client, which may need to fetch a couple dozen files at the same time (depending on how many sources are active and what files those sources' release files point to).

The following error occurs when sharing a single async::Client across all connections:

Is this something that could be handled better by reqwest? Perhaps return a NotReady when there's too many files open.

Sometimes, when a reqwest::blocking::Response is dropped, it appears to deadlock and the drop call never returns. On my setup, it appears to work roughly 70% of the time. Tested on latest master code as of right now.

fn main() {
    for _ in 0..10 {
        may_deadlock();
    }
}

fn may_deadlock() {
    let stream_url = "https://live-bauerse-fm.sharp-stream.com/nrj_instreamtest_se_mp3";

    let mut stream = {
        // setup
        let mut builder = reqwest::blocking::Client::builder();
        // note, I used a proxy:
        // builder = builder.proxy(reqwest::Proxy::all("----").unwrap());
        // get the stream
        builder = builder.connect_timeout(std::time::Duration::from_secs(2));
        builder = builder.timeout(std::time::Duration::from_secs(2));
        let client = builder.build().unwrap();
        let builder = client.get(stream_url);
        builder.send().unwrap()
    };

    println!("ok, reading some data!");
    let mut buffer = [0; 1400];
    use std::io::Read;
    stream.read(&mut buffer).unwrap();
    println!("attempting to drop stream. This will deadlock SOMETIMES...");
    std::mem::drop(stream);
    println!("we get here as expected ~70% of the time");
}

ok, reading some data! attempting to drop stream. This will deadlock SOMETIMES... we get here as expected ~70% of the time ok, reading some data! attempting to drop stream. This will deadlock SOMETIMES... we get here as expected ~70% of the time ok, reading some data! attempting to drop stream. This will deadlock SOMETIMES... we get here as expected ~70% of the time ok, reading some data! attempting to drop stream. This will deadlock SOMETIMES...

then, stuck.

This is untested, and based on my PR for hyper-native-tls (https://github.com/sfackler/hyper-native-tls/pull/7), but I'd like to know if you think this is the right direction.

This is a step toward fixing https://github.com/seanmonstar/reqwest/issues/15

Without the change, Error::into_io would always wrap self in io::Error, even if the source is already is an io::Error.

Signed-off-by: Jiahao XU [email protected]

Enables the user to create multiple requests with the same url without Url::clone, which could be using different methods, e.g. HEAD first then fallback to GET to check for existence of url, or simply because they want to create multiple request to the same url with the same method, and also avoids Url::clone in Request::try_clone.

This also avoid calling Url::clone in PendingRequest on creating Error or creating the Response, or when pushing into urls.

Plus

• cleanup some clippy warnings
• Impl Response::url_arc
• Impl Request::url_arc
• Impl IntoUrl for Arc<Url> and &Arc<Url>
• Impl Error::url_arc
• Apply changes to async_impl::Request to blocking::Request
• Apply changes to mod wasm

Signed-off-by: Jiahao XU [email protected]

Happens here https://github.com/seanmonstar/reqwest/blob/cdbf84feb1d86b8288796631f2120de5363b3ba9/src/async_impl/client.rs#L1544 due to expect call in https://github.com/seanmonstar/reqwest/blob/cdbf84feb1d86b8288796631f2120de5363b3ba9/src/into_url.rs#L67-L71

It would be nice to get an error that one could recover from, but before it happens, a simple doc change should be sufficient.

ref: https://github.com/denoland/deno/pull/17164

For curiosity, is there any plan to implement http signatures (https://datatracker.ietf.org/doc/html/draft-cavage-http-signatures-12) like https://docs.rs/http-sig/latest/http_sig/ ?

I am trying to follow your example for docker https://github.com/seanmonstar/reqwest/blob/master/examples/tor_socks.rs

Can you please give me a clue why it is not working

Error: Tor Proxy not found in the main.rs expect

main.rs

use reqwest::{Client, Proxy, Response};

#[tokio::main]
async fn main() {
    let proxy: Proxy = Proxy::all("socks5h://127.0.0.1:9050").expect("Tor Proxy not found");

    let client: Client = Client::builder()
        .proxy(proxy)
        .build()
        .expect("Fail to build proxy client");

    let res: Response = client
        .get("https://check.torproject.org")
        .send()
        .await
        .unwrap();

    println!("Status: {}", res.status());

    let text: String = res.text().await.unwrap();
    let is_tor: bool = text.contains("Congratulations. This browser is configured to use Tor.");
    println!("Is Tor: {}", is_tor);
}

Dockerfile

FROM rust:1.65-alpine3.16 as builder

RUN apk update && apk upgrade
RUN apk add build-base
RUN apk add pkgconfig
RUN apk add openssl-dev

WORKDIR /app

COPY Cargo.toml .
COPY .cargo .cargo/
COPY vendor vendor/
COPY src src/

RUN cargo build --release

###

FROM alpine:3.16

RUN apk update && apk upgrade
RUN apk add tor
RUN apk add torsocks
RUN apk add openrc

WORKDIR /app

COPY --from=builder /app/target/release/reqwest_test .
COPY start.sh .

RUN chmod +x start.sh

CMD ./start.sh

start.sh

rc-service tor start &

sleep 1

/app/reqwest_test