I am currently doing some load testing comparing my previous app, built in python, and this new application built in rust using your google-cloud-rust repo and with the rust application, Im getting the below error when sending a high volume of requests to pubsub. thread 'actix-rt|system:0|arbiter:0' panicked at 'calledResult::unwrap()on anErrvalue: GAX(Auth(HttpError(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(169.xxx.xxx.xxx)), port: None, path: "/computeMetadata/v1/instance/service-accounts/default/token", query: Some("scopes%3Dhttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpubsub.data"), fragment: None }, source: TimedOut })))'

the same number of requests are sent to the python app, which is forwarding the request to the same pubsub service in GCP, and I do not see any failures/errors on the python app.

Any chance you could help me understand why this is happening and how I can fix it.

Cheers

I'm working on a high-perf requirement pubsub client where I've been told it would be unacceptable to individually ack messages. I wasn't able to see examples for batch acking here, and from what I see going on internally in ReceivedMessage.ack() (reference), it calls the crate-private SubscriberClient.acknowledge() method which supports batches. Is this API exposed publicly somewhere in the package or ack-batching taken care of in some other way?

Issue

When running the below code block (from the readme) to instantiate Client struct

let client = Client::default().await.unwrap();

I get the following error:

GAX(Auth(HttpError(reqwest::Error { kind: Decode, source: Error("EOF while parsing a value", line: 1, column: 0) })))

I am using the google_auth crate from this repository utilising the below code block to set the credentials (from the readme)

    let credentials_path = PathBuf::from("./src/keys/dev-key.json");
    let service_account = CustomServiceAccount::from_file(credentials_path).expect("Service account threw an error");
    let authentication_manager = AuthenticationManager::from(service_account);
    let scopes = &["https://www.googleapis.com/auth/cloud-platform"];

Setting the credentials seems to be working fine; when I run the code in debug, I can see the below all populating with the correct values

My environment

rustc --version
rustc 1.62.1 (e092d0b6b 2022-07-16)

cargo --version
cargo 1.62.1 (a748cf5a3 2022-06-08)

My Cargo.toml file looks like this:

actix-web = "4.1.0"
awc = "3.0.0"
colored = "2.0.0"
dotenv = "0.15.0"
env_logger = "0.9.0"
futures = "0.3.21"
gcp_auth = "0.7.3"
google-cloud-gax = "0.7.0"
google-cloud-googleapis = {version = "0.4.0", features = ["pubsub"]}
google-cloud-pubsub = {version = "0.4.0", features = ["trace"]}
google_auth = "0.2.1"
prost-build = "0.10.4"
protobuf = "3.1.0"
serde = { version = "1.0.140", features = ["derive"] }
serde_json = "1.0.82"
tokio = "1.20.1"
validator = "0.16.0"
validator_derive = "0.16.0"

I want to be able to get all the objects from a particular bucket like the below documentation

https://cloud.google.com/storage/docs/json_api/v1/objects/list

Can this be implemented? I am trying to add it to the current code, but I need some help understanding how to go about this

This will cause CI to fail if multiple versions of a dependency is being used. This will prevent that someone accidentally adds several versions of a dependency.

In some cases there is no solution to having multiple versions, and in those cases an exception can be added in deny.yaml, I've added an exception for windows-sys where there is currently two versions in the dependency graph.

This puts tonic-build behind a feature, so it only needs to be included when generating the protos.

I also re-generated all protos with the latest googleapis

There seems to be no way to provide the google pubsub Client with a Project directly like how the cloud storage Client allows it. This PR simply adds the Project to the new constructor instead of calling google_cloud_auth::project() within it. The only difference right now is that the pubsub Client requires a owned Project but the cloud storage Client requires a &Project. This is because Environment::GoogleCloud requires an owned Project and Project does not have implement Clone or ToOwned.

Also, noting that adding Project to the constructor of pubsub Client does interfere with the usability of the emulator. There should be a work around for this and I am open for discussion on this.

I'm working on a Google pub/sub connector over at https://github.com/risingwavelabs/risingwave where sources can be added ad-hoc through a SQL interface. As such I feel it makes natural sense to not tie down credentials to files on the filesystem, but I don't think its possible yet with the google-cloud-auth crate. I can work on a PR to make this possible if it sounds good!

At work for high performance we have apis that operates in bulk not in single message. Maybe adding a new method in publisher that support an iterable of ReservedMessages could be useful to prevent send one by one message to the channel. Maybe an enum Reserved::Multi and Reserved::Single could be useful to detect if is single or not for the matching here.

Looking at the compose.rs, I see the build function doesn't give us the ability to specify a mime type. According th the google documentation, the mime type should be the type of the first object if not set, but this is not showing on the cloud console. Can you add the ability to specify a mime type?

pub(crate) fn build(base_url: &str, client: &Client, req: &ComposeObjectRequest) -> RequestBuilder {
    let url = format!(
        "{}/b/{}/o/{}/compose",
        base_url,
        req.bucket.escape(),
        req.destination_object.escape()
    );
    let builder = client.post(url).query(&req).json(&req.composing_targets);
    if let Some(e) = &req.encryption {
        e.with_headers(builder)
    } else {
        builder
    }
}

Using:

[dependencies]
google-cloud-storage = "0.6.1"
tokio = { version = "1.23.0", features=["rt-multi-thread"]}

We get:

error[E0432]: unresolved import `google_cloud_storage::client::http`

I'm wondering if you would be open to making the auth crate optional, so that is possible to provide your own auth implementation?

We use our own auth crate for all of our other gcp-related auth. It implements a sans-io approach so that we can control what http client is being used so that we can have the same http client throughout all of our projects.

We are trying to keep our dependency tree down as much as possible (as you might have noticed from my previous contributions), and for us it would be great to be able to use the same gcp auth implementation for all of our dependencies as well.

If you are open to it, I'm of course happy to do the work and implement it. But I wanted to check first that you are willing to merge such a feature.

I haven't looked into how to implement it, but I'm thinking that I would put the auth crate/implementation behind a feature (that is part of the default features), and then use some a trait or something that can be implement by other auth crates.

But I'll properly look into how to implement it if you think this is something you would accept.

Firstly, thank you so much for this library; it's great and very straightforward to use (I've used the pub-sub crate).

I think BigQuery is one of the most used products inside GCP, so it would be super helpful to have a native rust crate to interact with BQ. Thanks again 👍

Hello!

First of all, thanks for working on this library! I was wondering if you're thinking of adding Datastore support in the future releases. I'm currently using google-cloud-rs however it is not really maintained actively and missing some features (e.g. unindexed values).

It would be really nice to have a better alternative to google-cloud-rs and I guess you are on it 🐻