Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cluster-test] Add experiment to automatically generate CPU FlameGraph #2387

Closed
wants to merge 2 commits into from
Closed

[cluster-test] Add experiment to automatically generate CPU FlameGraph #2387

wants to merge 2 commits into from

Conversation

ankushagarwal
Copy link
Contributor

Summary

  • We generate load on the cluster and pick one validator and run perf against it to generate the FlameGraph
  • This FlameGraph is then copied locally to cluster-test.
  • The next step would be to upload this local file to S3

Test Plan

Tested on my cluster

fn apply(&self) -> BoxFuture<Result<()>> {
async move {
info!("GenerateCpuFlamegraph {}", self.instance);
let cmd = format!(r#"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need some sort of set -e to make it fail if any command fails?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that'll be a good idea

impl ExperimentParam for CpuFlamegraphParams {
type E = CpuFlamegraph;
fn build(self, cluster: &Cluster) -> Self::E {
let (perf_instance, _) = cluster.split_n_validators_random(1);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just make random_validator function in Cluster? Probably going to be used a lot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It already exists :-/

) -> BoxFuture<'a, anyhow::Result<Option<String>>> {
async move {
let buffer = Duration::from_secs(60);
let mut tx_emitter = context.tx_emitter.clone();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be rewritten in one task, so that you don't need to clone tx_emitter. Something like that:

let emit_future = tx_emitter.emit_txn_for(tx_emitter_duration, validator_instances);
let flame_graph_future = GenerateCpuFlamegraph::new(self.perf_instance.clone(), self.duration_secs).apply();
let flame_graph_future = tokio::time::delay_for(buffer).then(flame_graph_future);
join_all(emit_future, flame_graph_future).await?

Might need some .map to make sure both futures return same type, which is required by join_all

@andll
Copy link

andll commented Jan 31, 2020

About usage - do you plan to include flamegraph link to perf report in #changelog in next diffs?

@ankushagarwal
Copy link
Contributor Author

About usage - do you plan to include flamegraph link to perf report in #changelog in next diffs?

Yes, once the s3 setup is complete

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

Summary
We generate load on the cluster and pick one validator and run perf against it to generate the FlameGraph
This FlameGraph is then copied locally to cluster-test.
The next step would be to upload this local file to S3

Test Plan
Tested on my cluster
@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@ankushagarwal ankushagarwal deleted the flamegraph-cluster-test-2 branch February 4, 2020 21:04
@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

@diem-cla-bot
Copy link

diem-cla-bot bot commented Feb 4, 2020

Need CLA signature Needs signature

Thank you for your submission. We require all contributors to
sign the CLA before we can accept your contribution.

Have you signed the CLA already, but your status is still pending? Recheck CLA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants