This is a work-in-progress tool to gather a timely dataflow trace and render the dataflow graph.

Please give it a try!

• clone this repo, switch to this branch (graph-tool);
• start the tool with cargo run -- --source-peers 2 graph --out graph.html;
• in another shell, start the computation you'd like to visualize, and instruct timely to send trace data to the tool: TIMELY_WORKER_LOG_ADDR="127.0.0.1:51317" cargo run ... -- -w 2
• once the computation you'd like to visualize has started, press enter at the tool prompt;
• open graph.html with a modern browser.

It should look something like this:

Goal of this is to track the number of arranged tuples across all arrangements in the computation, as the computation runs. Right now the only thing is doing is ingesting Batch and Merge events into a Differential collection (arrangement_id, t, +-tuples) and reporting accumulated counts every second.

Remaining TODOs.

• [x] join with Operates to extract arrangement names
• [x] make output interval configurable
• [x] make differential logging port configurable

This is not as plug-n-play, because it requires augmenting a Differential computation with the following snippet:

if let Ok(addr) = ::std::env::var("DIFFERENTIAL_LOG_ADDR") {
    info!("enabled DIFFERENTIAL logging to {}", addr);

    if let Ok(stream) = ::std::net::TcpStream::connect(&addr) {
        use differential_dataflow::logging::DifferentialEvent;

        let writer = ::timely::dataflow::operators::capture::EventWriter::new(stream);
        let mut logger = ::timely::logging::BatchLogger::new(writer);

        worker.log_register()
           .insert::<DifferentialEvent,_>("differential/arrange", move |time, data| logger.publish_batch(time, data));
    } else {
        panic!("Could not connect to differential log address: {:?}", addr);
    }
}

Hello. I was trying to follow the example on the README.md but tdiag gets stuck.

1. One one terminal I execute: cargo run --release -- --source-peers 2 graph --out graph.html
2. One second terminal I execute: env TIMELY_WORKER_LOG_ADDR="127.0.0.1:51317" cargo run --release --example pagerank 1000 100000 -w 2
3. pagerank runs to completion.
4. tdiag acknowledges connection via:

Listening for 2 connections on 127.0.0.1:51317
Trace sources connected
Press enter to generate graph (this will crash the source computation if it hasn't terminated).

5. I press enter but tdiag hangs indefinitely.

Looking at the stack trace of tdiag there are two threads. The main thread is waiting on a thread join. Thread2 also seems stuck on await_events. Stack trace for Thread2:

futex_wait_cancelable 0x00007ffff7d9c376
__pthread_cond_wait_common 0x00007ffff7d9c376
__pthread_cond_wait 0x00007ffff7d9c376
std::sys::unix::condvar::Condvar::wait condvar.rs:73
std::sys_common::condvar::Condvar::wait condvar.rs:50
std::sync::condvar::Condvar::wait condvar.rs:200
std::thread::park mod.rs:923
<timely_communication::allocator::thread::Thread as timely_communication::allocator::Allocate>::await_events thread.rs:44
<timely_communication::allocator::generic::Generic as timely_communication::allocator::Allocate>::await_events generic.rs:99
timely::worker::Worker<A>::step_or_park worker.rs:216
timely::execute::execute::{{closure}} execute.rs:206
timely_communication::initialize::initialize_from::{{closure}} initialize.rs:269
std::sys_common::backtrace::__rust_begin_short_backtrace backtrace.rs:130
std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} mod.rs:475
<std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once panic.rs:318
std::panicking::try::do_call panicking.rs:297
__rust_try 0x000055555661a74d
std::panicking::try panicking.rs:274
std::panic::catch_unwind panic.rs:394
std::thread::Builder::spawn_unchecked::{{closure}} mod.rs:474
core::ops::function::FnOnce::call_once{{vtable-shim}} function.rs:232
<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once boxed.rs:1034
<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once boxed.rs:1034
std::sys::unix::thread::Thread::new::thread_start thread.rs:87
start_thread 0x00007ffff7d95609
clone 0x00007ffff7ed1103

Please lmk if I missed something when executing the commands?

Reports aggregate runtime for each scope/operator.

See the main README for usage instructions (no need for --output here).

Sample output:

$ cargo run -- --source-peers 4 profile
Listening for 4 connections on 127.0.0.1:51317
Trace sources connected
Press enter to stop collecting profile data (this will crash the source computation if it hasn't terminated).
[scope]	Dataflow	(id=0, addr=[0]):	1.7661819e-2 s
[scope]	Dataflow	(id=120, addr=[2]):	8.231392e-3 s
[scope]	Dataflow	(id=82, addr=[1]):	6.199795e-3 s
[scope]	Region	(id=116, addr=[1, 3]):	3.12838e-3 s
	Reduce	(id=50, addr=[0, 26]):	4.49143e-4 s
	Reduce	(id=60, addr=[0, 31]):	4.41773e-4 s
	Join	(id=147, addr=[2, 15]):	4.17792e-4 s
	Arrange	(id=8, addr=[0, 6]):	3.81e-4 s
	Concatenate	(id=66, addr=[0, 34]):	3.76647e-4 s
	Reduce	(id=10, addr=[0, 7]):	3.4767e-4 s
	Join	(id=99, addr=[1, 3, 7]):	3.43986e-4 s
	Arrange	(id=21, addr=[0, 12]):	3.39887e-4 s
	Concatenate	(id=16, addr=[0, 10]):	3.30095e-4 s
	Concatenate	(id=37, addr=[0, 20]):	3.1384e-4 s
	Arrange	(id=154, addr=[2, 18]):	2.75343e-4 s
	Arrange	(id=42, addr=[0, 22]):	2.70947e-4 s
	Map	(id=19, addr=[0, 11]):	2.66121e-4 s
[snip]

This is a subset of #1. It contains the functions to create EventReaders and a Replay operator that support early shutdown. make_readers supports both files and sockets as sources, borrowing from https://github.com/li1/snailtrail.

@li1 please give these a try! We may/will publish these as a crate, but for now you can use a git dependency.

Retractions were being filtered while still assessing the number of records, causing the calculations to not reflect progress made by merges. At the same time, the filtering was not applied to count outputs, so retractions there (when counts increased) were presented as positive occurrences (though they should all have been immediately followed by new larger values, as the previous filtering ensured no counts ever decreased).

A first draft of extending the README with usage information.

(easiest to look at directly: https://github.com/TimelyDataflow/diagnostics/tree/readme)

This PR upgrades the Timely and DD dependencies to 0.11, fixing a breaking change to the builder interface, and adds support for batch drop events, as mentioned in #12.

(@frankmcsherry )

Differential just got https://github.com/TimelyDataflow/differential-dataflow/pull/201, which makes it possible to track batches created in transient trances. Seems like differential arrangements could do something useful with that.

This fixes #8 .

Currently, tdiag won't build since it relies on the crates.io 0.1 release which still uses 0.9 dependencies. If you'd like me to, I can also bump the crate versions & connect dependency from 0.1.1-pre to e.g. 0.1.2-pre?

The documentation was manually overridden (I changed it to append the manual part to the auto-generated part), causing documentation for --output-interval to be omitted. Sub-second intervals were not supported. I fixed this, so now millisecond-granularity windows are supported, instead of just whole seconds.

@frankmcsherry:

It is hard to diagnose a "stuck" timely dataflow computation, where for some reason there is a capability (or perhaps message) in the system that prevents forward progress. In the system there is fairly clear information (in the progress tracking) about which pointstamps have non-zero accumulation, and although perhaps not strictly speaking a "visualization" we could imagine extracting and presenting this information.

@antiguru recently had a similar issue, in which he wanted to "complete" a dataflow without simply exiting the worker (to take some measurements), and when he attempts this the dataflow never reports completion. The root cause was ultimately that a forgotten input was left un-closed.

One idiom that seemed helpful here was to imagine a version of the dataflow graph that reports e.g. whether operators have been tombstoned or not (closed completely, memory reclaimed). This would reveal who was keeping a dataflow open, which is a rougher version of what is holding a dataflow back. We might also look for similar idioms that allow people to ask, for a given timestamp/frontier, which operators have moved past that frontier and which have not, revealing where in the dataflow graph a time is "stuck".

This may be needed for complex graphs like Frank's epic doop graph.

Various snippets from #1:

@frankmcsherry

I've been thinking a bit about how to present these, and one thought was: maybe it makes sense to have two nodes for the feedback node, and to not connect them other than visually. This maybe allows the graph to dangle a bit better, and reveals the acyclic definitions.

@comnik

I think this could be a good use case for a touch of interactivity, e.g. draw the nodes somewhat differently to indicate an outgoing / incoming feedback edge, and then highlight the pair when the user hovers on either of the two nodes.

As an experiment, I built an extra script for adding DataScript into the mix. This is intended to be completely opt-in, without changing anything about the current representation.

I also added a hook to re-render the whole thing reactively.

This should give us a low-overhead (no React!) way to experiment with a few more dynamic features, such as highlighting feedback edges.

It would be helpful to have scopes be exported as well, which would allow us to do things such as collapsing / expanding scopes.