@petehayes102 Tests for PHP bindings are failing with:-

configure: error: host_connect function not found in libinapi

This isn't related to this pull request though.

Fixes https://github.com/intecture/api/issues/34.

The Agent provides endpoints for the API that are routable via a ZeroMQ pipe. However all the metadata required to autogenerate endpoints and serialize the data for transit is contained within the API.

In order to reduce duplication and unnecessary admin overhead, a parser should be built to interpret API endpoints and autogenerate the required stubs (agent_endpoints.rs). A Rust module, agent.rs, would be the binary target responsible for initialising ZMQ sockets and starting the Agent service.

This issue is blocked by https://github.com/Intecture/api/issues/14.

Among other benefits, the 0.9 JSON value is a lot nicer to work with. This line in api/src/host/data/mod.rs:

me.as_object_mut().unwrap().insert("_payloads".into(), serde_json::to_value(payloads));

could be:

me["_payloads"] = json!(payloads);

At runtime, all status output (apart from errors) has to be done manually by the user. I.e. printing a message to stdout before and after a job so the user can keep track of Intecture's progress.

Intecture primitives should instead publish their status to a PUB socket which the user can subscribe to. This way the user is saved from having to write their own status output.

Currently in the PHP binding, using the Telemetry primitive requires two lines:

$telemetry = Telemetry::init($host);
$tdata = $telemetry->get();
echo $tdata['os']['platform'];

This sucks. A better way to do it would be to allow declaring a new Telemetry object rather than the unimplemented singleton pattern, and to extend the ArrayObject class. The result would look like this:

$telemetry = new Telemetry($host);
echo $telemetry['os']['platform'];

Currently the API relies on a static timeout when communicating with an Agent. This is totally unreliable and has to be set unreasonably high to prevent timeouts on intential long running processes.

Instead a heartbeat should be used over the API socket to ascertain the uptime of both API and Agent.

Currently API primitives achieve cross-platform compatibility by splitting API endpoints into target specific implementations. This results in one file per platform (i.e. CentOS, FreeBSD, Mac OS etc.), which can get quite bloated. Also, there is a deliniation between "local-run" (for the Agent) and "remote-run" (for the dev box) features which muddies the waters significantly.

Instead, we should keep primitives' target specific implementations in the same place as their public APIs, and refer to them by technology instead of OS (e.g. Service::systemd, Service::init etc.). A Provider trait will be used to expose an available() method to determine which providers are available for the platform.

OS targets become text file definitions that provide a list of default providers for each primitive. build.rs will autogenerate Rust code from these targets.

Dependency injection will be used to inject the correct Provider into each primitive upon initialisation. Remote and local features will be removed in favour of DI. For a connected Host, DI will select the Remote Provider. For a disconnected [local] host, DI will select a Provider based on their availability (Provider::available()), filtered by the OS target definition.

Currently Intecture does not provide a way to manage data associated with a payload.

Here is a very informal spec:

1. Nodes, payloads, tags etc. are all data files
   1. A data file with the type “node” will be the entry point for any build
   2. Apart from “node” type, types are arbitrary strings (do we want this?)
2. Data files can optionally have an “include” parameter for requiring other files/payloads
3. Data file items can have conditions that allow them to specify when they are applied
   1. Like CSS, the most specific item wins
   2. If two items have the same specificity, the higher order file wins - i.e. the winning file is the one that included the other file, thus is higher in the data hierarchy
4. Data file items can optionally merge with less specific items, rather than overwriting
   1. Merges will either append to an existing array, or create an array containing the two merged items

Currently the authentication server's endpoint (e.g. "auth.example.com:7101") is specified in both the project config file (project.json) as well as the Host::connect() signature. As it's a data point that is unlikely to change, and also not specific to a Host, it belongs in config and not in code.

The Host::connect implementation must be changed to read the auth_server parameter from an external source. Either:

1. project.json in the project root - requires including the zdaemon crate for config file helpers and duplicating config.rs from the CLI repo, or,
2. Preferably, pass in the value from the CLI => stdin of the user project. In this case an args handler would need to be written to retrieve and cache 'system' values, while exposing user defined values back to the user.

Currently we test user provided pointers to make sure they are valid. However we don't provide the same level of due diligence on struct member pointers before converting the struct into the Rust-native equivalent.

We need some way of testing structs succinctly that allows us to return a Result<> type that we can handle safely. One way of doing this is to throw out the native convert trait implementations and roll our own with a friendly return signature. This isn't ideal as we don't want to lose Rust's implicit conversions.

Another method is to create a trait that would be implemented by FFI structs to check their own members:

trait HasNull {
    fn has_null(&self) -> Option<Vec<&'static str>>;
}

impl HasNull for Ffi__MyStruct {
    fn has_null(&self) -> Option<Vec<&'static str>> {
        let mut null_fields = Vec::new();
        if self.ptr_member.is_null() {
            null_fields.push("ptr_member");
        }

        if null_fields.len() > 0 {
            Some(null_fields);
        } else {
            None
        }
    }
}

^^^ Firing from the hip!

The FFI still relies on some unwrap()s, especially in the convert trait implementations. As a failsafe, we should implement catch_unwrap() to trigger an error instead of crashing the thread.

Cases that I'm currently aware of:

• File::mv vs. File::copy - should also have File::cp and File::move with the long names as aliases
• DirectoryOpts vs FileOptions - Should be DirectoryOptions, deprecate DirectoryOpts
• Service::new_service() accepts a ServiceRunnable::Service OR ServiceRunnable::Command, which is misleading. Should perhaps be Service::new_runnable?

Some earlier versions of Debian /proc/cpuinfo do not contain all the key/values that Intecture expects to find. For example, "cpu cores" is missing on Debian Wheezy.

It seems that by using the lscpu command, we will get more predictable output across flavours/versions.

Currently payloads are executed as standalone processes with ZMQ pipes for IPC. This is bad for a number of reasons:

• For large projects, it can lead to a massive number of child processes being spawned
• Each project must be compiled individually, which is incredibly slow (assuming not interpreted)
• Data exchange is very poor and prevents programmers from exposing rich APIs to other payloads

Any new design must allow programmers to include same-language payloads as libraries instead of processes.

Challenges:

• Exchanging data across language barriers is complex
• Most developers will not want to write FFI code themselves for interop
• Ideally interpreted payloads should stay within the same process to allow shared memory, which would require hooks into the interpreter itself (e.g. reimplementing PHP's SAPI so that it can be called as a library from Rust)

Data macros (host::data::macros) are currently hardcoded, which had led to a lot of code duplication. A macro exists (need_macro!) to generate the required macros, however there is an issue with the eager expansion of $crate::.

See issue: https://users.rust-lang.org/t/crate-being-expanded-too-eagerly-for-nested-macros/7592