Create, share, fetch and model Atomic Data! This project consists of a graph database + server, a CLI and a rust library.

The current configuration setup for the server is not ideal:

• Lot of boilerplate code
• Both env variables and ~/.config/atomic/config.toml are used
• Running from CLI does not really instruct user to setup basic, required fields

Strategy

• [x] Upgrade to clap v3 (currently in beta)
• [x] Use a StructOpt based on the config.rs file
• [x] Set defaults and env names using decorators

Tried running the docker image, got this:

$ docker run -p 80:80 -p 443:443 -v atomic-storage:/atomic-storage --platform=linux/amd64 joepmeneer/atomic-server
exec /server/atomic-server-bin: no such file or directory

This should not happen.

First I should fix the more abstract issue (how is an image pushed that does not work?) and then the more specific one.

Thanks to @rasendubi for reporting this

atomic_lib depends on ureq which (is publicly exposed but internally) seems used only for `validate which seems deprecated (see #61).

atomic_server depends on awcwhich seems used only in relation to acme_lib (which itself depends on ureq so it seems odd a) that extra client calls are even necessary, and b) a different client library is used).

It seems sensible to flag the dependency on awc as optional, tied to feature https.

It seems sensible to use only one http library. Right now simplest is to drop awc since ureq is also used by acme_lib. Smarter might be to drop both awcand ureq and instead use generic-async-http-client which is also used by async-acme (see #192). If I understand correctly, using that with feature tokio-rustls reuses most of the underlying libraries.

If keeping ureq then please consider disabling feature cookies as (to the best of my knowledge) cookies are not being used neither for accessing atomic servers nor to access Letsencrypt. Then crate cookie_storage should only get pulled on with feature https (because apparently acme_lib has that feature always enabled).

Being able to search through data inside your personal atomic server seems like a nice feature to have. In this issue, I'd like to explore the requirements and some possible approaches for implementing a full text search service.

Wishes

• Find specific resources and their URLs extremely fast. This enables using it in things like semantic document editors, so instead of simply typing a word, we're inserting a specific URL of some thing. This will mean that items that are often used will come out 'on top'. To do this, we need a sorting algorithm that uses historical searches and selections. We also need some form of autocompletion.
• Be able to either ignore or include types of resources. Perhaps I'm looking for a specific person, and I don't want to see all documents with that persons name inside it.
• A persons first name in a "firstName" property should weigh more than that string in a "description". Perhaps make character count diminish relevance score.
• Iterative indexing on Commits / when resources are added to the store.

Approaches / implementation ideas

• The API itself should use atomic data, too. The Collections model is probably useful here, since it introduces paginated content.
• The Sonic crate offers performant search features, and simply returns a URL. The developer is planning on making it embeddable.
• It would be nice if it works as an installable plugin. This would require that plugins function as some sort of middleware handler, offer a custom endpoint, be able to write / access data.

Introduces the use of crossbeam_channel to add concurrent / parallel resource processing in db.all_resources.

Contains commits from #396 #394, credits to @AlexMikhalev. Rebased onto main, conflicts resolved. Seems to work.

PR Checklist:

• [x] Link to related issue
• [x] Add changelog entry linking to issue
• [ ] Added tests (if needed)
• [ ] (If new feature) added in description / readme

The server keeps going down since running 29.2, and it throws these errors:

[2021-12-11T23:44:04Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-11T23:44:04Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-11T23:44:04Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-11T23:44:04Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }

PR Checklist:

• [x] Link to related issue closes #204
• [x] Add changelog entry linking to issue
• [x] Added tests (if needed)
• [x] (If new feature) added in description / readme
• [x] Get Playwright working

Keep getting stuck on npx playwright test:

Please install @playwright/test package to use Playwright Test.
  npm install -D @playwright/test

But I just installed it!

      - run: npx playwright install --with-deps
      - run: npx playwright test "e2e.spec.js"

Runs fine locally, too. But for some reason.

I'm still learning Rust, and that means that I'm still not fully understanding traits and object safety.

We have a Store struct based on the Storelike trait, which contains all the data. We also have Resource, which have a reference to a Storelike.

pub struct Resource<'a> {
    propvals: PropVals,
    subject: String,
    classes: Option<Vec<Class>>,
    store: &'a dyn Storelike,
    commit: CommitBuilder,
}

This enables a more convenient API, so we can have:

resource.set_prop("prop", "val")

instead of requiring the store in every single call:

resource.set_prop(store, "prop", "val")

Since our Store is a trait, it's size is not known at compile time. From what I understand, this means that in methods on the Storelike trait where we create Resources, we need to explicitly state that the Store (Self) is Sized:

fn create_agent(&self, name: &str) -> AtomicResult<(String, String)>
    where
        Self: std::marker::Sized,
    {
        let mut agent = Resource::new_instance(urls::AGENT, self)?;
        Ok(())
    }

This where clause is now needed in all functions that create resources, and I feel like that is wrong, although I can't say for sure that it is.

I thought it might make sense to require Sized in the Storelike trait, like this: pub trait Storelike: Sized, and although that removes the need for the where clauses, it introduces another problem. In every signature where a reference to the store is made:

impl Collection {
    /// Constructs a Collection, which is a paginated list of items with some sorting applied.
    pub fn new(
        store: &dyn Storelike,
        collection_builder: crate::collections::CollectionBuilder,
    )

...this error appears:

the trait `storelike::Storelike` cannot be made into an object
the trait `storelike::Storelike` cannot be made into an objectrustc(E0038)
storelike.rs(47, 22): ...because it requires `Self: Sized`

One of the advantages of using and storing Commits, is that we have a full history of every single resource. However, we currently have no way of browsing this information.

Wishes

• I want to click on a link on a resource page to see all versions.
• I want to see a log of changes of some resource. This might be different from the versions.
• I want to see a previous version of some resource, and I want to share its URL.
• I want to restore a previous version of some resource, yet keep the previous changes.

Considerations

• For any list of things (commits for some resource, we should use the Collections model.

Approaches

Versions endpoint

example.com/versions?subject=someresource lists all the version for the resource included in the Subject query parameter. It returns a collection. Perhaps first define the Endpoints feature?

1. User requests a list of versions for some resource
2. A collections is constructed, containing all (or some subset of) the existing versions, one for each Commit. Each version points to a URL of a resource that needs to be constructed (e.g. example.com/versions?subject=someresource&version=signatureHashBlablabla)
3. user requests a specific version
4. That version is constructed by the server, using the commits

Introduce sequential TPF queries

Knowing which commits are linked to a resource, can be done using two TPF queries:

1. Get me all the items where is-a = commit
2. AND subject = mySubject

Some time ago, the great Ruben Verborgh told me that you can actually convert (all?) SPARQL queries into a set of sequential TPF queries. That's pretty cool. How should we create an interface to facilitate this? It probably requires some form of nesting queries, and some form of combined queries.

But... this kind of discussion quickly turns into a difficult discussion of which methods need to be included in the newly designed query language, and will kind of lead to some almost-turing-complete language, but not fully. Therefore, we might want to try a plugin-like approach, where the query logic resides in code.

Plugin (endpoint + extend resources)

Atomic Plugins are currently just a bunch of thoughts, but perhaps Versions is a logical first plugin. Plugins should be apps that can be added to an Atomic Server at runtime. They might be run using a WASM runtime. In any case, they need to add functionality to an Atomic Server.

Ideally, the Versioning plugin will have:

• An endpoint (see above) for constructing a version of a resource, and listing all versions / commits of a resource
• Maybe add some links to resources, so you can find the versions of a resource from the resource page.
• A way to restore previous versions (by producing a Commit that reverts some changes)

Constructing a version

A Version is a representation of some Resource at some point in time. Constructing one can (theoretically be done in several ways:

• Get the initial commit of a resource, and apply all commits until you get to the requested version
• Persist all versions and representations (costly!)
• Persist reverse actions, which describe how you can do the inverse of a Commit.

I feel like the first approach is the most logical.

I have a problem with this, though. Some resources don't have commits: for example, resources that are imported from a static AD3 file (such as the default atomic data).

I think this should simply be handled by the plugin. If you try to construct a version for a resource that has no commits, just return an error. Or should it return the static version that?

Option --config-dir or environment variable ATOMIC_CONFIG_DIR is used not only for configuration but also for ephemeral and precious data.

Please align with FreeDesktop Base Directory Specification, distinguishing between...

• Precious data
  • potentially large
  • candidate for privacy-protected backup routines
• Configuration data
  • typically lightweight
  • candidate for less privacy-oriented backup routines
  • often reasonable to reuse unedited across systems.
• Ephemeral cheap data like cache files
  • potentially large (but might support auto-dropping older entries to save space)
  • candidate for storing on extra-fast storage (e.g. SSD, or in special cases a RAM disk)
• Ephemeral expensive data like database indexes
  • potentially large (but might support auto-dropping older entries to save space)
  • might requiring downtime or locks to create
  • candidate for storing on extra-fast storage (e.g. SSD, or in special cases a RAM disk)
• execution-specific data
  • typically lightweight
  • might require root access to create and clean up

Concretely, I suggest these paths:

• config.toml -> $XDG_CONFIG_DIRS/atomic-data/server.toml
• trace-.json -> $XDG_LOG_DIRS/atomic-data/server/trace-.json
• atomic_server_process_id -> $XDG_RUNTIME_DIRS/atomic-data/server_process_id
• db/* -> $XDG_DATA_DIRS/atomic-data/server/db/*
• search_index/* -> $XDG_STATE_DIRS/atomic-data/server/search_index/*

Seems directories is the perfect library for this.

TODO:

• [x] Build the cache
• [x] Get rid of .unwrap() in update_member closures
• [x] Load from cache when possible
• [x] Update CollectionWatcher dynamically

PR Checklist:

• [x] Link to related issue #114
• [x] Add changelog entry linking to issue
• [x] Add tests
• [x] Add tests for authorization
• [x] Add tests for changing data, updating index (e.g. remove stuff)
• [x] More tests?

If I want a value from a resource in some specific type, I often have to do some explicit conversions:

// note the .to_int
let date_int = resource.get(urls::CREATED_AT)?.to_int();
// we still need to convert it
let date = new Date(date_val)

Easiest way is to implement try_into for value, so we can do:

let date: Date =  resource.get(urls::CREATED_AT)?.try_into()?;

But even better is if this could work:

let date =  resource.get::<Date>(urls::CREATED_AT)?;

I don't know if this is possible, but I've seen something similar before in serde.

The indexes are currently dealing just fine with queries that sort by property and / or value, but we don't support queries that iterate over all values (code)

I'm not sure if we should support these at all. The indexes can become absolutely huge. At this moment, we simply throw an error if a user watches such a query request.

e.g. here https://github.com/atomicdata-dev/atomic-data-rust/actions/runs/3609011868/jobs/6081946265#step:2:51

The following packages have unmet dependencies:
 libappindicator3-dev : Depends: libappindicator3-1 (= 12.10.1+20.10.20200706.1-0ubuntu1)
 libayatana-appindicator3-dev : Conflicts: libappindicator3-dev
E: Unable to correct problems, you have held broken packages.
Error: Process completed with exit code 100.

There are some systems that I'm working on / have worked on where tokens are needed:

1. Invites. An invite can be used one to many times, and it represents a right for a specific resource
2. E-mail confirmations. Sent once during registration. It confirms that one email address is correct
3. Public key reset link. Used when a user has lost their private key. This link represents a single-use token that allows adding a new public key to an Agent.

All these have some things in common:

• The token is publicly accessible - we can't use authentication to grant access
• The token must be unique. We could use URLs for this (as these constraints are already used).
• The token needs to be hard to guess.
• The tokens should not be findable through pubic collections. This means the read rights are limited to the creator, or nonexistent.
• The tokens can only be created under specific conditions. For example, users can only create an Invite for resource R if they have rights for resource R. E-mail confirmation tokens can only be created by the server - not by any user.

But there are also some differences:

• Invites can be used multiple times, and the others only once

Other thoughts and considerations:

• It is likely that we'll have different usecases for tokens in the future. For example vote tokens, credits, vouchers.
• We want to prevent that users create tokens that give rights that they cannot give. For Invites, this meant creating an Invites plugin that handles all commits done to Invite instances.

Approaches

Add some token helper functions in atomic-lib

If we continue to use Class Extender Plugins, we should make sure that most logic is shared. That means:

• Checking for existing tokens
• By default disallowing creation of tokens.

Add a token property with custom property level

• Makes it really simple to query for tokens, because we know they are generated through the plugin.
• Requires custom property-level validations, which is powerful, but potentially very costly if we allow it as a plugin abstraction. We currently only check for plugins at the class level.

Add a single /token endpoint

• Users pass a query param containing the token string (not a URL)
• Depending on the type of token, the server does what should be done
• Not sure how we need to deal with passing arguments... What if a token requires additional user input? We couldn't communicate these fields to the user. Nope, bad idea.

Protected properties

We can add an isProtected attribute to Property definitions. When this is set to true, we prevent users from editing this field. That way, we can know that nobody can create malicious invites, for example.

Signed tokens (e.g. JWTs)

Most of the current designs are stateful. They require that the server stores something and later retrieves this. This brings complexity - we need to store tokens, retrieve them, make sure they are unique, etc... Instead, we could let the server create signed tokens which contain all relevant information.

• I think all signed tokens need an expiration date
• Making them internally JSON-AD would make sense, but I also want serde compatbility. If we go for JSON-AD, then:
  • All should require a Class
  • Their @id fields are probably empty, as they are not hosted anywhere
• Tokens that are used should probably be stored, so they aren't used again. We could even store just the hash. I'm not sure when we could remove these, though. Perhaps store them indefinitely in a sled::Tree.
• JWT is the most wide used, but also criticized for its security.

In the case of an email reset, the JWT would contain the e-mail address and the Agent URL. For pubkey reset, only the Agent URL.

Invites are another story... I'm not sure if the JWT approach works there. Invites can be used multiple times. A counter has to be decremented when they are used.

Crates:

• jwt. Seems to provide everything
• jwt-simple, seems a bit more maintained and easy to use. Plays well with derive macros and serde.
• biscuit, improves on some JWT / JOSE design issues. But no JS crate (not sure if I need that though).
• branca is simple, but more constrained. No third party validation possible. I think that's fine in our use-case, though. The repo seems a bit empty (not many examaples), and it hasn't been maintained in 9 months.
• paseto, seems to fix jwt issues and has support for ed25519 (which we already use) see examples

To maintain lightweight for Atomic Server give user the opportunity to add S3 credenitals and endpoint with option to synchronise binary data with external S3 compatible servers. I would check how https://github.com/quickwit-oss/quickwit does it.

Give the user the ability to generate S3-compatible credentials: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and the ability create AWS compatible AWS_ENDPOINT to the user can populate Atomic Drive using standard AWS S3 compatible tools like rclone (or numerous UI clients).