A more detailed comparision with Gemini's Gemtext could be helpful. I found this so far:

| | subtext | gemtext | |-|--------|----| | text blocks | any line not starting with a sigil | any line not starting with #, >, => or * | blank lines | ignored | are valid text blocks | links | & link | => link optional-link-name | heading blocks | # heading | #, ## or ### followed by the heading | list blocks | - item | * item | quote blocks | > text | > text | space after sigil | required | optional | other sigils | reserved | become part of text blocks | preformatted | not supported | toggled with ``` | line breaks | \n but \r\n is also also supported | \r\n but \n is also also supported

I'm currently reworking some of the Rust subtext parser's AST representation to line it up more closely to the spec. I'm trying to reason about how to treat leading whitespace. This is important in the context of https://github.com/subconsciousnetwork/subtext/issues/31; if leading whitespace has presentational significance for a list block (it doesn't yet, but it might in the future), then some amount of leading whitespace ought to be captured context for that block. This may be true for other blocks even if that leading whitespace isn't used for presentational purposes in those cases.

At any rate, right now the Rust parser considers any leading whitespace is to be part of a Blank that precedes the content. But, this actually violates the intention of the spec in its own way, I think.

Would you be willing to clarify in the spec the way such leading whitespace that precedes a sigil on a given line should be treated?

Does \ at the start of a line escape the character, so it can be used at the start of a text block?:

\#
\&
\-
\\

If so, does it also act as escape character elsewhere like in Markdown?

Just adding this as a possible alternative to header keyvalue pairs, as discussed in #19.

That link mentions @ as a possible sigil, ie

@ foo: bar

Some alternatives -

.foo bar .foo=bar .foo: bar

I like this syntax as it's like property assignment in oo, and @ seems more appropriate for other uses.

For my project Neomem, I had been planning to just parse out any plain 'key: value' lines and treat them as metadata. Each item has a text representation similar to markdown, corresponding to a record in a database.

There's also 'key:: value' as in obsidian - I don't like that syntax though.

Which character(s) are considered a line break? LF? Also CRLF? All linebreaks defined by Unicode? I'd not include additional Unciode line breaks such as U+2029 so you can still use them for linebreaks within a block (if you your editor supports entering it instead of a normal line break)

This issue exists to document background and decisions w/r/t machine headers in Subtext. Related to #19. No action is necessary.

Background

Key-value metadata is a generally useful primitive (see If headers did not exist it would be necessary to invent them). In our design discussions, we broadly identified two kinds of key-value metadata:

• Machine metadata (tracked here in #41). this includes things like cache-control headers, content-type flags, user agent strings, etc. This kind of metadata is often invisible to the user and authored by the program.
• User metadata (tracked in #19). Anything authored by the user for their own purposes.

Both of these use-cases can technically be supported by the same mechanism. However, machine headers are often visually noisy and potentially confusing. We believe it is valuable to have separate mechanisms for these two features so that the mess of machine headers can be hidden from the user.

Design goals

• Should be expressive enough that an app can reify state as metadata, such that any internal caches, indexes, or databases can be regenerated from files.
• Should provide a content-type mechanism
• Ideally, plain text body content should be readable as plain text

Prior art and notes

HTTP-like

Packets, HTTP, and email all use the memo pattern for key-value metadata.

A document begins with a series of key: value lines, followed by an empty line. Lines after the first empty line are considered content.

Title: Floop
Date: 2022-01-15
exotic-header: {"msg": "you can put anything in header body"}

Content.

This design has been very successful in the wild, and our sense is that our design solution should be heavily influenced by this pattern.

Note: earlier version of HTTP allowed for multi-line headers through line folding, then later obsoleted this feature. We wish to avoid line folding in Subtext. See #40.

Observations:

• Pro: It's what you would write anyway
• Pro: Header keys are case-insensitive, so you can write them nice style
• Pro: It works for email and the web
• Pro: Header bodies are specified separately from header syntax. This makes headers a completely open-ended extension mechanism.
• Pro: different applications can write to the same file without conflicts provided they use their own header fields (e.g. subconscious-meta, obsidian-meta)
• Pro: Happens to fit with the way SQL thinks about JSON. Blobs of JSON belong to a specific column.
• Con: Trends toward bespoke DSLs for each header.
  • Pro: This is also a pro, because it allows for open-ended evolution.
  • Synthesis: we could strongly encourage all headers to be one of a few types: String, Number, Bool, JSON. At the same time, it would be technically possible to do things beyond these types.
• Con: Requires at least one header, or a line break at the beginning of the file

Sidecar files

Another option is to make metadata a separate file that references the content file. In this case, the obvious choice for metadata format would be JSON. Like the envelope format approach, no specification would be necessary for Subtext to support this.

Tradeoffs:

• Pro: Multiple apps can generate sidecar files without conflicts
• Con: they don't travel together with the file (not the biggest deal)

Prior art:

• iTunes used to (still?) generates its plist data store by crawling your music library.
• You can delete your plist, and the application will regenerate it.

Reference:

• https://en.wikipedia.org/wiki/Sidecar_file

This issue exists to document background research on header line folding, related to #19. No action is necessary.

Line folding

Email and HTTP header syntaxes are line-oriented, where each line is a header, and key and value are separated by :. Headers in email also support "line folding". Line folding means multi-line headers are allowed as long as the following line is preceded by at least one space.

Folded-Header: some value
  which continues on next line

HTTP also used to support line folding, but this was obsoleted in https://www.rfc-editor.org/rfc/rfc7230#section-3.2.4.

Benefits/costs of line folding

Line folding has some useful aspects.

The absolute best thing about YAML is the support for indented multi-line strings - so useful for embedding other languages inside of a YAML document, e.g. the giant hunk of JavaScript in my shots.yml file here (https://twitter.com/simonw/status/1501762626941947909)

However it comes with the cost of making parsing more complex. If parsers do not implement it, or implement it incorrectly, you end up with broken or malformed headers. This ended up happening in practice with HTTP, leading to the obsolescence of line-folding (see below).

Why was line folding obsoleted in http headers?

Some popular servers did not implement line-folding, or implemented it incorrectly, leading to complexities and issues. This practical issue lead to the obsoletion of line folding. Simplification of the spec was also cited as a reason.

Snipping a few relevant passages from the email archives and specs:

On 23 Aug 2015, at 5:44 am, Mike Bishop [email protected] wrote:

We’ve encountered a site (whom we shall leave nameless, unless they choose otherwise) that included a 0x0A20202020202020 sequence inside an HPACK value –carriage return followed by several spaces, which is to say line-folding. Edge rejects that (as will IIS if clients send it), because that’s an HTTP/1.1 legacy artifact. RFC 7540 just references RFC 7230 section 3.2 for header values, which in turn says in section 3.2.4:

Historically, HTTP header field values could be extended over multiple lines by preceding each extra line with at least one space or horizontal tab (obs-fold). This specification deprecates such line folding except within the message/http media type (Section 8.3.1). A sender MUST NOT generate a message that includes line folding (i.e., that has any field-value that contains a match to the obs-fold rule) unless the message is intended for packaging within the message/http media type.

While there’s no requirement for HTTP/1.1 clients to reject this pattern (for back-compat), it seems like an HTTP/2 implementation might want to hold its peers to that MUST NOT. Osama has additionally pointed out that being tolerant of this could make HTTP/1.1 to HTTP/2 conversion “wreak havoc and bugs.” In an offline thread with a few others, the general feeling seems to be that we should all change to reject this as broken header framing.

(Quoted in https://lists.w3.org/Archives/Public/ietf-http-wg/2015JulSep/0249.html)

Many implementations do not implement header line folding; should it be deprecated / removed? (https://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i77)

We deprecated folding, but couldn't require implementations to reject it, because (as you've seen) it still happens in the wild. (https://lists.w3.org/Archives/Public/ietf-http-wg/2015JulSep/0250.html)

At what level of the application should IP anonymity be built, or is it out of scope for this project? Use case: A neighborhood community of marginalized people needs to communicate privately in order to plan a protest and ensure that authorities cannot trace a social graph of IP connections to find the leader and pre-empt the protest.

Rust Subtext

This change proposes a prototype Subtext parser written in Rust.

Currently it is implemented as a basic recursive descent parser. No sophisticated matching (e.g., regex) and almost no backtracking is used in the implementation. This simplicity is afforded us by Subtext's relative simple grammar. However, many edge cases may yet be unrecognized at this time.

Included is a small, not-comprehensive suite of tests that cover various Subtext block formats.

Additionally, there is an example that prints out the AST of a given input Subtext file.

The parser has only one dependency: a Servo crate that makes it more ergonomic to exercise zero-copy string sharing.

This proposal does a radical rethink of Subtext link syntax, proposing to replace the current block links with inline links that are much simpler.

The gist:

• Links are wrapped in angle brackets <https://example.com>, and can appear anywhere in text.
• You can also just paste bare links https://example.com, and Subtext will automatically link them. This only works for http and https, not exotic protocols like <ipfs://asdfasdfasdfasdf>. Use angle brackets for the funky stuff.
• This is a /slashlink it's a shortcut for linking to internal pages. Slashlinks can also include paths, /like/this.

/slashlinks are like the #hashtag of links. A simple syntax for referencing things. Spiritually similar to the original camelcase WikiLink. The simplest thing that could possibly work.

Rendered: https://github.com/gordonbrander/subtext/blob/slashlink/rfcs/2021-10-07-slashlinks.md

This PR also introduces a user guide, and rewrites the specification to reflect the proposed changes.

I’m realizing that since #14 requires links to start with ./, ../, /, or scheme: it is possible to detect link blocks by sniffing the line with a regular expression.

We introduced this constraint to disambiguate wiki links from protocol links in #14. The disambiguation opens up an additional approach that was not obvious at the time of the RFC:

• Remove sigil from link blocks, restrict them to path and URL, sniff out block type with regular expression.
• Introduce a separate wiki link block.

This is very tempting because writing it looks like what you might do absent of any formal syntax. There are tradeoffs though. (TODO: Elaborate on these…)

This is a JavaScript parser for Subtext with streaming capability. Since I am using something similar for my own notes app, I thought I might as well provide a parser for this repo.

It uses the Web Streams API which means it can be used with compatible browsers and server runtimes like Node.js and Deno. The parser does not have any dependency.

I have included the example from the Rust parser and some tests from the Rust test suite.

You can run the example via npm run example and run tests via npm run test.

Feel free to suggest improvements or close the request if it does not match your requirements. If you prefer TypeScript, I could also make the adjustments.

I was wondering how one might go about writing an outline in subtext, or writing one with an editor that supported transclusions. You guys have probably already thought this through, but I've been wondering how I was gonna do it in Neomem also.

So, this is a bit pedantic, but it helped me understand the issue better.

This is a solved problem with things like Roam and Logseq, which both use lines as blocks, as I understand it, and have custom editors. I haven't tried Roam, and didn't stay long with Logseq.

Alternatives:

• Embedded outlines - could use indentation as in Python or Markdown. Space as a sigil character? This goes against the idea of a flat list of blocks. But it's lightweight, and simple for a text editor to work with.

• Transcluded items - each item could be a separate memo, and each parent embeds references to its children. This seems rigid though - like a strict folder hierarchy.

• Store link to parent in each child item. Also rigid.

• Graph of items - each item in the outline could be a separate memo, and links between them are stored in an edge table. This seems heavyweight for just a simple outline, but preserves the idea of a flat list of blocks. Different orderings are possible in the same db.

So if we want a set of notes that could be arranged in any linear order or hierarchy, storing them in a graph makes sense.

How would you transform a text outline like this to a graph version?

• home
  • ladder99 a data pipeline
    • task1
    • task2
  • neomem a data gateway
    • task3
    • task4
      • task5
      • task6

Flatten everything - each item to a node - eg with addr:contents for shorthand -

home ladder99: a data pipeline task1 task2 neomem: a data gateway task3 task4 task5 task6

and construct the edges between them - eg from, relation, to, props - which could accommodate ordered lists, trees, graphs, and have multiple hierarchies possible.

So say we have that list of nodes, and a list of edges like this -

home, parent, -null-, {} ladder99, parent, home, {order:1} task1, parent, ladder99, {order:1} task2, parent, ladder99, {order:2} neomem, parent, home, {order:2} task3, parent, neomem, {order:1} task4, parent, neomem, {order:2} task5, parent, task4, {order:1} task6, parent, task4, {order:2}

Now we've got a graph storing an ordered tree. If we need to rearrange things it's a matter of changing the numbers - floating point can go pretty far before needing to renumber a set of siblings.

You can define multiple trees/orderings on the same items, by using a different relation address, eg parent2, container, part-of, etc.

q. How would an editor render this?

It could traverse the graph structure, being wary of cycles. eg start at null - who are the children of null? home. who are the children of home? ladder99, neomem. It could do depth first traversal and add divs to a doc, with indentation. Use the order in the props set to order siblings.

q. How would an editor let you edit this tree/outline?

Say we have a rendering of the tree with each item having a text input for heading and a textarea for contents. Then changing the values is easy. Can reorder things by shortcut keys or mouse and updating order values. And can switch between different orderings by selecting the relation name/addr. It's easy to show a flat list of all the items, or any number of trees, or graphs.

For Neomem I had gotten as far as a flat list of items connected to a db - catch the up/down arrows when you're at the start/end of the textarea to move to prev/next item. Hadn't thought through outlines/graphs yet.

I haven't read enough about Subconscious to know how this all connects with the rest of the system. For Neomem each item would have a uuid, but you guys have content-addressing in mind? With some kind of address/alias system?

I'm still trying to catch up!

Anyway, as far as Subtext goes, I guess this doesn't have much bearing on it, other than that transcluding items is one way to make an outline...

Just an idea, could comment out a line/block by prefixing with ~. This would roughly correspond to a possible strikethrough representation.

Why comment out lines/blocks?

I often want a way to do this quickly in notes, to keep around things I want to disregard, or color differently, eg grayed out.

In orgmode can do with #, which matches shell and python, but conflicts with the heading sigil.

In markdown, have to use html comments <!-- foo -->, which is obviously very awkward.

Alternatives - ; as in lisp, ' as in basic, ! as in fortran (nah).

It would be nice to be able to link directly to a CID from Subtext. There are a few reasons why one might want to do this, but perhaps the most relevant one to Subtext is to enable a user to capture a "permalink" to a piece of content.

Note that when linking to a CID directly, peer key / pet name is no longer needed to resolve to the content ID. So, it's possible to imagine linking by CID as a stand-alone link (with trade-offs at the margins).

Some ideas for what this could look like:

• A stand-alone link type with some distinguishing sigil. Ex: %bafyrei...
• A specialization of a slashlink. Ex: /cid:bafyrei...
• Support for ipfs:// hyperlinks. Ex: ipfs://bafyrei...

Umbrella for:

• [x] Bold/italic #20 TODO: write up RFC
• [x] Wikilinks #29 TODO: write up RFC
• [ ] Hashtags #21
• [ ] Deep lists (implemented as line-oriented list plus whitespace "indent", similar in concept to heading levels)
• [ ] Deep headings

I am ambivalent about this, but the value would be in trying to hit a MAYA (most advanced yet acceptable) threshold of "good enough" surface-level compatibility with the unofficial Markdown flavors used in some wikis and TfTs, while retaining the advantage of having a programmatically useful flat-list block model.

Goals:

• Surface-level MAYA compatibility with some major markup (not all) features used in Markdown-flavored wikis and TfTs.
• "Mostly working" for the simplest markdown/wikilink markup cases. This will be a subjective criteria.

Non-goals:

• Compatibility. It is a non-goal to implement 1:1 feature compatibility with Markdown or Obsidian
• Strict subsetting. It is a non-goal to implement strict subset compatibility. This would be a "mostly works due to passing resemblance" kind of thing. The value is in the familiarity, not in 1:1 semantic equivalency.

Subtext deliberately takes a fundamentally different, line-oriented parsing strategy that enables you to treat lines as a flat list of blocks. This means that while it has a passing resemblance to Markdown, it will always be subtly different from Markdown at a deeper level (by design!)

Tradeoffs:

• Pro: conformability with existing infrastructure
• Pro: known/proven set of features (although I think many of these, such as deep headings are long-tail)
• Con: complicates Subtext
• Con: not a strict subset. There will be edge-cases.
• Con: adds an additional linking flavor

Related to #19.

Subtext should have a canonical JSON serialization. This would make it conformable to many other systems, and give parsers a default input/output interchange format.