A rust implementation of the csl-next model.

Per discussion on a CSL 1.0 test, some styles require author sort keys to be shortened as they are for display.

https://github.com/citation-style-language/test-suite/issues/60

So a vector/list of sort structs isn't enough, or the sort struct needs another parameter.

In looking through the style repo with the blunt instrument of ripgrep, here's some conclusions:

1. by far the most common primary sorting key is author, which is subject to shortening for display, and to substitution when not present
2. per the linked test, some styles want you to sort on the shortened list
3. almost without exception, substitutions for author are editor, title, translator.
4. two of those substitutions are contributor/names lists, so also subject to shortening

I've already extracted substitution and name list shortening to top-level config options.

So I think a small change like the following should work?

sort:
  shorten_author: true
  specs:
    - key: author
    - key: issuedYear
      order: descending

... or even:

sort:
  shorten_author: true
  keys:
    - author
    - issued.year-descending

Here's how biblatex does it, which is similar to my last option, but simpler, yet more options (see section 3.1.2 in general):

But note that it has minsortnames and variants, which means it's not just a boolean controlling the linked case. From the manual:

The first item considered in the sorting process is always the presort field of
the entry. If this field is undefined, biblatex will use the default value ‘mm’ as
a presort string. The next item considered is the sortkey field. If this field is
defined, it serves as the master sort key. Apart from the presort field, no further
data is considered in this case. If the sortkey field is undefined, sorting continues
with the name. The package will try using the sortname, author, editor, and
translator fields, in this order. Which fields are considered also depends on the
setting of the use<name> options. If all such options are disabled, the sortname
field is ignored as well. Note that all name fields are responsive to maxnames and
minnames. If no name field is available, either because all of them are undefined
or because all use<name> options are disabled, biblatex will fall back to the
sorttitle and title fields as a last resort. The remaining items are, in various
order: the sortyear field, if defined, or the first four digits of the year field
otherwise; the sorttitle field, if defined, or the title field otherwise; the
volume field.

So do something like they did, it might:

sort:
  rules: nty
  shorten:
    min: 4
    take: 2

E.g. would need to allow shorten in multiple places.

With #50, I somehow I broke sorting such that it reverses the order of processing.

Debugging output suggests it's correct, but the results say otherwise.

[Sort { key: Author, order: Ascending }, Sort { key: Year, order: Ascending }]
a_author: brown, david, lee, jane

Spent a long time try to fix it, but gave up for now.

I'm playing with this on the #50 branch, but it's tricky.

We need to support:

1. people and orgs
2. sorting and displaying
3. localized differences, including of 2, and of things like participles
4. independent formatting of people name parts
5. for it all ideally to be easy-to-use in the input format

Right now it's just:

author: ["Doe, Jane"]
parse: true, # default

One idea that just occurred to me, that I need think on more, is something like this:

pub enum Contributor {
    SimpleName(Vec<String>),
    StructuredName(Vec<Vec<String>>),
}

So then the first option would be as now, and assume sort and display are the same.

author: ["Mao Zedong"]

The second would add more structure; at minimum:

author: [ ["Doe", "Jane", "de"] ]

So there, the sort string would be derived by joining the items in the list, for example, and display would reorder them.

Need to pass the config options to these.

It may be it remains one parameter, but with four pieces:

• global
• citation
• bibliography
• component

... with a separate function to determine which to apply in a given context.

Way too premature, but worth thinking about now:

The most expensive operations will be sorting, grouping, etc.

But performance isn't that critical in the sort of batch-processing scenarios I tend to focus on; for example, processing markdown or tex documents from the command line.

Maybe there should be two modes, then: batch and interactive/server?

On the latter, it would be good to do some of these operations in parallel, and to perhaps cache the key data structures.

Citeproc-rs does this using rayon and salsa, so maybe look at that?

EDIT: I did add parallel sorting very easily. I need to add a benchmark so I can compare the results.

Basic rendering now works so:

1. [X] add it to CLI
2. [X] fix render_references so it returns an array of templates (also need to add a type for that)
3. [X] fix what seems to be a input date format issue (see error below)
4. [X] fill out the example style to demonstrate and test better.

Tackle these later:

4. [ ] add edtf:Date formatting
5. [ ] add disambiguation from hints
6. [ ] I think I want to remove the outer templateComponent property in the AST, but may do later

    {
      "templateComponent": {
        "contributor": "author",
        "form": "long",
        "rendering": null
      },
      "value": "United Nations"
    }

❯ target/debug/csln processor/examples/style.csl.yaml processor/examples/ex1.bib.yaml
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseError(TooShort)', bibliography/src/reference.rs:101:82
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

On EDTF, here's what the parse function returns, so will presumably need to match on this?

pub enum Edtf {
    DateTime,
    Date,
    YYear,
    Interval,
    IntervalFrom,
    IntervalTo,
}

I think I then need to use chrono to do the formatting, but I'm not sure:

https://docs.rs/chrono/latest/chrono/#formatting-and-parsing

Create three functions.

First, make_group_key:

Take a vector StyleGroupKey and concatenate the values returned from the respective field data using string_for_key.

Second, string_for_key:

When given a key return the string value from ProcReference::data.

Third, group_proc_references:

Group the vector returned by Processor::get_proc_references and add a ProcHint to each, consisting of group_index, group_length, and group_key; return a new vector of ProcReference.

Finally, add a test to processor_test.rs to confirm correct ProcHint::group_index, values using the examples/bibliography.yaml.

To make room for additional configuration options for sorting, move the template to a named template field, and add a couple of example parameters.

Refs: #64

If you're curious, @plk, here's the commit for adjusting the sort config per discussion.

sort:
  template:
    - author
    - issued-year

Much of the magic will happen with default values.

Depending on how complex it gets, I may add later a similar ability to reference pre-defined configs via a key.

sort: abc

Made progress on this, but stuck on what seems one last detail: passing the right data types to the formatting method.

I've spent hours trying to figure this out, without luck!

Also, I'm giving up this again.

No need for this here; just pass the enum option directly:

https://github.com/bdarcus/csln/blob/38645446c0ab8f5c20f5cb77a5fbebbdc44e1f0e/processor/src/lib.rs#L509-L512

An experiment ATM, that I'm merging for now. But it's not hooked up, and it needs work.

Adapt a few ideas, and some code, from Hayagriva, and separate out basic data types and formatting code from bibliography and processor.

I think ideally I want all the basic formatting of these datatypes to be here too, so the models become clean, and also easier to contribute to for non-programmers or rust devs, and it's all more modular.

For titles, adapted this from the ts model, which in turn is adapted from the CSL 1.1 branch xml schema.

export interface TitleStructured {
  full?: TitleString;
  main: TitleString;
  sub: TitleString[];
}

Still unsure on traits vs straight functions, etc. Masto response back-and-forth is very useful.

https://fosstodon.org/@digikata/110515566415939721

Here's one of the suggested examples to examine:

https://github.com/obsidiandynamics/stanza/blob/master/src/renderer.rs

And the markdown renderer:

https://github.com/obsidiandynamics/stanza/blob/master/src/renderer/markdown.rs

https://docs.rs/tabled/latest/tabled/

May want to consider this on sorting:

https://stackoverflow.com/questions/60916194/how-to-sort-a-vector-in-descending-order-in-rust/60916195#60916195

Like my fifth attempt at this ....

I've decided to add my own methods to parse edtf date-times, and also convert them to the sort of input formats needed by chrono or icu.

The first commit here adds those.

The parsing method returns an Edtf::Date regardless of whether it's valid, but if it's not, it's a dummy date with zero year.

Maybe there's a better way, but it'll do for now.

But this currently ONLY handles edtf:Date, so still work to do.

Seems the biblatex and latex world is a good place to look for how to do this.

First, this issue at their tracker:

https://github.com/plk/biblatex/issues/416

Second, this thread;:

https://tex.stackexchange.com/a/505649

It seems the minimum you need is a language field on entries.

@book{Sharoni1969,
 author = {ميخائيل، ملاك  and  الشاروني، حبيب},
 date = {1969},
 title = {المرجع فى قواعد اللغة القبطية},
 location = {الاسكندرية},
 publisher = {جمعية مارمينا العجايبي},
 langid = {arabic}
}
@book{Browning1983,
 author = {Browning, Robert},
 date = {1983},
 title = {Medieval and Modern Greek},
 publisher = {Cambridge University Press},
 langid = {english}
}

But that doesn't go far enough:

@misc{CBible2015,
 date = {2015},
 title = {\foreignlanguage{english}{Coptic Bible} الكتاب المقدس القبطي},
 langid = {arabic}
}

Rendering of that example:

So I think we also need to allow langids to be attached to alternate title fields.

See also #61

Beyond CSL, the other excellent package first released around the same time, and similarly ambitious, is biblatex.

It has struck me its design has some similarities to what I'm doing here.

Consider their long list of completely flat parameters, aka options (and see table I've attached below for how they map to scopes):

They've also been ahead of us on EDTF, and looks like already figured it out.

biblatex-options-table.pdf

I think I settled on a good foundation in #51, but left out a few things.

Maybe we should look at biblatex extended names (see p15 of manual):

@inproceedings{Hasselt2016,
  author    = {family=Hasselt, given=Hado P., prefix=van useprefix=false and Guez, Arthur and Hessel, Matteo and Mnih, Volodymyr and Silver, David},
}

Of note:

• prefix (aka CSL particle?)
• suffix
• useprefix (aka CSL non-dropping-particle?)
• can also specify initials by appending -i

See also #64.

Need to compare the csl-rs repo options side-by-side with these, and adjust as needed, add docs.

The idea is to transfer, and as needed adjust, the language from the 1.0 style spec.