A weekly dive into commonly used modules in the Rust ecosystem, with story flavor!

Scott Lyons

Last update: Aug 26, 2022

Related tags

Foreign Function Interface rust-module-of-the-week

Overview

Rust Module of the Week

A weekly dive into commonly used modules in the Rust ecosystem, with story flavor!

Build status

Release
Draft

The goal

The goal of this project is to bring the same concept as PyMOTW to the Rust world. PyMOTW was an invaluable resource for me when I was learning Python years ago, and I hope that I can help someone in a similar way. Each week we'll dive into a module and explore some of the functionality that we can find there while following along the adventures of some colourful characters.

Why the story?

I have always found that I learn better when there's a story behind something I'm learning, a purpose. I've looked at my fair share of documentation but I've found that a lot of code samples can be hard to follow without broader context. So I've decided that every issue will include the story of one or more characters and their problems that only Rust can solve.

Examples

See the examples README for how to run them.

Contributing

Feel free to fork and open Pull Requests about any past or potential future content! I would especially welcome any code feedbacks as I'm sure there'll be lots of chances to improve my code. A proper development guide is planned.

Comments

[Error] Compilation error in the first example

Hi,

First of all - great initiative.

I'm a complete Rust noob and this is exactly what I was looking for - an easily digestible, short articles on Rust stdlib.

Now, when I try to execute the first example:

use std::{fs, io};

const PHOTO_HOME: &str = "/home/user/motw/week-1/";

fn main() {
    let entries = fs::read_dir(PHOTO_HOME)?
        .map(|entry_res| entry_res.map(|entry| entry.path()))
        .collect::<Result<Vec<_>, io::Error>>()?;

    println!("{:?}", entries);
}

I get the following error:

error[E0277]: the `?` operator can only be used in a function that returns `Result` or `Option` (or another type that implements `FromResidual`)
  --> src/main.rs:6:43
   |
5  | / fn main() {
6  | |     let entries = fs::read_dir(PHOTO_HOME)?
   | |                                           ^ cannot use the `?` operator in a function that returns `()`
7  | |         .map(|entry_res| entry_res.map(|entry| entry.path()))
8  | |         .collect::<Result<Vec<_>, io::Error>>()?;
9  | |
10 | |     println!("{:?}", entries);
11 | | }
   | |_- this function should return `Result` or `Option` to accept `?`
   |
   = help: the trait `FromResidual<Result<Infallible, std::io::Error>>` is not implemented for `()`
   = note: required by `from_residual`

error[E0277]: the `?` operator can only be used in a function that returns `Result` or `Option` (or another type that implements `FromResidual`)
  --> src/main.rs:8:48
   |
5  | / fn main() {
6  | |     let entries = fs::read_dir(PHOTO_HOME)?
7  | |         .map(|entry_res| entry_res.map(|entry| entry.path()))
8  | |         .collect::<Result<Vec<_>, io::Error>>()?;
   | |                                                ^ cannot use the `?` operator in a function that returns `()`
9  | |
10 | |     println!("{:?}", entries);
11 | | }
   | |_- this function should return `Result` or `Option` to accept `?`
   |
   = help: the trait `FromResidual<Result<Infallible, std::io::Error>>` is not implemented for `()`
   = note: required by `from_residual`

Version of Rust in use is:

stable-x86_64-unknown-linux-gnu (default)
rustc 1.54.0 (a178d0322 2021-07-26)

Could you provide hints on what could be wrong in my example?

Thanks in advance!

opened by matejrisek 2

Fix clippy lints

In group_files_by_date the clone is not needed. or_insert_withis more efficient in the case where a value already exists, because the closure or function provided to it are only executed if needed, while or_insert always creates the value.

BTW keep up the good work. I really enjoyed those posts! :)

opened by Lesstat 1

Alternative implementation of visit_dirs

fn visit_dirs(dir: &Path) -> io::Result<Vec<PathBuf>> {
    let mut stack = vec![fs::read_dir(dir)?];
    let mut files = vec![];
    while let Some(dir) = stack.last_mut() {
        match dir.next().transpose()? { // r: dir.find_map(|r| r.ok())
            None => {
                stack.pop();
            }
            Some(dir) if dir.file_type().map_or(false, |t| t.is_dir()) => {
                stack.push(fs::read_dir(dir.path())?); // stack.extend(fs::read_dir(dir.path()))
            }
            Some(file) => files.push(file.path()),
        }
    }
    Ok(files)
}

Recursion in Rust isn't specially optimised, so iterative solutions are often faster (in this case 3x). I think it would be good to make a note of this.

Also, it's often better to just ignore errors returned from the ReadDir since they can have different permission flags etc. In these cases discarding the rest of the results can be overly pessimistic.

opened by Plecra 1

Wrapping up some build automation

As mentioned in #7 I wanted to have a clean build and release process. It's working for the draft branch, now it's time to bring it to the main branch before content changes/suggestions come in.

opened by slyons 0
Enable continuous testing for examples

Enable continuous testing by some means. Whether through doctests or unit tests it would behoove us to have automatic testing and use it as a barrier to any merges.

opened by slyons 0
Run code samples in CI and in the browser

#9 addresses a bug in one of the code samples. That made me thinking.

What if we the examples can be tested automatically, similar to Rust's doc tests? Than issue like the one reported in #9 can be prevented.

And what do you think about making the code sample executable in the web browser? The rust book offers that for many of it's examples.

opened by OrangeTux 1
# std::fs Part 2
Rudgal gets serious about organizing. Will have to include some sort of generating function so the moving/organizing can be demonstrated.

DirBuilder

Permissions

Copying/removing files.
opened by slyons 0

Releases(published)

v0.32.3(Dec 13, 2022)
0.32.3 (2022-12-12)

new:

N/A

changed

N/A

fixed:

Changed NMF to accept optional parameters nmf_alpha_W and nmf_alpha_H based on changes in scikit-learn==1.2.0.

Change ktrain.utils to check for TensorFlow before doing a version check, so that ktrain can be imported without TensorFlow being installed.

Source code(tar.gz)
Source code(zip)
v0.32.2(Dec 12, 2022)
0.32.2 (2022-12-12)

new:

N/A

changed

N/A

fixed:

Changed call to NMF to use alpha_W instead of alpha, as alpha parameter was removed in scikit-learn==1.2. (#470)

Source code(tar.gz)
Source code(zip)
v0.32.1(Dec 12, 2022)
0.32.1 (2022-12-11)

new:

N/A

changed

N/A

fixed:

In TensorFlow 2.11, the tf.optimizers.Optimizer base class points the new keras optimizer that seems to have problems. Users should use legacy optimizers in tf.keras.optimizers.legacy with ktrain (which evidently will never be deleted). This means that, in TF 2.11, supplying a string representation of an optimizer like "adam" to model.compile uses the new optimizer instead of the legacy optimizers. In these cases, ktrain will issue a warning and automatically recompile the model with the default tf.keras.optimizers.legacy.Adam optimizer.

Source code(tar.gz)
Source code(zip)
v0.32.0(Dec 9, 2022)
0.32.0 (2022-12-08)

new:

Support for TensorFlow 2.11. For now, as recommended in the TF release notes, ktrain has been changed to use the legacy optimizers in tf.keras.optimizers.legacy. This means that, when compiling Keras models, you should supply tf.keras.optimizers.legacy.Adam() instead of the string "adam".

Support for Python 3.10. Changed references from CountVectorizer.get_field_names to CountVectorizer.get_field_names_out. Updated supported versions in setup.py.

changed

N/A

fixed:

fixed error in docs

Source code(tar.gz)
Source code(zip)
v0.31.10(Oct 1, 2022)
0.31.10 (2022-10-01)

new:

N/A

changed

N/A

fixed:

Adjusted tika imports due to issue with /tmp/tika.log in multi-user scenario

Source code(tar.gz)
Source code(zip)
v0.31.9(Sep 24, 2022)
0.31.9 (2022-09-24)

new:

N/A

changed

N/A

fixed:

Adjustment for kwe

Fixed problem with importing ktrain without TensorFlow installed

Source code(tar.gz)
Source code(zip)
v0.31.8(Sep 8, 2022)
0.31.8 (2022-09-08)

new:

N/A

changed

N/A

fixed:

Fixed paragraph tokenization in AnswerExtractor

Source code(tar.gz)
Source code(zip)
v0.31.7(Aug 4, 2022)
0.31.7 (2022-08-04)

new:

N/A

changed

re-arranged dep warnings for TF

ktrain now pinned to transformers==4.17.0. Python 3.6 users can downgrade to transformers==4.10.3 and still use ktrain.

fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.31.6(Aug 2, 2022)
0.31.6 (2022-08-02)

new:

N/A

changed

updated dependencies to work with newer versions (but temporarily continue pinning to transformers==4.10.1)

fixed:

fixes for newer networkx

Source code(tar.gz)
Source code(zip)
v0.31.5(Aug 1, 2022)
0.31.5 (2022-08-01)

new:

N/A

changed

N/A

fixed:

fix release

Source code(tar.gz)
Source code(zip)
v0.31.4(Aug 1, 2022)
0.31.4 (2022-08-01)

new:

N/A

changed

TextPredictor.explain and ImagePredictor.explain now use a different fork of eli5: pip install https://github.com/amaiya/eli5-tf/archive/refs/heads/master.zip

fixed:

Fixed loss_fn_from_model function to work with DISABLE_V2_BEHAVIOR properly

TextPredictor.explain and ImagePredictor.explain now work with tensorflow>=2.9 and scipy>=1.9 (due to new eli5-tf fork -- see above)

Source code(tar.gz)
Source code(zip)
v0.31.3(Jul 16, 2022)
0.31.3 (2022-07-15)

new:

N/A

changed

added alnum check and period check to KeywordExtractor

fixed:

fixed bug in text.qa.core caused by previous refactoring of paragraph_tokenize and tokenize

Source code(tar.gz)
Source code(zip)
v0.31.2(May 20, 2022)
0.31.2 (2022-05-20)

new:

N/A

changed

added truncate_to argument (default:5000) and minchars argument (default:3) argument to KeywordExtractor.extract_keywords method.

added score_by argument to KeywordExtractor.extract_keywords. Default is freqpos, which means keywords are now ranked by a combination of frequency and position in document.

fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.31.1(May 17, 2022)
0.31.1 (2022-05-17)

new:

N/A

changed

Allow for returning prediction probabilities when merging tokens in sequence-tagging (PR #445)

added basic ML pipeline test to workflow using latest TensorFlow

fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.31.0(May 7, 2022)
0.31.0 (2022-05-07)

new:

The text.ner.models.sequence_tagger now supports word embeddings from non-BERT transformer models (e.g., roberta-base, codebert). Thanks to @Niekvdplas.

Custom tokenization can now be used in sequence-tagging even when using transformer word embeddings. See custom_tokenizer argument to NERPredictor.predict.

changed

[breaking change] In the text.ner.models.sequence_tagger function, the bilstm-bert model is now called bilstm-transformer and the bert_model parameter has been renamed to transformer_model.

[breaking change] The syntok package is now used as the default tokenizer for NERPredictor (sequence-tagging prediction). To use the tokenization scheme from older versions of ktrain, you can import the re and string packages and supply this function to the custom_tokenizer argument: lambda s: re.compile(f"([{string.punctuation}“”¨«»®´·º½¾¿¡§£₤‘’])").sub(r" \1 ", s).split().

Code base was reformatted using black and isort

ktrain now supports TIKA for text extraction in the text.textractor.TextExtractor package with the use_tika=True argument as default. To use the old-style text extraction based on the textract package, you can supply use_tika=False to TextExtractor.

removed warning about sentence pair classification to avoid confusion

fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.30.0(Mar 28, 2022)
0.30.0 (2022-03-28)

new:

ktrain now supports simple, fast, and robust keyphrase extraction with the ktran.text.kw.KeywordExtractor module

ktrain now only issues a warning if TensorFlow is not installed, insteading of halting and preventing further use. This means that pre-trained PyTorch models (e.g., text.zsl.ZeroShotClassifier) and sklearn models (e.g., text.eda.TopicModel) in ktrain can now be used without having TensorFlow installed.

text.qa.SimpleQA and text.qa.AnswerExtractor now both support PyTorch with optional quantization (use framework='pt' for PyTorch version)

text.zsl.ZeroShotClassifier, text.translation.Translator, and text.translation.EnglishTranslator all support a quantize argument.

pretrained image-captioning and object-detection via transformers are now supported

changed

reorganized imports

localized seqeval

The half parameter to text.translation.Translator, and text.translation.EnglishTranslator was changed to quantize and now supports both CPU and GPU.

TFDataset and SequenceDataset classes must not be imported as ktrain.dataset.TFDataset and ktrain.dataset.SequenceDataset.

fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.29.3(Mar 9, 2022)
0.29.3 (2022-03-09)

new:

NERPredictor.predict now includes a return_offsets parameter. If True, the results will include character offsets of predicted entities.

changed

In eda.TopicModel, changed lda_max_iter to max_iter and nmf_alpha to alpha

Added show_counts parameter to TopicModel.get_topics method

Changed qa.core._process_question to qa.core.process_question

In qa.core, added remove_english_stopwords and and_np parameters to process_question

The valley learning rate suggestion is now returned in learner.lr_estimate and learner.lr_plot (when suggest=True supplied to learner.lr_plot)

fixed:

save TransformerEmbedding model, tokenizer, and configuration when saving NERPredictor and reset te_model to facilitate loading NERPredictors with BERT embeddings offline (#423)

switched from keras2onnx to tf2onnx, which supports newer versions of TensorFlow

Source code(tar.gz)
Source code(zip)
v0.29.2(Feb 9, 2022)
0.29.2 (2022-02-09)

new:

N/A

changed

N/A

fixed:

added get_tokenizer call to TransformersPreprocessor._load_pretrained to address issue #416

Source code(tar.gz)
Source code(zip)
v0.29.1(Feb 8, 2022)
0.29.1 (2022-02-08)

new:

N/A

changed

pin to sklearn==0.24.2 due to breaking changes. This scikit-learn version change only really affects TextPredictor.explain. The eli5 fork supporting tf.keras updated for scikit-learn 0.24.2. To use scikit-learn==0.24.2, users must uninstall and re-install the eli5 fork with: pip install https://github.com/amaiya/eli5/archive/refs/heads/tfkeras_0_10_1.zip.

fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.29.0(Jan 29, 2022)
0.29.0 (2022-01-28)

new:

New vision models: added MobileNetV3-Small and EfficientNet. Thanks to @ilos-vigil.

changed

core.Learner.plot now supports plotting of any value that exists in the training History object (e.g., mae if previously specified as metric). Thanks to @ilos-vigil.

added raw_confidence parameter to QA.ask method to return raw confidence scores. Thanks to @ilos-vigil.

fixed:

pin to transformers==4.10.3 due to Issue #398

pin to syntok==1.3.3 due to bug with syntok==1.4.1 causing paragraph tokenization in qa module to break

properly suppress TF/CUDA warnings by default

ensure document fed to keras_bert tokenizer to avoid this issue

Source code(tar.gz)
Source code(zip)
v0.28.3(Nov 5, 2021)
0.28.3 (2021-11-05)

new:

speech transcription support

changed

N/A

fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.28.2(Oct 18, 2021)
0.28.2 (2021-10-17)

new:

N/A

changed

minor fix to installation due to pypi

fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.28.1(Oct 18, 2021)
0.28.1 (2021-10-17)

New:

N/A

Changed

added extra_requirements to setup.py

changed imports for summarization, translation, qa, and zsl in notebooks and tests

Fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.28.0(Oct 13, 2021)
0.28.0 (2021-10-13)

New:

text.AnswerExtractor is a universal information extractor powered by a Question-Answering module and capable of extracting user-specfied information from texts.

text.TextExtractor is a text extraction pipeline (e.g., convert PDFs to plain text)

Changed

changed transformers pin to transformers>=4.0.0,<=4.10.3

Fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.27.3(Sep 3, 2021)
0.27.3 (2021-09-03)

New:

N/A

Changed

-N/A

Fixed:

SimpleQA now can load PyTorch question-answering checkpoints

change API call to support newest causalnlp

Source code(tar.gz)
Source code(zip)
v0.27.2(Jul 28, 2021)
0.27.2 (2021-07-28)

New:

N/A

Changed

N/A

Fixed:

check for logits attribute when predicting using transformers

change raised Exception to warning for longer sequence lengths for transformers

Source code(tar.gz)
Source code(zip)
v0.27.1(Jul 20, 2021)
0.27.1 (2021-07-20)

New:

N/A

Changed

Added method parameter to tabular.causal_inference_model.

Fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.27.0(Jul 20, 2021)
0.27.0 (2021-07-20)

New:

Added tabular.causal_inference_model function for causal inference support.

Changed

N/A

Fixed:

N/A

Source code(tar.gz)
Source code(zip)
v0.26.5(Jul 16, 2021)
0.26.5 (2021-07-15)

New:

N/A

Changed

added query parameter to SimpleQA.ask so that an alternative query can be used to retrieve contexts from corpus

added chardet as dependency for stellargraph

Fixed:

fixed issue with TopicModel.build when threshold=None

Source code(tar.gz)
Source code(zip)
v0.26.4(Jun 23, 2021)
0.26.4 (2021-06-23)

New:

API documenation index

Changed

Added warning when a TensorFlow version of selected transformers model is not available and the PyTorch version is being downloaded and converted instead using from_pt=True.

Fixed:

Fixed utils.metrics_from_model to support alternative metrics

Check for AUC ktrain.utils "inspect" function

Source code(tar.gz)
Source code(zip)

Owner

Scott Lyons

GitHub

Rust Python modules for interacting with Metaplex's NFT standard.

Simple Metaplex Metadata Decoder Install the correct Python wheel for your Python version with pip: pip install metaplex_decoder-0.1.0-cp39-cp39-manyl

11 Mar 31, 2022

Facilitating high-level interactions between Wasm modules and JavaScript

wasm-bindgen Facilitating high-level interactions between Wasm modules and JavaScript. Guide | API Docs | Contributing | Chat Built with ?? ?? by The

5.9k Jan 8, 2023

Instrument and transform wasm modules.

wasm-instrument A Rust library containing a collection of wasm module instrumentations and transformations mainly useful for wasm based block chains a

31 Dec 16, 2022

Zinnia is a runtime for Filecoin Station modules. It provides a sandboxed environment to execute untrusted code on consumer-grade computers.

?? Zinnia Zinnia is a runtime for Filecoin Station modules. It provides a sandboxed environment to execute untrusted code on consumer-grade computers.

5 Jan 25, 2023

A programming environment that aims to help people learn how to program in JavaScript, while giving them a tour on how old computers and their limitations used to be.

This repository is for the new under renovation rewrite of the LIKO-12 project. The legacy version with the original stars and contributions is still

1k Jan 5, 2023

WASM-compiled built-in actors used by all Filecoin clients

Built-in Filecoin actors (v8) This repo contains the code for the on-chain built-in actors that power the Filecoin network starting from network versi

45 Jan 4, 2023

Rust library for build scripts to compile C/C++ code into a Rust library

A library to compile C/C++/assembly into a Rust library/application.

1.3k Dec 21, 2022

Minimal framework to inject the .NET Runtime into a process in Rust

錆の核 sabinokaku Minimal framework to inject the .NET Runtime into a process. Supports Windows and Linux. macOS support is complicated due to SIP, and w

8 Mar 6, 2022

Turns running Rust code into a serializable data structure.

WasmBox WasmBox turns running Rust code into a serializable data structure. It does this by compiling it to WebAssembly and running it in a sandbox. T

12 Dec 7, 2022

📦 Pack hundreds of Garry's Mod Lua files into just a handful

?? gluapack gluapack is a program that can pack hundreds of Garry's Mod Lua files into just a handful. Features Quick, easy and portable - perfect for

11 Aug 10, 2021

Inline CSS into style attributes

css-inline A crate for inlining CSS into HTML documents. It is built with Mozilla's Servo project components. When you send HTML emails, you need to u

122 Dec 30, 2022

A small application to convert a 3D model in .obj format into HTML and CSS

obj-to-html is a small application that converts a 3D model in .obj format into HTML and CSS that will display that model in a web browser, spinning a

7 Dec 30, 2022

Slitter is a C- and Rust-callable slab allocator implemented primarily in Rust, with some C for performance or to avoid unstable Rust features.

Slitter is a less footgunny slab allocator Slitter is a classically structured thread-caching slab allocator that's meant to help write reliable long-

133 Dec 5, 2022

A weekly dive into commonly used modules in the Rust ecosystem, with story flavor!

Related tags

Overview

Build status

The goal

Why the story?

Examples

Contributing

Comments

Releases(published)

v0.32.3(Dec 13, 2022)

0.32.3 (2022-12-12)

new:

changed

fixed:

v0.32.2(Dec 12, 2022)

0.32.2 (2022-12-12)

new:

changed

fixed:

v0.32.1(Dec 12, 2022)

0.32.1 (2022-12-11)

new:

changed

fixed:

v0.32.0(Dec 9, 2022)

0.32.0 (2022-12-08)

new:

changed

fixed:

v0.31.10(Oct 1, 2022)

0.31.10 (2022-10-01)

new:

changed

fixed:

v0.31.9(Sep 24, 2022)

0.31.9 (2022-09-24)

new:

changed

fixed:

v0.31.8(Sep 8, 2022)

0.31.8 (2022-09-08)

new:

changed

fixed:

v0.31.7(Aug 4, 2022)

0.31.7 (2022-08-04)

new:

changed

fixed:

v0.31.6(Aug 2, 2022)

0.31.6 (2022-08-02)

new:

changed

fixed:

v0.31.5(Aug 1, 2022)

0.31.5 (2022-08-01)

new:

changed

fixed:

v0.31.4(Aug 1, 2022)

0.31.4 (2022-08-01)

new:

changed

fixed:

v0.31.3(Jul 16, 2022)

0.31.3 (2022-07-15)

new:

changed

fixed:

v0.31.2(May 20, 2022)

0.31.2 (2022-05-20)

new:

changed

fixed:

v0.31.1(May 17, 2022)

0.31.1 (2022-05-17)

new:

changed

fixed: