tiktoken is a fast BPE tokeniser for use with OpenAI's models.
const tiktoken = require('tiktoken-node')
let enc = tiktoken.getEncoding("gpt2")
console.assert(enc.decode(enc.encode("hello world")) == "hello world")
// To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encodingForModel("text-davinci-003")
The open source version of tiktoken-node can be installed from npm:
npm install tiktoken-node
Include pre-built binary for tiktoken on Apple Silicon based macOS.
Should fix #2
Also opened up a PR here in case you want to use the PR started by @aarohmankad.
I didn't include a version update when I committed #5 because I wasn't sure of your have a release strategy for this library. I figured I'd create this PR for you if you'd like to just merge this and release it as v0.0.5. Otherwise, feel free to close this.
Thanks for putting this library together!
My code...
const tiktoken = require('tiktoken-node')
let enc = tiktoken.getEncoding("gpt2")
console.assert(enc.decode(enc.encode("hello world")) == "hello world")
My error...
error - Error: Cannot find module './dist/tiktoken-node.darwin-arm64.node'
Require stack:
- /Users/danielrasmuson/code/humata/app/node_modules/tiktoken-node/index.cjs
- /Users/danielrasmuson/code/humata/app/.next/server/pages/api/hello.js
- /Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/next-server.js
- /Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/dev/next-dev-server.js
- /Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/next.js
- /Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/lib/start-server.js
- /Users/danielrasmuson/code/humata/app/node_modules/next/dist/cli/next-dev.js
- /Users/danielrasmuson/code/humata/app/node_modules/next/dist/lib/commands.js
- /Users/danielrasmuson/code/humata/app/node_modules/next/dist/bin/next
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:985:15)
at Function.mod._resolveFilename (/Users/danielrasmuson/code/humata/app/node_modules/next/dist/build/webpack/require-hook.js:23:32)
at Function.Module._load (node:internal/modules/cjs/loader:833:27)
at Module.require (node:internal/modules/cjs/loader:1057:19)
at require (node:internal/modules/cjs/helpers:103:18)
at Object.<anonymous> (/Users/danielrasmuson/code/humata/app/node_modules/tiktoken-node/index.cjs:13:25)
at Module._compile (node:internal/modules/cjs/loader:1155:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10)
at Module.load (node:internal/modules/cjs/loader:1033:32)
at Function.Module._load (node:internal/modules/cjs/loader:868:12)
at Module.require (node:internal/modules/cjs/loader:1057:19)
at require (node:internal/modules/cjs/helpers:103:18)
at Object.tiktoken-node (/Users/danielrasmuson/code/humata/app/.next/server/pages/api/hello.js:22:18)
at __webpack_require__ (/Users/danielrasmuson/code/humata/app/.next/server/webpack-api-runtime.js:33:42)
at eval (webpack-internal:///(api)/./src/pages/api/hello.ts:5:18)
at Object.(api)/./src/pages/api/hello.ts (/Users/danielrasmuson/code/humata/app/.next/server/pages/api/hello.js:32:1)
at __webpack_require__ (/Users/danielrasmuson/code/humata/app/.next/server/webpack-api-runtime.js:33:42)
at __webpack_exec__ (/Users/danielrasmuson/code/humata/app/.next/server/pages/api/hello.js:42:39)
at /Users/danielrasmuson/code/humata/app/.next/server/pages/api/hello.js:43:28
at Object.<anonymous> (/Users/danielrasmuson/code/humata/app/.next/server/pages/api/hello.js:46:3)
at Module._compile (node:internal/modules/cjs/loader:1155:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10)
at Module.load (node:internal/modules/cjs/loader:1033:32)
at Function.Module._load (node:internal/modules/cjs/loader:868:12)
at Module.require (node:internal/modules/cjs/loader:1057:19)
at require (node:internal/modules/cjs/helpers:103:18)
at DevServer.runApi (/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/next-server.js:481:34)
at DevServer.handleApiRequest (/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/next-server.js:827:21)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Object.fn (/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/next-server.js:751:37)
at async Router.execute (/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/router.js:253:36)
at async DevServer.run (/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/base-server.js:384:29)
at async DevServer.run (/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/dev/next-dev-server.js:743:20)
at async DevServer.handleRequest (/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/base-server.js:322:20) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'/Users/danielrasmuson/code/humata/app/node_modules/tiktoken-node/index.cjs',
'/Users/danielrasmuson/code/humata/app/.next/server/pages/api/hello.js',
'/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/next-server.js',
'/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/dev/next-dev-server.js',
'/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/next.js',
'/Users/danielrasmuson/code/humata/app/node_modules/next/dist/server/lib/start-server.js',
'/Users/danielrasmuson/code/humata/app/node_modules/next/dist/cli/next-dev.js',
'/Users/danielrasmuson/code/humata/app/node_modules/next/dist/lib/commands.js',
'/Users/danielrasmuson/code/humata/app/node_modules/next/dist/bin/next'
]
}
Any suggestions for debugging?
Seems like there's some files missing in npm package for 0.0.3
"errorMessage": "Cannot find module '.../node_modules/tiktoken-node/index.cjs'",
Seems like index.cjs and index.mps are missing
I am trying to load it in deno with the help of esm.sh but it is giving error:
error: Uncaught Error: createRequire only supports 'file://' URLs for the 'filename' parameter. Received 'https://esm.sh/v111/[email protected]/deno/tiktoken-node.js'
throw new Error(
^
at createRequire (https://deno.land/[email protected]/node/module.ts:690:17)
at https://esm.sh/v111/[email protected]/deno/tiktoken-node.js:2:185
import statement
import tiktoken from "https://esm.sh/[email protected]";
Node-Rust Bridge Simple rust and node.js script to achieve a bridge between them. Only 1 bridge can be initialized per rust program. But node.js can h
ldk-node-sample Sample lightning node command-line app built on top of Ldk Node (similar to ldk-sample ). Installation git clone https://github.com/op
Polkadot Implementation of a https://polkadot.network node in Rust based on the Substrate framework. NOTE: In 2018, we split our implementation of "Po
⋰·⋰ Feeless What is Feeless? Feeless is a Nano cryptocurrency node, wallet, tools, and Rust crate. This is not the official project for Nano, only an
Statemint Implementation of Statemint, a blockchain to support generic assets in the Polkadot and Kusama networks. Statemint will allow users to: Depl
Basilisk node Local Development Follow these steps to prepare a local Substrate development environment ??️ Simple Setup Install all the required depe
dprint-node A node API for the dprint TypeScript and JavaScript code formatter. It's written in Rust for blazing fast speed. Usage Pass a file path an
Anmol Substrate Node ?? ??️ ?? Anmol is the First Cross-Chain NFT Toolkit, on Polkadot. Introducing: Moulds NFT Breeding Multi-Chain NFT Migration ink
Bovine Manage single node Rancher clusters with a single binary, bovine. % bovine run Pulling [rancher/rancher:latest], this may take awhile... Ranche
substrate-contracts-node This repository contains Substrate's node-template configured to include Substrate's pallet-contracts ‒ a smart contract modu
TRINCI TRINCI Blockchain Node. Requirements The required dependencies to build correctly the project are the following: clang libclang-dev (ver. 11 su
Substrate Node Template Generator A tool to generate stand-alone node templates of a customized Substrate clients used in "Substrate Library Extension
Subsocial Node by DappForce Subsocial is a set of Substrate pallets with web UI that allows anyone to launch their own decentralized censorship-resist
This repository contains the Rust implementation of Aleph Zero blockchain node based on the Substrate framework. Aleph Zero is an open-source layer 1
Ecoball Node is the Official Rust implementation of the Ecoball protocol. It is a fork of OpenEthereum - https://github.com/openethereum/
Webb DKG ??️ The Webb DKG ??✈️ ⚠️ Beta Software ⚠️ Running the DKG Currently the easiest way to run the DKG is to use a 3-node local testnet using dk
Substrate PoA A sample Substrate runtime for a PoA blockchain that allows: Dynamically add/remove authorities. Automatically remove authorities when t
key-node-list Doubly-linked list that stores key-node pairs. KeyNodeList is a doubly-linked list, it uses a hash map to maintain correspondence betwee
Contents Contents About Beta Releases Getting Started Known Issues Future Work Documentation Security License About Zebra is the Zcash Foundation's in
5 Apr 30, 2023
3 Nov 21, 2023
6.5k Jan 6, 2023
127 Dec 5, 2022
52 Dec 27, 2022
431 Dec 24, 2022
12 Aug 28, 2022
51 Feb 17, 2022
5 Sep 26, 2022
74 Nov 24, 2022
55 Dec 15, 2022
2 Jun 9, 2022
42 Dec 19, 2022
10 Jun 16, 2022
3 May 4, 2022
311 Dec 28, 2022