Using engines running outside of the browser

Overview

External engine (ALPHA)

Using engines running outside of the browser on https://lichess.org/analysis.

Work in progress 🔧 🔨

  • Implement MVP
  • Complete lichess-org/lila#10867
  • Implement safe-uci adapter
  • Build easily installable local providers

Official providers

remote-uci

Reference implementation in Rust. Cross platform command line application wrapping an UCI engine. Secure, but not robust against denial of service.

Minimal GUIs with bundled Stockfish for Linux, Windows and Mac

Planned

Protocol (still subject to change)

Overview

Lichess provides a reference implementation for an external engine provider. Third parties can also implement their own engine providers.

An external engine provider is a WebSocket server. To inform the client about the connection details, it triggers a navigation to an authorization endpoint, where the user can confirm that their client should use the given engine provider. The client will then open a WebSocket connection for each session with a chess engine.

The client sends UCI commands as text messages over the WebSocket connection. Each command is sent in its own WebSocket message, containing no line feeds or carriage returns.

The provider responds as if the client were exclusively communicating with a UCI engine, by sending UCI commands as individual WebSocket messages. copyprotection and registration are not supported.

Important considerations for providers

The most straight-forward implementation would be to forward all WebSocket messages to a UCI engine as a thin proxy. However, some important considerations arise that require dealing with UCI specifics and tracking the engine state.

  • ⚠️ With many engines, a malicious user who can execute arbitrary commands will be able to damage the host system, cause data loss, exfiltrate data, or even achieve arbitrary code execution.

    Recommendation: Use the safe-uci adapter as a wrapper around UCI engines. If possible, bind the server only on the loopback interface to limit the attack surface. Generate a strong secret for the engine registration and do not forget to check it.

  • Analysis is resource intensive. Be sure to put limits on CPU and memory usage and inforce them, in order for your system to stay responsive.

  • Network connections can be interrupted.

    Recommendation: Send pings over all WebSocket connections at intervals. If a client times out or disconnects, stop ongoing searches in order to prevent deep or infinite analysis from consuming resources indefinitely.

  • Clients may open multiple connections.

    Recommendation: Manage shared access to a single engine process. At each point, one of the WebSocket connections has an exclusive session with the engine. Track the engine state and options associated with each session.

    When receiving a message (except stop) on a connection, an exclusive session is requested for that connection. In order to switch sessions, end any ongoing search in the previous session (by injecting stop) and wait until any outstanding engine output has been delivered. Then issue ucinewgame, to ensure the following session is clean, and reapply any options associated with the session.

Register external engine

To inform the client about connection details, trigger a navigation to

https://lichess.org/analysis/external

with the following query parameters:

name default example description
url required ws://localhost:9670/ URL of the provider server. External engine registrations are stored in local storage, so this may refer to localhost without breaking on other devices.
secret required A secret token that the client should include in every connection request.
name required Stockfish 15 Short engine or provider name to show on the client.
maxThreads 1 8 Maximum number of threads supported for setoption name Threads .... Make sure to respect limits of the engine as well as the machine.
maxHash 16 1024 Maximum number of memory supported for setoption name Hash ... (MiB). Make sure to respect limits of the engine as well as the machine.
variants chess,atomic Comma-separated list of variants supported by setoption name UCI_Variant ..., if any.

Accepting connections

The client will open WebSocket connections to the url as provided in the registration above. It will set the following additional query parameters:

name description
secret The secret token as provided in the registration above. The provider must check and reject connection attempts if the token does not match.
session Each new tab or session will have a different identifier. Reconnections will reuse the identifier.

Engine requirements

To properly work on the Lichess analysis board, engines must support:

  • UCI_Chess960
  • MultiPV
  • info with
    • depth (reaching 6 must be fast)
    • multipv
    • score
    • nodes (with order of magnitude comparable to Stockfish)
    • time
    • pv
Comments
  • Why use ws?

    Why use ws?

    Hey guys,

    great that you implement external engine use! I wished that for lichess just yesterday. I wanted to ask why a ws is necessary, should not just a regular rest-adapter (in form of a additional layer before UCI) have done the job?

    Greetings, Niclas

    opened by NiclasSchwalbe 5
  • No authorization dialog showing

    No authorization dialog showing

    Hey, I have followed your installation instructions for Ubuntu, however when I open the link there is no dialog asking for authorization. Are there any settings in Lichess that need to be changed? I am using Firefox 105.0 together with Ubuntu 20.04.5 if that helps, remote-uci version is 1.0.0.

    Running the tool from the command line produces the following output:

    ~ remote-uci --engine stockfish        
    [2022-10-11T12:09:55Z INFO ] Starting engine "stockfish" ...
    [2022-10-11T12:09:55Z INFO ] 0 << uci
    [2022-10-11T12:09:56Z WARN ] 0 >> Stockfish 15 by the Stockfish developers (see AUTHORS file)
    [2022-10-11T12:09:56Z INFO ] 0 >> id name Stockfish 15
    [2022-10-11T12:09:56Z INFO ] 0 >> id author the Stockfish developers (see AUTHORS file)
    [2022-10-11T12:09:56Z WARN ] 0 >> 
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Debug Log File type string default 
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Threads type spin default 1 min 1 max 512
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Hash type spin default 16 min 1 max 33554432
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Clear Hash type button
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Ponder type check default false
    [2022-10-11T12:09:56Z INFO ] 0 >> option name MultiPV type spin default 1 min 1 max 500
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Skill Level type spin default 20 min 0 max 20
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Move Overhead type spin default 10 min 0 max 5000
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Slow Mover type spin default 100 min 10 max 1000
    [2022-10-11T12:09:56Z INFO ] 0 >> option name nodestime type spin default 0 min 0 max 10000
    [2022-10-11T12:09:56Z INFO ] 0 >> option name UCI_Chess960 type check default false
    [2022-10-11T12:09:56Z INFO ] 0 >> option name UCI_AnalyseMode type check default false
    [2022-10-11T12:09:56Z INFO ] 0 >> option name UCI_LimitStrength type check default false
    [2022-10-11T12:09:56Z INFO ] 0 >> option name UCI_Elo type spin default 1350 min 1350 max 2850
    [2022-10-11T12:09:56Z INFO ] 0 >> option name UCI_ShowWDL type check default false
    [2022-10-11T12:09:56Z INFO ] 0 >> option name SyzygyPath type string default <empty>
    [2022-10-11T12:09:56Z INFO ] 0 >> option name SyzygyProbeDepth type spin default 1 min 1 max 100
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Syzygy50MoveRule type check default true
    [2022-10-11T12:09:56Z INFO ] 0 >> option name SyzygyProbeLimit type spin default 7 min 0 max 7
    [2022-10-11T12:09:56Z INFO ] 0 >> option name Use NNUE type check default true
    [2022-10-11T12:09:56Z INFO ] 0 >> option name EvalFile type string default nn-6877cd24400e.nnue
    [2022-10-11T12:09:56Z INFO ] 0 >> uciok
    https://lichess.org/analysis/external?url=ws%3A%2F%2F127.0.0.1%3A9670%2Fsocket&secret=f628a6940430f4e1ca89c1643f617b06&name=Stockfish+15&maxThreads=12&maxHash=8192
    
    
    opened by MushroomMaula 4
  • External Engine Not Working on Arch Linux (Stuck on Loading)

    External Engine Not Working on Arch Linux (Stuck on Loading)

    Hi, I'm trying to run this but it gets stuck. The command I'm using is:

    remote-uci stockfish

    When I click the link in the terminal, I see this: image

    After I authorize and try to open one of my games, it gets stuck like this and never evaluates: image

    Any suggestions for fixes? This is my log:

    [2022-08-19T15:46:23Z WARN ] 0 >> Stockfish 15 by the Stockfish developers (see AUTHORS file)
    [2022-08-19T15:46:24Z INFO ] 0 >> id name Stockfish 15
    [2022-08-19T15:46:24Z INFO ] 0 >> id author the Stockfish developers (see AUTHORS file)
    [2022-08-19T15:46:24Z WARN ] 0 >> 
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Debug Log File type string default 
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Threads type spin default 1 min 1 max 512
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Hash type spin default 16 min 1 max 33554432
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Clear Hash type button
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Ponder type check default false
    [2022-08-19T15:46:24Z INFO ] 0 >> option name MultiPV type spin default 1 min 1 max 500
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Skill Level type spin default 20 min 0 max 20
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Move Overhead type spin default 10 min 0 max 5000
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Slow Mover type spin default 100 min 10 max 1000
    [2022-08-19T15:46:24Z INFO ] 0 >> option name nodestime type spin default 0 min 0 max 10000
    [2022-08-19T15:46:24Z INFO ] 0 >> option name UCI_Chess960 type check default false
    [2022-08-19T15:46:24Z INFO ] 0 >> option name UCI_AnalyseMode type check default false
    [2022-08-19T15:46:24Z INFO ] 0 >> option name UCI_LimitStrength type check default false
    [2022-08-19T15:46:24Z INFO ] 0 >> option name UCI_Elo type spin default 1350 min 1350 max 2850
    [2022-08-19T15:46:24Z INFO ] 0 >> option name UCI_ShowWDL type check default false
    [2022-08-19T15:46:24Z INFO ] 0 >> option name SyzygyPath type string default <empty>
    [2022-08-19T15:46:24Z INFO ] 0 >> option name SyzygyProbeDepth type spin default 1 min 1 max 100
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Syzygy50MoveRule type check default true
    [2022-08-19T15:46:24Z INFO ] 0 >> option name SyzygyProbeLimit type spin default 7 min 0 max 7
    [2022-08-19T15:46:24Z INFO ] 0 >> option name Use NNUE type check default true
    [2022-08-19T15:46:24Z INFO ] 0 >> option name EvalFile type string default nn-6877cd24400e.nnue
    [2022-08-19T15:46:24Z INFO ] 0 >> uciok
    
    bug 
    opened by agenttux 4
  • Error running `cargo run stockfish` within the `remote-uci` directory

    Error running `cargo run stockfish` within the `remote-uci` directory

    Whene executing cargo run stockfish within the remote-uci directory and RUST_BACKTRACE=full I get the following warnings and errors

    warning: unused import: `ProtocolError`
     --> src/engine.rs:9:18
      |
    9 | use crate::uci::{ProtocolError, UciIn, UciOption, UciOptionName, UciOut};
      |                  ^^^^^^^^^^^^^
      |
      = note: `#[warn(unused_imports)]` on by default
    
    warning: field is never read: `variant`
      --> src/engine.rs:18:5
       |
    18 |     variant: Variant,
       |     ^^^^^^^^^^^^^^^^
       |
       = note: `#[warn(dead_code)]` on by default
    
    warning: field is never read: `pos`
      --> src/engine.rs:19:5
       |
    19 |     pos: VariantPosition,
       |     ^^^^^^^^^^^^^^^^^^^^
    
    warning: `remote-uci` (lib) generated 3 warnings
        Finished release [optimized] target(s) in 30.60s
         Running `target/release/remote-uci stockfish`
    thread 'main' panicked at 'spawn engine: Os { code: 2, kind: NotFound, message: "No such file or directory" }', /Users/rkosko/Repos/external-engine/remote-uci/src/lib.rs:87:56
    stack backtrace:
       0:        0x1024537b0 - std::backtrace_rs::backtrace::libunwind::trace::hdeda4600dd070bcc
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
       1:        0x1024537b0 - std::backtrace_rs::backtrace::trace_unsynchronized::hc8148713b4dab4e7
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
       2:        0x1024537b0 - std::sys_common::backtrace::_print_fmt::haf0555383873bc65
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:66:5
       3:        0x1024537b0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h1fed2dd6a97f3bed
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:45:22
       4:        0x10246f958 - core::fmt::write::hb0e4304f3c645902
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/fmt/mod.rs:1196:17
       5:        0x10244e81c - std::io::Write::write_fmt::h5d1524415c342f9b
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/io/mod.rs:1654:15
       6:        0x102455060 - std::sys_common::backtrace::_print::h048609092df7cfde
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:48:5
       7:        0x102455060 - std::sys_common::backtrace::print::h4efb3f2f5d6d6b21
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:35:9
       8:        0x102455060 - std::panicking::default_hook::{{closure}}::ha90997e26eb3c9d7
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:295:22
       9:        0x102454d3c - std::panicking::default_hook::hb89340eaa54105a5
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:314:9
      10:        0x102455538 - std::panicking::rust_panic_with_hook::h1db23d4e5c129198
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:698:17
      11:        0x10245542c - std::panicking::begin_panic_handler::{{closure}}::h18e2e1878436e49e
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:588:13
      12:        0x102453c8c - std::sys_common::backtrace::__rust_end_short_backtrace::h146455e36557ba62
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:138:18
      13:        0x102455184 - rust_begin_unwind
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:584:5
      14:        0x10248a374 - core::panicking::panic_fmt::hb6f3c1bd9d609721
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panicking.rs:142:14
      15:        0x10248a418 - core::result::unwrap_failed::h2b21ee7518c0dc35
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/result.rs:1785:5
      16:        0x1022cbc3c - remote_uci::main::{{closure}}::h0e8cd6be5c0e29e3
      17:        0x1022e21a4 - tokio::park::thread::CachedParkThread::block_on::h1831e8f8b715e241
      18:        0x1022e23f8 - tokio::runtime::thread_pool::ThreadPool::block_on::h8fdb18da2558717c
      19:        0x1022ff5c4 - tokio::runtime::Runtime::block_on::h0bcab5cb5c7c396f
      20:        0x1022f4510 - remote_uci::main::h05678b5d29931c09
      21:        0x1022daf8c - std::sys_common::backtrace::__rust_begin_short_backtrace::h6c75583e673ad67b
      22:        0x1022ef044 - std::rt::lang_start::{{closure}}::ha82807876cc03ad9
      23:        0x102449fd0 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hfc87a1b486ea6796
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/ops/function.rs:280:13
      24:        0x102449fd0 - std::panicking::try::do_call::h7dcb1e4efbfac5dc
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:492:40
      25:        0x102449fd0 - std::panicking::try::h4ee76056fd63c1cd
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:456:19
      26:        0x102449fd0 - std::panic::catch_unwind::hbd5e8ad01dd6a9ee
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panic.rs:137:14
      27:        0x102449fd0 - std::rt::lang_start_internal::{{closure}}::h67dd3aa83aa43833
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/rt.rs:128:48
      28:        0x102449fd0 - std::panicking::try::do_call::h5c2aca003eb2a1cb
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:492:40
      29:        0x102449fd0 - std::panicking::try::h22495619ddde9ed3
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:456:19
      30:        0x102449fd0 - std::panic::catch_unwind::h9c037ef19fbc6aba
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panic.rs:137:14
      31:        0x102449fd0 - std::rt::lang_start_internal::h609ce79df789091f
                                   at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/rt.rs:128:20
      32:        0x1022f45f0 - _main
    

    I wasn't sure if I need to install the stockfish engine separately because it looks to be included in the src. Also I know this is alpha so just was dropping a note if you haven't seen this yet :)

    opened by koskorya 3
  • Facilitate specifying a custom External URL

    Facilitate specifying a custom External URL

    When navigating to http://<external-engine>:<port>, the url query parameter generated and used in the redirect relies on the value supplied by the --bind <BIND> option to the remote-uci command.

    In order to facilitate binding the service to Localhost and utilizing a reverse-proxy for handling incoming connections, an option to specify a custom address to be used in the generated ws:// address would be a welcome addition.

    Example Scenario

    I have deployed the external-engine docker image on a server using the equivalent to the command below. This command essentially provides access to the remote-uci api through a pre-configured Traefik reverse-proxy using the host name stockfish.example.com.

    docker run -it --rm \
        --label "traefik.enable=true" \
        --label "traefik.http.routers.stockfish.rule=Host(`stockfish.example.com`)"
        --label "traefik.http.services.stockfish.loadbalancer.server.port=9670" \
        --label "traefik.http.routers.stockfish.tls.certresolver=pdns" \
        -p 127.0.0.1:9670:9670 \
        external-engine \
            /usr/bin/remote-uci --bind 0.0.0.0:9670 --engine stockfish
    

    When navigating to https://stockfish.example.com using the above deployment, the redirect generated contains a url query parameter value that does not work (url=ws://0.0.0.0:9670/socket)

    https://lichess.org/analysis/external?url=ws%3A%2F%2F127.0.0.1%3A9670%2Fsocket&secret=<SHHHiTSaSecret>&name=Stockfish+15&maxThreads=8&maxHash=32768
    

    Proposed Solution

    Simply adding an optional --external-address command line option that is used to populate the URL template is likely sufficient for 99% of use-cases. An argument could be made for allowing more granular control by having the option to separately set the host and port separately.

    I haven't used Rust much in the past but I may take a stab at implementing this feature as an excuse to poke the language a bit.

    opened by RickyGrassmuck 2
  • Fix #18 by moving pipes to end of line instead of beginning

    Fix #18 by moving pipes to end of line instead of beginning

    At first I thought a file was missing from the repo but it turns out to be a powershell syntax error. I fixed it by moving the pipes in Deps.ps1 to the end of the line instead of the beginning.

    opened by jonbbbb 1
  • Authorization url uses wrong ip address

    Authorization url uses wrong ip address

    After running the docker script on Windows, the url it produces is: https://lichess.org/analysis/external?url=ws%3A%2F%2F0.0.0.0%3A9670%2Fsocket&secret=redacted&name=Stockfish+15&maxThreads=8&maxHash=8192

    The 0.0.0.0 value doesn't work. Manually changing it to 127.0.0.1 fixes it.

    opened by jonbbbb 1
  • Windows instructions not quite working

    Windows instructions not quite working

    After I run RunDocker.ps1 here's what I get:

    PS C:\Users\jonbbbb\dev\external-engine> .\RunDocker.ps1
    At C:\Users\jonbbbb\dev\external-engine\Deps.ps1:6 char:1
    + | Where-Object { -Not (Test-Path $(Split-Path $_ -Leaf)) }
    + ~
    An empty pipe element is not allowed.
    At C:\Users\jonbbbb\dev\external-engine\Deps.ps1:7 char:1
    + | ForEach-Object { Invoke-WebRequest $_ -OutFile $(Split-Path $_ -Lea ...
    + ~
    An empty pipe element is not allowed.
        + CategoryInfo          : ParserError: (:) [], ParseException
        + FullyQualifiedErrorId : EmptyPipeElement
    
    [+] Building 0.9s (12/25)
     => [internal] load build definition from Dockerfile                                                               0.0s
     => => transferring dockerfile: 32B                                                                                0.0s
     => [internal] load .dockerignore                                                                                  0.0s
     => => transferring context: 2B                                                                                    0.0s
     => [internal] load metadata for docker.io/library/rust:1.62.0-slim                                                0.3s
     => [internal] load metadata for docker.io/library/debian:bullseye-slim                                            0.3s
     => [internal] load build context                                                                                  0.0s
     => => transferring context: 4.28kB                                                                                0.0s
     => [stockfish 1/8] FROM docker.io/library/debian:bullseye-slim@sha256:5cf1d98cd0805951484f33b34c1ab25aac7007bb41  0.0s
     => CANCELED [remote-uci 1/7] FROM docker.io/library/rust:1.62.0-slim@sha256:8f2e8dc0537dd0e5f0d407ab2c7df2174019  0.5s
     => => resolve docker.io/library/rust:1.62.0-slim@sha256:8f2e8dc0537dd0e5f0d407ab2c7df217401974df95bfee5f95ba1135  0.0s
     => => sha256:8f2e8dc0537dd0e5f0d407ab2c7df217401974df95bfee5f95ba1135a8232fc6 984B / 984B                         0.0s
     => => sha256:91868727ce5f5fdf67966d42f1825af003c2d9fda1066aa161d8eabbe8d4061b 742B / 742B                         0.0s
     => => sha256:278f9d1dd2ba1d3c70802ba1cc4c6bf60de2bbac30283e33877ba401ef92523a 4.85kB / 4.85kB                     0.0s
     => => sha256:461246efe0a75316d99afdbf348f7063b57b0caeee8daab775f1f08152ea36f4 3.15MB / 31.37MB                    0.5s
     => => sha256:efc70690774ac945d5fab061fde25a741314641c293612ee3b1b0b156bb60540 3.15MB / 208.94MB                   0.5s
     => CACHED [stage-3 2/6] RUN apt-get update && apt-get install -y openssl                                          0.0s
     => CACHED [stockfish 2/8] RUN apt-get update && apt-get install -y xz-utils make                                  0.0s
     => CACHED [stockfish 3/8] WORKDIR /stockfish                                                                      0.0s
     => CACHED [stockfish 4/8] COPY stockfish .                                                                        0.0s
     => ERROR [stockfish 5/8] RUN cd vendor &&     sha256sum -c SHA256SUM &&     tar xf sde-external-9.0.0-2021-11-07  0.5s
    ------
     > [stockfish 5/8] RUN cd vendor &&     sha256sum -c SHA256SUM &&     tar xf sde-external-9.0.0-2021-11-07-lin.tar.xz &&     tar xf x86_64-linux-musl-native.tgz &&     mv nn-6877cd24400e.nnue Stockfish/src:
    #11 0.442 sde-external-9.0.0-2021-11-07-lin.tar.xz: FAILED open or read
    #11 0.442 x86_64-linux-musl-native.tgz: FAILED open or read
    #11 0.442 sha256sum: sde-external-9.0.0-2021-11-07-lin.tar.xz: No such file or directory
    #11 0.442 sha256sum: x86_64-linux-musl-native.tgz: No such file or directory
    #11 0.442 sha256sum: nn-6877cd24400e.nnue: No such file or directory
    #11 0.443 nn-6877cd24400e.nnue: FAILED open or read
    #11 0.443 sha256sum: WARNING: 3 listed files could not be read
    ------
    executor failed running [/bin/sh -c cd vendor &&     sha256sum -c SHA256SUM &&     tar xf sde-external-9.0.0-2021-11-07-lin.tar.xz &&     tar xf x86_64-linux-musl-native.tgz &&     mv nn-6877cd24400e.nnue Stockfish/src]: exit code: 1
    Unable to find image 'external-engine:latest' locally
    docker: Error response from daemon: pull access denied for external-engine, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
    See 'docker run --help'.
    PS C:\Users\jonbbbb\dev\external-engine>
    
    

    Looks like downloads.txt is missing from the repository?

    opened by jonbbbb 1
  • Stockfish eating up RAM after being used

    Stockfish eating up RAM after being used

    I have remote-uci and stockfish installed from AUR to my laptop (because non-external engine on the analysis board only detects 1 out of my 12 CPU cores). This installation includes a systemd service for the remote uci (and a socket too). After having used https://lichess.org/analysis with the external engine, stockfish ends up hanging in the background with all the memory that has been allocated to it.

    I have to sudo systemctl stop remote-uci.service or sudo pkill stockfish if I want to get my RAM back (or, alternatively, go back to the analysis board, and turn engine's memory slider all the way down to 16MB or so).

    I have tried different approaches to make the engine stop on its own: closing the tab instead of closing the browser, going to another lichess page, refreshing, turning evaluation off, disconnecting from the remote engine. Nothing seems to help.

    I'm not sure whether this is actually an issue or it's just me using the software in an unintended way. Also not sure whether this should go to lila issues or here (and whether the inaccurate detection for the number of CPU cores is a lila issue or just a browser limitation).

    opened by SandaruKasa 1
  • Loop sending

    Loop sending "isready" and "stop" commands when using Lc0

    Setup

    I installed external-engine via the debsource. I modified the config to use lc0 as engine. Lc0 uses the CPU network file, with standard configuration.

    Problem

    After searching on a position, it continues sending isready and stop commands. It filled my log files (total 40 Go) and broke my root partition.

    bug 
    opened by Virinas-code 1
  • Proposal: Docker image build optimizations

    Proposal: Docker image build optimizations

    Hello!

    While working on the pull requests submitted recently I found that building the docker image to test changes took up quite a bit of time mainly due to having to build the stockfish binaries.

    I ended up making some changes locally that actually made a pretty big difference in both the time it took to build things and also resulted in much smaller image sizes.

    Changes

    1. Split stockfish off into it's own separate docker image since it doesn't need to be rebuilt (this saves a good amount of time on the initial build) and based the final image on a scratch container that just contains the stockfish binaries and .deb
    2. Refactored the external-engine image build to simply copy over the stockfish binaries from it's new image.
    3. Switched to using a distroless image for the final layer in external-engine.

    Image Size Improvements

    Switching over to the distroless image for external-engine resulted in a significant reduction in the size of the final image.

    The current docker image that builds both stockfish and external-engine ends comes in around 170Mb.

    ❯ docker image ls | grep 'fat'
    external-engine                     fat-build         b51f1903898c   37 seconds ago      169MB
    

    Switching to the distroless image with just the binaries from external-engine and stockfish cuts that size almost in half.

    Note: the distroless-slim image below only has 1 of the 8 stockfish binaries so i'm just adding the two image sizes together for a rough estimate

    ❯ docker image ls | grep 'distroless-slim\|lichess-org/stockfish'
    lichess-org/stockfish               15                88cf4948be4d   38 minutes ago      60.9MB
    external-engine                     distroless-slim   e8c28ac942eb   23 hours ago        30.1MB
    

    Hosted Images

    My thought process around splitting off the stockfish container was that since it's an external project that doesn't change often, you can remove building it from the external-engine build process by building and pushing it to the Github container registry. That way you are only ever rebuilding the external-engine project itself resulting in significantly faster turnaround time when using a docker based development workflow.

    And since the Github build pipeline is already building a container image for external-engine, throwing it up on Githubs container registry will allow users like myself who will only ever deploy it with docker do so without having to build the image at deploy time.

    If this is something you'd consider pulling in just let me know and i'll cleanup what I have and submit a PR.

    opened by RickyGrassmuck 1
  • More Explanation for Windows

    More Explanation for Windows

    I had some troubles making it run for Windows. For example, python does not like folders with whitespaces like MY NAME FOLDER. Secondly, even more advanced users may not know they have to run pip install. Furthermore, adding enviroment variables in Windows sucks and I think it is more convenient to use --token

    opened by NiclasSchwalbe 0
  • Scale endgame default depth

    Scale endgame default depth

    Possibly this may involve a larger conversation with Lichess about how analysis is done, but in endgames (or shallow positions in general which search a low node count per depth) it may be desirable to search deeper than 25 plies without resorting to go infinite.

    opened by ddugovic 2
Owner
Lichess
The free and open source chess server
Lichess
Web Browser Engineering

This is a port of Web Browser Engineering series from Python to Rust done by Korean Rust User Group.

한국 러스트 사용자 그룹 36 Dec 12, 2022
Play Onitama in your browser, compete with friends or lose to a bot

Play Onitama in your browser, compete with friends or lose to a bot

Jack Adamson 52 Nov 12, 2022
secmem-proc is a crate designed to harden a process against low-privileged attackers running on the same system trying to obtain secret memory contents of the current process.

secmem-proc is a crate designed to harden a process against low-privileged attackers running on the same system trying to obtain secret memory contents of the current process. More specifically, the crate disables core dumps and tries to disable tracing on unix-like OSes.

null 3 Dec 19, 2022
A thread pool for running multiple tasks on a configurable group of threads.

Threadfin A thread pool for running multiple tasks on a configurable group of threads. Extra features: Dynamic pool size based on load Support for asy

Stephen M. Coakley 46 Dec 5, 2022
dm-jitaux is a Rust-based JIT compiler using modified auxtools, dmasm and Inkwell LLVM wrapper for boosting Byond DM performance without any hassle!

dm-jitaux is a Rust-based JIT compiler using modified auxtools, dmasm and Inkwell LLVM wrapper for boosting Byond DM performance without any hassle (such as rewriting/refactroing your DM code).

SS220 20 Dec 13, 2022
A Simple, But amazing telegram bot, Made using the Rust language!

Telegram bot in Rust A fun Telegram bot made using Rust language.

Deep Alchemy 2 Dec 21, 2021
Rust bindings to Cloudflare Worker KV Stores using wasm-bindgen and js-sys.

worker-kv Rust bindings to Cloudflare Worker KV Stores using wasm-bindgen and js-sys

Zeb Piasecki 39 Dec 4, 2022
Showing how to deploy a Terra smart contract using Chainlink Data Feeds

Chainlink Terra Developing Requirements This demo requires the following components: Rust: rustup with cargo 1.44.1+. rustc and cargo 1.44.1+. Install

SmartContract 6 Aug 22, 2022
Private swaps for Secret Network using a private entropy pool & differential privacy.

WIP SecretSwap: Anon Edition Private swaps for Secret Network! Uses private entropy pool for differential privacy when reporting pools sizes. Swap amo

Enigma 5 Apr 5, 2022
A template for kick starting a Cloudflare worker project using workers-rs.

Getting Started A template for kick starting a Cloudflare worker project using workers-rs. This template is designed for compiling Rust to WebAssembly

Abid Omar 1 Oct 13, 2021
twilight-interactions is a set of macros and utilities to work with Discord Interactions using twilight.

Twilight interactions twilight-interactions is a set of macros and utilities to work with Discord Interactions using twilight. Note: This crate is not

null 24 Dec 26, 2022
Modrinth API is a simple library for using, you guessed it, the Modrinth API in Rust projects

Modrinth API is a simple library for using, you guessed it, the Modrinth API in Rust projects. It uses reqwest as its HTTP(S) client and deserialises responses to typed structs using serde.

null 21 Jan 1, 2023
A translation of Brendan Galea's Vulkan tutorial into Rust using the ash crate

Rust Light Vulkan Engine This is a translation of Brendan Galea's Vulkan tutorial into rust using the Ash crate. Original tutorial: Brendan Galea's Yo

Mot 6 Dec 25, 2022
Some tools for streaming frames to rpi-rgb-led-matrix using ZeroMQ, written in Rust.

led_matrix_zmq Some tools for streaming frames to rpi-rgb-led-matrix using ZeroMQ, written in Rust. This repository includes: Rust client and server l

Dan 2 Sep 6, 2022
Cross-platform GUI written in Rust using ADB to debloat non-rooted android devices

Cross-platform GUI written in Rust using ADB to debloat non-rooted android devices. Improve your privacy, the security and battery life of your device.

w1nst0n 6.8k Jan 2, 2023
This contract implements simple vote for the best coffe in indonesia using near protocol.

vote-coffe-near Description This contract implements simple vote for the best coffe in indonesia using near protocol. Contract in contract/src/lib.rs

RickhySis25 1 Nov 15, 2021
Telegram bot for searching in Arch User Repository ( AUR ); Implemented using rust.

AurSearchBot A Telegram Inline Search Bot Written in Rust Introduction Telegram Bot that can search AUR ( Arch User Repository ) in inline mode. This

AlenPaulVarghese 3 Feb 15, 2022
A custom frontend for the TradeOgre exchange, written using Iced

Forest A custom frontend for the TradeOgre exchange, written using Iced. Why TradeOgre? It's a simple, non-KYC exchange that I found out about recentl

null 3 Aug 24, 2022
Simple async library for triggering IFTTT events using webhooks.

IFTTT Webhook A simple Rust async library for triggering IFTTT events using webhooks. Installation Installation can be performed using cargo add: carg

Leo Dutra 1 Mar 11, 2021