Execution of and interaction with external processes and pipelines

Overview

subprocess

Build Status docs.rs

The subprocess library provides facilities for execution of and interaction with external processes and pipelines, inspired by Python's subprocess module. subprocess is hosted on crates.io, with API Documentation on docs.rs.

Features

This library is about launching external processes with optional redirection of standard input, output, and error. It covers similar ground as the std::process standard library module, but with additional functionality:

  • The communicate family of methods for deadlock-free capturing of subprocess output/error to memory, while simultaneously feeding data to its standard input. Capturing supports optional timeout and read size limit.

  • Connecting multiple commands into OS-level pipelines.

  • Flexible redirection options, such as connecting standard streams to arbitrary files, or merging output streams like shell's 2>&1 and 1>&2 operators.

  • Non-blocking and timeout methods to wait on the process: poll, wait, and wait_timeout.

The crate has minimal dependencies to third-party crates, only requiring libc on Unix and winapi on Windows. It is intended to work on Unix-like platforms as well as on reasonably recent Windows. It is regularly tested on Linux, MacOS and Windows.

API Overview

The API is separated in two parts: the low-level Popen API similar to Python's subprocess.Popen, and the higher-level API for convenient creation of commands and pipelines. The two can be mixed, so it is possible to use builder to create Popen instances, and then to continue working with them directly.

While Popen loosely follows Python's subprocess module, it is not a literal translation. Some of the changes accommodate the differences between the languages, such as the lack of default and keyword arguments in Rust, and others take advantage of Rust's more advanced type system, or of additional capabilities such as the ownership system and the Drop trait. Python's utility functions such as subprocess.run are not included because they can be better expressed using the builder pattern.

The high-level API offers an elegant process and pipeline creation interface, along with convenience methods for capturing their output and exit status.

Examples

Spawning and redirecting

Execute an command and wait for it to complete:

let exit_status = Exec::cmd("umount").arg(dirname).join()?;
assert!(exit_status.success());

To prevent quoting issues and injection attacks, subprocess will not spawn a shell unless explicitly requested. To execute a command using the OS shell, like C's system, use Exec::shell:

Exec::shell("shutdown -h now").join()?;

Start a subprocess and obtain its output as a Read trait object, like C's popen:

let stream = Exec::cmd("find /").stream_stdout()?;
// Call stream.read_to_string, construct io::BufReader(stream) and iterate it
// by lines, etc...

Capture the output of a command:

let out = Exec::cmd("ls")
  .stdout(Redirection::Pipe)
  .capture()?
  .stdout_str();

Redirect standard error to standard output, and capture them in a string:

let out_and_err = Exec::cmd("ls")
  .stdout(Redirection::Pipe)
  .stderr(Redirection::Merge)
  .capture()?
  .stdout_str();

Provide some input to the command and read its output:

let out = Exec::cmd("sort")
  .stdin("b\nc\na\n")
  .stdout(Redirection::Pipe)
  .capture()?
  .stdout_str();
assert_eq!(out, "a\nb\nc\n");

Connecting stdin to an open file would have worked as well.

Pipelines

Popen objects support connecting input and output to arbitrary open files, including other Popen objects. This can be used to form pipelines of processes. The builder API will do it automatically with the | operator on Exec objects.

Execute a pipeline and return the exit status of the last command:

let exit_status =
  (Exec::shell("ls *.bak") | Exec::cmd("xargs").arg("rm")).join()?;

Capture the pipeline's output:

let dir_checksum = {
    Exec::shell("find . -type f") | Exec::cmd("sort") | Exec::cmd("sha1sum")
}.capture()?.stdout_str();

The low-level Popen type

let mut p = Popen::create(&["command", "arg1", "arg2"], PopenConfig {
    stdout: Redirection::Pipe, ..Default::default()
})?;

// Since we requested stdout to be redirected to a pipe, the parent's
// end of the pipe is available as p.stdout.  It can either be read
// directly, or processed using the communicate() method:
let (out, err) = p.communicate(None)?;

// check if the process is still alive
if let Some(exit_status) = p.poll() {
  // the process has finished
} else {
  // it is still running, terminate it
  p.terminate()?;
}

Querying and terminating

Check whether a previously launched process is still running:

let mut p = Exec::cmd("sleep").arg("2").popen()?;
thread::sleep(Duration::new(1, 0));
if p.poll().is_none() {
    // poll() returns Some(exit_status) if the process has completed
    println!("process is still running");
}

Give the process 1 second to run, and kill it if it didn't complete by then.

let mut p = Exec::cmd("sleep").arg("2").popen()?;
if let Some(status) = p.wait_timeout(Duration::new(1, 0))? {
    println!("process finished as {:?}", status);
} else {
    p.kill()?;
    p.wait()?;
    println!("process killed");
}

License

subprocess is distributed under the terms of both the MIT license and the Apache License (Version 2.0). See LICENSE-APACHE and LICENSE-MIT for details. Contributing changes is assumed to signal agreement with these licensing terms.

Comments
  • Popen is not Sendable on Windows

    Popen is not Sendable on Windows

    I got this error on Windows:

    error[E0277]: the trait bound `*mut std::os::raw::c_void: std::marker::Send` is not satisfied in `plugin::Plugin`
      --> src\monitor.rs:80:48
       |
    80 | pub fn monitor_signal(plugins: Vec<Plugin>) -> impl Future<Item = (), Error = io::Error> + Send {
       |                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `*mut std::os::raw::c_void` cannot be sent between threads safely
       |
       = help: within `plugin::Plugin`, the trait `std::marker::Send` is not implemented for `*mut std::os::raw::c_void`
       = note: required because it appears within the type `subprocess::win32::Handle`
       = note: required because it appears within the type `subprocess::popen::os::ExtChildState`
       = note: required because it appears within the type `subprocess::popen::ChildState`
       = note: required because it appears within the type `subprocess::Popen`
       = note: required because it appears within the type `plugin::Plugin`
       = note: required because of the requirements on the impl of `std::marker::Send` for `std::ptr::Unique<plugin::Plugin>`
       = note: required because it appears within the type `alloc::raw_vec::RawVec<plugin::Plugin>`
       = note: required because it appears within the type `std::vec::Vec<plugin::Plugin>`
    

    I have a struct Plugin with a member of type subprocess::Popen.

    Can Popen be Send?

    opened by zonyitoo 12
  • Interact with subprocess from a different thread using channels

    Interact with subprocess from a different thread using channels

    I'm trying to create something that works kind of like a shell using subprocess. A user enters a command in a webview and this executes the command in a subprocess inside a new thread. I'm trying to communicate with them using channels from the main thread and send messages from there to the subprocesses. I've managed to make stdout/err work like this but I'm having trouble with stdin.

    I've tried a lot of things, here's my current attempt:

    let mut p = Exec::cmd(command)
        .args(&args)
        .cwd(current_dir)
        .stdin(Redirection::Pipe)
        .stdout(Redirection::Pipe)
        .stderr(Redirection::Pipe)
        .popen()?;
    
    let mut stdin = vec![];
    
    while p.poll().is_none() {
        // trying to write stdin (this is not working)
        p.stdin
            .as_ref()
            .unwrap()
            .write_all(&stdin)
            .expect("Failed to write stdin");
    
        let mut reader = BufReader::new(p.stdout.as_ref().unwrap());
        let mut chunk = [0u8; 64];
        loop {
            let len = reader.read(&mut chunk)?;
            if len == 0 {
                break;
            }
            let chunk = &chunk[..len];
    
            // send stdout
            let event_result = tauri::event::emit(
                webview,
                "event",
                Some(Payload {
                    id: id.clone(),
                    chunk,
                }),
            );
            if let Err(err) = event_result {
                println!("Failed to send chunk: {}", err);
            } else {
                println!("Sent chunk");
            };
        }
    
        // receive stdin message
        let received = receiver.recv();
        if let Ok(message) = received {
            if message == "Exit" {
                println!("Exiting");
                break;
            }
    
            println!("Message received: {}", message);
            stdin = message.as_bytes().to_owned();
        } else {
            println!("No message or error");
        }
    }
    

    Please let me know if it's unclear what I'm trying to do. Any help would be appreciated, and thanks for your time!

    opened by martonlanga 11
  • Quoted strings on Windows adds additional quotes

    Quoted strings on Windows adds additional quotes

    I'm noticing that subprocess is doing a bit of extra work processing the arguments, and while helpful, it seems like it's not allowing me to encode certain types of calls.

    In Windows, for example, there is a difference between:

    echo foo

    and

    echo "foo"

    The latter of these doesn't seem be expressible in subprocess. If I add an argument that is "foo", subprocess will create a second set of quotations around the first, and then escape the first. This isn't what you want. If the quotes are at the outside and not inside, I don't believe you want to escape them.

    opened by jntrnr 9
  • Cannot compile on Windows

    Cannot compile on Windows

    Compiling on windows gives a bunch of

     expected enum `libc::c_void`, found enum `winapi::ctypes::c_void`
    

    I've tried with:

    • stable-x86_64-pc-windows-gnu (1.31)
    • stable-x86_64-pc-windows-msvc (1.30)
    • nightly-x86_64-pc-windows-msvc (1.32)
    opened by yorodm 9
  • communicate multiple times

    communicate multiple times

    I believe under the current implementation there is no way to call communicate multiple times to a subprocess that terminates on EOF. It would be nice if there was some sort of mechanism by which I could write to a subprocess without sending eof, such that I could read its response and send more based on the response. I apologize if this is already possible, or if my lack of understanding of the implementation is preventing me from seeing if this is impossible.

    opened by ShaneEverittM 8
  • Popen is not Sync on windows

    Popen is not Sync on windows

    help: within `subprocess::popen::Popen`, the trait `std::marker::Sync` is not implemented for `*mut bitflags::core::ffi::c_void`
       = note: required because it appears within the type `subprocess::win32::Handle`
       = note: required because it appears within the type `subprocess::popen::os::ExtChildState`
       = note: required because it appears within the type `subprocess::popen::ChildState`
       = note: required because it appears within the type `subprocess::popen::Popen`
    

    From looking at the code in win32.rs there is:

    unsafe impl Send for Handle {}
    

    My guess is that this should be solved by adding:

    unsafe impl Sync for Handle {}
    
    opened by palaviv 7
  • command error doesnt get caputred in the stderr

    command error doesnt get caputred in the stderr

    hey. very nice crate.

    im having here a very interesting situation

    i have this code; just trying to run a command that is not found on my system using the Exec::shell and get its output and err into 2 variables:

    extern crate subprocess;
    
    use subprocess::Exec;
    use subprocess::Redirection;
    
    fn main() {
        let out_and_err = Exec::shell("not_found")
            .stdout(Redirection::Pipe)
            .stderr(Redirection::Merge)
            .capture().unwrap();
    
        let out = out_and_err.stdout_str(); 
        // here out is the error message from shell: 
        // "sh: line 1: not_found: command not found\n"
        println!("{:?}", out);
    
    
        
        let err = out_and_err.stderr_str();
        // println!("{:?}", out);
        println!("{:?}", err); // <---- here the error is empty string ""
    }
    

    when i run it i do get:

    ❱  cargo run --example sub
        Blocking waiting for file lock on build directory
       Compiling rust_core v0.0.1 (~/Alexzander__/programming/rust/core)
        Finished dev [unoptimized + debuginfo] target(s) in 1.10s
         Running `target/debug/examples/sub`
    
    "sh: line 1: not_found: command not found\n"
    ""
    

    my questions are:

    • is this the expected behavior when the shell its returning command not found error?
    • shouldnt let err = out_and_err.stderr_str(); contain the message sh: line 1: not_found: command not found ?

    because when im running not_found in my shell its exiting with code 127:

    ❱  not_found
    zsh: command not found: not_found
    
    鉶(tmux)
     [~/Alexzander__/programming/rust/core]
     git:(main[-US?--UT])
    ✘ exit:(code[127])           <--------- here
    ❱
    
    
    opened by alexzanderr 6
  • Example improvement and Exec::cmd problem on Windows

    Example improvement and Exec::cmd problem on Windows

    First an idea to improve the example which might relevant for users of this crate that take the example from examples/sample.rs. Shell builtin commands on Windows, like e.g., dir, need to always be run through subprocess::Exec::shell. I used the following code to get the example working on Windows (if you like I'll be happy to open a Pull Request):

    extern crate subprocess;
    
    use std::io::{BufRead, BufReader};
    use subprocess::Exec;
    
    fn main() {
        #[cfg(not(target_family = "windows"))]
        let x = Exec::cmd("ls").stream_stdout().unwrap();
        #[cfg(target_family = "windows")]
        // on Windows shell builtins like dir need to be run through shell not cmd
        let x = Exec::shell("dir").stream_stdout().unwrap();
    
        let br = BufReader::new(x);
        for (i, line) in br.lines().enumerate() {
            println!("{}: {}", i, line.unwrap());
        }
    }
    

    Furthermore I think there is bug on Windows. Passing a command that is built on more then one word (meaning something surrounded by whitespace), does not work if it passed to subprocess::Exec::cmd. Simple example:

    fn main() {
        let x = Exec::shell("cargo").stream_stdout().unwrap();              // works
        let x = Exec::cmd("cargo").stream_stdout().unwrap();                // works
        let x = Exec::shell("cargo --version").stream_stdout().unwrap();    // works
        let x = Exec::cmd("cargo --version").stream_stdout().unwrap();      // does NOT work
    
        let br = BufReader::new(x);
        for (i, line) in br.lines().enumerate() {
            println!("{}: {}", i, line.unwrap());
        }
    }
    

    The error message for the non working example is:

    thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: IoError(Os { code: 2, kind: NotFound, message: "..." })', src\main.rs:7:13
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    

    For the reasons mentioned in the README.md I think it would be good if subprocess::Exec::cmd works well with that type of input on Windows.

    opened by swaldhoer 6
  • How to get stdout, stderr, exit_code and timeout

    How to get stdout, stderr, exit_code and timeout

    Hey, first thanks for a great project, its exactly what I needed.

    So I'm looking to execute processes which have no stdin, but I need to capture:

    • stdout
    • stderr
    • exit_code
    • and time out a process

    I've tried a whole bunch of things. The closest I've gotten is to use the Popen and use the communicate_bytes to get the first 3 of the bullets above, but it blocks and so the wait_timeout doesn't get called.

    I looked through other issues, but may have missed it. Any ideas on how to accomplish this?

    //This should timeout after 50ms, but blocks on `communicate_bytes` so the `wait_timeout` never can check
    #[test]
    fn test_sleep_with_timeout_blocks() {
        
        let timeout = Duration::from_millis(50);
        let args = &vec!("sh", "-c", "sleep 5");
    
        let mut p = Popen::create(
            args, 
            PopenConfig {
                stdout: Redirection::Pipe,
                stderr: Redirection::Pipe,
                ..Default::default()
            }).unwrap();
    
        // This blocks, so the wait_timeout cant get hit
        // I need the stdout and stderr
        let (maybe_out, maybe_err) = p.communicate_bytes(None).unwrap();
        
        let status = p.wait_timeout(timeout).unwrap();
    }
    
    opened by gregberns 6
  • why is Capture not exposed

    why is Capture not exposed

    I found myself wanting to use value given by Capture, and found that the struct is sort of private... it's not visible from outside the crate. Is it an oversight?

    opened by tshepang 6
  • add CREATE_NO_WINDOW for gui mode

    add CREATE_NO_WINDOW for gui mode

    //it will popup a black console windows in windows #![windows_subsystem = "windows"] extern crate subprocess;

    use subprocess::{Exec,Redirection}; use std::io::{BufReader, BufRead}; use std::io::prelude::*; use std::fs::File;

    fn main() { let out = Exec::cmd("dir") .stdout(Redirection::Pipe) .capture().unwrap() .stdout_str(); println!("{}",out); let mut file = File::create("out.txt").unwrap(); file.write_all(&out.into_bytes()).unwrap(); }

    opened by woodgear 6
  • Question on struck when interacting with subprocess

    Question on struck when interacting with subprocess

    I have a shell app. Generally, I would open it in a shell, then I input a command "ucci" to it, then it feedbacks some lines of strings. And I would still interact it with other commands until I input "quit", then it would quit.

    I wrote code like this:

    use subprocess::{Popen, PopenConfig, Redirection};
    
    fn main() -> io::Result<()> {
        let cmd = "./myShellApp.exe";
    
        // test crate
        let mut p = Popen::create(
            &[cmd],
            PopenConfig {
                stdin: Redirection::Pipe,
                stdout: Redirection::Pipe,
                ..Default::default()
            },
        )
        .expect("open app failed");
    
        let (out, err) = p.communicate(Some("ucci\n"))?;
        match out {
            Some(s) => println!("{}", s),
            None => todo!(),
        }
        match err{
            Some(e) => println!("{}", e),
            None => todo!(),
        }
    

    But the code just stuck at let (out, err) = p.communicate(Some("ucci\n"))?;. It seems that it would not quit until the sub process in terminated. How should I fix it?

    opened by dbsxdbsx 2
  • Fix test that calls setenv() without synchronization

    Fix test that calls setenv() without synchronization

    This test is flaky under the following interleaving:

    • exec_to_string() grabs MUTATE_ENV.
    • env_inherit_set() calls temp_env_var() which calls env::set_var().
    • exec_to_string() executes some asserts, one of which fails because the other test messed with the environment in a way exec_to_string() did not expect.
    • exec_to_string() panics and poisons MUTATE_ENV.
    • env_inherit_set() grabs the lock and panics because it's poisoned.

    Taking the lock before setenv fixes this.

    opened by mcy 0
  • would be nice to have an api to interrupt or kill a process group

    would be nice to have an api to interrupt or kill a process group

    Today subprocess gives you a way on unix to put the child in a new process group but, I think, if you terminate or kill it you will only signal the direct child. It would be nice to also expose a safe interface to killpg.

    (It's not a big deal as you can do the equivalent through libc or nix but it might be nice.)

    opened by sourcefrog 0
  • Add Process Creation Flags.

    Add Process Creation Flags.

    Adds the ability to set the process creation flags for Windows process via param in the PopenConfig struct. This is a workaround for #40, so you can, for example, pass flag 0x08000000, which hides the CMD on Windows, or any other flag you want.

    opened by videobitva 1
Owner
Hrvoje Nikšić
Hrvoje Nikšić
EmbedAnything is a powerful python library designed to streamline the creation and management of embedding pipelines

EmbedAnything is a powerful python library designed to streamline the creation and management of embedding pipelines. Built in Rust with no heavy dependencies.

Starlight 39 May 7, 2024
Framework for large distributed pipelines

Rain Rain is an open-source distributed computational framework for processing of large-scale task-based pipelines. Rain aims to lower the entry barri

Substantic 705 Dec 27, 2022
Write CI/CD pipelines using TypeScript

Katoa Katoa is a community fork of Cidada, a tool created by Fig which was sunset in late 2023 following acquisition by AWS. This fork and the underly

Katoa 47 Oct 6, 2023
procs makes it easy to find and manage system processes

procs procs makes it easy to find and manage system processes. Right now, the main usage is finding processes by the ports it is listening on, but mor

Kurt Wolf 3 Aug 29, 2022
a Rust library for running child processes

duct.rs Duct is a library for running child processes. Duct makes it easy to build pipelines and redirect IO like a shell. At the same time, Duct help

Jack O'Connor 633 Dec 30, 2022
A command-line tool to easily kill processes running on a specified port.

killport killport is a command-line utility for killing processes listening on specific ports. It's designed to be simple, fast, and effective. The to

Francisco Jiménez Cabrera 6 Mar 29, 2023
Kill processes protected by antivirus during offensive activities.

superman Kill everything. usage Options: -p, --pid <PID> Pid to kill -r Recursive kill process -t, --time <TIME> Kill interv

B1-TEAM 96 Jun 16, 2023
A program to share a TTY like a GPS UART between 2 processes.

TTYTEE - A process that exposes 2 copies of the same TTY. The initial use case for this crate has been sharing a single GPS device talking through an

Skyways 4 Jun 25, 2023
A lightweight terminal tool to manage processes in Unix machines.

TTV v0.0.1 TTV (term-task-viewer) is a lightweight tool to view and manage active processes in Unix machines. It provides an easy interface with vim-l

Caio Ishikawa 9 Aug 29, 2023
REC2 (Rusty External Command and Control) is client and server tool allowing auditor to execute command from VirusTotal and Mastodon APIs written in Rust. 🦀

Information: REC2 is an old personal project (early 2023) that I didn't continue development on. It's part of a list of projects that helped me to lea

Quentin Texier (g0h4n) 104 Oct 7, 2023
Rust library for integrating local LLMs (with llama.cpp) and external LLM APIs.

Table of Contents About The Project Getting Started Roadmap Contributing License Contact A rust interface for the OpenAI API and Llama.cpp ./server AP

Shelby Jenkins 4 Dec 18, 2023
AUR external package builder

AUR Build Server Goal This project aims to provide an external package making server based on any PKGBUILD based project. Right now it pulls AUR packa

Seïfane Idouchach 2 Sep 11, 2022
Cucumber testing framework for Rust. Fully native, no external test runners or dependencies.

Cucumber testing framework for Rust An implementation of the Cucumber testing framework for Rust. Fully native, no external test runners or dependenci

Cucumber Rust 393 Dec 31, 2022
Open-source, external CS2 cheat.

?? ProExt - A game enhancer for CS2 ?? Features: ESP ??️ Aimbot ?? Triggerbot ?? Crosshair ⌖ Radar ?? Bomb Timer ?? Spectator List ?? Styling ??️ ...a

Vytrol 5 Nov 29, 2023
The PC-based component of a two-part Linux driver for using a TI calculator as an external keyboard

Introduction i68apollo is the computer-based component of the two-part i68 (*I*nput from Motorola *68*000[fn:4]-based calculator) prototype userspace

Joseph Burke 4 Aug 17, 2024
Poisson intensity of limit order execution, calibration of parameters A and k using level 1 tick data

Poisson intensity of limit order execution, calibration of parameters A and k using level 1 tick data Description A limit order placed at a price St ±

0xCuteSocks 6 Apr 9, 2023
Low level access to processors using the AArch64 execution state.

aarch64-cpu Low level access to processors using the AArch64 execution state. Usage Please note that for using this crate's register definitions (as p

Rust Embedded 18 Jan 5, 2023
Shared execution environment for constructing 3D virtual spaces from the inside.

Hearth Hearth is a shared, always-on execution environment for constructing 3D virtual spaces from the inside. Come join our Discord server! The Histo

null 6 Jan 31, 2023
Yet another code execution engine written in Rust.

exec Yet another blazingly fast code execution engine written in Rust. Paths GET /api/v1/status GET /api/v1/runtimes POST /api/v1/execute POST /api/v1

Stefan Asandei 2 Jul 11, 2023