Beanstalk is a simple, fast work queue.

Overview

Build Status codecov

beanstalkd

Simple and fast general purpose work queue.

https://beanstalkd.github.io/

See doc/protocol.txt for details of the network protocol.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See CodeOfConduct.txt for details.

Quick Start

$ make
$ ./beanstalkd

also try,

$ ./beanstalkd -h
$ ./beanstalkd -VVV
$ make CFLAGS=-O2
$ make CC=clang
$ make check
$ make install
$ make install PREFIX=/usr

Requires Linux (2.6.17 or later), Mac OS X, FreeBSD, or Illumos.

Currently beanstalkd is tested with GCC and clang, but it should work with any compiler that supports C99.

Uses ronn to generate the manual. See http://github.com/rtomayko/ronn.

Subdirectories

  • adm - files useful for system administrators
  • ct - testing tool; vendored from https://github.com/kr/ct
  • doc - documentation
  • pkg - scripts to make releases

Tests

Unit tests are in test*.c. See https://github.com/kr/ct for information on how to write them.

Comments
  • AF_UNIX support for beanstalkd

    AF_UNIX support for beanstalkd

    Add the posibility of accepting connections from a local socket (bound to a filesystem path).

    To minimize portability issues, socket paths longer than 100 bytes are rejected. Systemd support has been added too, but not tested as I don't have access to any Fedora or similar.

    The behaviour of the new command line switch may be strange, but it has been implemented in a way to avoid script breakage when this option is not used.

    The automatic socket file replacement may be a bit surprising too. However, I found that implementing an automatic removal/cleanup on exit was a bit more intrusive.

    Any feedback will be appreciated.

    opened by etanol 42
  • create a 1.11 release to include the commits since 1.10

    create a 1.11 release to include the commits since 1.10

    Hi @JensRantil I wondered if it was possible to create a 1.11 release of beanstalkd, to include the commits already in master since 1.10.

    https://github.com/beanstalkd/beanstalkd/compare/v1.10...master

    NeedsFix 
    opened by tomponline 39
  • Beanstalk not purging old binlogs?

    Beanstalk not purging old binlogs?

    I know this was previously an issue, that was supposed to be fixed in 1.5, However, we're still seeing this happen.

    We're currently using beanstalk 1.10, and noticed that our binlog directory has grown to over 265GB in usage.

    When looking at beanstalk stats, we have roughly 10K jobs in a delayed state. Even with our maximum job size of 10MB (99.9% of jobs are going to be less than 1 or 2MB each), that only puts us at 100GB of storage.

    It appears as though jobs that have been purged, are not being removed from disk.

    Is there a safe way to clean up after this? How can we prevent it from happening?

    EDITING THIS FOR SOLUTION

    The solution to this was to update to beanstalkd 1.11. This seems to have resolved it for us.

    opened by laszlof 31
  • heap invariant compromised in connsched

    heap invariant compromised in connsched

    Function connsched was modifying field tickat while a Conn was still inside the heap. Wacky hijinks ensued.

    Original report follows.


    Reported in version 1.5:

    We are running beanstalkd on Ubuntu. Yesterday we received alerts saying the beanstalkd process stopped and our queues began backing up. This is the only error message I could find. Any idea what it means?

    May 13 11:44:39 bns6 kernel: [4658098.511165] beanstalkd[16793] general protection ip:40978b sp:7fff307194c0 error:0 in beanstalkd[400000+12000]
    

    https://groups.google.com/d/topic/beanstalk-talk/3uRiRBonVuE

    opened by kr 29
  • Beanstalk does not free memory

    Beanstalk does not free memory

    Hi,

    We're using beanstalk in a high-throughput application, in which tens if not hundreds of jobs per second are processed in multiple queues in beanstalkd.

    Inevitably, the beanstalkd process takes over the entire memory of the server it's running on. The rate at which this happens varies, it can take days or, just like now, half an hour.

    This looks an awful lot like issue #54, only that's closed now. Since this behaviour is affecting our application, I would like to readdress the issue.

    I was wondering, has any headway been made with issue #54? Perhaps you can advise me as to how to avoid the problem. Please tell what what other info I can supply.

    Thanks,

    Wim.

    opened by wimrijnders 23
  • core dump with beanstalkd 1.8

    core dump with beanstalkd 1.8

    Tom Guevin [email protected] wrote:

    We recently upgraded to version 1.8 from 1.4.3 and saw this core file under a high load situation:

    # /tmp/gdb /usr/bin/beanstalkd /var/core/12791.beanstalkd.core
    GNU gdb (GDB) 7.1-ubuntu
    Copyright (C) 2010 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "i486-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>...
    Reading symbols from /usr/bin/beanstalkd...(no debugging symbols found)...done.
    [New Thread 12791]
    
    warning: Can't read pathname for load map: Input/output error.
    Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no debugging symbols found)...done.
    Loaded symbols for /lib/tls/i686/cmov/libc.so.6
    Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
    Loaded symbols for /lib/ld-linux.so.2
    Core was generated by `/usr/bin/beanstalkd'.
    Program terminated with signal 11, Segmentation fault.
    #0  0x080519d2 in h_conn ()
    (gdb) bt
    #0  0x080519d2 in h_conn ()
    #1  0x08051a3e in prothandle ()
    #2  0x08052e88 in srvserve ()
    #3  0x08054a38 in main ()
    

    Any suggestions?

    opened by kr 20
  • add sunos.c for solaris/illumos support

    add sunos.c for solaris/illumos support

    I added preliminary solaris/illumos support with event ports in sunos.c as mentioned in #112. The other changes outside sunos.c were related to portability issues between BSD/Linux and SunOS.

    This branch was tested with make bench on the following platforms:

    • SmartOS joyent_20130629T040542Z with GCC 4.7.2
    • CentOS+OpenVZ 2.6.32-042stab076.7 with GCC 4.7.1
    • OS X 10.8.4 with clang 4.2

    Here's the related documentation for event ports:

    opened by potatosalad 17
  • improve tests coverage

    improve tests coverage

    codecov

    Right now tests coverage is at 32%. What is worse, some files are not tested at all: conn.c, file.c, linux.c, prot.c, walg.c

    https://codecov.io/gh/beanstalkd/beanstalkd/list/master/

    We might want to pick low hanging fruit by adding integration tests that cover wide area.

    Help Wanted Testing 
    opened by ysmolsky 15
  • beanstalkd is compiled without optimisations

    beanstalkd is compiled without optimisations

    I noticed beanstalkd is compiled without optimisations. Adding -O3 to CFLAGS gives a new warning that is converted to an error, but otherwise it compiles fine.

    Should optimisations be enabled?

    opened by Minnozz 14
  • consider adding active maintainers

    consider adding active maintainers

    @kr, would you consider the idea of adding more maintainers to the project?

    There are lots of requests for a release and there are many outstanding bugs in the beanstalkd. Some use it in production because it's fast and capable, but there are critical bugs for which people has to use workarounds and crutches. Users, me included, love this server and want to make it even better.

    I understand that you are busy and maybe lost interest, but please find some time to pass the torch to somebody who still cares.

    Thank you.

    CC: @JensRantil

    opened by ysmolsky 13
  • Update protocol.txt

    Update protocol.txt

    I've wrapped each command into quotes (except for those which had already been wrapped). So now each command declaration starts with:

    The "<command_name>" command
    
    opened by sergeyklay 13
  • Which is the best way to replicate between beanstalkd?

    Which is the best way to replicate between beanstalkd?

    Hello there!

    I have been testing coolbeans but it does not have all the commands implemented for beanstalkd, so using that project is not an option yet, and I was wondering which is the best way to replicate the data between beanstalkd servers.

    The idea is to have a Haproxy in front of it and only send the the traffic to one available beanstalkd, and if it fails, then send it to the next one, like (Redis + Sentinel + Haproxy) pattern, but the problem is that, to be able to do it replication need to be done, at least pasive replication. So my question is, do you have some idea or best practise to do this with beanstalkd?

    Thank you in advance!

    opened by achetronic 2
  • Error on building beanstalkd 1.12 source from spec file

    Error on building beanstalkd 1.12 source from spec file

    Hi ,

    I need beanstalkd 1.12 rpm file , so I am trying to build beanstalkd 1.12 source code from spec provided in source, but on build command i am getting error as- image

    If we remove buildconf.sh , it still gives error for /Configure file not found.

    Please suggest me to solve the build errors

    WaitingForInfo 
    opened by premagavarawad 2
  • binlogs continually increase when delaying jobs

    binlogs continually increase when delaying jobs

    Beanstalk versions: 1.11, 1.12, current master (4c275d5945299e4562389f9f2ca7c326173d6335)

    Steps to reproduce:

    1. Launch beanstalkd with binlogs enabled
    2. Fill a tube with 100k jobs
    3. Continually reserve and release the jobs with a delay
    4. Observe that over time, the binlog size increases

    I would expect that for a fixed size of jobs, the size of the binlogs remains fixed.

    Example perl code:

    Create jobs

    use strict;
    use warnings;
    use Beanstalk::Client;
    
    my $c = Beanstalk::Client->new();
    $c->use('test');
    
    for (1 .. 100_000) {
        $c->put({}, {});
    }
    

    Worker:

    use strict;
    use warnings;
    use Beanstalk::Client;
    
    my $c = Beanstalk::Client->new();
    $c->watch('test');
    
    while (1) {
        my $job = $c->reserve;
        $job->release({ delay => 20 });
    }
    

    Cause:

    I believe it's due to filewrjobshort being called every time the job becomes ready, these Jobrec dumps are never cleaned up until the job is deleted. https://github.com/beanstalkd/beanstalkd/blob/4c275d5945299e4562389f9f2ca7c326173d6335/walg.c#L186

    NeedsFix 
    opened by simon816 1
  • Document pipelining and its limitations

    Document pipelining and its limitations

    This issue was raised in @266 before where @JensRantil proposed following solution to the problem:

    Before considering this, have you first tested using a client that supports pipelining?

    @meox

    Originally posted by @ysmolsky in https://github.com/beanstalkd/beanstalkd/issues/427#issuecomment-507016097

    We should document somewhere that pipelining is supported and that it has limitations. Currently because of the protocol not using IDs in all responses pipelining can only be reliably implemented for reserve commands. Commands like bury or stats-job don't return an id and thus matching a response with its command is hard. (For stats-job you could look in the data, but that's a hack)

    opened by SamMousa 0
  • Format for lists and dictionary responses

    Format for lists and dictionary responses

    I think a somewhat simpler/easier way forward would be to restrict the output format of the existing commands from YAML to JSON. This ought to be a backwards-compatible change, since all YAML parsers are supposed to be able to read JSON. However, it would be worth testing the major client languages to double-check that they actually do the right thing in that case.

    Originally posted by @kr in https://github.com/beanstalkd/beanstalkd/issues/610#issuecomment-1003193205

    I've thought some more on this and I think going either the YAML or the JSON road is troublesome unless you want to pull in a proper library into beanstalkd.

    We could actually simplify the formats as well, since the beanstalkd protocol is a text based protocol and requires a custom parser / interpreter it's not much work to define the lists and dictionaries.

    Lists:

    • All entries are strings
    • Each entry is separate by a \n

    This gives you all you need to know, no parsing and more importantly no escaping required.

    Dictionary:

    • Each entry has a key and a value
    • Since we control the keys we can just put strict definitions on them; keys only consist of the alphabet a-z and the character -.
    • Values should be interpreted as string unless we specify that they are some other type explicitly.
    • Keys and values are separated by a :

    These could have trivial implementations in beanstalkd code and to not break existing clients we could implement these as new commands. For example stats-tubes-2 or stats-2.

    For me this has no priority by the way, I'm happy parsing the YAML in it's current state and since I'm assuming developers create tube names in a reasonable way (not containing user input) it should cause no issues.

    FeatureRequest Unplanned 
    opened by SamMousa 3
Owner
Beanstalkd
Beanstalk is a simple, fast work queue.
Beanstalkd
A lock-free thread-owned queue whereby tasks are taken by stealers in entirety via buffer swapping

Swap Queue A lock-free thread-owned queue whereby tasks are taken by stealers in entirety via buffer swapping. This is meant to be used [thread_local]

Thomas Sieverding 20 Sep 9, 2022
A better message queue built by rust

bettermq A better message queue built by rust I start this project to study Rust

Sun Junyi 13 Dec 16, 2022
twilight-interactions is a set of macros and utilities to work with Discord Interactions using twilight.

Twilight interactions twilight-interactions is a set of macros and utilities to work with Discord Interactions using twilight. Note: This crate is not

null 24 Dec 26, 2022
Rust implementation for Wlroots (Sway, Wayfire, Hikari, River, etc.) of Gnome Screenshot and Idle DBUS Server, which Upwork uses to capture the screen as proof of work.

?? upwork-wlroots-bridge ?? Rust Implementation for Wlroots (Sway, Wayfire, Hikari, River, etc.) of Gnome Screenshot and Idle DBUS Server (with extra

Daniel Moretti V. 4 Jan 2, 2023
Fast and simple PHP version manager written in rust

[WIP] phpup (PHP-up): Fast and Simple PHP version manager ⚑ Fast and simple PHP version manager, written in rust Features No requirements for system P

null 27 Dec 25, 2022
A simple and fast FRC autonomous path planner (designed for swerve drive)! (Desktop/Laptop only)

This is a website developed for planning autonomous paths for FRC robots. It is intended to be a simple and fast tool to create autos, which works offline at competitions.

Weaver Goldman 2 Jan 6, 2023
qn (quick note) is a simple, fast and user-friendly way to save notes πŸ¦€βš™οΈ

Quick Note qn Install This is currently for my personal use. I may push breaking changes at any time. If you want to use it, bring down the code and r

Code Smell 3 Jul 15, 2022
A dead-simple, extreme fast permission flag system for Rust with no dependencies

A dead-simple, extreme fast permission flag system for Rust with no dependencies

pan93412 2 Mar 17, 2022
A simple, fast and fully-typed JSPaste API wrapper for Rust

rspaste A simple, fast and fully-typed JSPaste API wrapper for Rust. aidak.tk Β» Installation Put the desired version of the crate into the dependencie

Aidak 2 May 17, 2022
This is a simple Telegram bot with interface to Firefly III to process and store simple transactions.

Firefly Telegram Bot Fireflies are free, so beautiful. (Les lucioles sont libres, donc belles.) ― Charles de Leusse, Les Contes de la nuit This is a s

null 13 Dec 14, 2022
MeiliSearch is a powerful, fast, open-source, easy to use and deploy search engine

MeiliSearch is a powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-the-box. For more information about features go to our documentation.

MeiliSearch 31.6k Dec 30, 2022
Extreme fast factor expression & computation library for quantitative trading in Python.

Extreme fast factor expression & computation library for quantitative trading in Python.

Weiyuan Wu 22 Dec 8, 2022
Novus - A blazingly fast and efficient package manager for windows.

Novus - A blazingly fast and efficient package manager for windows. Why Novus Swift Unlike any other package manager, Novus uses multithreaded downloads

Novus 197 Dec 18, 2022
A little bit fast and modern Ruby version manager written in Rust

A little bit fast and modern Ruby version manager written in Rust Features Pure Rust implementation not using ruby-build Cross-platform support (macOS

Takayuki Maeda 510 Jan 5, 2023
πŸš€ Fast and 100% API compatible postcss replacer, built in Rust

?? Fast and 100% API compatible postcss replacer, built in Rust

θΏ·ζΈ‘ 472 Jan 7, 2023
Build fast, reward everyone, and scale without friction.

Scrypto Language for building DeFi apps on Radix. Terminology Package: A collection of blueprints, compiled and published as a single unit. Blueprint:

Radix DLT 330 Dec 25, 2022
An implementation of Code Generation and Factoring for Fast Evaluation of Low-order Spherical Harmonic Products and Squares

sh_product An implementation of Code Generation and Factoring for Fast Evaluation of Low-order Spherical Harmonic Products and Squares (paper by John

Simon Brown 7 Dec 2, 2022
πŸ¦€πŸš€πŸ”₯ A blazingly fast and memory-efficient implementation of `if err != nil` πŸ”₯πŸš€πŸ¦€

?????? A blazingly fast and memory-efficient implementation of `if err != nil` ??????

Federico DamiΓ‘n Schonborn 6 Dec 30, 2022
πŸš€ Fleet is the blazing fast build tool for Rust

Fleet is the blazing fast build tool for Rust. Compiling with Fleet is up-to 5x faster than with cargo.

Dimension 2.2k Jan 6, 2023