Is the GIL seeing someone else? How's about repetitively calling and seeing how long it takes to answer?

Miles Granger

Last update: Jan 18, 2023

Related tags

Command-line gilknocker

Overview

GIL Knocker

pip install gilknocker

When you thought the GIL was available, and you find yourself suspecting it might be spending time with another.

You probably want py-spy, however if you're looking for a quick-and-dirty way to slip in a GIL contention metric within a specific chunk of code, this might help you.

How?

Unfortunately, there doesn't appear to be any explicit C-API for checking how busy the GIL is. PyGILState_Check won't really work, that's limited to the current thread. PyInterpreterState is an opaque struct, and the PyRuntimeState and other goodies are private in CPython.

So, in ~200 lines of Rusty code, I've conjured up a basic metric that seems to align with what is reported by py-spy when running the same test case. This works by spawning a thread which, at regular intervals, re-acquires the GIL and checks how long it took for the GIL to answer.

Note, the interval (interval_micros) is configurable. The lower the value, the more accurate the metric, but will be more likely to slow your program down.. because it will play a larger role in competing for the GIL's attention.

Use

Look at the tests

from gilknocker import KnockKnock

knocker = KnockKnock(interval_micros=1000, timeout_secs=1)
knocker.start()

... smart code here ...

knocker.contention_metric  # float between 0-1 indicating roughly how busy the GIL was.
knocker.reset_contention_metric()  # reset timers and meteric calculation

... some more smart code ...

knocker.stop()

knocker.contention_metric  # will stay the same after `stop()` is called.

How will this impact my program?

Short answer, it depends, but probably not much. As stated above, the more frequent the monitoring interval, the more likely GIL bound tasks will be affected. This is demonstrated in the benchmarks testing. Below is a summary of benchmarking two different functions, one which uses the GIL, and one which releases it. For interval=None this means no polling was used, effectively just running the function without gilknocker. Otherwise, the interval represents the value passed to KnockKnock(interval_micros=interval)

python -m pytest -v --benchmark-only benchmarks/ --benchmark-histogram

---------------------------------------------------------------------------------------------- benchmark: 12 tests ----------------------------------------------------------------------------------------------
Name (time in ms)                          Min                   Max                  Mean             StdDev                Median                 IQR            Outliers     OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_bench[a_lotta_gil-None]          697.6828 (1.0)        804.5402 (1.11)       755.6981 (1.06)     53.0970 (61.53)      777.1266 (1.09)     101.6509 (83.91)         2;0  1.3233 (0.95)          5           1
test_bench[a_lotta_gil-10]            707.0513 (1.01)       724.3552 (1.0)        714.4783 (1.0)       6.8460 (7.93)       715.2083 (1.0)       10.0545 (8.30)          2;0  1.3996 (1.0)           5           1
test_bench[a_lotta_gil-1000]          708.0325 (1.01)       742.4564 (1.02)       722.2247 (1.01)     12.6517 (14.66)      721.7707 (1.01)      12.5343 (10.35)         2;0  1.3846 (0.99)          5           1
test_bench[a_lotta_gil-10000]         716.1168 (1.03)       791.8905 (1.09)       733.0825 (1.03)     32.9744 (38.21)      717.7345 (1.00)      23.2516 (19.19)         1;1  1.3641 (0.97)          5           1
test_bench[a_lotta_gil-100000]        758.2248 (1.09)       760.4424 (1.05)       759.2441 (1.06)      0.8629 (1.0)        758.9144 (1.06)       1.2114 (1.0)           2;0  1.3171 (0.94)          5           1
test_bench[a_lotta_gil-100]           760.8787 (1.09)       839.1526 (1.16)       777.9811 (1.09)     34.2144 (39.65)      763.4823 (1.07)      20.4199 (16.86)         1;1  1.2854 (0.92)          5           1
test_bench[a_little_gil-None]       1,505.1989 (2.16)     1,510.2234 (2.08)     1,508.0564 (2.11)      1.8985 (2.20)     1,508.2229 (2.11)       2.5074 (2.07)          2;0  0.6631 (0.47)          5           1
test_bench[a_little_gil-100000]     1,506.0053 (2.16)     1,559.4051 (2.15)     1,531.3341 (2.14)     22.6875 (26.29)    1,524.5321 (2.13)      38.7802 (32.01)         2;0  0.6530 (0.47)          5           1
test_bench[a_little_gil-10000]      1,508.9686 (2.16)     1,521.0912 (2.10)     1,515.0701 (2.12)      5.5128 (6.39)     1,514.7033 (2.12)      10.3673 (8.56)          2;0  0.6600 (0.47)          5           1
test_bench[a_little_gil-1000]       1,534.0449 (2.20)     1,540.6296 (2.13)     1,537.8621 (2.15)      2.5307 (2.93)     1,538.5808 (2.15)       3.4261 (2.83)          2;0  0.6503 (0.46)          5           1
test_bench[a_little_gil-100]        1,566.4128 (2.25)     1,576.2634 (2.18)     1,569.6245 (2.20)      4.0978 (4.75)     1,567.4297 (2.19)       5.3087 (4.38)          1;0  0.6371 (0.46)          5           1
test_bench[a_little_gil-10]         1,587.1471 (2.27)     1,597.2920 (2.21)     1,592.0651 (2.23)      3.7001 (4.29)     1,591.2409 (2.22)       4.1942 (3.46)          2;0  0.6281 (0.45)          5           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

License

Unlicense or MIT, at your discretion.

Comments

Failed to join knocker thread

Ran into the same issue that was already reported in https://github.com/dask/distributed/issues/7290#issuecomment-1386241201

Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 07:07:06) [Clang 13.0.1 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from gilknocker import KnockKnock

In [2]: knocker = KnockKnock(1000, 1)

In [3]: knocker.start()

In [4]: knocker.stop()
---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 knocker.stop()

TimeoutError: Failed to join knocker thread.

I tried it a couple of times. Once it was successful but most of the times it failed. this was on version 0.1.0

opened by fjetter 3

Deadlock only in tests?

Running python tests/test_knockknock.py, where as a script, executes the busy gil variant test, never seems to deadlock. Can also add the other non-busy gil variant, and also doesn't seem to deadlock. :thinking:

However, pytest running those same tests will deadlock fairly often, maybe 20-30% of the time.

opened by milesgranger 1
Better signal for when reset is complete.

https://github.com/milesgranger/gilknocker/blob/aed7ec495570920e2e11818fa9c1e2843c330ac8/src/lib.rs#L95

It works, but it's a bit hacky and quite possibly won't work if a configured high frequency polling is in place while also under high GIL contention, then a sleep could easily miss the reset and contention metric could be above that hard-coded value.

opened by milesgranger 0
Support resetting of contention metric, fix deadlocking
Support method reset_contention_metric() which resets the contention metric and timers.

Fix deadlocking by internally spawning a thread specifically to try and obtain the GIL. This allows the monitoring thread to remain available for receiving messages.

Thereby also removing the flaky decorator and xfail markers.

Ref https://github.com/dask/distributed/issues/7290#issuecomment-1385998993
opened by milesgranger 0

Make contention_metric available for monitoring

Makes contention_metric a continuously updated value, so long as the monitoring thread is running.

Example from one of the updated tests:

        # usually ~0.9 on linux ~0.6 on windows
        assert knocker.contention_metric > 0.6

        # Now wait for it to 'cool' back down
        # by looping over some work which releases the GIL
        prev_cm = knocker.contention_metric
        for i in range(10):
            a_little_gil()
            assert knocker.contention_metric < prev_cm
            prev_cm = knocker.contention_metric

        # ~0.15 oN mY MaChInE.
        assert knocker.contention_metric < 0.2

Ref: https://github.com/dask/distributed/issues/7290#issuecomment-1385742756

opened by milesgranger 0

Releases(v0.2.1)

v0.2.1(Jan 18, 2023)
What's Changed

Better impl of contention metric reset finish signal by @milesgranger in https://github.com/milesgranger/gilknocker/pull/6

Full Changelog: https://github.com/milesgranger/gilknocker/compare/v0.2.0...v0.2.1
Source code(tar.gz)
Source code(zip)
v0.2.0(Jan 18, 2023)
What's Changed

Make contention_metric available for monitoring by @milesgranger in https://github.com/milesgranger/gilknocker/pull/2

Support resetting of contention metric, fix deadlocking by @milesgranger in https://github.com/milesgranger/gilknocker/pull/3

New Contributors

@milesgranger made their first contribution in https://github.com/milesgranger/gilknocker/pull/2

Full Changelog: https://github.com/milesgranger/gilknocker/compare/v0.1.0...v0.2.0
Source code(tar.gz)
Source code(zip)
v0.1.0(Jan 17, 2023)
Initial release

Known issues:

Occasional deadlock/timeout when stopping monitoring thread.

Source code(tar.gz)
Source code(zip)

Is the GIL seeing someone else? How's about repetitively calling and seeing how long it takes to answer?

Related tags

Overview

GIL Knocker

How?

Use

How will this impact my program?

License

Comments

Failed to join knocker thread

Deadlock only in tests?

Better signal for when reset is complete.

Support resetting of contention metric, fix deadlocking

Make contention_metric available for monitoring

Releases(v0.2.1)

v0.2.1(Jan 18, 2023)

What's Changed

v0.2.0(Jan 18, 2023)

What's Changed

New Contributors

v0.1.0(Jan 17, 2023)

Owner

Miles Granger

Stall tracking for Python's GIL and Trio tasks

Command-Line program that takes images and produces the copy of the image with a thin frame and palette made of the 10 most frequent colors.

ddi is a wrapper for dd. It takes all the same arguments, and all it really does is call dd in the background

lemmy-help is a emmylua parser as well as a CLI which takes that parsed tree and converts it into vim help docs.

`rusty_regex` takes an input string and produces a `regex` string representing what was provided.

Ideas => Creations, a multi-language CMS(Content Management System) based on Rust Web stacks, with long-term upgrade and maintenance.

argmax is a library that allows Rust applications to avoid Argument list too long errors (E2BIG) by providing a std::process::Command wrapper with a

🧠 A command-line utility for switching git branches more easily. Switch branches interactively or use a fuzzy search to find that long-forgotten branch name.

Fast tool to scan for valid 7-long imgur ids for the ArchiveTeam imgur efforts (not affiliated or endorsed)

Warp is a blazingly fast, Rust-based terminal that makes you and your team more productive at running, debugging, and deploying code and infrastructure.

interactcli-rs is a command-line program framework used to solve the problem of the integration of command-line and interactive modes, including functions such as unification of command-line interactive modes and sub-command prompts. The framework integrates clap and shellwords.

This is a simple lnd poller and web front-end to see and read boosts and boostagrams.

Sets of libraries and tools to write applications and libraries mixing OCaml and Rust

Downloads and provides debug symbols and source code for nix derivations to gdb and other debuginfod-capable debuggers as needed.

Tooling and library for generation, validation and verification of supply chain metadata documents and frameworks

A comprehensive collection of resources and learning materials for Rust programming, empowering developers to explore and master the modern, safe, and blazingly fast language.

ratlab is a programming platform designed loosely for hobbyist and masochist to analyse and design stuff and things that transform our world?

REC2 (Rusty External Command and Control) is client and server tool allowing auditor to execute command from VirusTotal and Mastodon APIs written in Rust. 🦀

Execution of and interaction with external processes and pipelines