hexyl is a simple hex viewer for the terminal. It uses a colored output to distinguish different categories of bytes

This pull request fixes #13 partially by adding the --columns option to manually set the number of columns. The default is 2 as with previous behavior.

Some notes:

• the short option is -w not -c as that option is already reserved for --bytes
• it passes cargo test
• I could have added the option to automatically set columns to terminal width, but that would require adding another dependency

The -n/--length flag is a great feature -- good especially for quickly checking if file headers match something. Another use case I can think of for limited output is inspecting, say, an entire block of some data from a file that's been dumped from a disk. Let's say I'm using Linux and reading the first block of a disk somewhere to determine its contents manually:

$ hexyl -n 512 "$disk_dump"
// Some output here...

I read the output and discover that there's an MBR at the beginning, with the first partition starting at logical block address 1. Sweet, let's mosey on over to 0x200 and read another block. I could implement this by using dd:

$ input_file="_viminfo" block_size=512 block_num=1
$ dd bs="$block_size" status=none skip="$block_num" count="$block_size" if="$input_file" \
    | hexyl

...but there's two issues with this:

• dd isn't usually available on Windows machines!
• Because we're using stdin, we can't get the correct set of offsets -- the "file" starts at 0x0 regardless of what parameters I gave dd.

Perhaps something like this spitballed set of options might help:

$ hexyl \
    --start 512      \ # Could also be written as 
                     \ #
                     \ # Could be bikeshed to `--begin`?
                     \
    --length 512     \
                     \
                     \ # One could use an end offset instead of a length:
    # --end 1024     \ # Could also be written as `-e 0x300

Having something similar to bat's --range could also be really handy, especially when combined with relative offsets (positive and negative):

$ hexyl --range 512:+512 # same as using `--skip 512 --length 512`
$ hexyl --range=-512: # read the last block
$ hexyl --block-size 4096 -1block: # like above, but use the block unit
$ hexyl --range 32:-32 # cut out a common header and footer for the input stream we don't care about

I would be more than happy to push implementation of this, since I've great personal interest in allowing more of my reverse engineering flow in the terminal. Let me know if you want me to hack on it!

Unresolved questions

• ~~Adding several more arguments that accept offsets/sizes might add pressure to create a system of units/radices a la xxd for the appropriate arguments. Where should the line be drawn in terms of what this project is willing to support?~~ We've added support for xb and xib with #44. That's as far as we've decided to go right now.

Hi,

Thank you for a new release. I was looking into updating the package for NetBSD and realized the man page is provided in markdown. Would it possible to provide the man page in the man page Unix format, i.e. like hexyl.1 or hexyl.1.gz ? It would be desirable to avoid the need of external build dependencies for a simple man page.

TIA

I did some profiling of hexyl using cargo flamegraph and identified some key performance optimizations that could be made.

The three methods that take up the most time are:

• write_all into the buffer and then stdout
• format! call in print_position_panel
• multiple uses of the write! and writeln! macro in print_* functions

The three commits in the pull request address these three issues.

The first replaces the manual buffers used by hexyl for reading and writing by BufReader and BufWriter, respectively. This required some major refactoring of code because of the way that buffer_line and raw_line were used. I feel that the refactoring was appropriate and made the code much easier to read, so it was worth it even without performance optimizations. With this addition, here is the benchmark:

| Command | Mean [s] | Min [s] | Max [s] | Relative | |:---|---:|---:|---:|---:| | hexyl data | 1.068 ± 0.015 | 1.045 | 1.084 | 1.23 ± 0.03 | | hexyl --no-squeezing data | 1.073 ± 0.035 | 1.041 | 1.164 | 1.23 ± 0.04 | | target/release/hexyl data | 0.895 ± 0.019 | 0.871 | 0.917 | 1.03 ± 0.03 | | target/release/hexyl --no-squeezing data | 0.871 ± 0.013 | 0.851 | 0.899 | 1.00 |

The second commit replaces the format! call in print_position_panel with a workaround. Previously, the function would take the position and format it as hexademical, then painted it if color was enabled. This was expensive! In order to avoid this, I used the same strategy as with the byte_hex_panel and byte_char_panel and precomputed the values in the new function so they would only be created once through byte_hex_panel_g and byte_char_panel_g. This adds a tiny amount of overhead to the startup but it's invisible in profiling. Here is the benchmark with both the first and second commit:

| Command | Mean [s] | Min [s] | Max [s] | Relative | |:---|---:|---:|---:|---:| | hexyl data | 1.181 ± 0.025 | 1.135 | 1.215 | 1.67 ± 0.05 | | hexyl --no-squeezing data | 1.122 ± 0.024 | 1.091 | 1.165 | 1.59 ± 0.05 | | target/release/hexyl data | 0.738 ± 0.015 | 0.713 | 0.767 | 1.04 ± 0.03 | | target/release/hexyl --no-squeezing data | 0.706 ± 0.014 | 0.687 | 0.729 | 1.00 |

The last and most dramatic optimization was the elimination of the write! macro sprinkled throughout the three print methods that are repeated every line, replacing it with the more direct write_all method on self.writer. This wasn't actually too difficult, because all of the write! macro calls didn't use any formatting machinery. I didn't remove every use of write! because profiling shows that it wasn't important outside of the loop. Here is the benchmark with all of these commits:

| Command | Mean [s] | Min [s] | Max [s] | Relative | |:---|---:|---:|---:|---:| | hexyl data | 1.062 ± 0.027 | 1.024 | 1.101 | 3.69 ± 0.11 | | hexyl --no-squeezing data | 1.072 ± 0.011 | 1.060 | 1.099 | 3.73 ± 0.07 | | target/release/hexyl data | 0.332 ± 0.007 | 0.323 | 0.343 | 1.15 ± 0.03 | | target/release/hexyl --no-squeezing data | 0.287 ± 0.005 | 0.282 | 0.299 | 1.00 |

And here is the final flamegraph:

It should probably handle SIGINT and terminate the output as if it was EOF (but maybe with some mark):

│00001410│ 57 37 f9 ae 0b ae 6c 8a ┊ df 0e d8 20 15 f1 d7 f6 │W7×ו×l×┊×•× •×××│
│00001420│ a5 5e 99 b0 bd bf 22 a7 ┊ b2 e0 ab ^C             | ×^×××"×┊×××^C   |

We've been benchmarking the performance of the tool without considering the rendering performance of the terminal. Specifically, I'm thinking about how we turn colors on and off again for every single hex pair and textual character. Ideally we wouldn't turn colors off if the next printed hex/char uses the same color.

I'm not really sure how to programmatically measure the terminal performance (and of course performance would change for different terminals), but it's worth at least trying to measure. Optimizing our color usages would be more overhead on our side and therefore slow down our benchmark (though perhaps not significantly) but if it produces faster rendering it might be worth it.

At the very least, we could investigate not printing the style suffix for each character, under the assumption that the style prefix for the next character will suffice (and then just printing the suffix prior to printing a frame character).

| Unit | Implemented? | Description | Examples | Suggested implementation | | ------- | ------------ | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Decimal | [x] | A decimal integer, which is equivalent to specifying a single byte unit for the count. | 23, 1024 | u64::from_str(...) | | Hex | [x] Implemented in #45. | A hexadecimal integer. Specified with a leading 0x. | 0x17, 0x100 | u64::from_str_radix(...) | | Blocks | [ ] | A single block, which is by default 512 bytes but configurable via config flag. | -b 512 -n 1block

N.B: one cannot use a block unit to define the block size. | Add a flag to optionally define block size, then check for a trailing block when parsing numbers. Multiply by block size. | | Bytes | [ ] | A byte size familiar to most IT professionals. Specified by B at the end of the count, and can include an optional magnitudinal spec like kilobytes (K) or megabytes (M). |

• 23B: 23 bytes
• 9KB: 9 kilobytes

Other open questions

• Should the block unit, which might be controversial, be scoped into another issue?
• Should a single leading - or + sign be supported?
  • Not sure how + is useful -- xxd's manual states that for the -s option + is useful only for stdin. Not sure what that means, though.

When printing the output of hexyl to a file (hexyl file.exe > file.txt), the file.txt file gets polluted by the ANSI escape sequences. Checking whether the application has been launched from inside the terminal or not, allows you to decide whether or not to use them.

I don't have access to a Windows machine, but atty claims that it has been tested on Windows.

> brew install hexyl
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 3 taps (homebrew/cask-versions, homebrew/core and homebrew/cask).
==> Updated Formulae
bash ✔               brew-php-switcher    fonttools            mariadb              telegraf
libidn2 ✔            buildifier           grv                  mercurial            tor
wget ✔               chakra               hcloud               mutt                 urdfdom_headers
alexjs               citus                jabba                neofetch             vegeta
amqp-cpp             clang-format         jenkins              nvm                  whois
aws-sdk-cpp          conan                kitchen-sync         picard-tools         wtf
azure-cli            console_bridge       libphonenumber       rabbitmq             yarn
bat                  emscripten           libpsl               serverless
bettercap            erlang@20            lmod                 sox
bitrise              exploitdb            lxc                  swiftlint

/usr/local/Homebrew/Library/Homebrew/config.rb:39:in `initialize': no implicit conversion of nil into String (TypeError)
        from /usr/local/Homebrew/Library/Homebrew/config.rb:39:in `new'
        from /usr/local/Homebrew/Library/Homebrew/config.rb:39:in `<top (required)>'
        from /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/local/Homebrew/Library/Homebrew/global.rb:25:in `<top (required)>'
        from /usr/local/Homebrew/Library/Homebrew/brew.rb:13:in `require_relative'
        from /usr/local/Homebrew/Library/Homebrew/brew.rb:13:in `<main>'

> hexyl
-bash: hexyl: command not found

macOS High Sierra Version ~10.13.16~ 10.13.6

MacBook Pro

Hi there, Thank you for this tool.

There is a background color making it difficult to see the characters on rxvt-unicode only.

rxvt-unicode with default settings:

maybe that looks not a big problem, but when I have dark terminal theme, that make it difficult to see:

That background color not appear on other terminals, just rxvt-unicode.

While using it for large file, the output will scrolling long-long time , and piping to more will output bad string . So, please add pager to output .

I decided to take a stab at #89 just to see how feasible it is. In order to facilitate this, I have temporarily changed the API of print_all to take Input with a new generic Box <dyn Read> variant as a fallback.

So far, this implementation is only able to use sparseness when there is non-sparse content at the end of the file, and the rest of the file, including the start, is sparse. Otherwise, it will cause strange errors.

It also only works for Linux, because this was the simplest OS to implement sparse support for.

How to test this:

truncate -s 16G startzero
echo "Lorem ipsum" >> startzero
cargo run --release -- startzero

The position will be slightly wrong but it will finish extremely quickly.

Tasks:

• [x] correct position indexing
• [x] handle fully sparse files
• [x] handle when the file starts with non-sparse data
• [x] handle non-sparse data interspersed in files (this might work already to some extent)
• [x] fully support Linux (caveat: does not work with BufReader)
• [ ] fully support Windows
• [ ] fully support MacOS

Adds the following options:

• --hex-inner-separator none Removes the inner separator from the hex display
• --text-inner-separator none Removes the inner separator from the text display
• --outer-border none Removes the outer border (including header, footer and left border, but currently leaves right border in)

TODO:

• [ ] Confirm naming of options and associated variables, structs etc.
• [ ] Confirm documentation in --help for new options
• [ ] Add changes to CHANGELOG.MD
• [ ] Add tests for new options
• [x] Check existing tests haven't broken

The hexyl --help description for the --block-size argument goes as follows:

    -o, --display-offset <N>    Add N bytes to the displayed file position. The
                                N argument can also include a unit (see
                                `--length` for details)
                                A negative value is valid and calculates an
                                offset relative to the end of the file.

However, when I attempt to run dd if=/dev/urandom bs=16 count=32 2>/dev/null | hexyl --skip=0xFF --display-offset=-0x2F, hexyl prints the following error:

Error: failed to parse `--display-offset` arg "-0x2F" as byte count

Caused by:
    negative offset specified, but only positive offsets (counts) are accepted in this context

Note: I'm doing a class assignment regarding creating tests (from scratch) for an open-source command line program, and I chose hexyl for it. You might have a couple more of these on the way, depending on what the assignment brings to light :)

I frequently have to dump data files (ADC output, for example) that don't just have byte-oriented data. It would be nice to be able to specify data width in the dump so I get the hex data grouped in the natural data size instead of having to do the little-endian two-step and mentally group indistinguishable bytes by 2 or 4 or whatever. Something like:

--word-size=1 (uint8_t, default) --word-size=2 (uint16_t) --word-size=4 (uint32_t) --word-size=8 (uint64_t) --word-size=16 (uint128_t)

That covers the common-ish types. If you want to be really brave you could do weird crap like 3-byte or 17 byte, but that is likely low return on investment.

Not all such data is little-endian, so an extra flag for those cases where word-size > 1 would be:

--little-endian (default) --big-endian

Also, interpretation could be signed or unsigned

--signed --unsigned (default)

Of course with this you'd drop the byte-oriented colouration (but maybe with --signed you'd highlight negative numbers in red or something).

Because hexyl truncates repeating sections, it would be nice to be able to have hexyl quickly skip over these sections instead of scanning them byte-by-byte.