When retrieving the Loader::program of cyberblogger from https://github.com/trailofbits/cb-multios (compiled with clang 8.0.1) an error Err("Index does not exist for set_entry") is returned from the line control_flow_graph.set_entry(block_indices[&function_address].0)?; at the end of translate_function_extended. This occurs because for the function at address 0x45008 returns from the translate_block call in the x86 decoder immediately due to a CS_ERR_OK. This leaves the translation block with no instructions which results in a bogus insertion to the block_indices at block_indices.insert(*result.0, (block_entry, block_exit)); in translate_function_extended because block entry is set to 0 and block exit is set to 0 as nothing was inserted into the overall Function CFG and the block_entry variable was never set.

My hacky fix for an empty translation block is up at https://github.com/2over12/falcon

I simply check if there are no instructions for a block and add a new empty CFG to the block if so. There probably should not be an empty function though, it is currently unclear to me why falcon_capstone immediately returns a CS_ERR_OK, but it also probably worth handling an empty function in some sort of sane way.

Hello!

So I've been reading project, really great work and I'm really excited for some of the stuff you're doing, can't wait to see more, no matter what gets decided!

On that note, as you guessed from the title, I'm hoping it might be possible to consolidate 0-N things from this, panopticon, and a theoretical new memory interval crate that I want to write, as well as some other things.

This is a huge, huge topic, and I likely won't hit on a lot of the points, but just getting the ball rolling is good I think, if only to see if you're interested, where you're headed with things, etc.

If you're not interested at all, that is totally fine of course :) Just wanted to see what you think

Generic IL/Function crate

So, for starters (and probably most controversially), reading through your source, particularly the il module, there is so much that I think could be refactored (along with panopticon) into a generic function/il rust crate.

I say controversial because it will likely be hard/tedious, but i do think it would be (extremely) beneficial.

It would also require probably the deepest amount of coordination, which could be hard.

Nevertheless, I think some prime candidates are the il, and the function objects. If we could somehow make Function<IL>, where IL is the intermediate language used, this could have really really cool benefits.

1. It would allow all of us to try out each other's IR
2. it would allow us to switch to another IR for a different task (perhaps one is more compact, hence faster analysis)
3. It would allow us to reuse the function logic and definitions and methods, consolidating bugs and developer effort into a single location (this and 2 are my prime motivation)

Its hard for me to state how great this could be if we were able to swap out IL's at will. It also just seems right from an engineering perspective, similar to backends on a compiler.

As it stands now in both codebases, I think this modification is almost trivially possible - except - the disassembler aspect.

But this isn't necessarily bad news!

For almost the exact reasons in 1-3, i think it would be really cool to allow function (or whatever it ends up being) to also be generic in the disassembler, allowing a more robust disassembler implementation (like capstone), or a home grown solution like panopticon, etc.

Again the benefits here are experimentation, can try different assembler for different IR backend, etc.

Doing this I think will require sketching out what a generic function + a generic disassembler would look like, and what would be the most flexible, and hence requires the most cooperation and assessment of current codebases dependencies and expectations etc., but long term I think it would be really cool, it would allow all our work to be pooled together and hence we'd all benefit.

While I think this will be the hardest part to refactor, coordinate and get right, I think it will actually have the most benefit; of course, this is just my opinion though :)

An interval tree crate, with a second crate geared towards binary memory intervals

I don't think this is controversial at all, and I think it would be invaluable. I want something like this already for bingrep, panopticon needs it, and i'm sure falcon could use it too.

Basically the idea is a:

[x..y) -> Value

Which is a datastructure that's created after the parser pass (or whenever you want, as long as you can send it a goblin binary), and which initially gets filled up with segment/section data; which ranges, what the name of the segment is, and perhaps what "kind of data" is there. We'd figure out what we want for a segment datatype, what information we'd need, etc. And of course, if its a central crate, when we need something new, we just extend it and everyone gets the benefit.

Similarly, and this would be the tricky part and where I want feedback, downstream users could also extend the memory ranges with their own tagging data, like [0xbeef..0xdead) -> FunctionRange, etc.

Even if some fancy runtime extendable type doesn't work out, even if we just agree on an enum in this crate which downstream clients use, I think this would be great code reuse and benefit everyone all around.

dynamic linker/runtime loader

Your loader looks really awesome!!! So i've been trying to get other persons to help create a relocator crate for a while, but no one is really interested in this stuff :laughing:

Anyway, at some future data I think panopticon wants to have this. So I've wanted to turn https://github.com/m4b/dryad into a library for quite some time. Basically I like working on that project and I'll find any excuse; also all that code going to waste would be sad.

So I'd like to propose potentially fusing falcon's runtime loader here with dryad, or vice versa, perhaps dryad becomes a lib, or i rip out parts of it via copy paste, whatever, and then that crate is refactored to be a library which downstream consumers like falcon and panopticon (and whoever really, who knows the applications!) can use it as their runtime linking and loading system.

Initial issue is i'll have to put the asm usage in dryad and bare functions behind feature flags, as it requires nightly, and its not nice to force that on downstream clients (which would be sad, since it's a pure rust toolchain dynamic linker that way!)

More things haven't even thought of

Anyway, that's my suggestion for 3 different things I think are candidate usecases to refactor out into shared dependencies for great good. I'm sure there are many other opportunities as well.

Let me know what you're thinking; as you can tell, I'm of the persuasion we should combine all of our powers and take over the universe :angel:

Thanks for reading this far, I know, it was a lot :)

/cc @flanfly

This project is very, very, relevant to my interests.

I see from the blog post you are using Capstone as the disassembler, which implements the ARM architecture. I'd like to take a stab at an implementation of ARM - at least for ARMv6 in Thumb mode (my current target).

I will use this issue to track an attempt at adding a module in falcon/lib/translator to map the Capstone ARM instruction API to Falcon IL.

If you've thought of this already and have ideas of where it might go wrong please chime in! I'm just familiarizing myself with the codebase now.

@emmanuel099 This adds an option to translators, allowing all unhandled instructions to be lifted to intrinsics. This is also an API-breaking change, so 0.5.0 bump.

Anything you want to throw in/change that may be API breaking?

Building off my previous pull request, and following the intent of making it easier for people to get started with Falcon, I've added a docker.sh script which will get the dependencies needed for Docker, install Docker, then build and run the container.

Because Docker doesn't seem to have networking available to the containers by default, I modified the Dockerfile.stretch to no longer attempt to reach out to the internet. Instead, the llvm snapshot key is downloaded on the host (in docker.sh) and Dockerfile.stretch just copies it into the container.

In testing this, I also found that the dependencies.sh was not compatible with Debian (which doesn't have sudo installed by default), so I also updated that script as well as setup.sh. The dependencies.sh script isn't run automatically, but the first thing I did when I went into the container was to run that so I would have everything I needed to run cargo build.

This change is motivated by the fact, that SMT solvers have builtin support for arithmetic shift right which is more efficient than the combination of shl, sub and shr.

I wanted the ability to run the code from within my VM, so I made a script to accomplish that. I also made a Vagrantfile so it was easier for me to test my changes. The setup.sh and dependencies.sh scripts can be used with Vagrant VMs, manually created VMs, physical machines, Docker images, etc. The general goal of this PR is to make it easier for people to get a Falcon environment set up.

Wanted to try this out (nice work btw!)

Do you have a simple cli program that takes a binary as input and outputs whatever?

If not, I'd suggest adding a simple prototype/reference program in lib/bin/main.rs so could be easy to try out, or perhaps something in examples/

Keep up the good work!

I've used "*" so that the project can be maintained against the latest libraries. I have tested it to be working on nightly build without any issues.

Additionally, I have used cargo clippy and cargo fix to fix linting issues.

• For both, rol and ror, the value assigned to ZF was more than one bit wide
• Replaced ZF shifting in rol by truncation (we are interested in the LSB instead of MSB)

Bumps regex from 1.5.4 to 1.5.6.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Falcon needs a good-and-proper Error enum. This will touch almost all of the codebase.

This is a very heavy lift, and is being forever/continually pushed to the right.