Attempt to make yara and yara-sys compatible with libyara 4.0.2

It turns out that keeping the 3.X compatibility will be harder than I though.

Closes #11

Hi,

I used the option vendored on the toml config file and all went good but once it parses any Yara rule that contains pe.number_of_signatures , it panics as follow:

thread 'main' panicked at 'Should have parsed rule: Compile(CompileErrors { errors: [CompileError { level: Error, filename: Some("C:\Users\user\yarafolder\test.yar"), line: 2009, message: "invalid field name "number_of_signatures"" }, CompileError

condition:
     for any i in (0 .. pe.number_of_signatures)

Toml configuration as follow:

[dependencies] yara = "0.16.0" yara-sys ={ version = "0.16.0", features = ["bundled-4_2_3","vendored"] }

what have I done to make not work :(?

When I'm compiling for windows x86, the following trivial test fails:

    #[test]
    fn check_yara_scan_mem() {
        const RULES: &str = r#"
            rule test {
                strings:
                    $a = { 42 }
                condition:
                    $a
            }
        "#;

        let rules = Compiler::new()
            .unwrap()
            .add_rules_str(RULES)
            .unwrap()
            .compile_rules()
            .unwrap();
        let matching = rules.scan_mem(&[0x00, 0x01, 0x42, 0x03], 10)
            .unwrap();
        assert_eq!(matching.len(), 1, "byte not found");
    }

check_yara_scan_mem panicked at 'called `Result::unwrap()` on an `Err` value: YaraError { kind: Unknown(53) }

On the line corresponding to rules_scan_mem().unwrap(). Error 53 corresponds to ERROR_CALLBACK_REQUIRED.

In yara-rust, the call boils down to this code in internals::scan :

    let result = unsafe {
        yara_sys::yr_rules_scan_mem(
            rules,
            mem.as_ptr(),
            mem.len().try_into().unwrap(),
            flags,
            Some(scan_callback),
            &p_callback as *const Box<dyn FnMut(CallbackMsg<'a>) -> CallbackReturn> as *mut _,
            timeout,
        )
    };

Some(scan_callback) is the function pointer to the callback, and p_callback is the user data that will be passed by libyara as an argument to scan_callback. (It's a pointer the actual callback closure that will be reconstructed by our scan_callback and then called, but this is not relevant here).

The thing is: Some(scan_callback) is definitely not a function pointer, and does not have the same size as a function pointer.

Here are the arguments that rust pushes on the stack before doing the FFI call:

[esp + 0x1c] = 0xa (timeout)
[esp + 0x18] = 0x1c23efc0 (&p_callback)
[esp + 0x14] = 0xebd820 (fn scan_callback)
[esp + 0x10] = 0x0 (tag for variant "Some")
[esp + 0x0c] = 0x0 (flags)
[esp + 0x08] = 0x4 (mem len]
[esp + 0x04] = 0x019b57d4 (mem addr)
[esp + 0x00] = 0x143b04c8 (rules)

This is one stack slot more than expected by the C function yr_rules_scan_mem, because of the space needed for the Option's variant tag. Because of this, yr_rules_scan_mem sees all variables shifted by one slot:

callback is now 0x0 (the tag for variant Some), user_data is now callback, and timeout amounts to the address of user_data as seconds.

Officially, yr_rules_scan_mem is defined in rules.h as

YR_API int yr_rules_scan_mem(
    YR_RULES* rules,
    const uint8_t* buffer,
    size_t buffer_size,
    int flags,
    YR_CALLBACK_FUNC callback,
    void* user_data,
    int timeout)

where YR_CALLBACK_FUNC is

typedef int (*YR_CALLBACK_FUNC)(
    YR_SCAN_CONTEXT* context,
    int message,
    void* message_data,
    void* user_data);

Definitely no Option here. However, bindgen (I'm using the bundled bindings) has generated:

pub type YR_CALLBACK_FUNC = ::std::option::Option<
    unsafe extern "C" fn(
        context: *mut YR_SCAN_CONTEXT,
        message: ::std::os::raw::c_int,
        message_data: *mut ::std::os::raw::c_void,
        user_data: *mut ::std::os::raw::c_void,
    ) -> ::std::os::raw::c_int,
>;

No idea where this Option comes from.

Somehow this bug is miraculously not triggered in x64, either because the ABI is different (no stack slots, args are passed via registers), or because Option<unsafe extern "C" fn(...)> is properly optimized to be the same size as the function pointer, just like Option<NonNull<T>> is.

For now my conclusion is that bindgen is at fault here. I'll try to pinpoint the reason, and when I find it I'll open an issue on their repo linking to this one.

hi,

with "vendored" feature enabled, I got the following build error on Macos

/Users/../.cargo/bin/cargo build --color=always --message-format=json-diagnostic-rendered-ansi --package .. --bin... Compiling yara-sys v0.8.0 error: failed to run custom build command for yara-sys v0.8.0 Caused by: process didn't exit successfully: /Users/../working/../target/debug/build/yara-sys-040eb9438488a67e/build-script-build (signal: 6, SIGABRT: process abort signal) --- stderr WARNING: /Users/../working/../target/debug/build/yara-sys-040eb9438488a67e/build-script-build is loading libcrypto in an unsafe way Process finished with exit code 101

Hi,

Thank you very much for creating this awesome crate. I add the yara crate to the toml and compile it but I faced this error.

error: failed to run custom build command for `yara-sys v0.4.2`

Caused by:
  process didn't exit successfully: `C:\Users\*****\Desktop\yara-rust\rsyara-scanner\target\debug\build\yara-sys-ab4856629aaf30fc\build-script-build` (exit code: 101)
  --- stdout
  cargo:rustc-link-lib=dylib=yara

  --- stderr
  wrapper.h:1:10: fatal error: 'yara.h' file not found
  wrapper.h:1:10: fatal error: 'yara.h' file not found, err: true
  thread 'main' panicked at 'Unable to generate bindings: ()', C:\Users\***\.cargo\registry\src\github.com-1ecc6299db9ec823\yara-sys-0.4.2\build.rs:106:14
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  

Is there a way to solve this issue!

Thanks.

• Reworked build.rs to support macOS, Linux, and now OpenBSD
• Static link to all dependencies needed by typical Yara package (cuckoo needs jansson, file needs magic)
• Update docs and links to reflect Yara 3.10

Fix: https://github.com/Hugal31/yara-rust/issues/40

I only added build without tests, since I am having problems with test on MacOS: https://github.com/ikrivosheev/yara-rust/runs/3270599324?check_suite_focus=true#step:4:204

A proposal to address #19, tell me what you think of it.

I've renamed bindings/yara-3.X.rs to bindings/yara-3.X-unix.rs, and generated bindings/yara-3.X-windows.rs with bindgen. Looking at the diff, you'll see that besides private internal structure layout and types, in the actually exposed API only the functions that take in a fd differ.

I've updated the yara-sys/build.rs to use the right file at compile time.

And finally I've added two scenarios in .tarvis-ci.yml:

TARGET=x86_64-pc-windows-msvc   YARA_FEATURES=vendored,bundled-3_11
TARGET=x86_64-pc-windows-msvc   YARA_FEATURES=vendored,bundled-3_8

The second one is technically invalid, since it uses 3.8 bindings on a 3.11 lib built by cc (vendored), but because of backcompability it should work.

Hello @Hugal31 , sorry for my english, I will try to describe what I want to do.

1. I kept the old api for backward compatibility
2. Callback api provides more control over scanning
3. I am writing an engine that will be integrated with python. To avoid converting from a Vec to PyList, I want to immediately add Rule to PyList

If this PR is Ok, I can write Callback API for Scanner

Added support for building and linking vendored libyara (useful when you want to statically link libyara).

I tested the crate's tests (with vendored feature) on Windows 10, macOS, and Ubuntu.

lmk if there's anything else you want me to change or test

Problem was describe here: https://github.com/Hugal31/yara-rust/issues/74. After fix drop, I got: Segmentation fault (core dumped), because context.matches was NULL and I fix it too.

This is needed to calculate the offset of a match when using iterators over blocks.

See base at https://yara.readthedocs.io/en/v4.2.3/capi.html?highlight=YR_RULES#c.YR_MATCH

CallbackMsg::ImportModule should be extended to access YR_MODULE_IMPORT data, namely module_name and module_data(_size).

Some modules accept additional module data, e.g. cuckoo module accepts json, which enables extended yara scanning.

This functionality is already exposed in python bindings (https://github.com/VirusTotal/yara-python/blob/master/yara-python.c#L618) and it would be lovely to have this functionality in this awesome rust crate :)

The unit test rules::test::rules_scan_proc failed on a CI build with the configuration ubuntu-20.04,vendored,bindgen,nightly. I ran it again and it passed.

May be related to #25.

I tested my application and see a huge growth of memory! Files to reproduce error: 16025.tar.gz.zip (this is simple txt files)

Simple code to reproduce:

use std::os::unix::fs::MetadataExt;
use std::sync::Arc;
use threadpool::ThreadPool;
use yara::Compiler;

const WORKERS: usize = 4;
const RULES: &str = "..."; // path to rules
const FILE: &str = "...";  // path to files to scan

fn main() {
    let pool = ThreadPool::new(WORKERS);
    let compiler = Compiler::new().unwrap();
    let compiler = compiler.add_rules_file(RULES).unwrap();
    let rules = Arc::new(compiler.compile_rules().unwrap());
    loop {
        let mut counter = 0;
        println!("Start iter");

        for file in globwalk::GlobWalkerBuilder::new(FILE, "**")
            .build()
            .unwrap()
        {
            if let Ok(file) = file {
                counter += 1;
                if file.metadata().unwrap().size() == 0 {
                    continue;
                }
                let rules = rules.clone();
                pool.execute(move || {
                    let mut scanner = rules.scanner().unwrap();
                    scanner.set_timeout(60);
                    let _ = scanner.scan_file(file.path());
                });
            }
        };
        pool.join()
        println!("Finish iter, files={}", counter);
        std::thread::sleep(std::time::Duration::from_secs(7));

    }
}

What am I doing wrong?

• error message

$ cargo build
   Compiling yara-sys v0.6.1
   Compiling yara v0.6.1
error: could not find native static library `yara`, perhaps an -L flag is missing?

• Cargo.toml

[dependencies]
yara = "0.6.1"
yara-sys = { version = "0.6.1", features = ["bundled-4_1_0"] }

Hi,

We're trying to cross-compile this little program:

use yara::Compiler;

const RULES: &str = r#"
    rule contains_rust {
      strings:
        $rust = "rust" nocase
      condition:
        $rust
"#;

fn main() {
    let mut compiler = Compiler::new().unwrap();
    compiler.add_rules_str(RULES)
        .expect("Should have parsed rule");
    let rules = compiler.compile_rules()
        .expect("Should have compiled rules");
    let results = rules.scan_mem("I love Rust!".as_bytes(), 5)
        .expect("Should have scanned");
    assert!(results.iter().any(|r| r.identifier == "contains_rust"));
}

to Windows from Linux by issuing:

cargo build --bin thomas --target x86_64-pc-windows-gnu

[package]
name = "thomas"
version = "0.1.0"
authors = ["Sonny <[email protected]>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[[bin]]
name = "thomas"
path = "src/mymain.rs"

[dependencies]
yara = "0.4.3"
yara-src = "0.1.2+3.11.0"

We have set the correct inc dir: YARA_INCLUDE_DIR="../yara-3.11.0/libyara/include"

The error we get is:

warning: version requirement `0.1.2+3.11.0` for dependency `yara-src` includes semver metadata which will be ignored, removing the metadata is recommended to avoid confusion
   Compiling yara v0.4.3
error[E0308]: mismatched types
   --> /volumes/SoonrRepo/Users/tclausen/.cargo/registry/src/github.com-1ecc6299db9ec823/yara-0.4.3/src/internals/compiler.rs:115:13
    |
115 |             handle,
    |             ^^^^^^ expected `i32`, found *-ptr
    |
    = note:     expected type `i32`
            found raw pointer `*mut c_void`

error[E0308]: mismatched types
   --> /volumes/SoonrRepo/Users/tclausen/.cargo/registry/src/github.com-1ecc6299db9ec823/yara-0.4.3/src/internals/scan.rs:130:13
    |
130 |             handle,
    |             ^^^^^^ expected `i32`, found *-ptr
    |
    = note:     expected type `i32`
            found raw pointer `*mut c_void`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0308`.
error: could not compile `yara`

To learn more, run the command again with --verbose.