By mirroring traffic to and from your machine, mirrord surrounds your local service with a mirror image of its cloud environment.

Adds a more graceful exit when the directory does not exist.

If the desired behavior is any different (for example, printing the absolute path of the nonexistent directory), I can update with the absolute path.

After installing the mirrord CLI, It didn't turn available through my terminal.

# Installed the CLI
$ npm install -g mirrord
...

# Checked where it have been installed
$ which mirrord
/home/tom/.nvm/versions/node/v17.6.0/bin/mirrord

# Attempted to use it
$ mirrord -v
/usr/bin/env: ‘npx ts-node’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines

# Checked for ts-node
$ which ts-node
/home/tom/.nvm/versions/node/v17.6.0/bin/ts-node

# Confirmed that is available
$ npx ts-node
>

If further information is required for your investigation, don't hesitate to ask me for it.

Alright my initial PR is adding just a hook for open, but in the process I also ended up refactoring the select! macro usage.

Before this becomes an even bigger PR, I'm pushing it to get some reviews.

This PR closes #35 adding tests conforming to the suite provided in #50

Current state of the tests: validation tests:

• run the node application with mirrord exec in a forked process
• send GET, POST, PUT, and DELETE requests and keep a counter
• POST validates data.txt, PUT creates a file with the data received, DELETE reads the file created by PUT, validates the data and removes the file.
• if all operations succeed, we exit with EXIT_SUCCESS
• since only a single job and pod is created, we verify that these resources exist and are removed after agent timeout finishes.

namespace tests [agent]:

• create a new namespace
• run a binary with agent namespace assigned the new namespace
• check if the job and the pod exist in the created namespace within the timeout limit
• run a binary with agent namespace assigned a non-existent namespace
• make sure the process crashes
• verify no resources have been created in either the default or the bad namespace

namespace tests [pod]:

• create a new namespace
• run a binary with pod namespace assigned the new namespace in the form of an env var
• verify the job exists

[email protected]:~/mirrord$ target/debug/mirrord extract blah 
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', mirrord-cli/src/main.rs:70:52
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

It would be nicer to handle the Result type gracefully and return an error or maybe prompt the user whether they want to create the directory (only useful for CLI users)

When mirrord exec is called with a simple flask API (https://gist.github.com/infiniteregrets/ba753ffb53ff8b98b2dd8f5051323919), where port 80/any other port is explicitly specified, the application seems to ignore that and instead assign a different port. This could perhaps be due to a call to dup or fcntl with F_DUPFD/F_DUPFD_CLOEXEC flags where mirrord is not aware of the new sockfd and probably getsockname returns a different port to the flask app. So we might need to add detours for the mentioned system calls to bring new sockfds to mirrord book-keeping. More context: Strace of flask API: https://gist.github.com/infiniteregrets/09773cc3f1aee7983f95fc800ff8d969 Agent logs when enforcing port 3689 in the Flask Application: https://gist.github.com/infiniteregrets/7c42aeae359e3a55c756f2ba9c7e1213

Description: I went to test the extension with an app written in Golang which runs outside of my cluster's default namespace. After clicking on Start mirrord in the status bar, only pods placed in the default namespace are listed. Because of this, I wasn't able to proceed with the tests.

Notes: I'm not sure what useful information I could add here since the debugging mode never got to receive forwarded connections. But feel free to ask anything for I'm willing to help =)

Bug Description

mirrord-cli uses the "blocking" feature of reqwest: https://github.com/metalbear-co/mirrord/blob/255fbe55262019f282010951d9d6f8158ec2530d/mirrord-cli/src/main.rs#L193

but does not enable that feature in its toml file https://github.com/metalbear-co/mirrord/blob/255fbe55262019f282010951d9d6f8158ec2530d/mirrord-cli/Cargo.toml#L26

should look something like this (docs):

reqwest = { version = "0.11.10", features = ["blocking"] }

Steps to Reproduce

1. Clone main branch (as of this commit)
2. Open project in dev container (I'm using VSCode remote containers extension)
3. run cargo check or cd mirrord-cli && cargo check
4. 💥 (see backtrace)

Backtrace

if let Ok(client) = reqwest::blocking::Client::builder().build() {
   |                         ^^^^^^^^ could not find `blocking` in `reqwest`

Relevant Logs

No response

Your operating system and version

local: macOS Monterey version 12.4, container: Debian OS variant buster

Local process

N/A

Local process version

No response

Additional Info

If you have already been working with the source code it may be necessary to rebuild your container to simulate this issue.

I'm not sure if I would call it "cached", but after I solved this issue once "in passing" i was unable to make it happen again regardless of cargo clean ext.

The agent image once built is tagged as test:latest, this results in an Elapsed error if the image is not loaded to minikube by any chance, so we should not be hardcoding this. It would be nice to be able to specify the image as an environment variable and read from it, and do the same in the CI.

openat implementation missing.

Some of the changes in -agent come from @infiniteregrets branch (might be a bit outdated by now). Will update those as well after getting some review on file + refactors I made.

Bug Description

We currently don't hook close, meaning that if a program runs close on a socket we manage we don't delete it from our managed fds.

Steps to Reproduce

N/A

Backtrace

No response

Your operating system and version

N/A

Your Node version (node --version)

N/A

Additional Info

No response

Bug Description

mirrord's VSCode extension fails to debug the local process.

Steps to Reproduce

• Add this configuration to your launch.json:

 {
      "type": "lldb",
      "request": "launch",
      "name": "ops.py: Debug file ops",
      "program": "python3",
      "args": ["${workspaceFolder}/tests/python-e2e/ops.py"],
      "cwd": "${workspaceFolder}",
      "sourceMap": {},      
      "env": {
        "DYLD_INSERT_LIBRARIES": "${workspaceFolder}/target/debug/libmirrord_layer.dylib",
        "LD_PRELOAD": "${workspaceFolder}/target/debug/libmirrord_layer.so",
        "RUST_LOG": "warn,mirrord=debug",
        "MIRRORD_AGENT_IMPERSONATED_POD_NAME": "http-echo-deployment-64887f7cdb-4lwrs",
        "MIRRORD_AGENT_RUST_LOG": "warn,mirrord=debug",
        "MIRRORD_FILE_OPS": "true",
        "RUST_BACKTRACE": "short", 
        "MIRRORD_ACCEPT_INVALID_CERTIFICATES": "true",
        "MIRRORD_AGENT_IMAGE": "runtime"   
      }
    }

• Start debugging and choose the impersonated pod

Backtrace

2022-06-20T17:26:58.258050Z  INFO mirrord_layer: Initializing mirrord-layer!
2022-06-20T17:26:58.259302Z  WARN mirrord_layer::pod_api: Accepting invalid certificates
2022-06-20T17:26:59.597035Z DEBUG mirrord_layer::sockets: getpeername hooked
2022-06-20T17:26:59.597069Z DEBUG mirrord_layer::sockets: getsockname hooked
2022-06-20T17:26:59.597090Z DEBUG mirrord_layer::sockets: accept4 hooked
2022-06-20T17:26:59.597106Z DEBUG mirrord_layer::sockets: dup3 hooked
2022-06-20T17:26:59.597122Z DEBUG mirrord_layer::sockets: accept hooked
FFF
======================================================================
FAIL: test_lseek (__main__.FileOpsTest)
Seeks character by character in a file with Lorem Ipsum text in "/tmp" and verifies the concatenation of the text.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mehula/mirrord/tests/python-e2e/ops.py", line 27, in test_lseek
    self.assertFalse(self._check_path_exists_on_host(file_path))
AssertionError: True is not false

======================================================================
FAIL: test_openat (__main__.FileOpsTest)
Opens a directory i.e. "/tmp", then opens a file in temp using openat give the directory file desciptor for "/tmp".
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mehula/mirrord/tests/python-e2e/ops.py", line 43, in test_openat
    self.assertFalse(self._check_path_exists_on_host(file_path))
AssertionError: True is not false

======================================================================
FAIL: test_read_write_family (__main__.FileOpsTest)
Reads data from a file in "/tmp" and verifies the text expected is the same as the text written.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mehula/mirrord/tests/python-e2e/ops.py", line 14, in test_read_write_family
    self.assertFalse(self._check_path_exists_on_host(file_path))
AssertionError: True is not false

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=3)
[email protected]:~/mirrord$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS     AGE
http-echo-deployment-64887f7cdb-4lwrs   1/1     Running   7 (6d ago)   6d4h
mirrord-agent-kpjldmsafe-2zrr4          1/1     Running   0            3s
mirrord-agent-ykgeipboyw-2g4tj          1/1     Running   0            4s
[email protected]:~/mirrord$ kubectl logs mirrord-agent-kpjldmsafe-2zrr4
2022-06-20T17:27:00.470051Z DEBUG mirrord_agent: main -> Initializing mirrord-agent.
2022-06-20T17:27:00.470269Z DEBUG mirrord_agent::sniffer: packet_worker -> setting namespace
2022-06-20T17:27:00.470333Z DEBUG mirrord_agent::file: file_worker -> Setting namespace
2022-06-20T17:27:00.471149Z DEBUG mirrord_agent::runtime: set_namespace -> fd 12
2022-06-20T17:27:00.471166Z DEBUG mirrord_agent::sniffer: preparing sniffer
2022-06-20T17:27:00.471169Z DEBUG mirrord_agent::sniffer: prepare_sniffer -> Preparing interface.
2022-06-20T17:27:00.594518Z DEBUG mirrord_agent: start -> Connection accepted from 127.0.0.1:51646
2022-06-20T17:27:01.597675Z DEBUG mirrord_agent: peer_handler -> daemon_stream.next received a message None
2022-06-20T17:27:01.597730Z DEBUG mirrord_agent: start_agent -> Peer 0 closed
2022-06-20T17:27:01.597750Z DEBUG mirrord_agent: ClientMessage::Close -> peer id 0 sent close
2022-06-20T17:27:01.597779Z DEBUG mirrord_agent::sniffer: packet_worker -> setting ports []
2022-06-20T17:27:01.597783Z DEBUG mirrord_agent::sniffer: packet_worker -> empty ports, setting dummy bpf
[email protected]:~/mirrord$ kubectl logs mirrord-agent-ykgeipboyw-2g4tj
2022-06-20T17:26:59.111089Z DEBUG mirrord_agent: main -> Initializing mirrord-agent.
2022-06-20T17:26:59.111252Z DEBUG mirrord_agent::sniffer: packet_worker -> setting namespace
2022-06-20T17:26:59.111253Z DEBUG mirrord_agent::file: file_worker -> Setting namespace
2022-06-20T17:26:59.112627Z DEBUG mirrord_agent::runtime: set_namespace -> fd 12
2022-06-20T17:26:59.112638Z DEBUG mirrord_agent::sniffer: preparing sniffer
2022-06-20T17:26:59.112641Z DEBUG mirrord_agent::sniffer: prepare_sniffer -> Preparing interface.
2022-06-20T17:26:59.593772Z DEBUG mirrord_agent: start -> Connection accepted from 127.0.0.1:51642
2022-06-20T17:27:01.634394Z DEBUG mirrord_agent: peer_handler -> daemon_stream.next received a message None
2022-06-20T17:27:01.634527Z DEBUG mirrord_agent: ClientMessage::Close -> peer id 0 sent close
2022-06-20T17:27:01.634581Z DEBUG mirrord_agent::sniffer: packet_worker -> setting ports []
2022-06-20T17:27:01.634591Z DEBUG mirrord_agent::sniffer: packet_worker -> empty ports, setting dummy bpf
2022-06-20T17:27:01.634536Z DEBUG mirrord_agent: start_agent -> Peer 0 closed

### Relevant Logs

_No response_

### Your operating system and version

[email protected]:~/mirrord$ uname -a Linux mehul-machine 5.13.0-1029-azure #34~20.04.1-Ubuntu SMP Thu Jun 9 12:37:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

### Local process

python3 tests/python-e2e/ops.py

### Local process version

[email protected]:~/mirrord$ python3 -V Python 3.8.10

### Additional Info

_No response_

Right now the E2E uses something we found online (http echo) but that means that the TCP/HTTP session behavior can be different leading to odd reactions. I think building our own docker image for the test would be better. It should be something that accepts the same APIs that our tests use, have files that we expect to have, etc.

Long story: Right now we work on TCP level, and therefore if the application running locally and the application running remotely are different, one can close the TCP connection for each HTTP request and the other can reuse it. By using the same app we eliminate randomness.

The test https://github.com/metalbear-co/mirrord/blob/main/mirrord-agent/tests/blackbox.rs#L23 is currently being ignored, a fix would be to enable/disable file operations through a flag

With 2.2.0 release, we need e2e tests for the following functions:

• [x] open
• [ ] fopen
• [ ] fdopen
• [x] openat
• [x] read
• [ ] fread
• [ ] fileno
• [x] lseek
• [x] write

Bug Description

mirrord-cli in release (downloaded from the script/compiled binary) gets killed when executing the process on macOS

Steps to Reproduce

1. download release build
2. mirrord exec --pod-name test-pod python
3. kill

Backtrace

no backtrace :(

Relevant Logs

No response

Your operating system and version

21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:37 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T6000 arm64

Local process

python

Local process version

No response

Additional Info

No response