Hi,

I tried to send my files to S3 but got the following error:

Full backtrace:

thread '' panicked at 'called Option::unwrap() on a None value', file/src/send.rs:72:34 stack backtrace: 0: 0x103726302 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd2e8cbde22b780fc 1: 0x1037484fa - core::fmt::write::hd6692086cdd356a7 2: 0x10371e6bc - std::io::Write::write_fmt::h6043124a2486acbb 3: 0x103727c5b - std::panicking::default_hook::{{closure}}::h87a12b8b06887dd7 4: 0x103727967 - std::panicking::default_hook::h7f68dad17e0bfaa4 5: 0x10372827f - std::panicking::rust_panic_with_hook::hd57427cbbfc3717a 6: 0x103728182 - std::panicking::begin_panic_handler::{{closure}}::h33aab6d04e2bba70 7: 0x103726798 - std::sys_common::backtrace::__rust_end_short_backtrace::h0e7a76f927db9964 8: 0x103727e8d - _rust_begin_unwind 9: 0x10377d193 - core::panicking::panic_fmt::hcf6f3c517c6f3cb3 10: 0x10377d077 - core::panicking::panic::h46977cf6deabee02 11: 0x103164a83 - core::ops::function::impls::<impl core::ops::function::FnOnce for &mut F>::call_once::h3ae85f760bd22e23 12: 0x1031cf9cb - <alloc::vec::Vec as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter::hb4f07769796b97ee 13: 0x103169a52 - xvc_file::send::cmd_send::h71b8f1414f8bdc26 14: 0x103184e87 - xvc_file::run::h6f27c3454c6ddc25 15: 0x102ff6786 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h86d8253b795cae5f 16: 0x102fd3323 - std::sys_common::backtrace::__rust_begin_short_backtrace::h07d3b68a3599eace 17: 0x102fc7d3c - core::ops::function::FnOnce::call_once{{vtable.shim}}::h8b20dc35e1a5fb59 18: 0x10372e1b7 - std::sys::unix::thread::Thread::new::thread_start::haa45038b11bc331d 19: 0x7fff203768fc - __pthread_start thread 'main' panicked at 'called Result::unwrap() on an Err value: Any { .. }', lib/src/cli/mod.rs:273:6 stack backtrace: 0: 0x103726302 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd2e8cbde22b780fc 1: 0x1037484fa - core::fmt::write::hd6692086cdd356a7 2: 0x10371e6bc - std::io::Write::write_fmt::h6043124a2486acbb 3: 0x103727c5b - std::panicking::default_hook::{{closure}}::h87a12b8b06887dd7 4: 0x103727967 - std::panicking::default_hook::h7f68dad17e0bfaa4 5: 0x10372827f - std::panicking::rust_panic_with_hook::hd57427cbbfc3717a 6: 0x1037281c3 - std::panicking::begin_panic_handler::{{closure}}::h33aab6d04e2bba70 7: 0x103726798 - std::sys_common::backtrace::__rust_end_short_backtrace::h0e7a76f927db9964 8: 0x103727e8d - _rust_begin_unwind 9: 0x10377d193 - core::panicking::panic_fmt::hcf6f3c517c6f3cb3 10: 0x10377d2f5 - core::result::unwrap_failed::ha988429942445917 11: 0x102fbe283 - xvc::cli::dispatch::h6cb0b458cd3e5b16 12: 0x102f75284 - xvc::main::h02643c3fbc259b82 13: 0x102f7518b - std::sys_common::backtrace::__rust_begin_short_backtrace::h6b49ecb8fbe72698 14: 0x102f751b8 - std::rt::lang_start::{{closure}}::h76c1ee7dd2921ad7 15: 0x10371890e - std::rt::lang_start_internal::ha5deaf08dab8765b 16: 0x102f752cf - _main

Remote tests use feature flags to prevent to run every time the test suite runs.

These test could be run manually, either by per remote or in total.

• [ ] Enter all relevant credentials as secrets to Github Actions
• [ ] Create a new CI file that has separate testing for each of the remotes, via matrix

• Remove xvc file fetch and merge its functionality as an option in xvc file bring.

• Add --checkout-as option to xvc file bring to pass to xvc file checkout.

• parse(from_occurrences) -> ArgMatches::Count

• command and subcommand #[clap(...)] -> #[command(...)]

• argument #[clap(...)] -> #[arg(...)]

This PR also makes changes to XvcStorageOperations trait to return a new struct in init, with an older Guid. This allows to reinit with an already existing directory.

• Removes unused dependencies with cargo udeps
• Upgrade all dependencies, including breaking serde_yaml and tokio
• Adds clean up to tests to delete the repository director

The default behavior for xvc file list should be to show actual info.

Add a --cached option to show the cached info.

Add a --diff option to show the differences between cached and actual info only.

Show versions of a specific file. It can show other metadata like dates and size as well.

In the future, other commands like recheck, copy, move may also receive version strings to operate on specific versions.

Tests on Github Actions fail sometimes for the different metadata reported.

For example, the tests for git-branches.md fails when run for xvc file list

---- expected: stdout
++++ actual:   stdout
   1      - C=[..]
        1 + C<	2022-12-07 18:57:05 	                  19	data.txt	        
   2    2 | 
stderr:

This test passes on my local mac. It looks the fs behavior to report the results are different per OS.

Entity counter record itself even if it's not changed. We need some kind of "dirty bit" mechanism to check whether to save to a new file.

This will prevent most of the spurious Git commits after read only commands.

This gist of the experiments is to compare them. We need a diff facility to compare inputs and results across runs.

We don't have to limit this comparison to defined outputs in the pipeline. Any file changed between two runs can be diffed.

There can be three types of diffs:

• Unstructured diffs: This is for binary files that we don't recognize. Only the content digest is reported.
• Structured diffs: For a file format that we can parse, we can report the individual differences across runs. JSON, YAML or any other format that we can parse for results can be reported as structured diff.
• Text diffs: This is for the source code files that may have lead to changes in other files.

The workflow is as follows:

• User has a bunch of files, source, params, data, model, etc.
• User modifies some of these manually. e.g. updating the source code.
• User modifies some of these with xvc exp run --input-param command.
• User runs a command (or pipeline) on the files.
• Xvc clones/rechecks/copies files from original to a directory in .xvc/exp/KEYWORD-RANDOMSTRING-TIMESTAMP directory.
• Xvc links the original cache.
• Xvc creates a .xvc-exp directory to store experiment specific data.
• Xvc modifies the files with the given modification option.
  • --input-param params.yaml params.my-param 123,124,135 creates 3 experiments, each changing params.yaml::params.my-param to a given value.
• Xvc runs the given command (or pipeline) in the directory
• Xvc stores the updated artifacts in the common cache, symlinking the results.
• User asks for results diffed from the original.
• Xvc compares each of the directories for the changed files.
• Xvc shows unstructured files digest strings.
• Xvc shows structured files changed values.
• Xvc shows text file diffs similar to Git.

All results must be reported in JSON. Tables may be built from this JSON.

The second facility xvc exp provides is to modify structured files quickly for each experiment.

xvc exp run --input-param file.yaml dict.key value1,value2,value3 will parse file.yaml, update dict.key with value1 and run an experiment, update with value2 and run another, update with value3 and run another.

xvc exp run --input-param file.json dict.key '0;5;100' will run experiments with 0,5,10,15,20,...,100 (inclusive).

Files to be modified are JSON, YAML1.2 and TOML files. (Anything serde can read/write is possible in theory.)

We can extend this functionality to regex. --input-regex file.txt 'my_var = (.*)' 0;0.1;1 updates $1 in regex with the values.

We can also use --command-template for this. xvc exp run --command-template 'python train.py ${{EXP_VALUE}}' 0;0.2;10 will run python train.py with parameters 0, 0.2, 0.4, .... in different experiments.

If there are more than one --input-param, --input-regex, --command-template parameters, we build permutations of values. xvc exp run --input-param file.yaml dict.key 1,2,3 --input-param another.yaml another.key 5,6,7 will run 9 experiments.

There may be three subcommands for xvc exp run.

• xvc exp run pipeline --name: (xvcerp) Runs a pipeline command with the given parameters. (xvc pipeline run --name)
• xvc exp run command 'cmd': Runs a generic command as experiment
• xvc exp run template 'cmd ${{EXP_VALUE_1}} ${{EXP_VALUE_2}} 1,2,3 4,5,6 runs a command by substituing values to the command string.

--input-param and --input-regex options are available to all three of these. Maybe instead of --input-param, it's better to use --update-param and --update-regex. Maybe we can merge these, but I don't like to have corner cases.

--keyword will set the KEYWORD portion of experiment names. By default, this is exp. User may want to set to a searchable name.

The updated params, and run commands are stored in .xvc-exp directory. It may contain the exact script that was run.