The Declarative Data Generator

Required Functionality Currently, users must initialize a "workspace" before using synth to generate data. The rationale for this was that synth would be able to write files for both import and possibly generation (e.g. with #33) and having a directory for itself would reduce the risk of accidentally overwriting files. However, both requiring a call to "synth init" and the generated directories, even if empty, make it harder to set up tests and worsen the user experience.

Proposed Solution Recognizing that we put the baby out with the bathwater, we should not require user interaction unless there's a problem. So at the very least we could remove the synth init command and subsequent check for the .synth directory and instead add a prompt to ask the user whenever overwriting a file. Even better, we could record the files and creation timestamp whenever synth creates a new file and not issue a prompt when a file of that list gets overwritten. We should however keep the list of generated files somewhat short; storing up to 1000 files should be enough for everybody (famous last words). When the list gets full, we could check for removed files, otherwise remove the oldest entries.

Use case Improve user experience, simplify testing and reproducibility (because the tests no longer need to clean up .synth folders).

Required Functionality

Proposing adding schema to synth import command to build a data model from your Postgres database schema. Currently it goes to public schema and works but need a way to specify schema. Also, when it finds non table objects in schema, it continues to fail. I had to remove those objects temporarily for import to work.

$ synth import tpch --from postgres://user:pass@localhost:5432/tpch Building customer collection... Building primary keys... Building foreign keys... Ingesting data for table customer... 10 rows done.

Proposed Solution Provide a switch for selecting schema from existing database. If you can also add functionality to add table section would be super. Besides while selecting schema, based on integrity constraints, if it can add dependencies to collection automatically would help save lots of development time.

Use case Have many schema in existing database and would like to select schema. It continues to fail when it encounters non table kind of objects. Also, adding integrity constraints to each collection is very laborious task.

Required Functionality Whenever synth panics, the user should get the option to send an automated bug message with another option to give their contact info so we can get back to them. This will help our users be more active when bugs occur and help us get more and better bug reports.

Proposed Solution A custom panic handler can get some information (synth version, operating system, etc.) to format and send to an endpoint we need to set up.

Use case Improve our reaction to bugs, make synth better faster

Required Functionality An import and export of CSV files.

Proposed Solution The CSV format has some surprising complexity. Nonetheless, an import could at least use headers (if any) of a CSV file and try to match certain things about the values (e.g. are they all digits?). The export should have an option to declare the delimiter and perhaps quoting / escaping.

Use case

• Creating a schema (that can later be refined) from a set of CSV files
• Generating a set of CSV files from a schema.

Describe the bug The telemetry builds fail because of a typo.

To Reproduce

1. Build with telemetry feature
2. See error

error[E0432]: unresolved import `crate::utils::META_OS`
  --> synth/src/cli/telemetry.rs:20:5
   |
20 | use crate::utils::META_OS;
   |     ^^^^^^^^^^^^^^^^^^^^^ no `META_OS` in `utils`

I will make a PR that fixes this. I will also have to make a new release.

1. Utilizes the weights when choosing the next variant.
2. Use swap_remove which have O(1) since we don't care about the order.

Small demo

synth/examples/random_variants on  hbina-ISSUE-149-utilize-weights-in-oneof on ☁️  (ap-southeast-1) 
❯ cargo run --bin synth -- generate random --size 1 | jq | rg '1' | wc -l
warning: /home/hbina/git/synth/synth/Cargo.toml: unused manifest key: target.i686-pc-windows-msvc.rustflags
warning: /home/hbina/git/synth/synth/Cargo.toml: unused manifest key: target.x86_64-pc-windows-msvc.rustflags
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s
     Running `/home/hbina/git/synth/target/debug/synth generate random --size 1`
9930

synth/examples/random_variants on  hbina-ISSUE-149-utilize-weights-in-oneof on ☁️  (ap-southeast-1) 
❯ cargo run --bin synth -- generate random --size 1 | jq | rg '2' | wc -l
warning: /home/hbina/git/synth/synth/Cargo.toml: unused manifest key: target.i686-pc-windows-msvc.rustflags
warning: /home/hbina/git/synth/synth/Cargo.toml: unused manifest key: target.x86_64-pc-windows-msvc.rustflags
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `/home/hbina/git/synth/target/debug/synth generate random --size 1`
70

Closes https://github.com/getsynth/synth/issues/149

Required Functionality synth as of now requires the namespaces to be in order when having dependencies (for example by foreign key relations in a database) between them. A topological sort would solve this.

Proposed Solution The most used algorithm is Kahn's which the toposort-scc crate already implements. An implementation could either (in accordance with the Apache-2.0 license) be derived from that or depend on the crate.

Use case With the topological sort and a function to know whether a namespace depends on another, we may even parallelize generation for multiple namespaces.This would also incidentally allow us to emit a good error message if the dependency graph between namespaces is cyclic.

Hi, I created a schema with 11 fields of different types. The PG table consist of a serial field + 11 fields with same names and types as the schema .

Now when I try generate to the postgres : synth generate --collection test_data --to postgres://admin:1122@localhost:5432/data ~/db_test/data/

I get the error as mentioned below : - Caused by: 0: db error: ERROR: VALUES lists must all be the same length 1: ERROR: VALUES lists must all be the same length

When I pipe the generated values to a normal file ...I noticed that some row have 10values...and some rows even have 9 values.. So not all the generated rows have 11 values. Why this happens ....how to solve ?

1. Schema (if applicable)

	"type":"array",
	"length":{
		"type":"number",
		"subtype":"u64",
		"constant":10000
	},
	"content":{
		"type":"object",
		
		"special_id":{
			"optional":false,
			"type":"number",
			"subtype":"u64",
			"range":{
				"low":1,
				"high":20000,
				"step":2
			}
		},
		"first_name":{"optional":false,"type":"string","faker":{"generator":"first_name"}},
		"surname":{"optional":false,"type":"string","faker":{"generator":"last_name"}},
		"nickname":{"optional":false,"type":"string","faker":{"generator":"sentence"}},
		"ages":{
			"optional":false,
			"type":"number",
			"subtype":"u64",
			"range":{
				"low":18,
				"high":45,
				"step":1
			}			
		},
		"department":{
			"optional":false,
			"type":"string",
			"pattern": "(Accounting|Managment|IT||Sales)"
		},
		"job":{
			"optional":false,
			"type":"string",
			"faker":{
				"generator":"job"
			}
		},
		"ident":{
			"optional": true,
			"type":"string",
			"uuid":{}
		},
		"email_address":{
			"optional":false,
			"type":"string",
			"faker":{
				"generator":"company_email"
			}
		},
		"company_web":{
			"optional":true,
			"type":"string",
			"faker":{
				"generator":"safe_domain_name"
			}
		},
		"active":{
			"type":"bool",
			"frequency":0.5
		}
	}	
}

Describe the bug One of the tests on .json from a tutorial is failing.

To Reproduce Steps to reproduce the behavior: Have the cachix github action run.

See error

---- docs_blog_2021_08_31_seeding_databases_tutorial_dot_md stdout ----
/build/source/synth/tmp/docs/blog/2021-08-31-seeding-databases-tutorial:224 has a JSON only code block that will be skipped
thread 'docs_blog_2021_08_31_seeding_databases_tutorial_dot_md' panicked at 'Unable to open the namespace "docs/blog/2021-08-31-seeding-databases-tutorial"

Caused by:
    0: at file docs/blog/2021-08-31-seeding-databases-tutorial/735.json
    1: Failed to parse collection
    2: expected `,` or `}` at line 4 column 38
should contain one of the following errors: {
    "unknown variant `date_time`",
}', synth/tests/docs.rs:75:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    docs_blog_2021_08_31_seeding_databases_tutorial_dot_md

Expected behavior No error was expected.

Environment (please complete the following information): Github Actions

Additional context The only change between failing and passing was a minor change to documentation. I don't know why it fails now.

Describe the bug Nightly changed it's API so .chain() is now .sources(). This broke compilation.

To Reproduce Steps to reproduce the behavior:

1. cargo +nightly build --bin synth
2. See error

|
44 |         let mut chain = original.chain().collect::<Vec<_>>();
   |                                  ^^^^^ `&(dyn StdError + 'static)` is not an iterator
   |
  ::: /home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/error.rs:31:1
   |
31 | pub trait Error: Debug + Display {
   | -------------------------------- doesn't satisfy `(dyn StdError + 'static): Iterator`
   |
   = note: the following trait bounds were not satisfied:
           `&(dyn StdError + 'static): Iterator`
           which is required by `&mut &(dyn StdError + 'static): Iterator`
           `(dyn StdError + 'static): Iterator`
           which is required by `&mut (dyn StdError + 'static): Iterator`

Expected behavior Successful compilation.

Additional context This can be fixed easily by changing .chain() to .sources(). I will make a PR fixing this soon.

Required Functionality Currently JSON is supported and CSV is going to be supported (https://github.com/getsynth/synth/issues/33), well JSON Lines is like CSV, but with the power of JSON, it's very used to export big chunks of data with columns with complex values not supported by CSV, and where JSON is inefficient. Because one line is one record like CSV, it's much efficient than JSON to import, and it's used among other things because it's really easy to build a JSON line from a JSON file that is filtered / post-processed with tools like jq.

Proposed Solution Support it, and because only one type of data like CSV is supported, either assume that the file only has one type of data, or allow to set as argument the name of the column that determine the type of data (this also applies to CSV). E.g. in CouchDB there are no tables or collections, everything is store in the same database, but by convention the field "type" is used to discriminate what is the type of the record: users, transactions, claims...

Use case As mentioned above, it's easy to generate and work with JSON Lines. E.g. here is a transformation example made with jq. :

$ echo '{ "transactions": [ {"tx":123, "amount": 100, "cust_id": 444}, {"tx":123, "amount": 100, "cust_id": 444} ]  }' \
    | jq '.transactions[]|{tx:.tx,val:.amount}' -c
{"tx":123,"val":100}
{"tx":123,"val":100}

Another example: although the documentation of mongoexport says it export to JSON, in reality the output format is JSON Lines.

Required Functionality Make some of Synth usable from typescript.

Proposed Solution Synth as a native Deno module, probably with the ability to call a Synth server to work on the browser.

Use case Graphgen has expressed interest in using some of our internals.

Required Functionality I want to use synth on a PostgreSQL DB, however, it does not feature a converter for bytea.

Proposed Solution Implement a converter for bytea.

Use case I want to import my database in synth, but I'm currently blocked. I'm not interested in a representative generated value for the columns, a default value would suffice.

Required Functionality I would appreciate for the functionality of new import - import from Swagger spec (and Async / CloudEvents if this is possible). That would help a lot with data generation for mocks/stubs and performance testing

Proposed Solution

Use case Generating mocks/stubs for performance testing of APIs and fast data for POC validations

Required Functionality Importing DB scheme is a complex task with many possible points of failure. I always believe that something is better than nothing - Allowing to ignore errors and import part of the scheme can become very handy. In some cases it would be even worth to fill the gaps manually until the DB will have better support.

Proposed Solution There are two main options as I see it, Synth may offer both:

• skip-failed-tables - build partial scheme, skipping the problematic tables and other tables dependent on them.
• ignore-errors - add the failed tables with missing data so the user can fill the missing. generate should validate that the user fixed the problems and exit accordingly.

Both options should be suggested to the user upon import failure, along with detailed report contains the problematic table/column name and cause - if the table was skipped because it dependent on problematic table - it should be specified in the cause

Use case

• Scan fails, the user see that the failure is related to two small tables and chooses to use skip-failed-tables as suggested in the error
• Scan fails, the user see that many tables are were skipped because they are depended in one problematic table. The user chooses to use the ignore-errors and fix the problematic table

Required Functionality When attempting to import a database the could be potentially more than a single issue. Exiting after the first import issue can be frustrating as a single issue may hide many others. The import should keep running, keep collecting errors and finally exit with a list of problems.

Proposed Solution

• Collect errors to array during the import
• Before dumping the scheme, check if the array is not empty
• Finally exit listing all the collected import errors

Use case I have few postgres DBs, I got unimplemented converter errors - one on each DB. I had to write small script that collects all columns types from all the DBs and look at Synth source to understand which other types are yet supported, as the first error is masking the others. Potentially, there could be other errors except unsupported columns, so the adaptation of Synth to my DBs could last for long time.