Hello I am trying to use winsfs to create 3d SFS and have so far been unable to get it to converge and output a sfs file. winsfs -v -t 20 popAtlAst.out.saf.idx popAtlOc.out.saf.idx popGulfAst.out.saf.idx > popAtlAst_AtlOc_GulfAst.sfs You mention that winsfs usually takes less than 10 epochs to converge (although perhaps this was only for 2d?) I recently had an aborted run where it had hit INFO [windowem] Finished epoch 619

Do you recommend adding the --max epoch parameter for 3d sfs?

Hello, I just installed winsfs on a cluster using the cargo thing and I ran into this error: error: invalid or unsupported SAF magic number (found '[73, 61, 66, 76, 34, 00, 00, 00]', expected '[73, 61, 66, 76, 33, 00, 00, 00]')

The example (SAFA and B) ran perfectly. I tried with different SAF files obtained from ANGSD (angsd version: 0.937-108-gbb2e2d7 (htslib: 1.14-9-ge769401)) in different populations and with different filters in ANGSD and I always got the same error. I googled but...

Note that I am not super confident in my dataset because 1) of some strange pattern in terms of depths (bumpy from 1 to 100 depth then going to the usual "normal" distribution like and 2) of the results of the realSFS which are also bumpy to some point (e.g. for one pop of 11 individuals: 477252479.919277 1275341.790921 1311721.463687 1150418.711882 1038184.248378 818146.03548 707177.638546 636220.983638 565128.098679 524291.968394 482128.873608 350877.267486 0 0 0 0 0 0 0 0 0 0 0 ) So I am increasing the iterations in realSFS (default parameter with 30 it and 500000 sites) but I also wanted to test whether winsfs gives different results... and that's where the error came in the game.

Thank you for any comment you may have.

I saw that there is now support for Fst calculation with the commit named "Add Fst". But I don't see how to get it from the program.

How to get the Fst values from 2 saf files from 2 populations and plot the results in R?

Also, when plotting the SFS values in R, is it required to remove the first and last values (e.g., barplot(winsfs[-c(1, length(winsfs))]), as this is what I've seen for the realSFS output)? (assume that the winsfs is the output of the program based on a saf file, the values are pasted below)

Folded:

winsfs = c(128459.496517, 37807.43237, 19595.541448, 13794.172787, 11914.055373, 
11591.036125, 10717.842869, 10659.416557, 10619.180001, 9878.407667, 
9693.903963, 9162.882716, 7863.178519, 9416.350486, 8264.15024, 
7746.684849, 7423.467346, 7521.058327, 7142.004797, 7026.281412, 
6516.114025, 6935.251862, 6729.684416, 6037.666072, 6045.952302, 
6472.325573, 6216.954428, 6035.96378, 5911.554285, 6503.866442, 
6635.114531, 10348.583203, 7124.833748, 5177.385221, 4065.965662, 
4468.497129, 3837.587458, 3920.093395, 4342.735254, 3935.650826, 
4049.177171, 3887.896187, 4008.457721, 4412.583478, 4270.295593, 
4998.608504, 3509.831612, 4812.221073, 4710.699949, 4670.200019, 
4608.496482, 4876.892382, 4554.280017, 5063.93489, 5305.382852, 
5178.426539, 6011.698312, 5845.579826, 6383.94782, 7191.321642, 
7156.951056, 8282.703951, 8816.750871, 9978.439809, 11285.247227, 
13067.576935, 24704.074097, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)

Thanks!

Hi Guys,

I am trying to install winsfs at our local Redhat cluster and I am experiencing some problems. Everything goes well until this it starts to Compiling winsfs-core v0.1.0.

I am just posting the error messages below to check whether this is something you have seen before:

cargo install --git https://github.com/malthesr/winsfs --root /projects/mjolnir1/apps/winsfs Updating git repositoryhttps://github.com/malthesr/winsfsInstalling winsfs-cli v0.6.0 (https://github.com/malthesr/winsfs#f1d0f0be) Updating git repositoryhttps://github.com/malthesr/angsd-io.gitUpdating crates.io index Downloaded crossbeam-channel v0.5.6 Downloaded crossbeam-utils v0.8.11 Downloaded crossbeam-deque v0.8.2 Downloaded unicode-ident v1.0.3 Downloaded os_str_bytes v6.2.0 Downloaded hashbrown v0.12.3 Downloaded proc-macro2 v1.0.43 Downloaded clap v3.2.16 Downloaded syn v1.0.99 Downloaded quote v1.0.21 Downloaded libc v0.2.127 Downloaded clap_derive v3.2.15 Downloaded crossbeam-epoch v0.9.10 Downloaded 13 crates (1.5 MB) in 8.00s Compiling autocfg v1.1.0 Compiling cfg-if v1.0.0 Compiling libc v0.2.127 Compiling version_check v0.9.4 Compiling once_cell v1.13.0 Compiling crossbeam-utils v0.8.11 Compiling proc-macro2 v1.0.43 Compiling cc v1.0.73 Compiling unicode-ident v1.0.3 Compiling quote v1.0.21 Compiling crc32fast v1.3.2 Compiling syn v1.0.99 Compiling rayon-core v1.9.3 Compiling scopeguard v1.1.0 Compiling adler v1.0.2 Compiling hashbrown v0.12.3 Compiling log v0.4.17 Compiling byteorder v1.4.3 Compiling ppv-lite86 v0.2.16 Compiling either v1.7.0 Compiling heck v0.4.0 Compiling os_str_bytes v6.2.0 Compiling bitflags v1.3.2 Compiling strsim v0.10.0 Compiling termcolor v1.1.3 Compiling textwrap v0.15.0 Compiling miniz_oxide v0.5.3 Compiling clap_lex v0.2.4 Compiling memoffset v0.6.5 Compiling crossbeam-epoch v0.9.10 Compiling indexmap v1.9.1 Compiling rayon v1.5.3 Compiling proc-macro-error-attr v1.0.4 Compiling proc-macro-error v1.0.4 Compiling simple_logger v2.2.0 Compiling crossbeam-channel v0.5.6 Compiling flate2 v1.0.24 Compiling libdeflate-sys v0.8.0 Compiling crossbeam-deque v0.8.2 Compiling getrandom v0.2.7 Compiling num_cpus v1.13.1 Compiling atty v0.2.14 Compiling rand_core v0.6.3 Compiling rand_chacha v0.3.1 Compiling rand v0.8.5 Compiling libdeflater v0.8.0 Compiling noodles-bgzf v0.12.0 Compiling clap_derive v3.2.15 Compiling angsd-io v0.1.0 (https://github.com/malthesr/angsd-io.git?rev=c9d36cd#c9d36cd6) Compiling winsfs-core v0.1.0 (/home/jsd606/.cargo/git/checkouts/winsfs-06f8d3aa94bc2f69/f1d0f0b/winsfs-core) error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/adaptors.rs:35:22 | 35 | impl<const N: usize, T, F, I> Em<N, I> for Inspect<T, F> | -----------------^--^--^- help: reorder the parameters: lifetimes, then types, then consts:<T, F, I, const N: usize>`

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/adaptors.rs:49:22 | 49 | impl<const N: usize, T, F, R> StreamingEm<N, R> for Inspect<T, F> | -----------------^--^--^- help: reorder the parameters: lifetimes, then types, then consts: <T, F, R, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/standard_em.rs:44:22 | 44 | impl<const N: usize, I> Em<N, I> for StandardEm | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <I, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/standard_em.rs:53:22 | 53 | impl<const N: usize, R> StreamingEm<N, R> for StandardEm | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <R, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/standard_em.rs:66:22 | 66 | impl<const N: usize, I> Em<N, I> for StandardEm | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <I, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/window_em.rs:18:37 | 18 | pub struct WindowEm<const N: usize, T> { | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <T, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/window_em.rs:17:10 | 17 | #[derive(Clone, Debug, PartialEq)] | ^^^^^ help: reorder the parameters: lifetimes, then types, then consts: <T: ::core::clone::Clone, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/window_em.rs:17:17 | 17 | #[derive(Clone, Debug, PartialEq)] | ^^^^^ help: reorder the parameters: lifetimes, then types, then consts: <T: ::core::fmt::Debug, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/window_em.rs:17:24 | 17 | #[derive(Clone, Debug, PartialEq)] | ^^^^^^^^^ help: reorder the parameters: lifetimes, then types, then consts: <T: ::core::cmp::PartialEq, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/window_em.rs:24:22 | 24 | impl<const N: usize, T> WindowEm<N, T> { | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <T, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/window_em.rs:40:22 | 40 | impl<const N: usize, T> EmStep for WindowEm<N, T> | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <T, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/window_em.rs:47:22 | 47 | impl<const N: usize, I, T> Em<N, I> for WindowEm<N, T> | -----------------^--^- help: reorder the parameters: lifetimes, then types, then consts: <I, T, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em/window_em.rs:73:22 | 73 | impl<const N: usize, R, T> StreamingEm<N, R> for WindowEm<N, T> | -----------------^--^- help: reorder the parameters: lifetimes, then types, then consts: <R, T, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em.rs:37:32 | 37 | fn inspect<const N: usize, F>(self, f: F) -> Inspect<Self, F> | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <F, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em.rs:46:30 | 46 | pub trait Em<const N: usize, I>: EmStep { | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <I, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/em.rs:95:39 | 95 | pub trait StreamingEm<const N: usize, R>: EmStep | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <R, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/saf/iter/sites.rs:46:22 | 46 | impl<const N: usize, T> IntoSiteIterator for T | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <T, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/saf/iter/sites.rs:98:22 | 98 | impl<const N: usize, T> IntoParallelSiteIterator for T | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <T, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/lib.rs:49:43 | 49 | pub(crate) trait ArrayExt<const N: usize, T> { | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <T, const N: usize>

error: type parameters must be declared prior to const parameters --> winsfs-core/src/lib.rs:63:22 | 63 | impl<const N: usize, T> ArrayExt<N, T> for [T; N] { | -----------------^- help: reorder the parameters: lifetimes, then types, then consts: <T, const N: usize>

error[E0658]: default values for const generic parameters are experimental --> winsfs-core/src/em/standard_em.rs:22:39 | 22 | pub struct StandardEm { | ^^^^^^^ | = note: see issue #44580 https://github.com/rust-lang/rust/issues/44580 for more information

error[E0658]: default values for const generic parameters are experimental --> winsfs-core/src/sfs.rs:95:49 | 95 | pub struct Sfs<const N: usize, const NORM: bool = true> { | ^^^^^^ | = note: see issue #44580 https://github.com/rust-lang/rust/issues/44580 for more information

error[E0658]: destructuring assignments are unstable --> winsfs-core/src/saf.rs:453:38 | 453 | Ordering::Less => (i, j) = (j, i), | ------ ^ | | | cannot assign to this expression | = note: see issue #71126 https://github.com/rust-lang/rust/issues/71126 for more information

Compiling clap v3.2.16 For more information about this error, try rustc --explain E0658. error: could not compile winsfs-core due to 23 previous errors warning: build failed, waiting for other jobs to finish... error: failed to compile winsfs-cli v0.6.0 (https://github.com/malthesr/winsfs#f1d0f0be), intermediate artifacts can be found at /tmp/cargo-installf3ERZa

Caused by: build failed`

Currently, only up to three dimensions are supported. More was requested as part of #3 , and would probably be a good thing to have.

It shouldn't be particularly difficult to add higher dimensions, but it'll need some testing to make sure it looks reasonable, doesn't regress in the future, etc. I'll look at adding this in the near future.

winsfs has been updated to work from the banded SAF representation that angsd now outputs by default, in addition to the old format. The internal representation, however, is not banded. That is, the values that are implicitly zero in the banded SAF files are filled in by explicit zeros when reading. This is correct, but it is inefficient in terms of run-time and memory usage for very large sample sizes.

I'll be adding an internal, banded SAF representation at some point to deal with this, and I'm opening this issue to track this work.