File "/home/CS/Documents/test/rust/venv/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 627, in load_checkpoint_in_model
raise ValueError(
ValueError: At least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`.

pipeline::new return an error when I try to create pipeline. the error is "invalid type: map, expected u32". (maybe something about the "diffusers.json" file) I use the python script "h2pyke.py" convert a model. here is the "diffusers.json" in the converted model: (I format it use vscode.)

{
    "pipeline": "stable-diffusion",
    "framework": "onnx",
    "tokenizer": {
        "type": "CLIPTokenizer",
        "path": "tokenizer.json",
        "model-max-length": 77,
        "bos-token": 49406,
        "eos-token": 49407
    },
    "feature-extractor": {
        "resample": 3,
        "size": {
            "shortest_edge": 224
        },
        "crop": [
            {
                "height": 224,
                "width": 224
            },
            {
                "height": 224,
                "width": 224
            }
        ],
        "crop-center": true,
        "rgb": true,
        "normalize": true,
        "resize": true,
        "image-mean": [
            0.48145466,
            0.4578275,
            0.40821073
        ],
        "image-std": [
            0.26862954,
            0.26130258,
            0.27577711
        ]
    },
    "text-encoder": {
        "path": "text_encoder.onnx"
    },
    "unet": {
        "path": "unet.onnx"
    },
    "vae": {
        "encoder": "vae_encoder.onnx",
        "decoder": "vae_decoder.onnx"
    },
    "hashes": {
        "text-encoder": "edf2f0ea013cd652905d00e50e89db1a",
        "unet": "ffbefc01413f4e2e1fb01ef8767d53d9",
        "vae-encoder": "9ceb9c41e1fee3c67f13e74c6e65570b",
        "vae-decoder": "a0c8bb5e2fe170feb647d38602db3a63",
        "safety-checker": "b88096c404ad8d6f52daba2995c87fe8"
    },
    "safety-checker": {
        "path": "safety_checker.onnx"
    }
}

My code is here:

let env = match OrtEnvironment::builder().with_name("Stable Diffusion").build() {
            Ok(e) => Arc::new(e),
            Err(e) => {
                simple_message_box::create_message_box(&format!("Cannot init ORT environment: {}", e), "Error");
                panic!("Cannot init ORT environment: {}", e);
            }
        };
        let scheduler = match DDIMScheduler::stable_diffusion_v1_optimized_default() {
            Ok(s) => Arc::new(s),
            Err(e) => {
                simple_message_box::create_message_box(&format!("Cannot init scheduler: {}", e), "Error");
                panic!("Cannot init scheduler: {}", e);
            }
        };
        let pipeline = match StableDiffusionPipeline::new(
            &env,
            "./models/",
            &StableDiffusionOptions::default()
        ) {
            Ok(p) => p,
            Err(e) => {
                simple_message_box::create_message_box(&format!("Cannot init pipeline: {}", e), "Error");
                panic!("Cannot init pipeline: {}", e);
            }
            
        };

And it panic with a message:"Cannot init pipeline: invalid type: map, expected u32".

When I am running script from https://github.com/pykeio/diffusers/blob/main/examples/stable-diffusion.rs . Its using up my CPU and system Memory. I don't see the GPU being utilized

I have the following setup:

May Cargo.toml looks like this

[dependencies]
pyke-diffusers = "0.1"

Output of nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 527.56       Driver Version: 527.56       CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:0A:00.0  On |                  N/A |
|  0%   50C    P5    38W / 220W |    988MiB /  8192MiB |     11%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Output of nvcc -v

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Fri_Dec_17_18:28:54_Pacific_Standard_Time_2021
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0

From C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include\cudnn_version.h version is 8.6.0

#define CUDNN_MAJOR 8
#define CUDNN_MINOR 6
#define CUDNN_PATCHLEVEL 0

I have successfully run CUDA workloads on this machine outside of this library

From documentation I converted the SD1.5 model as follows:

To convert a float32 model from HF (recommended for CPU inference):

 python3 scripts/hf2pyke.py runwayml/stable-diffusion-v1-5 ~/pyke-diffusers-sd15/

Then I ran the https://github.com/pykeio/diffusers/blob/main/examples/stable-diffusion.rs

May Cargo.toml looks like this

[dependencies]
pyke-diffusers = "0.1"

Ran the command from documentation

To convert a float16 model from HF (recommended for GPU inference):

python3 scripts/hf2pyke.py --fp16 runwayml/stable-diffusion-v1-5@fp16 ~/pyke-diffusers-sd15-fp16/

• [x] DDIM
• [x] DDPM
• [x] DPM-solver, multistep & singlestep (mostly complete: dynamic thresholding not implemented)
• [x] Euler a
• [x] Euler
• [ ] IPNDM
• [ ] k-DPM2-a
• [ ] k-DPM2
• [ ] Karras VE
• [x] LMS
• [ ] PNDM/PLMS
• [ ] ~~SDE-VE~~
• [ ] ~~SDE-VP~~
• [ ] VQ Diffusion

• [x] Stable Diffusion text-to-image
• [x] "Safe" Stable Diffusion
• [ ] Stable Diffusion image-to-image
• [ ] Stable Diffusion inpainting
• [ ] Stable Diffusion v2 upscaling
• [ ] Stable Diffusion v2 image variation

Pipelines to explore

May or may not be implemented depending on complexity.

• [ ] Stable Diffusion outpainting
• [ ] Latent Diffusion
• [ ] Latent Diffusion super-resolution
• [ ] Unconditional Latent Diffusion
• [ ] Cycle Diffusion
• [ ] Versatile Diffusion
• [ ] Dance Diffusion