Let's save the 3DS and Wii U eShop!

Overview

saveShop

The eShop for the 3DS and Wii U is closing down on 27 March 2023. Let's save it!

saveShop is a PC tool that scrapes the eShop for metadata and media files (screenshots/videos). A web-based game list is included to explore the scraped data.

Demo screenshot

Why?

The data presented on the eShop provides historical context to games that you wouldn't find by just playing them, such as game descriptions, trailers, user-ratings, and more.

These things present interesting background to a game's release:

  • What features were considered innovative at the time? What catch-phrases did the developers highlight?
  • What screenshots and trailers were released alongside the game release to show off the game's strengths?
  • Did the gaming community welcome or reject the title at the time? How widely was it played?

Sadly, for one reason or another, many preservation communities don't bother archiving this historical context. It would be a shame to lose it just because its significance wasn't widely understood at the time. With saveShop, your own personal archive is in close reach!

Legal disclaimer

This tool cannot be used to download games. The authors of this tool do not host public archives.

Use of this tool may be subject to legislation in your country and/or the country the accessed servers reside in. Terms of service may apply. Text and data mining may be considered legal for the purposes of pattern and trend analysis, but speak to your lawyer if in doubt about your legal situation.

Build & Usage

The Rust package manager Cargo is required to build saveShop. Run the following for usage instructions:

cargo run -- --help

For most subcommands, the server region(s) to fetch data from must be specified using --regions.

For full metadata access, you will need to provide the 3DS client certificate (see below). If video data is dumped (fetch-media --fetch-videos), saveShop can auto-convert moflex videos to mp4 using the convert-media subcommand (requires FFmpeg to be installed).

Incomplete runs can be resumed, but the initial rescan will take some time.

Extract 3DS client certificate

The 3DS client certificate ("ClCertA") is required to access metadata from Ninja servers. The certificate can be dumped in either of three ways. In all cases, the certificate must be converted to PEM format after dumping.

Option 1: Dump fully decrypted certificate from a 3DS using ccrypt

The most convenient option to dump the certificate is to run ccrypt on a 3DS.

Option 2: Dump certificate from a 3DS

Using GodMode9, navigate to CTRNAND/title/0004001b/00010002/content/00000000.app, select NCCH image options and then Mount image to drive. Enter the romfs directory in the mounted image. You should see two files, ctr-common-1-cert.bin and ctr-common-1-key.bin. Copy each file to the SD card and transfer it to a PC.

These files are AES encrypted using 3DS key slot 0xd. This is the same key you'd put into the slot0x0DKeyN line of an aes_keys.txt file used by Citra. To decrypt the data on PC, run these commands (where <aeskey_0x0d> must be replaced with the 32-hex-digit AES key):

dd if=ctr-common-1-cert.bin of=ctr-common-1-cert.bin.iv bs=1 count=16
dd if=ctr-common-1-cert.bin of=ctr-common-1-cert.bin.contents bs=1 skip=16
dd if=ctr-common-1-key.bin of=ctr-common-1-key.bin.iv bs=1 count=16
dd if=ctr-common-1-key.bin of=ctr-common-1-key.bin.contents bs=1 skip=16
openssl enc -aes-128-cbc -nosalt -d -in ctr-common-1-cert.bin.contents -K <aeskey_0x0d> -iv `xxd -p ctr-common-1-cert.bin.iv` > ctr-common-1-cert.dec
openssl enc -aes-128-cbc -nosalt -d -in ctr-common-1-key.bin.contents -K <aeskey_0x0d> -iv `xxd -p ctr-common-1-key.bin.iv` > ctr-common-1-key.dec

Option 3: Dump certificate from NUS

It's assumed you know what you're doing here. Don't forget to decrypt the data using the CLCertA's title key. Like in Option 2, this data must additionally be decrypted with AES key 0xd.

After dumping the certificate: Converting to PEM format

First, ensure the dumping process was successful using:

curl --cert-type DER --key-type DER --cert ctr-common-1-cert.dec --key ctr-common-1-key.dec --insecure https://ninja.ctr.shop.nintendo.net/ninja/ws/country/US

Then, combine the two decrypted files into a single PEM file:

openssl x509 -in ctr-common-1-cert.dec -inform DER -out ctr-common-1-cert.pem -outform PEM
openssl rsa -in ctr-common-1-key.dec -inform DER -out ctr-common-1-key.pem -outform PEM
cat ctr-common-1-cert.pem ctr-common-1-key.pem > ctr-common-1.pem
rm ctr-common-1-cert.pem ctr-common-1-key.pem

Again verify everything went smoothly using:

curl --cert ctr-common-1.pem --insecure https://ninja.ctr.shop.nintendo.net/ninja/ws/country/US

You can now pass this certificate to saveShop using the --cert option.

Viewing results

A web-app is included to explore scraped contents. Copy index.html to the directory you ran saveShop in and open a local HTTP server (e.g. by running python3 -m http.server). You should then be able to view the data by navigating to localhost:8000 in your web browser.

TODO

  • Improve Wii U eShop scraping
You might also like...
Save cli commands and fuzzy find them later
Save cli commands and fuzzy find them later

crow - cli command memorizer What is crow? | Installation | Usage | FAQ What is crow? crow (command row) is a CLI tool to help you memorize CLI comman

A diff-based data management language to implement unlimited undo, auto-save for games, and cloud-apps which needs to retain every change.

Docchi is a diff-based data management language to implement unlimited undo, auto-save for games, and cloud-apps which needs to save very often. User'

Create tasks and save notes offline from your terminal

Create tasks and save notes offline from your terminal

Save decryption/encryption and transfer utility for Automachef

automachef-save Automachef by HermesInteractive encrypts it's save files with the user's account ID (Steam, Epic) or a static key (GOG, Twitch). The I

Quickly save and retrieve values for shell scripts.

Quickly save and retrieve values for shell scripts.

Start and stop system for applications to save your budget on hourly billing VPS.

Start and stop system (STT) Start and stop system for applications to save your budget on hourly billing VPS. Service A service consists of start/stop

Save image from your clipboard 📋 as an image file directly from your command line! 🔥

Clpy 📋 Save copied image from clipboard as an image file directly from your command line! Note It works only on windows as of now. I'll be adding sup

A git command to quickly save your local changes in case of earthquake !

git-eq (aka git earthquake) Earthquakes are part of the daily life in many countries like in Taiwan. git-eq is a simple git command to quickly save yo

Slay the Spire save deobfuscator / reobfuscator

SpireSaver This is a command-line Slay the Spire save file deobfuscator / reobfuscator that I threw together in a couple of hours. I used andrewsnyder

Comments
  • Some Wii U directories have no content

    Some Wii U directories have no content

    Thanks for the tool! I tried to download the Wii U metadata for the DE region, but it's always running into a panic when it tried to download directory 1090723 (which seems to be empty).

    Fetching metadata for directory 1090723 (5 out of 53)
    thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/main.rs:540:56
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    

    Looks like saveShop doesn't like empty directories?

    My Rust coding expierence it very limited so unfortunately I can't do a proper PR myself. This is what I do in my local build but it feels and looks not really optimal

    diff --git a/src/main.rs b/src/main.rs
    index 8556ee3..c1212b7 100644
    --- a/src/main.rs
    +++ b/src/main.rs
    @@ -17,8 +17,8 @@ use serde::de::DeserializeOwned;
     use std::sync::Mutex;
     use once_cell::sync::OnceCell;
     
    -const shop_id: i32 = 1; // 3DS
    -// const shop_id: i32 = 2; // Wii U?
    +// const shop_id: i32 = 1; // 3DS
    +const shop_id: i32 = 2; // Wii U?
     
     // Used to avoid rate-limiting. Lower at your own risk.
     const FETCH_DELAY: time::Duration = time::Duration::from_secs(1);
    @@ -535,8 +535,29 @@ async fn handle_directory_content(client: &reqwest::Client, directory_id: &str,
             let mut file = File::create(format!("samurai/{}/{}/directory/paginated/{}%3Foffset%3D{}", locale.region, locale.language, directory_id, offset)).unwrap();
             write!(file, "{}", &resp)?;
     
    -        let doc: DirectoryDocument = quick_xml::de::from_str(&resp).unwrap();
    +        let tmp = quick_xml::de::from_str(&resp);
    +        if tmp.is_err() {
    +            full_list.push(resp);
    +            directory_info = Some(DirectoryDocument {
    +                directory : NodeDirectory{
    +                    id: "".to_string(),
    +                    name: "".to_string(),
    +                    icon_url: None,
    +                    banner_url: "".to_string(),
    +                    contents : None,
    +                }
    +            });
    +            break;
    +        }
    +        let doc: DirectoryDocument = tmp.unwrap();
     
    +        if doc.directory.contents.is_none() {
    +            if offset == 0 {
    +                full_list.push(resp);
    +                directory_info = Some(doc);
    +                break;
    +            }
    +        }
             let contents = doc.directory.contents.as_ref().unwrap();
     
             println!("  Directory contents {}-{}, {} total", offset, offset + contents.length.unwrap_or(contents.total) - 1, contents.total);
    @@ -918,8 +939,11 @@ async fn fetch_metadata(client: &reqwest::Client, locale: &Locale, args: &Args,
             println!("Fetching metadata for directory {} ({} out of {})", directory_id, index + 1, directory_ids.len());
             let directory: DirectoryDocument = handle_directory_content(&client, &directory_id, &locale).await?;
             let directory = directory.directory;
    -        assert!(directory.contents.is_some());
    -        for content in directory.contents.unwrap().content {
    +        if !directory.contents.is_some() {
    +            println!("###### Failed to fetch metadata for directory {}", directory_id);
    +        }
    +        let dummy = NodeContents { content: Vec::new(), length: Some(0), total: 0, offset: Some(0) };
    +        for content in directory.contents.unwrap_or(dummy).content {
                 match content.title_or_movie {
                     NodeTitleOrMovie::Title(title) => if !title_ids.contains(&title.id) { title_ids.push(title.id) },
                     NodeTitleOrMovie::Movie(movie) => if !movie_ids.contains(&movie.id) { movie_ids.push(movie.id) },
    
    
    opened by Maschell 6
  • Theme Shop support?

    Theme Shop support?

    Never thought about it until now, but it would maybe be a good idea to add support for scraping the Theme Shop. Correct me if I'm wrong, but I believe that it will also go down along side with the eShop

    opened by KLanausse 7
Owner
null
A bit like tee, a bit like script, but all with a fake tty. Lets you remote control and watch a process

teetty teetty is a wrapper binary to execute a command in a pty while providing remote control facilities. This allows logging the stdout of a process

Armin Ronacher 259 Jan 3, 2023
Terminal UI for leetcode. Lets you browse questions through different topics. View, solve, run and submit questions from TUI.

Leetcode TUI Use Leetcode in your terminal. Why this TUI: My motivation for creating leetcode-tui stemmed from my preference for tools that are lightw

Akarsh 8 Aug 10, 2023
koyo is a cli tool that lets you run commands as another user. It is similar to doas or sudo.

koyo is a cli tool that lets you run commands as another user. It is similar to doas or sudo.

null 3 Nov 27, 2021
ISG lets you use YouTube as cloud storage for ANY files, not just video

I was working on this instead of my finals, hope you appreciate it. I'll add all relevant executables when I can Infinite-Storage-Glitch AKA ISG (writ

HistidineDwarf 3.6k Feb 23, 2023
⚡️(cd with env) Is a configurable cd wrapper that lets you define your environment per directory.

⚡️cdwe (cd with env) A simple configurable cd wrapper that provides powerful utilities for customizing your envionment per directory. (For ZSH / BASH

teo 20 Aug 6, 2023
An anyrun plugin that lets you search NixOS options.

anyrun-nixos-options An anyrun plugin that lets you search NixOS options. how 2 build? nix build ... or cargo build optionally :) Configuration This p

Michał 4 Aug 24, 2023
Lets you tweak Assassin's Creed Mirage in various ways.

Mirage Tweaks Lets you tweak Assassin's Creed Mirage in various ways. Currently supports adjusting the eject height and sprint speed. Usage Get the la

Assassin's Creed Community 3 Nov 3, 2023
Uses the cardano mini-protocols to receive every block and transaction, and save them to a configurable destination

cardano-slurp Connects to one or more cardano-node's, streams all available transactions, and saves them to disk (or to S3) in raw cbor format. Usage

Pi Lanningham 16 Jan 31, 2023
Scriptable tool to read and write UEFI variables from EFI shell. View, save, edit and restore hidden UEFI (BIOS) Setup settings faster than with the OEM menu forms.

UEFI Variable Tool (UVT) UEFI Variable Tool (UVT) is a command-line application that runs from the UEFI shell. It can be launched in seconds from any

null 4 Dec 11, 2023
🤖 just is a handy way to save and run project-specific commands.

just just is a handy way to save and run project-specific commands. (非官方中文文档,这里,快看过来!) Commands, called recipes, are stored in a file called justfile

Casey Rodarmor 8.2k Jan 5, 2023