Depends on: https://github.com/torrust/torrust-index-backend/pull/87

I've created the basic scaffolding of the command. My plan is:

• Migrate all the table records.
• Insert torrents in DB from uploads dir.

Potential problems are:

• AUTOINCREMENTS
• Foreign keys

UPDATE 2022-11-30

Tasks

• [x] Transfer torrent_categories
• [x] Transfer torrent_user
• [x] Allow login for the transferred users using the Pbkdf2 for hashing the password
• [x] Use date and time in today_iso8601 function instead of only the date
• [x] Transfer torrent_tracker_keys
• [x] Remove unused content from dir upgrades. We do not need the MySQL stuff for this migration. There was no MySQL support for version v1.0.0
• [x] Transfer torrent_torrents. See below the subtasks for this task
• [x] Transfer torrent_torrent_files. It was not used in v1.0.0
• [x] Make registration_date optional and leave it empty for imported users
• [x] Add a new DB field imported_date to the torrust_users table with the importation date for user records imported with this command
• [x] Refactor
• [x] Add automated tests
• [x] Intensive testing
• [x] Write documentation explaining how to upgrade
• [x] Rename destiny DB to target DB.

Subtasks from issue #84

This rework is needed because @ldpr found a bug in the current version that affects the upgrader. We have to fix that bug and update the "upgrader".

• [x] Fix issue #84
• [x] Rework: remove the torrust_torrent_tracker_stats::tracker_url column. DISCARDED.
• [x] Populate table torrust_torrent_tracker_stats with one record per torrent.

Transfer torrents subtasks

• [x] Fill table torrust_torrents
• [x] Fill table torrust_torrent_files
• [x] Fill table torrust_torrent_announce_urls
• [x] Fill table torrust_torrent_info

Testing subtasks

Assertions for integration test:

• [x] torrust_users table
• [x] torrust_user_authentication table
• [x] torrust_user_profiles table
• [x] torrust_tracker_keys table
• [x] torrust_torrents table
• [x] torrust_torrent_files table
• [x] torrust_torrent_info table
• [x] torrust_torrent_announce_urls table

Some cases to test:

• [x] Torrent with pieces.

Using a single tracker URL or multiple URLs:

• [x] Torrent with announce (single tracker url).
• [x] Torrent with announce_list (multiple tracker URLs).

Containing one file or multiple files:

• [x] Torrent with one file.
• [x] Torrent with multiple files.

Add tests for the application behaviour changed:

• [x] User can use have the password hashed with "argon2id" or a "pbkdf2-sha256". Add a test for the "verify_password" function.

Things I'm not going to test:

• [ ] Torrent with MD5 checksum?. I do not know how to include this MD5 checksum in the torrent file.
• [ ] Torrent with root_hash (BEP-0030).
• [ ] The user registration date can be null for users that have been imported. We do not test it for MySQL since the upgrader is only for SQLite.

The tracker has an endpoint to generate new keys.

The response for that endpoint was like:

{
  "key": "IaWDneuFNZi8IB4MPA3qW1CD0M30EZSM",
  "valid_until": 1674804892
}

But now is like this:

{
  "key": "IaWDneuFNZi8IB4MPA3qW1CD0M30EZSM",
  "valid_until": 
    { 
      "secs": 1674804892, 
      "nanos": 423855037
    }
}

The change was introduced in this commit cab093c026cf253ba6f7208211953a25b3d1c91b by @da2ce7.

The struct:

pub struct AuthKey {
    pub key: String,
    pub valid_until: Option<u64>,
 }

changed to:

pub struct AuthKey {
    pub key: String,
    pub valid_until: Option<DurationSinceUnixEpoch>,
}

Since we automatically build the json objects exposed in the API, that change was exposed to API consumers.

The backend cannot parse the new response, and keys are not stored in the table torrust_tracker_keys. So the function TrackerService::retrieve_new_tracker_key is called every time we use the TrackerService.

There have been quite some database changes between V1 and V2 of the backend and this renders the database of a backend running V1 incompatible with backend V2. The biggest change we made to the database is the way torrents are stored. In V1, torrents were stored as actual .torrent files in a dedicated uploads folder on the backend server. As of V2, all torrents are completely stored in the database.

To make sure all V1 users can move to V2 of the backend, we want to provide some kind of migration script. This script would take a V1 database (dump) and torrent uploads folder as input and should be able to convert this data to a new database dump that is suitable for the backend V2.

We will start working on this script when we believe V2 is in a usable state.

Endpoint: http://0.0.0.0:3000/torrents?page_size=20&page=0&sort=UploadedDesc&categories=&search=

Expected response:

{
  "data": {
    "total": 2,
    "results": [
      {
        "torrent_id": 1,
        "uploader": "user01",
        "info_hash": "9e0217d0fa71c87332cd8bf9dbeabcb2c2cf3c4d",
        "title": "A Mandelbrot Set 2048x2048px picture",
        "description": "A beautiful Mandelbrot Set picture in black and white. \n - Hybrid torrent V1 and V2. \n - Single-file torrent. \n - Public. \n - More than one tracker URL. \n ",
        "category_id": 10,
        "date_uploaded": "2022-11-04 07:19:18",
        "file_size": 9219566,
        "seeders": 0,
        "leechers": 0
      },
      {
        "torrent_id": 2,
        "uploader": "user01",
        "info_hash": "0902d375f18ec020f0cc68ed4810023032ba81cb",
        "title": "Two Mandelbrot Set 2048x2048px pictures",
        "description": "Two beautiful Mandelbrot Set pictures in black and white. \n - Hybrid torrent V1 and V2. \n - Multiple-files torrent. \n - Private. \n - Only one tracker URL. \n ",
        "category_id": 10,
        "date_uploaded": "2022-11-04 07:19:18",
        "file_size": 9219566,
        "seeders": 0,
        "leechers": 0
      }
    ]
  }
}

Actual response:

{
  "data": {
    "total": 1,
    "results": [
      {
        "torrent_id": 1,
        "uploader": "user01",
        "info_hash": "9e0217d0fa71c87332cd8bf9dbeabcb2c2cf3c4d",
        "title": "A Mandelbrot Set 2048x2048px picture",
        "description": "A beautiful Mandelbrot Set picture in black and white. \n - Hybrid torrent V1 and V2. \n - Single-file torrent. \n - Public. \n - More than one tracker URL. \n ",
        "category_id": 10,
        "date_uploaded": "2022-11-04 07:19:18",
        "file_size": 9219566,
        "seeders": 0,
        "leechers": 0
      }
    ]
  }
}

How to reproduce

1. Run this backend test: cargo test upgrades_data_from_version_v1_0_0_to_v2_0_0 -- --nocapture. It's still only available on PR #77 77. That will produce a tests DB in thetests/upgrades/from_v1_0_0_to_v2_0_0/output dir.
2. Run the tracker
3. Run the backend using the previously generated database.
4. Run the frontend
5. Load this URL: http://localhost:8080/torrents/trending. You will see only one torrent A Mandelbrot Set 2048x2048px picture.
6. Click on the torrent name to go to the torrent details page and back to the previous list page. You will now see two torrents.
7. If you go to the DB and delete all records in the torrust_toprrent_tracker_stats, you will see only one result again.

Conclusion

There must be a problem with the SQL sentence. Maybe there is an inner join that uses that table. The torrent that does not appear is private if that helps find the issue.

Attachments

The bug. THe list with only one torrent.

The torrent detail page.

The correct list page with two torrents.

The stats table with one record. ONce you have that record, the list page is right.

I've successfully installed the three components of Torrust (tracker, index-backend, index-frontend) and have it all appearing in a client side browser, but can't log in.

My index-backend data.db has a table torrust_users which is empty, and I can populate it with data via the command line sqlite3 tool, but don't know how to hash a password correctly to be recognised as valid and let me log in.

Is there a command line tool to do this for me? Alternatively is there a simple openssl style command to generate the hashed password?

[moved from here torrust/torrust-index/issues/82]

this is a pretty painful bug, i actually thought there was an issue open already.

in short:

user logs in
user doesn't log out and keeps the same session active for X amount of weeks
eventually the session will time out (but they will still show as logged in) and any torrents they download won't work - they show as "400 error unauthenticated" - AKA no passkey.

Having users do this:

Delete the torrent from the client
Sign out from the site
Sign back in again and download the torrent fresh
re-add new torrent - this fixes it.

This would be a huge QOL improvement for us - again, thanks so much guys, you're the best

MySQL would be a better default database to use for the index. It has a lot more features than SQLite3 and can handle heavy traffic more efficiently. I think however that we should not yet drop support for SQLite3.

Running this command cargo install sqlx-cli I get this error:

Compiling radix_trie v0.2.1
error: failed to run custom build command for `openssl-sys v0.9.74`

Caused by:
  process didn't exit successfully: `/tmp/cargo-install9r6wn4/release/build/openssl-sys-b7846f0d7e417835/build-script-main` (exit status: 101)
  --- stdout
  cargo:rustc-cfg=const_fn
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  run pkg_config fail: "`\"pkg-config\" \"--libs\" \"--cflags\" \"openssl\"` did not exit successfully: exit status: 1\nerror: could not find system library 'openssl' required by the 'openssl-sys' crate\n\n--- stderr\nPackage openssl was not found in the pkg-config search path.\nPerhaps you should add the directory containing `openssl.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'openssl' found\n"

  --- stderr
  thread 'main' panicked at '

  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/sfackler/rust-openssl
  and include information about your system as well as this message.

  $HOST = x86_64-unknown-linux-gnu
  $TARGET = x86_64-unknown-linux-gnu
  openssl-sys = 0.9.74

  ', /home/josecelano/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-sys-0.9.74/build/find_normal.rs:191:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: failed to compile `sqlx-cli v0.6.0`, intermediate artifacts can be found at `/tmp/cargo-install9r6wn4`

Caused by:
  build failed

• Ubuntu: 21.10
• Cargo: cargo 1.60.0
• rustc 1.60.0

Link: https://torrust.com/torrust-web-backend/about/ On this line: https://github.com/torrust/torrust-index-backend/blob/main/README.md?plain=1#L37

## Documentation

You can read the documentation [here](https://torrust.github.io/torrust-documentation/torrust-web-backend/about/).

## Installation

It was added because the tracker API was changed, but that change was reverted here:

https://github.com/torrust/torrust-tracker/pull/109

and it is not used anymore.

When a new torrent is uploaded, we must update the tracker torrent stats in the torrust_torrent_tracker_stats table.

It worked from the UI but not for the DB migration script because the frontend makes a request to get the torrent info after the upload, and that endpoint updates the torrent tracker stats.

It must update the stats even if that endpoint is not called after uploading a new torrent.

Tasks

• [x] After uploading a new torrent, update its tracker stats.
• [x] Change SQL query for torrent list result if we allow torrents without stats.

Potencial refactors

• Extract TrackerApiClient from TrackerService
• Rename TrackerService::get_torrent_info to TrackerService::update_torrent_tracker_stats

I've only added the config file. You can run it locally if you want. I used the IDE to check typos, and this file is very useful for ignoring false positives.

In the future, we can force the check with a workflow if we want.

Add a .cspell.json file in the root dir:

{
  "version": "0.2",
  "language": "en",
  "ignorePaths": [
    ".vscode",
    "**/.git/**",
    "**/vscode-extension/**",
    "megalinter"
  ],
  "words": [
  ]
}

Same as we did for the tracker here: https://github.com/torrust/torrust-tracker/issues/111

this would go hand in hand with #74 & #73

for example, if a torrent gets uploaded, reported or deleted, a webhook is triggered so we have a way to keep track of user actions.

(sorry, i had an issue open for this before but couldn't find it, my apologies if it exists elsewhere, i might just be blind)

1. It would be great if we could hard-limit who can upload torrents to only uploaders/moderators/admins.

2. Currently, only the uploader can edit torrents they upload. if we could have a secondary group under admin that's able to edit/delete any/all torrents it'd be a great QOL improvement.

Further, allowing admin to edit all torrents would be nice too.