A project for automatically generating and maintaining Debian repositories from a TOML spec.

Overview

Debian Repository Builder

A simple utility for constructing and maintaining Debian repositories. Configuration of a repo is based on the directory hierarchy, and a TOML configuration file. Real world repos to demonstrate may be found bellow.

Key Features

  • Can import pre-built Debian archives from external sources
  • Can repackage pre-built Debian archives
  • Can build packages from various sources, using sbuild
  • Supports multiple components within a suite
  • Supports migrating packages between components
  • Processes Debian archives within the pool in parallel.
  • Generates distribution file archives in parallel
  • Generates Packages, Sources, Contents, and Release dist files.
  • Repos will be apt-file-compatible

Todo

  • Generate Appstream + Icon archives

Directory Structure

The root directory of a debrep-based repo will contain the following directories:

  • assets/: where files that need to be linked at build-time are stored
    • cache/: files which debrep downloads from external sources, and should be cached between runs
    • share/: files that can be shared across packages, and are specified in the TOML config
    • packages/: files which are automatically linked to the build before building
    • replace/${suite}/${component}/${arch}/package/files/: Repackage prepackaged archives
      • DEBIAN: control archive files to replace
      • data: data archive files to replace
  • build/${suite}/: debrep performs all builds within this directory.
    • Every file is linked / sourced here at build time.
    • After each successful build, files are moved into the repo.
  • debian/${suite}/: contains the debian configuration for each source package that needs one.
    • The directories within must have the same name as the source package they reference.
    • Each package directory contains the entire contents of the debian directory for that package.
  • metapackages/${suite/: place your metapackage.cfg equivs files in here.
    • On build, they'll be generated and placed into the repo.
  • record/${suite}/: keeps tabs on what source packages have been built
  • repo/: Contains the archive & associated dist and pool directories for each
  • suites/${suite}.toml: Configuration files for each repo to build.

Highly Parallel Distribution File Generation

Since this tool is written in Rust, one of the key focuses has been on making it do as much as it can in parallel, as fast as it can do it. It uses thread pools, parallel iterators, and state machines to achieve that goal. Each component of a suit; each architecture in those components; and each package in those architectures are all processed in parallel. Data from each archive is also processed in parallel, and the final stage of processing that data into information and writing it to various compressed archives is done in parallel as well. Our tool should be fast with large archives.

Source Building Support

Packages can be generated from sources so long as you provide the debian files necessary -- either by using existing debian files in the upstream archive or git repository, or by providing your own through a variety of means.

Components Support

Managing components are supported by this utility! There's currently a default_component variable for the config, which will designate where packages will be stored by default. The migrate subcommand can be used to move packages between components. After moving packages, the dist files will be re-generated.

Contents Generation

Tools like apt-file require the the repository stores Contents archives, which it will download and read from to find which packages contain what files in a repository. This tool will process and generate these files in parallel as it is also processing the Packages archives.

Repo Structure

This is what you can expect to see after a successful build. You may sync the dists and pool directories to your package server to make your repository accessible to other machines.

repo/
  dists/
    cosmic/
    bionic/
      Contents-amd64
      Contents-amd64.gz
      Contents-amd64.xz
      InRelease
      proposed/
        binary-amd64/
          Packages
          Packages.gz
          Packages.xz
          Release
      main/
        binary-amd64/
          Packages
          Packages.gz
          Packages.xz
          Release
        source/
          Sources
          Sources.gz
          Sources.xz
        Release
        Release.gpg
  pool/
    cosmic/
    bionic/
      proposed/
        binary-amd64/
          p/
            package/
              package...
      main/
        binary-amd64/
          p/
            package/
              package_version_amd64.buildinfo
              package_version_amd64.changes
              package_version_amd64.deb
              package-dbgsym_version_amd64.ddeb
        source/
          p/
            package/
              package_version.dsc
              package_version.tar.xz

Usage

Create / update a Debian repository

debrep build [ -f | --force ]
debrep build packages <PACKAGES>... [ -f | --force ]
debrep build pool
debrep build dist

Migrate packages between components

debrep migrate package1 package2 pacakge3 --from proposed --to main

Clean up old packages

debrep clean

Remove packages

debrep remove <PACKAGES>...
Comments
  • Fix VS Code packaging (missing fields in Packages archive)

    Fix VS Code packaging (missing fields in Packages archive)

    There are some optional fields that weren't being added to the Packages archive. This would cause packages that have those fields to give repeated update messages.

    opened by mmstick 3
  • support multi-architecture cross-compiling

    support multi-architecture cross-compiling

    hi @mmstick, first up thanks for a sweet tool for Debian packaging! :100:

    i'd like to use debrep for building Debian packages for a project PeachCloud :peach: :cloud: , which will run on (Raspberry Pi) ARM devices, so i'm keen to add cross-compilation support to deprep.

    here's a pull request that seems to work:

    • adds architectures field to suite config, which defaults to [amd64, i386]
    • runs sbuild for each architecture (in a iter.try_for_each)
    • architecture matches support all official Debian architectures: https://www.debian.org/ports/

    here's a demo using this to build a basic Rust package for amd64, i386, arm64, and armhf: https://github.com/peachcloud/peach-packages.

    happy to implement this feature however you want, let me know if anything is not okay, or if you want something done differently, or if i'm missing a better way to do this, etc. i kept the changes pretty minimal to get it working and start the conversation, but for example at the moment this compiles every architecture in series and then writes the record object without any architecture information, maybe you have a better idea.

    cheers! :smiley_cat:

    opened by ahdinosaur 1
  • Update file locations for records, logs, and debian packages to support suites

    Update file locations for records, logs, and debian packages to support suites

    Currently testing locally

    • Built source packages will now be recorded at /records/${suite}/package-name
    • Builds now occur at /build/${suite}/
    • Metapackages are now moved to /metapackages/${suite}/
    • Package logs will be retained for each suite at /logs/${suite}/package-name
    • Debian packaging data will now be separated at /debian/${suite}/package-name

    This will fix source builds on cosmic so that the CUDA packages will build on both bionic and cosmic separately.

    opened by mmstick 1
  • Sync packages from third party repos

    Sync packages from third party repos

    • Refactoring the config module to separate the source, repo, and direct structures.
    • Added a new repos field for handling repo-based sources.
    • Added a new repos module to repo::download for fetching sources from a repo.
    • Split the web crawling functionality out to a new url-scraper, url-crawler, and apt-repo-crawler crate

    Example

    [[repos]]
    repo = "http://ppa.launchpad.net/js-reynaud/kicad-5/ubuntu/pool/main/"
    version = { is = "ubuntu18.04" }
    arch = { not = "i386" }
    
    [[repos]]
    repo = "http://ppa.launchpad.net/freecad-maintainers/freecad-stable/ubuntu/pool/main/"
    version = { is = "ubuntu18.04" }
    arch = { not = "i386" }
    
    # Peek GTK application
    [[repos]]
    repo = "http://ppa.launchpad.net/peek-developers/stable/ubuntu/pool/main/"
    version = { is = "ubuntu18.04" }
    arch = { not = "i386"}
    
    # Updated thunderbird packages, and firefox-esr
    [[repos]]
    repo = "http://ppa.launchpad.net/mozillateam/ppa/ubuntu/pool/main/"
    version = { is = "18.04" }
    arch = { is = "amd64|all" }
    
    opened by mmstick 1
  • Fix package versions of sources built from git

    Fix package versions of sources built from git

    Debian versions require either the timestamp or revision count to be included in the version in order for packages to be properly incremented when building git updates. Timestamps can be fetched with git show -s --format=%ct, and revision count with git rev-list --count --first-parent HEAD.

    opened by mmstick 0
  • Support building deb packages from .dsc, fetching git branches and commits, & appending git commit revisions to git-based packages

    Support building deb packages from .dsc, fetching git branches and commits, & appending git commit revisions to git-based packages

    Closes #33

    • dsc files may now be specified as a possible source location variant
    • The git source location variant now supports specifying a branch and commit
    • Enable automatic changelog updates of git sources, to append the commit to the changelog

    Example

    [DEBUG] debrep: running "git" "-C" "/home/mmstick/Sources/repo-curated-free/build/cosmic/piserver" "rev-parse" "master"
    [DEBUG] debrep: running "dch" "-D" "cosmic" "-l" "~40a78a" "-c" "/home/mmstick/Sources/repo-curated-free/build/cosmic/piserver/debian/changelog" "automatic build of commit 40a78a"
    [INFO] debrep: building piserver at changelog version 0.3~40a78a1
    
    building ion-shell at changelog version 1.0.0-alpha0-cosmic2~0bff9e1
    
    [[source]]
    name = "ion-shell"
    version = "1.0.0-alpha0-cosmic2"
    
        [source.location]
        dsc = "https://launchpad.net/~mmstick76/+archive/ubuntu/${name}/+sourcefiles/${name}/${version}/${name}_${version}.dsc"
    
    [[source]]
    name = "ion-shell"
    build_on = "changelog"
    
        [source.location]
        git = "https://gitlab.redox-os.org/redox-os/ion/"
        branch = "master"
        commit = "0bff9e12"
    
    opened by mmstick 0
  • Extra Repos and Keys

    Extra Repos and Keys

    • This is required to support building Tensorflow 1.11 for CUDA 10.0
    • Also fixes extra sbuild package dependencies, when multiple versions exist

    Closes #15

    opened by mmstick 0
  • Select the correct package versions

    Select the correct package versions

    When generating dist files, older versions were getting picked up instead of new versions. This will use the version comparison function that the repo fetching config also uses, so that 10.0 is considered newer than 9.2.

    opened by mmstick 0
  • Fetch & build sources from dsc files

    Fetch & build sources from dsc files

    To better support importing software which has already been packaged for Debian.

    • [ ] Support fetching dsc files remotely with dget
    • [ ] Support building from locally-stored dsc / tar files
    • [ ] Support applying additional quilt patches
    enhancement bite-sized 
    opened by mmstick 0
  • Multi-suite configuration support

    Multi-suite configuration support

    The lone sources.toml file has been done away with to now offer a suites directory for storing toml configs for each suite to be built. No changes were made outside of the location for storing suite configrations. The new layout will be:

    suites/
        bionic.toml
        cosmic.toml
    

    The parallel-getter dependency has also been updated to use the version on Crates.io.

    opened by mmstick 0
  • Fix repo version comparison

    Fix repo version comparison

    • Fixes version comparisons, as found when attempting to backport rustc 1.28.0 from cosmic.
    • No longer assumes that received apt entries will be in order
    opened by mmstick 0
  • Git-based Continuous Packaging Integration

    Git-based Continuous Packaging Integration

    • [ ] Provide a GitHub / GitLab repo, or a list of individual repos
    • [ ] Provide a destination repo which we can compare against / submit to
    • [ ] Fetch each repo and their branches to the "CPI" system
    • [ ] Watch for new branches, and updated branches
    • [ ] Generate a list of repos that need to be built
    • [ ] Generate packages, and submit packages
    enhancement 
    opened by mmstick 0
  • Improve apt mirroring support

    Improve apt mirroring support

    It should be possible to:

    • Specify the base URL of a repo
    • Specify the suite to pull from
    • Fetch packages from the packages list(s) for that suite

    The implementation belongs in apt-fetcher.

    enhancement 
    opened by mmstick 0
  • Incremental update of Packages and Content files

    Incremental update of Packages and Content files

    [ ] Support capability to determine what packages were changed between the last dists update [ ] Insert details into the dists file at the correct location, rather than regenerating them from scratch

    enhancement help wanted bite-sized 
    opened by mmstick 0
  • Automatic changelog updates

    Automatic changelog updates

    • [x] Support automatically generating and appending an entry to a package's changelog (debchange)
    • [ ] Support repacking a remote deb archive with an updated changelog
    enhancement good first issue bite-sized 
    opened by mmstick 0
Owner
Pop!_OS
An Operating System by System76
Pop!_OS
`haproxy_autconfd` is a daemon that automatically assembles a HAProxy config and restarts HAProxy if the config changes

haproxy_autconfd Welcome to haproxy_autconfd ?? haproxy_autconfd is a daemon that automatically assembles a HAProxy config and restarts HAProxy if the

null 0 Nov 9, 2021
This project now lives on in a rewrite at https://gitlab.redox-os.org/redox-os/parallel

MIT/Rust Parallel: A Command-line CPU Load Balancer Written in Rust This is an attempt at recreating the functionality of GNU Parallel, a work-stealer

Michael Murphy 1.2k Nov 20, 2022
Your project’s nix-env [maintainer=@Profpatsch]

lorri https://github.com/nix-community/lorri lorri is a nix-shell replacement for project development. lorri is based around fast direnv integration f

Nix community projects 442 Jan 1, 2023
A simple and fast download accelerator, written in Rust

zou A simple and fast download accelerator, written in Rust Zou is a Snatch fork by @k0pernicus. Snatch is a fast and interruptable download accelerat

Antonin Carette 173 Dec 4, 2022
GTK application for browsing and installing fonts from Google's font archive

Font Finder This project is a from-scratch implementation of TypeCatcher in Rust. It is a GTK3 application for browsing through and installing fonts f

Michael Murphy 252 Dec 26, 2022
Linux Kernel Manager and Activity Monitor 🐧💻

Linux Kernel Manager and Activity Monitor ?? ?? The kernel is the part of the operating system that facilitates interactions between hardware and soft

Orhun Parmaksız 1.7k Jan 5, 2023
Utility that takes logs from anywhere and sends them to Telegram.

logram Utility that takes logs from anywhere and sends them to Telegram. Supports log collection from files, journald and docker containers. More abou

Max Eliseev 85 Dec 22, 2022
A safe and ergonomic alternative to rm

rip (Rm ImProved) rip is a command-line deletion tool focused on safety, ergonomics, and performance. It favors a simple interface, and does not imple

Kevin Liu 781 Jan 7, 2023
A simple, fast and user-friendly alternative to 'find'

fd [中文] [한국어] fd is a program to find entries in your filesytem. It is a simple, fast and user-friendly alternative to find. While it does not aim to

David Peter 25.8k Dec 30, 2022
Untrusted IPC with maximum performance and minimum latency. On Rust, on Linux.

Untrusted IPC with maximum performance and minimum latency. On Rust, on Linux. When is this Rust crate useful? Performance or latency is crucial, and

null 72 Jan 3, 2023
CLI utility to move (or rename) your files to a new location and redirect all of its symbolic links, to the new path

Move Links CLI utility to move (or rename) your files to a new location and redirect all of its symbolic links, to the new path (or name). Usage execu

Ben Mefteh 18 May 22, 2022
A tool to simplify reprovisioning a fresh OS. Installs packages and manages dotfiles.

[[TOC]] Comtrya This is better, yes? Warning This tool is ridiculously early in its development. I'm building out features as I migrate my dotfiles ov

David McKay 2 Dec 29, 2022
A tool to simplify reprovisioning a fresh OS. Installs packages and manages dotfiles.

Comtrya This is better, yes? Warning This tool is ridiculously early in its development. I'm building out features as I migrate my dotfiles over. Thos

Comtrya 272 Jan 7, 2023
Parallel finance a decentralized lending protocol built on top of the Polkadot ecosystem. Our unique approach will allow users to earn "double interests" from staking and lending their tokens simultaneously.

Parallel Finance A new Cumulus-based Substrate node, ready for hacking ?? Getting Started Follow these steps to get started with the Cumulus Template

parallel-finance 100 Dec 17, 2022
idf-env tool helps set up and manage ESP-IDF installations

idf-env Tool for maintaining ESP-IDF environment. Quick start Install serial drivers for ESP boards on Windows. Execute following command in PowerShel

Espressif Systems 19 Dec 4, 2022
A system handler to get information and interact with processes written in Rust

A system handler to get information and interact with processes written in Rust

Guillaume Gomez 1.1k Jan 3, 2023
It is a backup tool that creates backups and stores them on an object storage

Hold My Backup It is a backup tool that creates backups and stores them on an object storage. By default it uses minio but you can use AWS: S3 as well

Taylan Dogan 13 Feb 17, 2022
desktop automation, screen capture, input listen/simulation, asar compress, color picker for nodejs and electron based on rust

desktop automation, screen capture, input listen/simulation, asar compress, color picker for nodejs and electron based on rust. 基于 Rust / WASM 提供截图、取色、键鼠事件监听模拟、压缩解压、图像处理、获取已安装应用等跨平台功能的现代异步 Nodejs 模块,占用空间小, 安装便捷, 使用简单, 高性能, 资源占用极小, 可取代 iohook 和 robotjs

null 48 Dec 15, 2022
Reproducible builds, dev envs and deployments.

?? Toros An implementation of Nix in Rust. Syntax support: With NixEL Interpreter support: Int Binding (aliasing) Let-in (flat bindings without interp

Kevin Amado 35 Dec 23, 2022