View this in action at http://hab-wip.surge.sh

Definitely not perfect. Always more polish to do!

This affects the following issues:

• #3317
• #3316
• #3341

Ok,

It's not a totally massive addition, but I'm totally not a web developer. If anyone can pull this down and build it locally and check it out that would be good. I've verified its a reasonable experience.

As there are no blog articles in this PR in order to verify how all of the things look with an actual blog post you'll need to add some markdown files to the www/source/blog directory.

If there are any glaringly obscene things in my css or slim templates please let me know.

Ran into a situation yesterday where in one of our gossip rings some services had started reporting their sys.ip as 127.0.0.1.

For example:

            "ip": "127.0.0.1",
            "hostname": "<hostname>",
            "gossip_ip": "192.168.100.20",
            "gossip_port": 9638,
            "http_gateway_ip": "0.0.0.0",
            "http_gateway_port": 9631
          },

This caused bound services to render the wrong IP in configurations/hooks, breaking interaction. In one case, a single node had both PostgreSQL and a backend API. PostgreSQL was showing the correct IP, while the backend API was showing 127.0.0.1.

I was able to correct the issue by unloading and loading the effected services. This is a long running ring that has updates for services pushed into it via CI/Chef. Host OS is Centos 7.

I have some debug logs I can send on request (not wanting to post directly here as they are live servers).

One thing specifically needs to go into this branch before merge. The quicklink cards got boogered up somehow and I was unable to deduce what broke them.

Ideally, we need to also add continuous scroll and a pinned sidebar to the docs but that can go in after the refactor.

Signed-off-by: eeyun [email protected]

• [x] The OS (including version) where you are running any of the Habitat commands.

  • Ubuntu 16.04 LTS, Linux 4.4.0-38-generic, x86_64
  • hab 0.10.2/20160930230245

• [x] Debug/backtrace of the command you are trying to run. You can set the following environment variables before running the hab command to generate a trace:

  • https://gist.github.com/lilianmoraru/e242716d0e633fb9f6012caf329c4a72
  • After it loaded(you can scroll to the end for this) I did ls /hab/cache/ so you can see that there is no keys folder inside it. After that I exited and entered the studio again(less output there) and listed /hab/cache again.

• [x] Current Habitat environment variables where the hab command or supervisor is running. These can be gathered using:

  • Does not print anyhing(no variables except for AUTH).

• [x] If this is a key related issue, please include the list of files (including user/group permissions) in /hab/cache/keys and $HOME/.hab/cache/keys via ls -la.

  • ls -la /hab/cache/keys

  ls -la /hab/cache/keys
  total 32
  drwxr-xr-x 2 root root 4096 oct  8 09:41 .
  drwxr-xr-x 6 root root 4096 iul 23 10:34 ..
  -rw-r--r-- 1 root root   75 iul 17 15:20 core-20160612031944.pub
  -rw-r--r-- 1 root root   75 sep 15 00:54 core-20160810182414.pub
  -r-------- 1 root root   77 iul 23 10:30 lilian-20160723073001.pub
  -r-------- 1 root root  121 iul 23 10:30 lilian-20160723073001.sig.key
  -r-------- 1 root root   77 oct  8 09:41 lilian-20161008064147.pub
  -r-------- 1 root root  121 oct  8 09:41 lilian-20161008064147.sig.key
  

  • ls -la $HOME/.hab/cache.keys

  ls -la $HOME/.hab/cache/keys
  total 16
  drwxr-xr-x 2 root root 4096 oct  8 09:10 .
  drwxr-xr-x 3 root root 4096 oct  8 09:09 ..
  -rw-r--r-- 1 root root   75 oct  8 09:09 core-20160612031944.pub
  -rw-r--r-- 1 root root   75 oct  8 09:10 core-20160810182414.pub
  

I generate the keys with sudo and start the studio with sudo. I have always used it that way, it doesn't work just now.

I am following the lesson track as described at https://learn.chef.io/modules/try-habitat#/

I have created an origin named nick-oliver and all installation criteria are followed and all seem to be completing correctly, up to the point where I enter: curl -s 127.0.0.1 | grep 'The time is' The track says that there is supposed to be output, but when I run the curl command, I get no output.

I then attempt to do the hab pkg upload, entering the correct .hart file name in the ./results directory. At this point I get the error: No auth token specified

I am running CentOS Linux release 7.5.1804 (Core)

RUST_LOG=debug RUST_BACKTRACE=1 hab
DEBUG 2018-05-16T20:14:25Z: habitat_common::ui: InputStream(stdin): { is_a_terminal(): true }
DEBUG 2018-05-16T20:14:25Z: habitat_common::ui: OutputStream(stdout): { is_colored(): true, supports_color(): true, is_a_terminal(): true }
DEBUG 2018-05-16T20:14:25Z: habitat_common::ui: OutputStream(stderr): { is_colored(): true, supports_color(): true, is_a_terminal(): true }
DEBUG 2018-05-16T20:14:25Z: hab: clap cli args: ["hab"]
DEBUG 2018-05-16T20:14:25Z: hab: remaining cli args: []
DEBUG 2018-05-16T20:14:25Z: hab::analytics: Event: v=1&tid=UA-6369228-7&cid=526351ec-186a-49f6-a6cd-1636b2bd0840&t=event&aip=1&an=hab&av=0.55.0%2F20180321220925&ds=cli--hab&ec=clierror&ea=MissingArgumentOrSubcommand--hab--
DEBUG 2018-05-16T20:14:25Z: hab::analytics: Creating directory /home/nick.oliver/.hab/cache/analytics
DEBUG 2018-05-16T20:14:25Z: hab::analytics: Creating file /home/nick.oliver/.hab/cache/analytics/event-1526501665.479903065.txt
hab 0.55.0/20180321220925

Authors: The Habitat Maintainers <[email protected]>
-----------------------------------------------------
env | grep HAB | grep -v HAB_AUTH_TOKEN
HAB_ORIGIN=nick-oliver

~/.hab/cache/keys$ ls -lash
total 8.0K
   0 drwxrwxr-x. 2 nick.oliver nick.oliver  84 May 16 14:39 .
   0 drwxrwxr-x. 6 nick.oliver nick.oliver  59 May 16 15:05 ..
4.0K -r--------. 1 nick.oliver nick.oliver  82 May 16 14:39 nick-oliver-20180516193924.pub
4.0K -r--------. 1 nick.oliver nick.oliver 126 May 16 14:39 nick-oliver-20180516193924.sig.key

ls -lash ~/.hab
total 4.0K
   0 drwxrwxr-x.  4 nick.oliver nick.oliver   28 May 16 14:39 .
4.0K drwx------. 31 nick.oliver nick.oliver 4.0K May 16 15:03 ..
   0 drwxrwxr-x.  6 nick.oliver nick.oliver   59 May 16 15:05 cache
   0 drwxrwxr-x.  2 nick.oliver nick.oliver   21 May 16 14:38 etc

I would appreciate any assistance you can provide.

This change is designed to make a Studio even more temporary and ephemeral for build-related tasks. Now when performing a build using the hab-studio build subcommand, a directory local to your current working directory is created called ./results which will have a copy of the produces Habitat artifact on success.

Local artifact copy

A ./results/ directory is now created in the same directory you invoke hab-studio build or hab-studio enter from and contains a copy of the produced Habitat artifact file. Because the parent directory of ./results/ has been bind-mounted into the Studio's chroot, this means the the ./results/ directory will appear locally outside the Studio and will persist after a Studio has been torn down.

For example:

results
|-- core-bzip2-1.0.6-20160517193810-x86_64-linux.hart
|-- last_build.env
`-- logs
    `-- bzip2.2016-05-17-193809.log

Modifed build report

Additionally a very build report (building on #509) is also written to the ./results directory as ./results/last_build.env. The intended user of this report is a build automation process, CI/CD environment, etc. The format is shell-compatible variable assignments of the form:

<var>=<val>

Where <var> is a variable and <val> is its value, delimited with an equals sign. The current contents of this build report are as follows:

• pkg_origin - the artifact origin, such as core
• pkg_name - the artifact name, such as bzip2
• pkg_version - the artifact version, such as 1.0.6
• pkg_release - the artifact release, such as 20160517193810
• pkg_ident - the artifact's package identifier, such as core/bzip2/1.0.6/20160517193810

For example:

pkg_origin=core
pkg_name=bzip2
pkg_version=1.0.6
pkg_release=20160517193810
pkg_ident=core/bzip2/1.0.6/20160517193810
pkg_artifact=core-bzip2-1.0.6-20160517193810-x86_64-linux.hart

Note that the order of these variables will not be guarenteed, only that the variable names and their values will be present, one per line. Any tooling which consume this report should take measures to not assume any line ordering of the file's contents.

Build logs with hab-studio build

This change also puts a session recording-style log (captured via the script(1) program) in a ./results/logs subdirectory.

Habitat plan.sh subdirectory default convention

This change also introduces a happy-path directory naming convention if you choose to place your plan.sh and related files in a subdirectory under your project's root. In short, simply call the directory habitat/ and you're good to go.

The hab-plan-build.sh program takes an argument as before which is a path to a directory containing a plan.sh file, however on failure, it will additionally check for habitat/plan.sh before giving up.

An example might help to clarify. Let's assume we are creating Habitat packages for a Ruby application which exists in its own git repository (not important, but only that the directory we're sitting in contains the root of the application).

A simple "flat" style of arranging the Habitat-related files is:

.
|-- config
|   `-- env.rb
|-- default.toml
|-- hooks
|   `-- run
|-- lib
|-- plan.sh
`-- test

A "nested" style of arranging the Habitat-related files is:

.
|-- habitat
|   |-- config
|   |   `-- env.rb
|   |-- default.toml
|   |-- hooks
|   |   `-- run
|   `-- plan.sh
|-- lib
`-- test

Calling hab-studio build in the root of this project will produce identical results without having to specify the habitat/ path in the "nested" style.

The same applies to non-application Plans such as software/library Plans. There is no need for hooks, config, etc in these style packages so most of the time it is simple enough to put the plan.sh in the root of that project. However, the "nested" style also works here too. To each, their own.

It should also be noted that calling the build program inside a Studio (via hab-studio enter) also yields this behavior.

Signed-off-by: Fletcher Nichol [email protected]

Since the habitat repo has been split into habitat, builder, core, etc., we have some duplicated infrastructure. This has led to some issues such as the travis libsodium build being broken. Issues like this are likely to occur more in the future, so it would be helpful to find a sustainable way to share this code

Aha! Link: https://chef.aha.io/features/APPDL-51

with a node application with a MongoDB database.

This uses the default mongo DB of test - debating whether I should add in info about creating your own DB.

Signed-off-by: Nell Shamrell [email protected]

This is the first pass at adding a metadata storage to the Depot server

• Repo server stores metadata in lmdb, an embedded in-memory database
• Added PackageIdent struct which encapsulates functionality which was sprinkled throughout modules in the codebase for working with package identifiers. This struct is passed as an argument to many functions which previously took either four &strs or two &strs and two Option<&str>s
• Added DataStore struct which encapsulates the lmdb environment and all databases
• Added PkgDatabase which is populated from reading the metadata files of package archives
• Added PkgIndex which is a special database that allows sorted duplicate data to form an index. This database can be queried to find the latest version of a package using a cursor.
• Added ViewDatabase which contains the names of repositories and the packages which are a part of them. This allows us to segregate particular packages to only being available in certain environments while not duplicating data on disk for the actual package archive.
• Added data_store and data_object modules which contain an abstraction on top of the FFI layer of lmdb
• Add new command line option depot-repair
• Add new command line option repo-create
• Add new command line option repo-list
• Uploading a package which already exists on the depot will no longer result in a forced upload
• Moved package archive specific functions to package::archive module from package

Known Issues

• Starting multiple transactions from different threads is not currently working. Additional child transactions can be started, but not root transactions. This behaviour should be possible within LMDB.
• Cursor functions don't return useable values for keys, but the data of the current item is good to read and available

Previously, hab-sup --version would exit with 86. Now, we exit 0 for both --version and --help.

Note, however, we will still exit with an error if the help is displayed because of an error such as a bad subcommand or bad option.

You may ask, "Do we need to print the error in the case of HelpDisplayed or VersionDisplayed?" In the case of HelpDisplayed the answer is yes because the help output isn't actually output to stdout but rather to a buffer that is contained in the error. In the case of VersionDisplay the answer is "kinda". The version is printed to stdout but without a newline, so printing the empty error prints a newline, which is a bit dirty but kept the function simple.

E.g.: When running on Linux, "hab pkg promote foo/bar/1.2.3/20220102030405", if successful, will always report "Promoted package foo/bar/1.2.3/20220102030405 (x86_64-linux)" ...regardless of whether the package is Linux or Windows.

Converse is also true, when running on Windows, the platform reported is always x86_64-windows regardless of the package's actual platform.

The CORRECT package is pro/demoted, this is a report-only issue.

Bumps notify from 4.0.17 to 5.0.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps serde_yaml from 0.8.26 to 0.9.4.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Command that is running: hab pkg build . Setting pkg_source in a plan.sh file to either a git URL or a zipped file pkg_source="https://location/of/file.tar.bz2" and having do_download() overridden seems to not use the do_download() nor does it try to download the zipped file. The output is always:

Cloning into 'repo'...
Username for '[https://repo-location.com'](https://repo-location.com%27/): fatal: could not read Username for '[https://repo-location.com'](https://repo-location.com%27/): Success

• The OS (including version) where you are running any of the Habitat commands.

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

• Debug/backtrace of the command you are trying to run. You can set the following environment variables before running the hab command to generate a trace:

Running the hab command with the RUST env vars didn't product a stacktrace. I think maybe because it is just a failure to download from a git remote.

$ hab --version
hab 1.6.521/20220603154827

• [ ] Current Habitat environment variables where the hab command or supervisor is running.

   hab-studio: Exported: HAB_AUTH_TOKEN=[redacted]
   hab-studio: Exported: HAB_ORIGIN=[redacted]
   hab-studio: Exported: HAB_BLDR_URL=https://bldr.biome.sh/
   hab-studio: Exported: AWS_DEFAULT_REGION=[redacted]
   hab-studio: Exported: AWS_SECRET_ACCESS_KEY=[redacted]
   hab-studio: Exported: HAB_STUDIO_HOST_ARCH=[redacted]
   hab-studio: Exported: AWS_ACCESS_KEY_ID=[redacted]
   hab-studio: Exported: GITLAB_USER=[redacted]
   hab-studio: Exported: GITLAB_TOKEN=[redacted]

My goal was to have the hab pkg build . command pull from a remote private repo (using either the username:token option in the URL or the SSH clone option. It doesn't seem to matter what I put into the do_download() method or the pkg_source variable.

Summary

This is a proposal for introducing the idea of habitat package types. There will primarily be two types of packages:

• Standard Packages : These are currently the only type of packages we have. A standard package is built in a cleanroom environment and it is guaranteed to run in any environment once it has been built. This is possible because every dependency of a standard package is also another standard package.
• Native Packages: This is a new type of package. Native packages are built in the host environment that the habitat packaging tool gets run in. These packages transfer the responsibility of managing runtime dependencies to the user. A native package is essentially a thin wrapper around an existing application that makes use of hooks to define how to manage the application lifecycle.

Motivation

The Habitat Supervisor depends on .hart packages to define how a service must be managed through the use of hooks. As a result we have a hard dependency on the availability of habitat packages to be able to leverage the supervisor on a new platform. We would like to break this dependency to enable a separation between Habitat's packaging and Habitat's supervisor runtime capabilities. Users could make use of the Habitat Supervisor runtime without the need for a managed set of packaged dependencies for their desired platform (eg: ARM, Solaris, M1 Macs) shortening their on-boarding time.

Design

To enable support for native packages, the HAB_FEAT_NATIVE_PACKAGE_SUPPORT environment variable must be set to true.

To create a Native Package you must:

• Put # package_type: "native" as the topmost line of your plan.sh / plan.ps1 file.
• You cannot have the following plan parameters:
  • pkg_deps
  • pkg_build_deps
  • pkg_interpreters
• You must have a run hook defined for your package.
• You can use any interpreter for your hooks. You must use a shebang line at the start of all your hook files with an appropriate path to the interpreter you wish to use.
• There will be a new Metafile in a package called PKG_TYPE. It will contain native or standard to indicate the type of package.

Behavior Changes

• Invoking hab pkg build will automatically run the hab plan build script on the host system when it detects that the plan builds a native package.
• Invoking build within a studio environment for a native package plan will result in an error message suggesting the user to run hab pkg build outside the studio.
• Adding any unsupported plan parameter such as pkg_deps will result in a build time error message.
• Native Package hooks will have to ensure that the interpreter they use and any commands that are used in them are available in the runtime environment.

Design Issues Considered

• Supporting dependencies between native packages is likely to be error prone as the user has to ensure that they always build and run the package in an environment that is compatible with all dependencies. Hence we will also not be supporting native packages that are libraries at this time.
• We need to introduce a new concept here (ie: Package Type) in order to have a clean way to detect and prevent mixing up of packages meant to work in different environments.
• There is no way to have both a native and a standard package plan file in the same folder, due to the existing rules of writing plan files.
• We chose to use a magic comment in the header vs bash/ps variable because the latter would require us to parse and interpret the script to evaluate the package type. This is troublesome as we require to make a decision on whether to build the plan on the host system or within a studio environment prior to actually running the plan. Parsing a magic comment is a simpler and more reliable way to do this versus running the plan through an interpreter runtime for this purpose.

Things yet to be decided

• How will the builder APIs have to be modified to support this.