This replaces gitbook with mdbook.

It currently uses the mdbook-dtmo wrapper, which bundles plugins for mermaid and ToC. Binary releases for Linux, Mac and Windows are provided on the release page (though Windows untested). It's based on the latest git version of mdbook, which includes relevant bug fixes. I intend to update it to a stable released version of mdbook as soon as it gets a release.

Upstreaming/changing how we do the mermaid/toc processing is tracked in #186.

Known remaining issues:

• I had to add empty chapters for some parts, see commit https://github.com/mozilla/firefox-data-docs/commit/ffdc0b09fc81aa9cec0d398a37d6d20380680016
• I had to remove the top-level "Bug template" link and moved it to the "Contributing" chapter

Live view: https://badboy.github.io/firefox-data-docs/ Fixes #162, #149

I can hold this back until the currently open PRs are merged and rebase it on top of that or I can help with rebasing the PRs after the fact.

Added cookbook for telemetry events design best practices. RE this bug

Not sure where in the docs this should live, so put it into cookbooks by default.

DAU shows strong weekly seasonality. Some of this is due to users who are only present Monday-Friday (in their timezone). Weekday-only clients likely have a different usecase to all-week users: they may be on work or school computers. And post-COVID we saw a drop in weekday-only clients and an increase in all-week clients - so it’s worth tracking these separately.

“Weekday regulars v1” and “All-week regulars v1” are sub-segments of regular users v3: we do not try to classify non-regular users, because we may not have enough data points at hand to confidently classify a non-regular user as predominantly using the browser on weekdays. A broader usage threshold than “regular users v3” could have been chosen here, but nevertheless some threshold would have needed to be chosen: so for the sake of simplicity let’s just compute this for regular users v3.

Timezones: the submission_date of a ping does not necessarily match the local day that the browser was used. Monday in Australia starts on Sunday UTC. Therefore we should allow some flexibility around what is considered a “weekend”: for an individual client the weekend could be Friday/Saturday, Saturday/Sunday, or Sunday/Monday. Looking at Australian/NZ clients, we see that there are many more weekday-only users if we allow Sunday/Monday UTC to be considered the weekend, and not many more weekday-only users if we allow Friday/Saturday UTC to be considered the weekend. For US clients, the effect is reversed and shrunken.

The upshot is that we need to include Sunday/Monday UTC as “the weekend” so that we fairly represent AU/NZ weekday-only users. Including Friday/Saturday UTC as “the weekend” is less essential, but seems to add little noise to the segment so let’s just do it.

Out of the last 27 days, we can count the number of weekend days that a client was active, using the SQL snippet:

BIT_COUNT(
            cls.days_seen_bits & 0x0FFFFFFE & (
                ((udf.bits28_from_string('0000011000001100000110000011') << 14)
                    + udf.bits28_from_string('00000110000011')
                ) >> (8 - EXTRACT(DAYOFWEEK FROM cls.submission_date)))
        )

where cls refers to the clients_last_seen table. (We can add OR statements with copies of this snippet that add/subtract 1 from the day of week, to handle the other tizemones)

If we plot DAU for the weekday-only users (i.e. where the above snippet evaluates to 0), then we see a strongly-seasonal curve, albeit one that is nonzero on weekends because as usual we segment today’s data only on historical behaviour.

If we relax our strictness and allow one historical day of weekend use in the past 27 days, then we capture a lot more weekday use and only a little more weekend use (see weekday_only_dau vs almost_weekday_only_dau on https://sql.telemetry.mozilla.org/queries/71353/source#179227). If we allow two historical days of weekend use in the past 27 days, then we capture more weekday use but start to include substantial weekend use. And the complement of this segment (all-week regular users) still has a lot of weekly seasonality - so I think two days is too much.

Finally, we should decide whether to have segments both for “weekday regulars” and “all-week regulars”, or just define “weekday regulars” and let people compute “all-week regulars” as all clients who are Regular Users v3 but not a weekday regular. It’s probably going to be easier for people if we do both for them.

This is my initial pass on BigQuery documentation. Please r? and provide any suggestions and changes.

I think we need more examples but also believe those should probably live in the dataset documentation for the tables.

This document isn't ready to be submitted yet, but I want to get some early feedback. This doc keeps getting pre-empted by competing work and I have a suspicion that this document may trigger some larger process discussions. I'd like to get those conversations started now.

This needs a lot of organization, but most of the content is there. High level commentary welcome! Let's be sure to use reviewable since I'll likely revise this document a few times.

You can preview the rendered documentation here

This change is 

I will be shopping around this workflow to some Data Scientists for review. It is intended to empower them to explore new feature usage definitions in a manner that has a clear path to being included in clients_last_seen if it proves useful.

I'm coming to the conclusion that a general clients_all_time or feature_usage_all_time table is not feasible in the short term, but the technique is still useful for rapid prototyping of bit patterns that can be used to investigate new user segmentation, feature usage definitions, etc.

Relevant to the Data Warehouse daily aggregations sub-project.

This needs more work, probably, but this is the ~minimal edit to avoid pointing people at the Nightly table for now.

aside: I love editing pipe tables by hand; it's great

@SuYoungHong r?

Hey Su, I've updated this doc to reflect the current state of the Activity-Stream dataset.

Change highlights:

• Update all the document links to the firefox source as the Github repo has been archived
• Update the database and table names as now we've switched to Bigquery from Redshift
• Some caveats are now invalidated with the database migration
• Update the sample queries

Let me know what you think.

Thanks!

Create a place to build out definitions for standard metrics, as well as improving discoverability by bringing it to the top of the navigation and linking from the Getting Started page.

The definitions are based directly on GUD.

This is a cookbook that guides the reader through retention analysis. This is the most recent draft after an informal review by the majority of the Product Data Science Team.