Push Duty

The pushing of the server rotates each week to another developer. Current rotation is:

  • bsilverberg

  • eviljeff

  • mat

Check out the Add-ons calendar for a list of events.

Before the push

The code that will go in production on Thursday is tagged on Tuesday at 09:00 PT. The following repositories are tagged:

Project Dependencies

Project dependencies are not tagged as part of the push duty responsibilities. If you’re working on a feature in a project that’s a dependency of a project e.g. addons-linter, then it’s your responsibility to make a release and update the project that consumes that dependency in time for the tag.

This way we can ensure that:

  • Dependency packages are built and released in time for the tag.

  • The new feature in the new version of a package has been validated on -dev.

Making multiple releases of a package during a weekly milestone is totally fine since this helps with testing smaller sets of changes.

Security Fixes

Security fixes for addons-server live on a separate private repository:

To make merging easier, when making a pull request against this repository, the remote branch should not be published to one’s fork but to the repository itself. Once the PR has been reviewed, it should not be merged right away. Instead, merging to main is part of push duty and happens right before tagging:

$ git checkout main
$ git pull
$ git fetch security
$ git merge security/<branch-name>
$ git diff upstream/main
$ git push upstream main
$ git push security main

Note

mozilla/addons-server-security main branch should never be pushed to directly without pushing to mozilla/addons-server main first - the two should always stay in sync.

This means the merge or edit buttons in github web UI must never be used in that repository.

Note

Here we are using upstream and security remotes, which point to mozilla/addons-server and mozilla/addons-server-security, respectively. If your configuration is different you can substitute upstream and security for whatever you call the mozilla/addons-server and mozilla/addons-server-security repositories’ remotes.

Tag the repos

Tags are of the format: YYYY.MM.DD,

Note

The date is the date of the push, not the date of tagging.

Note

Once addons-frontend has been tagged a new docker image will be built on CircleCI and is required to deploy to stage.

It’s usually the main branch that is tagged:

$ git checkout main
$ git pull
$ git tag 2015.09.10
$ git push upstream 2015.09.10

Note

Here we are using “upstream” as the remote. If yours is different you can substitute “upstream” for whatever you call the mozilla/addons-server repo remote.

Get a compare link from github to compare this tag to the last tag. Add that compare link to the push doc so that people can clearly see what is pushing.

If tagging the main branch can’t be done (some feature is already on main, but not ready for production), then the commits that need to be released should be cherry-picked

If you’re adding cherry-picks to a tag that already exists, it makes sense to create a new tag rather than overwrite the old one. The reason for this is that re-using a tag makes it less easy to see the process that was involved in arriving at that tag. Also, it’s entirely possible to make a mistake by using an old tag that exists locally rather than the newer version on the remote when tags are re-used.

When creating a new tag you can use the format YYYY.MM.DD-SUFFIX where suffix is a number that’s incremented with each revision. The first time this is done will look like this:

$ git checkout 2015.09.03
$ git cherry-pick <commit hash> # as many times as you need
$ git tag 2015.09.10-1
$ git push upstream 2015.09.10-1

And the second:

$ git checkout 2015.09.03-1
$ git cherry-pick <commit hash> # as many times as you need
$ git tag 2015.09.10-2
$ git push upstream 2015.09.10-2

Then update the push doc with the new comparison link for the updated tag.

Extension Workshop

The tag naming scheme for extension-workshop is a bit different. When tagging for stage, add the suffix -stage to the tag name. For example:

$ git checkout master
$ git pull
$ git tag 2022.02.10-stage
$ git push upstream 2022.02.10-stage

Note

When pushing to stage you must create a tag with the -stage suffix. If you create and push a tag without that suffix you will be deploying to production.

Push to stage

Our infrastructure automates pushing the tags to stage once the tags have been pushed up to the repository.

The only required step is to check that the tag has deployed by looking out for the automated push messages in the internal slack channel.

You can also check /__version__ or /__frontend_version__ on a given service which shows the currently deployed revision and tag e.g:

Note that for Extension Workshop, pushing a tag to upstream with a name including the -stage suffix, will automatically deploy the tag to stage. You should manually verify Extension Workshop on stage after the push has completed. Visit https://extensionworkshop.allizom.org and view any pages that have been changed since the last push to verify they exist and are rendering properly.

Extract locales

Once you are done pushing the tags to stage:

  • Run the ./bin/run-l10n-extraction command in addons-frontend repository (documentation).

  • Run the ./scripts/extract-l10n.sh command in addons-server repository.

Because the addons-server is meant to be used inside the docker container, it doesn’t have access to your git credentials, so you’ll need to enter the container, run the script, exit the container and push the branch created by script. The addons-frontend one does that for you, but in both cases you’ll need to open the pull request yourself.

Before the push

Keep an eye out for any blocking bugs. Add them to the etherpad and find someone to work on them.

Push

The tag is pushed to production by ops (wezhou), once approved by QA (Krupa), on Thursdays. It is the responsibility of the push hero to follow-up with QA and ops, and be around during the push for any unexpected issues.

Extension Workshop

We push to extension-workshop on prod manually by creating a pushing a tag without the --stage suffix. For example:

$ git checkout 2022.02.10-stage
$ git tag 2022.02.10
$ git push upstream 2022.02.10

You should manually verify Extension Workshop on prod after the push has completed. Visit https://extensionworkshop.com and view any pages that have been changed since the last push to verify they exist and are rendering properly.

Monitoring the push

The best places to monitor the results of the push are:

The site performance graphs should show no large spikes or changes. Sentry should show no new errors, although it will show intermittent errors and the occasional error as the push occurs.

Once complete

Create a new github document for the next push, for example:

https://github.com/mozilla/addons/blob/main/releases/2019/05/09.md

You can use this handy template:

# AMO Release Thursday 13th September 2018

This week's push hero is @GithubUsername

## Notable things shipping:

## Blockers:

## Cherry-picks:

<!-- Link to the actual commits, NOT merge commits. The commits need to appear
in chronological order so that `git cherry-pick` will apply them correctly. -->

## Pushing:

- https://github.com/mozilla/addons-server/compare/YYYY.MM.DD...YYYY.MM.DD
- https://github.com/mozilla/addons-frontend/compare/YYYY.MM.DD...YYYY.MM.DD
- https://github.com/mozilla/addons-code-manager/compare/YYYY.MM.DD...YYYY.MM.DD

## Before we push:

## Before we start:

## Before we promote:

## After we're done:

Set the topic of the AMO Matrix channel and #remora slack channel to include the new github doc link and the relevant nickname of next week’s push hero.

Future Goals

Move to continuous deployment and change the way this is done dramatically.