- contribute
- release process
- Upgrading the Tor Browser
The big picture
The Tails build system downloads a set of Tor Browser tarballs from a location specified in config/chroot local-includes/usr/share/tails/tbb-dist-url.txt, and compares their hash with previously verified ones found in config/chroot local-includes/usr/share/tails/tbb-sha256sums.txt.
Once released officially, Tor Browser tarballs can be found in
a permanent (?)
location.
However, when upgrading Tor Browser for an imminent Tails release, we
generally have to use Tor Browser tarballs that are under QA and not
officially released yet. So, we have to retrieve them from another,
temporary location, such as
http://people.torproject.org/~mikeperry/builds/. If we hard-coded
this temporary URL in tbb-dist-url.txt
, then our release tag would
only be buildable for as long the tarballs stay in that place, which
at best is a few months.
To solve this, we host ourselves the Tor Browser tarballs we need, and point to this permanent location for anything that we tag.
Still, one can set an arbitrary download location in
tbb-dist-url.txt
, which should provide all the flexibility needed
for development purposes, e.g. when importing an alpha version with no intention
to include it in a tagged Tails release, in which case a stable URL is
not needed and we can point directly to the upstream URL.
Occasionally, the new Tor Browser requires adjustments to the corresponding AppArmor profile: AppArmor policy.
Upgrade Tor Browser in Tails
Wait
To check if the desired version is available:
Set
TBB_VERSION
to the version of Tor Browser you are waiting for, e.g. 42.0.12.Run this script:
```sh ( set -eu typeset -a tbbUrls tbbUrls=( 'https://archive.torproject.org/tor-package-archive/torbrowser/?C=M;O=D' 'https://www.torproject.org/dist/torbrowser/?C=M;O=D' 'https://people.torproject.org/~'{boklm,gk,ma1,morgan,pierov}'/builds/?C=M;O=D' 'https://tb-build-0'{2,3}'.torproject.org/~'{boklm,ma1,morgan,pierov}'/builds/torbrowser/release/unsigned/' ) while true do if (for u in "${tbbUrls[@]}"; do echo "query ${u}" >&2 elinks -no-numbering -dump "$u" | sed -e "s+^+[${u//+/ }]+" ; done ) | grep -F -e "${TBB_VERSION?}" -e "$(date +%F)" -e "$(date +%F -d yesterday)" --color=always then echo FOUND notify-send "Release dir" 'found' fi sleep 10m done ) ```
Verify
Set TBB_DIST_URL
to the chosen URL, and set TBB_VERSION
to the desired Tor
Browser version, for example:
TBB_DIST_URL=https://people.torproject.org/~mikeperry/builds/4.5-build5/
TBB_VERSION="$(basename "${TBB_DIST_URL:?}")"
Set TBB_TAILS_TICKET
to the number of the ticket about updating Tor
Browser, and TBB_IMPORT_BRANCH
to the branch we'll work on, which
must include the -force-all-tests
suffix:
TBB_TAILS_TICKET=123456
TBB_IMPORT_BRANCH=${TBB_TAILS_TICKET:?}-tor-browser-${TBB_VERSION%%-*}-force-all-tests
Check out the new branch:
git checkout -b ${TBB_IMPORT_BRANCH:?} origin/stable
Set TBB_SIGNER_SUFFIX
to the nickname of the Tor Browser developer who signed
the sha256sums-unsigned-build.txt
file, preceeded by a dash; the detached
signature at $TBB_DIST_URL
is named
sha256sums-unsigned-build.txt.asc${TBB_SIGNER_SUFFIX}
:
TBB_SIGNER_SUFFIX='-boklm'
Note: Nowadays it's quite common that the detached signature is called sha256sums-unsigned-build.txt.asc
, with no suffix.
If that's the case, just set it to an empty string:
TBB_SIGNER_SUFFIX=''
Fetch the version's hash file and its detached signature, and verify with GnuPG:
(
set -e
gpg --refresh-keys EF6E286DDA85EA2A4BA7DE684E2C6E8793298290
for f in sha256sums-unsigned-build.txt{.asc${TBB_SIGNER_SUFFIX?},}; do
wget "${TBB_DIST_URL:?}/$f" -O "$f"
done
gpg --verify sha256sums-unsigned-build.txt{.asc${TBB_SIGNER_SUFFIX?},}
)
Build the list of tarballs we want and their hashes, so this information is available at build time, when the tarballs are fetched:
(
grep --color=never -E \
"\stor-browser-linux-x86_64-${TBB_VERSION%-build*}\.tar\.xz$" \
sha256sums-unsigned-build.txt
) > config/chroot_local-includes/usr/share/tails/tbb-sha256sums.txt
Set download url
If TBB_DIST_URL
supports http
Then update the URL to the one chosen above:
echo "${TBB_DIST_URL:?}" | sed "s,^https://,http://," > \
config/chroot_local-includes/usr/share/tails/tbb-dist-url.txt
We cannot use HTTPS: building Tails uses a caching HTTP proxy by default and there's no way such a proxy caches data without MitM'ing connections. However, it is of no consequence since we verify the checksum file.
Else
Nowadays, tarballs are usually put on a web server, such as
https://tb-build-*.torproject.org/
, that doesn't support http: it will
redirect to https.
If this is the case, you must put those tarballs on a server that allows us to use plain HTTP.
We cannot use such a server directly, so we need to (at least temporarily) self-host this.
If you can access torbrowser-archive.git
echo "${TBB_DIST_URL:?}" > \
config/chroot_local-includes/usr/share/tails/tbb-dist-url.txt
git commit config/chroot_local-includes/usr/share/tails/tbb-*.txt \
-m "Upgrade Tor Browser to ${TBB_VERSION:?} (refs: tails#${TBB_TAILS_TICKET})" && \
git show
Please refer to Self-hosted Tor Browser tarballs archive section.
If you can't access it
Download the relevant files:
(
grep --color=never -Eo \
'\stor-browser-linux-x86_64-${TBB_VERSION%-build*}\.tar\.xz$' \
sha256sums-unsigned-build.txt \
) | sed -re "s@^\s*@${TBB_DIST_URL:?}@" | xargs wget -c
Now, put them on some webserver that you can control. Let's say you uploaded them to
http://myserver.net/misc/tbb/
. Put that URL in config/chroot_local-includes/usr/share/tails/tbb-dist-url.txt
Commit
Lastly, commit:
git commit config/chroot_local-includes/usr/share/tails/tbb-*.txt \
-m "Upgrade Tor Browser to ${TBB_VERSION:?} (refs: tails#${TBB_TAILS_TICKET})" && \
git show
If this new Tor Browser is meant to be included in a Tails release, then that's not enough: as explained above, we need to host the corresponding tarballs ourselves, so read on the next section.
Sync with the upstream wrapper scripts
Adapt our config/chroot_local-includes/usr/local/bin/tor-browser
and/or
config/chroot_local-includes/usr/local/lib/tails-shell-library/tor-browser.sh
for recent changes made in the Tor Browser build Git repo
(git clone https://gitlab.torproject.org/tpo/applications/tor-browser-build.git
),
on the maint-X.Y-desktop
branch (if it does not exist, fall back to maint-X.Y
):
git log -p \
projects/firefox/abicheck.cc \
projects/firefox/start-firefox \
projects/browser/RelativeLink/start-browser
Then apply any relevant change, e.g. to:
environment variables;
commandline options passed to the
firefox
executable;required libstdc++6 version bumps; if there's been any change upstream, look for
abicheck
inconfig/chroot_local-hooks/10-tbb
and adjust that hook as needed.
We also document (as code comments that includes the analysis, in
config/chroot_local-includes/usr/local/lib/tails-shell-library/tor-browser.sh
)
each such environment variable (and similar) that we consider
irrelevant and skip, both to avoid others duplicating the analysis and
to make it explicit during the review so it can be validated.
Self-hosted Tor Browser tarballs archive
Initial setup
First, install git-annex.
Then, make sure you have an entry for git.tails.net
in
your ~/.ssh/config
. For details, see ISO_history.mdwn
in tails/rm.
Then, clone the metadata repository and initialize git-annex:
git clone gitolite@git.tails.net:torbrowser-archive.git && \
cd torbrowser-archive && \
git annex init
You now have a lot of (dangling) symlinks in place of the files that are available in this git-annex repo.
To synchronize your local git-annex metadata with the remote, run:
git annex sync --no-content
Set up environment variables
Make sure you still have the environment variables defined in the previous section set.
Make
TAILS_GIT_REPO
point to the main Tails Git repository checkout wheretbb-dist-url.txt
is being worked on, for example:TAILS_GIT_REPO="$HOME/tails/git"
Make
TBB_ARCHIVE
point to your local git annex working copy of our Tor Browser archive, for example:TBB_ARCHIVE="$HOME/tails/torbrowser-archive"
Import a new set of Tor Browser tarballs
Download and verify all the tarballs we need:
DL_DIR=$(mktemp --tmpdir -d "tor-browser-${TBB_VERSION}.XXXXXXXXXX") CHROOT_INCLUDES="config/chroot_local-includes" TBB_SHA256SUMS_FILE="${CHROOT_INCLUDES:?}/usr/share/tails/tbb-sha256sums.txt" TBB_DIST_URL_FILE="${CHROOT_INCLUDES:?}/usr/share/tails/tbb-dist-url.txt" cd "${TAILS_GIT_REPO:?}" && git checkout "${TBB_IMPORT_BRANCH:?}" TBB_TARBALLS_BASE_URL="$(cat "${TBB_DIST_URL_FILE:?}" | sed "s,^http://,https://,")" current_branch=$(git -C "${TAILS_GIT_REPO:?}" branch | awk '/^\* / { print $2 }') for branch in "$current_branch" ; do git -C "${TAILS_GIT_REPO:?}" show "$branch:${TBB_SHA256SUMS_FILE:?}" \ | while read expected_sha256 tarball; do ( cd "$DL_DIR" echo "Retrieving '${TBB_TARBALLS_BASE_URL:?}/${tarball:?}'..." curl --remote-name --retry 20 --continue-at - \ "${TBB_TARBALLS_BASE_URL:?}/${tarball:?}" ) done ( cd "${DL_DIR:?}" && \ git -C "${TAILS_GIT_REPO:?}" show "$branch:${TBB_SHA256SUMS_FILE:?}" \ | sha256sum -c - ) done
Import the tarballs into your local Git annex:
( cd "${TBB_ARCHIVE:?}" && \ mkdir "${TBB_VERSION:?}" && cd "${TBB_VERSION:?}" && \ git annex import --duplicate "${DL_DIR:?}/"* "${TAILS_GIT_REPO:?}/"sha256sums-* )
Commit and push your changes
(
cd "${TBB_ARCHIVE:?}" && \
git commit -m "Add Tor Browser ${TBB_VERSION:?}" && \
git annex sync --no-content && \
git annex copy --to origin -- "${TBB_VERSION:?}" && \
git annex sync --no-content
)
Adjust the URL in the main Git repository
(
cd "${TAILS_GIT_REPO:?}" && \
git checkout "${TBB_IMPORT_BRANCH:?}"
current_branch=$(git branch | awk '/^\* / { print $2 }')
for branch in "${current_branch:?}" ; do
git checkout "${branch:?}" && \
echo "http://torbrowser-archive.tails.net/${TBB_VERSION:?}/" > \
config/chroot_local-includes/usr/share/tails/tbb-dist-url.txt && \
git commit config/chroot_local-includes/usr/share/tails/tbb-dist-url.txt \
-m "Fetch Tor Browser from our own archive" && \
git show
done
)
Wait for the synchronization
Before you can build Tails using Tor Browser from our own archive, you need to wait for the tarballs to appear on https://torbrowser-archive.tails.net/?C=M&O=D. This can take up to 10 minutes:
(
url="$(cat config/chroot_local-includes/usr/share/tails/tbb-dist-url.txt)"
while [ -z "$(elinks -dump -no-references -no-numbering "$url" | grep 'tor-browser-linux-x86_64.*MiB')" ]
do
sleep 5
done
)
Push the updated branch
git push -o merge_request.create --set-upstream \
origin "${TBB_IMPORT_BRANCH:?}"
Clean up
(
cd "${TBB_ARCHIVE:?}" && \
git annex drop -- "${TBB_VERSION:?}" && \
git annex sync --no-content && \
rm -rf "${DL_DIR:?}" && \
cd "${TAILS_GIT_REPO:?}" && \
rm -f sha256sums-unsigned-build.txt{,.asc${TBB_SIGNER_SUFFIX}}
)