Build Fennec frontend fast with mach artifact!
Nota bene: this post supercedes Build Fennec frontend fast!
It’s easy! But there is a pre-requisite: you need to enable Gregory Szorc’s mozext Mercurial extension  first. mozext is part of Mozilla’s version-control-tools repository; run mach mercurial-setup to make sure your local copy is up-to-date, and then add the following to the .hg/hgrc file in your source directory:
[extensions] mozext = /PATH/TO/HOME/.mozbuild/version-control-tools/hgext/mozext
Then, run hg pushlogsync. Mercurial should show a long (and slow) progress bar . From now on, each time you hg pull, you’ll also maintain your local copy of the pushlog.
Now, open your mozconfig file and add:
ac_add_options --disable-compile-environment mk_add_options MOZ_OBJDIR=./objdir-frontend
(That last line uses a different object directory — it’s worth experimenting with a different directory so you can go back to your old flow if necessary.)
Then mach build and mach build mobile/android as usual. When it’s time to package an APK, use:
mach artifact install && mach package
instead of mach package . Use mach install like normal to deploy to your device!
After running mach artifact install && mach package once, you should find that mach gradle-install, mach gradle app:installDebug, and developing with IntelliJ (or Android Studio) work like normal as well.
The integration currently requires Mercurial. Mozilla’s release engineering runs a service mapping git commit hashes to Mercurial commit hashes; mach artifact should be able to use this service to provide automatic binary artifact management for git users.
mach artifact install is your main entry point: run this to automatically inspect your local repository, determine good candidate revisions, talk to the Task Cluster index service to identify suitable build artifacts, and download them from Amazon S3. The command caches heavily, so it should be fine to run frequently; and the command avoids touching files except when necessary, so it shouldn’t invalidate builds arbitrarily.
The reduction in build time comes from --disable-compile-environment: this tells the build system to never build C++ libraries (libxul.so and friends) . On my laptop, a clobber build with this configuration completes in about 3 minutes . This configuration isn’t well tested, so please file tickets blocking Bug 1159371.
Run mach artifact to see help.
I’m seeing problems with pip
Your version of pip may be to old. Upgrade it by running pip install --upgrade pip.
I’m seeing problems with hg
Does hg log -r pushhead('fx-team') work? If not, there’s a problem with your mozext configuration. Check the pre-requisites again.
What version of the downloaded binaries am I using?
mach artifact last displays the last artifact installed. You can see the local file name; the URL the file was fetched from; the Task Cluster job URL; and the corresponding Mercurial revision hash. You can use this to get some insight into the system.
Where are the downloaded binaries cached?
Everything is cached in ~/.mozbuild/package-frontend. The commands purge old artifacts as new artifacts are downloaded, keeping a small number of recently used artifacts.
I’m seeing weird errors and crashes!
Since your local build and the upstream binaries may diverge, lots of things can happen. If the upstream binaries change a C++ XPCOM component, you may see a binary incompatibility. Such a binary incompatibility looks like:
You should update your tree (using hg pull -u --rebase or similar) and run mach build && mach artifact install && mach package again.
How can I help debug problems?
There are two commands to help with debugging: print-cache and clear-cache. You shouldn’t need either; these are really just to help me debug issues in the wild.
This work builds on the contributions of a huge number of people. First, @indygreg supported this effort from day one and reviewed the code. He also wrote mozext and made it easy to access the pushlog locally. None of this happens without Greg. Second, the Task Cluster Index team deserves kudos for making it easy to download artifacts built in automation. Anyone who’s written a TBPL scraper knows how much better the new system is. Third, I’d like to thank @liucheia for testing this with me in Whistler, and /u/vivek for proof-reading this blog post.
In my blog post The Firefox for Android build system in 2015, the first priority was making it easier to build Firefox for Android the first time. The second priority was reducing the edit-compile-test cycle time. The mach artifact work described here drastically reduces the first compile-test cycle time, and subsequent compile-test cycles after pulling from the upstream repository. It’s hitting part of the first priority, and part of the second priority. Baby steps.
The Firefox for Android team is always making things better for contributors! Get involved with Firefox for Android.
- Wed 1 July 2015: Initial version.
- Mon 6 July 2015: fix typo in link to Vivek. Thanks, sfink!
|||I can’t find documentation for mozext anywhere, but http://gregoryszorc.com/blog/2013/07/22/mercurial-extension-for-gecko-development/ includes a little information. mach artifact uses mozext to manage the pushlog.|
|||The long (and slow) download is fetching a local copy of the pushlog, which records who pushed what commits when to the Mozilla source tree. mach artifact uses the pushlog to determine good candidate revisions (and builds) to download artifacts for.|
|||We should make this happen automatically.|
|||In theory, --disable-compile-environment also means we don’t need a host C++ toolchain (e.g., gcc targeting Mac OS X) nor a target C++ toolchain (e.g., the Android NDK). This is not my primary motivation but I’m happy to mentor a contributor who wanted to test this and make sure it works! It would be a nice win: you could get a working Fennec build with fewer (large!) dependencies.|
|||I intend to profile mach build in this case and try to improve it. Much of the build is essentially single-threaded in this configuration, including compiling the Java sources for Fennec. Splitting Fennec into smaller pieces and libraries would help, but that is hard. See for example Bug 1104203.|