Build Fennec frontend fast!

Wed 06 May 2015 / tagged: android, build system, fennec

Nota bene: this post has been superceded by Build Fennec frontend fast with mach artifact!

Quick start

Temporarily, you’ll need to pull the new mach command into your source tree:

$ hg pull -r 3c0fb13b77b8460e56a31054a5f005bb3e4cdda1 https://reviewboard-hg.mozilla.org/gecko/
$ hg up 3c0fb13b77b8460e56a31054a5f005bb3e4cdda1
$ hg rebase -d fx-team

It’s easy! Open your mozconfig file and add:

ac_add_options --disable-compile-environment

You might want to also use a different object directory:

mk_add_options MOZ_OBJDIR=./objdir-frontend

Then mach build and mach build mobile/android as usual. When it’s time to package an APK, use:

mach package-frontend

instead of mach package. Use mach install like normal to deploy to your device!

After running mach package-frontend once, mach gradle-install, mach gradle app:installDebug, and developing with IntelliJ (or Android Studio) work as well.

Disclaimer

This only works when you are building Fennec (Firefox for Android) and developing JavaScript and/or Fennec frontend Java code!

You have been warned.

Options

mach package-frontend takes a couple of options that let you control downloading:

--force-remote-binaries, -r

Download and use remote binaries without comparing local and remote timestamps.
--force-local-binaries, -l

Do not download remote binaries; use local cached binaries.

Troubleshooting

What version of the downloaded binaries am I using?

Look in ~/.mozbuild/package-frontend/stage/assets/fennec-*.json for build ID, timestamps, and additional configuration.

I’m seeing weird errors and crashes!

Since your local build and the upstream binaries may diverge, lots of things can happen. If the upstream binaries change a C++ XPCOM component, you may see a binary incompatibility. Such a binary incompatibility looks like:

E GeckoConsole(5165)          [JavaScript Error: "NS_ERROR_XPC_GS_RETURNED_FAILURE: Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]" {file: "resource://gre/modules/Services.jsm" line: 23}]

You should update your tree (using hg pull -u --rebase or similar) and run mach build && mach package-frontend again.

Technical details

There are two things happening here. The reduction in build time comes from --disable-compile-environment: this tells the build system to never build C++ libraries (libxul.so and friends) [1]. On my laptop, a clobber build with this configuration completes in about 3 minutes [2]. This configuration isn’t well tested, so please file tickets blocking Bug 1159371.

The new mach package-frontend downloads pre-built binaries [3], copies them into your object directory, and then runs the regular packaging code to include them in an APK. This new mach command is tracked at Bug 1162191. A successful run looks something like:

$ ./mach package-frontend
 0:00.23 wget --timestamping --quiet --recursive --level=1 --no-directories --no-parent --accept fennec-*.en-US.android-arm.txt http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/
 0:00.82 wget --timestamping http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/geckolibs-20150506030206.aar
--2015-05-06 16:20:13--  http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/geckolibs-20150506030206.aar
Resolving ftp.mozilla.org... 63.245.215.56, 63.245.215.46
Connecting to ftp.mozilla.org|63.245.215.56|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22891694 (22M) [text/plain]
Saving to: ‘geckolibs-20150506030206.aar’

100%[=======================================================================================>] 22,891,694  3.84MB/s   in 11s

2015-05-06 16:20:25 (1.93 MB/s) - ‘geckolibs-20150506030206.aar’ saved [22891694/22891694]
 0:12.20 unzip -u /Users/nalexander/.mozbuild/package-frontend/geckolibs-20150506030206.aar -d /Users/nalexander/.mozbuild/package-frontend/stage
 0:12.37 /Users/nalexander/Mozilla/gecko/objdir-droid/_virtualenv/bin/python /Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/action/process_install_manifest.py --no-remove --no-remove-all-directory-symlinks --no-remove-empty-directories /Users/nalexander/Mozilla/gecko/objdir-droid/dist/bin /Users/nalexander/.mozbuild/package-frontend/geckolibs.manifest
From /Users/nalexander/Mozilla/gecko/objdir-droid/dist/bin: Kept 12 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:12.51 /usr/bin/make -C . -j8 -s -w package
 0:13.00 make: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid'
 0:13.04 make[1]: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid/mobile/android/installer'
 0:13.57 make[2]: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid/mobile/android/installer'
 0:14.02 make[3]: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid/mobile/android/installer'
 <snip>
 0:29.88 make: Leaving directory `/Users/nalexander/Mozilla/gecko/objdir-droid'
 0:29.89 /usr/local/bin/terminal-notifier -title Mozilla Build System -group mozbuild -message Packaging complete

If the upstream binaries haven’t changed, new binaries aren’t downloaded, so it’s reasonable to use mach package-frontend as a drop-in replacement for mach package:

$ ./mach package-frontend
 0:00.23 wget --timestamping --quiet --recursive --level=1 --no-directories --no-parent --accept fennec-*.en-US.android-arm.txt http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/
 0:00.52 wget --timestamping http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/geckolibs-20150506030206.aar
--2015-05-06 16:27:02--  http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/geckolibs-20150506030206.aar
Resolving ftp.mozilla.org... 63.245.215.46, 63.245.215.56
Connecting to ftp.mozilla.org|63.245.215.46|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22891694 (22M) [text/plain]
Server file no newer than local file ‘geckolibs-20150506030206.aar’ -- not retrieving.
 0:00.58 unzip -u /Users/nalexander/.mozbuild/package-frontend/geckolibs-20150506030206.aar -d /Users/nalexander/.mozbuild/package-frontend/stage
 0:00.59 /Users/nalexander/Mozilla/gecko/objdir-droid/_virtualenv/bin/python /Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/action/process_install_manifest.py --no-remove --no-remove-all-directory-symlinks --no-remove-empty-directories /Users/nalexander/Mozilla/gecko/objdir-droid/dist/bin /Users/nalexander/.mozbuild/package-frontend/geckolibs.manifest
From /Users/nalexander/Mozilla/gecko/objdir-droid/dist/bin: Kept 12 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.72 /usr/bin/make -C . -j8 -s -w package
 0:01.25 make: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid'
 <snip>
 0:17.30 /usr/local/bin/terminal-notifier -title Mozilla Build System -group mozbuild -message Packaging complete

Conclusion

In my blog post The Firefox for Android build system in 2015, the first priority was making it easier to build Firefox for Android the first time. So I landed mach bootstrap for Fennec, and we turned an onerous process into a one-liner that works for most people [4]. The second priority was reducing the edit-compile-test cycle time. The work described here drastically reduces the first compile-test cycle time, and subsequent compile-test cycles after pulling from the upstream tree. It’s hitting part of the first priority, and part of the second priority. Baby steps.

The Firefox for Android team is always making things better for contributors! Get involved with Firefox for Android.

Discussion is best conducted on the mobile-firefox-dev mailing list and I’m nalexander on irc.mozilla.org/#mobile and @ncalexander on Twitter.

Changes

Fri 8 May 2015: Thanks @michaelcomella for pointing out a bad Bugzilla ticket number.

Notes

[1] In theory, --disable-compile-environment also means we don’t need a host C++ toolchain (e.g., gcc targeting Mac OS X) nor a target C++ toolchain (e.g., the Android NDK). This is not my primary motivation but I’m happy to mentor a contributor who wanted to test this and make sure it works! It would be a nice win: you could get a working Fennec build with fewer (large!) dependencies.

[2]	I intend to profile `mach build` in this case and try to improve it. Much of the build is essentially single-threaded in this configuration, including compiling the Java sources for Fennec. Splitting Fennec into smaller pieces and libraries would help, but that is hard. See for example Bug 1104203.

[3] By default, the downloaded binaries are stored in ~/.mozbuild/package-frontend. The remote binaries change frequently — several times a day — and are not yet purged from your local cache after a reasonable time. In future, we’ll download less (maybe once a day? maybe only according to the latest revision in your local tree?) and delete old binaries after some set time (one week?).

[4]	This was tracked by Bug 1108771. It’s made a huge difference, and I should know: I used to field build problems in #mobile every few days. Now, I field build problems every few months.

About Nick Alexander

Mathematician. Mozillian. Runner. Master of Disguise.