We should build Firefox for Android with an external build system
For some time, I have been advocating for building Fennec with a full-featured Android build system. At one time, the state of the art was Ant; now, the solution blessed by Google is Gradle [1]; both Facebook (buck) and Twitter (pants) have built and published open-source build systems; and Google (again!) has built a separate proprietary internal build system (blaze) of their own. The major lesson I take from the proliferation of these tools is:
The best organizations invest in their tooling to make their people more productive.
However, this needs to be countered with the inescapable reality that build systems, and especially Android build systems, are big investments.
As the lone Firefox for Android build peer, I do not support Mozilla making a big investment into building our own Android build system.
For the sake of argument, however, such a Mozilla-developed build system would look like a blend of custom moz.build and a recursive Make backend. The amount of new and un-tested build system code would be very high; we’d incur serious risk, because all that new code needs to support a delicate combination of features that are constantly moving; and in all likelihood we wouldn’t achieve even 10% of what the existing solutions achieve in a calendar year [2].
Rather than diving deep into a particular illustration of a feature that would be difficult to express in such a Mozilla-developed build system, let me give instead a whirlwind tour of features we could take advantage of — today — if we used an existing solution.
Caveat: I have evaluated Gradle and buck extensively. I have not evaluated pants. (Blaze is not available.) Not everything claimed will be true for all external build systems.
Wins for building Fennec
Real Android libraries, with resources and manifests.
These libraries could, at first, just be third-party libraries that we want to build against. (We’ve so far held off using, or modified to be "part of" Fennec, any third-party libraries that reference Android resources.)
In future, we could build Fennec as a set of loosely-coupled libraries. (In some ways we do: GeckoView, Stumbler, Search Activity, Background Services… we’re doing this in a limited way.) Such small libraries let us test faster and parallelize the development process.
There’s a possibility to reduce risk here, too: I claim that we can improve our existing practice of landing new features behind a build flag by developing mostly self-contained Android libraries and then including those libraries behind a build flag. This lets developers land the code in the tree, and possibly even compile and unit test on TBPL, but not actually ship in Fennec until the build flag is flipped.
Merging resources and manifests is a solved problem; let’s use an existing solution to do it.
Faster builds.
Our existing build system supports compiling our existing libraries in parallel, but it doesn’t support DEXing in parallel [3]. Moving to an existing build system that compiles and DEXes in parallel will buy us a faster build even without separating our libraries. We get big wins (which I will try to quantify in the near future) if we do a little more work and split our libraries further.
Simpler Proguard support.
We’re about to land Bug 1106593, which addresses the fact that we weren’t Proguarding everything we should have been. It adds two bespoke stages to our build. This type of change would be simpler if we were using an existing implementation (which would have certainly Done the Right Thing) and only had to modify a Proguard configuration.
Trivial MultiDex support.
Fennec is pushing up against the classes.dex size limit. Google recently added MultiDex support, which allows further run-time loaded classesN.dex files. Gradle exposes the additional build configuration with a handful of lines of code. Someday, we’ll probably need this support.
Support for shrinking Android resources
Gradle exposes a non-trivial build step for resource shrinking. We want it.
Support for building multiple App configurations.
By configurations, I mean building a standard release version of Fennec, and a debug (non-Proguarded) version of Fennec for JUnit 3 testing, and possibly even multiple additional versions (resource constrained; API limited) all with one "TBPL build job". Such build configurations reduce automation machine load and reduce the risk of build flags being orphaned.
There’s a similar, but limited, feature known as split APKs that we might want too.
Wins for contributors
Simpler build bootstrapping.
Gradle in particular has first class for bootstrapping itself via the Gradle wrapper. This wrapper prepares the Gradle part of the build environment automatically and reduces the time needed to get a build started.
First class dependency management.
External build systems, in particular Gradle, integrate with Maven/Ivy repositories for consuming dependencies. That means we can specify, in the tree, what versions we currently support and the build system will do the work of fetching, configuring, and installing them. It’s a faster first build and fewer frustrating dependency chases trying to figure out what changed and how your local machine is different from the official TBPL configuration.
Better IDE integration.
Gradle integrates, mostly tightly, with IDEA IntelliJ and Google’s Android Studio. This is a big win for new contributors (just open mobile/android in your IDE to build) and a huge boon for seasoned developers who get the power of IntelliJ for their daily work.
Built in support for building and publishing Javadocs for our own use.
- Some parts of the code are well-commented; others are not. Perhaps we
- want to document some of our trickier code flows in a public place?
Built in support for producing Android lint output.
Catches bugs! For free!
Wins for GeckoView
Existing build systems have better systems for updating only specific targets.
Right now, the moz.build based build system produces essentially all artifacts at build time. That includes test APKs, packaging the GeckoView library, an example application, etc. It’s slow! Most of the time we just want the ability to package GeckoView; we really just want a fresh Fennec APK.
Built in code to build a GeckoView AAR file
This includes first class support for publishing the artifact to a Maven repository. This would eliminate the need for https://ci.mozilla.org/job/mozilla-central-geckoview/configure.
Support for producing Java source JARs and Javadoc JARs for GeckoView.
As the GeckoView library gains consumers, providing source JARs and Javadoc JARs will make consumers lives significantly easier.
Conclusion
Help me make our builds better! Discussion is best conducted on the mobile-firefox-dev mailing list and I’m nalexander on irc.mozilla.org and @ncalexander on Twitter.
Notes
| [1] | Truthfully, Gradle with Google’s own rapidly developing Android-Gradle plugin. But this was the case with Ant, too: Google shipped an Ant build.xml that consumers extended and customized. |
| [2] | That is, if Mozilla even was willing to pay a single developer to work on such a build system for a whole year. As it stands, we don’t pay a single full-time equivalent to work exclusively on any part of the build system. We get a lot of glandium’s time, a little of gps’s time, some of mshal’s time, some of my time, etc, etc. |
| [3] | To be clear, we could support DEXing in parallel and merging the results, but it’s yet another layer of complexity in our Make-based build system. Add in supporting MultiDex and producing debug and release artifacts… let’s not. |
Changes
- Mon 05 January 2015: Thanks to @rnewman for surfacing spelling and grammatical errors.
About Nick Alexander
Mathematician. Mozillian. Runner. Master of Disguise.
