The Firefox for Android build system in 2015
Me and my colleagues @lucasratmundo, @mleibovic, @michaelcomella, and vivekb attended the Community Building discussion at #mozlandia (notes and slides are available). @mhoye presented his thinking about community building and engagement at Mozilla and beyond. I interpreted Mike’s presentation through a bifurcated lens: I came away feeling that there are social aspects to community engagement, such as providing positive rewards and recognition; and there are technical aspects to community engagement, such as maintaining documentation and simplifying tooling requirements .
People like @lsblakk are able to bring new people into our community with phenomenal outreach programs like the Ascend Project, but that’s not my skill-set. I deeply admire the social work Lukas (and others!) are doing, but I personally am most able to empower the Mozilla community to own Firefox for Android by addressing the technical aspects Mike discussed.
Making it easier to contribute to Firefox for Android
In this context, the following initiatives will drive the Firefox for Android tooling:
- making it easier to build Firefox for Android the first time;
- reducing the edit-compile-test cycle time;
- making the Firefox for Android development process look and feel like the standard Android development process.
Making it easier to build Firefox for Android the first time
One strong claim made by mhoye — and supported by many others in the room — is that mach bootstrap has significantly reduced the technical accessibility barrier to building Firefox for Desktop. We need to implement mach bootstrap for Firefox for Android.
For those who don’t know, mach bootstrap is a script that prepares the Firefox build environment, including fetching, updating, and installing the pre-requisites needed for building Firefox. It automates the (often difficult!) task of fetching dependencies; ensures that known-good versions of dependencies are installed; and sets development environment defaults. mach bootstrap is the first thing that should be run in a fresh Firefox source tree .
Firefox for Android has more complicated dependencies than Firefox for Desktop, including some that cannot be easily distributed or installed: the Java development kit and run-time environment, the Android SDK and NDK; Google’s Play Services libraries, etc. We can save new contributors a long dependency chase before they see a positive result. In addition, seasoned developers spend an unknown-but-large amount of time discovering that the required dependencies have advanced. Pinning the build to known-good versions, failing the build when said versions are not present, and providing mach bootstrap to update to known-good versions will reduce this frustration.
A contributor started writing a shell script that does the work of mach bootstrap. Bug 1108771 tracks building upon this effort. I’ve also filed Bug 1108782 to track pinning the Firefox for Android build requirements to known-good versions.
Reducing the Firefox for Android edit-compile-test cycle time
The previous work will make it faster to build Firefox for Android the first time, because we won’t build C++ libraries. We’re also going to invest in making each incremental build faster, and there’s some low-hanging fruit here. Right now, the most costly parts of our build are compiling individual JAR libraries and DEXing all of the resulting JAR libraries. Every time we split our JAR libraries, we can parallelize a small part of our build and reduce the wall-clock time of our Java compilation. Right now we could split our single third-party JAR library and save ourselves compile time. And we’re very close to being able to split the Background Services (Sync, Firefox Accounts, Firefox Health Report, etc) library out of Fennec proper, which will save even more compile time.
Improving our DEXing time is more difficult. Android’s DEX processor is a byte-code transformation step that turns Java’s byte-code into Dalvik VM byte-code. For historical reasons, we DEX the entirety of Fennec’s byte-code in one DEX invocation, and it’s both a single-process bottleneck and terribly expensive. For some time, it has been possible to DEX each individual library in parallel and to merge the resulting DEX files. All modern Android build system (such as buck or Gradle) support this. We could support this in the Firefox for Android build system as well, but I think we should move to use a more featured build system under the hood instead. Android build systems are very complicated; we don’t want to write our own, and we definitely don’t want to write our own in Make syntax. In 2015, we’ll push to use a full-featured build tool that brings this DEX-time improvement. More on this in a future post.
Making the Firefox for Android development process "standards compliant"
This point is a more philosophical point than the others. Firefox for Android wins when we engage our community. The community of Android developers is large and constantly pushing the boundaries of what’s possible on a device. We want to tap into that well-spring of talent and innovation, and every thing we do that’s non-standard to an Android developer makes it harder for us to do this. Contributor @fedepaol wrote a blog post about how difficult this used to be.
The good news is, we’re getting better: we have rudimentary Gradle support and you can use IntelliJ now. But we still have a long, long way to go. We’ve got lots of easy wins just waiting for us: tickets like Bug 1107811 will go a long way towards making the Fennec "Android standards compliant" IntelliJ experience better. I have a technical plan to land in-source-tree IntelliJ configurations, so developers can open mobile/android directly in IntelliJ and get to a working Firefox for Android APK in the IDE in one step.
At a lower level, Tickets like Bug 1074258 will let us use the IntelliJ design view more easily, and landing Android JUnit 3 Instrumentation test runners in automation (Bug 1064004) will make local testing significantly easier than the convoluted Robocop process we have right now. The list goes on and on.
The Firefox for Android team moved strongly towards easier builds and ubiquitous tooling in 2014. 2015 is going to be even better. We’re going to improve our technical experience in (at least!) three ways: making the first build easier; making the next builds faster; and unlocking the power of the standard Android developer toolchain.
|||I believe there is an over-arching third aspect, that of the system in which we do our work and interact with the community, but right-here-right-now I don’t feel empowered to change this. Systemic change requires making community engagement part of every team’s top-level goals, and achieving such goals requires resources that are allocated well above my pay-grade.|
|||In fact, the bootstrapper does not even require a source check-out — you can download just the script and it will fetch enough to bootstrap itself. So it’s more accurate to say just bootstrap rather than mach bootstrap, but mach bootstrap has won the vocabulary battle in this arena.|
|||glandium has written a very informative blog post about the future of the Firefox build system at http://glandium.org/blog/?p=3318. The section relevant to this discussion is Specialized incremental builds.|