@ncalexanderhttps://www.ncalexander.net/blog/2017-05-31T16:00:00-07:00Tofino, data storage, and how we got to Mentat2017-05-31T16:00:00-07:00ncalexandertag:www.ncalexander.net,2017-05-31:blog/2017/05/31/tofino-data-storage-and-how-we-got-to-mentat/<!-- -*- Mode: rst; fill-column: 100; -*- -->
<!-- PELICAN_END_SUMMARY -->
<p><a class="reference external" href="https://github.com/mozilla/tofino">Project Tofino</a> was an attempt by Mozilla to develop a new web
browser. I think Mark Mayo, Senior Vice President of Firefox, <a class="reference external" href="https://medium.com/project-tofino/browsers-innovators-dilemma-and-project-tofino-ef634c6164f0">described Tofino best</a>:</p>
<blockquote>
[W]e’re working on browser prototypes that look and feel almost nothing like the current
Firefox… [we’re trying to solve] the kinds of problems people have that aren’t currently solved
by <em>anybody’s</em> browser product.</blockquote>
<p>My colleague Richard Newman and I were responsible for designing and implementing a data storage
layer to back Tofino. The engineering organizations I’ve been a part of and have witnessed at
Mozilla do not have a culture of lessons learned, final reports, and post-mortems. I’m a
context-building history-oriented learner so I’d like to change that, one little bit, by providing
my biased account of the forces that resulted in us proposing Mentat to meet Tofino’s data storage needs.</p>
<div class="section" id="data-storage-in-tofino">
<h2>Data storage in Tofino</h2>
<p>Richard and I have a lot of experience storing data for browsers (which I’ll return to), and Richard
<a class="reference external" href="https://160.twinql.com/different-kinds-of-storage/">captured some of his thinking about storage in a blog post</a>. All told, we wanted storage that:</p>
<ul class="simple">
<li>could handle existing Firefox data like bookmarks, history, and passwords;</li>
<li>was easy to evolve to handle new Tofino data like page thumbnails, fulltext web content indexes,
and A/B testing results;</li>
<li>performed well on an under-powered Windows tablet (the kind you might buy at a big-box store).</li>
</ul>
<p>However, we also carry the scars from implementing multiple Firefox Sync clients. For those not
intimately familiar with browsers and Firefox, Sync is a distributed system that co-ordinates
browser data (bookmarks, history, passwords, etc) across your devices (Desktop, Android, iOS). Sync
in general is a very hard problem and <a class="reference external" href="https://160.twinql.com/syncing-and-storage-on-three-platforms/">Firefox Sync is a poor rendition of a sync solution</a> <a class="footnote-reference" href="#sync" id="id1">[1]</a>. Richard and I knew
that Tofino would eventually evolve to become a Firefox Sync client (or grow a similar Sync
solution). So we also wanted storage that:</p>
<ul class="simple">
<li>would support an eventual Firefox Sync client.</li>
</ul>
<p>To meet Tofino’s needs we proposed and are prototyping <a class="reference external" href="https://github.com/mozilla/mentat">Project Mentat</a>, a data store designed for embedded client applications. The
<a class="reference external" href="https://medium.com/project-tofino/introducing-datomish-a-flexible-embedded-knowledge-store-1d7976bff344">initial blog post announcing Mentat (née Datomish)</a>
does a great job framing the data problems Tofino and Firefox face and how we’re trying to solve
them. In a nutshell, Mentat:</p>
<ul class="simple">
<li>is designed to be embedded into client applications;</li>
<li>manages a strong schema and helps you evolve that schema over time;</li>
<li>separates readers from writers, allowing to optimize critical queries;</li>
<li>maintains a historical log, supporting three-way merges when syncing.</li>
</ul>
</div>
<div class="section" id="how-we-got-to-mentat">
<h2>How we got to Mentat</h2>
<div class="section" id="ancient-history-2011-2016">
<h3>Ancient history [2011-2016]</h3>
<p>Any good origin story starts before the beginning, so in that spirit, let us start with ancient
history that predates Tofino. For simplicity, I’ll refer to myself and Richard Newman as the team
<a class="footnote-reference" href="#team" id="id2">[2]</a>. Our experience led us to to design a storage system supporting (and eventually integrated
with) the Firefox Sync implementation. We got here in stages:</p>
<ul class="simple">
<li>Firefox for Desktop is the cautionary tale: this is what happens when storage and syncing are not integrated;</li>
<li>Firefox for Android is the second system that learned some architectural lessons but "didn’t go far
enough"; and</li>
<li>Firefox for iOS gets it "mostly right".</li>
</ul>
<p>To set the scene, Richard inherited Firefox Sync, the Firefox for Desktop implementation. The
initial development was done in an add-on <a class="footnote-reference" href="#weaveaddon" id="id3">[3]</a>; it used observer notifications to witness
changes relevant to Sync and track those changes in memory to propagate upstream. The code landed
with this architecture and the foundational decision to rely on observer notifications resulted in
significant issues <a class="footnote-reference" href="#desktopfixes" id="id4">[4]</a>:</p>
<ul class="simple">
<li>not all data sources produced the notifications required for Sync;</li>
<li>notifications were lost at start-up, shutdown, and when Sync itself had bugs;</li>
<li>the notification flow is inherently non-transactional <a class="footnote-reference" href="#transaction" id="id5">[5]</a>.</li>
</ul>
<p>Richard and I wrote the Firefox for Android Sync implementation. Richard owned <a class="footnote-reference" href="#modules" id="id6">[6]</a> Sync
and, eventually, the Firefox for Android storage implementation. Sync was built as a stand-alone
process from day one, factored out of the browser front-end via the Android <cite>ContentProvider</cite>
storage abstraction <a class="footnote-reference" href="#contentprovider" id="id7">[7]</a>. Unfortunately, we didn’t appreciate how significant
transactional syncing would be, and we made Firefox for Android’s data storage "live": the browser
and Sync update the store non-transactionally — potentially at the same time — leading to subtle <span class="caps">UI</span>
conflicts and missed changes in Sync <a class="footnote-reference" href="#androidfixes" id="id8">[8]</a>.</p>
<p>Richard also owned the Firefox for iOS storage and Firefox Sync implementations. The two systems
were designed to support each other from day one. The storage system is:</p>
<ul class="simple">
<li>entirely transactional and robust in the face of extreme behaviours;</li>
<li>well factored, enabling performance improvements to key user interactions due to the single point
of responsibility for data;</li>
<li>significantly less likely to lose or corrupt Sync data than the other implementations.</li>
</ul>
<p>These performance improvements are most notable in the iOS top sites implementation. Top sites is
the panel of most frecent <a class="footnote-reference" href="#frecent" id="id9">[9]</a> sites shown every time the user taps the <span class="caps">URL</span> bar to navigate
somewhere new. The iOS top sites panel displays instantly, since the underlying store keeps a
materialized view (a read-only computed cache) in memory and updates it efficiently independent of
the user interface.</p>
</div>
<div class="section" id="enter-tofino-april-2016">
<h3>Enter Tofino [April 2016]</h3>
<p>Tofino was pitched as an entirely new browser product. We anticipated a data footprint similar to
Firefox for iOS, so we started to express the architectural lessons we had learned from the two
previous storage implementations using web technologies <a class="footnote-reference" href="#swift" id="id10">[10]</a>. The team started to build an
Electron/Node.js desktop application. We quickly discovered (or perhaps, realized) that a
client/server architecture is the best model for our web technology implementation. We started to
develop a User Agent service: an always available backend that stored browsing data for the user and
leveraged that data to help the user exploit the web. We settled into a familiar pattern,
recognizable to anyone who has built a Web App:</p>
<ul class="simple">
<li><span class="caps">UI</span> -> transport -> <span class="caps">UA</span> service</li>
<li><span class="caps">UI</span> updates locally (optimistic update)</li>
<li><span class="caps">UA</span> service -> transport -> notifies <span class="caps">UI</span> to update locally (authoritative update)</li>
</ul>
<p>where the transport was variously <span class="caps">HTTP</span> requests, Web Socket messages, or Electron’s <span class="caps">IPC</span>.</p>
<p>Tofino almost immediately pivoted to product and user research. Suddenly there was much more
interest in capturing event streams, and an immediate need to support rapid prototyping. We started
to build a Node.js service, backed by SQLite, to store events, materialize views, and update and
publish changes to clients.</p>
</div>
<div class="section" id="tofino-product-evolution-september-2016">
<h3>Tofino product evolution [September 2016]</h3>
<p>We quickly observed that adding data to our hand-written SQLite store required migrating the SQLite
store forward rapidly. Each data type we wanted to add required a back-and-forth between the
product owner, the front-end team, and the storage team. In response, we started to investigate
event stores that might do this work for us. Unfortunately, most offerings were:</p>
<ul class="simple">
<li>not embeddable (targeted the Java Virtual Machine (<span class="caps">JVM</span>), or were intended to scale horizontally in
the cloud); or</li>
<li>more general than we were comfortable with (graph stores like Neo4J generally don’t query for time
ranges efficiently; document stores like Mongo generally don’t support strong schemas); or</li>
<li>not general enough (key-value stores like <span class="caps">LMDB</span> and LevelDB don’t support strong schemas and
high-level queries); or</li>
<li>not mature enough to ship to a market of Firefox’s size (side-projects like Cayley).</li>
</ul>
<p>Evaluating this technical landscape, Richard and I found the ideas behind Cognitect’s <a class="reference external" href="http://www.datomic.com/">Datomic</a> most compelling. Datomic:</p>
<ul class="simple">
<li>is assertion (event) oriented;</li>
<li>maintains a full transaction log;</li>
<li>exposes an expressive, extendable schema;</li>
<li>models row-oriented (relational) data efficiently;</li>
<li>is flexible enough to model graph-oriented data.</li>
</ul>
<p>Sadly, Datomic targets the <span class="caps">JVM</span> and we don’t see a path to shipping 200+ Mb of <span class="caps">VM</span> and database to
Firefox’s market. (In addition, Datomic is not open-source, making it an awkward cultural fit for Mozilla.)</p>
<p>Concurrently, the product and front-end teams wanted to rapidly prototype using tools like GraphQL.
Our research into GraphQL suggested that performance would be poor and very difficult to address,
but we see Datomic’s query and transact syntax as enabling experimentation by the front-end team
similar in spirit (if not expression) to GraphQL, and also possible to make performant.</p>
<p>To research the ideas by Datomic, we started to adapt Nikita Prokopov’s awesome <a class="reference external" href="https://github.com/tonsky/datascript">DataScript</a>, a Clojure{Script} Datomic-alike that transpiles to
JavaScript and can run in the browser and in Node.js. DataScript is an in-memory store, not
suitable for Firefox-sized work loads, and fundamentally synchronous (which is a big problem in the
highly concurrent JavaScript browser environment).</p>
</div>
<div class="section" id="datomish-july-2016">
<h3>Datomish [July 2016]</h3>
<p>The work to make DataScript asynchronous, backed by a persistent store (a flat file, IndexedDB,
SQLite) was close to a rewrite, but we still felt that Datomic’s model was compelling for our
requirements — particularly our emphasis on experimentation and managing change. So we started to
build a Clojure{Script} Datomic-alike. This short-lived prototype was named Datomish. Datomish:</p>
<ul class="simple">
<li>re-used key pieces of DataScript’s source code;</li>
<li>persisted to SQLite;</li>
<li>translated Datomic’s Datalog queries to SQLite queries.</li>
</ul>
<p>The main idea was to reduce the unknown performance of Datomic’s Datalog queries to the better known
performance of SQLite’s <span class="caps">SQL</span> queries. The Datomish prototype convinced us that the SQLite
translation could be done efficiently and yield performant queries against the working-sets we
expect to witness in Firefox in the wild. It was flexible in the ways we wanted, and early
experience suggested that the store was performant enough to back the Tofino browser and related
experiments. However, the Datomish prototype was not fit for greater purposes in three regards:</p>
<ul class="simple">
<li>the transpiled JavaScript could not be reasonably shipped in a product with Firefox’s audience;</li>
<li>the ClojureScript prototype suffered from emergent memory leaks due to subtle bugs with our use of
ClojureScript’s persistent data structures and communicating sequential processes implementation,
and exposed impedance mismatches between JavaScript and ClojureScript; and</li>
<li>we felt that ClojureScript and JavaScript were not the right technologies to back data storage for
Android or iOS.</li>
</ul>
</div>
<div class="section" id="the-end-of-tofino-december-2016">
<h3>The end of Tofino [December 2016]</h3>
<p>At this point, the Tofino product experiment was stopped. None of the <span class="caps">UX</span> experiments and prototypes
were deemed worthy of future investment. The people working on Tofino joined the people working on
Datomish (me and Richard) to prototype a Datomic-alike written in Rust. We hoped to:</p>
<ul class="simple">
<li>ship in Firefox for Desktop;</li>
<li>be able to ship on Android and iOS;</li>
<li>improve on the performance and robustness of the Clojure{Script} implementation.</li>
</ul>
<p>In response, the newly enlarged team rapidly stood up a Rust version of Datomish, which we named
Project Mentat.</p>
</div>
<div class="section" id="redirection-april-2017">
<h3>Redirection [April 2017]</h3>
<p>Throughout Q1 2016, we focused on implementing the core features of Project Mentat. However, senior
management redirected effort away from "ship in Firefox for Desktop" and toward two alternate goals:</p>
<ul class="simple">
<li>prototype the new Firefox <span class="caps">UI</span> (Photon), using lessons learned from the existing Firefox <span class="caps">UI</span>
(Australis) implementation expressed using React in Tofino;</li>
<li>re-focus on Mentat as a component of new product experiences, in the same way that Tofino had
focused on new product experiences.</li>
</ul>
<p>In response, we re-focused the people who had been working on Tofino onto the "Photino" Photon <span class="caps">UI</span>
prototype <a class="footnote-reference" href="#photinoblurb" id="id11">[11]</a>.</p>
<p>Crucially, we continued to build Mentat as a store that:</p>
<ul class="simple">
<li>focused on flexibility and schema evolution; and</li>
<li>could meet performance requirements through suitable abstractions rather than manual tinkering; and</li>
<li>would support a robust Sync solution (that was not necessarily Firefox Sync);</li>
</ul>
<p>while additionally proposing an architectural split between the user interface, browser data, and
web rendering platform that we believe Mozilla should invest in across all its browser offerings.</p>
</div>
<div class="section" id="status-may-2017">
<h3>Status [May 2017]</h3>
<p>Where does the Project Mentat codebase stand? As of May 2017 we’ve implemented basic transacting:</p>
<ul class="simple">
<li>assertion and retraction with <tt class="docutils literal">:db/add</tt> and <tt class="docutils literal">:db/retract</tt></li>
<li>map notation like <tt class="docutils literal">{:db/id ... :some/attribute :some/value <span class="pre">...}</span></tt></li>
<li>foundational data types like <tt class="docutils literal">:db.type/long</tt>, <tt class="docutils literal">:db.type/string</tt>, <tt class="docutils literal">:db.type/keyword</tt>, etc</li>
<li>cardinality constraints with <tt class="docutils literal">:db.cardinality/one</tt> and <tt class="docutils literal">:db.cardinality/many</tt></li>
<li>custom identifiers with <tt class="docutils literal">:db/ident</tt></li>
<li>schema mutation equivalent to Datomic’s <tt class="docutils literal">:db.install/attribute</tt></li>
<li>temporary identifier resolution like Datomic`s <tt class="docutils literal">{:db/id "tempid"}</tt></li>
<li><tt class="docutils literal">:db.unique/identity</tt> and upserts</li>
</ul>
<p>and basic querying:</p>
<ul class="simple">
<li>accepting a large subset of Datomic’s Datalog query language</li>
<li>non-trivial joining with <tt class="docutils literal">:or</tt> and <tt class="docutils literal"><span class="pre">:or-join</span></tt></li>
<li>negation with <tt class="docutils literal">:not</tt> and <tt class="docutils literal"><span class="pre">:not-join</span></tt></li>
<li>interpolating input values with <tt class="docutils literal">:in</tt></li>
<li>projecting scalar, vector, tuple, and relation results</li>
<li>ordering and limiting result sets</li>
<li>some non-trivial query pruning and type-aware optimization</li>
</ul>
<p>The Rust implementation is not yet as full featured as the Clojure{Script} prototype:</p>
<ul class="simple">
<li>no support for aggregates like <tt class="docutils literal">(count)</tt>, <tt class="docutils literal">(max)</tt>, and <tt class="docutils literal">(min)</tt> in queries;</li>
<li>no fulltext search with <tt class="docutils literal">:db/fulltext true</tt> attributes in queries;</li>
<li>no schema registration and migration layer on top of the basic store;</li>
<li>no transactor loop and transaction listeners.</li>
</ul>
<p>But we think what is implemented is robust and has a clear path to production. We anticipate it
will take roughly 3 months to land Mentat into Firefox for Desktop and to back a simple store like
logins or form history using the new technology. The most significant work will be managing the
application life cycles and locking and concurrency; the foundational work for applying transactions
and querying the store is essentially complete.</p>
</div>
<div class="section" id="review-may-2017">
<h3>Review [May 2017]</h3>
<p>Mentat is currently running an architectural review gauntlet to decide whether Mozilla will invest
further or cancel the project. The architecture review has focused primarily on whether Mentat can
address the wide-ranging short-term storage architectural failings that hold Firefox for Desktop
down. Unfortunately, we didn’t expect to need to do so at this time — indeed, we were told not to!
— so we’re fighting to justify our approach as I type.</p>
<p>We still believe that Mentat can solve some of Firefox’s storage problems right now and can evolve
to solve most of the others, but we have not engaged directly with most of the concrete problems the
architecture review process foresee. I hope we’ll get time to do so, but if we don’t — this blog
post can serve as a Project Mentat pre-mortem.</p>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>I hope this narrative explains the forces that shaped the arc we followed to get to Mentat, and that
not all of the problems we faced, solutions we explored, and artifacts we created are lost immediately.</p>
<p>Thanks to Richard Newman (<a href="https://twitter.com/rnewman">@rnewman</a>) for being the animating force behind Mentat and this work.
Thanks to Joe Walker for being a great manager to work with and for. And thanks to Mark Mayo and
the Tofino team for trying a new thing, regardless of the outcome. Innovator’s dilemma is a real thing.</p>
<p>Many thanks to Joe Walker, who provided initial feedback on this blog post before it was intended
for public consumption; and to early readers and reviewers Richard Newman, Grisha Kruglov, Ralph
Giles, and Francois Marier, who provided valuable feedback and compelled me to expand or rewrite
many sections.</p>
<p>Discussion is best conducted on <span class="caps">IRC</span>: I’m nalexander in <tt class="docutils literal"><span class="pre">irc.mozilla.org/#mentat</span></tt> and on <a class="reference external" href="https://mozilla.slack.com/messages/mentat">Slack
(Mozilla Corporation only)</a>, and I’m <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Wed 31 May 2017: Incorporated suggestions from Francois Marier.</li>
<li>Mon 29 May 2017: Incorporated suggestions from Grisha Kruglov, Ralph Giles, and Richard Newman.</li>
<li>Wed 24 May 2017: Initial version.</li>
</ul>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
<table class="docutils footnote" frame="void" id="sync" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td><p class="first">Syncing is a large and active area of academic research; the <a class="reference external" href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type">Wikipedia article for
conflict-free replicated datatypes</a> is a good place to begin.</p>
<p>Firefox Sync is even more disadvantaged than most systems:</p>
<ul class="simple">
<li>to preserve Firefox user’s privacy, all data is encrypted in such a way that the cloud storage
part of the system (owned and operated by Mozilla) cannot help the clients manage conflicts in the data;</li>
<li>the <span class="caps">HTTP</span> <span class="caps">API</span> that Firefox Sync uses to upload and download data from cloud storage does not
make it easy to robustly fetch and report all data. These issues are being addressed with the
introduction of atomic uploads, batch downloads that can be interrupted and resumed, etc.</li>
</ul>
<p class="last">However, at a higher layer, the Firefox Sync <em>client protocol</em> makes it very difficult to
actually implement Firefox Sync. For example, every Firefox Sync client is expected to recognize
when the cloud store has failed and bootstrap a new cloud store, uploading all historical data.
These ancient decisions make it very difficult to implement a Firefox Sync client that can
interoperate with existing Firefox Sync clients.</p>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="team" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td><p class="first">In addition, Brian Grinstead, Victor Porof, Jordan Santell, Joe Walker, and Emily Toop
(all Mozilla Corporation employees) have contributed to Mentat. We also gratefully acknowledge
feedback and support from several Rust library authors:</p>
<ul class="last simple">
<li>Markus Westerland (<a class="reference external" href="https://github.com/Marwes/combine">combine</a>)</li>
<li>John Gallagher (<a class="reference external" href="https://github.com/jgallagher/rusqlite">rusqlite</a>)</li>
<li>Kevin Mehall (<a class="reference external" href="https://github.com/kevinmehall/rust-peg">rust-peg</a>)</li>
</ul>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="weaveaddon" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>The code that grew to become Firefox Sync was developed by the Mozilla Services
team in an add-on. It was originally called <a class="reference external" href="https://blog.mozilla.org/labs/2007/12/introducing-weave/">Weave</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="desktopfixes" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>Many of these issues are now being resolved: the Firefox for Desktop Sync team
has been building support for robust syncing into the underlying data stores for some time. See
in particular <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1258127">Bug 1258127</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="transaction" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td><p class="first">By <em>transactional</em>, I mean that a series of storage operations either all succeed
and are committed in one atomic operation, or none succeed and they are all dropped. I use
<em>transactional</em> and <em>atomic</em> interchangeably.</p>
<p>To see that an observer notification-based system is <strong>not</strong> transactional, suppose that a single
user operation both bookmarks a site and visits it to fetch its current title. Firefox Sync
might witness a notification saying the site was bookmarked and a second notification saying that
the site was visited. Now suppose that there was a race between syncing bookmarks from a
different client and the notification flow. It’s possible that the different client has already
bookmarked the site. That means the first notification (the bookmarked notification) should be
dropped, since the site is already bookmarked. But the second notification (the visit
notification) will not be dropped, even though logically it should be dropped as well.</p>
<p>This particular example is artificial, but these types of scenarios become more and more
prevalent as the scope of synced data expands.</p>
<p class="last">See <a class="reference external" href="https://en.wikipedia.org/wiki/Atomicity_(database_systems)">the Wikipeda article on atomicity</a> for more.</p>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="modules" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id6">[6]</a></td><td>For folks not familiar with Mozilla’s development process, different parts of the
source code are split into <em>modules</em>. Each module has a designated owner, who stewards that part
of the code base. They triage tickets, set priorities, delegate reviews, and are the ultimate
decision maker for changes impacting the module. See <a class="reference external" href="https://wiki.mozilla.org/Modules">https://wiki.mozilla.org/Modules</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="contentprovider" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id7">[7]</a></td><td>Originally, Firefox for Android did not own its data store: we intended to use
the Android system Browser data stores. Firefox for Android quickly outgrew the system Browser
data stores and evolved its own storage; unfortunately, the Android <cite>ContentProvider</cite> <span class="caps">API</span> is
essentially non-transactional <em>across</em> separate <cite>ContentProvider</cite> instances, and there was a
period where Firefox for Android supported both the external storage and the internal storage,
which led us to ape the system Browser data stores and not structure our internal storage in the
ways that best supported our browser and Firefox Sync.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="androidfixes" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id8">[8]</a></td><td>Most of these issues have been addressed in the browser: we have a well
abstracted <cite>BrowserDB</cite> interface that transactionally updates the underlying stores. And the
Sync implementation is actively evolving to update a clone of the underlying store before
atomically committing changes.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="frecent" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id9">[9]</a></td><td>A portmanteau blending <em>frequent</em> and <em>recent</em>; see
<a class="reference external" href="https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Places/Frecency_algorithm">https://developer.mozilla.org/en-<span class="caps">US</span>/docs/Mozilla/Tech/Places/Frecency_algorithm</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="swift" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id10">[10]</a></td><td>Why not start from the iOS storage and Sync implementation? We considered doing so.
However, the iOS implementation was built in Apple’s Swift language, and Tofino had a strong
technology experimentation position: Mozilla wanted to evaluate building a browser using "modern
web technologies". Swift at that time had been open-sourced only 6 months earlier (December
2015) and desktop support was patchy. Rust is an area of technology investment at Mozilla; Swift
is not. In addition, the Firefox for iOS implementation is not flexible enough to support the
rapid prototyping phase we wanted for Tofino.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="photinoblurb" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id11">[11]</a></td><td>To Victor Porof, Brian Grinstead, and Joe Walker’s credit, <a class="reference external" href="https://github.com/victorporof/tofino">Photino</a> has become an active test bed for the Photon <span class="caps">UX</span> team.</td></tr>
</tbody>
</table>
</div>
Firefox "artifact builds" for Mac OS X2015-12-31T16:00:00-08:00ncalexandertag:www.ncalexander.net,2015-12-31:blog/2015/12/31/firefox-artifact-builds-for-mac/<!-- -*- Mode: rst; fill-column: 100; -*- -->
<!-- PELICAN_END_SUMMARY -->
<p>I’m thrilled to announce support for Mac <span class="caps">OS</span> X <em>artifact builds</em>. Artifact builds trade expensive
compile times for (variable) download times and some restrictions on what parts of the Firefox
codebase can be modified. For Mac <span class="caps">OS</span> X, the downloaded binaries are about 100Mb, which might take
just a minute to fetch. The hard restriction is that only the non-compiled parts of the browser can
be developed, which means that artifact builds are really only useful for front-end developers. The
Firefox for Android front-end team has been using artifact builds with great success for almost a
year (see <a class="reference external" href="http://www.ncalexander.net/blog/2015/07/02/build-fennec-frontend-fast-with-mach-artifact/">Build Fennec frontend fast with mach artifact!</a> and my other posts on this blog).</p>
<p>I intend to update the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Simple_Firefox_build"><span class="caps">MDN</span> documentation</a> and the build bootstrapper (see
<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1221200">Bug 1221200</a>) as soon as I can, but in the meantime, here’s a quick start guide.</p>
<div class="section" id="quick-start">
<h2>Quick start</h2>
<p>You’ll need to have run <tt class="docutils literal">mach <span class="pre">mercurial-setup</span></tt> and installed the <tt class="docutils literal">mozext</tt> extension (see Bug
1234912). In your mozconfig file, add the lines</p>
<div class="highlight"><pre>ac_add_options --enable-artifact-builds
mk_add_options MOZ_OBJDIR=./objdir-artifact
</pre></div>
<p>You’ll want to run <tt class="docutils literal">mach configure</tt> again to make sure the change is recognized. This sets
<tt class="docutils literal"><span class="pre">--disable-compile-environment</span></tt> and opts you in to running <tt class="docutils literal">mach artifact install</tt>
automatically.</p>
<p>After this, you should find that <tt class="docutils literal">mach build</tt> downloads and installs the required artifact
binaries automatically, based off your current Mercurial commit. To test, just try</p>
<div class="highlight"><pre>./mach build && ./mach run
</pre></div>
<p>After the initial build, incremental <tt class="docutils literal">mach build <span class="caps">DIR</span></tt> should also maintain the state of the
artifact binaries — even across <tt class="docutils literal">hg commit</tt> and <tt class="docutils literal">hg pull && hg update</tt>.</p>
<p>You should find that <tt class="docutils literal">mach build faster</tt> works as expected, and that the occasional <tt class="docutils literal">mach build
browser/app/repackage</tt> is required.</p>
</div>
<div class="section" id="restrictions">
<h2>Restrictions</h2>
<p>Oh, so many. Here are some of the major ones:</p>
<ul class="simple">
<li>Right now, artifact builds are only available to developers working on Mac <span class="caps">OS</span> X Desktop builds
(<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1207890">Bug 1207890</a>) and Firefox for Android builds. I expect Linux support to follow shortly (tracked
in <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1236110">Bug 1236110</a>). Windows support is urgently needed but I don’t yet know how much work it will be
(tracked in Bug 1236111).</li>
<li>Right now, artifact builds are only available to Mercurial users. There’s no hard technical
reason they can’t be made available to git users, and I expect it to happen eventually, but it’s
non-trivial and really needs a dedicated git-using engineer to scratch her own itch. This is
tracked by Bug 1234913.</li>
<li>Artifact builds don’t allow developing the C++ source code. As soon as you need to change a
compiled component, you’ll need a regular build. Unfortunately, things like Telemetry are
compiled (but see tickets like Bug 1206117).</li>
<li>Artifact builds are somewhat heuristic, in that the downloaded binary artifacts may not correspond
to your actual source tree perfectly. That is, we’re not hashing the inputs and mapping to a
known binary: we’re choosing binaries from likely candidates based on your version control status
and pushes to Mozilla automation. Binary mismatches for Fennec builds are rare (but do exist,
see, for example, <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1222636">Bug 1222636</a>), but I expect them to be much more common for Desktop builds.
Determining if an error is due to an artifact build is a black art. We’ll all have to learn what
the symptoms look like (often, binary component <span class="caps">UUID</span> mismatches) and how to minimize them.</li>
<li>Support for running tests is limited. I don’t work on Desktop builds myself, so I haven’t really
explored this. I expect a little work will be needed to get xpcshell tests running, since we’ll
need to arrange for a downloaded xpcshell binary to get to the right place at the right time.
Please file a bug if some test suite doesn’t work so that we can investigate.</li>
</ul>
</div>
<div class="section" id="troubleshooting">
<h2>Troubleshooting</h2>
<p>The command that installs binaries is <tt class="docutils literal">mach artifact install</tt>. Start by understanding what
happens when you run</p>
<div class="highlight"><pre>./mach artifact install --verbose
</pre></div>
<p>See <a class="reference external" href="http://www.ncalexander.net/blog/2015/07/02/build-fennec-frontend-fast-with-mach-artifact/#troubleshooting">the troubleshooting section of my older blog post</a>
for more. As a last resort, the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Simple_Firefox_for_Android_build#I_want_to_work_on_the_front-end">Firefox for Android <span class="caps">MDN</span> documentation</a>
may be helpful.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Thanks to Gregory Szorc (<a href="https://twitter.com/indygreg">@indygreg</a>) and Mike Hommey for reviewing this work. Many thanks to Mark
Finkle (<a href="https://twitter.com/mfinkle">@mfinkle</a>) for providing paid time for me to pursue this line of work and to the entire
Firefox for Android team for being willing guinea pigs.</p>
<p>There’s a huge amount of work to be done here, and I’ve tried to include Bugzilla ticket links so
that interested folks can contribute or just follow along. Dan Minor will be picking up some of
this artifact build work in the first quarter of 2016.</p>
<p>Mozilla is always making things better for the front-end teams and our valuable contributors! <a class="reference external" href="https://developer.mozilla.org/en/docs/Introduction">Get
involved with code contribution at Mozilla</a>!</p>
<p>Discussion is best conducted on the <a class="reference external" href="https://lists.mozilla.org/listinfo/dev-builds">dev-builds mailing list</a> and I’m nalexander on
<tt class="docutils literal"><span class="pre">irc.mozilla.org/#developers</span></tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Thu 31 December 2015: Initial version.</li>
</ul>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
</div>
nalexander:community update, part the second2015-07-15T15:00:00-07:00ncalexandertag:www.ncalexander.net,2015-07-15:blog/2015/07/15/nalexander-community-update-part-the-second/<!-- -*- Mode: rst; fill-column: 100; -*- -->
<!-- :status: draft -->
<!-- PELICAN_END_SUMMARY -->
<div class="section" id="active-projects">
<h2>Active projects</h2>
<p>Here’s some of the projects I’m currently offering that are seeing active progress.</p>
<div class="section" id="when-it-s-personal-firefox-account-profile-avatars">
<h3>When it’s personal: Firefox Account profile avatars</h3>
<p>Super-contributor <a href="https://mozillians.org/u/vivek">/u/vivek</a> has been working on all aspects of integrating Firefox Account profile
avatar images into Fennec. This work is broadly tracked in <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1150964">Bug 1150964</a>, and there are lots of
pieces: network layer fetching; storage and caching; managing update broadcasts; and implementing
<span class="caps">UI</span>. This project is the first OAuth-authenticated Firefox Account service in Fennec (our native
Reading List implementation didn’t ship) and is likely to be the first WebChannel consumer in Fennec
as well!</p>
<p>This project is extra special to me because Vivek came to me and asked (in his usual under-stated
manner) if he could "do all the work" for this feature. Vivek and I had collaborated on a lot of
tickets, but I had been hoping to work with a contributor on a project scoped larger than one or two
tickets. This project is the first time that I have gotten to engage with a contributor on an
ongoing basis. Where we talked about expectations (for both of us!) and timelines up front. Where
I expect to turn maintainership of the code over to Vivek and he’s going to own it. And it is my
sincere hope that Vivek will mentor new contributors to improve that code.</p>
</div>
<div class="section" id="paying-down-technical-debt-deprecating-the-android-sync-clients-database">
<h3>Paying down technical debt: deprecating the android-sync clients database</h3>
<p>Contributor <a href="https://mozillians.org/u/ahmedkhalil">/u/ahmedkhalil</a> has been chewing through tickets that simplify the handling of clients
and tabs from other devices (as shown in Fennec’s Synced Tabs panel). This project isn’t as well
tracked as some of the other ones I’m writing about today, partly because I didn’t set the scope on
day one — Ahmed arrived at the tickets himself. And what a path! Ahmed and I started doing some
build system tickets (if you use the new <tt class="docutils literal">mach artifact</tt> command to <a class="reference external" href="http://www.ncalexander.net/blog/2015/07/02/build-fennec-frontend-fast-with-mach-artifact/">Build Fennec frontend fast
with mach artifact!</a>, you’re using some of Ahmed’s <span class="caps">AAR</span> packaging code); and then we took a strange
and ultimately unsuccessful trip into bookmark exporting; and then we did some other minor tickets.
I fully expect Ahmed to push into the dark corners of the Fennec Sync implementation and refactor
some of our oldest, least touched code in the clients engine. I got Ahmed into this with the lure
of front-end user-visible Synced Tabs improvements and he may end up in the least user-visible part
of the code base!</p>
</div>
<div class="section" id="understanding-the-fennec-connected-experience-sync-metrics">
<h3>Understanding the Fennec connected experience: Sync metrics</h3>
<p>The Fennec Sync product is a "mature product", if by mature you mean that nobody modifies the code.
However, the newly revitalized Sync team (bandleader: Chris Karlof) is leading a wide-ranging
project to understand the Sync experience across Firefox products. This will be a qualitative and
quantitative project, and I’m partnering with new contributor <a href="https://twitter.com/aminban">@aminban</a> to collect quantitative
metrics about Fennec Sync on Android. This work is broadly tracked at <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1180321">Bug 1180321</a>. This is a very
paralellizable project; most of the individual tickets are independent of each other. I’m hoping
to work with Amin on a few tickets and then have him help mentor additional contributors to flesh
out the rest of the work.</p>
</div>
</div>
<div class="section" id="help-wanted">
<h2>Help wanted</h2>
<p>But I also have some projects in the hopper that need … a certain set of skills.</p>
<div class="section" id="plain-old-java-projects">
<h3>Plain Old Java Projects</h3>
<p>These are projects for front-end developers that require Java (and maybe JavaScript) skills.</p>
<ul class="simple">
<li>The Firefox Accounts team had an idea to email <span class="caps">QR</span> codes to make it easier for Fennec users to
connect to their Firefox Account. I made some notes and tracked the idea at <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1178364">Bug 1178364</a>. It’s a
wide ranging project that might need some co-ordination with the Firefox Accounts team, but I work
with those folks frequently and we can make it happen. This is a really interesting project with
lots of moving pieces. It needs Java and some JavaScript skills, and the ability to get creative
while testing.</li>
<li>I’ve been talking to <a href="https://mozillians.org/u/anatal">/u/anatal</a> about implementing the WebSpeech <span class="caps">API</span> in Fennec. André has plans to
develop an <em>offline</em> (meaning, on the device) implementation, but shipping such an implementation
in Fennec is hard due to the size of the model files required. An <em>online</em> implementation that
used Google’s Android Speech implementation would be easier to ship. This would be a really
interesting project because you’d be implementing a web <span class="caps">API</span> exposed to web content! That is,
you’d actually be <em>building the web platform</em>. You’d need some Java and JavaScript skills;
preferably some experience with the Android Speech APIs; and we’d both learn some Gecko web engine
internals and read a lot of <span class="caps">W3C</span> specifications.</li>
</ul>
</div>
<div class="section" id="engagement-projects">
<h3>Engagement Projects</h3>
<p>These projects might not end up in the Fennec codebase, but they’re valuable and require folks with
special skills.</p>
<ul class="simple">
<li>I want to expose better metrics about the Fennec team’s contributor experience. I hate to say the
word dashboard but… a dashboard! Tracking things like number of new tickets created in the
Firefox for Android component, number of new mentor tickets, number of new good first bugs, number
of new contributors arriving, etc. I think most of this can be extracted from Bugzilla with some
clever queries, but I don’t really know how to do it, and I really don’t know how to display the
data in a useful form. This might be a simple client-side web page that does some <a class="reference external" href="https://wiki.mozilla.org/Bugzilla:REST_API">Bugzilla Rest
<span class="caps">API</span></a> queries and uses <a class="reference external" href="http://d3js.org/">d3.js</a> or
similar to format the results. Or it could be a set of Mediawiki <tt class="docutils literal"><bugzilla></tt> queries that we
can put in the <a class="reference external" href="https://wiki.mozilla.org/Mobile/Notes">mobile team weekly meeting notes</a>. This
is really open-ended and could grow into a larger community engagement role with the Fennec team.</li>
<li>I want to do some Android community outreach to understand barriers to Fennec (code) contribution.
I’m aware that not building on Windows is probably a big deal (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1169873">Bug 1169873</a>), but I don’t know how
big a deal. And I’m aware (painfully!) of how awkward it is to get started with Fennec, but I
don’t know which parts Android developers find the worst. (For example: these developers probably
have the Android <span class="caps">SDK</span> (if not the Android <span class="caps">NDK</span>) installed already.) This might look like a "Getting
started with Fennec development" session in your location. But I’d also like to know how Android
developers feel about Fennec as a product, and whether Android developers are even interested in
the web in the way that Mozilla is representing. If you are connected to Android developers
(maybe through a meetup group?) and would be interested in doing some outreach, contact me.</li>
</ul>
</div>
<div class="section" id="build-system-projects">
<h3>Build system Projects</h3>
<p>Build system hackers are a rare breed. But there’s so much low-hanging fruit here that can make a
big difference to our daily development.</p>
<ul class="simple">
<li>I have several Gradle-related build tickets. I want to get rid of <tt class="docutils literal">mach <span class="pre">gradle-install</span></tt>, and
make it so that every Fennec build has an automatically maintained Gradle configuration without
additional commands. Part of this will be making the Gradle configuration more dynamic, so that
you don’t have to run <tt class="docutils literal">mach package</tt> before running <tt class="docutils literal">mach <span class="pre">gradle-install</span></tt>. I’d like to find a
way to share bits of the <tt class="docutils literal">.idea</tt> directory. I’d like to move the Gradle configuration files out
of the object directory, so that clobber builds don’t destroy your Gradle configuration. These
projects require Python skills.</li>
<li>I have lots of <tt class="docutils literal">mach artifact</tt> follow-up tickets. Read <a class="reference external" href="http://www.ncalexander.net/blog/2015/07/02/build-fennec-frontend-fast-with-mach-artifact/">Build Fennec frontend fast with mach
artifact!</a> to get an idea of what <tt class="docutils literal">mach artifact</tt> is, but in a nutshell it downloads and caches
binary artifacts built in Mozilla automation so that you don’t have to compile C++ to build
Fennec. It turns a 20 minute build into a 5 minute build. I’d like to support git, and improve
the caching layer, and make the system more configurable, and support Desktop front-end builds,
and… These projects require Python skills.</li>
<li>I want to move <tt class="docutils literal">build/mobile/robocop</tt> into <tt class="docutils literal">mobile/android/tests/browser/robocop</tt>. And
convert it to <tt class="docutils literal">moz.build</tt>. This will both making testing better (no more forgetting to build
Robocop!) and it also make it easier to conditionally compile tests. If you’re interested, start
with <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=938659">Bug 938659</a> and <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1180104">Bug 1180104</a>. This project requires basic Make and Python skills.</li>
</ul>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>I’d like to thank all the contributors who make my job a pleasure, especially those mentioned in
this blog post.</p>
<p>The Firefox for Android team is always making things better for contributors! <a class="reference external" href="https://wiki.mozilla.org/Mobile/Get_Involved">Get involved with
Firefox for Android</a>.</p>
<p>Discussion is best conducted on the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a> and I’m nalexander on
<tt class="docutils literal"><span class="pre">irc.mozilla.org/#mobile</span></tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Sun 5 July 2015: Initial version.</li>
</ul>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
</div>
nalexander:community update, part the first2015-07-05T15:00:00-07:00ncalexandertag:www.ncalexander.net,2015-07-05:blog/2015/07/05/nalexander-community-update-part-the-first/<!-- -*- Mode: rst; fill-column: 100; -*- -->
<!-- :status: draft -->
<!-- PELICAN_END_SUMMARY -->
<p>The part of my job that is special — the part I wouldn’t get working away from Mozilla — is
enabling community code contributors to participate in and own the direction of Fennec (Firefox for
Android). There are <a class="reference external" href="https://support.mozilla.org/kb/contributor-news-resources">so</a> <a class="reference external" href="https://sendto.mozilla.org/">many</a> <a class="reference external" href="https://github.com/rust-lang/rust/issues">ways</a> to contribute to
Mozilla, but this post I’ll limit myself to <a class="reference external" href="https://wiki.mozilla.org/Mobile/Get_Involved#Hacking">Fennec code contribution</a> because it’s the on-ramp that the Fennec
team has put the most effort into, and it’s my own personal cause.</p>
<p>In <a class="reference external" href="http://www.ncalexander.net/blog/2015/07/15/nalexander-community-update-part-the-second/">part the second</a>, I give a project status update and advertise a variety of new projects to the
community. If you’re interested, contact me!</p>
<div class="section" id="new-contributors">
<h2>New contributors</h2>
<p>Mozilla’s very own <a href="https://twitter.com/mhoye">@mhoye</a> has been very active encouraging new contributors. Mike (and others —
I’m looking at you, <a href="https://twitter.com/lastontheboat">@lastontheboat</a>) pioneered marking tickets as [good first bugs]; and pushed to add
the mentor field to Bugzilla; and maintains <a href="https://twitter.com/StartMozilla">@StartMozilla</a>. Mike works tirelessly to widen the
funnel of new contributors approaching the Mozilla project. In large part Mike’s internal
evangelism has worked: we think we know some things that work to help people make their <em>first</em> contribution:</p>
<ul class="simple">
<li>we have improved our on-boarding documentation;</li>
<li>we have made it easier to build Fennec the first time (<tt class="docutils literal">mach bootstrap</tt>) and to read the code
base (Gradle integration);</li>
<li>we have made it easier to find [good first bug] tickets and mentors;</li>
<li>and we have committed to making #mobile a friendly, welcoming space for potential contributors.</li>
</ul>
<p>We’re now looking to grow our existing contributors. How do we work with contributors to move them
from "new arrival" to "valued contributor"? What’s the value exchange?</p>
</div>
<div class="section" id="capturing-unicorns">
<h2>Capturing unicorns</h2>
<p>I’ve been thinking about this for a few years now. Mobile team — which really means Fennec team,
since the Firefox for iOS is very new — has captured some unicorns <a class="footnote-reference" href="#unicorns" id="id1">[1]</a>. I’m thinking of
<a href="https://mozillians.org/u/capella">/u/capella</a>, who is de-facto owner of the Fennec input front-end code, and <a href="https://mozillians.org/u/vivek">/u/vivek</a> — if I didn’t
include someone, it wasn’t intentional. We’re truly lucky to have had the benefit of their
contributions for as long as we have. But we don’t really <em>understand</em> how we caught them. And the
thing about unicorns is that you only need to capture one unicorn to look like a great unicorn
hunter. Mobile team is right at that point: we’ve had some success attracting code contributors,
but we don’t really know how we did it — and we don’t have a strategy to attract more.</p>
<p>Recently I’ve had success attracting and retaining high-value code contributors with the following
tactic. I’ve started offering medium-sized projects that let me and a contributor collaborate on an
area of the code base over time. By medium-sized I really mean not <em>just</em> a [good first bug] or a
[good next bug]; I mean an area of work that’s somewhat open-ended and will develop deeper expertise
in at least one part of the code base. (I’m trying to keep these projects well-scoped, but that’s a
battle for me as an engineer: scoping work is <em>hard</em>.) I mean a few things by <em>high-value code
contributor</em>: principally, a repeat contributor who might grow into a reviewer and module peer.</p>
<p>This approach requires some up-front work on my part. It’s not easy writing a bug with enough
context to be meaningful to someone without deep context into how Fennec is built and where our team
strategy is leading us. It’s not easy to file dependent tickets that are correctly "chunked", and
to provide enough links and technical details to guide a reasonable implementation. It’s especially
not easy trying to anticipate high-level problems in parts of the code that I myself don’t know much
(or anything!) about.</p>
<p>But the reward can be great. High-value contributors have time and skills that we want. In
exchange, they want things: experience; an opportunity to work with excellent engineers; the chance
to contribute to the Open Web; references; etc. The folks with that mix need to be challenged; they
need to see that their contribution leads somewhere: to an implemented feature, to a performance
improvement, to a better future. I want to give a pathway to that future. Contributions on-ramps
need to lead to contribution in-roads.</p>
<p>On a purely practical level, after mentoring a [good first bug] to completion, it’s really hard to
find a [good next bug]! Often, there’s nothing in the same or a similar area. Or there’s no ticket
(that I know about!) that is a reasonable challenge. And as a contributor, there’s no satisfaction
in fixing typos or doing trivial variable substitutions more than once or twice. With a scoped
project somewhat specified up front, I always have at least an <em>idea</em> of what could come next.</p>
<p>I haven’t yet tried to make one of these projects self-contained, in the sense of having a [good
first bug], and then some [good next bugs], and then some meatier implementation details. But I
think it’s a reasonable model and I intend to try it.</p>
</div>
<div class="section" id="other-thoughts">
<h2>Other thoughts</h2>
<p>There’s so much more to this discussion. I think some of my success retaining contributors is that
I put in a lot of time in #mobile answering questions. I get to know the people I work with —
where they live, what they do every day, what kinds of change they want to see in the project. I’ll
reach out to them if I find tickets that are good fits for them. If I could scale this high-touch
approach, I would!</p>
<p>I don’t claim to know much about our contributor motivations. I think we do a good job of
recognizing our contributors inside our team but an absolutely terrible job recognizing our
contributors in the Mozilla community and in the larger Android community. We have essentially no
formal mentoring (outside of internships, which are paid) or formal recognition (such as writing
letters of recommendation). I day-dream about a "new contributor survey" that would quickly let us
learn about our community and match contributors to mentors, tickets, and the outcomes that they want.</p>
<p>I’m really interested in understanding the health of our contributor community. We know that
keeping the pipeline of [good first bugs] and mentored tickets wide helps, but we don’t measure that
flow. That’s all looking inwards, to our own community. What if we looked outwards? How much
would outreach to targeted Android communities, projects, and even specific developers help? Could
we further Fennec’s mission by targeting key web projects and developers? Could we attract key
people to evangelize Fennec? We don’t know.</p>
<p>Finally, I haven’t really seen this approach used in other parts of Mozilla, although I hear
rumblings that <a href="https://twitter.com/redheadedcuban">@redheadedcuban</a> and the A-Team do something like this. If your team does this, let
me know!</p>
<div class="section" id="conclusion">
<h3>Conclusion</h3>
<p>The Firefox for Android team is always making things better for contributors! <a class="reference external" href="https://wiki.mozilla.org/Mobile/Get_Involved">Get involved with
Firefox for Android</a>.</p>
<p>Discussion is best conducted on the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a> and I’m nalexander on
<tt class="docutils literal"><span class="pre">irc.mozilla.org/#mobile</span></tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="changes">
<h3>Changes</h3>
<ul class="simple">
<li>Sun 5 July 2015: Initial version.</li>
</ul>
</div>
<div class="section" id="notes">
<h3>Notes</h3>
<table class="docutils footnote" frame="void" id="unicorns" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>I use the term <em>unicorn</em> to mean an extremely capable code contributor, in the most
positive sense — someone who might be unique.</td></tr>
</tbody>
</table>
</div>
</div>
Build Fennec frontend fast with mach artifact!2015-07-02T20:30:00-07:00ncalexandertag:www.ncalexander.net,2015-07-02:blog/2015/07/02/build-fennec-frontend-fast-with-mach-artifact/<!-- -*- Mode: rst; fill-column: 100; -*- -->
<!-- PELICAN_END_SUMMARY -->
<p>Nota bene: this post supercedes <a class="reference external" href="http://www.ncalexander.net/blog/2015/05/06/build-fennec-frontend-fast/">Build Fennec frontend fast!</a></p>
<div class="section" id="quick-start">
<h2>Quick start</h2>
<p>It’s easy! But there is a pre-requisite: you need to enable Gregory Szorc’s <em>mozext</em> Mercurial
extension <a class="footnote-reference" href="#mozext" id="id1">[1]</a> first. <em>mozext</em> is part of Mozilla’s <em>version-control-tools</em> repository; run
<tt class="docutils literal">mach <span class="pre">mercurial-setup</span></tt> to make sure your local copy is up-to-date, and then add the following to
the <tt class="docutils literal">.hg/hgrc</tt> file in your source directory:</p>
<div class="highlight"><pre>[extensions]
mozext = /PATH/TO/HOME/.mozbuild/version-control-tools/hgext/mozext
</pre></div>
<p>Then, run <tt class="docutils literal">hg pushlogsync</tt>. Mercurial should show a long (and slow) progress bar <a class="footnote-reference" href="#pushlog" id="id2">[2]</a>.
From now on, each time you <tt class="docutils literal">hg pull</tt>, you’ll also maintain your local copy of the pushlog.</p>
<p>Now, open your <tt class="docutils literal">mozconfig</tt> file and add:</p>
<div class="highlight"><pre>ac_add_options --disable-compile-environment
mk_add_options MOZ_OBJDIR=./objdir-frontend
</pre></div>
<p>(That last line uses a different object directory — it’s worth experimenting with a different
directory so you can go back to your old flow if necessary.)</p>
<p>Then <tt class="docutils literal">mach build</tt> and <tt class="docutils literal">mach build mobile/android</tt> as usual. When it’s time to package an <span class="caps">APK</span>, use:</p>
<div class="highlight"><pre>mach artifact install && mach package
</pre></div>
<p>instead of <tt class="docutils literal">mach package</tt> <a class="footnote-reference" href="#package" id="id3">[3]</a>. Use <tt class="docutils literal">mach install</tt> like normal to deploy to your device!</p>
<p>After running <tt class="docutils literal">mach artifact install && mach package</tt> once, you should find that <tt class="docutils literal">mach
<span class="pre">gradle-install</span></tt>, <tt class="docutils literal">mach gradle app:installDebug</tt>, and developing with IntelliJ (or Android Studio)
work like normal as well.</p>
</div>
<div class="section" id="disclaimer">
<h2>Disclaimer</h2>
<p>This only works when you are building Fennec (Firefox for Android) and developing JavaScript and/or
Fennec frontend Java code! If you’re building Firefox for Desktop, this won’t help you. If you’re
building C++ code, this won’t help you.</p>
<p>The integration currently requires Mercurial. Mozilla’s release engineering runs a service mapping
git commit hashes to Mercurial commit hashes; <tt class="docutils literal">mach artifact</tt> should be able to use this service
to provide automatic binary artifact management for git users.</p>
</div>
<div class="section" id="discussion">
<h2>Discussion</h2>
<p><tt class="docutils literal">mach artifact install</tt> is your main entry point: run this to automatically inspect your local
repository, determine good candidate revisions, talk to the Task Cluster index service to identify
suitable build artifacts, and download them from Amazon S3. The command caches heavily, so it
should be fine to run frequently; and the command avoids touching files except when necessary, so it
shouldn’t invalidate builds arbitrarily.</p>
<p>The reduction in build time comes from <tt class="docutils literal"><span class="pre">--disable-compile-environment</span></tt>: this tells the build
system to never build C++ libraries (<tt class="docutils literal">libxul.so</tt> and friends) <a class="footnote-reference" href="#toolchain" id="id4">[4]</a>. On my laptop, a
clobber build with this configuration completes in about 3 minutes <a class="footnote-reference" href="#improvements" id="id5">[5]</a>. This
configuration isn’t well tested, so please file tickets blocking Bug 1159371.</p>
</div>
<div class="section" id="troubleshooting">
<h2>Troubleshooting</h2>
<p>Run <tt class="docutils literal">mach artifact</tt> to see help.</p>
<div class="section" id="i-m-seeing-problems-with-pip">
<h3>I’m seeing problems with <tt class="docutils literal">pip</tt></h3>
<p>Your version of <tt class="docutils literal">pip</tt> may be to old. Upgrade it by running <tt class="docutils literal">pip install <span class="pre">--upgrade</span> pip</tt>.</p>
</div>
<div class="section" id="i-m-seeing-problems-with-hg">
<h3>I’m seeing problems with <tt class="docutils literal">hg</tt></h3>
<p>Does <tt class="docutils literal">hg log <span class="pre">-r</span> <span class="pre">pushhead('fx-team')</span></tt> work? If not, there’s a problem with your <em>mozext</em> configuration.
Check the pre-requisites again.</p>
</div>
<div class="section" id="what-version-of-the-downloaded-binaries-am-i-using">
<h3>What version of the downloaded binaries am I using?</h3>
<p><tt class="docutils literal">mach artifact last</tt> displays the last artifact installed. You can see the local file name; the
<span class="caps">URL</span> the file was fetched from; the Task Cluster job <span class="caps">URL</span>; and the corresponding Mercurial revision
hash. You can use this to get some insight into the system.</p>
</div>
<div class="section" id="where-are-the-downloaded-binaries-cached">
<h3>Where are the downloaded binaries cached?</h3>
<p>Everything is cached in <tt class="docutils literal"><span class="pre">~/.mozbuild/package-frontend</span></tt>. The commands purge old artifacts as new
artifacts are downloaded, keeping a small number of recently used artifacts.</p>
</div>
<div class="section" id="i-m-seeing-weird-errors-and-crashes">
<h3>I’m seeing weird errors and crashes!</h3>
<p>Since your local build and the upstream binaries may diverge, lots of things can happen. If the
upstream binaries change a C++ <span class="caps">XPCOM</span> component, you may see a <em>binary incompatibility</em>. Such a
binary incompatibility looks like:</p>
<div class="highlight"><pre>E GeckoConsole(5165) [JavaScript Error: "NS_ERROR_XPC_GS_RETURNED_FAILURE: Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]" {file: "resource://gre/modules/Services.jsm" line: 23}]
</pre></div>
<p>You should update your tree (using <tt class="docutils literal">hg pull <span class="pre">-u</span> <span class="pre">--rebase</span></tt> or similar) and run <tt class="docutils literal">mach build && mach
artifact install && mach package</tt> again.</p>
</div>
<div class="section" id="how-can-i-help-debug-problems">
<h3>How can I help debug problems?</h3>
<p>There are two commands to help with debugging: <tt class="docutils literal"><span class="pre">print-cache</span></tt> and <tt class="docutils literal"><span class="pre">clear-cache</span></tt>. You
shouldn’t need either; these are really just to help me debug issues in the wild.</p>
</div>
</div>
<div class="section" id="acknowledgements">
<h2>Acknowledgements</h2>
<p>This work builds on the contributions of a huge number of people. First, <a href="https://twitter.com/indygreg">@indygreg</a> supported this
effort from day one and reviewed the code. He also wrote <em>mozext</em> and made it easy to access the
pushlog locally. None of this happens without Greg. Second, the Task Cluster Index team deserves
kudos for making it easy to download artifacts built in automation. Anyone who’s written a <span class="caps">TBPL</span>
scraper knows how much better the new system is. Third, I’d like to thank <a href="https://twitter.com/liucheia">@liucheia</a> for testing
this with me in Whistler, and <a href="https://mozillians.org/u/vivek">/u/vivek</a> for proof-reading this blog post.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>In my blog post <a class="reference external" href="http://www.ncalexander.net/blog/2014/12/08/the-firefox-for-android-build-system-in-2015/">The Firefox for Android build system in 2015</a>, the first priority was making it
easier to build Firefox for Android the first time. The second priority was reducing the
edit-compile-test cycle time. The <tt class="docutils literal">mach artifact</tt> work described here drastically reduces the
<em>first</em> compile-test cycle time, and subsequent compile-test cycles after pulling from the upstream
repository. It’s hitting part of the first priority, and part of the second priority. Baby steps.</p>
<p>The Firefox for Android team is always making things better for contributors! <a class="reference external" href="https://wiki.mozilla.org/Mobile/Get_Involved">Get involved with
Firefox for Android</a>.</p>
<p>Discussion is best conducted on the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a> and I’m nalexander on
<tt class="docutils literal"><span class="pre">irc.mozilla.org/#mobile</span></tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Wed 1 July 2015: Initial version.</li>
<li>Mon 6 July 2015: fix typo in link to Vivek. Thanks, sfink!</li>
</ul>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
<table class="docutils footnote" frame="void" id="mozext" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>I can’t find documentation for <em>mozext</em> anywhere, but
<a class="reference external" href="http://gregoryszorc.com/blog/2013/07/22/mercurial-extension-for-gecko-development/">http://gregoryszorc.com/blog/2013/07/22/mercurial-extension-for-gecko-development/</a> includes a
little information. <tt class="docutils literal">mach artifact</tt> uses mozext to manage the pushlog.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="pushlog" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>The long (and slow) download is fetching a local copy of the <em>pushlog</em>, which records
who pushed what commits when to the Mozilla source tree. <tt class="docutils literal">mach artifact</tt> uses the pushlog to
determine good candidate revisions (and builds) to download artifacts for.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="package" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>We should make this happen automatically.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="toolchain" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>In theory, <tt class="docutils literal"><span class="pre">--disable-compile-environment</span></tt> also means we don’t need a host C++
toolchain (e.g., gcc targeting Mac <span class="caps">OS</span> X) nor a target C++ toolchain (e.g., the Android <span class="caps">NDK</span>).
This is not my primary motivation but I’m happy to mentor a contributor who wanted to test this
and make sure it works! It would be a nice win: you could get a working Fennec build with fewer
(large!) dependencies.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="improvements" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td>I intend to profile <tt class="docutils literal">mach build</tt> in this case and try to improve it. Much of
the build is essentially single-threaded in this configuration, including compiling the Java
sources for Fennec. Splitting Fennec into smaller pieces and libraries would help, but that is
hard. See for example <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1104203">Bug 1104203</a>.</td></tr>
</tbody>
</table>
</div>
Build Fennec frontend fast!2015-05-06T02:30:00-07:00ncalexandertag:www.ncalexander.net,2015-05-06:blog/2015/05/06/build-fennec-frontend-fast/<!-- -*- Mode: rst; fill-column: 100; -*- -->
<!-- PELICAN_END_SUMMARY -->
<p>Nota bene: this post has been superceded by <a class="reference external" href="http://www.ncalexander.net/blog/2015/07/02/build-fennec-frontend-fast-with-mach-artifact/">Build Fennec frontend fast with mach artifact!</a></p>
<div class="section" id="quick-start">
<h2>Quick start</h2>
<p>Temporarily, you’ll need to pull the new mach command into your source tree:</p>
<div class="highlight"><pre>$ hg pull -r 3c0fb13b77b8460e56a31054a5f005bb3e4cdda1 https://reviewboard-hg.mozilla.org/gecko/
$ hg up 3c0fb13b77b8460e56a31054a5f005bb3e4cdda1
$ hg rebase -d fx-team
</pre></div>
<p>It’s easy! Open your <tt class="docutils literal">mozconfig</tt> file and add:</p>
<div class="highlight"><pre>ac_add_options --disable-compile-environment
</pre></div>
<p>You might want to also use a different object directory:</p>
<div class="highlight"><pre>mk_add_options MOZ_OBJDIR=./objdir-frontend
</pre></div>
<p>Then <tt class="docutils literal">mach build</tt> and <tt class="docutils literal">mach build mobile/android</tt> as usual. When it’s time to package an <span class="caps">APK</span>, use:</p>
<div class="highlight"><pre>mach package-frontend
</pre></div>
<p>instead of <tt class="docutils literal">mach package</tt>. Use <tt class="docutils literal">mach install</tt> like normal to deploy to your device!</p>
<p>After running <tt class="docutils literal">mach <span class="pre">package-frontend</span></tt> once, <tt class="docutils literal">mach <span class="pre">gradle-install</span></tt>, <tt class="docutils literal">mach gradle
app:installDebug</tt>, and developing with IntelliJ (or Android Studio) work as well.</p>
</div>
<div class="section" id="disclaimer">
<h2>Disclaimer</h2>
<p>This only works when you are building Fennec (Firefox for Android) and developing JavaScript and/or
Fennec frontend Java code!</p>
<p>You have been warned.</p>
</div>
<div class="section" id="options">
<h2>Options</h2>
<p><tt class="docutils literal">mach <span class="pre">package-frontend</span></tt> takes a couple of options that let you control downloading:</p>
<ul>
<li><p class="first"><tt class="docutils literal"><span class="pre">--force-remote-binaries</span></tt>, <tt class="docutils literal"><span class="pre">-r</span></tt></p>
<p>Download and use remote binaries without comparing local and remote timestamps.</p>
</li>
<li><p class="first"><tt class="docutils literal"><span class="pre">--force-local-binaries</span></tt>, <tt class="docutils literal"><span class="pre">-l</span></tt></p>
<p>Do not download remote binaries; use local cached binaries.</p>
</li>
</ul>
</div>
<div class="section" id="troubleshooting">
<h2>Troubleshooting</h2>
<div class="section" id="what-version-of-the-downloaded-binaries-am-i-using">
<h3>What version of the downloaded binaries am I using?</h3>
<p>Look in <tt class="docutils literal"><span class="pre">~/.mozbuild/package-frontend/stage/assets/fennec-*.json</span></tt> for build <span class="caps">ID</span>, timestamps, and
additional configuration.</p>
</div>
<div class="section" id="i-m-seeing-weird-errors-and-crashes">
<h3>I’m seeing weird errors and crashes!</h3>
<p>Since your local build and the upstream binaries may diverge, lots of things can happen. If the
upstream binaries change a C++ <span class="caps">XPCOM</span> component, you may see a <em>binary incompatibility</em>. Such a
binary incompatibility looks like:</p>
<div class="highlight"><pre>E GeckoConsole(5165) [JavaScript Error: "NS_ERROR_XPC_GS_RETURNED_FAILURE: Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]" {file: "resource://gre/modules/Services.jsm" line: 23}]
</pre></div>
<p>You should update your tree (using <tt class="docutils literal">hg pull <span class="pre">-u</span> <span class="pre">--rebase</span></tt> or similar) and run <tt class="docutils literal">mach build && mach
<span class="pre">package-frontend</span></tt> again.</p>
</div>
</div>
<div class="section" id="technical-details">
<h2>Technical details</h2>
<p>There are two things happening here. The reduction in build time comes from
<tt class="docutils literal"><span class="pre">--disable-compile-environment</span></tt>: this tells the build system to never build C++ libraries
(<tt class="docutils literal">libxul.so</tt> and friends) <a class="footnote-reference" href="#toolchain" id="id1">[1]</a>. On my laptop, a clobber build with this configuration
completes in about 3 minutes <a class="footnote-reference" href="#improvements" id="id2">[2]</a>. This configuration isn’t well tested, so please file
tickets blocking Bug 1159371.</p>
<p>The new <tt class="docutils literal">mach <span class="pre">package-frontend</span></tt> downloads pre-built binaries <a class="footnote-reference" href="#purging" id="id3">[3]</a>, copies them into your
object directory, and then runs the regular packaging code to include them in an <span class="caps">APK</span>. This new mach
command is tracked at <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1162191">Bug 1162191</a>. A successful run looks something like:</p>
<div class="highlight"><pre>$ ./mach package-frontend
0:00.23 wget --timestamping --quiet --recursive --level=1 --no-directories --no-parent --accept fennec-*.en-US.android-arm.txt http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/
0:00.82 wget --timestamping http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/geckolibs-20150506030206.aar
--2015-05-06 16:20:13-- http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/geckolibs-20150506030206.aar
Resolving ftp.mozilla.org... 63.245.215.56, 63.245.215.46
Connecting to ftp.mozilla.org|63.245.215.56|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22891694 (22M) [text/plain]
Saving to: ‘geckolibs-20150506030206.aar’
100%[=======================================================================================>] 22,891,694 3.84MB/s in 11s
2015-05-06 16:20:25 (1.93 MB/s) - ‘geckolibs-20150506030206.aar’ saved [22891694/22891694]
0:12.20 unzip -u /Users/nalexander/.mozbuild/package-frontend/geckolibs-20150506030206.aar -d /Users/nalexander/.mozbuild/package-frontend/stage
0:12.37 /Users/nalexander/Mozilla/gecko/objdir-droid/_virtualenv/bin/python /Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/action/process_install_manifest.py --no-remove --no-remove-all-directory-symlinks --no-remove-empty-directories /Users/nalexander/Mozilla/gecko/objdir-droid/dist/bin /Users/nalexander/.mozbuild/package-frontend/geckolibs.manifest
From /Users/nalexander/Mozilla/gecko/objdir-droid/dist/bin: Kept 12 existing; Added/updated 0; Removed 0 files and 0 directories.
0:12.51 /usr/bin/make -C . -j8 -s -w package
0:13.00 make: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid'
0:13.04 make[1]: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid/mobile/android/installer'
0:13.57 make[2]: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid/mobile/android/installer'
0:14.02 make[3]: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid/mobile/android/installer'
<snip>
0:29.88 make: Leaving directory `/Users/nalexander/Mozilla/gecko/objdir-droid'
0:29.89 /usr/local/bin/terminal-notifier -title Mozilla Build System -group mozbuild -message Packaging complete
</pre></div>
<p>If the upstream binaries haven’t changed, new binaries aren’t downloaded, so it’s reasonable to use
<tt class="docutils literal">mach <span class="pre">package-frontend</span></tt> as a drop-in replacement for <tt class="docutils literal">mach package</tt>:</p>
<div class="highlight"><pre>$ ./mach package-frontend
0:00.23 wget --timestamping --quiet --recursive --level=1 --no-directories --no-parent --accept fennec-*.en-US.android-arm.txt http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/
0:00.52 wget --timestamping http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/geckolibs-20150506030206.aar
--2015-05-06 16:27:02-- http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/en-US/geckolibs-20150506030206.aar
Resolving ftp.mozilla.org... 63.245.215.46, 63.245.215.56
Connecting to ftp.mozilla.org|63.245.215.46|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22891694 (22M) [text/plain]
Server file no newer than local file ‘geckolibs-20150506030206.aar’ -- not retrieving.
0:00.58 unzip -u /Users/nalexander/.mozbuild/package-frontend/geckolibs-20150506030206.aar -d /Users/nalexander/.mozbuild/package-frontend/stage
0:00.59 /Users/nalexander/Mozilla/gecko/objdir-droid/_virtualenv/bin/python /Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/action/process_install_manifest.py --no-remove --no-remove-all-directory-symlinks --no-remove-empty-directories /Users/nalexander/Mozilla/gecko/objdir-droid/dist/bin /Users/nalexander/.mozbuild/package-frontend/geckolibs.manifest
From /Users/nalexander/Mozilla/gecko/objdir-droid/dist/bin: Kept 12 existing; Added/updated 0; Removed 0 files and 0 directories.
0:00.72 /usr/bin/make -C . -j8 -s -w package
0:01.25 make: Entering directory `/Users/nalexander/Mozilla/gecko/objdir-droid'
<snip>
0:17.30 /usr/local/bin/terminal-notifier -title Mozilla Build System -group mozbuild -message Packaging complete
</pre></div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>In my blog post <a class="reference external" href="http://www.ncalexander.net/blog/2014/12/08/the-firefox-for-android-build-system-in-2015/">The Firefox for Android build system in 2015</a>, the first priority was making it
easier to build Firefox for Android the first time. So I landed <tt class="docutils literal">mach bootstrap</tt> for Fennec, and
we turned an onerous process into a one-liner that works for most people <a class="footnote-reference" href="#bootstrap" id="id4">[4]</a>. The second
priority was reducing the edit-compile-test cycle time. The work described here drastically reduces
the <em>first</em> compile-test cycle time, and subsequent compile-test cycles after pulling from the
upstream tree. It’s hitting part of the first priority, and part of the second priority. Baby steps.</p>
<p>The Firefox for Android team is always making things better for contributors! <a class="reference external" href="https://wiki.mozilla.org/Mobile/Get_Involved">Get involved with
Firefox for Android</a>.</p>
<p>Discussion is best conducted on the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a> and I’m nalexander on
<tt class="docutils literal"><span class="pre">irc.mozilla.org/#mobile</span></tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Fri 8 May 2015: Thanks <a href="https://twitter.com/michaelcomella">@michaelcomella</a> for pointing out a bad Bugzilla ticket number.</li>
</ul>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
<table class="docutils footnote" frame="void" id="toolchain" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>In theory, <tt class="docutils literal"><span class="pre">--disable-compile-environment</span></tt> also means we don’t need a host C++
toolchain (e.g., gcc targeting Mac <span class="caps">OS</span> X) nor a target C++ toolchain (e.g., the Android <span class="caps">NDK</span>).
This is not my primary motivation but I’m happy to mentor a contributor who wanted to test this
and make sure it works! It would be a nice win: you could get a working Fennec build with fewer
(large!) dependencies.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="improvements" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>I intend to profile <tt class="docutils literal">mach build</tt> in this case and try to improve it. Much of
the build is essentially single-threaded in this configuration, including compiling the Java
sources for Fennec. Splitting Fennec into smaller pieces and libraries would help, but that is
hard. See for example <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1104203">Bug 1104203</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="purging" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>By default, the downloaded binaries are stored in <tt class="docutils literal"><span class="pre">~/.mozbuild/package-frontend</span></tt>.
The remote binaries change frequently — several times a day — and are not yet purged from your
local cache after a reasonable time. In future, we’ll download less (maybe once a day? maybe
only according to the latest revision in your local tree?) and delete old binaries after some set
time (one week?).</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="bootstrap" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>This was tracked by <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1108771">Bug 1108771</a>. It’s made a huge difference, and I should
know: I used to field build problems in #mobile every few days. Now, I field build problems
every few months.</td></tr>
</tbody>
</table>
</div>
Smoother Landings: How to update and test Fennec feature build flags2015-02-13T11:30:00-08:00ncalexandertag:www.ncalexander.net,2015-02-13:blog/2015/02/13/how-to-update-and-test-fennec-feature-build-flags/<!-- -*- Mode: rst; fill-column: 80; -*- -->
<!-- PELICAN_END_SUMMARY -->
<p>First, have you read my earlier blog post <a class="reference external" href="http://www.ncalexander.net/blog/2014/07/09/how-to-land-a-fennec-feature-behind-a-build-flag/">Bumpy Landings</a>? No? Go read it now! I’ll wait.</p>
<p>Good, you’ve read <a class="reference external" href="http://www.ncalexander.net/blog/2014/07/09/how-to-land-a-fennec-feature-behind-a-build-flag/">Bumpy Landings</a>, so you’re familiar with adding and updating
Fennec build flags.</p>
<p>The <strong>first major point</strong>: changing build settings — your <tt class="docutils literal">mozconfig</tt> file,
the <tt class="docutils literal">configure.in</tt> script, sundry <tt class="docutils literal">.m4</tt> files, application-specific
configuration files like <tt class="docutils literal">mobile/android/confvars.sh</tt> — requires
<em>re-configuration</em> to recognize your changes. You need to run:</p>
<pre class="literal-block">
mach configure
</pre>
<p>Unfortunately, after re-configuration, you really need to rebuild your entire
tree. The issue is that <tt class="docutils literal">mach configure</tt> updates a variety of build settings
and information (including your feature flag) and the build system does not
track, at a fine-grained level, what is the minimal set of things that need to
rebuilt when such settings change. Hence, you need to rebuild everything to be
safe, which is termed a <em>clobber build</em>.</p>
<p>The good news is that you can avoid clobber builds in some situations. First,
let’s see how to find the value of a build flag. Then we’ll see how to update a
value and do a limited re-build of the impacted source code.</p>
<div class="section" id="interrogating-the-build-settings">
<h2>Interrogating the build settings</h2>
<p>The build system maintains the settings you see in <tt class="docutils literal">Makefile.in</tt> and
<tt class="docutils literal">moz.build</tt> files in <tt class="docutils literal">$<span class="caps">OBJDIR</span>/config.status</tt>. The only time that
<tt class="docutils literal">config.status</tt> is written is by <tt class="docutils literal">mach configure</tt> (which may, on occasion,
be triggered by other <tt class="docutils literal">mach</tt> commands, including <tt class="docutils literal">mach build</tt>). So to
verify that your <tt class="docutils literal">config.status</tt> is up-to-date, you can always re-run <tt class="docutils literal">mach
configure</tt> manually.</p>
<p>The <tt class="docutils literal">config.status</tt> file is just a specially formatted Python script. Open it
up and you’ll see there’s no magic:</p>
<div class="highlight"><pre><span class="c">#!/Users/nalexander/Mozilla/gecko/objdir-droid/_virtualenv/bin/python</span>
<span class="c"># coding=utf-8</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">types</span>
<span class="n">topsrcdir</span> <span class="o">=</span> <span class="s">'''/Users/nalexander/Mozilla/gecko'''</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isabs</span><span class="p">(</span><span class="n">topsrcdir</span><span class="p">):</span>
<span class="n">rel</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">),</span> <span class="n">topsrcdir</span><span class="p">)</span>
<span class="n">topsrcdir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">rel</span><span class="p">)</span>
<span class="n">topsrcdir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">normpath</span><span class="p">(</span><span class="n">topsrcdir</span><span class="p">)</span>
<span class="n">topobjdir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">))</span>
<span class="n">defines</span> <span class="o">=</span> <span class="p">[(</span><span class="n">name</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">value</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="p">[</span>
<span class="p">(</span><span class="s">''' ANDROID '''</span><span class="p">,</span> <span class="s">' 1 '</span><span class="p">),</span>
<span class="o">...</span>
<span class="p">]</span> <span class="p">]</span>
<span class="n">substs</span> <span class="o">=</span> <span class="p">[(</span><span class="n">name</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">value</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">types</span><span class="o">.</span><span class="n">StringTypes</span><span class="p">)</span> <span class="k">else</span> <span class="n">value</span><span class="p">)</span> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="p">[</span>
<span class="p">(</span><span class="s">''' SHELL '''</span><span class="p">,</span> <span class="s">r''' /bin/sh '''</span><span class="p">),</span>
<span class="o">...</span>
<span class="p">]</span>
<span class="n">__all__</span> <span class="o">=</span> <span class="p">[</span><span class="s">'topobjdir'</span><span class="p">,</span> <span class="s">'topsrcdir'</span><span class="p">,</span> <span class="s">'defines'</span><span class="p">,</span> <span class="s">'non_global_defines'</span><span class="p">,</span> <span class="s">'substs'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">args</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">([(</span><span class="n">name</span><span class="p">,</span> <span class="nb">globals</span><span class="p">()[</span><span class="n">name</span><span class="p">])</span> <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">__all__</span><span class="p">])</span>
<span class="kn">from</span> <span class="nn">mozbuild.config_status</span> <span class="kn">import</span> <span class="n">config_status</span>
<span class="n">config_status</span><span class="p">(</span><span class="o">**</span><span class="n">args</span><span class="p">)</span>
</pre></div>
<p>The <strong>second major point</strong>: you can always determine what the build system
thinks a build setting by extracting values from either the <tt class="docutils literal">defines</tt> or the
<tt class="docutils literal">substs</tt> list of <tt class="docutils literal">$<span class="caps">OBJDIR</span>/config.status</tt>. The <tt class="docutils literal">substs</tt> list corresponds
to the <tt class="docutils literal"><span class="caps">CONFIG</span></tt> dictionary in <tt class="docutils literal">moz.build</tt> files. (It’s less obvious how
<tt class="docutils literal">defines</tt> works.)</p>
</div>
<div class="section" id="tweaking-a-build-setting">
<h2>Tweaking a build setting</h2>
<p>I’m going to take a work-in-progress patch from <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1132185">Bug 1132185</a> as my example. I’ve
applied the patch locally and pushed it to the review mercurial repository. You
can <a class="reference external" href="https://reviewboard-hg.mozilla.org/gecko/rev/006d3619a5da">view the commit</a> or fetch it with:</p>
<blockquote>
hg pull <a class="reference external" href="https://reviewboard-hg.mozilla.org/review">https://reviewboard-hg.mozilla.org/review</a> -r 006d3619a5da</blockquote>
<p>The changeset itself is small and self-contained — exactly how we like build
system changes. The patch:</p>
<ul class="simple">
<li>declares a new flag, <tt class="docutils literal">MOZ_ANDROID_TAB_QUEUE</tt>, in <tt class="docutils literal">configure.in</tt>;</li>
<li>exposes that flag to the preprocessor in <tt class="docutils literal">mobile/android/base/moz.build</tt>;</li>
<li>consumes the flag in the preprocessed source input <tt class="docutils literal">mobile/android/base/AppConstants.java.in</tt>;</li>
<li>and turns the flag on, for all builds and all trains, in <tt class="docutils literal">mobile/android/confvars.sh</tt>.</li>
</ul>
<p>There’s a weirdness here in the way that <tt class="docutils literal">mozconfig</tt> options are interpolated
into <tt class="docutils literal">configure.in</tt> which means that <em>setting a default value does not do what
you would expect</em>. So let’s remove the default in <tt class="docutils literal">configure.in</tt>:</p>
<div class="highlight"><pre><span class="gh">diff --git a/configure.in b/configure.in</span>
<span class="gd">--- a/configure.in</span>
<span class="gi">+++ b/configure.in</span>
<span class="gu">@@ -3938,17 +3938,16 @@ MOZ_WEBSMS_BACKEND=</span>
MOZ_ANDROID_NEW_TABLET_UI=
<span class="gd">-MOZ_ANDROID_TAB_QUEUE=</span>
ACCESSIBILITY=1
</pre></div>
<p>For the purposes of testing, that last is not desirable. So let’s go into
<tt class="docutils literal">mobile/android/confvars.sh</tt> and comment out those lines:</p>
<div class="highlight"><pre><span class="gh">diff --git a/mobile/android/confvars.sh b/mobile/android/confvars.sh</span>
<span class="gd">--- a/mobile/android/confvars.sh</span>
<span class="gi">+++ b/mobile/android/confvars.sh</span>
<span class="gu">@@ -75,17 +75,17 @@ MOZ_WEBGL_CONFORMANT=1</span>
# Enable the tab queue. This will go away in <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1132507">Bug 1132507</a>.
<span class="gd">-MOZ_ANDROID_TAB_QUEUE=1</span>
<span class="gi">+# MOZ_ANDROID_TAB_QUEUE=1</span>
</pre></div>
<p>Now, let’s run <tt class="docutils literal">mach configure</tt> and check the build settings in
<tt class="docutils literal">config.status</tt>:</p>
<div class="highlight"><pre>chocho:gecko nalexander$ ./mach configure && grep MOZ_ANDROID_TAB_QUEUE objdir-chrome/config.status
0:00.23 /usr/bin/make -f client.mk -s configure
0:01.00 Generating /Users/nalexander/Mozilla/gecko/configure using autoconf
0:01.28 cd /Users/nalexander/Mozilla/gecko/objdir-chrome
0:01.28 /Users/nalexander/Mozilla/gecko/configure
0:01.64 Adding configure options from /Users/nalexander/Mozilla/gecko/mozconfig-chrome
0:01.64 --target=arm-linux-androideabi
0:01.64 --enable-application=mobile/android
0:01.64 --with-android-ndk=/usr/local/Cellar/android-ndk/r8e-darwin-x86_64
0:01.64 --with-android-sdk=/usr/local/Cellar/android-sdk/22.3/platforms/android-21
0:01.77 loading cache ./config.cache
...
(''' MOZ_ANDROID_TAB_QUEUE ''', r''' '''),
</pre></div>
<p>Observe that we see a match for <tt class="docutils literal">MOZ_ANDROID_TAB_QUEUE</tt>, but there’s an empty
value. This match is in the <tt class="docutils literal">substs</tt> list — it means the variable is known,
but not set. When we test for existence, we’ll look for a non-empty value.</p>
<p>Now let’s add it to our local <tt class="docutils literal">mozconfig</tt> <a class="footnote-reference" href="#export" id="id1">[1]</a>:</p>
<div class="highlight"><pre><span class="gh">diff --git a/mozconfig b/mozconfig</span>
<span class="gd">--- a/mozconfig 2015-02-13 12:29:09.000000000 -0800</span>
<span class="gi">+++ b/mozconfig 2015-02-13 12:29:05.000000000 -0800</span>
<span class="gu">@@ -8,2 +8,4 @@</span>
mk_add_options MOZ_OBJDIR=./objdir-chrome
<span class="gi">+</span>
<span class="gi">+MOZ_ANDROID_TAB_QUEUE=1</span>
</pre></div>
<p>Re-run configure, and you should see the variable have a value:</p>
<div class="highlight"><pre>chocho:gecko nalexander$ ./mach configure && grep MOZ_ANDROID_TAB_QUEUE objdir-chrome/config.status
...
(''' MOZ_ANDROID_TAB_QUEUE ''', ' 1 '),
(''' MOZ_ANDROID_TAB_QUEUE ''', r''' 1 '''),
</pre></div>
<p>(The first value is in the <tt class="docutils literal">substs</tt> list; the second in the <tt class="docutils literal">defines</tt> list.
There’s a technical difference that I won’t get into here.)</p>
</div>
<div class="section" id="selective-rebuilds-in-mobile-android">
<h2>Selective rebuilds in <tt class="docutils literal">mobile/android</tt></h2>
<p>For the purposes of this section, I’m going to assume that we’re experimenting
with a new build flag that is Firefox for Android (Fennec) only. That means we
expect all impacts of the new build flag to be localized to the
<tt class="docutils literal">mobile/android</tt> directory, and in turn that lets us rebuild less of the tree.</p>
<p>In the <tt class="docutils literal">mobile/android</tt> directory, we’re very careful to preprocess as little
as possible, and we maintain our build dependencies quite actively, which makes
testing <tt class="docutils literal">mobile/android</tt>-only changes generally straight-forward. That brings
us to the <strong>third major point</strong>: you should always see your build changes after
running <tt class="docutils literal">mach build mobile/android</tt>.</p>
<p>You can verify by manually inspecting the following generated files in the
object directory:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">$(<span class="caps">OBJDIR</span>)/mobile/android/base/AndroidManifest.xml</span></tt></li>
<li><tt class="docutils literal"><span class="pre">$(<span class="caps">OBJDIR</span>)/mobile/android/base/res/values/strings.xml</span></tt></li>
<li><tt class="docutils literal"><span class="pre">$(<span class="caps">OBJDIR</span>)/mobile/android/base/generated/preprocessed/org/mozilla/gecko/AppConstants.java</span></tt></li>
<li><tt class="docutils literal"><span class="pre">$(<span class="caps">OBJDIR</span>)/dist/bin/modules/AppConstants.jsm</span></tt></li>
</ul>
<p>For example, adding and removing the <tt class="docutils literal">MOZ_ANDROID_TAB_QUEUE=1</tt> line in my
<tt class="docutils literal">mozconfig</tt> toggles the boolean in the following block of
<tt class="docutils literal">AppConstants.java</tt> for me:</p>
<div class="highlight"><pre> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">boolean</span> <span class="n">MOZ_ANDROID_TAB_QUEUE</span> <span class="o">=</span>
<span class="c1">//@line 178 "/Users/nalexander/Mozilla/gecko/mobile/android/base/AppConstants.java.in"</span>
<span class="kc">true</span><span class="o">;</span>
<span class="c1">//@line 182 "/Users/nalexander/Mozilla/gecko/mobile/android/base/AppConstants.java.in"</span>
</pre></div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>The Firefox for Android team is always making things better for contributors —
including shining light on the dark corners of the build system. <a class="reference external" href="https://wiki.mozilla.org/Mobile/Get_Involved">Get involved
with Firefox for Android</a> or
help build and test <a class="reference external" href="https://github.com/mozilla/firefox-ios">Firefox for iOS</a>.</p>
<p>Discussion is best conducted on the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a>
and I’m nalexander on <tt class="docutils literal">irc.mozilla.org</tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Mon 16 February 2015: Thanks to <a href="https://twitter.com/MartynHaigh">@MartynHaigh</a> for pointing out that I showed an
incorrect <tt class="docutils literal">mozconfig</tt> snippet using <tt class="docutils literal">mk_add_option</tt>. I’ve corrected the
text and included a small discussion <a class="footnote-reference" href="#export" id="id2">[1]</a>.</li>
</ul>
<table class="docutils footnote" frame="void" id="export" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label">[1]</td><td><em>(<a class="fn-backref" href="#id1">1</a>, <a class="fn-backref" href="#id2">2</a>)</em> Not specifying a default value in <tt class="docutils literal">configure.in</tt> lets you set the value in
your <tt class="docutils literal">mozconfig</tt> easily. You can either specify <tt class="docutils literal">MOZ_ANDROID_TAB_QUEUE=1</tt>
or <tt class="docutils literal">export MOZ_ANDROID_TAB_QUEUE=1</tt>. <strong>Nota bene</strong>: using <tt class="docutils literal">mk_add_option</tt>
form does not work! (That’s for exporting settings to <tt class="docutils literal">client.mk</tt>, which is
a different code path. Confusing, I know.)</td></tr>
</tbody>
</table>
</div>
Building Firefox for Android: I want my IDE!2015-01-08T16:20:00-08:00ncalexandertag:www.ncalexander.net,2015-01-08:blog/2015/01/08/building-firefox-for-android/<p>The Mozilla Corporation as a whole has been investing in a deliberate
2015 planning process, and some teams, in particular Cloud Services,
have done a fantastic job of circulating their first quarter goals. I
really applaud their efforts to be transparent, so I figured I would
mimic them (the sincerest form of flattery and all that).</p>
<div class="section" id="end-of-the-firefox-for-android-37-cycle">
<h2>End of the Firefox for Android 37 cycle</h2>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Discussion is best conducted on the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a>
and I’m nalexander on <tt class="docutils literal">irc.mozilla.org</tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
</div>
What I intend to work on for the Firefox 38 cycle2015-01-06T16:20:00-08:00ncalexandertag:www.ncalexander.net,2015-01-06:blog/2015/01/06/what-i-intend-to-work-on-for-the-firefox-38-cycle/<p>The Mozilla Corporation as a whole has been investing in a deliberate
2015 planning process, and some teams, in particular Cloud Services,
have done a fantastic job of circulating their first quarter goals. I
really applaud their efforts to be transparent, so I figured I would
mimic them (the sincerest form of flattery and all that).</p>
<p>I was quite deliberate when choosing my Firefox 36 cycle goals, and then
fell off the wagon for the 37 cycle. I found writing my goals for the
36 cycle personally helpful, but I only shared them with my direct
supervisor (<a href="https://twitter.com/mfinkle">@mfinkle</a>). One way that I surfaced them more broadly was
that I included my goals as top-level items in <a class="reference external" href="https://wiki.mozilla.org/Mobile/Notes/22-Oct-2014#nalexander">my section of the
weekly team meeting wiki page</a>
and
<a class="reference external" href="https://wiki.mozilla.org/Mobile/Notes/29-Oct-2014#nalexander">updated</a>
<a class="reference external" href="https://wiki.mozilla.org/Mobile/Notes/05-Nov-2014#nalexander">my</a>
<a class="reference external" href="https://wiki.mozilla.org/Mobile/Notes/12-Nov-2014#nalexander">status</a>
<a class="reference external" href="https://wiki.mozilla.org/Mobile/Notes/19-Nov-2014#nalexander">over</a>
<a class="reference external" href="https://wiki.mozilla.org/Mobile/Notes/26-Nov-2014#nalexander">time</a>.</p>
<p>In the spirit of transparency, and because I find knowing what my
colleagues are building interesting, here’s what I anticipate working on
for the reminder of the 37 cycle and the 38 cycle.</p>
<div class="section" id="end-of-the-firefox-for-android-37-cycle">
<h2>End of the Firefox for Android 37 cycle</h2>
<p>I have been pushing the Firefox for Android work that will migrate Old
Sync users to Firefox Accounts. The work is tracked by meta <a class="reference external" href="https://bugzilla.mozilla.org/showdependencytree.cgi?id=migratesyncandroid&hide_resolved=0">Bug migratesyncandroid</a>.</p>
<p>I’ve landed the Firefox Accounts client-side <span class="caps">UI</span> code that prompts the
user to finish migrating by logging in (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1098667">Bug 1098667</a>). I’ve landed the
Old Sync code that recognizes if another device in the same device
constellation has migrated and uploaded a migration sentinel (Bug 1091189).</p>
<p>The final piece of functionality is to message Old Sync users who have
not migrated (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=895526">Bug 895526</a>). We expect this set to be small, since all
Android Old Sync users needed a Desktop Old Sync device in order to
pair their Android device, and those users are certainly going to be
convinced by our Desktop messaging. We hope. This messaging only
needs to be on the release channel before we start deprecating the Old
Sync server farm, which is some number of releases after the start of
migration messaging (Firefox 37).</p>
<p>There’s an open request from rnewman for some tracking of the
Android-side migration process, be it <span class="caps">FHR</span>, telemetry, or whatever. I
intend to investigate this but I haven’t committed to it yet.</p>
<p>Sometime in the far future, we can rip out the existing Android Sync
code (tracked by meta Bug 957845).</p>
</div>
<div class="section" id="firefox-for-android-38-cycle">
<h2>Firefox for Android 38 cycle</h2>
<p>We intend to expose our first Firefox Account attached service
(Reading List) on Firefox for Android in the 38 cycle. The auth layer
will be FxA OAuth. I’ve written some of the OAuth code that we’ll
use; <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1117829">Bug 1117829</a> tracks rehabilitating the old code and adding the
Android Account authenticator to it. This is a hard blocker for the
Reading List service, so it’s a high priority for me. (But there’s no
particular reason another developer couldn’t pick it up.)</p>
<p>We have ongoing build system battles to fight and continual feature
and improvement requests. Solidifying <em>mach bootstrap</em> for
mobile/android provides outsize value relative to my time invested. I
will try to maintain and incrementally improve the IntelliJ
configuration that I have glued together, but really pushing the
Gradle/IntelliJ integration forward is not going to happen in the 38
cycle. It is my opinion that investing effort into the build system
to use new Google-provided features (split Google Play services,
shrinkResources, etc) is a bad use of our time. <a class="reference external" href="http://www.ncalexander.net/blog/2015/01/05/we-should-build-firefox-for-android-with-an-external-build-system/">We should instead
put the effort into transitioning our build to Gradle (or a
competitor) and get these incremental features cheaply</a>.
Given this assessment, I do not expect to spend time improving our <span class="caps">APK</span>
size this cycle.</p>
</div>
<div class="section" id="firefox-for-ios-38-cycle">
<h2>Firefox for iOS 38 cycle</h2>
<p>I’ll be building as much of the Firefox Accounts front-end and back-end
as I can get done for Firefox for iOS. This is a soup-to-nuts affair,
roughly tracked by meta Bug 1092561.</p>
<p>The first order of business is to decide between a native <span class="caps">UI</span> for signing
in and signing up or whether to host this <span class="caps">UI</span> on the web, using a hybrid
jelly-doughnut approach. We built a native <span class="caps">UI</span> for Firefox for Android,
but we’ve used a hosted web <span class="caps">UI</span> in Firefox for Desktop. Both work; both
have advantages and disadvantages. I’ll be evaluating and making a
recommendation for which approach to pursue in the first weeks of the 38 cycle.</p>
<p>Concurrently, I’ll be writing the <span class="caps">HTTP</span> clients and token-passing
plumbing required to keep a configured Firefox Account in a good state
and prepare the client to Sync. The abstractions we built for this in
Firefox for Android have weathered very well; this will be a "judicious
re-implementation" of the same ideas into Swift.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>That’s what I’m going to be doing. But I’m always trying to make
things better for contributors and <a class="reference external" href="https://bugzilla.mozilla.org/buglist.cgi?bug_status=__open__&email1=nalexander%40mozilla.com&emailbug_mentor1=1&emailtype1=exact">I love to mentor tickets</a>.
<a class="reference external" href="https://wiki.mozilla.org/Mobile/Get_Involved">Get involved with Firefox for Android</a> or help build and
test <a class="reference external" href="https://github.com/mozilla/firefox-ios">Firefox for iOS</a>.</p>
<p>Discussion is best conducted on the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a>
and I’m nalexander on <tt class="docutils literal">irc.mozilla.org</tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
</div>
We should build Firefox for Android with an external build system2015-01-05T16:20:00-08:00ncalexandertag:www.ncalexander.net,2015-01-05:blog/2015/01/05/we-should-build-firefox-for-android-with-an-external-build-system/<p>For some time, I have been advocating for building Fennec with a
full-featured Android build system. At one time, the state of the art
was Ant; now, the solution blessed by Google is <a class="reference external" href="http://tools.android.com/tech-docs/new-build-system/user-guide">Gradle</a>
<a class="footnote-reference" href="#android-gradle" id="id1">[1]</a>; both Facebook (<a class="reference external" href="http://facebook.github.io/buck/">buck</a>) and Twitter (<a class="reference external" href="http://pantsbuild.github.io/">pants</a>)
have built and published open-source build systems; and Google
(again!) has built a separate proprietary internal build system
(<a class="reference external" href="http://google-engtools.blogspot.ca/2011/06/build-in-cloud-accessing-source-code.html">blaze</a>) of their own. The major lesson I take from the
proliferation of these tools is:</p>
<p><strong>The best organizations invest in their tooling to make their people
more productive.</strong></p>
<p>However, this needs to be countered with the inescapable reality that
build systems, and especially Android build systems, are big investments.</p>
<p>As the lone Firefox for Android build peer, I do not support Mozilla
making a big investment into building our own Android build system.</p>
<p>For the sake of argument, however, such a Mozilla-developed build
system would look like a blend of custom <tt class="docutils literal">moz.build</tt> and a recursive
Make backend. The amount of new and un-tested build system code would
be very high; we’d incur serious risk, because all that new code needs
to support a delicate combination of features that are constantly
moving; and in all likelihood we wouldn’t achieve even 10% of what the
existing solutions achieve in a calendar year <a class="footnote-reference" href="#resourcing" id="id2">[2]</a>.</p>
<p>Rather than diving deep into a particular illustration of a feature
that would be difficult to express in such a Mozilla-developed build
system, let me give instead a whirlwind tour of features we could take
advantage of — <strong>today</strong> — if we used an existing solution.</p>
<p>Caveat: I have evaluated <em>Gradle</em> and <em>buck</em> extensively. I have not
evaluated <em>pants</em>. (<em>Blaze</em> is not available.) Not everything
claimed will be true for all external build systems.</p>
<div class="section" id="wins-for-building-fennec">
<h2>Wins for building Fennec</h2>
<div class="section" id="real-android-libraries-with-resources-and-manifests">
<h3>Real Android libraries, with resources and manifests.</h3>
<p>These libraries could, at first, just be third-party libraries that we
want to build against. (We’ve so far held off using, or modified to be
"part of" Fennec, any third-party libraries that reference Android resources.)</p>
<p>In future, we could build Fennec as a set of loosely-coupled
libraries. (In some ways we do: GeckoView, Stumbler, Search Activity,
Background Services… we’re doing this in a limited way.) Such small
libraries let us test faster and parallelize the development process.</p>
<p>There’s a possibility to reduce risk here, too: I claim that we can
improve our existing practice of landing new features behind a build
flag by developing mostly self-contained Android libraries and then
including those libraries behind a build flag. This lets developers
land the code in the tree, and possibly even compile and unit test on
<span class="caps">TBPL</span>, but not actually ship in Fennec until the build flag is flipped.</p>
<p>Merging resources and manifests is a solved problem; let’s use an
existing solution to do it.</p>
</div>
<div class="section" id="faster-builds">
<h3>Faster builds.</h3>
<p>Our existing build system supports compiling our existing libraries in
parallel, but it doesn’t support DEXing in parallel <a class="footnote-reference" href="#dex" id="id3">[3]</a>. Moving
to an existing build system that compiles and DEXes in parallel will
buy us a faster build even without separating our libraries. We get
big wins (which I will try to quantify in the near future) if we do a
little more work and split our libraries further.</p>
</div>
<div class="section" id="simpler-proguard-support">
<h3>Simpler Proguard support.</h3>
<p>We’re about to land <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1106593">Bug 1106593</a>, which addresses the fact that we
weren’t Proguarding everything we should have been. It adds two
bespoke stages to our build. This type of change would be simpler if
we were using an existing implementation (which would have certainly
Done the Right Thing) and only had to modify a Proguard configuration.</p>
</div>
<div class="section" id="trivial-multidex-support">
<h3>Trivial MultiDex support.</h3>
<p>Fennec is pushing up against the <tt class="docutils literal">classes.dex</tt> size limit. Google
recently added <a class="reference external" href="http://developer.android.com/tools/building/multidex.html">MultiDex</a> support, which allows further run-time
loaded <tt class="docutils literal">classesN.dex</tt> files. Gradle exposes the additional build
configuration with a handful of lines of code. Someday, we’ll
probably need this support.</p>
</div>
<div class="section" id="support-for-shrinking-android-resources">
<h3>Support for shrinking Android resources</h3>
<p>Gradle exposes a non-trivial build step for <a class="reference external" href="http://tools.android.com/tech-docs/new-build-system/resource-shrinking">resource shrinking</a>. We
want it.</p>
</div>
<div class="section" id="support-for-building-multiple-app-configurations">
<h3>Support for building multiple App configurations.</h3>
<p>By <em>configurations</em>, I mean building a standard release version of
Fennec, and a debug (non-Proguarded) version of Fennec for JUnit 3
testing, and possibly even multiple additional versions (resource
constrained; <span class="caps">API</span> limited) all with one "<span class="caps">TBPL</span> build job". Such build
configurations reduce automation machine load and reduce the risk of
build flags being orphaned.</p>
<p>There’s a similar, but limited, feature known as <a class="reference external" href="http://tools.android.com/tech-docs/new-build-system/user-guide/apk-splits">split APKs</a> that we
might want too.</p>
</div>
</div>
<div class="section" id="wins-for-contributors">
<h2>Wins for contributors</h2>
<div class="section" id="simpler-build-bootstrapping">
<h3>Simpler build bootstrapping.</h3>
<p>Gradle in particular has first class for bootstrapping itself via the
<a class="reference external" href="http://www.gradle.org/docs/current/userguide/gradle_wrapper.html">Gradle wrapper</a>. This wrapper prepares the Gradle part of the build
environment automatically and reduces the time needed to get a build started.</p>
</div>
<div class="section" id="first-class-dependency-management">
<h3>First class dependency management.</h3>
<p>External build systems, in particular Gradle, integrate with Maven/Ivy
repositories for consuming dependencies. That means we can specify,
in the tree, what versions we currently support and the build system
will do the work of fetching, configuring, and installing them. It’s
a faster first build and fewer frustrating dependency chases trying to
figure out what changed and how your local machine is different from
the official <span class="caps">TBPL</span> configuration.</p>
</div>
<div class="section" id="better-ide-integration">
<h3>Better <span class="caps">IDE</span> integration.</h3>
<p>Gradle integrates, mostly tightly, with <span class="caps">IDEA</span> IntelliJ and Google’s
Android Studio. This is a big win for new contributors (just open
<tt class="docutils literal">mobile/android</tt> in your <span class="caps">IDE</span> to build) and a huge boon for seasoned
developers who get the power of IntelliJ for their daily work.</p>
</div>
<div class="section" id="built-in-support-for-building-and-publishing-javadocs-for-our-own-use">
<h3>Built in support for building and publishing Javadocs for our own use.</h3>
<dl class="docutils">
<dt>Some parts of the code are well-commented; others are not. Perhaps we</dt>
<dd>want to document some of our trickier code flows in a public place?</dd>
</dl>
</div>
<div class="section" id="built-in-support-for-producing-android-lint-output">
<h3>Built in support for producing Android lint output.</h3>
<p>Catches bugs! For free!</p>
</div>
</div>
<div class="section" id="wins-for-geckoview">
<h2>Wins for GeckoView</h2>
<div class="section" id="existing-build-systems-have-better-systems-for-updating-only-specific-targets">
<h3>Existing build systems have better systems for updating only specific targets.</h3>
<p>Right now, the <tt class="docutils literal">moz.build</tt> based build system produces essentially all
artifacts at build time. That includes test APKs, packaging the
GeckoView library, an example application, etc. It’s slow! Most of
the time we just want the <em>ability</em> to package GeckoView; we really
just want a fresh Fennec <span class="caps">APK</span>.</p>
</div>
<div class="section" id="built-in-code-to-build-a-geckoview-aar-file">
<h3>Built in code to build a GeckoView <span class="caps">AAR</span> file</h3>
<p>This includes first class support for publishing the artifact to a
Maven repository. This would eliminate the need for
<a class="reference external" href="https://ci.mozilla.org/job/mozilla-central-geckoview/configure">https://ci.mozilla.org/job/mozilla-central-geckoview/configure</a>.</p>
</div>
<div class="section" id="support-for-producing-java-source-jars-and-javadoc-jars-for-geckoview">
<h3>Support for producing Java source JARs and Javadoc JARs for GeckoView.</h3>
<p>As the GeckoView library gains consumers, providing source JARs and
Javadoc JARs will make consumers lives significantly easier.</p>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Help me make our builds better! Discussion is best conducted on the
<a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a> and I’m nalexander on
<tt class="docutils literal">irc.mozilla.org</tt> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
<table class="docutils footnote" frame="void" id="android-gradle" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Truthfully, Gradle with Google’s own rapidly
developing Android-Gradle plugin. But this was the case with Ant,
too: Google shipped an Ant <tt class="docutils literal">build.xml</tt> that consumers extended
and customized.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="resourcing" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>That is, if Mozilla even was willing to pay a single
developer to work on such a build system for a whole year. As it
stands, we don’t pay a single full-time equivalent to work
exclusively on <em>any</em> part of the build system. We get a lot of
glandium’s time, a little of gps’s time, some of mshal’s time, some
of my time, etc, etc.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="dex" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>To be clear, we could support DEXing in parallel and merging
the results, but it’s yet another layer of complexity in our
Make-based build system. Add in supporting MultiDex and producing
debug and release artifacts… let’s not.</td></tr>
</tbody>
</table>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Mon 05 January 2015: Thanks to <a href="https://twitter.com/rnewman">@rnewman</a> for surfacing spelling and
grammatical errors.</li>
</ul>
</div>
The Firefox for Android build system in 20152014-12-08T16:20:00-08:00ncalexandertag:www.ncalexander.net,2014-12-08:blog/2014/12/08/the-firefox-for-android-build-system-in-2015/<p>Me and my colleagues <a href="https://twitter.com/lucasratmundo">@lucasratmundo</a>, <a href="https://twitter.com/mleibovic">@mleibovic</a>, <a href="https://twitter.com/michaelcomella">@michaelcomella</a>, and
vivekb attended the Community Building discussion at #mozlandia
(<a class="reference external" href="https://etherpad.mozilla.org/Portland-Community-Building">notes</a> and <a class="reference external" href="http://bespokeio.com/moz/portland/">slides</a> are available). <a href="https://twitter.com/mhoye">@mhoye</a> presented his thinking
about community building and engagement at Mozilla and beyond. I
interpreted Mike’s presentation through a bifurcated lens: I came away
feeling that there are <em>social</em> aspects to community engagement, such
as providing positive rewards and recognition; and there are
<em>technical</em> aspects to community engagement, such as maintaining
documentation and simplifying tooling requirements <a class="footnote-reference" href="#systemic" id="id1">[1]</a>.</p>
<p>People like <a href="https://twitter.com/lsblakk">@lsblakk</a> are able to bring new people into our community
with phenomenal outreach programs like the <a class="reference external" href="http://ascendproject.org/">Ascend Project</a>, but
that’s not my skill-set. I deeply admire the <em>social</em> work Lukas (and
others!) are doing, but I personally am most able to empower the
Mozilla community to own Firefox for Android by addressing the
<em>technical</em> aspects Mike discussed.</p>
<div class="section" id="making-it-easier-to-contribute-to-firefox-for-android">
<h2>Making it easier to contribute to Firefox for Android</h2>
<p>In this context, the following initiatives will drive the Firefox for
Android tooling:</p>
<ol class="arabic simple">
<li>making it easier to build Firefox for Android the first time;</li>
<li>reducing the edit-compile-test cycle time;</li>
<li>making the Firefox for Android development process look and feel
like the standard Android development process.</li>
</ol>
<div class="section" id="making-it-easier-to-build-firefox-for-android-the-first-time">
<h3>Making it easier to build Firefox for Android the first time</h3>
<p>One strong claim made by mhoye — and supported by many others in the
room — is that <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Simple_Firefox_build/Linux_and_MacOS_build_preparation">mach bootstrap</a> has significantly reduced the
technical accessibility barrier to building Firefox for Desktop. We
need to implement <tt class="docutils literal">mach bootstrap</tt> for Firefox for Android.</p>
<p>For those who don’t know, <tt class="docutils literal">mach bootstrap</tt> is a script that prepares
the Firefox build environment, including fetching, updating, and
installing the pre-requisites needed for building Firefox. It
automates the (often difficult!) task of fetching dependencies;
ensures that known-good versions of dependencies are installed; and
sets development environment defaults. <tt class="docutils literal">mach bootstrap</tt> is the first
thing that should be run in a fresh Firefox source tree <a class="footnote-reference" href="#bootstrap" id="id2">[2]</a>.</p>
<p>Firefox for Android has more complicated dependencies than Firefox for
Desktop, including some that cannot be easily distributed or
installed: the Java development kit and run-time environment, the
Android <span class="caps">SDK</span> and <span class="caps">NDK</span>; Google’s Play Services libraries, etc. We can
save new contributors a long dependency chase before they see a
positive result. In addition, seasoned developers spend an
unknown-but-large amount of time discovering that the required
dependencies have advanced. Pinning the build to known-good versions,
failing the build when said versions are not present, and providing
<tt class="docutils literal">mach bootstrap</tt> to update to known-good versions will reduce this frustration.</p>
<p>A contributor started writing a <a class="reference external" href="https://github.com/ibrahimdraidia/FennecEnvSetup/blob/master/FennecSetup.sh">shell script</a> that does the work of
<tt class="docutils literal">mach bootstrap</tt>. <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1108771">Bug 1108771</a> tracks building upon this effort.
I’ve also filed <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1108782">Bug 1108782</a> to track pinning the Firefox for Android
build requirements to known-good versions.</p>
</div>
<div class="section" id="reducing-the-firefox-for-android-edit-compile-test-cycle-time">
<h3>Reducing the Firefox for Android edit-compile-test cycle time</h3>
<p>Firefox for Android is an unusual Android application: a large C++
library backing a medium-sized Java front-end, all plumbed together
with a JavaScript-based message passing system. Right now, building
the C++ library takes roughly 12 minutes on my system. Building the
Java front-end takes roughly 2 minutes, and the JavaScript parts are
essentially free. In 2015, glandium has taken a first quarter goal to
make it possible to build Firefox (for Desktop and for Android)
without building that large C++ library at all <a class="footnote-reference" href="#glandium" id="id3">[3]</a>. In the
future, purely front-end developers (<span class="caps">XUL</span>/JavaScript developers on
Desktop; Java/JavaScript developers on Android) will download and
cache the C++ build artifacts and build the application on top of the
cached artifacts. Firefox for Android is really well-suited to this
mode of operation because our dependencies are so well-defined. I’ve
filed <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1093242">Bug 1093242</a> to track part of this work.</p>
<p>The previous work will make it faster to build Firefox for Android the
first time, because we won’t build C++ libraries. We’re also going to
invest in making each incremental build faster, and there’s some
low-hanging fruit here. Right now, the most costly parts of our build
are compiling individual <span class="caps">JAR</span> libraries and DEXing all of the resulting
<span class="caps">JAR</span> libraries. Every time we split our <span class="caps">JAR</span> libraries, we can
parallelize a small part of our build and reduce the wall-clock time
of our Java compilation. Right now we could split our single
third-party <span class="caps">JAR</span> library and save ourselves compile time. And we’re
very close to being able to split the Background Services (Sync,
Firefox Accounts, Firefox Health Report, etc) library out of Fennec
proper, which will save even more compile time.</p>
<p>Improving our DEXing time is more difficult. Android’s <span class="caps">DEX</span> processor
is a byte-code transformation step that turns Java’s byte-code into
Dalvik <span class="caps">VM</span> byte-code. For historical reasons, we <span class="caps">DEX</span> the entirety of
Fennec’s byte-code in one <span class="caps">DEX</span> invocation, and it’s both a
single-process bottleneck and terribly expensive. For some time, it
has been possible to <span class="caps">DEX</span> each individual library in parallel and to
merge the resulting <span class="caps">DEX</span> files. All modern Android build system (such
as <a class="reference external" href="http://facebook.github.io/buck/">buck</a> or <a class="reference external" href="http://tools.android.com/tech-docs/new-build-system/user-guide">Gradle</a>) support this. We could support this in the
Firefox for Android build system as well, but I think we should move
to use a more featured build system under the hood instead. Android
build systems are very complicated; we don’t want to write our own,
and we definitely don’t want to write our own in Make syntax. In
2015, we’ll push to use a full-featured build tool that brings this
<span class="caps">DEX</span>-time improvement. More on this in a future post.</p>
</div>
<div class="section" id="making-the-firefox-for-android-development-process-standards-compliant">
<h3>Making the Firefox for Android development process "standards compliant"</h3>
<p>This point is a more philosophical point than the others. Firefox for
Android wins when we engage our community. The community of Android
developers is large and constantly pushing the boundaries of what’s
possible on a device. We want to tap into that well-spring of talent
and innovation, and every thing we do that’s non-standard to an
Android developer makes it harder for us to do this. Contributor
<a href="https://twitter.com/fedepaol">@fedepaol</a> wrote a <a class="reference external" href="http://fedepaol.github.io/blog/2013/07/29/contributing-to-firefox-for-android/">blog post</a> about how difficult this used to be.</p>
<p>The good news is, we’re getting better: we have rudimentary Gradle
support and you can use IntelliJ now. But we still have a long, long
way to go. We’ve got lots of easy wins just waiting for us: tickets
like <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1107811">Bug 1107811</a> will go a long way towards making the Fennec
"Android standards compliant" IntelliJ experience better. I have a
technical plan to land in-source-tree IntelliJ configurations, so
developers can open <tt class="docutils literal">mobile/android</tt> directly in IntelliJ and get to
a working Firefox for Android <span class="caps">APK</span> in the <span class="caps">IDE</span> in one step.</p>
<p>At a lower level, Tickets like <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1074258">Bug 1074258</a> will let us use the
IntelliJ design view more easily, and landing Android JUnit 3
Instrumentation test runners in automation (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1064004">Bug 1064004</a>) will make
local testing significantly easier than the convoluted Robocop process
we have right now. The list goes on and on.</p>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>The Firefox for Android team moved strongly towards easier builds and
ubiquitous tooling in 2014. 2015 is going to be even better. We’re
going to improve our technical experience in (at least!) three ways:
making the first build easier; making the next builds faster; and
unlocking the power of the standard Android developer toolchain.</p>
<p>Join us! Discussion is best conducted on the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev
mailing list</a> and I’m nalexander on <tt class="docutils literal">irc.mozilla.org</tt> and
<a href="https://twitter.com/ncalexander">@ncalexander</a> on Twitter.</p>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
<table class="docutils footnote" frame="void" id="systemic" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>I believe there is an over-arching third aspect, that
of the <em>system</em> in which we do our work and interact with the
community, but right-here-right-now I don’t feel empowered to change
this. Systemic change requires making community engagement part of
every team’s top-level goals, and achieving such goals requires
resources that are allocated well above my pay-grade.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="bootstrap" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>In fact, the bootstrapper does not even require a
source check-out — you can download just the script and it will fetch
enough to bootstrap itself. So it’s more accurate to say just
<em>bootstrap</em> rather than <em>mach bootstrap</em>, but <em>mach bootstrap</em> has won
the vocabulary battle in this arena.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="glandium" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>glandium has written a very informative blog post about
the future of the Firefox build system at
<a class="reference external" href="http://glandium.org/blog/?p=3318">http://glandium.org/blog/?p=3318</a>. The section relevant to this
discussion is <em>Specialized incremental builds</em>.</td></tr>
</tbody>
</table>
</div>
Building Fennec with Gradle and IntelliJ: first steps2014-10-23T22:00:00-07:00ncalexandertag:www.ncalexander.net,2014-10-23:blog/2014/10/23/building-fennec-with-gradle-and-intellij-first-steps/<!-- -*- Mode: rst; fill-column: 80; -*- -->
<p>Developing Fennec with Eclipse has been working well for quite some time now,
but Eclipse is officially no longer supported by Google and the new standard is
to build with Gradle and to edit in Android Studio or IntelliJ. I recently
landed <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=intellij"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1041395">Bug 1041395</a></a> and friends, which makes it easy to build Fennec with
Gradle. Here is a companion demonstration screencast.</p>
<div class="vimeo" align="left"><iframe width="800" height="450" src="https://player.vimeo.com/video/109896097" frameborder="0"></iframe></div><div class="section" id="instructions">
<h2>Instructions</h2>
<div class="section" id="prerequisites">
<h3>Prerequisites</h3>
<p>You need the <em>Android Support Repository</em> and <em>Google Repository</em> installed.
These install local Maven repositories of <span class="caps">SDK</span> libraries that the Gradle build
depends on. To install the repositories:</p>
<div class="highlight"><pre>android update sdk --no-ui --filter "extra-google-m2repository,extra-android-m2repository"
</pre></div>
<p>You can also install the repositories manually: use the <tt class="docutils literal">android</tt> tool and
select them from the <em>Extras</em> section.</p>
<a class="reference external image-reference" href="https://www.ncalexander.net/blog/images/Gradle.01.android.Maven.repositories.jpg"><img alt="" class="align-center" src="https://www.ncalexander.net/blog/images/Gradle.01.android.Maven.repositories.jpg" /></a>
</div>
<div class="section" id="test-a-build-with-gradle">
<h3>Test a build with Gradle</h3>
<div class="highlight"><pre>./mach build && ./mach package
cd $OBJDIR/mobile/android/gradle
./gradlew build
</pre></div>
<p>The debug <span class="caps">APK</span> will be at <tt class="docutils literal"><span class="pre">$<span class="caps">OBJDIR</span>/mobile/android/base/app/build/outputs/apk/app-debug.apk</span></tt>.</p>
</div>
<div class="section" id="test-a-build-with-intellij">
<h3>Test a build with IntelliJ</h3>
<p>The <tt class="docutils literal">$<span class="caps">OBJDIR</span>/mobile/android/gradle</tt> directory can be imported into IntelliJ as follows:</p>
<ul class="simple">
<li>File > Import Project</li>
<li>[select <tt class="docutils literal">$<span class="caps">OBJDIR</span>/mobile/android/gradle</tt>]</li>
<li>Import project from external model > Gradle</li>
<li>[select Use default Gradle wrapper]</li>
</ul>
<p>When prompted, do not add any files to git. You may need to re-open the
project, or restart IntelliJ, to pick up a compiler language-level change.</p>
</div>
</div>
<div class="section" id="technical-overview">
<h2>Technical overview</h2>
<div class="section" id="caveats">
<h3>Caveats</h3>
<ul class="simple">
<li>The Gradle build will "succeed" but crash on start up if the object directory
has not been properly packaged.</li>
<li>Changes to preprocessed source code and resources (namely, <tt class="docutils literal">strings.xml.in</tt>
and the accompanying <span class="caps">DTD</span> files) are not recognized.</li>
<li>There’s no support for editing JavaScript.</li>
</ul>
</div>
<div class="section" id="how-the-gradle-project-is-laid-out">
<h3>How the Gradle project is laid out</h3>
<p>To the greatest extent possible, the Gradle configuration lives in the source
directory. The only Gradle configuration that lives in the object directory is
installed when building the <tt class="docutils literal">mobile/android/gradle</tt> directory.</p>
<p>At the time of writing, their are three sub-modules: <em>app</em>, <em>base</em>, and <em>thirdparty</em>.</p>
<p><em>app</em> is the Fennec wrapper; it generates the <strong>org.mozilla.fennec.R</strong> resource
package. <em>base</em> is the Gecko code; it generates the <strong>org.mozilla.gecko.R</strong>
resource package. Together, <em>app</em> and <em>base</em> address the "two package
namespaces" that has plagued Fennec from day one.</p>
<p>Due to limitations in the Android Gradle plugin, all test code is shoved into
the <em>app</em> module. (The issue is that, at the time of writing, there is no
support for test-only APKs.) For no particular reason, the compiled C/C++
libraries are included in the <em>app</em> module; they could be included in the <em>base</em>
module. I expect <em>base</em> to rebuilt slightly more frequently than <em>app</em>, so I’m
hoping this choice will allow for faster incremental builds.</p>
<p><em>thirdparty</em> is the external code we use in Fennec; it’s built as an Android
library but uses no resources. It’s separate simply to allow the build system
to cache the compiled and pre-dexed artifacts, hopefully allowing for faster
incremental builds.</p>
</div>
<div class="section" id="recursive-make-backend-details">
<h3>Recursive make backend details</h3>
<p>The <tt class="docutils literal">mobile/android/gradle</tt> directory writes the following into
<tt class="docutils literal">$<span class="caps">OBJDIR</span>/mobile/android/gradle</tt>:</p>
<ol class="arabic simple">
<li>the Gradle wrapper;</li>
<li><tt class="docutils literal">gradle.properties</tt>;</li>
<li>symlinks to certain source and resource directories.</li>
</ol>
<p>The Gradle wrapper is written to make it easy to build with Gradle from
the object directory. The wrapper is <a class="reference external" href="http://www.gradle.org/docs/current/userguide/gradle_wrapper.html">intended to be checked into
version control</a>.</p>
<p><tt class="docutils literal">gradle.properties</tt> is the single source of per-object directory Gradle
configuration, and provides the Gradle configuration access to
configure/moz.build variables.</p>
<p>The symlinks are not necessary for the Gradle build itself, but they
prevent nested directory errors and incorrect Java package scoping when
the Gradle project is imported into IntelliJ. Because IntelliJ treats
the Gradle project as authoritative, it’s not sufficient to fix these
manually in IntelliJ after the initial import — IntelliJ reverts to the
Gradle configuration after every build. Since there aren’t many
symlinks, I’ve done them in the Makefile rather than at a higher level
of abstraction (like a moz.build definition, or a custom build backend).
In future, I expect to be able to remove all such symlinks by making our
in-tree directory structures agree with what Gradle and IntelliJ expect.</p>
</div>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
<p>Many thanks to <a class="reference external" href="http://perplexedturnip.wordpress.com/">ckitching</a> for doing the first work on developing Fennec with
IntelliJ. Thanks also to <a class="reference external" href="https://twitter.com/MartynHaigh">mhaigh</a> for championing IntelliJ as the tool of
choice for the Fennec front-end team, and to <a class="reference external" href="https://twitter.com/DigDug2K">wesj</a> for circulating updated
IntelliJ usage notes.</p>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Fri 14 Nov 2014: Included Maven repository prerequisites. Thanks to
<em>mcomella</em> for reporting the underlying issue and to <span class="caps">IRC</span> user <em>rogeliodh</em> for
the command line one-liner.</li>
</ul>
</div>
Build your own browser: A Maven repository for GeckoView2014-07-10T16:20:00-07:00ncalexandertag:www.ncalexander.net,2014-07-10:blog/2014/07/10/build-your-own-browser-a-maven-repository-for-geckoview/<!-- -*- Mode: rst; fill-column: 80; -*- -->
<p>GeckoView is a project that lets you embed the Gecko rendering engine into your
Android App. Slowly but surely, we’ve making this process easier. It’s now
<em>really</em> easy to include GeckoView in your Gradle-based application, thanks to a
new Maven repository hosting Nightly GeckoView builds.</p>
<p>GeckoView is a long-time Fennec (Firefox for Android) side-project: you can see
the <a class="reference external" href="https://wiki.mozilla.org/Mobile/Projects/GeckoView">the GeckoView project page</a> and <a class="reference external" href="http://starkravingfinkle.org/blog/2013/10/geckoview-embedding-gecko-in-your-android-application/">the first GeckoView blog post</a>. The
first sample code, <a class="reference external" href="https://github.com/mfinkle/geckobrowser">the original geckobrowser</a> is still working too, but in the
year since the initial development, progress has been slow.</p>
<p>I think part of the reason that progress has been slow is that it’s quite
difficult to embed GeckoView into an App — at least, it’s quite tricky if you
use it how it’s packaged on Mozilla’s build infrastructure <a class="footnote-reference" href="#packaged" id="id1">[1]</a>. Now,
there’s an easier way that takes advantage of Gradle’s excellent <a class="reference external" href="http://www.gradle.org/docs/current/userguide/artifact_dependencies_tutorial.html">support for
Maven repositories</a>.</p>
<div class="section" id="a-maven-repository-for-geckoview">
<h2>A Maven repository for GeckoView</h2>
<p>A <a class="reference external" href="https://ci.mozilla.org/job/mozilla-central-geckoview/">Jenkins job</a>, running on <tt class="docutils literal">ci.mozilla.org</tt>, builds a <a class="reference external" href="http://tools.android.com/tech-docs/new-build-system/aar-format">new <span class="caps">AAR</span> (Android
ARchive) library</a> file <a class="footnote-reference" href="#aar" id="id2">[2]</a>. The new job runs every night at <span class="caps">5AM</span> Pacific;
Nightly builds are usually kicked off between 2 and <span class="caps">3AM</span> Pacific, so the
artifacts should usually be fresh.</p>
<p>The <span class="caps">AAR</span> files produced are versioned appropriately <a class="footnote-reference" href="#version" id="id3">[3]</a> and then published
in the Maven repository hosted at
<a class="reference external" href="https://ci.mozilla.org/job/mozilla-central-geckoview/mozilla-central_Maven_Repository">https://ci.mozilla.org/job/mozilla-central-geckoview/mozilla-central_Maven_Repository</a>
The <span class="caps">AAR</span> artifacts are pushed with <tt class="docutils literal">groupId=org.mozilla.geckoview</tt> and
<tt class="docutils literal">artifactId=library</tt>; to refer to the latest <span class="caps">AAR</span> in Gradle, use:</p>
<pre class="literal-block">
repositories {
maven {
url 'https://ci.mozilla.org/job/mozilla-central-geckoview/mozilla-central_Maven_Repository'
}
}
dependencies {
compile 'com.android.support:support-v4:19.+'
compile 'org.mozilla.geckoview:library:+'
}
</pre>
<p>That’s it; that’s all you need to build against GeckoView <a class="footnote-reference" href="#latest" id="id4">[4]</a> <a class="footnote-reference" href="#corruption" id="id5">[5]</a>. For a worked
example, keep reading.</p>
</div>
<div class="section" id="example-an-updated-geckobrowser">
<h2>Example: an updated geckobrowser</h2>
<p>You can follow along with the repository at <a class="reference external" href="https://github.com/ncalexan/geckobrowser-gradle">https://github.com/ncalexan/geckobrowser-gradle</a>.</p>
<p>As of Android 19, the <tt class="docutils literal">android create project</tt> tool can create Gradle
projects, so let’s use it:</p>
<div class="highlight"><pre>~/Mozilla/geckobrowser-gradle $ android create project \
-a MainActivity -k org.mozilla.geckobrowser -t android-19 -g -p . -v 0.12
Error: Project folder '.' is not empty. Please consider using 'android update' instead.
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/java
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/java/org/mozilla/geckobrowser
Added file ./src/main/java/org/mozilla/geckobrowser/MainActivity.java
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/instrumentTest/java
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/instrumentTest/java/org/mozilla/geckobrowser
Added file ./src/instrumentTest/java/org/mozilla/geckobrowser/MainActivityTest.java
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/res
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/res/values
Added file ./src/main/res/values/strings.xml
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/res/layout
Added file ./src/main/res/layout/main.xml
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/res/drawable-xhdpi
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/res/drawable-hdpi
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/res/drawable-mdpi
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/src/main/res/drawable-ldpi
Added file ./src/main/AndroidManifest.xml
Added file ./build.gradle
Created directory /Users/nalexander/Mozilla/geckobrowser-gradle/gradle/wrapper
</pre></div>
<p>My new project is in directory <tt class="docutils literal"><span class="pre">geckobrowser-gradle</span></tt> and the original is in
directory <tt class="docutils literal">geckobrowser</tt>. Copy the Android resources (<tt class="docutils literal">res/</tt> directory)
verbatim, and take <tt class="docutils literal">MainActivity.java</tt> from <a class="reference external" href="https://github.com/mfinkle/geckobrowser">the original geckobrowser</a>; it
will need some light editing:</p>
<div class="highlight"><pre>~/Mozilla/geckobrowser-gradle $ rm -rf src/main/res && cp -R ../geckobrowser/res src/main
~/Mozilla/geckobrowser-gradle $ cp ../geckobrowser/src/com/starkravingfinkle/geckobrowser/MainActivity.java \
src/main/java/org/mozilla/geckobrowser/MainActivity.java
</pre></div>
<p>We’re almost there; we just need to rename the package. Update the Java package
in <tt class="docutils literal">MainActivity.java</tt> like so:</p>
<div class="highlight"><pre><span class="gd">--- a/src/main/java/org/mozilla/geckobrowser/MainActivity.java</span>
<span class="gi">+++ b/src/main/java/org/mozilla/geckobrowser/MainActivity.java</span>
<span class="gu">@@ -1,4 +1,4 @@</span>
<span class="gd">-package com.starkravingfinkle.geckobrowser;</span>
<span class="gi">+package org.mozilla.geckobrowser;</span>
import org.mozilla.gecko.GeckoView;
import org.mozilla.gecko.GeckoView.Browser;
</pre></div>
<p>After building and pushing to device using <tt class="docutils literal">./gradlew build installDebug</tt>, I
took the following screen capture:</p>
<img alt="" class="align-center" src="https://www.ncalexander.net/blog/images/geckobrowser-gradle.png" />
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>That’s a browser — minimal but functional — built around GeckoView! And it
uses "Android standard" packaging techniques: no fussing with manually managed
<span class="caps">ZIP</span> files. I’ve worked on improving packaging Fennec and GeckoView so that
things like this were at least feasible, and now that I’ve done it, I’d like to
congratulate the Android Tools team for steadily improving the Android packaging
story. It’s still not <em>perfect</em> <a class="footnote-reference" href="#perfect" id="id6">[6]</a>, but it handles most functional
requirements and is a huge improvement over the frustrating limitations of
earlier iterations.</p>
<p>Finally, a plea: GeckoView needs consumers to drive its development. The Fennec
team tries to move the project forward when possible, but without guiding lights
and community involvement (other than Fennec itself, and the Fennec team),
making GeckoView more functional always plays a poor second fiddle to making
Fennec more awesome than it already is. But we need to bring the Open Web to
more places than those where Fennec can take it, and for that, we need
GeckoView, and your help.</p>
</div>
<div class="section" id="notes">
<h2>Notes</h2>
<table class="docutils footnote" frame="void" id="packaged" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Every mozilla-central Nightly build includes GeckoView, in the
form of two zip files. The first, <tt class="docutils literal">geckoview_library.zip</tt>, contains compiled
code (Java JARs) and Android resources that need to be included in the embedding
App. The second, <tt class="docutils literal">geckoview_assets.zip</tt>, contains specially compressed native
code libraries (<tt class="docutils literal">.so</tt> files) that need to be copied into the embedding App’s
<tt class="docutils literal">assets/</tt> directory. Who likes manual dependency management? Not me!</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="aar" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>It’s not well-documented, but an <tt class="docutils literal">assets/</tt> directory packaged into
an <span class="caps">AAR</span> will be merged with the embedding application’s assets, and we take
advantage of this fact.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="version" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>The Maven version is the Gecko build id of the corresponding Nightly
build; for example, <tt class="docutils literal">20140710071924</tt> is a version from today.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="latest" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>Technically, the incantation <tt class="docutils literal">org.mozilla.geckoview:library:+</tt>
means to find the <em>latest</em> version of the <tt class="docutils literal">library</tt> artifact (in this case,
the appropriate <span class="caps">AAR</span>) in the group <tt class="docutils literal">org.mozilla.geckoview</tt>. That’s living
on the bleeding edge, to be sure. I <em>think</em> the current Jenkins job saves
each Nightly it produces, but I’m not <em>certain</em>, and in any case, each <span class="caps">AAR</span> is
roughly 35Mb, so I’m going to have start culling pretty much immediately.
I’m not sure what the right way to host such a large Maven repository is, so
the version scheme is likely to change in the (near) future. If you depend
on a specific version, download and install it into your local Maven
repository manually.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="corruption" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td>I did witness a transient error downloading from the Maven
repository, perhaps during the first download. The content length was
mis-reported and the <span class="caps">AAR</span> file was valid but contained redundant data. Let me
know if you see this; I have only witnessed it the one time.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="perfect" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id6">[6]</a></td><td>For example there is only patchy Gradle plugin documentation, and
with the shift to Gradle, there is little support for library dependency
management at the <tt class="docutils literal">aapt</tt> level. This matters for Fennec, since we don’t
use Gradle, but would like to participate in the <span class="caps">AAR</span> ecosystem.</td></tr>
</tbody>
</table>
</div>
Bumpy landings: How to land a Fennec feature behind a build flag2014-07-09T10:24:00-07:00ncalexandertag:www.ncalexander.net,2014-07-09:blog/2014/07/09/how-to-land-a-fennec-feature-behind-a-build-flag/<!-- -*- Mode: rst; fill-column: 80; -*- -->
<p>Fennec (Firefox for Android) features are staged and ride the trains (Nightly,
Aurora, and Beta) before reaching the Release audience. Features that land on
Nightly may — or may not — continue to Aurora. To support rapid Nightly
development, while letting code mature before it reaches Aurora, you should land
your new feature behind a runtime preference or a build flag. Here’s a guide to
landing behind such a flag.</p>
<div class="section" id="examples">
<h2>Examples</h2>
<p>We’ll use the following ticket as an example of landing a new build flag, code
behind the flag, and build system integration:</p>
<ul class="simple">
<li><a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864">Bug 1021864</a></a></a></a></a> landed the code and Android resources for the <em>Search
Activity</em>. The Search Activity is an Android activity that depends on
GeckoView. As such, it’s built as part of the main Fennec code-base, since
GeckoView is essentially the same as Fennec. The build flag is
<tt class="docutils literal">MOZ_ANDROID_SEARCH_ACTIVITY</tt>.</li>
</ul>
<p>You can also look at the following tickets, which are being actively developed
during the Firefox for Android 33 cycle.</p>
<ul class="simple">
<li><a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1024708"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1024708">Bug 1024708</a></a> will land the code and Android manifest integration for the
<em>Mozilla Stumbler</em>. The stumbler is an Android background service that
uploads location sensor data to Mozilla. That location sensor data is then
reflected to device users through the Mozilla Location Service. This helps
device users locate themselves in the world, without requiring a <span class="caps">GPS</span> lock.
Since the stumbler presents no <span class="caps">UI</span> directly, it’s built as a separate Java <span class="caps">JAR</span>,
and integrated into the Fennec <span class="caps">APK</span> via manifest fragments.</li>
<li><a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1033560"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1033560">Bug 1033560</a></a> "flipped the switch" to enable flinging videos to Google
Chromecast devices. (The code landed in a long sequence of earlier tickets.)
The Chromecast source code is part of the main Fennec code base, and parts of
it are compiled (or manifest fragments included) conditionally.</li>
</ul>
</div>
<div class="section" id="guide-to-landing">
<h2>Guide to landing</h2>
<p>Let’s look at <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864">Bug 1021864</a></a></a></a></a>. This ticket landed as a series of 5 commits
<a class="footnote-reference" href="#commits" id="id1">[1]</a>.</p>
<div class="section" id="build-flags">
<h3>Build flags</h3>
<p>The interesting commits of <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864">Bug 1021864</a></a></a></a></a> are the <a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/7d404850f5f2">second</a> and the <a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/9395fb24971c">fifth</a>.
The <a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/7d404850f5f2">second</a> commit adds a build flag, <tt class="docutils literal">MOZ_ANDROID_SEARCH_ACTIVITY</tt>, that
defaults to not being set:</p>
<div class="highlight"><pre><span class="gd">--- a/configure.in</span>
<span class="gi">+++ b/configure.in</span>
<span class="gu">@@ -3915,16 +3915,17 @@ if test -n "$MOZ_RTSP"; then</span>
fi
MOZ_LOCALE_SWITCHER=
<span class="gi">+MOZ_ANDROID_SEARCH_ACTIVITY=</span>
ACCESSIBILITY=1
</pre></div>
<p>And enables it just for <tt class="docutils literal">mobile/android</tt>:</p>
<div class="highlight"><pre><span class="gd">--- a/mobile/android/confvars.sh</span>
<span class="gi">+++ b/mobile/android/confvars.sh</span>
<span class="gu">@@ -69,8 +69,11 @@ fi</span>
# Enable second screen using native Android libraries
MOZ_NATIVE_DEVICES=
<span class="gi">+</span>
<span class="gi">+# Enable the Search Activity.</span>
<span class="gi">+MOZ_ANDROID_SEARCH_ACTIVITY=1</span>
</pre></div>
</div>
<div class="section" id="guarding-a-feature-behind-the-build-flag">
<h3>Guarding a feature behind the build flag</h3>
<p>Use the <a class="reference external" href="https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/buildsystem/preprocessor.html">preprocessor</a> to conditionally include code (and other
includes, etc) behind the build flag. For example, the <a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/3e8565120b1b">first</a> commit landed
all the code for the initial version of the Search Activity with no build
integration at all. Then, the <a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/a7a8ea9e736b">third</a> commit exposed the build flag:</p>
<div class="highlight"><pre><span class="gd">--- a/mobile/android/base/locales/moz.build</span>
<span class="gi">+++ b/mobile/android/base/locales/moz.build</span>
<span class="gu">@@ -1,6 +1,8 @@</span>
# -*- Mode: python; c-basic-offset: 4; indent-tabs-mode: nil; tab-width: 40 -*-
# vim: set filetype=python:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
<span class="gi">+if CONFIG['MOZ_ANDROID_SEARCH_ACTIVITY']:</span>
<span class="gi">+ DEFINES['MOZ_ANDROID_SEARCH_ACTIVITY'] = 1</span>
</pre></div>
<p>and used the flag in a preprocessed Android resource file:</p>
<div class="highlight"><pre><span class="gd">--- a/mobile/android/base/strings.xml.in</span>
<span class="gi">+++ b/mobile/android/base/strings.xml.in</span>
<span class="gu">@@ -3,16 +3,19 @@</span>
<!-- This Source Code Form is subject to the terms of the Mozilla Public
- License, v. 2.0. If a copy of the MPL was not distributed with this
- file, You can obtain one at http://mozilla.org/MPL/2.0/. -->
<string name="android_package_name_for_ui">@ANDROID_PACKAGE_NAME@</string>
<span class="gi">+</span>
<span class="gi">+#ifdef MOZ_ANDROID_SEARCH_ACTIVITY</span>
<span class="gi">+#include ../search/strings/search_strings.xml.in</span>
<span class="gi">+#endif</span>
<span class="gi">+</span>
#include ../services/strings.xml.in
</pre></div>
<p>The <a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/bc6a8b22530a">fourth</a> commit includes the bulk of the integration: the Java code itself
is built and included in Fennec by the code changes in
<tt class="docutils literal">mobile/android/base/moz.build</tt> and <tt class="docutils literal">mobile/android/base/Makefile.in</tt>.</p>
</div>
<div class="section" id="landing-disabled">
<h3>Landing disabled</h3>
<p>The <a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/9395fb24971c">fifth</a> commit <em>disables</em> the Search Activity. This is because we wanted
to land build-time disabled, work out initial issues using local and try builds,
and then build-time enable when the feature stabilizes.</p>
<div class="highlight"><pre><span class="gd">--- a/mobile/android/confvars.sh</span>
<span class="gi">+++ b/mobile/android/confvars.sh</span>
# Enable second screen using native Android libraries
MOZ_NATIVE_DEVICES=
<span class="gd">-# Enable the Search Activity.</span>
<span class="gd">-MOZ_ANDROID_SEARCH_ACTIVITY=1</span>
<span class="gi">+# Don't enable the Search Activity.</span>
<span class="gi">+# MOZ_ANDROID_SEARCH_ACTIVITY=1</span>
</pre></div>
</div>
</div>
<div class="section" id="guide-to-enabling">
<h2>Guide to enabling</h2>
<p>To enable a feature behind a build time flag, we merely need to flip the switch
in <tt class="docutils literal">mobile/android/confvars.sh</tt>. This can be a conditional flip; for
example, to enable only in Nightly:</p>
<pre class="literal-block">
if test "$NIGHTLY_BUILD"; then
MOZ_ANDROID_SEARCH_ACTIVITY=1
else
MOZ_ANDROID_SEARCH_ACTIVITY=
fi
</pre>
<p>Likewise, to enable only when not in Release or Beta:</p>
<pre class="literal-block">
if test ! "$RELEASE_BUILD"; then
MOZ_ANDROID_SEARCH_ACTIVITY=1
else
MOZ_ANDROID_SEARCH_ACTIVITY=
fi
</pre>
<p>See <a class="reference external" href="https://wiki.mozilla.org/Platform/Channel-specific_build_defines">https://wiki.mozilla.org/Platform/Channel-specific_build_defines</a> for details
on the relevant flags.</p>
</div>
<div class="section" id="footnotes">
<h2>Footnotes</h2>
<table class="docutils footnote" frame="void" id="commits" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td><a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1021864">Bug 1021864</a></a></a></a></a> landed as a series of 5 commits:</td></tr>
</tbody>
</table>
<ul class="simple">
<li><a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/3e8565120b1b">https://hg.mozilla.org/integration/fx-team/rev/3e8565120b1b</a></li>
<li><a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/7d404850f5f2">https://hg.mozilla.org/integration/fx-team/rev/7d404850f5f2</a></li>
<li><a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/a7a8ea9e736b">https://hg.mozilla.org/integration/fx-team/rev/a7a8ea9e736b</a></li>
<li><a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/bc6a8b22530a">https://hg.mozilla.org/integration/fx-team/rev/bc6a8b22530a</a></li>
<li><a class="reference external" href="https://hg.mozilla.org/integration/fx-team/rev/9395fb24971c">https://hg.mozilla.org/integration/fx-team/rev/9395fb24971c</a></li>
</ul>
</div>
How the Android Eclipse build system integration works2014-07-08T10:45:00-07:00ncalexandertag:www.ncalexander.net,2014-07-08:blog/2014/07/08/how-the-android-eclipse-integration-works/<p>Firefox for Android (Fennec) can be built with Eclipse, but it’s a delicate
dance. This post runs through the technical details of what happens, and when,
during an Eclipse build.</p>
<p><strong>This is not intended to be a guide to using Eclipse to build Fennec</strong>. For
such a guide, see <a class="reference external" href="https://wiki.mozilla.org/Mobile/Fennec/Android/Eclipse">https://wiki.mozilla.org/Mobile/Fennec/Android/Eclipse</a>.</p>
<div class="section" id="theory">
<h2>Theory</h2>
<ol class="arabic">
<li><p class="first">The <tt class="docutils literal">RecursiveMakeBackend</tt> writes <tt class="docutils literal">Makefile</tt> and <tt class="docutils literal">backend.mk</tt> files
into the object directory for every directory in the tree. If the
corresponding source directory’s <tt class="docutils literal">moz.build</tt> file includes Eclipse project
definitions, then the <tt class="docutils literal">backend.mk</tt> includes special "Eclipse-only"
recursive make targets, like:</p>
<pre class="literal-block">
ANDROID_ECLIPSE_PROJECT_FennecResourcesBranding: .aapt.deps .locales.deps
$(call py_action,process_install_manifest,\
--no-remove --no-remove-all-directory-symlinks --no-remove-empty-directories\
/Users/nalexander/Mozilla/gecko-dev/objdir-droid/android_eclipse/FennecResourcesBranding\
/Users/nalexander/Mozilla/gecko-dev/objdir-droid/android_eclipse/FennecResourcesBranding.manifest)
</pre>
<p>These targets are always written (based on the Eclipse projects defined in
the <tt class="docutils literal">moz.build</tt> files).</p>
</li>
<li><p class="first">The <tt class="docutils literal">AndroidEclipseBackend</tt> writes Eclipse project files and support files
to the top-level <tt class="docutils literal">android_eclipse</tt> directory of the object directory. The
directory layout looks something like:</p>
<pre class="literal-block">
android_eclipse
Fennec.manifest
Fennec
.classpath
.externalToolBuilders
.project
.settings
gen
lint.xml
project.properties
</pre>
<p>At this point, the project files are in place, but things are barren — there
are no source files, Android resource files, or Android manifest.</p>
</li>
<li><p class="first">The object directory is built and packaged using <tt class="docutils literal">mach build && mach
package</tt>. This prepares the C/C++ layer and writes libraries (<tt class="docutils literal">.so</tt>
files) that Fennec requires.</p>
</li>
<li><p class="first">The Eclipse project files written include directions for a special builder
plugin [#plugin] to run as the first step of every build. Each time Eclipse
requests a build (for example, after a file is modified), the builder plugin
takes whatever action is required to prepare the Eclipse project for building.</p>
<p>Currently, the plugin:</p>
<ol class="arabic simple">
<li>checks if anything (interesting) has changed</li>
<li>runs the single recursive make target <em>ANDROID_ECLIPSE_PROJECT_Project</em>
written by the recursive make backend (if necessary)</li>
<li>marks Eclipse resources as needing to be refreshed (if necessary).</li>
</ol>
<p>After this, the regular Eclipse/Android build steps happen: processing the
Android manifest, packaging Android resources, building Java files, etc.</p>
</li>
</ol>
</div>
<div class="section" id="practice">
<h2>Practice</h2>
<p>Let’s dig in to what the plugin really does. All the integration glue is in the
<em>ANDROID_ECLIPSE_PROJECT_Project</em> recursive make target. This target is really
an aggregate target that does two things: install files and aggregate dependencies.</p>
<div class="section" id="install-needed-files">
<h3>Install needed files</h3>
<p>The target calls the Python <tt class="docutils literal">process_install_manifest</tt> build action to
install needed files.</p>
<p>For each Eclipse project, the eclipse backend writes an <em>install manifest</em>
file (named <tt class="docutils literal">Project.manifest</tt>) that contains directions for files to copy
and symlink into the project directory. When I noted earlier that the
project directory was "barren", we’re seeing it before this target has run,
and specifically before this manifest is installed.</p>
<p>To compare, let’s execute:</p>
<pre class="literal-block">
$ mach build $OBJDIR/mobile/android/base/ANDROID_ECLIPSE_PROJECT_Fennec
/usr/bin/make -C /Users/nalexander/Mozilla/gecko-dev/objdir-droid -j8 -s backend.RecursiveMakeBackend
/usr/bin/make -C mobile/android/base -j8 -s ANDROID_ECLIPSE_PROJECT_Fennec
...
From /Users/nalexander/Mozilla/gecko-dev/objdir-droid/android_eclipse/Fennec:\
Kept 1 existing; Added/updated 8; Removed 0 files and 0 directories.
</pre>
<p>That last line is the output from installing the manifest; it has populated
the Fennec directory:</p>
<pre class="literal-block">
android_eclipse
Fennec.manifest
Fennec
.classpath
.externalToolBuilders
.project
.settings
AndroidManifest.xml
assets -> /Users/nalexander/Mozilla/gecko-dev/objdir-droid/dist/fennec/assets
gen
generated -> /Users/nalexander/Mozilla/gecko-dev/objdir-droid/mobile/android/base/generated
java -> /Users/nalexander/Mozilla/gecko-dev/mobile/android/search/java
libs
lint.xml
project.properties
res
src
thirdparty -> /Users/nalexander/Mozilla/gecko-dev/mobile/android/thirdparty
</pre>
</div>
<div class="section" id="add-build-system-dependencies">
<h3>Add build system dependencies</h3>
<p>The target may depend on additional recursive make targets, as specified in the
<tt class="docutils literal">moz.build</tt> file using the <tt class="docutils literal">recursive_make_targets</tt> list. So, for example,
looking at <tt class="docutils literal">mobile/android/base/moz.build</tt>, I can see that the
<em>FennecResourcesGenerated</em> Eclipse project depends on the targets that capture
dependencies on the Android manifest and all resources:</p>
<pre class="literal-block">
# Captures dependencies on Android manifest and all resources.
generated_recursive_make_targets = ['.aapt.deps', '.locales.deps']
generated = add_android_eclipse_library_project('FennecResourcesGenerated')
generated.recursive_make_targets += generated_recursive_make_targets
</pre>
<p>These additional recursive make targets are defined deeper in the build system:
in this case, in <tt class="docutils literal">mobile/android/base/Makefile.in</tt>.</p>
<p>In this way, the project-specific recursive make target both prepares the
project directory, and provides a flexible extension point for additional build
system dependencies. The custom Eclipse plugin merely invokes this make target
(like we did above, with <tt class="docutils literal">mach build</tt>).</p>
</div>
<div class="section" id="an-important-note-about-build-stages">
<h3>An important note about build stages</h3>
<p>It’s important to note that the packaging step <strong>must</strong> happen before the plugin
runs: the per-project recursive make targets depend on the build and package
artifacts. In the <em>vanilla</em> build system, the one built on <tt class="docutils literal">mach</tt> and
recursive make, there are two build stages:</p>
<ul class="simple">
<li>configure and build-backend time</li>
<li>build (and incremental build) time</li>
<li>package time</li>
</ul>
<p>Due to the unusual requirements of the Android build system, we add two new stages:</p>
<ul class="simple">
<li>configure and build-backend time</li>
<li>Android build-backend time</li>
<li>build (and incremental build) time</li>
<li>package time</li>
<li>Eclipse incremental build time</li>
</ul>
<p>At Android build-backend time, we don’t have the artifacts produced by at
package time. That’s why we have the install manifests detailed above, which
process package artifacts lazily; and that’s why the Eclipse projects produced
by the Android build-backend look barren. There has been a good deal of
confusion about this lazy approach (which I think results from not having the
Eclipse plugin installed), manifesting as reports like
<a class="reference external" href="https://mail.mozilla.org/pipermail/mobile-firefox-dev/2014-July/000788.html">https://mail.mozilla.org/pipermail/mobile-firefox-dev/2014-July/000788.html</a>.</p>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>There are a fair number of moving parts:</p>
<ul class="simple">
<li>two Python build backends, that work in concert;</li>
<li>staged build targets, that can’t be run at build-backend time;</li>
<li>coupled make targets and Eclipse project builders, joined by a custom Eclipse plugin.</li>
</ul>
<p>And yet, it all basically works. Don’t move too quickly or make any loud noises.</p>
</div>
DEPRECATED: How to connect Firefox for Android to self-hosted Firefox Account and Firefox Sync servers2014-07-05T20:01:00-07:00ncalexandertag:www.ncalexander.net,2014-07-05:blog/2014/07/05/how-to-connect-firefox-for-android-to-self-hosted-services/<div class="section" id="this-blog-post-is-deprecated">
<h2><span class="caps">THIS</span> <span class="caps">BLOG</span> <span class="caps">POST</span> <span class="caps">IS</span> <span class="caps">DEPRECATED</span></h2>
<p><strong>Firefox for Android 44 and above no longer require an addon to
connect to custom Firefox Account and Firefox Sync servers.</strong></p>
<p>The documentation at
<a class="reference external" href="https://docs.services.mozilla.com/howtos/run-sync-1.5.html">https://docs.services.mozilla.com/howtos/run-sync-1.5.html</a> and
<a class="reference external" href="https://docs.services.mozilla.com/howtos/run-fxa.html">https://docs.services.mozilla.com/howtos/run-fxa.html</a> has been updated
and is now the canonical documentation for configuring custom servers
in Firefox for Android.</p>
<p>The contents of this blog post have been maintained for interested
parties to inspect.</p>
</div>
<div class="section" id="original-content">
<h2>Original content</h2>
<p>Firefox 29 was a <em>huge</em> release. The two largest items were, by most measures,
the new Australis Desktop theme, and the new Firefox Accounts sign-in to Firefox
Sync. In the rush to land the new Firefox Account sign-in, we pared our feature
set aggressively. One of the things that got delayed was support in Firefox for
Android for connecting to self-hosted Firefox Account auth servers, and to
self-hosted Firefox Sync servers. I’m thrilled to announce that support for such
self-hosted servers has <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1022748">just</a> <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1022749">landed</a> in Firefox Nightly, and
should make it to release as part of Firefox 33.</p>
</div>
<div class="section" id="background">
<h2>Background</h2>
<a class="reference external image-reference" href="https://www.ncalexander.net/blog/images/02.Get.Started.png"><img alt="" class="align-right" src="https://www.ncalexander.net/blog/images/02.Get.Started.jpg" /></a>
<p>Historically, connecting Firefox to a self-hosted Firefox Sync server was — if
not easy — at least possible and supported in all products, including Firefox
for Android. The new Firefox Accounts system introduces a redesigned, easy to
use sign-up/sign-in flow that <a class="reference external" href="https://blog.mozilla.org/services/2014/02/07/a-better-firefox-sync/">delivers great user value</a>, but it’s
complicated to host your own servers. Instead of talking to a single Firefox
Sync server, you need to talk to a Firefox Accounts auth server and a Firefox
Sync server, both working in tandem to provide your service. Dan Callahan has
been leading the effort to clarify the <a class="reference external" href="https://blog.mozilla.org/services/2014/05/08/firefox-accounts-sync-1-5-and-self-hosting/">self-hosted server side of the story</a>,
and we have <a class="reference external" href="https://github.com/callahad/selfhosted-sync">work in progress documentation</a> explaining how to use self-hosted
servers (in Firefox Desktop).</p>
</div>
<div class="section" id="guide">
<h2>Guide</h2>
<p>Firefox for Android 33 lets you</p>
<ul class="simple">
<li>specify your Firefox Account and Sync servers before connecting your device to
an account, and</li>
<li>see what servers your device is talking to after you have connected your device.</li>
</ul>
<p>Let’s see how to use the new features.</p>
<!-- .. image:: {filename}/images/03.Sign.Up.jpg -->
<!-- :align: right -->
<!-- :target: {filename}/images/03.Sign.Up.png -->
<div class="section" id="determine-your-custom-server-urls">
<h3>Determine your custom server URLs</h3>
<p>These URLs are a bit magic, and it’s not currently possible to change them on
your Android device without deleting the Android Account and starting again.</p>
<div class="section" id="firefox-account-auth-server">
<h4>Firefox Account auth server</h4>
<p>The Firefox Account auth server <span class="caps">URL</span> should look like:</p>
<table border="1" class="docutils">
<colgroup>
<col width="50%" />
<col width="50%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">Type</th>
<th class="head">Auth server <span class="caps">URL</span></th>
</tr>
</thead>
<tbody valign="top">
<tr><td>Custom</td>
<td><a class="reference external" href="http://domain.com:5001/v1">http://domain.com:5001/v1</a></td>
</tr>
<tr><td>Mozilla production</td>
<td><a class="reference external" href="https://api.accounts.firefox.com/v1">https://api.accounts.firefox.com/v1</a></td>
</tr>
<tr><td>Mozilla testing</td>
<td><a class="reference external" href="https://api-accounts.stage.mozaws.net/v1">https://api-accounts.stage.mozaws.net/v1</a></td>
</tr>
</tbody>
</table>
<p>This should match the <tt class="docutils literal">identity.fxaccounts.auth.uri</tt> preference in your
Desktop Firefox configuration.</p>
<p>You can get a decent idea if your <span class="caps">URL</span> is correct by doing an <span class="caps">HTTP</span> <span class="caps">POST</span> with
<tt class="docutils literal">/account/login</tt> appended to your auth server <span class="caps">URL</span>. You should get an <span class="caps">HTTP</span>
400 response, like:</p>
<pre class="literal-block">
chocho:~ nalexander$ curl -X POST https://api.accounts.firefox.com/v1/account/login
{"code":400,"errno":107,"error":"Bad Request","message":"Invalid parameter in request body",...
</pre>
</div>
<div class="section" id="firefox-sync-token-server">
<h4>Firefox Sync token server</h4>
<p>The Firefox Sync token server <span class="caps">URL</span> should look like:</p>
<table border="1" class="docutils">
<colgroup>
<col width="50%" />
<col width="50%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">Type</th>
<th class="head">Token server <span class="caps">URL</span></th>
</tr>
</thead>
<tbody valign="top">
<tr><td>Custom</td>
<td><a class="reference external" href="http://domain.com:5001/token/1.0/sync/1.5">http://domain.com:5001/token/1.0/sync/1.5</a></td>
</tr>
<tr><td>Mozilla production</td>
<td><a class="reference external" href="https://token.services.mozilla.com/1.0/sync/1.5">https://token.services.mozilla.com/1.0/sync/1.5</a></td>
</tr>
<tr><td>Mozilla testing</td>
<td><a class="reference external" href="https://token.stage.mozaws.net/1.0/sync/1.5">https://token.stage.mozaws.net/1.0/sync/1.5</a></td>
</tr>
</tbody>
</table>
<p>This should match the <tt class="docutils literal">identity.sync.tokenServerURI</tt> preference in your
Desktop Firefox configuration. (Prior to Firefox 42, the token server
preference name for Firefox Desktop was <tt class="docutils literal">services.sync.tokenServerURI</tt>. While
the old preference name will work in Firefox 42 and later, the new preference is
recommended: the old preference name will be reset when the user signs out from
Sync, potentially causing confusion.)</p>
<p>If you are using a custom token server, it’s "public <span class="caps">URL</span>" should be set to
<tt class="docutils literal"><span class="pre">http://domain.com:5001</span></tt>.</p>
<p>You can get a decent idea if your <span class="caps">URL</span> is correct by doing an <span class="caps">HTTP</span> <span class="caps">GET</span> after
dropping the <tt class="docutils literal">1.0/sync/1.5</tt> from your token server <span class="caps">URL</span>. You should get a list
of services; for example:</p>
<pre class="literal-block">
chocho:~ nalexander$ curl https://token.services.mozilla.com
{"services": {"sync": ["1.5"]}, "auth": "https://token.services.mozilla.com"}
</pre>
</div>
</div>
<div class="section" id="install-firefox-for-android-33">
<h3>Install Firefox for Android 33</h3>
<p>First, let’s install Firefox version 33 (or higher). You can download
a recent Firefox Nightly <a class="reference external" href="http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android/">here</a>. Download
the <tt class="docutils literal"><span class="pre">fennec-33.0a1.multi.android-arm.apk</span></tt> file and install it on
your device. Open Firefox.</p>
<!-- You might see a Firefox Sync promotion. If you were to tap on it, or if you -->
<!-- tapped the menu and navigated to *Settings > Sync*, you would start the Firefox -->
<!-- Account sign-up flow. After you tap *Get Started*, you'd see the first screen -->
<!-- of the flow. -->
</div>
<div class="section" id="install-the-fxa-custom-server-addon-firefox-for-android-add-on">
<h3>Install the <em>fxa-custom-server-addon</em> Firefox for Android add-on</h3>
<p>Tap <em>back</em> several times to return to Firefox. We need to install a custom
Firefox for Android add-on called <a class="reference external" href="https://addons.mozilla.org/en-US/android/addon/fxa-custom-server-addon/">fxa-custom-server-addon</a>. This add-on lets
us specify self-hosted servers <a class="footnote-reference" href="#addon" id="id1">[1]</a>.</p>
<p>Tap the link above in Firefox for Android, and click the (hopefully green!) <em>Add
to Firefox</em> button. Since this add-on is hosted at the <a class="reference external" href="https://addons.mozilla.org/en-US/android/">Firefox for Android
Add-ons site</a>, you should not see a warning when you install. (If you do,
<em>Allow</em> the installation, and then <em>Install</em> the downloaded add-on — and please
let me know.)</p>
<p>After you see a toast saying that the add-on is installed, tap the menu. You
should see a new <em>Custom Firefox Account</em> menu item, right at the bottom.</p>
<!-- .. image:: {filename}/images/06.New.Menu.Item.jpg -->
<!-- :align: right -->
<!-- :target: {filename}/images/06.New.Menu.Item.png -->
</div>
<div class="section" id="launch-the-sync-set-up-flow-with-self-hosted-server-urls">
<h3>Launch the Sync set-up flow with self-hosted server URLs</h3>
<a class="reference external image-reference" href="https://www.ncalexander.net/blog/images/07.Dialog.png"><img alt="" class="align-right" src="https://www.ncalexander.net/blog/images/07.Dialog.jpg" /></a>
<p>From the menu, tap the new <em>Custom Firefox Account</em> menu item. Now
you can enter your self-hosted server URLs! The <em>Save</em> button closes the
dialog but keeps your entered URLs for next time; it makes it easy to
copy-and-paste your URLs from elsewhere. When you’ve got your URLs correct, tap
<em>Launch Setup</em>. (It can help to flip your device to landscape mode.)</p>
<p>You should skip right past the <em>Get Started</em> screen and go directly to
the sign-up/sign-in flow. Most importantly, you should see big boxes
loudly announcing that you are using non-standard server URLs
<a class="footnote-reference" href="#boxes" id="id2">[2]</a>. If you tap <em>Already have an account? Sign in</em>, you’ll see
the sign-in flow, still with the non-standard server <span class="caps">URL</span> boxes.</p>
<a class="reference external image-reference" href="https://www.ncalexander.net/blog/images/10.Custom.Sign.In.Details.png"><img alt="" class="align-left" src="https://www.ncalexander.net/blog/images/10.Custom.Sign.In.Details.jpg" /></a>
<p>The server URLs I’ve entered are Mozilla’s staging test servers. Our
quality assurance team uses these servers to verify that new versions
of the Sync client and server are working before they get
released. The accounts and data on Mozilla’s staging servers are
frequently deleted, so <strong>you shouldn’t use these staging test
servers</strong>. (You should either self-host your own servers — that’s why
you’re here — or use Mozilla’s standard production servers.)</p>
<p>I have already created an account on Mozilla’s staging test servers,
so I’m going to sign-in to it. I tap <em>Sign in</em>, and I see that my
account has already been verified. I tap <em>Back to browsing</em>, and I’m
back in Firefox, with my shiny new Firefox Account ready to Sync.</p>
<p><strong>Note:</strong> When this was written, there was a <strong>bug with custom servers with
non-standard ports</strong> (in Firefox for Android 33 and 34 only). <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1046020"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1046020">Bug 1046020</a></a>
fixes an issue where custom servers with non-standard ports (i.e., servers with
a non-80 port for <span class="caps">HTTP</span> or non-443 port for <span class="caps">HTTPS</span>) failed to sync. The failure
looked like an authentication error when requesting a token from the token
server. The fix is in Nightly 34, has been uplifted to Firefox 33, and should
be present in Aurora 33 by the end of August. Many thanks to user Ben Curtis
for reporting and fixing this issue!</p>
</div>
<div class="section" id="verify-that-firefox-sync-is-using-self-hosted-server-urls">
<h3>Verify that Firefox Sync is using self-hosted server URLs</h3>
<a class="reference external image-reference" href="https://www.ncalexander.net/blog/images/11.Custom.Status.png"><img alt="" class="align-right" src="https://www.ncalexander.net/blog/images/11.Custom.Status.jpg" /></a>
<p>Let’s verify that our device is connected to a Firefox Account and is
healthy by inspecting our server URLs in the Firefox Account
settings. Tap the menu, and then <em>Settings > Sync</em>. Observe that we
have two new sections, labeled <em>Account server</em> and <em>Sync server</em>. You
should see the self-hosted server URLs you entered in the
sign-up/sign-in flow <a class="footnote-reference" href="#changing" id="id3">[3]</a>.</p>
<p>You can also watch the <tt class="docutils literal">adb logcat</tt> while syncing to see what URLs are being
used <a class="footnote-reference" href="#logcat" id="id4">[4]</a>.</p>
<p>The <em>fxa-custom-server-addon</em> has done its job, and you can safely remove it
entirely: tap the menu and then select <em>Tools > Add-ons</em> to uninstall it.</p>
</div>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>Firefox 33 allows you to</p>
<ul class="simple">
<li>specify self-hosted Firefox Account and Sync servers when connecting a device, and</li>
<li>see what servers your device is connected to.</li>
</ul>
<p>We hope Firefox for Android works well with your self-hosted servers!</p>
</div>
<div class="section" id="footnotes">
<h2>Footnotes</h2>
<table class="docutils footnote" frame="void" id="addon" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>You can see the <em>fxa-custom-server-addon</em>‘s <a class="reference external" href="https://github.com/ncalexan/fxa-custom-server-addon">source code</a> and you can write
your own add-on using the brand new <a class="reference external" href="https://developer.mozilla.org/en-US/Add-ons/Firefox_for_Android/API/Accounts.jsm">Accounts.jsm add-on <span class="caps">API</span></a> that we built
to support these features.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="boxes" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>The boxes are red because we absolutely don’t want a new user to
accidentally use a non-standard, non-Mozilla server when they didn’t
choose to do so explicitly. If changing the server is <em>too</em> easy,
or the change is not visible, there is a possible attack on the
user’s private Sync data.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="changing" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td><p class="first">Changing the (self-hosted) servers <em>after</em> you’ve connected your
device is a non-goal. If you want to change servers, delete your
account using the Android Settings App, and create a new one with
the updated server URLs.</p>
<p>Why is this? There are two main reasons.</p>
<ul>
<li><p class="first">Sync on Android maintains a number of caches and partial sync
states to provide an efficient sync experience. It’s very
difficult to sync smoothly through a change of servers.</p>
</li>
<li><p class="first">Sync on Android is not like Desktop. It’s not written in C++ and
JavaScript, like the rest of the Firefox Desktop; <a class="reference external" href="https://mail.mozilla.org/pipermail/sync-dev/2014-July/000914.html">as my colleague Richard
says</a>,</p>
<blockquote>
<p>It’s best to think of Sync on Android as being a separate pure-Java
application that’s bundled with Firefox. Sync doesn’t use the same
network stack, or any of the Gecko features that you might be used to from desktop.</p>
</blockquote>
</li>
</ul>
<p class="last">This means that we can’t just edit a few preferences using <tt class="docutils literal">about:config</tt> or
some JavaScript. We would have to provide a Java user-interface (or a
Gecko-to-Java bridge that respects Sync’s runtime lifecycle) to allow such
changes, and that is an on-going maintenance burden.</p>
</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="logcat" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td><p class="first">The sure-fire way to know what Sync on Android is really doing is to observe the
Android device log using <tt class="docutils literal">adb logcat</tt>.</p>
<p>You’ll want to bump your log-level:</p>
<pre class="literal-block">
adb shell setprop log.tag.FxAccounts VERBOSE
</pre>
<p>Then, you can observe the log using:</p>
<pre class="literal-block">
adb logcat | grep FxAccounts
</pre>
<p>It’s best to observe the log while you force a sync from the Android Settings
App. You should see output like:</p>
<pre class="literal-block">
D FxAccounts(...) fennec :: BaseResource :: HTTP GET https://token.stage.mozaws.net/1.0/sync/1.5
...
D FxAccounts(...) fennec :: BaseResource :: HTTP GET https://sync-4-us-east-1.stage.mozaws.net/...
</pre>
<p class="last">See <a class="reference external" href="http://160.twinql.com/how-to-file-a-good-android-sync-bug">How to file a good Android Sync bug</a> for details.</p>
</td></tr>
</tbody>
</table>
</div>
<div class="section" id="changes">
<h2>Changes</h2>
<ul class="simple">
<li>Mon 21 July 2014: Updated link to <a class="reference external" href="https://addons.mozilla.org/en-US/android/addon/fxa-custom-server-addon/">fxa-custom-server-addon</a> to point to
<a class="reference external" href="https://addons.mozilla.org/en-US/android/">Firefox for Android Add-ons site</a>. Thanks to <span class="caps">IRC</span> user <em>zombie</em> for testing!</li>
<li>Mon 18 August 2014: Added note about bug with custom server ports.</li>
<li>Wed 5 August 2015: Per <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1003708">Bug 1003708</a>, replaced <tt class="docutils literal">services.sync.tokenServerURI</tt>
with <tt class="docutils literal">identity.sync.tokenServerURI</tt>.</li>
<li>Mon 1 February 2016: deprecated entire post and linked to new documentation.</li>
</ul>
</div>
Adding assets to the Fennec APK file2014-07-02T17:42:00-07:00ncalexandertag:www.ncalexander.net,2014-07-02:blog/2014/07/02/adding-assets-to-the-fennec-apk-file/<p>The Fennec Android package file includes static assets in the <span class="caps">APK</span> root, and in
the <em>assets</em> directory. User <em>jkraml</em> recently asked how to add <em>new</em> assets,
and we should write down how it works.</p>
<p>First, a word on how packaging (and re-packaging!) works. There are three phases:</p>
<ul class="simple">
<li><em>build</em>: the build installs files into <tt class="docutils literal">$(FINAL_TARGET)</tt>, which is usually
<tt class="docutils literal">$<span class="caps">OBJDIR</span>/dist/bin</tt>;</li>
<li><em>stage</em>: the Python packager runs and moves files from <tt class="docutils literal">$(FINAL_TARGET)</tt>
into the staging area, which is usually <tt class="docutils literal">$<span class="caps">OBJDIR</span>/dist/fennec</tt>;</li>
<li><em>package</em>: the packaging scripts zip (part of) the staging area into the <span class="caps">APK</span>.</li>
</ul>
<div class="section" id="adding-a-file-to-the-apk-root">
<h2>Adding a file to the <span class="caps">APK</span> root</h2>
<p>As an example, I’m going to take the build actions in
<tt class="docutils literal">mobile/android/base</tt>. The links below lead to a few example commits hosted on github.</p>
<p>First, <a class="reference external" href="https://github.com/ncalexan/gecko-dev/commit/0fdd414e6693c5421e7611366b3c33b36ea12578#diff-1">create</a>
<tt class="docutils literal">mobile/android/base/example.cert</tt>.</p>
<p>Then:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/ncalexan/gecko-dev/commit/0fdd414e6693c5421e7611366b3c33b36ea12578#diff-0">build</a>: as part of the <em>libs</em> target, install the new file;</li>
<li><a class="reference external" href="https://github.com/ncalexan/gecko-dev/commit/0fdd414e6693c5421e7611366b3c33b36ea12578#diff-2">stage</a>: add the new file to <tt class="docutils literal"><span class="pre">mobile/android/installer/package-manifest.in</span></tt>;</li>
<li><a class="reference external" href="https://github.com/ncalexan/gecko-dev/commit/0fdd414e6693c5421e7611366b3c33b36ea12578#diff-3">package</a>: add the new file to <tt class="docutils literal">$(DIST_FILES)</tt> so that it gets packed into the <span class="caps">APK</span> root.</li>
</ul>
<p>Then run:</p>
<pre class="literal-block">
mach build-backend &&
mach build mobile/android/base &&
mach package
</pre>
<p>and you should see your new file in the root of <tt class="docutils literal"><span class="pre">$<span class="caps">OBJDIR</span>/dist/fennec-*.apk</span></tt>.</p>
</div>
<div class="section" id="adding-an-entire-directory-to-the-apk-root">
<h2>Adding an entire directory to the <span class="caps">APK</span> root</h2>
<p><a class="reference external" href="https://mozillians.org/pt-BR<a href="https://mozillians.org/u/anatal">/u/anatal</a>/">Andre Natal</a> asked how to add a directory to the <span class="caps">APK</span> root. It’s a little
trickier than an individual file, mostly because installing a directory into the
package staging area requires care. However, you can see that the <a class="reference external" href="https://github.com/ncalexan/gecko-dev/commit/de2ac193e297f21ed35498147030c24e86017989">form is very
similar</a>.</p>
</div>
<div class="section" id="adding-a-file-to-the-assets-directory">
<h2>Adding a file to the <em>assets</em> directory</h2>
<p>This is a tiny bit trickier:</p>
<ul class="simple">
<li><a class="reference external" href="https://github.com/ncalexan/gecko-dev/commit/0fdd414e6693c5421e7611366b3c33b36ea12578#diff-0">build</a>: exactly the same as before;</li>
<li><a class="reference external" href="https://github.com/ncalexan/gecko-dev/commit/243687e6e3588507b434f207679604a72c42b4d4#diff-0">stage</a>: similar, but observe that the new file is added in a section with <tt class="docutils literal"><span class="pre">destdir="assets"</span></tt>;</li>
<li><a class="reference external" href="https://github.com/ncalexan/gecko-dev/commit/243687e6e3588507b434f207679604a72c42b4d4#diff-1">package</a>:
a new list of files packed into <tt class="docutils literal">assets/</tt>.</li>
</ul>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>It’s not hard to add files to the Fennec <span class="caps">APK</span> root or the assets directory. As
always, you can reach me (<tt class="docutils literal">nalexander</tt>) directly on <tt class="docutils literal">irc.mozilla.org</tt>,
channel <tt class="docutils literal">#mobile</tt>; on <a class="reference external" href="https://twitter.com/ncalexander">the Twitters</a>; and
via the <a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">public Fennec mailing list</a>.</p>
</div>
<div class="section" id="updates">
<h2>Updates</h2>
<ul class="simple">
<li>Monday 18 August 2014: Added section and commit showing how to add a directory
to the <span class="caps">APK</span> root.</li>
</ul>
</div>
Better Fennec builds with an Eclipse plugin2014-06-24T21:32:00-07:00ncalexandertag:www.ncalexander.net,2014-06-24:blog/2014/06/24/better-fennec-builds-with-an-eclipse-plugin/<p>I’ve been working on making building Fennec better with Eclipse.</p>
<p>tl;dr: soon, the output of <tt class="docutils literal">mach <span class="pre">build-backend</span> <span class="pre">-b=AndroidEclipse</span></tt> will require
a new plugin to work. The steps to get started are <a class="reference external" href="https://www.ncalexander.net/blog/2014/02/14/getting-started-with-the-android-eclipse-build-backend-for-fennec/">the same as before</a>,
<strong>except that you’ll need to install a new Eclipse plugin</strong>. The plugin can be
installed from the Eclipse update site at
<a class="reference external" href="http://people.mozilla.org/~nalexander/eclipse/update-site/">http://people.mozilla.org/~nalexander/eclipse/update-site/</a>.</p>
<p>I’ve taken some work of <a class="reference external" href="https://github.com/thebnich/fennec-adt">my colleague Brian Nicholson’s</a> and modified it to provide a generic
"run this command in Eclipse" that lets us have faster, more predictable builds,
with better error reporting. This is about to land as <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1029232"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1029232">Bug 1029232</a></a>.</p>
<p>What the patches on that bug do is replace the Eclipse "ExternalToolBuilder"
invocations that we were using with a brand-new Eclipse plugin (plugin, feature,
and update site hosted at <a class="reference external" href="https://github.com/ncalexan/fennec-eclipse">https://github.com/ncalexan/fennec-eclipse</a>). The
plugin handles ignoring superfluous build requests; these were what led to the
long (or even infinite!) build cycles, where Eclipse would continually build and
rebuild projects. The plugin also marks errors in the <em>Problems</em> view, and shows
output in the <em>Android</em> console output log.</p>
<p>I’ve uploaded a few mid-length screencasts showing how to install the plugin and
demonstrating how the plugin is faster for editing preprocessed resources. The
last two are of general interest: one shows a few advanced features of the
Eclipse debugger and Android plugin; the other shows some of the Android layout
features of the Android plugin.</p>
<p>As always, thanks to Brian Nicholson and my testers, especially Mike Comella and
Richard Newman.</p>
<div class="vimeo" align="left"><iframe width="800" height="450" src="https://player.vimeo.com/video/99067397" frameborder="0"></iframe></div><div class="vimeo" align="left"><iframe width="800" height="450" src="https://player.vimeo.com/video/99067164" frameborder="0"></iframe></div><div class="vimeo" align="left"><iframe width="800" height="450" src="https://player.vimeo.com/video/99068253" frameborder="0"></iframe></div><div class="vimeo" align="left"><iframe width="800" height="450" src="https://player.vimeo.com/video/99068834" frameborder="0"></iframe></div><div class="vimeo" align="left"><iframe width="800" height="450" src="https://player.vimeo.com/video/99093138" frameborder="0"></iframe></div>Developing Fennec with Eclipse mini-tutorials2014-03-24T01:56:00-07:00ncalexandertag:www.ncalexander.net,2014-03-24:blog/2014/03/24/eclipse-mini-tutorials/<p><a class="reference external" href="https://www.ncalexander.net/blog/2013/11/05/building-fennec-with-eclipse-ide-demo/">Using Eclipse to develop Fennec</a> is pretty
great, but some folks need <em>even more</em> convincing. Here are ten
mini-tutorials — videos less than two minutes long each — showing
some handy navigation techniques and editing tips.</p>
<p>I can think of loads more Eclipse and Fennec topics to cover. Let me know if
you’d like me to say a few words on a particular topic, or (<em>even better!</em>) if
you’d like to contribute a short video on a particular topic. As always, I’m
<em>nalexander</em> on <span class="caps">IRC</span> and <a href="https://twitter.com/ncalexander">@ncalexander</a> on <a class="reference external" href="https://twitter.com/ncalexander">the Twitter</a>. Thanks to everybody using Eclipse to make
Fennec the best browser for your Android device!</p>
<div class="section" id="getting-started">
<h2>Getting started</h2>
<div class="section" id="configuring-the-java-browsing-perspective">
<h3>01. Configuring the Java Browsing perspective</h3>
<p>Learn how to switch perspectives and configure Eclipse to channel the Smalltalk
<span class="caps">IDE</span>.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/87540429" frameborder="0"></iframe></div></div>
<div class="section" id="importing-the-fennec-eclipse-projects">
<h3>02. Importing the Fennec Eclipse projects</h3>
<p>Learn how to import and build the Fennec Eclipse projects, and how to use <em>Clean
Build</em> to fix build inconsistencies.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/87542142" frameborder="0"></iframe></div></div>
</div>
<div class="section" id="navigating-the-fennec-code-base">
<h2>Navigating the Fennec code base</h2>
<div class="section" id="open-type-and-resource">
<h3>03. Open Type and Resource</h3>
<p>Learn how to navigate directly to Java code and resources, without using the
mouse, by using <em>Open…</em> and wildcards.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/87546096" frameborder="0"></iframe></div></div>
<div class="section" id="finding-and-following-references">
<h3>04. Finding and following references</h3>
<p>Divine where code is called from, where members are initialized, and where a
class is constructed.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/87546359" frameborder="0"></iframe></div></div>
<div class="section" id="code-outline">
<h3>05. Code outline</h3>
<p>Quickly see and navigate to points in the current file.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/89866481" frameborder="0"></iframe></div></div>
</div>
<div class="section" id="editing-tips">
<h2>Editing tips</h2>
<div class="section" id="leveraging-auto-complete-and-quick-fix">
<h3>06. Leveraging auto-complete and quick fix</h3>
<p>Lazy coders have more fun.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/89867156" frameborder="0"></iframe></div></div>
<div class="section" id="fixing-names-and-imports">
<h3>07. Fixing names and imports</h3>
<p>Smart coders do automatic tasks automatically.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/89867702" frameborder="0"></iframe></div></div>
</div>
<div class="section" id="running-and-debugging-fennec">
<h2>Running and debugging Fennec</h2>
<div class="section" id="running-fennec">
<h3>08. Running Fennec</h3>
<p>Learn how to configure an emulator or device to run Fennec from within Eclipse.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/89870513" frameborder="0"></iframe></div></div>
<div class="section" id="debugging-fennec">
<h3>09. Debugging Fennec</h3>
<p>See how to set breakpoints, debug Fennec, and inspect the running environment.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/89870752" frameborder="0"></iframe></div></div>
<div class="section" id="running-browser-instrumentation-tests">
<h3>10. Running browser instrumentation tests</h3>
<p>Run Fennec’s instrumentation test suites and get beautiful JUnit 3 output in Eclipse.</p>
<div class="vimeo" align="left"><iframe width="420" height="315" src="https://player.vimeo.com/video/89872049" frameborder="0"></iframe></div></div>
</div>
Getting started with the Android Eclipse build backend for Fennec2014-02-14T07:07:00-08:00ncalexandertag:www.ncalexander.net,2014-02-14:blog/2014/02/14/getting-started-with-the-android-eclipse-build-backend-for-fennec/<p>I’m thrilled to announce that tonight the Android Eclipse build
backend for Fennec landed on mozilla-inbound. This work has been a
long time coming; you can see the progression of ideas at <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=853045"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=853045">Bug 853045</a></a>.</p>
<div class="section" id="prerequisities">
<h2>Prerequisities</h2>
<p>You’ll need:</p>
<ul class="simple">
<li>An appropriate version of Eclipse. I’ve tested both <em>Kepler Service Release 1
(Build id: 20130919-0819)</em> and <em>Luna Release (Build id: 20131219-0014)</em> on Mac
<span class="caps">OS</span> X.</li>
<li>To bump Eclipse’s memory limits, as described in <a class="reference external" href="http://stackoverflow.com/a/11093228">this stack overflow
answer</a>. If you don’t, you’ll
hit <span class="caps">GC</span> issues because Android’s dex takes a good deal of memory.</li>
<li>To install the <a class="reference external" href="http://developer.android.com/tools/sdk/eclipse-adt.html">Eclipse <span class="caps">ADT</span>
plugin</a>.</li>
<li>Your <tt class="docutils literal">mozconfig</tt> to include
<tt class="docutils literal"><span class="pre">--enable-application=mobile/android</span></tt>.</li>
</ul>
</div>
<div class="section" id="quick-start">
<h2>Quick start</h2>
<p>Build and package your tree:</p>
<pre class="literal-block">
mach build && mach package
</pre>
<p>Generate the Android Eclipse project files with:</p>
<pre class="literal-block">
mach build-backend --backend=AndroidEclipse
</pre>
<p>On success, you should see output containing a line like:</p>
<pre class="literal-block">
Wrote 5 Android Eclipse projects to /Users/nalexander/Mozilla/gecko/objdir-droid/android_eclipse; 5 created; 0 updated
</pre>
<p>Open Eclipse. If you haven’t already, you’ll be prompted to enter a
workspace location. After you do so, you’ll see something like the
following (minus the <em>About</em> dialog):</p>
<p><img alt="Kepler welcome screen" src="http://people.mozilla.org/~nalexander/01_kepler_welcome.jpg" /></p>
<p>In a fresh Eclipse workspace, go to <em>File > Import…</em> and select
<em>General > Existing Projects into Workspace</em>:</p>
<p><img alt="Kepler file import screen" src="http://people.mozilla.org/~nalexander/02_kepler_file_import.jpg" /></p>
<p>Enter the path printed by
<tt class="docutils literal">mach <span class="pre">build-backend</span> <span class="pre">--backend=AndroidEclipse</span></tt>. You can safely ignore
the <em>BackgroundInstrumentationTests</em> project. Click <span class="caps">OK</span>.</p>
<p><img alt="Kepler projects checked screen" src="http://people.mozilla.org/~nalexander/03_kepler_projects_checked.jpg" /></p>
<p>You should see a good deal of action in the console, some errors, and
lots of repeated build steps. Eventually, this should settle down:</p>
<p><img alt="Kepler successful build screen" src="http://people.mozilla.org/~nalexander/04_kepler_successful_build.jpg" /></p>
<p>Now you should be able to begin editing the Java source code of Fennec and
Robocop. Saving should trigger an incremental build, which will print output to
the console (which, unfortunately, gets continually erased and can’t be easily
saved) before completing. These incremental builds should be pretty fast; on my
system, a trivial change takes about a second to compile.</p>
<p>These compilation steps do not actually build and dex an Android <span class="caps">APK</span>. To do so,
right click the <em>Fennec</em> project and select <em>Run as… > Android
Application</em>. You should see console logs saying that Fennec is being packaged
and deployed to the device. Some time later you should have a freshly built
Fennec re-started on your device. For me, this takes about 45 seconds — Eclipse
is dexing several megs of Java .class files, signing about 20 megs of libraries,
and then sending the resulting <span class="caps">APK</span> to my device’s storage.</p>
</div>
<div class="section" id="get-out-of-jail-card">
<h2>Get out of jail card</h2>
<p>It is possible (easy) to confuse Eclipse’s builder. In this case, select all the
projects you have open, right click, and select <em>Refresh</em>. Then, go to <em>Project
> Clean…</em> and clean all projects. You should see a lot of churn, but
eventually you should get a clean build. In rare cases (including changes to
<tt class="docutils literal">moz.build</tt> files) the Eclipse project files will be stale and you’ll need to re-run:</p>
<pre class="literal-block">
mach build-backend --backend=AndroidEclipse
</pre>
<p>Remember to refresh all projects and clean build after re-running the Android
Eclipse backend.</p>
</div>
<div class="section" id="future">
<h2>Future</h2>
<p>Some things for future work:</p>
<ul class="simple">
<li>making the Eclipse build integration smoother and more predictable;</li>
<li>making it easier to use Eclipse and rebuild <em>omni.ja</em>;</li>
<li>support for building and running tests from Eclipse;</li>
<li>squashing warnings, so we can see when we add new warnings;</li>
<li>documentation;</li>
<li>a getting started/best practices guide for Eclipse;</li>
<li>a build backend for IntelliJ project files.</li>
</ul>
</div>
<div class="section" id="thanks">
<h2>Thanks</h2>
<p>This work couldn’t have happened without the initial work and testing of
<a class="reference external" href="https://twitter.com/theBNich">bnicholson</a> and the tireless reviews and
encouragement of <a class="reference external" href="https://twitter.com/indygreg">gps</a>.</p>
</div>
<div class="section" id="feedback">
<h2>Feedback</h2>
<p>Feedback is always appreciated: discussion is best conducted on the
<a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev</a>
mailing list and I’m <a class="reference external" href="https://twitter.com/ncalexander">nalexander</a> on
<span class="caps">IRC</span>. File tickets concerning the build system in <em>Core :: Build Config</em> and <span class="caps">CC</span> me.</p>
</div>
Building Fennec with Eclipse — instructions2013-11-07T22:04:00-08:00ncalexandertag:www.ncalexander.net,2013-11-07:blog/2013/11/07/building-fennec-with-eclipse-instructions/<p>Here are terse instructions for getting my work in progress for building
Fennec with Eclipse running on your local developer machine. This is a
companion to <a class="reference external" href="http://www.ncalexander.net/blog/2013/11/05/building-fennec-with-eclipse-ide-demo/">my screencast about building Fennec with
Eclipse</a>.</p>
<ol class="arabic">
<li><p class="first">Start with a fresh checkout of some repository that merges into
<a class="reference external" href="http://hg.mozilla.org/mozilla-central/">mozilla-central</a>
(<em>mozilla-central</em>, <em>mozilla-inbound</em>, <em>fx-team</em>, …)</p>
<pre class="literal-block">
$ hg clone http://hg.mozilla.org/mozilla-central/
...
</pre>
<p>This repo doesn’t really need to be fresh, but you shouldn’t have any
uncommitted changes in your working directory.</p>
</li>
<li><p class="first">Now, we’ll pull in my development changes. I’ve pushed these out to
Greg Szorc’s <em>gecko-collab</em> repository, which is a modern Mercurial
configuration for doing Gecko development. I’m personally using some
of the advanced features, namely <a class="reference external" href="http://mercurial.selenic.com/wiki/ChangesetEvolution">changeset
evolution</a>,
but you don’t need to use any of that to get and use my changes. (The
warning below about <em>obsolete feature not enabled</em> is part of the
feature mismatch, but you can safely ignore it.)</p>
<pre class="literal-block">
$ hg pull -u -r nalexander/bug-853045-mach-projectify http://hg.gregoryszorc.com/gecko-collab
pulling from http://hg.gregoryszorc.com/gecko-collab
obsolete feature not enabled but 136 markers found!
searching for changes
adding changesets
adding manifests
adding file changes
added 12 changesets with 88 changes to 108 files (+1 heads)
adding remote bookmark aurora/COMM193a4_20100510_RELBRANCH
...
205 files updated, 0 files merged, 63 files removed, 0 files unresolved
</pre>
<p>If we check the log, we’ll see something like the following. (This
will change as patches get updated and merged into mainline.)</p>
<pre class="literal-block">
$ hg log -l 13 -f
obsolete feature not enabled but 136 markers found!
changeset: 154029:7d7660bea4a9
bookmark: nalexander/bug-853045-mach-projectify
tag: tip
user: Nick Alexander
date: Fri Nov 01 08:37:02 2013 -0700
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=853045">Bug 853045</a> - Implement |mach projectify-{create,update}|. r=nalexander,bnicholson,gps
changeset: 154028:6b66afd5896f
user: Nick Alexander
date: Wed Nov 06 21:47:59 2013 -0500
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646">Bug 934646</a></a></a></a> - Declare Android resource directories relative to $SRCDIR. r=glandium
changeset: 154027:77edf07ea0ad
user: Nick Alexander
date: Thu Nov 07 15:01:54 2013 -0500
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646">Bug 934646</a></a></a></a> - Replace hard-coded Android resource ID. r=mfinkle
changeset: 154026:8ef4bd5cc691
user: Nick Alexander
date: Wed Nov 06 17:52:52 2013 -0500
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646">Bug 934646</a></a></a></a> - Delete unused resources. r=mfinkle
changeset: 154025:5eeb79bb1eef
user: Nick Alexander
date: Wed Nov 06 17:52:52 2013 -0500
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934646">Bug 934646</a></a></a></a> - Always include Crash Reporter resources in Android APK. r=mfinkle
changeset: 154024:e5e64d42fc42
user: Nick Alexander
date: Wed Nov 06 17:10:26 2013 -0500
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=567884"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=567884">Bug 567884</a></a> - Part 2: Use INSTALL_TARGETS to install icons. r=glandium
changeset: 154023:b2455b8d9b93
user: Nick Alexander
date: Wed Nov 06 16:52:49 2013 -0500
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=567884"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=567884">Bug 567884</a></a> - Part 1: Rename branding resources. r=glandium
changeset: 154022:1543ac7c37cc
user: Nick Alexander
date: Wed Nov 06 16:22:11 2013 -0500
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300">Bug 933300</a></a></a> - Review comments. r=glandium
changeset: 154021:733edde0bf13
user: Nick Alexander
date: Thu Oct 31 09:32:02 2013 -0700
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300">Bug 933300</a></a></a> - Part 2: Generate generated/org/mozilla/gecko/R.java. r=glandium
changeset: 154020:e453968d23a9
user: Nick Alexander
date: Wed Oct 30 23:08:09 2013 -0700
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=933300">Bug 933300</a></a></a> - Part 1: Generate preprocessed code into generated/org/mozilla/*. r=glandium
changeset: 154019:6fae0e6e230f
user: Nick Alexander
date: Wed Nov 06 11:42:03 2013 -0500
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934864">Bug 934864</a></a> - Add REPORT_BUILD do INSTALL_TARGETS and PP_TARGETS
changeset: 154018:7caf5356dd7e
parent: 153961:7433abfef863
user: Mike Hommey
date: Tue Nov 05 23:10:13 2013 +0900
summary: <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934864"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=934864">Bug 934864</a></a> - Add option to make INSTALL_TARGETS and PP_TARGETS keep the original path when copying/preprocessing
changeset: 153961:7433abfef863
parent: 153927:f73dd492c34c
parent: 153960:844f6bf9bc17
user: Carsten "Tomcat" Book
date: Thu Nov 07 15:50:39 2013 +0100
summary: merge b2g-inbound to mozilla-central
</pre>
</li>
<li><p class="first">Now we need to configure our build directory. If you’ve already
created a <tt class="docutils literal">mozconfig</tt> file for Fennec, great; if not, here’s mine.
At a minimum, you’ll need to update the paths to the Android <span class="caps">SDK</span> and
<span class="caps">NDK</span>.</p>
<pre class="literal-block">
$ echo mozconfig
# Build Fennec
ac_add_options --enable-application=mobile/android
# Android
ac_add_options --target=arm-linux-androideabi
ac_add_options --with-android-ndk="$HOME/Mozilla/Android/android-ndk-r8e"
ac_add_options --with-android-sdk="$HOME/Mozilla/Android/android-sdks/platforms/android-16"
# Treat warnings as errors in directories with FAIL_ON_WARNINGS.
ac_add_options --enable-warnings-as-errors
ac_add_options --disable-optimize
ac_add_options --disable-crashreporter
ac_add_options --disable-debug-symbols
mk_add_options MOZ_OBJDIR=./objdir-droid
mk_add_options AUTOCLOBBER=1
mk_add_options MOZ_MAKE_FLAGS=-j7
ac_add_options --disable-elf-hack
</pre>
</li>
<li><p class="first">If you look at the log above, you can see that my changes are based
on top of the <em>mozilla-central</em> hg commit <tt class="docutils literal">7433abfef863</tt>.
Navigating to <a class="reference external" href="https://tbpl.mozilla.org/?rev=7433abfef863">https://tbpl.mozilla.org/?rev=7433abfef863</a> and clicking
on the green <em>B</em> beside <em>Android 2.2 Opt</em> pops up a frame at the
bottom of the page. Click <em>go to build directory</em> (in the bottom left
corner of the page) and download the file
<tt class="docutils literal"><span class="pre">fennec-28.0a1.en-<span class="caps">US</span>.android-arm.apk</span></tt>. Finally, unzip that file to,
say, <tt class="docutils literal"><span class="pre">../tmp</span></tt>. You should get a directory that looks like:</p>
<pre class="literal-block">
$ ls ../tmp
AndroidManifest.xml
META-INF
application.ini
assets
classes.dex
lib
package-name.txt
platform.ini
recommended-addons.json
removed-files
res
resources.arsc
</pre>
</li>
<li><p class="first">Now we’re ready to create the Eclipse projects:</p>
<pre class="literal-block">
$ ./mach projectify-create -i eclipse -w ~/Documents/test --bindir ../tmp mobile/android/base
0:02.66 Using /Users/ncalexan/Mozilla/moz for source dir and /Users/ncalexan/Mozilla/moz/objdir-droid for object dir.
0:02.66 Using mobile/android/base for relative source dir.
0:02.66 Using /Users/ncalexan/Mozilla/moz for source dir and /Users/ncalexan/Mozilla/moz/objdir-droid for object dir.
0:02.66 Using mobile/android/base for relative source dir.
0:03.21 /usr/bin/make -C /Users/ncalexan/Mozilla/moz/objdir-droid/mobile/android/base -j8 -s -w generated/org/mozilla/fennec_ncalexan/App.java generated/org/mozilla/fennec_ncalexan/WebApp.java generated/org/mozilla/fennec_ncalexan/WebApps.java generated/org/mozilla/gecko/AppConstants.java generated/org/mozilla/gecko/R.java generated/org/mozilla/gecko/SysInfo.java generated/org/mozilla/gecko/background/announcements/AnnouncementsConstants.java generated/org/mozilla/gecko/background/common/GlobalConstants.java generated/org/mozilla/gecko/background/healthreport/HealthReportConstants.java generated/org/mozilla/gecko/mozglue/GeckoLoader.java generated/org/mozilla/gecko/sync/SyncConstants.java generated/org/mozilla/gecko/widget/GeckoEditText.java generated/org/mozilla/gecko/widget/GeckoImageButton.java generated/org/mozilla/gecko/widget/GeckoImageView.java generated/org/mozilla/gecko/widget/GeckoLinearLayout.java generated/org/mozilla/gecko/widget/GeckoRelativeLayout.java generated/org/mozilla/gecko/widget/GeckoTextSwitcher.java generated/org/mozilla/gecko/widget/GeckoTextView.java
0:03.86 find /Users/ncalexan/Mozilla/moz/mobile/android/projectify/eclipse -name *.launch
0:03.93 Preprocessing 11 files.
0:04.41 find /Users/ncalexan/Mozilla/moz/mobile/android/base -name *.java
0:04.70 Preprocessing 5 files.
0:04.72 Creating 9 symlinks.
</pre>
</li>
<li><p class="first">Finally (!), you should be able to open up the newly created
projects in Eclipse, following <a class="reference external" href="https://www.ncalexander.net/blog/2013/11/05/building-fennec-with-eclipse-ide-demo/">my screencast about building Fennec
with Eclipse</a>. Good luck!</p>
</li>
</ol>
Building Fennec with Eclipse — IDE demo2013-11-05T04:37:00-08:00ncalexandertag:www.ncalexander.net,2013-11-05:blog/2013/11/05/building-fennec-with-eclipse-ide-demo/<p>I’ve recorded a screencast demo of some of the work <a class="reference external" href="https://twitter.com/ncalexander">me</a> and <a class="reference external" href="https://twitter.com/theBNich">bnicholson</a> have been doing on building Fennec with
Eclipse. (IntelliJ will follow; there are few small fixes I haven’t ported to
IntelliJ yet <a class="reference external" href="#f1">[1]</a>.)</p>
<p>The patches I have applied are available on Greg Szorc’s <em>gecko-collab</em> server
<a class="reference external" href="#2">[2]</a>. I’ve written up <a class="reference external" href="http://www.ncalexander.net/blog/2013/11/07/building-fennec-with-eclipse-instructions/">brief instructions for getting started</a>.
The meta bug tracking this project is <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=924961"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=924961">Bug 924961</a></a>.</p>
<p>You can <a class="reference external" href="https://vimeo.com/78590282">watch</a> or <a class="reference external" href="http://people.mozilla.org/~nalexander/Building.Fennec.with.Eclipse.IDE.Demo.webm">download</a>
(11 megs) the video.</p>
<ol class="arabic simple">
<li>Things I can remember that need to be updated to support IntelliJ (patches welcome!):<ul>
<li>need to update IntelliJ’s <tt class="docutils literal">build.xml</tt> file;</li>
<li>need to support excluding files from IntelliJ source directories;</li>
<li>need to make sure IntelliJ lint settings ignore errors about calling too
recent Android APIs.</li>
</ul>
</li>
<li>Updated Thursday, November 7, 2013: These were available from <a class="reference external" href="http://hg.mozilla.org/users/nalexander_mozilla.com/gecko-patches/file/c48622878590">my patch queue
repository</a>
but these are now stale and future updates will be pushed only to Greg’s
<em>gecko-collab</em> repository.</li>
</ol>
My vision for building mobile/android2013-10-17T20:16:00-07:00ncalexandertag:www.ncalexander.net,2013-10-17:blog/2013/10/17/my-vision-for-the-mobileandroid-build-system/<p>My colleague Gregory Szorc (gps on <span class="caps">IRC</span>) just blogged about Mozilla’s
<a class="reference external" href="http://gregoryszorc.com/blog/2013/10/15/the-state-of-the-firefox-build-system-%282013-q3-review%29/">Q3 build system progress</a>
and started a conversation about the <a class="reference external" href="https://groups.google.com/d/msg/mozilla.dev.builds/PUebwBF6Ewk/DZGQdhDe9CQJ">Q4 build system goals</a>.
This post will elaborate on Greg’s writing by describing my vision for
one of those goals: Generate <span class="caps">IDE</span> project files for Fennec. This is
intended to be vision more than a statement of the work required (but
you can follow the <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=924961">meta bug</a>).</p>
<p>The current state of developing mobile/android? Text editors. It’s text
editors all the way down. Now, I’m an emacs man; and I have nothing
against the vi(m) wielding barbarians I work with; but Java development
is best done with the support of a stiff drink and a sharp <span class="caps">IDE</span>.</p>
<div class="section" id="the-big-goal">
<h2>The big goal</h2>
<p>The big goal is to enable our main front-end developers, the ones who
work with the Java <span class="caps">UI</span> source and resources, to live in the <span class="caps">IDE</span> of their
choice (provided their choice is IntelliJ or Eclipse). To live within
the <span class="caps">IDE</span> means: to edit source and resources, compile the Java sources,
build the Android <span class="caps">APK</span>, deploy it to device, and debug it live; all
within the <span class="caps">IDE</span>. A related goal is to make it easier for new contributors
— those with Java and Android experience, but no prior contact with
Mozilla — to pitch in with Fennec front-end development. Federico
Paolinelli (fedepaol on <span class="caps">IRC</span>) has a great <a class="reference external" href="http://mytechaddiction.blogspot.ca/2013/07/contributing-to-firefox-for-android.html">blog
post</a>
about his experience as a new contributor, which everybody should read.
While I think lowering the bar to entry is extremely important, I claim
that right now we need to focus on driving improvements for our core
development team.</p>
</div>
<div class="section" id="test-suites">
<h2>Test suites</h2>
<p>Some sub-goals I have relate to doing similar things with smaller pieces
of mobile/android. A few of these have to do with test suites:</p>
<p>I recently landed a test suite (building, but not yet running on <span class="caps">TBPL</span>)
called background that is built on the JUnit 3 instrumentation testing
framework. This is a natural candidate for its own <span class="caps">IDE</span> project: this
suite is developed in the android-sync git repository as a separate
project! Configuring an <span class="caps">IDE</span> project for this suite should be straight
forward. This suite is a testing ground for sibling suite, to be called
browser, that will test non-<span class="caps">UI</span> front-end code (think things like
databases, parsing responses from the update server, and uploading
telemetry). Using the lessons from the background suite, we should be
able to ramp up the browser suite quickly. The pay-off here is getting
non-<span class="caps">UI</span> tests into a faster to execute and (hopefully!) more reliable
testing framework on <span class="caps">TBPL</span>, which should reduce intermittent orange pain
for the sherrifs and reduce load on releng infrastructure. This project
needs love to get it running on <span class="caps">TBPL</span>, but making this suite work in the
<span class="caps">IDE</span> is ripe for contribution <em>right now</em>.</p>
<p>We have a test suite called robocop that is built on the Robotium
framework for driving front-end <span class="caps">UI</span> tests. Robocop is notoriously
difficult to even <em>run</em> locally, and updating it is Kafka-esque ordeal.
We would dearly love to create a robocop <span class="caps">IDE</span> project to let our
developers modify and run individual tests in that suite from within the
<span class="caps">IDE</span>. There are some technical barriers thrown up by the build system and
there are some process hurdles: the test suite needs an ambient <span class="caps">HTTP</span>
server and a test profile configured to talk to that server; but in
theory we can drive this process from within the <span class="caps">IDE</span>. One idea I have
not had a chance to explore: can we implement a custom Android
TestRunner to stand up and teardown the ambient <span class="caps">HTTP</span> server? This test
runner would only be used by these <span class="caps">IDE</span> projects. Making this happen will
take a little longer than getting background and browser into the <span class="caps">IDE</span>.</p>
</div>
<div class="section" id="splitting-mobile-android-base">
<h2>Splitting <tt class="docutils literal">mobile/android/base</tt></h2>
<p>Some aspirations I have around pieces of the <tt class="docutils literal">mobile/android</tt> tree:</p>
<p>I would like to build the third-party libraries Sync relies on, and
third-party libraries that future Fennec might rely on, as an
independent <span class="caps">IDE</span> project. This code evolves much more slowly than
mobile/android/base and we should not be paying the compilation and
especially dexing costs during normal development.</p>
<p>I would like to deprecate the android-sync repository. It has served a
valuable purpose and been incredibly useful — I don’t think Android
Sync would exist without it — but if we could develop
<tt class="docutils literal">mobile/android/services</tt> as a separate <span class="caps">IDE</span> project, with the test
suite at <tt class="docutils literal">mobile/android/tests/background/junit3</tt> as another <span class="caps">IDE</span>
project, perhaps we can avoid paying the git-to-hg overhead. (Of course,
we’d lose the github integration — but that’s a separate issue.)</p>
<p>Finally, a pie-in-the-sky goal:</p>
<p>I would like to support an "app tier only" build. That is, a new
developer would download the latest Fennec Nightly build from <span class="caps">TBPL</span>,
clone mozilla-central, run <tt class="docutils literal">mach bootstrap mobile/android</tt>, and be
able to develop the front-end code <em>without</em> building libxul.so and the
other C++ libraries. Mike Hommey (glandium on <span class="caps">IRC</span>) has ideas for how to
make this reality, but this is truly future vision.</p>
</div>
<div class="section" id="conclusion">
<h2>Conclusion</h2>
<p>This is an exciting time for Fennec front-end developers. We’re on the
cusp of unleashing better tooling and better testing. In future posts,
I’ll dig into some of the technical details of how we’re going to get
over the hump.</p>
<p>I’d like to recognize the great work that Brian Nicholson (bnicholson on
<span class="caps">IRC</span>) has done in service of these goals, but the list of supporters is
too long to recite (not least, the build system peers who review!).</p>
<p>Feedback is always appreciated: discussion is best conducted on the
<a class="reference external" href="https://mail.mozilla.org/listinfo/mobile-firefox-dev">mobile-firefox-dev mailing list</a> and I’m
nalexander on <span class="caps">IRC</span>.</p>
</div>