F-Droid Server
**************

This manual is for the F-Droid repository server tools.

Copyright © 2010, 2011, 2012 Ciaran Gultnieks Copyright © 2011 Henrik Tunedal, Michael Haas, John Sullivan

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".

Overview
********

The F-Droid server tools provide various scripts and tools that are used to maintain the main F-Droid application repository. You can use these same tools to create your own additional or alternative repository for publishing, or to assist in creating, testing and submitting metadata to the main repository.

System Requirements
*******************

The system requirements for using the tools will vary depending on your intended usage. At the very least, you'll need:

  * GNU/Linux

  * Python 2.x

  * The Android SDK

If you intend to build applications from source you'll also need most, if not all, of the following:

  * All available SDK platforms and tools installed in the Android SDK, but *not* the proprietary components.

  * The Android NDK

  * Ant

  * Ant Contrib Tasks (Debian package ant-contrib)

  * Maven (Debian package maven2)

  * JavaCC (Debian package javacc)

  * JDK (Debian package openjdk-6-jdk)

  * VCS clients: svn, git, git-svn, hg, bzr

  * A keystore for holding release keys. (Safe, secure and well backed up!)

If you intend to use the 'Build Server' system, for secure and clean builds (highly recommended), you will also need:

  * VirtualBox (debian package virtualbox-ose)

  * Ruby

  * Vagrant and Vagrant-snap

  * Paramiko (debian package python-paramiko)

Setup
*****

Because the tools and data will always change rapidly, you will almost certainly want to work from a git clone of the tools at this stage. To get started:

     git clone git://gitorious.org/f-droid/fdroidserver.git

You now have lots of stuff in the fdroidserver directory, but the most important is the 'fdroid' command script which you run to perform all tasks. This script is always run from a repository data directory, so the most sensible thing to do next is to put your new fdroidserver directory in your PATH.

Data
====

To do anything, you'll need at least one repository data directory. It's from this directory that you run the fdroid command to perform all repository management tasks. You can either create a brand new one, or grab a copy of the data used by the main F-Droid repository:

     git clone git://gitorious.org/f-droid/fdroiddata.git

Regardless of the intended usage of the tools, you will always need to set up some basic configuration details. This is done by creating a file called config.py in the data directory. You should do this by copying the example file (config.sample.py) from the fdroidserver project to your data directory and then editing according to the instructions within.

Once configured in this way, all the functionality of the tools is accessed by running the fdroid command. Run it on its own to get a list of the available sub-commands.

You can follow any command with --help to get a list of additional options available for that command.

     fdroid update --help

Simple Binary Repository
************************

If you want to maintain a simple repository hosting only binary APKs obtained and compiled elsewhere, the process is quite simple:

 1. Set up the server tools, as described in Setup.

 2. Make a directory for your repository. This is the directory from which you will do all the work with your repository. Create a config file there, called config.py, by copying the config.sample.py from the server project and editing it.

 3. Within that, make a directory called repo and put APK files in it.

 4. Run fdroid update.

 5. If it reports that any metadata files are missing, you can create them in the metadata directory and run it again.

 6. To ease creation of metadata files, run fdroid update with the -c option. It will create 'skeleton' metadata files that are missing, and you can then just edit them and fill in the details.

 7. Then, if you've changed things, run fdroid update again.

 8. Running fdroid update adds an Icons directory into the repo directory, and also creates the repository index (index.xml, and also index.jar if you've configured the system to use a signed index).

 9. Publish the resulting contents of the repo directory to your web server.

Following the above process will result in a repo directory, which you simply need to push to any HTTP (or preferably HTTPS) server to make it accessible.

While some information about the applications (and versions thereof) is retrieved directly from the APK files, most comes from the corresponding file in the metadata directory. The metadata file covering ALL versions of a particular application is named package.id.txt where package.id is the unique identifier for that package.

See the Metadata chapter for details of what goes in the metadata file. All fields are relevant for binary APKs, EXCEPT for 'Build Version' entries, which should be omitted.

Building Applications
*********************

Instead of (or as well as) including binary APKs from external sources in a repository, you can build them directly from the source code.

Using this method, it is is possible to verify that the application builds correctly, corresponds to the source code, and contains only free software. Unforunately, in the Android world, it seems to be very common for an application supplied as a binary APK to present itself as Free Software when in fact some or all of the following are true:

 1. The source code (either for a particular version, or even all versions!) is unavailable or incomplete.

 2. The source code is not capable of producing the actual binary supplied.

 3. The 'source code' contains binary files of unknown origin, or with proprietary licenses.

For this reason, source-built applications are the preferred method for the main F-Droid repository, although occasionally for technical or historical reasons, exceptions are made to this policy.

When building applications from source, it should be noted that you will be signing them (all APK files must be signed to be installable on Android) with your own key. When an application is already installed on a device, it is not possible to upgrade it in place to a new version signed with a different key without first uninstalling the original. This may present an inconvenience to users, as the process of uninstalling loses any data associated with the previous installation.

The process for managing a repository for built-from-source applications is very similar to that described in the Simple Binary Repository chapter, except now you need to:

 1. Include Build Version entries in the metadata files.

 2. Run fdroid build to build any applications that are not already built.

 3. Run fdroid publish to finalise packaging and sign any APKs that have been built.

More about build.py
===================

When run without any parameters, fdroid build will build any and all versions of applications that you don't already have in the repo directory (or more accurately, the unsigned directory. There are various other things you can do. As with all the tools, the --help option is your friend, but a few annotated examples and discussion of the more common usage modes follows:

To build a single version of a single application, you could run the following:

     ./fdroid build --package=org.fdroid.fdroid --vercode 16

This attempts to build version code 16 (which is version 0.25) of the F-Droid client. Many of the tools recognise this --package parameter, allowing their activity to be limited to just a single package.

If the build above was succesful, two files will have been placed in the unsigned directory:

     org.fdroid.fdroid_16.apk
     org.fdroid.fdroid_16_src.tar.gz

The first is the (unsigned) APK. You could sign this with a debug key and push it direct to your device or an emulator for testing. The second is a source tarball containing exactly the source that was used to generate the binary.

If you were intending to publish these files, you could then run:

     ./fdroid publish

The source tarball would move to the repo directory (which is the directory you would push to your web server). A signed and zip-aligned version of the APK would also appear there, and both files would be removed from the unsigned directory.

If you're building purely for the purposes of testing, and not intending to push the results to a repository, at least yet, the --test option can be used to direct output to the tmp directory instead of unsigned. A similar effect could by achieved by simply deleting the output files from unsigned after the build, but with the risk of forgetting to do so!

Along similar lines (and only in conjunction with --test, you can use --force to force a build of a Disabled application for test purposes, where normally it would be completely ignored.

Direct Installation
===================

You can also build and install directly to a connected device or emulator using the --install switch. If you do this without using --package and --vercode then all versions of all packages will be installed (with each individual version overwriting the previous!). In most cases, this will not be what you want to do, so execution will stop straight away. However, you can override this if you're sure that's what you want, by using --all.

Importing Applications
**********************

To help with starting work on including a new application, fdroid import will take a URL and optionally some other parameters, and attempt to construct as much information as possible by analysing the source code. Basic usage is:

     ./fdroid import --url=http://address.of.project

For this to work, the URL must point to a project format that the script understands. Currently this is limited to one of the following:

 1. Gitorious - https://gitorious.org/PROJECTNAME/REPONAME

 2. Github - https://github.com/USER/PROJECT

 3. Google Code - http://code.google.com/p/PROJECT/ Supports git, svn and hg repos.

     Some Google Code projects have multiple repositories, identified by a dropdown list on the source/checkout page. To access one other than the default, specify its name using the --repo switch.

 4. Bitbucket - https://bitbucket.org/USER/PROJECT/

 5. Git - git://REPO

Depending on the project type, more or less information may be gathered. For example, the license will be retrieved from a Google Code project, but not a GitHub one. A bare repo url, such as the git:// one, is the least preferable optional of all, since you will have to enter much more information manually.

If the import is successful, a metadata file will be created. You will need to edit this further to check the information, and fill in the blanks.

If it fails, you'll be told why. If it got as far as retrieving the source code, you can inspect it further by looking in tmp/importer where a full checkout will exist.

A frequent cause of initial failure is that the project directory is actually a subdirectory in the repository. In this case, run the importer again using the --subdir option to tell it where. It will not attempt to determine this automatically, since there may be several options.

Metadata
********

Information used by update.py to compile the public index comes from two sources:

 1. the APK files in the repo directory, and

 2. the metadata files in the metadata directory.

The metadata files are simple, easy to edit text files, always named as the application's package ID with '.txt' appended.

Note that although the metadata files are designed to be easily read and writable by humans, they are also processed and written by various scripts. They are capable of rewriting the entire file when necessary. Even so, the structure and comments will be preserved correctly, although the order of fields will be standardised. (In the event that the original file was in a different order, comments are considered as being attached to the field following them). In fact, you can standardise all the metadata in a single command, without changing the functional content, by running:

     fdroid rewritemetadata

The following sections describe the fields recognised within the file.

Category
========

A single category for the application to be placed in. There is no fixed list of categories - both the client and the web site will automatically show any categories that exist in any applications. However, if your metadata is intended for the main F-Droid repository, you should probably use one of the existing categories (look at the site/client), or discuss the proposal to add a new one.

License
=======

The license for the application.

Common values:

  * GPLv2 GNU GPL version 2

  * GPLv2+ GNU GPL version 2 or later

  * GPLv3 GNU GPL version 3

  * GPLv3+ GNU GPL version 3 or later

  * GPL An unspecified GPL version. Use this only as a last resort.

  * AGPL Afferro GPL version 3.

  * Apache2 Apache 2

  * MIT MIT X11 license

  * BSD BSD license - the original '4-clause' version.

  * NewBSD BSD license - the new, or modified, version.

Name
====

The name of the application. Normally, this field should not be present since the application's correct name is retrieved from the APK file. However, in a situation where an APK contains a bad or missing application name, it can be overridden using this.

Web Site
========

The URL for the application's web site.

Source Code
===========

The URL to view or obtain the application's source code. This should be something human-friendly. Machine-readable source-code is covered in the 'Repo' field.

Issue Tracker
=============

The URL for the application's issue tracker. Optional, since not all applications have one.

Donate
======

The URL to donate to the project. This should be the project's donate page if it has one.

It is possible to use a direct PayPal link here, if that is all that is available. However, bear in mind that the developer may not be aware of that direct link, and if they later changed to a different PayPal account, or the PayPal link format changed, things could go wrong. It is always best to use a link that the developer explicitly makes public, rather than something that is auto-generated 'button code'.

FlattrID
========

The project's Flattr (http://flattr.com) ID, if it has one. This should be a numeric ID, such that (for example) https://flattr.com/thing/xxxx leads directly to the page to donate to the project.

Bitcoin
=======

A bitcoin address for donating to the project.

Summary
=======

A brief summary of what the application is.

Description
===========

A full description of the application. This can span multiple lines, and is terminated by a line containing a single '.'.

Repo Type
=========

The type of repository - for automatic building from source. If this is not specified, automatic building is disabled for this application. Possible values are:

  * git

  * svn

  * git-svn

  * hg

  * bzr

  * srclib

Repo
====

The repository location. Usually a git: or svn: URL, for example.

The git-svn option connects to an SVN repository, and you specify the URL in exactly the same way, but git is used as a back-end. This is preferable for performance reasons, and also because a local copy of the entire history is available in case the upstream repository disappears. (It happens!)

For a Subversion repo that requires authentication, you can precede the repo URL with username:password and those parameters will be passed as --username and --password to the SVN checkout command. (This works only for plain svn and not for git-svn - one of the very few cases where using svn is advisable).

Build Version
=============

Any number of these fields can be present, each specifying a version to automatically build from source. The value is a comma-separated list. For example:

Build Version:0.12,3,651696a49be2cd7db5ce6a2fa8185e31f9a20035

The above specifies to build version 0.12, which has a version code of 3. The third parameter specifies the tag, commit or revision number from which to build it in the source repository.

If the commit version starts with a !, that version is not built. Instead, everything after the ! is used as a reason why it can't be built. The purpose of this feature is to allow non-buildable releases (e.g. the source is not published) to be flagged, so the scripts don't generate repeated messages about them. (And also to record the information for review later).

In addition to the three, always required, parameters described above, further parameters can be added (in name=value format) to apply further configuration to the build. These are:

subdir=<path>
     Specifies to build from a subdirectory of the checked out source code. Normally this directory is changed to before building,

bindir=<path>
     Normally the build output (apk) is expected to be in the bin subdirectory below the ant build files. If the project is configured to put it elsewhere, that can be specified here, relative to the base of the checked out repo.

oldsdkloc=yes
     The sdk location in the repo is in an old format, or the build.xml is expecting such. The 'new' format is sdk.dir while the VERY OLD format is sdk-location. Typically, if you get a message along the lines of: "com.android.ant.SetupTask cannot be found" when trying to build, then try enabling this option.

target=<target>
     Specifies a particular SDK target, when the source doesn't. This is likely to cause the whole build.xml to be rewritten, which is fine if it's a 'standard' android file or doesn't already exist, but not a good idea if it's heavily customised.

rm=<relpath>
     Specifies the relative path of file to delete before the build is done. The path is relative to the base of the build directory - i.e. the root of the directory structure checked out from the source respository - not necessarily the directory that contains AndroidManifest.xml.

antcommand=xxx
     Specify an alternate ant command (target) instead of the default 'release'.

forceversion=yes
     If specified, the package version in AndroidManifest.xml is replaced with the version name for the build as specified in the metadata.

     This is useful for cases when upstream repo failed to update it for specific tag, or to build an arbitrary revision.

forcevercode=yes
     If specified, the package version code in the AndroidManifest.xml is replaced with the version code for the build. See also forceversion.

update=xxx
     By default, 'android update project' is used to generate or update the build.xml file. Specifying update=no bypasses that.

     Specifiying update=force forces rebuilding of the build.xml file at the same time - this is frequently needed with r14 of the Android platform tools. Be aware of any customisations in build.xml when using update=force.

     Otherwise, value can be a semicolon-separated list of directories in which to run 'android update project' relative to the main application directory (which may include 'subdir' parameter). Default value is '.', and passing non-default value may be useful for multi-component projects. Note that --subprojects switch is automatically passed to 'android update project', so using explicit list may be needed only for peculiar source layouts.

initfun=yes
     Enables a selection of mad hacks to make com.funambol.android build. Probably not useful for any other application.

buildjni=[yes|no|<dir list>]
     Enables building of native code via the ndk-build script before doing the main ant build. The value may be a list of directories relative to the main application directory in which to run ndk-build, or 'yes' which corresponds to '.' . Using explicit list may be useful to build multi-component projects.

     The build and scan processes will complain (refuse to build) if this parameter is not defined, but there is a jni directory present. If the native code is being built by other means, you can specify no here to avoid that. However, if the native code is actually not required, remove the directory instead (using prebuild for example).

submodules=yes
     Use if the project (git only) has submodules - causes git submodule init and update to be executed after the source is cloned.

encoding=xxxx
     Adds a java.encoding property to local.properties with the given value. Generally the value will be 'utf-8'. This is picked up by the SDK's ant rules, and forces the Java compiler to interpret source files with this encoding. If you receive warnings during the compile about character encodings, you probably need this.

prebuild=xxxx
     Specifies a shell command (or commands - chain with &&) to run before the build takes place. Backslash can be used as an escape character to insert literal commas, or as the last character on a line to join that line with the next. It has no special meaning in other contexts; in particular, literal backslashes should not be escaped.

     You can use $$name$$ to substitute the path to a referenced srclib - see the srclib directory for details of this.

     You can use $$SDK$$ and $$NDK$$ to substitute the paths to the android SDK and NDK directories respectively.

init=xxxx
     As for 'prebuild', but runs on the source code BEFORE any other processing takes place.

     You can use $$SDK$$ and $$NDK$$ to substitute the paths to the android SDK and NDK directories respectively.

novcheck=yes
     Don't check that the version name and code in the resulting apk are correct by looking at the build output - assume the metadata is correct. This takes away a useful level of sanity checking, and should only be used if the values can't be extracted.

fixtrans=yes
     Modifies any instances of string resources that use multiple formatting arguments, but don't use positional notation. For example, "Hello %s, %d" becomes "Hello %1$s, %2$d". Newer versions of the Android platform tools enforce this sensible standard. If you get error messages relating to that, you need to enable this.

fixapos=yes
     Like fixtrans, but deals with an even older issue relating to 'unescaped apostrophes' in translation strings.

maven=yes
     Build with maven instead of ant

patch=x
     Apply patch(es). 'x' names one (or more - comma-seperated) files within a directory below the metadata, with the same name as the metadata file but without the extension. Each of these patches is applied to the code in turn.

extlibs=a;b;c
     Specifies a list of external libraries (jar files) from the build/extlib library, which will be placed in the libs directory of the project. Separate items with semicolons.

srclibs=a@r;b@r1;
     Specifies a list of source libraries (kept up to date using version control) from a predefined set. Separate items with semicolons, and each item is of the form name@rev where name is the predefined source library name and rev is the revision in source control to use. You can then also use $$name$$ in the prebuild command to substitute the relative path to the library directory.

     The available source libraries are current hard-coded in common.py. This will later be data-driven.

Another example, using extra parameters:

Build Version:1.09.03,10903,45,subdir=Timeriffic,oldsdkloc=yes

AntiFeatures
============

This is optional - if present, it contains a comma-separated list of any of the following values, describing an AntiFeature the application has:

  * Ads - the application contains advertising.

  * Tracking - the application tracks and reports your activity to somewhere without your consent.

  * NonFreeNet - the application promotes a non-Free network service.

  * NonFreeAdd - the application promotes non-Free add-ons.

  * NonFreeDep - the application depends on a non-Free application (e.g. Google Maps) - i.e. it requires it to be installed on the device, but does not include it.

Disabled
========

If this field is present, the application does not get put into the public index. This allows metadata to be retained while an application is temporarily disabled from being published. The value should be a description of why the application is disabled.

Requires Root
=============

Set this optional field to "Yes" if the application requires root privileges to be usable. This lets the client filter it out if the user so desires.

Update Check Mode
=================

This determines the method using for determining when new releases are available - in other words, the updating of the Current Version and Current Version Code fields in the metadata by the fdroid checkupdates process.

Valid modes are:

  * None - No checking is done

  * RepoManifest - The AndroidManifest.xml file in the most recent commit in the source repository is checked. The appropriateness of this method depends on the development process used by the application's developers. You should not specify this method unless you're sure it's appropriate. (Also, it currently only works for git repositories).

  * Tags - The AndroidManifest.xml file in all tagged revisions in the source repository is checked, looking for the highest version code. The appropriateness of this method depends on the development process used by the application's developers. You should not specify this method unless you're sure it's appropriate. (Also, it currently only works for git repositories).

  * Market - This is being phased out and should not be used!

Current Version
===============

The name of the version that is current. There may be newer versions of the application than this (e.g. betas), and there will almost certainly be older ones. This should be the one that is recommended for general use.

This field is normally automatically updated - see Update Check Mode.

Current Version Code
====================

The version code corresponding to the Current Version field. Both these fields must be correct and matching.

This field is normally automatically updated - see Update Check Mode.

Update Processing
*****************

Detecting
=========

There are various mechanisms in place for automatically detecting that updates are available for applications, with the Update Check Mode field in the metadata determining which method is used for a particular application.

Running the fdroid checkupdates command will apply this method to each application in the repository and update the Current Version and Current Version Code fields in the metadata accordingly.

As usual, the -p option can be used with this, to restrict processing to a particular application.

Note that this only updates the metadata such that we know what the current published/recommended version is. It doesn't make that version available in the repository - for that, see the next section.

Adding
======

Adding updates (i.e. new versions of applications already included in the repository) happens in two ways. The simple case is applications where the APK files are binaries, retrieved from a developer's published build. In this case, all that's required is to place the new binary in the Repo directory, and the next run of fdroid update will pick it up.

For applications built from source, it is necessary to add a new Build Version line to the metadata file. At the very least, the version name, version code and commit will be different. It is also possible that the additional build flags will change between versions.

For processing multiple updates in the metadata at once, it can be useful to run fdroid update --interactive. This will check all the applications in the repository, and where updates are required you will be prompted to [E]dit the metadata, [I]gnore the update, or [Q]uit altogether.

Build Server
************

The Build Server system isolates the builds for each package within a clean, isolated and secure throwaway virtual machine environment.

Rationale
=========

Building applications in this manner on a large scale, especially with the involvement of automated and/or unattended processes, could be considered a dangerous pastime from a security perspective. This is even more the case when the products of the build are also distributed widely and in a semi-automated ("you have updates available") fashion.

Assume that an upstream source repository is compromised. A small selection of things that an attacker could do in such a situation:

 1. Use custom ant build steps to execute virtually anything as the user doing the build.

 2. Access the keystore.

 3. Modify the built apk files or source tarballs for other applications in the repository.

 4. Modify the metadata (which includes build scripts, which again, also includes the ability to execute anything) for other applications in the repository.

Through complete isolation, the repurcussions are at least limited to the application in question.

Aside from security issues, there are some applications which have strange requirements such as custom versions of the NDK. It would be impractical (or at least extremely messy) to start modifying and restoring the SDK on a multi-purpose system, but within the confines of a throwaway single-use virtual machine, anything is possible.

Setting up a build server
=========================

Integrating the build server setup into the main scripts is a work in progress. Some things may not work properly yet. Talk to CiaranG if you're trying to use this and have problems.

In addition to the basic setup previously described, you will also need a Vagrant-compatible Debian Squeeze base box called 'debian6-32'. You can create one of these for yourself from standard Debian installation media, as the specification for what's required to be Vagrant-compatible is very well defined. This is the sensible and secure way to do it, since you know what's in it. If you insist on taking a shortcut, ask CiaranG for his on the forum or in IRC.

With this base box installed, you can then do:

     ./makebuildserver.sh

This will take a long time - most of it spent installing the necessary parts of the Android SDK for all the various platforms. Luckily you only need to do it occasionally.

Once it's complete you'll have a new base box called 'buildserver' which is what's used for the actual builds. You can then build packages as normal, but with the addition of the --server flag to fdroid build to instruct it to do all the hard work within the virtual machine, which is reset to a completely clean state for every package built.

GNU Free Documentation License
******************************

     [file fdl.texi not included]