rsync improvements for fdroid server update
This is a couple of improvements to how `fdroid server update` uses `rsync`.
(also, please remove the branch when accepting any of my merge requests)
This allows the SSH key used to sync with the server to be specified via
the config.py or the command line. I need it for running automated tests
and setups.
rsync uses the modification time and size of the file when deciding whether
to update a file. These are relatively easy to control in malicious code,
so instead make rsync use a full MD5 checksum when decided whether the
index needs to be updated. I suppose we could add an option to use
checksum checking on all files, but since the signed repo already provides
a checksum check, it seems not worth the added load on the process.
Also, renamed 'index' to 'indexxml' to make it clear what is the XML and
what is the JAR.
Test updates and related bug fixes
I just set up some big tests of generating repos based on feeding as many random APKs into `fdroid update` as possible. On our jenkins server, the tests copy all of the APKs that the jenkins server has generated and builds a repo from them. This process caught lots of little glitches in the whole process. While these little glitches are usually caused by problematic APKs, `fdroid update` should handle them gracefully. Hopefully this set of fixes accomplishes that.
It is not necessarily a good idea to try to distribute system APKs via
FDroid, but `fdroid update` should just ignore APKs it cannot handle rather
than die and prevent a repo from being fully created. This is necessary to
handle the automatic creation of repos, like for debug builds from a
Jenkins server.
Using this example app which does not have <uses-sdk>:
https://android.googlesource.com/platform/development/+log/master/samples/ApiDemos/assets/HelloActivity.apk
aapt then returns "sdkVersion:'IceCreamSandwich'". minSdkVersion is only
ever supposed to be an integer, so this is a bizarre APK. It is included
only as a binary in the git repo for Android sample code. But who knows
what else is out there, so report and error and carry on with the update
process.
Some APKs can be corrupt or some system APKs do not have all the normal
info. Instead of quitting, `fdroid update` skips the non-parsable APKs and
optionally deletes them if --delete-unknown is specified.
FDroidPopen outputs by default, this should be controlled by the --verbose
flag so that most of the time, only meaningful messages are shown like
errors and such. For command output that should be shown everytime,
output=True can be set.
Make sure that fdroid can find aapt in the current config, otherwise exit
with an error. Some users don't have build_tools set, and their SDK does
not include the build-tools in the default versioned dir, so this should
warn them of what is wrong.
Since the new metadata created by -c is added after the existing metadata
was already parsed, `fdroid update -c` was not doing a complete update of
the repo. This moves the metadata creation as early as possible, then
reruns the metadata parsing if new metadata was created.
refs #12https://gitlab.com/fdroid/fdroidserver/issues/12
This adds the option --delete-unknown for the current default behavior of
`fdroid update`: to delete any unknown APKs. Instead, it just outputs a
warning about the unknown APKs and suggests -c for adding it.
Fixes#8https://gitlab.com/fdroid/fdroidserver/issues/8
