This is some very messy logic built up since 2010. This will all go away
once we have a python3 version of androguard available.
The removed imports and `dir(APK)` is to silence pyflakes
closes#303
This is a little omission. keys that are used in metadata/*.yml all start
with an UpperCase letter, but in fdroidserver, index-v1.json, and
fdroidclient, it is all camelCase with lowercase first letter. The keys
from the 'localized' section are currently never in metadata/*.yml, so
these keys never get downcase. This change will break fdroidclient
versions that do not also have this change, but since we're in alpha, that
should be fine.
If support for a 'localized' section is added to metadata/*.yml, then the
keys there should probably be UpperCase CamelCase to match the other keys.
This was failing on environments that did not have any LANG or LC_* locale
variables set. This is a valid setup, and is common in headless setups, so
it needs to be handled.
This also adds a new pass of the test suite without the locale env vars set
so that this situation is also tests on gitlab-ci, not only gpjenkins.
The error this caused was:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-18: ordinal not in range(128)
Google has their own utility for verifying APK signatures on a desktop
machine since Java's jarsigner is bad for the task. For example, it
acts as if an unsigned APK validates. And to check whether an APK is
unsigned using jarsigner is difficult.
apksigner also does the v2 signatures, so it will have to be used
eventually anyway. It is already in Debian/stretch and can be
available in jessie-backports if need be.
https://android.googlesource.com/platform/tools/apksighttps://packages.debian.org/apksigner
There are two SHA1 git commit IDs that fdroidserver includes in the builds
it makes: fdroidserverid and buildserverid. Originally, these were inserted
into AndroidManifest.xml, but that makes the build not reproducible. So
instead they are included as separate files in the APK's META-INF/ folder.
If those files exist in the signed APK, they will be part of the signature
and need to also be included in the unsigned APK for it to validate.
The ZIP format allows multiple entries with the exact same filename, and on
top of that, it does not allow deleting or updating entries. To make the
`fdroid verify` procedure failsafe, it needs to create a new temporary APK
that is made up on the contents of the "unsigned APK" and the signature
from the "signed APK". Since it would be possible to give a signed APK as
in the unsigned one's position, `fdroid verify` was not able to update the
signature since it was just adding the new signature to the end of the ZIP
file. When reading a ZIP, the first entry is used.
This is a bit different than index.jar: instead of their being index.xml
and index_unsigned.jar, the presense of index-v1.json means that there is
unsigned data. That file is then stuck into a jar and signed by the
signing process. index-v1.json is never published to the repo. It is
included in the binary transparency log, if that is enabled.
Using datetime instances as the internal format makes it much easier to
convert between the formats needed for index.xml and index-v1. apkcache
still uses time tuples and known_apks.txt still uses the ISO date.
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc). This still generates the exact same index.xml as before.
This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats. UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
FDroidPopen is used for running many commands - from git to gradle to
custom commands via flags like build=. When any of these invoke calls to
custom build systems or upstream programs/scripts, it's not safe to
assume that the output will be utf8.
Unfortunately, this currently leads to crashes and failed builds:
ERROR: Could not build app org.kiwix.kiwixmobile due to unknown error: Traceback (most recent call last):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 1155, in main
options.onserver, options.refresh):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 951, in trybuild
build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 648, in build_local
p = FDroidPopen(['bash', '-x', '-c', cmd], cwd=root_dir)
File "/home/vagrant/fdroidserver/fdroidserver/common.py", line 1786, in FDroidPopen
result.output = result.output.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 5397290: invalid start byte
One way to fix this would be to use one of the python libraries that
guess an encoding. But a much safer option is to tell the decode method
to ignore non-utf8 bytes, as opposed to crashing on them.
A .asc or .sig file is a detached PGPG signature, `fdroid gpgsign`
generates them. It makes no sense for them to be ever treated as a file
for distribution.
This also adds to forgotten forms of index files.
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order. To preserve order, the data must be either
a tuple () or list [].
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.
The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.
Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
Since `fdroid update` parses the output of aapt, and since aapt's output
format changes in non-compatible ways, test to make sure that the version
of aapt is new enough to prevent mystery stacktraces. This only prints a
warning since in many cases, the old version will work just fine.
When a git repo has a .fdroid.yml file in it, and `fdroid build` is run
directly in that git repo, then this uses the file path as the remote for
the git repo in build/appid that is actually built. That makes it possible
to run builds of commits that are only local, and makes things a whole lot
faster.
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`. If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.
This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code. This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control. This should make it easier to
spread around the maintenance load.
This makes sure there is a GPG signature on any file that is included in
the repo, including APKs, OBB, source tarballs, media files, OTA update
ZIPs, etc. Having a GPG signature is more important on non-APK files since
they mostly do not have any signature mechanism of their own.
This also adds basic tests of adding non-APK/OBB files to a repo with
`fdroid update`.
closes#232
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not. This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.
https://gitlab.com/fdroid/privileged-extension/issues/9
