The new policy is to move APKs with invalid signatures to the archive,
and only add those APKs to the archive's index if they have valid MD5
signatures.
closes#323closes#292
In April 2017, Oracle's jarsigner and Google's apksigner both switched to
considering any APK signature that uses MD5 as unsigned. Any old build
is likely to have a MD5 signature. This sets up the archive as the only
place where these "disabled algorithms" are allowed in the repo, and
marks any APK signed by a "disabled algorithm" as having a "known
vulnerability"
This also now automatically moves APKs with invalid signatures to the
archive section.
#323
This is useful for parsing APK files, which can include packageName,
versionCode, and optionally 7 char signing key ID (i.e. <sig>).
This also can set the packageName and versionCoe for non APK files, so
that it is easy to assign them to metadata files, and to allow for
upgrades by setting the versionCode in the filename.
This is some very messy logic built up since 2010. This will all go away
once we have a python3 version of androguard available.
The removed imports and `dir(APK)` is to silence pyflakes
closes#303
This is a little omission. keys that are used in metadata/*.yml all start
with an UpperCase letter, but in fdroidserver, index-v1.json, and
fdroidclient, it is all camelCase with lowercase first letter. The keys
from the 'localized' section are currently never in metadata/*.yml, so
these keys never get downcase. This change will break fdroidclient
versions that do not also have this change, but since we're in alpha, that
should be fine.
If support for a 'localized' section is added to metadata/*.yml, then the
keys there should probably be UpperCase CamelCase to match the other keys.
This was failing on environments that did not have any LANG or LC_* locale
variables set. This is a valid setup, and is common in headless setups, so
it needs to be handled.
This also adds a new pass of the test suite without the locale env vars set
so that this situation is also tests on gitlab-ci, not only gpjenkins.
The error this caused was:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-18: ordinal not in range(128)
Google has their own utility for verifying APK signatures on a desktop
machine since Java's jarsigner is bad for the task. For example, it
acts as if an unsigned APK validates. And to check whether an APK is
unsigned using jarsigner is difficult.
apksigner also does the v2 signatures, so it will have to be used
eventually anyway. It is already in Debian/stretch and can be
available in jessie-backports if need be.
https://android.googlesource.com/platform/tools/apksighttps://packages.debian.org/apksigner
There are two SHA1 git commit IDs that fdroidserver includes in the builds
it makes: fdroidserverid and buildserverid. Originally, these were inserted
into AndroidManifest.xml, but that makes the build not reproducible. So
instead they are included as separate files in the APK's META-INF/ folder.
If those files exist in the signed APK, they will be part of the signature
and need to also be included in the unsigned APK for it to validate.
The ZIP format allows multiple entries with the exact same filename, and on
top of that, it does not allow deleting or updating entries. To make the
`fdroid verify` procedure failsafe, it needs to create a new temporary APK
that is made up on the contents of the "unsigned APK" and the signature
from the "signed APK". Since it would be possible to give a signed APK as
in the unsigned one's position, `fdroid verify` was not able to update the
signature since it was just adding the new signature to the end of the ZIP
file. When reading a ZIP, the first entry is used.
This is a bit different than index.jar: instead of their being index.xml
and index_unsigned.jar, the presense of index-v1.json means that there is
unsigned data. That file is then stuck into a jar and signed by the
signing process. index-v1.json is never published to the repo. It is
included in the binary transparency log, if that is enabled.
Using datetime instances as the internal format makes it much easier to
convert between the formats needed for index.xml and index-v1. apkcache
still uses time tuples and known_apks.txt still uses the ISO date.
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc). This still generates the exact same index.xml as before.
This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats. UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
FDroidPopen is used for running many commands - from git to gradle to
custom commands via flags like build=. When any of these invoke calls to
custom build systems or upstream programs/scripts, it's not safe to
assume that the output will be utf8.
Unfortunately, this currently leads to crashes and failed builds:
ERROR: Could not build app org.kiwix.kiwixmobile due to unknown error: Traceback (most recent call last):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 1155, in main
options.onserver, options.refresh):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 951, in trybuild
build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 648, in build_local
p = FDroidPopen(['bash', '-x', '-c', cmd], cwd=root_dir)
File "/home/vagrant/fdroidserver/fdroidserver/common.py", line 1786, in FDroidPopen
result.output = result.output.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 5397290: invalid start byte
One way to fix this would be to use one of the python libraries that
guess an encoding. But a much safer option is to tell the decode method
to ignore non-utf8 bytes, as opposed to crashing on them.
A .asc or .sig file is a detached PGPG signature, `fdroid gpgsign`
generates them. It makes no sense for them to be ever treated as a file
for distribution.
This also adds to forgotten forms of index files.
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order. To preserve order, the data must be either
a tuple () or list [].
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.
The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.
Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
