A Binary Transparency Log is a append only log of all binaries published by
a repo. This is useful for people to find whether the binary they have
matches what F-Droid has published, and also makes it more difficult for
the published history to be changed without notice, or for a server to give
specific users custom malware binaries.
https://www.eff.org/deeplinks/2014/02/open-letter-to-tech-companies
FDroidPopen is used for running many commands - from git to gradle to
custom commands via flags like build=. When any of these invoke calls to
custom build systems or upstream programs/scripts, it's not safe to
assume that the output will be utf8.
Unfortunately, this currently leads to crashes and failed builds:
ERROR: Could not build app org.kiwix.kiwixmobile due to unknown error: Traceback (most recent call last):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 1155, in main
options.onserver, options.refresh):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 951, in trybuild
build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 648, in build_local
p = FDroidPopen(['bash', '-x', '-c', cmd], cwd=root_dir)
File "/home/vagrant/fdroidserver/fdroidserver/common.py", line 1786, in FDroidPopen
result.output = result.output.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 5397290: invalid start byte
One way to fix this would be to use one of the python libraries that
guess an encoding. But a much safer option is to tell the decode method
to ignore non-utf8 bytes, as opposed to crashing on them.
Since it is now possible to build and include arbitrary files, like OTA
update ZIP files, the update procedure needs to look for non-APK files that
match the packageName_versionCode pattern of fdroid-generated files.
!193
admin#14
privileged-extension#9
`fdroid build` will generate source tarballs, and now with support for
adding any file to a repo, we need to explicitly ignore the fdroid-
generated source tarballs. If people want to include source tarballs in a
repo, they still can, as long as that source tarball doesn't use the
`fdroid build` tarball naming scheme.
This forces <uses-permission maxSdkVersion=""> to be an integer in the
internal dict, and forces it to have no decimal point in XML. Having it as
an integer in the internal dict means data will pass directly through to
the fdroidclient Apk instance, where it is ultimately an integer.
fdroidclient should handle no minSdkVersion fine, since it just parses the
text in <sdkver></sdkver> as an int, and uses a default value if there is
an Exception.
A .asc or .sig file is a detached PGPG signature, `fdroid gpgsign`
generates them. It makes no sense for them to be ever treated as a file
for distribution.
This also adds to forgotten forms of index files.
This lets index-v1 be parsed directly into class instances because the
field/instance var names match exactly. The original index v0 element
must retain the 'lastupdated' name for backwards compatibility.
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order. To preserve order, the data must be either
a tuple () or list [].
In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean. For now,
make the YAML and JSON metadata produce the same internal data as .txt.
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.
The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.
Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
Python is heavily based on its core data types, and dict is one of the more
important ones. Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded. This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client. This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.
This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:
* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'
Inspired by:
https://goodcode.io/articles/python-dict-object/
This makes it a lot easier to work with Build instances with parsing and
dumping libraries, since they expect only core Python types (dict, list,
tuple, str, etc)
ad2b9b99c2 put this in the wrong place, it
was running it on the buildserver host rather than in the buildserver VM
itself, where the builds actually run.
refs #148
This is to clean up the code for the introduction of a new index format.
This also makes it so that only repo/categories.txt is generated, and not
archive/categories.txt.
Currently, the only thing that I can find that reads categories.txt is the
wordpress plugin, e.g. wp-fdroid. And it only reads repo/categories.txt
not archive/categories.txt.
* New command `dscanner`, enables one to scan signed APKs with Drozer
* Drozer is a dynamic vulnerability scanner for Android
* Drozer runs in a emulator or on-device, this new `dscanner` command...
* starts a docker image with Drozer and the Android Emulator pre-installed,
* loads the signed APK into the emulator
* activates Drozer automated tests for the APK
* gathers the report output and places it next to the original APK
* The Drozer docker image can be:
* cached locally for re-use (just don't run --clean*)
* retrieved from dockerhub.com for more efficient runtime
* or be built from scratch (in the new "./docker" directory)
* New "Vulnerability Scanning" documentation section (run gendocs.sh)
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work. Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
