8k is too small, e.g. when make -jN errors or when there are a lot of scanner
errors and warnings. 16k should be better. Also, use "[...]" to make it
obvious that the output is truncated.
Initially, the scanner used libmagic which used magic numbers in the file's
content to detect what kind of file it appears to be. Since that library isn't
available on all systems, we added support for two other libraries, mimetypes
amongst them.
The issue with mimetypes is that it only uses the file's extension, not its
actual content. So this ends in variable behaviour depending on what system
you're using fdroidserver on. For example, an executable binary without
extension would be ignored if mimetypes was being used.
We now drop all libraries - mimetypes too as it depends on the system's
mime.types file - and instead check extensions ourselves. On top of that, do
a simple binary content check to find binary executables that don't have an
extension.
The new in-house code without any dependencies doesn't add any new checks, so
no builds should break. The current checks still work:
% fdroid scanner app.openconnect:1029
[...]
Found executable binary at assets/raw/armeabi/curl
Found executable binary at assets/raw/mips/curl
Found executable binary at assets/raw/x86/curl
Found JAR file at lib/XposedBridgeApi-54.jar
Found JAR file at libs/acra-4.5.0.jar
Found JAR file at libs/openconnect-wrapper.jar
Found JAR file at libs/stoken-wrapper.jar
Found shared library at libs/armeabi/libopenconnect.so
Found shared library at libs/armeabi/libstoken.so
Found shared library at libs/mips/libopenconnect.so
Found shared library at libs/mips/libstoken.so
Found shared library at libs/x86/libopenconnect.so
Found shared library at libs/x86/libstoken.so
For a bit repo like f-droid.org, it makes sense to standardize on a single
format for metadata files. This adds support for enforcing a single data
format, or a reduced set of data formats. So f-droid.org would run like
this if it changed to YAML:
accepted_formats = ['txt', 'yaml']
Then once everything was converted to YAML, it could look like this:
accepted_formats = ['yaml']
For devs that want to build and distribute nightly builds of their apps
using the fdroid tools. The core idea here is to make the fdroidserver
tool suite the single set of tools for all types of builds and releases.
That will hopefully drive more free software developers to make f-droid.org
an core channel for official releases.
On OSX, when Java is not installed, it'll fail to create the keystore, but
then give an error from chmod failing. This changes things so that the
missing Java is reported instead.
There are two forms of python-magic, one included in libmagic that is
default on Debian, and another Python wrapper for libmagic that is
called 'python-magic' on pypi. Those both rely on the compiled binary
library libmagic. For platforms without good package management, fallback
to using the built-in 'mimetypes' library if 'magic' is not available.
This supports OSX, Windows, and BSD.
Python libraries work better in Python than running external commands, and
it also makes the code much more portable. For example, the GNU and BSD
sed commands have different, and sometimes conflicting, flags.
This also reworks the regexp patterns to be more tightly focused, and not
change the same variable name in comments or elsewhere.
To make the core tools portable to platforms like Mac OS X and Windows,
remove the dependency on wget and instead use Python Requests, which
probably has better performance anyway.
make it really easy to upgrade unsigned repos to signed
As a key step to removing support for unsigned repos from fdroidclient (https://gitlab.com/fdroid/fdroidclient/issues/12), this merge request makes `fdroid update` require a signing key. If there is no keystore, it'll prompt the user to create one using `fdroid update --create-key`.
This closes#13
See merge request !48
