Commit graph

1053 commits

Author SHA1 Message Date
Michael Pöhn
2c87b5e6f9 deploy build logs: no timestamps 2019-07-06 16:48:56 +02:00
Michael Pöhn
f30983368c build: rsync buildlogs to <webroot>/repo 2019-07-06 16:48:56 +02:00
relan
781b55f777 checkupdates: match whole words
Match only whole words when looking for versionCode, versionName, etc. in
manifests. A real build.gradle example:

    flutterVersionCode = '1'
    flutterVersionName = '1.0'
    ...
    defaultConfig {
        versionCode 53
        versionName "2.0.3"
    }

Before this change checkupdates was erroneously getting version code and
version name from the first two lines and failing to find a new release.
2019-06-13 19:51:44 +03:00
Hans-Christoph Steiner
17dc231dc9 update: fix running without androguard
Soon, we can rip out all the aapt parsing stuff, but not yet!
2019-02-03 15:50:39 +01:00
Hans-Christoph Steiner
2907aa3ea3 fallback to v3 signatures if v1 and v2 are not available 2019-02-01 09:18:00 +01:00
Hans-Christoph Steiner
d96f5ff660 support APK Signature V2 when apksigner is installed
This was done with much help from @uniqx.  This is the first level of
supporting APK Signatures v1, v2, and v3.  This is enough to include
APKs with any combo of v1/v2/v3 signatures.  For this to work at all,
apksigner and androguard 3.3.3+ must be installed.

closes #399
2019-02-01 09:17:56 +01:00
Hans-Christoph Steiner
ea84014f9b make signer_fingerprint() accept DER encoded certs
androguard's v2 signature methods return DER-encoded certificates.  The DER-
encoded certificates are parsed from JAR Signature's raw "Signature Block
File".
2019-02-01 08:50:57 +01:00
Hans-Christoph Steiner
1f346b3149 force_exit() to make exit work with all fdroid build conditions
The build command has to use some threading stuff to handle the timeout and
locks.  This seems to prevent the command from exiting, unless this hack is
used.
2018-11-14 14:27:32 +01:00
Hans-Christoph Steiner
57556aceee remove redundant open() arg: encoding='utf8'
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
2018-10-19 15:01:34 +02:00
Hans-Christoph Steiner
d2216b756b fix typos in strings 2018-09-24 17:09:15 +02:00
Hans-Christoph Steiner
6b57cb6b7c fix strict Application ID checks
* upper case letters are allowed at all positions
* there must be a "." separator
2018-09-24 17:07:27 +02:00
Hans-Christoph Steiner
11d46072ab use androguard primitives to speed up finding debuggable flag
androguard parses the whole APK before handing the instance back, this uses
the primitives to just find the <application android:debuggable=""> value,
then stop parsing.

#557
2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
a3cecc16a3 use partial androguard binary XML parsing to speed up APK ID lookup
Normally, androguard parses the entire APK before it is possible to get any
values from it.  This uses androguard primitives to only attempt to parse
the AndroidManifest.xml, then to quit as soon as it gets what it needs.
This greatly speeds up the parsing (1 minute vs 60 minutes).

fdroid/fdroidserver#557
2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
fa09337b4b APK_ID_TRIPLET_REGEX only matches first line of aapt output
Stop expensive aapt parsing after the first line when looking with
APK_ID_TRIPLET_REGEX.  As is seen with the `aapt dump badging` output files
in tests/build-tools/, the first line is the only line that will ever match.

#557
2018-09-21 14:56:32 +02:00
Hans-Christoph Steiner
807bf3d26b build: reuse common methods for getting metadata from APKs
This splits out the code that gets the list of native ABIs supported, then
uses the standard methods for the rest.
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
487c4d02f3 handle package: line output from aapt v28
fdroid/fdroiddata!3484
fdroid/fdroiddata!3562
fdroid/fdroidserver!548
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
9d12b1dc61 add strict, tested validation of Android/F-Droid package names
Android has stricter rules than Java for Package Names, but anything the
Python regex thinks is valid must be valid according to Java's rules too.

https://developer.android.com/studio/build/application-id
2018-09-07 14:17:39 +02:00
Hans-Christoph Steiner
9bccb2c73e temp fallback to built-in ElementTree if defusedxml is not there 2018-09-03 23:02:31 +02:00
Hans-Christoph Steiner
11b3e5be3a update: throw exception for APKs with invalid Application ID
Android Application IDs must be valid Java Package Names.  While the build
tools likely validate the Application ID, it is possible to manually create
a malicious APK.
2018-09-03 22:56:08 +02:00
Hans-Christoph Steiner
5d161cc9fd validate appid when reading metadata files
The metadata file must be named after the Application ID of the app it is
describing, and Android Application IDs must be valid Java Package Names.
2018-09-03 22:56:08 +02:00
Hans-Christoph Steiner
3ffe2860f3 gitlab-ci: add 'bandit' security scanner to all runs
bandit is used by Radically Open Security and is part of the GitLab Ultimate
Static Application Security Testing (SAST) suite.

https://docs.gitlab.com/ee/user/project/merge_requests/sast.html
2018-08-29 17:48:06 +02:00
Hans-Christoph Steiner
4d13a904f3 use defusedxml to avoid DoS attacks while loading XML 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
cc94ebca30 use global constant for the 'xmlns:android' XML namespace 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
a089614225 checkupdates: remove magic number 99999999 from HTTP checks
It is vestigal from old code and no longer is needed.
2018-08-29 17:24:24 +02:00
Hans-Christoph Steiner
4503e7a92a replace unneeded eval() call and support negative versionCodes 2018-08-29 17:24:04 +02:00
Michael Pöhn
4dcfa95d6e fix: jarsigner and keytool lookup regression 2018-08-21 03:05:43 +02:00
Cyril Russo
81641b4628 Fixed precommit checks 2018-08-20 11:17:36 +02:00
Cyril Russo
4303b0fac1 Apply suggestion from @uniqx to using shutil.which instead of iterating path by hand 2018-08-17 20:14:54 +02:00
Cyril Russo
ace33bcfc0 Reverted the change in the default config.py
Improved the detection of keytool and jarsigner by also searching the PATH environment variable
2018-08-17 16:46:46 +02:00
Cyril Russo
4f21045cad Fix keytool not found on MacOSX (when using Java from Apple)
Fix keytool parsing error due to localisation keytool -list command.
Always fallback to english so the parsing makes sense.
2018-08-17 15:32:29 +02:00
Michael Pöhn
f53d1abcce fix: make sure ensure_final_value always has a clearly defined return value 2018-08-14 11:07:44 +02:00
Michael Pöhn
74ae6a6a3b set locale explicitly when using keytool 2018-08-03 04:36:00 +02:00
Hans-Christoph Steiner
4352aa2792 update: pass resourceIds through if they can't be deferenced
People do odd tricks with `aapt --rename-manifest-package` and androguard
doesn't handle them all.

* https://github.com/androguard/androguard/issues/503
2018-07-17 12:37:13 +02:00
Marcus Hoffmann
9889a98dea
build: improve gradle experience
This expands the gradle wrapper shell script used by the buildserver for
usage outside the buildserver environment. It also allows downloading
whitelisted versions of gradle if they are not yet deployed to the
buildserver by simply upsating the copy of fdroidserver (in contrast to
having to reprovision the whole buildserver).

We first move the buildserver/gradle shell script to the repo root
as gradlew-fdroid, as it's an fdroid specific gradle wrapper.
We also now sync it inside the build VM before each build.

We then add a list of whitelisted gradle distributions taken from the
makebuildserver script.

The script additionally now reads two env vars which tell it where to
expect installed versions of gradle and where it might store downloaded
gradle .zip files. Both of those are configurable from config.py. As the
first should normally just be a subdir of the second it's not exposed in
the example config.py but only used by the buildserver config.py.

Default config now uses this internal gradle wrapper but a path to a
custom wrapper or specific gradle distribution can still be set from
config.py.

Closes fdroid/fdroidserver#98
Ref: fdroid/fdroidserver#370
2018-07-11 11:49:46 +02:00
Hans-Christoph Steiner
d20e8613fe build: only test for all Android SDK paths where it is needed
Now that androguard can be used, the android-sdk is no longer required for
most operations.
2018-06-25 10:52:12 +02:00
Hans-Christoph Steiner
c3b361c91b remove 'from distutils.util import strtobool' dependency
Most of distutils is being removed from the Python standard library, and
this is totally not needed here.
2018-06-25 10:52:12 +02:00
Hans-Christoph Steiner
8f2ee4bd1d run 'zipalign' using standard flags used by Gradle Android Plugin
Nice find by @equeim!  -p was added in build-tools-23.0.0

https://developer.android.com/studio/publish/app-signing#sign-manually

closes #288
2018-06-19 15:40:48 +02:00
Michael Pöhn
752f6a4ba2 applied review suggestions for deploying (build) logs 2018-06-19 15:18:13 +02:00
Michael Pöhn
88e64df3ef deploying build logs to server after each individual build run 2018-06-19 12:24:52 +02:00
Michael Pöhn
4c53c71fcf added helper function for uploading build logs with rsync 2018-06-19 12:24:52 +02:00
Hans-Christoph Steiner
ff90c0246e fix PEP8 W605 invalid escape sequence
Python 3.7 will get a lot stricter with escape sequences.  They must be
valid.

* https://lintlyci.github.io/Flake8Rules/rules/W605.html
* https://docs.python.org/3/whatsnew/3.6.html#deprecated-python-behavior
2018-05-29 13:51:47 +02:00
Hans-Christoph Steiner
e6d5260c3c fix PEP8: W504 line break after binary operator 2018-05-29 12:31:56 +02:00
Hans-Christoph Steiner
b0a5ec5c1a workaround crash in diffoscope when verifying APKs
On versions of diffoscope before 87, like the version included in Ubuntu
xenial LTS, it would crash saying:

ValueError: max_diff_block_lines (100) cannot be smaller than max_page_diff_block_lines (128)

https://bugs.debian.org/875451
2018-05-25 11:48:01 +02:00
Hans-Christoph Steiner
cef73e5794 remove unused variable, as reported by pyflakes 2018-05-25 10:32:01 +02:00
Torsten Grote
c6f3aed003
Fix reproducible builds
This commit fixes two bugs with reproducible builds:
* Files added by the buildserver to META-INF (fdroidserverid and buildserverid)
  were causing signature verification to always fail when --on-server was used.
  Since they are not needed anymore, they are no longer added to APKs.
* When showing a diff between both APK files, `jar xf` did not extract
  the full APK properly which was causing useless diffs.
  Instead of using jar, python's zipfile library is used instead.
2018-05-24 16:09:10 -03:00
Michael Pöhn
6b1f242d25 added tests for common.calculate_math_string 2018-05-22 13:24:37 +02:00
Michael Pöhn
24b20d7668 use simple ast+operator based calculator for evaluating Vercode Operation 2018-05-22 13:08:14 +02:00
Hans-Christoph Steiner
040b8d8d5d prefer latest version of Java, its no longer necessary to prefer 7
closes #474
2018-05-04 21:38:06 +02:00
Hans-Christoph Steiner
7eb622930e update Java path searches to also match v10, v11, etc.
This is still pretty conversative, since its not easy to test on all those
platforms.

#474
2018-05-04 21:37:41 +02:00
J-Jamet
d6f1de2649 Fix applicationIdSuffix / versionNameSuffix #455 2018-05-04 12:24:16 +02:00