Commit graph

3132 commits

Author SHA1 Message Date
Hans-Christoph Steiner
09fa49a7a3
make get_android_tools_versions() search ndk_paths from config 2021-05-28 09:13:38 +02:00
Hans-Christoph Steiner
7a1d236c8d
only support zipballs in NDK provisioning
Since I discovered there is an r10e zipball, this can now get all NDKs
in zipball form.
fdroid/android-sdk-transparency-log@447fea86e7

closes #902
2021-05-28 09:13:36 +02:00
Hans-Christoph Steiner
9f77044d0d
auto-detect NDKs installed in standard paths
'ndk_paths' will be automatically filled out from well known sources
like $ANDROID_HOME/ndk-bundle and $ANDROID_HOME/ndk/*.  If a required
version is missing in the buildserver VM, it will be automatically
downloaded and installed into the standard $ANDROID_HOME/ndk/
directory.  Manually setting it here will override the auto-detected
values.  The keys can either be the "release" (e.g. r21e) or the
"revision" (e.g. 21.4.7075529).

https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
* sdkmanager installs "ndk;12.3.4567890" into $ANDROID_SDK_ROOT/ndk/
* sdkmanager installs "ndk-bundle" into $ANDROID_SDK_ROOT/ndk-bundle/
2021-05-28 09:13:34 +02:00
Hans-Christoph Steiner
4686c06f62 metadata: allow ndk: to be str or list of release or revision
There are two version numbers used for NDKs: the "release" and the
"revision".  The "release" is used in the download URL and zipball and the
"revision" is used in the source.properties and the gradle ndkVersion field.

Also, there are some builds which need multiple NDKs installed, so this
makes it possible to have a list of release/revision entries in build.ndk.
This does not yet add full support since _fdroidserver/build.py_ will also
need changes.
2021-05-28 09:13:27 +02:00
Hans-Christoph Steiner
93145a43fb buildserver: use standardized paths for the NDKs
This should help with reproducibility since some tools like Python still
include the build paths in the binaries.
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/919#note_578180986

The default ANDROID_SDK_ROOT base dir of /opt/android-sdk is hard-coded in
buildserver/Vagrantfile.  The $ANDROID_HOME/ndk subdir is where Android
Studio will install the NDK into versioned subdirs.
https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
2021-05-26 09:21:52 +02:00
Hans-Christoph Steiner
9fc2a23713
build: remove default NDK, closes #717 2021-05-25 17:06:26 +02:00
Hans-Christoph Steiner
69fcd6a024
build: auto-download missing NDKS if they're known and can be verified
refs #517 #717
2021-05-25 17:06:24 +02:00
Hans-Christoph Steiner
7a7ecbf9dc
move sha256sum() and sha256base64() to common 2021-05-25 17:06:23 +02:00
Hans-Christoph Steiner
0fbd04f1c2
remove redundant imports and pointless code 2021-05-25 17:06:19 +02:00
Jochen Sprickerhof
84b6617faf Don't return Gradle variables as string
This is used in checkupdates and currently sets the AutoName to
something like ${displayName}:

4ae415f73c
2021-05-13 12:53:19 +00:00
Jochen Sprickerhof
31c0078b53 [status_json] Add list of archivePolicy0 2021-05-13 12:30:53 +00:00
Jochen Sprickerhof
33fa918a69 [status_json] Don't list disabled builds as needs updated
Closes: #898
2021-05-13 12:30:53 +00:00
Jochen Sprickerhof
22c32752de Don't remove logs of disabled builds
We need them to debug problems.
2021-05-13 11:49:55 +00:00
Hans-Christoph Steiner
0b0bc803b7 build: --test now keeps unsigned APKs in tmp/ that fail to reproduce
Before, whenever an unsigned APK failed to reproduce, it was just deleted.
That makes debugging hard.  This makes it keep the unsigned APK, which is
written in tmp/ when using --test.

@jspricke this is related to !864
2021-04-16 09:46:23 +02:00
Hans-Christoph Steiner
6097caef91
build: fix typo in regex for Binaries: verification 2021-04-16 09:40:21 +02:00
Jochen Sprickerhof
f01a3caf77 Bump METADATA_VERSION for apksigner transition
We switched to apksigner in 50f0534d but old apks where still verified
with jarsigner (or an old apksigner version). Bumping the
METADATA_VERSION to force a rebuild of apkcache.
Hopefully this resolves de.chagemann.regexcrossword getting the
KnownVuln, DisabledAlgorithm tags.
2021-04-14 22:08:26 +00:00
Felix C. Stegerman
b08d54cb90
fix metadata_find_signing_files() 2021-04-14 23:52:13 +02:00
Hans-Christoph Steiner
5346ea3c19 gitlab-ci: prevent dualing linters: pyflakes vs mypy
* pyflakes says: "typing.Dict' imported but unused"
* mypy says: "Name 'Dict' is not defined"
2021-04-14 23:18:05 +02:00
Hans-Christoph Steiner
2946c90dd4 publish: rename vars to match naming in JAR Signature docs
https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html

closes #892
2021-04-14 23:01:47 +02:00
Hans-Christoph Steiner
416790306f common.py: update copyrights 2021-04-14 23:01:47 +02:00
Hans-Christoph Steiner
4308a48717 apksigcopier: remove exclude_all_meta logic
This option is unneeded and unused in fdroidserver, and confusing.

https://github.com/obfusk/apksigcopier/pull/36

# Conflicts:
#	fdroidserver/apksigcopier.py
#	fdroidserver/common.py
2021-04-14 23:01:47 +02:00
Hans-Christoph Steiner
f5792d4f49 apksigcopier: purge main() to avoid confusion 2021-04-14 23:01:47 +02:00
Hans-Christoph Steiner
d28c1c0441 force apksigcopier "AUTO" to select sigs by available files 2021-04-14 23:01:47 +02:00
Felix C. Stegerman
202fd8b25a
vendor & use apksigcopier v0.4.0-12-g93d8e14 2021-04-14 21:06:20 +02:00
Felix C. Stegerman
67a0f3ae5b
use subclass hack for better ZIP cloning
See https://bugs.python.org/issue43547 for more info on the details.

thanks to @obfusk for the technique
2021-04-14 15:17:56 +02:00
Felix C. Stegerman
0a4995cd3b read_pkg_args(): allow appid_vercode.apk in addition to appid:vercode 2021-04-10 07:18:28 +00:00
Felix C. Stegerman
3e557a1a8a
check for invalid appids passed to commands 2021-04-08 10:31:33 +02:00
Sylvia van Os
24dd6740e1 Use tag if known
This ensures that we actually use the tag someone created when a new tag
is detected.
2021-04-07 09:41:03 +00:00
Jochen Sprickerhof
df3b6ea65c Create log file for VCSException
In case there is a VCS error (git tag not found), create a log file in
the F-Droid repo so it is shown on the monitor (currently 404).
2021-04-06 12:45:41 +00:00
Jochen Sprickerhof
49a593ac5f Excluded disabled metadata in failedBuilds 2021-04-06 09:44:11 +00:00
Benedikt Brückmann
8f0712e89b Add a mypy run to the pipeline 2021-04-02 08:57:56 +00:00
Hans-Christoph Steiner
b69b95103e add complete tests for finding apksigner; fix minor detection bug
find_apksigner() was preferring the oldest valid version rather than the
newest.
2021-03-23 21:01:53 +01:00
Hans-Christoph Steiner
9135dfbeed verify: include diffoscope setup info in output JSON 2021-03-23 11:01:40 +01:00
Hans-Christoph Steiner
8b393189c1 include /etc/issue.net in status JSON to identify the base system 2021-03-23 11:01:40 +01:00
Hans-Christoph Steiner
7ec2a8ebb3 add apksigner, jarsigner, and keytool to relevant status JSON 2021-03-22 22:31:29 +01:00
Hans-Christoph Steiner
903a7396b1 switch to preferring apksigner, requiring for fdroid publish 2021-03-22 22:30:49 +01:00
Hans-Christoph Steiner
8a9852209b update: do not die when androguard files to extract icon
https://jenkins.debian.net/job/reproducible_fdroid_test/422
https://jenkins.debian.net/job/reproducible_fdroid_test/423

```
Traceback (most recent call last):
  File "../fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 227, in main
    raise e
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 208, in main
    mod.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2316, in main
    apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1746, in process_apks
    use_date_from_apk, ada, True)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1619, in process_apk
    apk = scan_apk(apkfile)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1385, in scan_apk
    scan_apk_androguard(apk, apk_file)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1527, in scan_apk_androguard
    resource_id = arsc.get_id(apk['packageName'], icon_id)
  File "/usr/lib/python3/dist-packages/androguard/core/bytecodes/axml/__init__.py", line 1668, in get_id
    self._analyse()
  File "/usr/lib/python3/dist-packages/androguard/core/bytecodes/axml/__init__.py", line 1321, in _analyse
    ate = self.packages[package_name][nb + 3 + nb_i]
IndexError: list index out of range

```
2021-03-22 18:16:34 +01:00
Hans-Christoph Steiner
50f0534d87 publish: always use apksigner to sign APKs, closes #880 2021-03-22 18:16:34 +01:00
Hans-Christoph Steiner
08cde5c2e6 code formatting: black --skip-string-normalization --line-length 100 2021-03-22 18:16:34 +01:00
Jochen Sprickerhof
ea9f917ddc Support versionCode and versionName in brackets
Example: versionCode(29)
https://github.com/florisboard/florisboard/blob/v0.3.9/app/build.gradle.kts#L24
2021-03-19 13:58:02 +01:00
Christian Beier
553daf8552 update: take build subdir into account when looking for triple-t data
Implemented as a fallback solution when parsing settings.gradle
and globbing the build dir with "'*', 'src', '*', 'play'" did not
yield results.

Before, the logic would not find triple-t metadata in projects where
settings.gradle is in a subdirectory or 'src' is found at a directory
depth != 2.

Closes fdroid/fdroiddata#2295
2021-03-17 13:19:44 +00:00
Hans-Christoph Steiner
dbf80ad771
metadata: remove git repo handling from metadata parsing
This is a vestige of implementing builds from a .fdroid.yml file directly
in the app's source repo.  It was never fully complete and seems to not be
used in any apps in fdroiddata.  This makes `fdroid build --all` runs much
faster since it does not need to do any git handling for apps that do not
have any new builds to run.

4e8e29794
948689281
a4e431080
e37be9b06
e775d330c
2021-03-17 13:44:24 +01:00
Hans-Christoph Steiner
78842e9cc2 scanner: should not exit with error when apkanalyzer fails
apkanalyzer produces useful output when it can run, but it does not
support all recent JDK versions, and also some DEX versions, so this
cannot count on it to always produce useful output or even to run
without exiting with an error.

211dd65ff0 was based on false
assumptions that apkanalyzer can always produce output.

fdroiddata!8585
fdroiddata!8584
2021-03-15 10:54:29 +01:00
Hans-Christoph Steiner
5695993521 build: include useful successful build IDs in status JSON, closes #825 2021-03-11 18:52:31 +01:00
Felix C. Stegerman
259ca51001 use vagrantfile instead of 'builder/Vagrantfile' in get_clean_builder() 2021-03-10 09:12:52 +00:00
Hans-Christoph Steiner
8bb2c7d4b1 metadata: stop setting up source repo when running lint/rewritemeta
broken in 54e8867a7e
2021-03-09 17:55:07 +01:00
Hans-Christoph Steiner
211dd65ff0 scanner: show error if scan_binary fails to run apkanalyzer 2021-03-09 15:55:45 +01:00
Hans-Christoph Steiner
649347dcd4 common: properly parse version from NDK's source.properties 2021-03-09 15:55:45 +01:00
Hans-Christoph Steiner
6a875a1821 update: stop extracting and storing XML icons, they're useless
APKs that only have XML icons in them should include an icon.png in the
metadata or fastlane/triple-t

closes #344
2021-03-09 15:40:30 +01:00
Hans-Christoph Steiner
850e244526 index: rename vars to match the dict key 'signer'
'sig' matches the old deprecated entry for the signer fingerprint that
uses the custom fingerprint algorithm.
2021-03-09 15:40:30 +01:00