Commit graph

3118 commits

Author SHA1 Message Date
Marcus Hoffmann
145ba9db54 fix apksigner smartcardoptions
apksigner documents the options as --ks-provider-class and --ks-provider-arg
those seem to be accepted but fail when actually making a signature with
weird internal exceptions. The new options actually work.
From: https://geoffreymetais.github.io/code/key-signing/#scripting
2020-09-09 22:51:08 +02:00
Marcus Hoffmann
74af61f255 remove providerName from default smartcardoptions
apksigner doesn't recognize the SunPKCS11-OpenSC set via providerName
Neither jarsigner nor apksigner need this to work.
2020-09-09 22:50:58 +02:00
Marcus Hoffmann
768a91370c publish: use apksigner for signing apks with targetSDK>=30
This makes apksigner a hard requirement of the signing procedure.
We'll first try to find a globally installed version from PATH and if
that's not available fall back to using a version from build-tools.

Future TODO: always sign with apksigner, blocked on signature transplant
support for apksigv2/v3

Closes fdroid/fdroidserver#634
Closes fdroid/fdroidserver#827
2020-09-09 18:09:28 +02:00
Marcus Hoffmann
c90a72e14d scanner: check for test in path relative to build dir
This was degrading scanner errors to warnings whenever the path from the
current running install of fdroidserver contained test, as has been
happening in the CI image builder:

https://gitlab.com/fdroid/ci-images-server/-/blob/master/test#L6
2020-08-30 20:09:19 +02:00
Marcus Hoffmann
34717fe88a fix typoes 2020-08-25 21:32:54 +02:00
Marcus Hoffmann
d07b4123e9 scanner: docstrings for handleproblem functions 2020-08-25 21:32:29 +02:00
Marcus Hoffmann
a656be82ae update: calculate added date for an app over all apks
This was accidentally changed in !756 because the functionality was
hidden in `apply_info_from_latest_apk` which is a less than stellar name
for something that also applies infos from app->apk and in this case did
apply info from *oldest* apk->app.
So instead move that into a separate step.

Note: This restores the previous behaviour. There's discussion in #801
on further changes to make the added date also work for repos which
don't keep an archive at all.
2020-08-25 21:32:29 +02:00
Marcus Hoffmann
d5311fff09 vcs: don't fail when git remote set-head fails
There's valid use-cases for setups where set-head --auto fails. This
happens when building an app from a gitlab CI checkout where no remote
tracking branches are setup.

This isn't really a fatal error. When a remote HEAD exists we'll
continue setting it and if none exists and something requires this being
set up later on (either a build script or fdroid checkupdates) then
we'll fail later on with "origin/HEAD not being known to git".
By not failing early we allow the majority of use-cases that don't need
a remote HEAD to continue with just a warning.

The setup in which this can be reproduced is as follows:
(This is roughly what gitlab runner does when setting up a git checkout
for CI)
- mkdir test && cd test
- git init
- git remote add https://gitlab.com/Bubu/fdroidclassic.git
- git fetch --all
- git checkout db0d2a9a5d1d89101a344169013ac5d518185f31
- mkdir nested_repo && cd nested_repo
- git clone .. .
- git remote set-head origin --auto
  > error: Cannot determine remote HEAD
2020-08-25 21:32:29 +02:00
Marcus Hoffmann
03e723b1af fix crash when scanner wants to remove the same file more than once
A file can be flagged for multiple problems (i.e. multiple unknown maven
repos in one build.gradle file that is included in a scandelete path).

The scanner will try to delete it once for every problem detected, we
don't really care, as long as the file is gone.

fixes fdroid/fdroidserver#759
2020-08-25 21:32:29 +02:00
Marcus Hoffmann
78491a0a5b add used extlibs to scanignore path
fixes fdroid/fdroidserver#795
2020-08-25 21:32:29 +02:00
Marcus
8801d37649 Merge branch 'publish_json' into 'master'
Some publish.py improvements

Closes #820

See merge request fdroid/fdroidserver!787
2020-08-25 12:33:39 +00:00
Marcus Hoffmann
a114c73c2d publish: factor out the signing key creation into a method 2020-08-24 19:34:08 +02:00
Marcus Hoffmann
7813a17cf8 publish: extract a few functions out of main
publish is currently not reusable from other modules as everything is
happening in main. It's also not testable from python unittests.

There's already a function for getting the key_alias, so we can use
that.

Introduce tests for the split out functions.
2020-08-24 19:34:08 +02:00
Marcus Hoffmann
eaca3d5faa publish: better json reporting
* newKeyAliases wasn't providing any useful information
* generatedKeys now contains the used keyalias as well
* signedApks now also records the used keyalias for each apk
2020-08-24 19:34:08 +02:00
Marcus Hoffmann
ca86c18e33 publish: reformat 2020-08-24 19:34:08 +02:00
Hans-Christoph Steiner
0b92e60266 handle file type detection using Pythonic methods
This ditches the custom common.get_extension() for straight core Python
methods.  This should make the code closer to Python conventions.  For
example, pathlib also includes the "." in the extension it returns.
2020-08-24 17:05:49 +02:00
Hans-Christoph Steiner
da31120b5a build: refactor missing srclibs error reporting for only .yml 2020-08-24 17:05:49 +02:00
Michael Pöhn
5c3db9a7cc rewritemeta: overwrite existing metadata only if no exception occurred 2020-08-20 20:40:15 +02:00
Michael Pöhn
c45ef453fd remove SUPPORTED_FORMATS list from rewritemeta 2020-08-20 20:40:15 +02:00
Michael Pöhn
fac033314a fix a comment and a warning 2020-08-20 20:40:15 +02:00
Michael Pöhn
f5a5fffb10 purge accepted_formats from config 2020-08-20 20:40:15 +02:00
Michael Pöhn
f8bc51399e remove txt from nightly default supported metadata formats 2020-08-20 20:40:15 +02:00
Michael Pöhn
4dc503ed28 remove txt form accepted format list 2020-08-20 20:40:15 +02:00
Michael Pöhn
fd2cfb0c7d remove code for copying txt srclibs to buildserver 2020-08-20 20:40:15 +02:00
Michael Pöhn
681a27546c fix linking to yml metadata in wiki 2020-08-20 20:40:15 +02:00
Michael Pöhn
cb368a674c remove txt srclib support 2020-08-20 20:40:15 +02:00
Michael Pöhn
c8f25c2652 remove txt metadata support 2020-08-20 20:40:15 +02:00
Michael Pöhn
2ec90bb490 remove support for rewriting to txt 2020-08-20 20:40:15 +02:00
Michael Pöhn
bc1398f594 remove json metadata support 2020-08-20 20:40:15 +02:00
Marcus Hoffmann
ba28b44ae7 common: use androguard to figure out minSDK version
Closes: #816
2020-08-19 19:49:39 +02:00
Marcus Hoffmann
a8e9653b96 update: make --create-key work with a HSM 2020-08-14 17:27:58 +02:00
Marcus Hoffmann
004d13a48a make publish and update work with a smartcard HSM
Followup to fdroid/fdroidserver!779.

We need to add smartcardoptions to every call to keytool and jarsigner
as well as handle when keypass not being required and not allowed for
pkcs11 keystores.
2020-08-14 17:27:58 +02:00
Marcus Hoffmann
066978cbcf publish: use common signing method
This is currently still jarsigner based but will at least use sha256
when possible
2020-08-14 15:07:04 +02:00
Marcus Hoffmann
6128f93d26 publish: keystore "NONE" is a special case and doesn't need to exist 2020-08-14 15:06:33 +02:00
Marcus
11236deec6 Merge branch 'patch-1' into 'master'
Spelling: Deploy key, stripping, SSH

See merge request fdroid/fdroidserver!781
2020-08-13 15:03:42 +00:00
Allan Nordhøy
df7bc77465 Spelling: Deploy key, stripping, SSH 2020-08-10 10:53:23 +00:00
Hans-Christoph Steiner
f779ce276a 'keypass' is not required in config if using a HSM 2020-08-07 15:30:33 +02:00
Hans-Christoph Steiner
d213c8b37c update: validate smartcardoptions when using a HSM for the keystore 2020-08-07 15:30:33 +02:00
Hans-Christoph Steiner
7dcf4f5680 index: smartcardoptions is a list or tuple, but only lists can be added 2020-08-07 15:30:33 +02:00
Hans-Christoph Steiner
54257f8f6a init: generate opensc-fdroid.cfg rather than copying from examples
This file is so simple, it is better to skip the complexity of trying to
find the installed examples folder, especially when considering Windows and
macOS, with their odd paths.
2020-08-07 15:30:33 +02:00
Hans-Christoph Steiner
a6c60a908a import: add support for kivy buildozer projects 2020-08-07 12:48:15 +02:00
Hans-Christoph Steiner
960d31af2a import: standardize on 'appid' var name for Application ID
The rest of the fdroidserver code uses 'appid' or 'packageName'.  The official
Android name is "Application ID".  "Package Name" is the Java term, and it
is used in Android in Java code.
2020-07-30 14:57:19 +02:00
Hans-Christoph Steiner
10fa912c16 deploy: remove git remotes not enabled in servergitmirrors
If the repo operator removes an item from servergitmirrors, it should auto-
matically remove the remote.
2020-07-30 14:57:19 +02:00
Jochen Sprickerhof
d8b73dd241 Add timstamp to logs 2020-07-04 18:56:10 +02:00
Marcus
1902bfb6fc Merge branch 'fix_760' into 'master'
update.py (and index.py) refactoring

Closes #524 and #760

See merge request fdroid/fdroidserver!756
2020-06-24 22:37:51 +00:00
Marcus Hoffmann
07caa88920 don't include the localized metadata things for /archive
We haven't done this so far and it's a potential big change in archive
size and update performance.
2020-06-25 00:07:42 +02:00
Marcus Hoffmann
d720c99ae5 refactor update.py:main
This makes update.py:main a lot saner by removing a lot of the
implicit assumptions between the different stages of generating the
repository index.

* mostly unify repo and archive processing, that means the archive is
  now actually getting the same treatment regarding i.e. fastlane data.
  Previously the archive didn't get considered at all here.
* already filter the list of apps to include in a repo in update.py and
  give that prefiltered list to index. This makes sure we actually only
  copy fastlane/triple-t/etc. stuff for apps ending up in the index.
  This both, can save a lot of time if there are a lot of old /build
  dirs lying around and doesn't clutter /repo with things that aren't
  referenced from the index.

Closes fdroid/fdroidserver#524
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
e66683720b move index sorting to index module
This is a historic detail of the index format, so move it there.
For wiki update and status json there's really no reason why this should
be done in alphabetic app name order. Use the default sort order by
appid.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
8c71637d43 update: make copy_triple_t_store_metadata and insert_localized_app_metadata not assume /repo
This will enable copying the localized metadata to the archive as well.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
ee4ee85cbd update:archive_old_apks: handle apps with no CVC
If an app doesn't have a CVC, we can just skip any special archive
handling.
Also rename weirdly named `res` to `apkList`.
2020-06-24 23:53:36 +02:00