Commit graph

3306 commits

Author SHA1 Message Date
Marcus Hoffmann
7813a17cf8 publish: extract a few functions out of main
publish is currently not reusable from other modules as everything is
happening in main. It's also not testable from python unittests.

There's already a function for getting the key_alias, so we can use
that.

Introduce tests for the split out functions.
2020-08-24 19:34:08 +02:00
Marcus Hoffmann
eaca3d5faa publish: better json reporting
* newKeyAliases wasn't providing any useful information
* generatedKeys now contains the used keyalias as well
* signedApks now also records the used keyalias for each apk
2020-08-24 19:34:08 +02:00
Marcus Hoffmann
ca86c18e33 publish: reformat 2020-08-24 19:34:08 +02:00
Hans-Christoph Steiner
0b92e60266 handle file type detection using Pythonic methods
This ditches the custom common.get_extension() for straight core Python
methods.  This should make the code closer to Python conventions.  For
example, pathlib also includes the "." in the extension it returns.
2020-08-24 17:05:49 +02:00
Hans-Christoph Steiner
da31120b5a build: refactor missing srclibs error reporting for only .yml 2020-08-24 17:05:49 +02:00
Michael Pöhn
5c3db9a7cc rewritemeta: overwrite existing metadata only if no exception occurred 2020-08-20 20:40:15 +02:00
Michael Pöhn
c45ef453fd remove SUPPORTED_FORMATS list from rewritemeta 2020-08-20 20:40:15 +02:00
Michael Pöhn
fac033314a fix a comment and a warning 2020-08-20 20:40:15 +02:00
Michael Pöhn
f5a5fffb10 purge accepted_formats from config 2020-08-20 20:40:15 +02:00
Michael Pöhn
f8bc51399e remove txt from nightly default supported metadata formats 2020-08-20 20:40:15 +02:00
Michael Pöhn
4dc503ed28 remove txt form accepted format list 2020-08-20 20:40:15 +02:00
Michael Pöhn
fd2cfb0c7d remove code for copying txt srclibs to buildserver 2020-08-20 20:40:15 +02:00
Michael Pöhn
681a27546c fix linking to yml metadata in wiki 2020-08-20 20:40:15 +02:00
Michael Pöhn
cb368a674c remove txt srclib support 2020-08-20 20:40:15 +02:00
Michael Pöhn
c8f25c2652 remove txt metadata support 2020-08-20 20:40:15 +02:00
Michael Pöhn
2ec90bb490 remove support for rewriting to txt 2020-08-20 20:40:15 +02:00
Michael Pöhn
bc1398f594 remove json metadata support 2020-08-20 20:40:15 +02:00
Marcus Hoffmann
ba28b44ae7 common: use androguard to figure out minSDK version
Closes: #816
2020-08-19 19:49:39 +02:00
Marcus Hoffmann
a8e9653b96 update: make --create-key work with a HSM 2020-08-14 17:27:58 +02:00
Marcus Hoffmann
004d13a48a make publish and update work with a smartcard HSM
Followup to fdroid/fdroidserver!779.

We need to add smartcardoptions to every call to keytool and jarsigner
as well as handle when keypass not being required and not allowed for
pkcs11 keystores.
2020-08-14 17:27:58 +02:00
Marcus Hoffmann
066978cbcf publish: use common signing method
This is currently still jarsigner based but will at least use sha256
when possible
2020-08-14 15:07:04 +02:00
Marcus Hoffmann
6128f93d26 publish: keystore "NONE" is a special case and doesn't need to exist 2020-08-14 15:06:33 +02:00
Marcus
11236deec6 Merge branch 'patch-1' into 'master'
Spelling: Deploy key, stripping, SSH

See merge request fdroid/fdroidserver!781
2020-08-13 15:03:42 +00:00
Allan Nordhøy
df7bc77465 Spelling: Deploy key, stripping, SSH 2020-08-10 10:53:23 +00:00
Hans-Christoph Steiner
f779ce276a 'keypass' is not required in config if using a HSM 2020-08-07 15:30:33 +02:00
Hans-Christoph Steiner
d213c8b37c update: validate smartcardoptions when using a HSM for the keystore 2020-08-07 15:30:33 +02:00
Hans-Christoph Steiner
7dcf4f5680 index: smartcardoptions is a list or tuple, but only lists can be added 2020-08-07 15:30:33 +02:00
Hans-Christoph Steiner
54257f8f6a init: generate opensc-fdroid.cfg rather than copying from examples
This file is so simple, it is better to skip the complexity of trying to
find the installed examples folder, especially when considering Windows and
macOS, with their odd paths.
2020-08-07 15:30:33 +02:00
Hans-Christoph Steiner
a6c60a908a import: add support for kivy buildozer projects 2020-08-07 12:48:15 +02:00
Hans-Christoph Steiner
960d31af2a import: standardize on 'appid' var name for Application ID
The rest of the fdroidserver code uses 'appid' or 'packageName'.  The official
Android name is "Application ID".  "Package Name" is the Java term, and it
is used in Android in Java code.
2020-07-30 14:57:19 +02:00
Hans-Christoph Steiner
10fa912c16 deploy: remove git remotes not enabled in servergitmirrors
If the repo operator removes an item from servergitmirrors, it should auto-
matically remove the remote.
2020-07-30 14:57:19 +02:00
Jochen Sprickerhof
d8b73dd241 Add timstamp to logs 2020-07-04 18:56:10 +02:00
Marcus
1902bfb6fc Merge branch 'fix_760' into 'master'
update.py (and index.py) refactoring

Closes #524 and #760

See merge request fdroid/fdroidserver!756
2020-06-24 22:37:51 +00:00
Marcus Hoffmann
07caa88920 don't include the localized metadata things for /archive
We haven't done this so far and it's a potential big change in archive
size and update performance.
2020-06-25 00:07:42 +02:00
Marcus Hoffmann
d720c99ae5 refactor update.py:main
This makes update.py:main a lot saner by removing a lot of the
implicit assumptions between the different stages of generating the
repository index.

* mostly unify repo and archive processing, that means the archive is
  now actually getting the same treatment regarding i.e. fastlane data.
  Previously the archive didn't get considered at all here.
* already filter the list of apps to include in a repo in update.py and
  give that prefiltered list to index. This makes sure we actually only
  copy fastlane/triple-t/etc. stuff for apps ending up in the index.
  This both, can save a lot of time if there are a lot of old /build
  dirs lying around and doesn't clutter /repo with things that aren't
  referenced from the index.

Closes fdroid/fdroidserver#524
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
e66683720b move index sorting to index module
This is a historic detail of the index format, so move it there.
For wiki update and status json there's really no reason why this should
be done in alphabetic app name order. Use the default sort order by
appid.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
8c71637d43 update: make copy_triple_t_store_metadata and insert_localized_app_metadata not assume /repo
This will enable copying the localized metadata to the archive as well.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
ee4ee85cbd update:archive_old_apks: handle apps with no CVC
If an app doesn't have a CVC, we can just skip any special archive
handling.
Also rename weirdly named `res` to `apkList`.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
03881154c6 metadata: make linkresolver an actual object
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.

Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
2020-06-24 23:53:35 +02:00
Hans-Christoph Steiner
5fce16c847
remove unused, confusing arg to common.getsrclib() 2020-06-24 22:40:27 +02:00
Hans-Christoph Steiner
d0f426e076
replace $$srclib$$ with an absolute path
closes #725
2020-06-24 22:40:26 +02:00
Hans-Christoph Steiner
1e6f99988a fix typo when looking for build.gradle.kts 2020-06-24 21:43:58 +02:00
Hans-Christoph Steiner
238f048257 update: fix crash when liberapay: or open_collective: not in FUNDING.yml
closes #799
2020-06-24 21:39:43 +02:00
Hans-Christoph Steiner
62c8fd5999
add Liberapay: field with username as data
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs.  Now it is a username.  So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration.  LiberapayID: will not override
Liberapay:.  Clients are expected to prefer Liberapay: over LiberapayID:
2020-06-16 15:35:28 +02:00
Hans-Christoph Steiner
0183592526
update: insert donation links based on FUNDING.yml
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links.  Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:.  Specifying those in the metadata file takes
precedence over the FUNDING.yml.  This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata.  This lets the
git repo maintain those specific donations links themselves.

https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files

The test file was generated using:

```python
import os, re, yaml

found = dict()
for root, dirs, files in os.walk('.'):
    for f in files:
        if f == 'FUNDING.yml':
            with open(os.path.join(root, f)) as fp:
                data = yaml.safe_load(fp)
            for k, v in data.items():
                if k not in found:
                    found[k] = set()
                if not v:
                    continue
                if isinstance(v, list):
                    for i in v:
                        found[k].add(i)
                else:
                    found[k].add(v)

            with open('gather-funding-names.yaml', 'w') as fp:
                output = dict()
                for k, v in found.items():
                    output[k] = sorted(v)
                yaml.dump(output, fp, default_flow_style=False)
```
2020-06-16 15:35:26 +02:00
Marcus
f84818c15e scanner: add a simple scan for blacklisted classes after build step
add com.android.billing to blacklist, see
https://gitlab.com/fdroid/fdroiddata/-/issues/2070#note_360611289
2020-06-15 18:03:19 +00:00
Marcus
6986e73506 Merge branch 'allow_appcenter' into 'master'
scanner: allow microsoft appcenter

See merge request fdroid/fdroidserver!764
2020-06-15 10:46:51 +00:00
Marcus Hoffmann
670328c9e5 scanner: allow microsoft appcenter, except appcenter-push
This sdk is open source. It was added because appcenter has a dependency
to play-services. It's possible though to build an app using appcenter
that doesn't pull in play services, so we can't blanket ban the sdk.

The appcenter-push modules has obvious refenrences to firebase, so it's
safe to error on that.

Ref: https://phabricator.wikimedia.org/T254980
2020-06-15 12:24:46 +02:00
Marcus
12f4ae3039 Merge branch 'fix_791' into 'master'
scanner: fix wrong path being passed to function

Closes #791

See merge request fdroid/fdroidserver!767
2020-06-13 11:03:30 +00:00
Marcus Hoffmann
3a9f1e845c scanner: fix wrong path being passed to function
Also make rename the parameter in safe_path to make it clear that this
is just a relative path.

Closes fdroid/fdroidserver#791.
2020-06-13 03:01:54 +02:00