Commit graph

3118 commits

Author SHA1 Message Date
Marcus Hoffmann
03881154c6 metadata: make linkresolver an actual object
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.

Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
2020-06-24 23:53:35 +02:00
Hans-Christoph Steiner
5fce16c847
remove unused, confusing arg to common.getsrclib() 2020-06-24 22:40:27 +02:00
Hans-Christoph Steiner
d0f426e076
replace $$srclib$$ with an absolute path
closes #725
2020-06-24 22:40:26 +02:00
Hans-Christoph Steiner
1e6f99988a fix typo when looking for build.gradle.kts 2020-06-24 21:43:58 +02:00
Hans-Christoph Steiner
238f048257 update: fix crash when liberapay: or open_collective: not in FUNDING.yml
closes #799
2020-06-24 21:39:43 +02:00
Hans-Christoph Steiner
62c8fd5999
add Liberapay: field with username as data
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs.  Now it is a username.  So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration.  LiberapayID: will not override
Liberapay:.  Clients are expected to prefer Liberapay: over LiberapayID:
2020-06-16 15:35:28 +02:00
Hans-Christoph Steiner
0183592526
update: insert donation links based on FUNDING.yml
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links.  Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:.  Specifying those in the metadata file takes
precedence over the FUNDING.yml.  This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata.  This lets the
git repo maintain those specific donations links themselves.

https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files

The test file was generated using:

```python
import os, re, yaml

found = dict()
for root, dirs, files in os.walk('.'):
    for f in files:
        if f == 'FUNDING.yml':
            with open(os.path.join(root, f)) as fp:
                data = yaml.safe_load(fp)
            for k, v in data.items():
                if k not in found:
                    found[k] = set()
                if not v:
                    continue
                if isinstance(v, list):
                    for i in v:
                        found[k].add(i)
                else:
                    found[k].add(v)

            with open('gather-funding-names.yaml', 'w') as fp:
                output = dict()
                for k, v in found.items():
                    output[k] = sorted(v)
                yaml.dump(output, fp, default_flow_style=False)
```
2020-06-16 15:35:26 +02:00
Marcus
f84818c15e scanner: add a simple scan for blacklisted classes after build step
add com.android.billing to blacklist, see
https://gitlab.com/fdroid/fdroiddata/-/issues/2070#note_360611289
2020-06-15 18:03:19 +00:00
Marcus
6986e73506 Merge branch 'allow_appcenter' into 'master'
scanner: allow microsoft appcenter

See merge request fdroid/fdroidserver!764
2020-06-15 10:46:51 +00:00
Marcus Hoffmann
670328c9e5 scanner: allow microsoft appcenter, except appcenter-push
This sdk is open source. It was added because appcenter has a dependency
to play-services. It's possible though to build an app using appcenter
that doesn't pull in play services, so we can't blanket ban the sdk.

The appcenter-push modules has obvious refenrences to firebase, so it's
safe to error on that.

Ref: https://phabricator.wikimedia.org/T254980
2020-06-15 12:24:46 +02:00
Marcus
12f4ae3039 Merge branch 'fix_791' into 'master'
scanner: fix wrong path being passed to function

Closes #791

See merge request fdroid/fdroidserver!767
2020-06-13 11:03:30 +00:00
Marcus Hoffmann
3a9f1e845c scanner: fix wrong path being passed to function
Also make rename the parameter in safe_path to make it clear that this
is just a relative path.

Closes fdroid/fdroidserver#791.
2020-06-13 03:01:54 +02:00
Licaon_Kter
72fde0f10b Detect .gitmodules and add submodules build line 2020-06-11 16:36:54 +03:00
Marcus Hoffmann
294799e2f5 scanner: fix options handling
closes fdroid/fdroidserver#789
2020-06-11 11:44:00 +02:00
Hans-Christoph Steiner
0837289935 scanner: fix regex for matching URLs in gradle maven{} blocks
closes #465

This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml

pat = re.compile(r'\smaven\s*{[^}]+}')

finds = set()
for root, dirs, files in os.walk('.'):
    for f in files:
        if '.gradle' in f:
            with open(os.path.join(root, f), errors='surrogateescape') as fp:
                contents = fp.read()
            for m in pat.findall(contents):
                finds.add(m)

with open('finds.yaml', 'w') as fp:
    yaml.dump(sorted(finds), fp, default_flow_style=False)

```
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
6590f3869e scanner: error/warn on dex/gz/zip, closes #394 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
d7b3bca1e7 build: pass --verbose flag through to scanner 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
ee54dbce87 scanner: safely check options, for things using this API
https://gitlab.com/fdroid/fdroidserver/-/issues/771#note_353495799
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
75acf63efa scanner: remove all gradle wrapper files 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
2f5d780c49 scanner: always setup JSON data structure so functions work as API 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
84f225f2f2 scanner: make problem descriptions translationable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
d898ad0412 scanner: make AARs and JARs trigger an error
refs #491
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
3bc246ccad scanner: ignore well known image types that are set executable 2020-06-10 19:42:23 +02:00
Michael Pöhn
a97b3ca4dd implement plugin system review suggestsions 2020-06-10 18:43:23 +02:00
Michael Pöhn
77167e098e plugin system: regex instead of import bases plugin parsing 2020-06-10 18:43:11 +02:00
Michael Pöhn
b257a3411a stick to default python sys.path 2020-06-10 12:11:47 +02:00
Michael Pöhn
3a3803ea2d raise excepten when starting broken plugin 2020-06-10 12:11:47 +02:00
Michael Pöhn
bf815251ec rough plugin system implementation 2020-06-10 12:11:47 +02:00
Marcus Hoffmann
581e433832 strip_and_copy_image: abort on broken symlinks
Also guard against other processes removing the files we are about to
copy.

closes fdroid/fdroidserver#783
2020-06-04 12:58:11 +02:00
Hans-Christoph Steiner
13d9a122bf metadata: validate STRING and INT build field types
This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
2020-06-02 22:06:01 +02:00
Hans-Christoph Steiner
b2daf96284 lint: fix pyflakes 2020-06-02 22:05:18 +02:00
Hans-Christoph Steiner
6c5887e906
lint: make --force-yamllint error if yamllint is not installed
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/753#note_353829401
2020-06-02 21:56:05 +02:00
Hans-Christoph Steiner
6030445be0 logging.warn() was deprecated in Python 3.3, use logging.warning()
sed -i 's,logging\.warn(,logging.warning(,g' fdroid */*.*

https://docs.python.org/3.3/library/logging.html#logging.Logger.warning
2020-05-27 08:35:14 +02:00
Hans-Christoph Steiner
67332d83a5 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot

This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes #771.  It was previously merged in !748 then
reverted in 68c072c72e
2020-05-27 08:35:14 +02:00
relan
9ff77cfd1a build: clean buildSrc/build
The buildSrc directory contains custom build logic written in Kotlin.
Before this change we had to 'scandelete' buildSrc/build in the build
recipes becase 'gradle clean' leaves binary artifacts there.
2020-05-26 07:37:20 +00:00
relan
ddfbd1cc47 build: fix directories removal
The 'dirs' array contains a single-level listing of a directory, e. g.
['app', 'build', 'build.gradle', 'gradle', '.gradle']. Multi-component
paths like 'build/tmp' could never be found in this array and thus were
never removed.

Call shutil.rmtree() without checking that the argument is in 'dirs'. If
it exists and is a directory, it'll be removed. Otherwise shutil.rmtree()
will do nothing.
2020-05-26 07:37:20 +00:00
Marcus Hoffmann
68c072c72e Revert "scanner: add --json option for outputting machine readable results"
This reverts commit cdaf62e5d9.

See: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/748#note_347769371
2020-05-24 10:24:40 +02:00
Hans-Christoph Steiner
cdaf62e5d9 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
2020-05-14 21:58:48 +02:00
Hans-Christoph Steiner
a0e3b01e94 metadata: parsed srclibs must always return a dict as the container 2020-05-14 16:28:13 +02:00
Hans-Christoph Steiner
410901d3bd Merge branch 'yamllint' into 'master'
yamllint

See merge request fdroid/fdroidserver!721
2020-05-14 12:36:19 +00:00
Hans-Christoph Steiner
df563d339a
fix pep8 E741 ambiguous variable name 'l' 2020-05-14 12:11:36 +02:00
Hans-Christoph Steiner
1106795583 deploy: stop uploading to androidobservatory if its already up there 2020-05-14 11:55:09 +02:00
Hans-Christoph Steiner
30f2d62597 import: fix --omit-disable flag, its a boolean 2020-05-14 11:55:09 +02:00
Hans-Christoph Steiner
ba854ab24a support Cordova's gradle file
https://cordova.apache.org/docs/en/latest/guide/platforms/android/index.html
2020-05-14 11:55:09 +02:00
Hans-Christoph Steiner
77367f1bff server: disable extraneous HTTP errors in upload_apk_to_virustotal
This should be in the lowest level so that both the low level call,
upload_apk_to_virustotal(), and the next level up, upload_to_virustotal(),
both inherit this.
2020-05-14 11:55:09 +02:00
Torsten Grote
db96753902
Fix nightly --archive-older command line option 2020-05-11 10:23:26 -03:00
Michael Pöhn
810a6587df add --force-yamllint option to lint 2020-04-25 17:39:23 +02:00
Michael Pöhn
b076e8cba7 add srclib spec parser 2020-04-25 17:22:48 +02:00
Michael Pöhn
ed2c5f6f5b run yamllint on srclibs when running 'fdroid lint' 2020-04-25 15:57:21 +02:00
Michael Pöhn
975538a7a7 make yamllint optional 2020-04-24 15:47:47 +02:00