Commit graph

3677 commits

Author SHA1 Message Date
Hans-Christoph Steiner
960d31af2a import: standardize on 'appid' var name for Application ID
The rest of the fdroidserver code uses 'appid' or 'packageName'.  The official
Android name is "Application ID".  "Package Name" is the Java term, and it
is used in Android in Java code.
2020-07-30 14:57:19 +02:00
Hans-Christoph Steiner
10fa912c16 deploy: remove git remotes not enabled in servergitmirrors
If the repo operator removes an item from servergitmirrors, it should auto-
matically remove the remote.
2020-07-30 14:57:19 +02:00
Jochen Sprickerhof
d8b73dd241 Add timstamp to logs 2020-07-04 18:56:10 +02:00
Marcus
1902bfb6fc Merge branch 'fix_760' into 'master'
update.py (and index.py) refactoring

Closes #524 and #760

See merge request fdroid/fdroidserver!756
2020-06-24 22:37:51 +00:00
Marcus Hoffmann
07caa88920 don't include the localized metadata things for /archive
We haven't done this so far and it's a potential big change in archive
size and update performance.
2020-06-25 00:07:42 +02:00
Marcus Hoffmann
d720c99ae5 refactor update.py:main
This makes update.py:main a lot saner by removing a lot of the
implicit assumptions between the different stages of generating the
repository index.

* mostly unify repo and archive processing, that means the archive is
  now actually getting the same treatment regarding i.e. fastlane data.
  Previously the archive didn't get considered at all here.
* already filter the list of apps to include in a repo in update.py and
  give that prefiltered list to index. This makes sure we actually only
  copy fastlane/triple-t/etc. stuff for apps ending up in the index.
  This both, can save a lot of time if there are a lot of old /build
  dirs lying around and doesn't clutter /repo with things that aren't
  referenced from the index.

Closes fdroid/fdroidserver#524
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
e66683720b move index sorting to index module
This is a historic detail of the index format, so move it there.
For wiki update and status json there's really no reason why this should
be done in alphabetic app name order. Use the default sort order by
appid.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
8c71637d43 update: make copy_triple_t_store_metadata and insert_localized_app_metadata not assume /repo
This will enable copying the localized metadata to the archive as well.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
ee4ee85cbd update:archive_old_apks: handle apps with no CVC
If an app doesn't have a CVC, we can just skip any special archive
handling.
Also rename weirdly named `res` to `apkList`.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann
03881154c6 metadata: make linkresolver an actual object
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.

Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
2020-06-24 23:53:35 +02:00
Hans-Christoph Steiner
5fce16c847
remove unused, confusing arg to common.getsrclib() 2020-06-24 22:40:27 +02:00
Hans-Christoph Steiner
d0f426e076
replace $$srclib$$ with an absolute path
closes #725
2020-06-24 22:40:26 +02:00
Hans-Christoph Steiner
1e6f99988a fix typo when looking for build.gradle.kts 2020-06-24 21:43:58 +02:00
Hans-Christoph Steiner
238f048257 update: fix crash when liberapay: or open_collective: not in FUNDING.yml
closes #799
2020-06-24 21:39:43 +02:00
Hans-Christoph Steiner
62c8fd5999
add Liberapay: field with username as data
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs.  Now it is a username.  So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration.  LiberapayID: will not override
Liberapay:.  Clients are expected to prefer Liberapay: over LiberapayID:
2020-06-16 15:35:28 +02:00
Hans-Christoph Steiner
0183592526
update: insert donation links based on FUNDING.yml
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links.  Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:.  Specifying those in the metadata file takes
precedence over the FUNDING.yml.  This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata.  This lets the
git repo maintain those specific donations links themselves.

https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files

The test file was generated using:

```python
import os, re, yaml

found = dict()
for root, dirs, files in os.walk('.'):
    for f in files:
        if f == 'FUNDING.yml':
            with open(os.path.join(root, f)) as fp:
                data = yaml.safe_load(fp)
            for k, v in data.items():
                if k not in found:
                    found[k] = set()
                if not v:
                    continue
                if isinstance(v, list):
                    for i in v:
                        found[k].add(i)
                else:
                    found[k].add(v)

            with open('gather-funding-names.yaml', 'w') as fp:
                output = dict()
                for k, v in found.items():
                    output[k] = sorted(v)
                yaml.dump(output, fp, default_flow_style=False)
```
2020-06-16 15:35:26 +02:00
Marcus
f84818c15e scanner: add a simple scan for blacklisted classes after build step
add com.android.billing to blacklist, see
https://gitlab.com/fdroid/fdroiddata/-/issues/2070#note_360611289
2020-06-15 18:03:19 +00:00
Marcus
6986e73506 Merge branch 'allow_appcenter' into 'master'
scanner: allow microsoft appcenter

See merge request fdroid/fdroidserver!764
2020-06-15 10:46:51 +00:00
Marcus Hoffmann
670328c9e5 scanner: allow microsoft appcenter, except appcenter-push
This sdk is open source. It was added because appcenter has a dependency
to play-services. It's possible though to build an app using appcenter
that doesn't pull in play services, so we can't blanket ban the sdk.

The appcenter-push modules has obvious refenrences to firebase, so it's
safe to error on that.

Ref: https://phabricator.wikimedia.org/T254980
2020-06-15 12:24:46 +02:00
Marcus
12f4ae3039 Merge branch 'fix_791' into 'master'
scanner: fix wrong path being passed to function

Closes #791

See merge request fdroid/fdroidserver!767
2020-06-13 11:03:30 +00:00
Marcus Hoffmann
3a9f1e845c scanner: fix wrong path being passed to function
Also make rename the parameter in safe_path to make it clear that this
is just a relative path.

Closes fdroid/fdroidserver#791.
2020-06-13 03:01:54 +02:00
Licaon_Kter
72fde0f10b Detect .gitmodules and add submodules build line 2020-06-11 16:36:54 +03:00
Marcus Hoffmann
294799e2f5 scanner: fix options handling
closes fdroid/fdroidserver#789
2020-06-11 11:44:00 +02:00
Hans-Christoph Steiner
0837289935 scanner: fix regex for matching URLs in gradle maven{} blocks
closes #465

This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml

pat = re.compile(r'\smaven\s*{[^}]+}')

finds = set()
for root, dirs, files in os.walk('.'):
    for f in files:
        if '.gradle' in f:
            with open(os.path.join(root, f), errors='surrogateescape') as fp:
                contents = fp.read()
            for m in pat.findall(contents):
                finds.add(m)

with open('finds.yaml', 'w') as fp:
    yaml.dump(sorted(finds), fp, default_flow_style=False)

```
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
6590f3869e scanner: error/warn on dex/gz/zip, closes #394 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
d7b3bca1e7 build: pass --verbose flag through to scanner 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
ee54dbce87 scanner: safely check options, for things using this API
https://gitlab.com/fdroid/fdroidserver/-/issues/771#note_353495799
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
75acf63efa scanner: remove all gradle wrapper files 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
2f5d780c49 scanner: always setup JSON data structure so functions work as API 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
84f225f2f2 scanner: make problem descriptions translationable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
d898ad0412 scanner: make AARs and JARs trigger an error
refs #491
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
3bc246ccad scanner: ignore well known image types that are set executable 2020-06-10 19:42:23 +02:00
Michael Pöhn
a97b3ca4dd implement plugin system review suggestsions 2020-06-10 18:43:23 +02:00
Michael Pöhn
77167e098e plugin system: regex instead of import bases plugin parsing 2020-06-10 18:43:11 +02:00
Michael Pöhn
b257a3411a stick to default python sys.path 2020-06-10 12:11:47 +02:00
Michael Pöhn
3a3803ea2d raise excepten when starting broken plugin 2020-06-10 12:11:47 +02:00
Michael Pöhn
bf815251ec rough plugin system implementation 2020-06-10 12:11:47 +02:00
Marcus Hoffmann
581e433832 strip_and_copy_image: abort on broken symlinks
Also guard against other processes removing the files we are about to
copy.

closes fdroid/fdroidserver#783
2020-06-04 12:58:11 +02:00
Hans-Christoph Steiner
13d9a122bf metadata: validate STRING and INT build field types
This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
2020-06-02 22:06:01 +02:00
Hans-Christoph Steiner
b2daf96284 lint: fix pyflakes 2020-06-02 22:05:18 +02:00
Hans-Christoph Steiner
6c5887e906
lint: make --force-yamllint error if yamllint is not installed
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/753#note_353829401
2020-06-02 21:56:05 +02:00
Hans-Christoph Steiner
6030445be0 logging.warn() was deprecated in Python 3.3, use logging.warning()
sed -i 's,logging\.warn(,logging.warning(,g' fdroid */*.*

https://docs.python.org/3.3/library/logging.html#logging.Logger.warning
2020-05-27 08:35:14 +02:00
Hans-Christoph Steiner
67332d83a5 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot

This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes #771.  It was previously merged in !748 then
reverted in 68c072c72e
2020-05-27 08:35:14 +02:00
relan
9ff77cfd1a build: clean buildSrc/build
The buildSrc directory contains custom build logic written in Kotlin.
Before this change we had to 'scandelete' buildSrc/build in the build
recipes becase 'gradle clean' leaves binary artifacts there.
2020-05-26 07:37:20 +00:00
relan
ddfbd1cc47 build: fix directories removal
The 'dirs' array contains a single-level listing of a directory, e. g.
['app', 'build', 'build.gradle', 'gradle', '.gradle']. Multi-component
paths like 'build/tmp' could never be found in this array and thus were
never removed.

Call shutil.rmtree() without checking that the argument is in 'dirs'. If
it exists and is a directory, it'll be removed. Otherwise shutil.rmtree()
will do nothing.
2020-05-26 07:37:20 +00:00
Marcus Hoffmann
68c072c72e Revert "scanner: add --json option for outputting machine readable results"
This reverts commit cdaf62e5d9.

See: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/748#note_347769371
2020-05-24 10:24:40 +02:00
Hans-Christoph Steiner
cdaf62e5d9 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
2020-05-14 21:58:48 +02:00
Hans-Christoph Steiner
a0e3b01e94 metadata: parsed srclibs must always return a dict as the container 2020-05-14 16:28:13 +02:00
Hans-Christoph Steiner
410901d3bd Merge branch 'yamllint' into 'master'
yamllint

See merge request fdroid/fdroidserver!721
2020-05-14 12:36:19 +00:00
Hans-Christoph Steiner
df563d339a
fix pep8 E741 ambiguous variable name 'l' 2020-05-14 12:11:36 +02:00