Commit graph

138 commits

Author SHA1 Message Date
Hans-Christoph Steiner
78842e9cc2 scanner: should not exit with error when apkanalyzer fails
apkanalyzer produces useful output when it can run, but it does not
support all recent JDK versions, and also some DEX versions, so this
cannot count on it to always produce useful output or even to run
without exiting with an error.

211dd65ff0 was based on false
assumptions that apkanalyzer can always produce output.

fdroiddata!8585
fdroiddata!8584
2021-03-15 10:54:29 +01:00
Hans-Christoph Steiner
211dd65ff0 scanner: show error if scan_binary fails to run apkanalyzer 2021-03-09 15:55:45 +01:00
mimi89999
01699db376
Add androidx.work:work-gcm to non-free dependencies 2021-01-11 10:44:48 +01:00
Jochen Sprickerhof
a9c380b2b8 Ignore src/test only in path_in_build_dir
Found while debugging Debian autopkgtests.
2021-01-04 21:20:31 +01:00
Izzy
434a23f822
add Google Play Core to non-free/proprietary dependencies 2021-01-04 17:39:57 +01:00
Hans-Christoph Steiner
bf25b4ca03 eliminate app.builds everywhere, it should be app['Builds']
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds".  This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().

The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.

```
sed -i \
 -e 's/app\.builds/app.get('Builds', \[\])/g' \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
 -e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
 -e "s/app\['builds'\]/app.get('Builds', [])/g" \
 */*.*
```
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner
05cd8c6810 scanner: expose "usual suspects" patterns for use in an API 2020-10-07 18:56:17 +02:00
Hans-Christoph Steiner
5e1377c77a standardize on "Application ID" in UI text 2020-10-01 23:01:55 +02:00
Marcus Hoffmann
c90a72e14d scanner: check for test in path relative to build dir
This was degrading scanner errors to warnings whenever the path from the
current running install of fdroidserver contained test, as has been
happening in the CI image builder:

https://gitlab.com/fdroid/ci-images-server/-/blob/master/test#L6
2020-08-30 20:09:19 +02:00
Marcus Hoffmann
d07b4123e9 scanner: docstrings for handleproblem functions 2020-08-25 21:32:29 +02:00
Marcus Hoffmann
03e723b1af fix crash when scanner wants to remove the same file more than once
A file can be flagged for multiple problems (i.e. multiple unknown maven
repos in one build.gradle file that is included in a scandelete path).

The scanner will try to delete it once for every problem detected, we
don't really care, as long as the file is gone.

fixes fdroid/fdroidserver#759
2020-08-25 21:32:29 +02:00
Hans-Christoph Steiner
0b92e60266 handle file type detection using Pythonic methods
This ditches the custom common.get_extension() for straight core Python
methods.  This should make the code closer to Python conventions.  For
example, pathlib also includes the "." in the extension it returns.
2020-08-24 17:05:49 +02:00
Marcus
f84818c15e scanner: add a simple scan for blacklisted classes after build step
add com.android.billing to blacklist, see
https://gitlab.com/fdroid/fdroiddata/-/issues/2070#note_360611289
2020-06-15 18:03:19 +00:00
Marcus
6986e73506 Merge branch 'allow_appcenter' into 'master'
scanner: allow microsoft appcenter

See merge request fdroid/fdroidserver!764
2020-06-15 10:46:51 +00:00
Marcus Hoffmann
670328c9e5 scanner: allow microsoft appcenter, except appcenter-push
This sdk is open source. It was added because appcenter has a dependency
to play-services. It's possible though to build an app using appcenter
that doesn't pull in play services, so we can't blanket ban the sdk.

The appcenter-push modules has obvious refenrences to firebase, so it's
safe to error on that.

Ref: https://phabricator.wikimedia.org/T254980
2020-06-15 12:24:46 +02:00
Marcus Hoffmann
3a9f1e845c scanner: fix wrong path being passed to function
Also make rename the parameter in safe_path to make it clear that this
is just a relative path.

Closes fdroid/fdroidserver#791.
2020-06-13 03:01:54 +02:00
Marcus Hoffmann
294799e2f5 scanner: fix options handling
closes fdroid/fdroidserver#789
2020-06-11 11:44:00 +02:00
Hans-Christoph Steiner
0837289935 scanner: fix regex for matching URLs in gradle maven{} blocks
closes #465

This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml

pat = re.compile(r'\smaven\s*{[^}]+}')

finds = set()
for root, dirs, files in os.walk('.'):
    for f in files:
        if '.gradle' in f:
            with open(os.path.join(root, f), errors='surrogateescape') as fp:
                contents = fp.read()
            for m in pat.findall(contents):
                finds.add(m)

with open('finds.yaml', 'w') as fp:
    yaml.dump(sorted(finds), fp, default_flow_style=False)

```
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
6590f3869e scanner: error/warn on dex/gz/zip, closes #394 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
ee54dbce87 scanner: safely check options, for things using this API
https://gitlab.com/fdroid/fdroidserver/-/issues/771#note_353495799
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
75acf63efa scanner: remove all gradle wrapper files 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
2f5d780c49 scanner: always setup JSON data structure so functions work as API 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
84f225f2f2 scanner: make problem descriptions translationable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
d898ad0412 scanner: make AARs and JARs trigger an error
refs #491
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
3bc246ccad scanner: ignore well known image types that are set executable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
6030445be0 logging.warn() was deprecated in Python 3.3, use logging.warning()
sed -i 's,logging\.warn(,logging.warning(,g' fdroid */*.*

https://docs.python.org/3.3/library/logging.html#logging.Logger.warning
2020-05-27 08:35:14 +02:00
Hans-Christoph Steiner
67332d83a5 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot

This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes #771.  It was previously merged in !748 then
reverted in 68c072c72e
2020-05-27 08:35:14 +02:00
Marcus Hoffmann
68c072c72e Revert "scanner: add --json option for outputting machine readable results"
This reverts commit cdaf62e5d9.

See: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/748#note_347769371
2020-05-24 10:24:40 +02:00
Hans-Christoph Steiner
cdaf62e5d9 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
2020-05-14 21:58:48 +02:00
Hans-Christoph Steiner
df563d339a
fix pep8 E741 ambiguous variable name 'l' 2020-05-14 12:11:36 +02:00
Hans-Christoph Steiner
ad92b4c678 scanner: add --force option for scanning disabled apps/builds 2020-02-13 13:58:13 +01:00
licaon-kter
135c92f3d0 Add Microsoft Appcenter SDK to forbidden list 2020-02-02 03:50:54 +02:00
Hans-Christoph Steiner
b95f66a806
scanner: only allow HTTPS versions of the whitelist 2019-11-13 11:59:24 +01:00
Mike Hardy
3c398e2aa6 remove redundant google analytics match
'google.*play.*services' and 'firebase' will catch both manners of including analytics I think
https://developers.google.com/android/guides/setup#split
https://firebase.google.com/support/release-notes/android#latest_sdk_versions

This has the desired side effect of also allowing the libre analytics library
'net.mikehardy:google-analytics-java' and 'net.mikehardy:google-analytics-java7'
2019-11-02 18:40:58 +00:00
relan
bd96a17ef2 scanner: blacklist Bugly
The artifact is licensed under "The Bugly Software License, Version 1.0"
[1], but link to the full text [2] is broken. LICENSE file in the source
code repository is empty.

I guess this library is non-free.

[1] https://mvnrepository.com/artifact/com.tencent.bugly/crashreport/2.8.6.0
[2] http://bugly.qq.com/licenses/LICENSE-1.0.txt
[3] https://github.com/BuglyDevTeam/Bugly-Android
2019-09-28 07:58:19 +03:00
relan
3a7ad650e3 scanner: fix local Debian Maven repo handling
The resulting regex was 'https?://file:///usr/share/maven-repo' causing
scanner error.
2019-08-29 21:32:54 +03:00
Hans-Christoph Steiner
f0460dea6e scanner: allow local Debian Maven repo file:///usr/share/maven-repo
It is now possible to build an app using only things in Debian.  Since the
buildserver will always control the contents of file:///usr/share/maven-repo,
it is the most safe repo there is.
2019-05-01 21:16:15 +02:00
Pierre Rudloff
b3754af137 Add cloudrail to blacklist 2019-02-07 10:26:03 +01:00
Hans-Christoph Steiner
57556aceee remove redundant open() arg: encoding='utf8'
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
2018-10-19 15:01:34 +02:00
Marcus Hoffmann
a6dfbc6e2e scanner: add facebook sdk to forbidden libraries
These are sourceavailable but not under a free license.
I made sure that this matches only the facebook sdk's from here:
https://github.com/facebook/facebook-android-sdk and not some real open
source libraries by facebook (fresco, stetho, ...). These seem to be
under a different namespace.

fdroid/fdroidserver#534
2018-08-10 15:20:01 +02:00
Marcus Hoffmann
753e9c8175 scanner: properly combine all gradle compile commands with flavors 2018-07-25 19:00:34 +02:00
Marcus Hoffmann
c44ab190bd scanner: fix reformatting fail
This was broken by 44e8f425d :-(.
2018-07-25 19:00:37 +02:00
Marcus Hoffmann
3a6942ca77
scanner: whitelist www.jitpack.io
Only the non-www version was recognized so far.

Closes #498
2018-06-11 12:40:55 +02:00
Hans-Christoph Steiner
44e8f425d5 fix PEP8 formatting issue
fdroidserver/scanner.py:35:34: E241 multiple spaces after ','
fdroidserver/scanner.py:36:35: E241 multiple spaces after ','
fdroidserver/scanner.py:37:30: E241 multiple spaces after ','
fdroidserver/scanner.py:38:41: E241 multiple spaces after ','
fdroidserver/scanner.py:39:30: E241 multiple spaces after ','
fdroidserver/scanner.py:40:38: E241 multiple spaces after ','
fdroidserver/scanner.py:41:38: E241 multiple spaces after ','
2018-03-08 13:55:07 +01:00
relan
44d17663fd scanner: support Gradle plugin 3.0 syntax for dependencies
See https://developer.android.com/studio/build/gradle-plugin-3-0-0-migration.html#new_configurations
2018-03-08 15:23:15 +03:00
Hans-Christoph Steiner
32213ef040 scanner: allow running without versionCode and as API
This lets `fdroid scanner my.package.name` run without requiring that the
versionCode is also specified.  It also allows scanner.scan_source() to be
called as a function in the public API of fdroidserver.
2018-01-26 10:18:41 +01:00
Jan Berkel
62d1c672f3 whitelist some open-source firebase libs 2017-12-14 22:03:48 +01:00
Hans-Christoph Steiner
ffc91e301a make _ always be the gettext function, nothing else
This avoids hard bugs where the _() function gets overidden by a str or
something else.
2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
ab02a6fcc3 Revert "scanner: promote jar and aar files present in the repo to errors"
This reverts commit 5f5d3ea896.

These new scanner rules for JARs causes most builds on fdroiddata to
break.  For example, at least the past 20 releases of
org.fdroid.fdroid.  fdroiddata needs to be cleaned up and prepared for
this before we can include 5f5d3ea896

I'm running fdroid scanner on the whole archive now, I'll post the log
to an issue once its done.  Ignoring test files would eliminate some
of these build failures, at least for most org.fdroid.fdroid builds.

refs #!325
2017-10-25 22:55:47 +02:00
Hans-Christoph Steiner
53ce81179c Merge branch 'error_on_jars' into 'master'
RFC: Error on jars

See merge request fdroid/fdroidserver!325
2017-10-12 11:46:59 +00:00