Commit graph

154 commits

Author SHA1 Message Date
relan
bd96a17ef2 scanner: blacklist Bugly
The artifact is licensed under "The Bugly Software License, Version 1.0"
[1], but link to the full text [2] is broken. LICENSE file in the source
code repository is empty.

I guess this library is non-free.

[1] https://mvnrepository.com/artifact/com.tencent.bugly/crashreport/2.8.6.0
[2] http://bugly.qq.com/licenses/LICENSE-1.0.txt
[3] https://github.com/BuglyDevTeam/Bugly-Android
2019-09-28 07:58:19 +03:00
relan
3a7ad650e3 scanner: fix local Debian Maven repo handling
The resulting regex was 'https?://file:///usr/share/maven-repo' causing
scanner error.
2019-08-29 21:32:54 +03:00
Hans-Christoph Steiner
f0460dea6e scanner: allow local Debian Maven repo file:///usr/share/maven-repo
It is now possible to build an app using only things in Debian.  Since the
buildserver will always control the contents of file:///usr/share/maven-repo,
it is the most safe repo there is.
2019-05-01 21:16:15 +02:00
Pierre Rudloff
b3754af137 Add cloudrail to blacklist 2019-02-07 10:26:03 +01:00
Hans-Christoph Steiner
57556aceee remove redundant open() arg: encoding='utf8'
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
2018-10-19 15:01:34 +02:00
Marcus Hoffmann
a6dfbc6e2e scanner: add facebook sdk to forbidden libraries
These are sourceavailable but not under a free license.
I made sure that this matches only the facebook sdk's from here:
https://github.com/facebook/facebook-android-sdk and not some real open
source libraries by facebook (fresco, stetho, ...). These seem to be
under a different namespace.

fdroid/fdroidserver#534
2018-08-10 15:20:01 +02:00
Marcus Hoffmann
753e9c8175 scanner: properly combine all gradle compile commands with flavors 2018-07-25 19:00:34 +02:00
Marcus Hoffmann
c44ab190bd scanner: fix reformatting fail
This was broken by 44e8f425d :-(.
2018-07-25 19:00:37 +02:00
Marcus Hoffmann
3a6942ca77
scanner: whitelist www.jitpack.io
Only the non-www version was recognized so far.

Closes #498
2018-06-11 12:40:55 +02:00
Hans-Christoph Steiner
44e8f425d5 fix PEP8 formatting issue
fdroidserver/scanner.py:35:34: E241 multiple spaces after ','
fdroidserver/scanner.py:36:35: E241 multiple spaces after ','
fdroidserver/scanner.py:37:30: E241 multiple spaces after ','
fdroidserver/scanner.py:38:41: E241 multiple spaces after ','
fdroidserver/scanner.py:39:30: E241 multiple spaces after ','
fdroidserver/scanner.py:40:38: E241 multiple spaces after ','
fdroidserver/scanner.py:41:38: E241 multiple spaces after ','
2018-03-08 13:55:07 +01:00
relan
44d17663fd scanner: support Gradle plugin 3.0 syntax for dependencies
See https://developer.android.com/studio/build/gradle-plugin-3-0-0-migration.html#new_configurations
2018-03-08 15:23:15 +03:00
Hans-Christoph Steiner
32213ef040 scanner: allow running without versionCode and as API
This lets `fdroid scanner my.package.name` run without requiring that the
versionCode is also specified.  It also allows scanner.scan_source() to be
called as a function in the public API of fdroidserver.
2018-01-26 10:18:41 +01:00
Jan Berkel
62d1c672f3 whitelist some open-source firebase libs 2017-12-14 22:03:48 +01:00
Hans-Christoph Steiner
ffc91e301a make _ always be the gettext function, nothing else
This avoids hard bugs where the _() function gets overidden by a str or
something else.
2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
ab02a6fcc3 Revert "scanner: promote jar and aar files present in the repo to errors"
This reverts commit 5f5d3ea896.

These new scanner rules for JARs causes most builds on fdroiddata to
break.  For example, at least the past 20 releases of
org.fdroid.fdroid.  fdroiddata needs to be cleaned up and prepared for
this before we can include 5f5d3ea896

I'm running fdroid scanner on the whole archive now, I'll post the log
to an issue once its done.  Ignoring test files would eliminate some
of these build failures, at least for most org.fdroid.fdroid builds.

refs #!325
2017-10-25 22:55:47 +02:00
Hans-Christoph Steiner
53ce81179c Merge branch 'error_on_jars' into 'master'
RFC: Error on jars

See merge request fdroid/fdroidserver!325
2017-10-12 11:46:59 +00:00
Hans-Christoph Steiner
6c7b987239 fix string formats that are ambiguous for translators 2017-09-16 13:19:38 +02:00
Hans-Christoph Steiner
fa0d618d2f rename 'app-id' to standard Android 'applicationId'
* https://developer.android.com/studio/build/application-id.html
* https://sites.google.com/a/android.com/tools/tech-docs/new-build-system/applicationid-vs-packagename

This only changes the term in the human texts, not var names or CLI flags.
2017-09-15 11:41:51 +02:00
Hans-Christoph Steiner
278d67d960 implement gettext localization
This allows all the text to be localized via Weblate.  This is a quick
overview of all the strings, but there are certainly some that were left
out.

closes #342
2017-09-15 11:39:00 +02:00
Hans-Christoph Steiner
cb10f0df09 standardize os.walk() var names based on Python 3.5 docs
There were multiple conventions used in the code, but mostly it was already
using the convention from the docs, so this converts things to using that
convention:

https://docs.python.org/3/library/os.html#os.walk
2017-09-15 11:37:21 +02:00
Marcus Hoffmann
5f5d3ea896 scanner: promote jar and aar files present in the repo to errors
This makes sure we we don't include any prebuilt code in the build process.
2017-09-11 17:05:24 +02:00
Marcus Hoffmann
15c4492625 remove gradle-wrapper.jar during scan
We make sure to not use the gradle wrapper anyway so this should have no
impact.
2017-09-11 17:05:21 +02:00
Marcus Hoffmann
af2eb2bd50 scanner: warn on aar files 2017-09-11 17:05:16 +02:00
Marcus Hoffmann
5d9eb9b295 scanner: rename variables, use os.path.relpath
* rename fd -> path_in_build_dir
* rename fp -> filepath
* use os.path.reldir instead of string manipulation
2017-09-11 16:49:39 +02:00
Marcus Hoffmann
1ee3f9aad5 scanner: more descriptive variable names from os.walk
This uses the names from python documentation of os.walk().
2017-09-11 16:49:30 +02:00
Boris Kraut
c056b95fd5 scanner: allow google maven 2017-05-26 15:51:38 +02:00
Hans-Christoph Steiner
17cd07f1a7 fix pylint unused-argument 2017-05-23 22:34:16 +02:00
Torsten Grote
1fcd8e63a3
Replace sys.exit() in non-main functions by exceptions
Also move all exceptions into one module
2017-05-22 16:51:11 -03:00
Hans-Christoph Steiner
6c2cf2ccdd convert internal dict keys to match fdroidclient's Apk class
This syncs up the field names between the fdroiddata .yml files, the keys
used in the implementation in fdroidserver, the index data format, and the
final data structures in fdroidclient.  This makes it easier for devs to
follow, and makes the Jackson parsing library automatically handle
converting the data from the index file to Java instances.

This bumps the metadata version since the apkcache will have to be
discarded.

Here are the name changes:
* apkname --> apkName
* id --> packageName
* sha256 --> hash
* version --> versionName
* versioncode --> versionCode

tests/repo/index.xml was changed only to bump the metadata version
from 17 to 18.
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
e0f39a7e7b rename Build fields: version -> versionName, vercode -> versionCode
Since the YAML/JSON/etc. field names are now exactly the same as the field
names used in the internal dict in the Build class, this is a global rename

This keeps with the standard names used in Android:
https://developer.android.com/guide/topics/manifest/manifest-element.html
2017-02-24 11:01:01 +01:00
Boris Kraut
daafc422ae scanner: add firebase to usual suspect list, closes #259 2017-02-23 00:41:00 +01:00
Boris Kraut
c87cdb91db scanner: allow oss.jfrog.org/artifactory/oss-snapshot-local 2017-02-13 20:24:35 +01:00
Daniel Martí
82b1d7ad14 all: make newer pycodestyle happy
Apparently the "two empty lines" rule is now stricter.
2016-11-15 20:55:06 +00:00
est31
af22f18296 Scanner: handle utf8 errors more gracefully
Fixes #226
2016-11-02 16:58:59 +01:00
Hans-Christoph Steiner
d522988d8c make metadata exceptions optional based on CLI flag
In many cases, there are times where metadata errors need to be ignored, or
at least not stop the command from running.  For example, there will
inevitably be new metadata fields added, in which case a packaged version
of fdroidserver will throw errors on each one.  This adds a standard -W
flag to customize the response: ignore, default, or error.

* by default, the errors are still errors
* `fdroid readmeta -W` will just print errors
* `fdroid readmeta -Wignore` will not even print errors

https://gitlab.com/fdroid/fdroidserver/issues/150
2016-09-12 12:55:48 +02:00
Hans-Christoph Steiner
2b6d692f06 use UTF8 as default instead of ASCII for .java .gradle pom.xml
.java .gradle and XML files all can use any encoding.  Most code is ASCII,
but authors' names, etc. can easily be non-ASCII.  UTF-8 is by far the most
common file encoding.  While UTF-8 is the default encoding inside the code
in Python 3, it still has to deal with the real world, so the encoding
needs to be explicitly set when reading and writing files. So this switches
fdroidserver to expect UTF-8 instead of ASCII when parsing these files. For
now, this commit means that we only support UTF-8 encoded *.java, pom.xml
or *.gradle files.  Ideally, the code would detect the encoding and use the
actual one, but that's a lot more work, and its something that will not
happen often. We can cross that bridge when we come to it.

One approach, which is taken in the commit when possible, is to keep the
data as `bytes`, in which case the encoding doesn't matter.

This also fixes this crash when parsing gradle and maven files with
non-ASCII chars:

ERROR: test_adapt_gradle (__main__.BuildTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/fdroidserver-eighthave/tests/build.TestCase", line 59, in test_adapt_gradle
    fdroidserver.build.adapt_gradle(testsdir)
  File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/build.py", line 445, in adapt_gradle
    path)
  File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/common.py", line 188, in regsub_file
    text = f.read()
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9460: ordinal not in range(128)
2016-06-07 20:13:54 +02:00
Daniel Martí
fc21dbc667 Replace remaining file() usage 2016-03-10 16:43:37 +00:00
Daniel Martí
920ae4692f Port all imports to python3 2016-03-10 16:43:37 +00:00
Daniel Martí
75419c1f04 Replace iteritems() with items() 2016-03-10 16:43:37 +00:00
Daniel Martí
99edd64372 Switch all headers to python3 2016-03-10 16:43:36 +00:00
Daniel Martí
163044648c Allow commonsware and gradle plugin repos 2016-02-17 13:00:05 +00:00
Hans-Christoph Steiner
a28ede2b67 PEP8 fix E225 missing whitespace around operator 2016-02-15 23:06:31 +01:00
Daniel Martí
1ff4f63de3 scanner: Allow clojars.org maven repository
See https://github.com/inorichi/tachiyomi/issues/46.
2016-01-24 18:13:45 +00:00
NeroBurner
af38f151a2 Change print-function to have brackets 2015-12-29 20:06:46 +01:00
Daniel Martí
7a7d36feef scanner: remove jni check
buildjni=no is the default. Moreover, some apps now run ndk-build via
gradle, so this was just in the way.
2015-12-04 19:32:01 +01:00
Daniel Martí
bf8518ee8f Rework build into a class
This simplifies usage, goes from

    build['flag']
to
    build.flag

Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.

As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.

While at it, unify "build", "thisbuild", "info", "thisinfo", etc into
just "build".
2015-12-02 20:03:11 +01:00
Daniel Martí
ab614ab442 Rework app into a class
This simplifies usage, goes from

	app['Foo']
to
	app.Foo

Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.

As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
2015-11-28 17:11:05 +01:00
Daniel Martí
b84688af7a scanner: don't warn about ignored files 2015-11-12 20:02:40 +01:00
Daniel Martí
af7178f2fe scanner: Ignore certain binary executable files
These were warnings, so the behaviour doesn't really change.
2015-11-10 21:59:54 +01:00
Daniel Martí
c94b0342ad scanner: ignore .DS_Store files
They are binary and sometimes executable, so they trigger false
positives.
2015-11-10 21:49:46 +01:00