Commit graph

2903 commits

Author SHA1 Message Date
Hans-Christoph Steiner
5d161cc9fd validate appid when reading metadata files
The metadata file must be named after the Application ID of the app it is
describing, and Android Application IDs must be valid Java Package Names.
2018-09-03 22:56:08 +02:00
Hans-Christoph Steiner
3011953d0e convert apkcache from pickle to JSON
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle.  For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed.  That is not possible with JSON.

This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.

closes #163
2018-09-03 18:07:40 +02:00
Hans-Christoph Steiner
3ffe2860f3 gitlab-ci: add 'bandit' security scanner to all runs
bandit is used by Radically Open Security and is part of the GitLab Ultimate
Static Application Security Testing (SAST) suite.

https://docs.gitlab.com/ee/user/project/merge_requests/sast.html
2018-08-29 17:48:06 +02:00
Hans-Christoph Steiner
4d13a904f3 use defusedxml to avoid DoS attacks while loading XML 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
cc94ebca30 use global constant for the 'xmlns:android' XML namespace 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
3e1d313b7c mark manually sanitized input so bandit doesn't complain 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
2edc68d6cd mark all required permissions so bandit doesn't complain 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
8d3e82913e mark all required usages of MD5 so bandit doesn't complain 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
a089614225 checkupdates: remove magic number 99999999 from HTTP checks
It is vestigal from old code and no longer is needed.
2018-08-29 17:24:24 +02:00
Hans-Christoph Steiner
5d77fd97ee use posixpath.join() for paths on the buildserver
This fixes bandit misdetection of hardcoded /tmp dir.  posixpath.join() is
good to use anyway, it highlights what is on the remote server, vs what is
local.  Local paths should use os.path.join() to support Windows, etc.
posixpath is built in since Python 3.4, maybe earlier
2018-08-29 17:24:04 +02:00
Hans-Christoph Steiner
4503e7a92a replace unneeded eval() call and support negative versionCodes 2018-08-29 17:24:04 +02:00
Michael Pöhn
ee4945cb84 remove unused YamlLoader optimization 2018-08-29 11:25:01 +02:00
Michael Pöhn
e7a34807f7 use yaml.safe_load for parsing metadata 2018-08-29 00:33:58 +02:00
Michael Pöhn
4dcfa95d6e fix: jarsigner and keytool lookup regression 2018-08-21 03:05:43 +02:00
Cyril Russo
81641b4628 Fixed precommit checks 2018-08-20 11:17:36 +02:00
Cyril Russo
4303b0fac1 Apply suggestion from @uniqx to using shutil.which instead of iterating path by hand 2018-08-17 20:14:54 +02:00
Cyril Russo
ace33bcfc0 Reverted the change in the default config.py
Improved the detection of keytool and jarsigner by also searching the PATH environment variable
2018-08-17 16:46:46 +02:00
Cyril Russo
4f21045cad Fix keytool not found on MacOSX (when using Java from Apple)
Fix keytool parsing error due to localisation keytool -list command.
Always fallback to english so the parsing makes sense.
2018-08-17 15:32:29 +02:00
Hans-Christoph Steiner
05be4bc814 Merge branch '471-lint-check-unknown-keys' into 'master'
check for unknown app fields and build flags when parsing yml

Closes #471

See merge request fdroid/fdroidserver!554
2018-08-14 10:50:59 +00:00
Michael Pöhn
6a06944945 better exception and waring message when yaml metadata checks fail; build flags check optional 2018-08-14 12:31:12 +02:00
Michael Pöhn
0d005ab83a make parse_yaml_metadata ignore empty input 2018-08-14 12:31:12 +02:00
Michael Pöhn
e83221eb03 improve yaml build flag check cost; added unit test for unknown app fields, builds flags checks when parsing yaml 2018-08-14 12:31:12 +02:00
Michael Pöhn
25b4a47951 check for unknown app fields and build flags when parsing yml 2018-08-14 12:31:12 +02:00
Michael Pöhn
f53d1abcce fix: make sure ensure_final_value always has a clearly defined return value 2018-08-14 11:07:44 +02:00
Marcus
cf3665b328 Merge branch 'facebook_sdk' into 'master'
scanner: add facebook sdk to forbidden libraries

Closes #534

See merge request fdroid/fdroidserver!542
2018-08-10 13:36:44 +00:00
Marcus Hoffmann
a6dfbc6e2e scanner: add facebook sdk to forbidden libraries
These are sourceavailable but not under a free license.
I made sure that this matches only the facebook sdk's from here:
https://github.com/facebook/facebook-android-sdk and not some real open
source libraries by facebook (fresco, stetho, ...). These seem to be
under a different namespace.

fdroid/fdroidserver#534
2018-08-10 15:20:01 +02:00
Hans-Christoph Steiner
a01d11ba64 Merge branch '530-lint-issue-with-disable' into 'master'
Resolve "Lint issues with 'disable: ...` in YAML"

Closes #530

See merge request fdroid/fdroidserver!550
2018-08-03 09:07:09 +00:00
Michael Pöhn
74ae6a6a3b set locale explicitly when using keytool 2018-08-03 04:36:00 +02:00
Michael Pöhn
1e7d9b854d never interpret "disable" as bool in yaml metadata 2018-08-02 17:31:58 +02:00
Licaon_Kter
da0c541dec Lint - the error should hint that the right word is Categories 2018-07-30 22:56:57 +00:00
Michael Pöhn
27d382e1f4 clean up globals in signatures.py 2018-07-30 13:07:07 +02:00
mimi89999
d76c5dc859
signatures: Don't pass config to extract
Closes #536
2018-07-29 17:45:45 +02:00
Hans-Christoph Steiner
481a6274f0 Merge branch 'archive_icons' into 'master'
update: populate app level info for archive only from archived apks

Closes #365

See merge request fdroid/fdroidserver!537
2018-07-27 13:09:13 +00:00
Marcus Hoffmann
753e9c8175 scanner: properly combine all gradle compile commands with flavors 2018-07-25 19:00:34 +02:00
Marcus Hoffmann
c44ab190bd scanner: fix reformatting fail
This was broken by 44e8f425d :-(.
2018-07-25 19:00:37 +02:00
Areeb Jamal
7bdffde804 build: replace exclude with filter during source tar creation
Replace the deprecated exclude argument with filter in build.py
allowing users of Python 3.7 to use the fdroid CLI

Closes #531
2018-07-25 19:05:30 +05:30
Marcus Hoffmann
957b6b8d19
update: populate app level info for archive only from archived apks
Fixes fdroid/fdroidserver#365
2018-07-19 10:44:14 +02:00
Hans-Christoph Steiner
84dfd6b06a update: more reliable handling of XMLNS with androguard
Sometimes androguard returns the XMLNS as entirely empty, which would make
it an invalid APK since normally the 'android' name is mapped to the
'http://schemas.amazon.com/apk/res/android' value.  Occasionally, a
different key is used.

closes fdroid/fdroidserver#515
2018-07-17 12:37:13 +02:00
Hans-Christoph Steiner
4352aa2792 update: pass resourceIds through if they can't be deferenced
People do odd tricks with `aapt --rename-manifest-package` and androguard
doesn't handle them all.

* https://github.com/androguard/androguard/issues/503
2018-07-17 12:37:13 +02:00
Hans-Christoph Steiner
0e0a5019ab update: little trick to avoid some cases of XML app icons
refs #344
2018-07-17 12:37:13 +02:00
Hans-Christoph Steiner
75e9a9e176 update: support versionCode values in hex 2018-07-17 12:37:13 +02:00
Hans-Christoph Steiner
c8e61aa02d update: handle random drawable folder names when parsing icons
The app com.android.acehk.aceapp37423 had this invalid stuff in it:

     6165  2014-03-28 12:52   res/drawable-320dpi/ic_launcher.png
2018-07-17 12:37:13 +02:00
Hans-Christoph Steiner
56a5d868f1 update: purge --editor and --interactive, they were removed in 2013
48296df5b0
2018-07-17 12:37:13 +02:00
Michael Pöhn
b6ad6724cc Revert "publish: avoid double removal of dev-supplied reference binary"
This reverts commit db5ed26517.

Doh, messed up the rebase...
2018-07-13 00:07:58 +02:00
Hans-Christoph Steiner
58b81b5ca6 publish: strict regex replacement for .binary.apk
. matches anything \. matches the char "."
2018-07-12 23:53:05 +02:00
Hans-Christoph Steiner
68fb3d2bd9 fix typos 2018-07-12 23:53:05 +02:00
Michael Pöhn
db5ed26517 publish: avoid double removal of dev-supplied reference binary 2018-07-12 23:53:05 +02:00
Michael Pöhn
cdc2147de9 put .binary.apk files into sub-directory 2018-07-12 23:52:46 +02:00
Michael Pöhn
fab02a49dc Merge branch 'better_gradle_support' into 'master'
Better gradle support

Closes #98

See merge request fdroid/fdroidserver!527
2018-07-11 15:20:19 +00:00
Marcus Hoffmann
9889a98dea
build: improve gradle experience
This expands the gradle wrapper shell script used by the buildserver for
usage outside the buildserver environment. It also allows downloading
whitelisted versions of gradle if they are not yet deployed to the
buildserver by simply upsating the copy of fdroidserver (in contrast to
having to reprovision the whole buildserver).

We first move the buildserver/gradle shell script to the repo root
as gradlew-fdroid, as it's an fdroid specific gradle wrapper.
We also now sync it inside the build VM before each build.

We then add a list of whitelisted gradle distributions taken from the
makebuildserver script.

The script additionally now reads two env vars which tell it where to
expect installed versions of gradle and where it might store downloaded
gradle .zip files. Both of those are configurable from config.py. As the
first should normally just be a subdir of the second it's not exposed in
the example config.py but only used by the buildserver config.py.

Default config now uses this internal gradle wrapper but a path to a
custom wrapper or specific gradle distribution can still be set from
config.py.

Closes fdroid/fdroidserver#98
Ref: fdroid/fdroidserver#370
2018-07-11 11:49:46 +02:00