Commit graph

2607 commits

Author SHA1 Message Date
Michael Pöhn
a787d123a2 fdroid build: hide --on-server cli flag 2019-01-29 11:23:45 +01:00
Hans-Christoph Steiner
217e6a7c30 metadata: fix pyflakes "use ==/!= to compare str, bytes, and int literals"
$ ./hooks/pre-commit || export EXITVALUE=1
WARNING: ruby is not installed, using dummy placeholder!
fdroidserver/metadata.py:1180: use ==/!= to compare str, bytes, and int literals
fdroidserver/metadata.py:1184: use ==/!= to compare str, bytes, and int literals
fdroidserver/metadata.py:1186: use ==/!= to compare str, bytes, and int literals
fdroidserver/metadata.py:1189: use ==/!= to compare str, bytes, and int literals
ERROR: pyflakes tests failed!
2019-01-27 23:00:00 +01:00
Hans-Christoph Steiner
aa47e93f36 Revert "Revert "lint: check if CurrentVersion is older than oldest build entry""
This reverts commit a5ec1703c9.

fdroid/fdroiddata@42bb872b46
2019-01-21 23:00:33 +01:00
Hans-Christoph Steiner
1174b14d1e Merge branch 'neonsoftware/master' into master
buildserver: include build-tools 28.0.3
  Added test aapt output files for build-tools 28.0.3
  update.py: Still aapt output parsing, setting regex to catch 'name=' without prefixes, needed for build-tools 28.0.3

fdroid/fdroidserver!591
2019-01-21 22:39:35 +01:00
Francesco Cervigni
984d276c1b Added test aapt output files for build-tools 28.0.3 2019-01-13 19:27:02 +01:00
Marcus Hoffmann
a5ec1703c9 Revert "lint: check if CurrentVersion is older than oldest build entry"
This reverts commit 78473e7dab.

This is currently blocked by some apps in fdroiddata being reset to a
broken state.
2019-01-10 14:48:52 +01:00
Marcus Hoffmann
e1c547cfdf Revert "Revert "Merge branch 'random-fixes' into 'master'""
This reverts commit f6f2fb0b89.

Only one of the included commit should have been reverted.
2019-01-10 14:48:29 +01:00
Marcus
f6f2fb0b89 Revert "Merge branch 'random-fixes' into 'master'"
This reverts merge request !605
2019-01-10 13:46:49 +00:00
Francesco Cervigni
64445520b0 update.py: Still aapt output parsing, setting regex to catch 'name=' without prefixes, needed for build-tools 28.0.3
build-tools 28.0.3 added a new field in the end 'compileSdkVersionCodename=', which also accidentally ends with the string 'name='.
The purpose of this regex was to catch the 'packageName' field, which is in ht eaapt ouput the exact ' name=', therefore added whe non-caracter \W prefix match.

sample aapt output (28.0.3):
package: name='com.a.b.app' versionCode='1' versionName='1.0' compileSdkVersion='28' compileSdkVersionCodename='9'

previously, regex was catching second occurence, so '9'

See merge request !582
2019-01-08 22:46:47 +01:00
Hans-Christoph Steiner
76b026e572 lint: check whether the locale was included in an f-droid.org URL 2019-01-03 21:00:01 +01:00
Hans-Christoph Steiner
78473e7dab lint: check if CurrentVersion is older than oldest build entry
This causes F-Droid to never install this app, except for manually via the
Versions list in AppDetails.
2019-01-03 21:00:01 +01:00
Hans-Christoph Steiner
a2aef721d8 add timeout to net.http_get() and index.download_repo_index()
The requests docs recommend this:
http://docs.python-requests.org/en/master/user/quickstart/#timeouts

And mirror-monitor was hanging forever on a bad mirror.
2019-01-03 20:26:40 +01:00
Hans-Christoph Steiner
33b097a3aa deploy: upload to VirusTotal using index-v1.json or index-v1.jar
* A mirror of the repo will have only index-v1.jar.
* A buildserver with separate signing might have only index-v1.json
2018-12-17 14:33:02 +01:00
Hans-Christoph Steiner
869cc114a3 create index.get_index_from_jar() for parsing index-v1.json
Also removes redundant comments, should be in the doc string!
2018-12-17 14:30:05 +01:00
Hans-Christoph Steiner
13016c5d63 verify: add --output-json option to generate status reports
This has to do custom dict/encoder/decoder tricks in order to make sure
that each run of `fdroid verify` on the same files doesn't add duplicate
entries.
2018-12-07 13:13:10 +01:00
Hans-Christoph Steiner
e85256e248 verify: rename var remoteapk to remote_apk to fit other style 2018-12-07 11:19:26 +01:00
Hans-Christoph Steiner
51d961630b fix aapt scraping of <uses-permission> with maxSdkVersion
3e0d1beb09 changed this logic a bit, and it
wasn't quite right.  Then changing the SDK Versions to integers everywhere
seemed to bring this out more.
2018-12-06 13:42:45 +01:00
Hans-Christoph Steiner
653d5fbd88 update: fix running on trusty and things with old PIL/Pillow 2018-12-06 12:49:48 +01:00
Michael Pöhn
31ca2092a1 yaml parsing: script build flags can now be lists 2018-11-29 21:53:37 +01:00
Michael Pöhn
c15a7508e7 write yaml script metadata as lists 2018-11-29 21:53:37 +01:00
Michael Pöhn
afdc0be954 yaml build flag prebuild: keep writing string rathern than list of strings when theres just 1 entry 2018-11-29 21:53:37 +01:00
Michael Pöhn
942de28fa5 yaml metadata: split prebuild build field to list 2018-11-29 21:53:37 +01:00
Michael Pöhn
ddcab79817 fix .vagrant directory based vm provider lookup 2018-11-16 16:53:01 +01:00
Hans-Christoph Steiner
1f346b3149 force_exit() to make exit work with all fdroid build conditions
The build command has to use some threading stuff to handle the timeout and
locks.  This seems to prevent the command from exiting, unless this hack is
used.
2018-11-14 14:27:32 +01:00
Hans-Christoph Steiner
117d63cca5 update: use 'replace' mode to handle non-UTF8 description files
Fastlane/Triple-T app store metadata files must be in UTF-8.  Before this
would crash if they were not.  This changes the handling to just replace
the non-UTF8 chars with a �.  Here's the stacktrace:

CRITICAL: Unknown exception found!
Traceback (most recent call last):
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 164, in <module>
    main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 159, in main
    raise e
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 138, in main
    mod.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2010, in main
    insert_localized_app_metadata(apps)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 885, in insert_localized_app_metadata
    os.path.join(root, f))
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 700, in _set_localized_text_entry
    text = fp.read()[:limit]
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 94: invalid start byte
2018-10-19 15:06:16 +02:00
Hans-Christoph Steiner
57556aceee remove redundant open() arg: encoding='utf8'
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
2018-10-19 15:01:34 +02:00
Hans-Christoph Steiner
1d13bbf6b6 import: ensure categories from CLI flag are always a list
categories must always be a list in YAML  closes #472
2018-10-11 11:59:42 +02:00
Hans-Christoph Steiner
bfdf581201 import: use valid placeholder values for versionCode/versionName
fdroid/fdroidserver!559
closes fdroid/fdroidserver#548
2018-10-10 16:02:34 +02:00
Michael Pöhn
aea2589c67 metadata: remove redundant build flags list creation
picked out of fdroid/fdroidserver!559
2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner
8b251da79f update: do not set targetSdkVersion if the APK is missing it
699b3e4c69 got it wrong for targetSdkVersion.
Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for
com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk.
F-Droid never really supported running on android-1 or android-2, so it
seems pointless to debug support for them.
2018-10-10 15:29:02 +02:00
Hans-Christoph Steiner
58b14279af rewritemeta: rewrite AntiFeature str entries like Categories 2018-10-09 23:49:27 +02:00
Hans-Christoph Steiner
6e6afeec65 unset Categories should be empty by default, not ['None']
This is an ancient, bizarre vestige.  This makes the TYPE_LIST fields
behave the same, e.g. AntiFeatures and Categories.
2018-10-09 23:39:09 +02:00
Hans-Christoph Steiner
85993eb2f8 lint: check fields for proper type, e.g. list vs. string
fdroid/fdroidserver#578
2018-10-09 23:31:24 +02:00
Hans-Christoph Steiner
2113e29082 throw error on ancient, obsolete BuildVersion/UseBuilt metadata fields 2018-10-09 22:23:42 +02:00
Hans-Christoph Steiner
d2216b756b fix typos in strings 2018-09-24 17:09:15 +02:00
Hans-Christoph Steiner
6b57cb6b7c fix strict Application ID checks
* upper case letters are allowed at all positions
* there must be a "." separator
2018-09-24 17:07:27 +02:00
Hans-Christoph Steiner
6cf8fec057 update: correct error message about Android AppID 2018-09-24 17:07:27 +02:00
Hans-Christoph Steiner
11d46072ab use androguard primitives to speed up finding debuggable flag
androguard parses the whole APK before handing the instance back, this uses
the primitives to just find the <application android:debuggable=""> value,
then stop parsing.

#557
2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
a3cecc16a3 use partial androguard binary XML parsing to speed up APK ID lookup
Normally, androguard parses the entire APK before it is possible to get any
values from it.  This uses androguard primitives to only attempt to parse
the AndroidManifest.xml, then to quit as soon as it gets what it needs.
This greatly speeds up the parsing (1 minute vs 60 minutes).

fdroid/fdroidserver#557
2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
fa09337b4b APK_ID_TRIPLET_REGEX only matches first line of aapt output
Stop expensive aapt parsing after the first line when looking with
APK_ID_TRIPLET_REGEX.  As is seen with the `aapt dump badging` output files
in tests/build-tools/, the first line is the only line that will ever match.

#557
2018-09-21 14:56:32 +02:00
Hans-Christoph Steiner
4c4da3d176 update: remove 'SET' debug antifeature, closes #565
oops, this slipped in in 3011953d0e

Bump the metadata version to purge the cache
2018-09-18 10:57:10 +02:00
Hans-Christoph Steiner
3e0d1beb09 update: finish porting permissions regex to remove group names
Finish the job from 3011953d0e

closes #568
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
807bf3d26b build: reuse common methods for getting metadata from APKs
This splits out the code that gets the list of native ABIs supported, then
uses the standard methods for the rest.
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
487c4d02f3 handle package: line output from aapt v28
fdroid/fdroiddata!3484
fdroid/fdroiddata!3562
fdroid/fdroidserver!548
2018-09-18 10:55:51 +02:00
Michael Pöhn
c95cbfdf9c Merge branch 'use_available_buildserver_box' into 'master'
vmtools: use whatever buildserver box is available

Closes #497

See merge request fdroid/fdroidserver!574
2018-09-18 08:52:51 +00:00
Michael Pöhn
c005d8c5f4 more detailed error message and handling when fetching buildserverid in build.py 2018-09-13 11:11:18 +02:00
Marcus Hoffmann
1b95452c14 vmtools: use whatever buildserver box is available
When we don't have an already working builder/.vagrant directory
look for available buildserver vagrant boxes because we'll need to
import one.

Abort if there's no buildserver box.

Fixes: #497
2018-09-12 14:12:12 +02:00
Marcus Hoffmann
1e90b6118e vmtools: fix comment typos 2018-09-12 12:59:11 +02:00
Marcus Hoffmann
b0cce0b8c4 vmtools: actually use the chosen provider
Fixes #561
2018-09-12 12:59:11 +02:00
Hans-Christoph Steiner
9d12b1dc61 add strict, tested validation of Android/F-Droid package names
Android has stricter rules than Java for Package Names, but anything the
Python regex thinks is valid must be valid according to Java's rules too.

https://developer.android.com/studio/build/application-id
2018-09-07 14:17:39 +02:00