Commit graph

3118 commits

Author SHA1 Message Date
Hans-Christoph Steiner
40fac10ebc update: extract and store XML icons
These can then be used by the client.

#344
#392
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
5713b54e0b update: find alternate PNGs for apps that have an XML app icon
Apps can now use an XML icon, but if the app supports older Android
versions, it'll also contain PNG versions of the same icon.  This finds
those PNGs and uses them instead.

#344
closes #392
fdroiddata#913
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
1f99a27a9c Revert "wiki: include per-app link to all related activity on gitlab.com"
This reverts commit a57f17b276.

Turns out this was totally useless, this can be fully handled in
https://f-droid.org/wiki/page/Template:App since it already has the
Application ID there.
2018-02-12 12:15:56 +01:00
Hans-Christoph Steiner
dc26e7f79f git-svn: check HTTPS connection with Python Requests
git-svn will put up the "Reject/Accept" prompt if it encounters a bad HTTPS
certificate.  I could find no way to stop it from doing that.  So instead,
this checks the HTTPS connection with an HTTP HEAD request first.
2018-02-12 12:07:24 +01:00
Hans-Christoph Steiner
a1075f45cc git-svn: require working HTTPS for all Subversion URLs
Subversion does not verify each commit as strongly as git does, so HTTPS is
really important.  Also, there is the possibility of injecting code into
`fdroid checkupdate` calls if plain HTTP is used.
2018-02-12 12:07:24 +01:00
Hans-Christoph Steiner
dd93ee6c9b git: use /bin/true for 'askpass' to prevent all password prompts
This uses both the env vars and the command line options to ensure
that it works with as many versions of git as possible.  Also, git-svn
uses the env vars, but not necessarily the command line options.

This uses /bin/true to pretend that it succesfully got the password.
If password auth is truly required, then it will fail further on down
the line.
2018-02-12 12:07:24 +01:00
Hans-Christoph Steiner
574fa15fce git: make explicit that git configs are calling cmd line utilities
These are not boolean values, but command line utilities which return a
guaranteed exit status.
2018-02-12 12:07:24 +01:00
Hans-Christoph Steiner
c67ed5e85f git-svn: use '--' to isolate user input in command lines 2018-02-12 12:07:24 +01:00
Hans-Christoph Steiner
a40b79eb9d import: fix crash when importing from local git repo 2018-02-12 12:05:27 +01:00
Hans-Christoph Steiner
559adb207b import: get build dir from settings.gradle
Most projects set a single build dir in settings.gradle, so its worth
trying here.
2018-02-12 12:05:27 +01:00
Hans-Christoph Steiner
64938d9104 import: switch to .yml as default metadata format 2018-02-12 12:05:27 +01:00
Hans-Christoph Steiner
62db94d96e Merge branch 'build-36h' into 'master'
build: set overall timeout to 36 hours

See merge request fdroid/fdroidserver!460
2018-02-12 11:06:34 +00:00
Hans-Christoph Steiner
d4e64bda4c Merge branch 'timeout_flag' into 'master'
build: clear timeout flag before every build

See merge request fdroid/fdroidserver!455
2018-02-12 12:01:46 +01:00
Hans-Christoph Steiner
2238b5cafd Merge branch 'metadata-write-int' into 'master'
metadata: write values of TYPE_INT fields

See merge request fdroid/fdroidserver!456
2018-02-12 10:57:38 +00:00
Hans-Christoph Steiner
ef9b89f4ec Merge branch 'remove-ndk-r9b' into 'master'
makebuildserver: remove NDK r9b to save 1.6 GB of disk space

See merge request fdroid/fdroidserver!459
2018-02-12 10:55:02 +00:00
relan
c0f323e343 build: set overall timeout to 36 hours
Currently f-droid.org has a lot of pending builds and big delays
between "fdroid build --all" runs. Bump overall build time limit from
12 hours to 36 hours to increase buildserver throughput.
2018-02-12 13:45:53 +03:00
relan
76da21f121 makebuildserver: remove NDK r9b to save 1.6 GB of disk space
NDK r9b is used by only one app (net.gorry.android.input.nicownng) that
was last updated in 2015.
2018-02-12 09:16:55 +03:00
relan
6f295cb3d3 makebuildserver: upgrade NDK r16 to r16b 2018-02-11 10:20:42 +03:00
relan
4d56c6c262 metadata: write values of TYPE_INT fields
Otherwise we end up with empty values after "fdroid rewritemeta".
2018-02-11 09:51:53 +03:00
Marcus Hoffmann
39a5637df2
build: clear timeout flag before every build
This fixes the weird logging behaviour where after a build that timed
out all subsequent failed builds would also show up as timed out.
2018-02-07 20:47:50 +01:00
Hans-Christoph Steiner
05eef5a454 metadata: add new Translation app field for URL for contributions
This is a field requested from a bunch of translators so they can easily
find where to translate apps:
https://forum.f-droid.org/t/translation-field-for-app-entries/1403/5
2018-02-05 15:45:12 +01:00
relan
946a1461f2 common: use /dev/null as stdin when calling subprocess.Popen()
We always want to run all utilities non-interactively. By default
subprocess.Popen() inherits stdin descriptor from parent process, i.e.
when fdroid is run from an interactive shell, subprocesses may expect
input from it.

Reading from /dev/null immediately returns EOF, failing any user prompt
and preventing us from hang.
2018-02-05 15:34:42 +03:00
Hans-Christoph Steiner
5db86215f6 lint: check srclibs has name and @ in it
fdroiddata!2863
2018-01-31 13:33:36 +01:00
csagan5
427427481e Change --resetserver to --reset-server for consistency 2018-01-30 20:45:03 +01:00
csagan5
cf54c9514c Option --resetserver does nothing; have it used instead 2018-01-30 14:50:35 +01:00
relan
86f34ee70a build: make per-build hard time limit customizable
Add "timeout=n" metadata field that overrides build timeout (in seconds).
The default is 7200, i.e. 2 hours.
2018-01-30 11:14:50 +03:00
Hans-Christoph Steiner
f24cf7f71b vmtools: use standard imports: os.remove()
The rest of the code uses os.remove()
2018-01-26 10:18:42 +01:00
Hans-Christoph Steiner
4463bf238d vmtools: use standard imports: os.path.join()
The rest of the code uses os.path.join()
2018-01-26 10:18:42 +01:00
Hans-Christoph Steiner
099c98bc60 metadata: fix crash when there are duplicate metadata files
The string had the variable 'appid' while the format() call had only the
variable 'path'.  This also standardizes on 'appid' like the rest of the
code.
2018-01-26 10:18:42 +01:00
Hans-Christoph Steiner
b851d49d24 shell=True is too dangerous to allow; there are unfiltered user inputs
There are all sorts of unfiltered user inputs like tag and branch names in
source repos.  If those names are fed into popen calls that use shell=True,
that opens up a wide range of exploits.  All core operations should never
use shell=True.
2018-01-26 10:18:41 +01:00
Hans-Christoph Steiner
07cdf848d7 use '--' in source vcs calls to protect against malicious input
This is a quick and very incomplete addition of '--' to command line calls
to source VCSs like git and hg that could manipulated by malicious
tag/branch names or other vectors.

These were all manually tested by calling the command lines on my own
machine.
2018-01-26 10:18:41 +01:00
Hans-Christoph Steiner
32213ef040 scanner: allow running without versionCode and as API
This lets `fdroid scanner my.package.name` run without requiring that the
versionCode is also specified.  It also allows scanner.scan_source() to be
called as a function in the public API of fdroidserver.
2018-01-26 10:18:41 +01:00
Hans-Christoph Steiner
53f603bf30 lint: check description for forbidden HTML tags: iframe, link, script, etc. 2018-01-26 10:18:41 +01:00
Hans-Christoph Steiner
a57f17b276 wiki: include per-app link to all related activity on gitlab.com 2018-01-26 10:18:41 +01:00
Hans-Christoph Steiner
513c95894c build: remove unused, unmaintained Kivy build method
This code has never been used and contains some insecure uses of shell=True
Building Kivy apps should be done with the buildozer=yes method.  The
buildozer method should probably be moved to a provisioner once that is in
place.
2018-01-23 23:16:05 +01:00
Hans-Christoph Steiner
62ddab7edd buildserver: remove Qt installer, its huge, outdated, and being replaced
The currently included Qt has known security issues and is outdated.  This
can now be replaced by downloading and installing the Qt installer using
the sudo= build field.  @relan's provisioner system will also replace this
once that's done.  There are only two apps that currently use the Qt stuff:

* csd.qtproject.minesweeper
* org.openorienteering.mapper
2018-01-23 20:28:26 +01:00
Marcus Hoffmann
fa43066f8d
build: add global soft timeout of 12 hours
Only start new builds for 12 hours. This ensures we publish new builds
often enough even on long backlogs.

This could be made configurable at a later point.
2018-01-22 16:02:49 +01:00
Marcus Hoffmann
80e121d182
build: log timeouts to the wiki 2018-01-22 16:02:49 +01:00
Marcus Hoffmann
85985074d4
build: enable watchdog timer for each build that kills in 2 hours
This introduces locking for the commonly used vagrant functions in
vmtools because vagrant fails when another vagrant command is
already running.
2018-01-22 16:01:20 +01:00
Marcus Hoffmann
9a4f3ac019
Revert "build: bump max_apps_per_run to 50"
This reverts commit 56a53055be.

Revert "build: limit --all to 10 apps at a time"

This reverts commit afc5cc6b6a.
2018-01-22 15:53:45 +01:00
Hans-Christoph Steiner
486ee25708 wiki: log build start/stop time, command line, RAM, and processor count 2018-01-22 14:00:16 +01:00
Hans-Christoph Steiner
ef69bbff34 wiki: log server start/stop times and command line 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
f841ec745f wiki: move checkupdates wiki log to separate function 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
010f1c5029 log installed android sdk versions for update and checkupdates 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
e163c09e26 move get_android_tools_versions functions to common 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
4beb2d52e9 wiki: log update start/stop time and command line 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
5b92820ff3 wiki: fix bug updating Repository Maintenance
site.pages doesn't seem to exist anywhere, site.Pages is used throughout.
2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
22c6acc026 wiki: log appids as checkupdates goes through them 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
fc4f5a79a7 wiki: log checkupdates start/stop time and command line for each run 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
df51a6e999 common.get_wiki_timestamp() for posting timestamps to wiki log pages 2018-01-22 13:49:10 +01:00