This expands the gradle wrapper shell script used by the buildserver for
usage outside the buildserver environment. It also allows downloading
whitelisted versions of gradle if they are not yet deployed to the
buildserver by simply upsating the copy of fdroidserver (in contrast to
having to reprovision the whole buildserver).
We first move the buildserver/gradle shell script to the repo root
as gradlew-fdroid, as it's an fdroid specific gradle wrapper.
We also now sync it inside the build VM before each build.
We then add a list of whitelisted gradle distributions taken from the
makebuildserver script.
The script additionally now reads two env vars which tell it where to
expect installed versions of gradle and where it might store downloaded
gradle .zip files. Both of those are configurable from config.py. As the
first should normally just be a subdir of the second it's not exposed in
the example config.py but only used by the buildserver config.py.
Default config now uses this internal gradle wrapper but a path to a
custom wrapper or specific gradle distribution can still be set from
config.py.
Closesfdroid/fdroidserver#98
Ref: fdroid/fdroidserver#370
On versions of diffoscope before 87, like the version included in Ubuntu
xenial LTS, it would crash saying:
ValueError: max_diff_block_lines (100) cannot be smaller than max_page_diff_block_lines (128)
https://bugs.debian.org/875451
This commit fixes two bugs with reproducible builds:
* Files added by the buildserver to META-INF (fdroidserverid and buildserverid)
were causing signature verification to always fail when --on-server was used.
Since they are not needed anymore, they are no longer added to APKs.
* When showing a diff between both APK files, `jar xf` did not extract
the full APK properly which was causing useless diffs.
Instead of using jar, python's zipfile library is used instead.
For some reason, the parser stopped working intermittently, even
though the format has been the same since aapt 23 or earlier. Then
also, some of the test cases pointed to symlinks that were no longer
generated, and one test app now has a blank versionName.
Strange that this wasn't caught in the gitlab-ci runs. !484
FAIL: test_get_api_id_aapt (__main__.CommonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./common.TestCase", line 578, in testA_get_api_id_aapt
self.assertEqual(versionName, vn)
AssertionError: '0.1' != "0.1' platformBuildVersionName='4.3.1-1425645"
- 0.1
+ 0.1' platformBuildVersionName='4.3.1-1425645
This file is written freshly each time before use, so it does not need
to be ekpt around. It was the only file making the fdroiddata.git
repo dirty on the f-droid.org infrastructure.
This also adds stricter file permissions to avoid an attacker changing
those settings during operation.
"SVN follows HTTP 301 redirects to svn+ssh:// URLs. As a result, an
innocent looking HTTP URL can be used to trigger a Command Execution with a
301 redirect."
https://blog.recurity-labs.com/2017-08-10/scm-vulns.html#third-round-svn-and-mercurial
I scanned fdroiddata and found no suspicious redirects. Here's how:
grep -A1 '^Repo *Type: *git-svn' *.txt *.yml| sed -n 's,.*Repo:\(.*\),\1,p' > /tmp/urls.txt
import requests
with open('/tmp/urls.txt') as fp:
for line in fp:
try:
r = requests.head(line.strip())
print(r.status_code, line)
except requests.exceptions.SSLError:
print('SSLError', line)
This should have less of a change of matching bad things.
thanks to @stf for the report. I ran tests comparing the original vs these
new patterns, and it was a 100% match. So at least it didn't make things
worse.
Here's the test script:
#!/usr/bin/env python3
import os
import re
old_vcsearch_g = re.compile(r'''.*[Vv]ersionCode[ =]+["']*([0-9]+)["']*''').search
old_vnsearch_g = re.compile(r'.*[Vv]ersionName *=* *(["\'])((?:(?=(\\?))\3.)*?)\1.*').search
old_psearch_g = re.compile(r'.*(packageName|applicationId) *=* *["\']([^"]+)["\'].*').search
new_vcsearch_g = re.compile(r'''.*[Vv]ersionCode\s*=?\s*["']*([0-9]+)["']*''').search
new_vnsearch_g = re.compile(r'''.*[Vv]ersionName\s*=?\s*(["'])((?:(?=(\\?))\3.)*?)\1.*''').search
new_psearch_g = re.compile(r'''.*(packageName|applicationId)\s*=*\s*["']([^"']+)["'].*''').search
old = re.compile(r'.*(packageName|applicationId) *=* *["\']([^"]+)["\'].*').search
new = re.compile(r'''.*(packageName|applicationId)\s*=*\s*["']([^"']+)["'].*''').search
for root, dirs, files in os.walk('build'):
for f in files:
if f.endswith('.gradle'):
with open(os.path.join(root, f)) as fp:
for line in fp:
for old, new in ((old_vcsearch_g, new_vcsearch_g),
(old_vnsearch_g, new_vnsearch_g),
(old_psearch_g, new_psearch_g)):
found_old = old(line)
found_new = new(line)
oldresult = None
newresult = None
if found_old or found_new:
if found_old:
oldresult = found_old.groups()
#print('OLD', oldresult)
if found_new:
newresult = found_new.groups()
#print('NEW', newresult)
if oldresult != newresult:
print('--------------------------------')
print(f, oldresult, newresult)
git-svn will put up the "Reject/Accept" prompt if it encounters a bad HTTPS
certificate. I could find no way to stop it from doing that. So instead,
this checks the HTTPS connection with an HTTP HEAD request first.
Subversion does not verify each commit as strongly as git does, so HTTPS is
really important. Also, there is the possibility of injecting code into
`fdroid checkupdate` calls if plain HTTP is used.
This uses both the env vars and the command line options to ensure
that it works with as many versions of git as possible. Also, git-svn
uses the env vars, but not necessarily the command line options.
This uses /bin/true to pretend that it succesfully got the password.
If password auth is truly required, then it will fail further on down
the line.
We always want to run all utilities non-interactively. By default
subprocess.Popen() inherits stdin descriptor from parent process, i.e.
when fdroid is run from an interactive shell, subprocesses may expect
input from it.
Reading from /dev/null immediately returns EOF, failing any user prompt
and preventing us from hang.
This is a quick and very incomplete addition of '--' to command line calls
to source VCSs like git and hg that could manipulated by malicious
tag/branch names or other vectors.
These were all manually tested by calling the command lines on my own
machine.
The currently included Qt has known security issues and is outdated. This
can now be replaced by downloading and installing the Qt installer using
the sudo= build field. @relan's provisioner system will also replace this
once that's done. There are only two apps that currently use the Qt stuff:
* csd.qtproject.minesweeper
* org.openorienteering.mapper
