For some reason, the parser stopped working intermittently, even
though the format has been the same since aapt 23 or earlier. Then
also, some of the test cases pointed to symlinks that were no longer
generated, and one test app now has a blank versionName.
Strange that this wasn't caught in the gitlab-ci runs. !484
FAIL: test_get_api_id_aapt (__main__.CommonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./common.TestCase", line 578, in testA_get_api_id_aapt
self.assertEqual(versionName, vn)
AssertionError: '0.1' != "0.1' platformBuildVersionName='4.3.1-1425645"
- 0.1
+ 0.1' platformBuildVersionName='4.3.1-1425645
This file is written freshly each time before use, so it does not need
to be ekpt around. It was the only file making the fdroiddata.git
repo dirty on the f-droid.org infrastructure.
This also adds stricter file permissions to avoid an attacker changing
those settings during operation.
"SVN follows HTTP 301 redirects to svn+ssh:// URLs. As a result, an
innocent looking HTTP URL can be used to trigger a Command Execution with a
301 redirect."
https://blog.recurity-labs.com/2017-08-10/scm-vulns.html#third-round-svn-and-mercurial
I scanned fdroiddata and found no suspicious redirects. Here's how:
grep -A1 '^Repo *Type: *git-svn' *.txt *.yml| sed -n 's,.*Repo:\(.*\),\1,p' > /tmp/urls.txt
import requests
with open('/tmp/urls.txt') as fp:
for line in fp:
try:
r = requests.head(line.strip())
print(r.status_code, line)
except requests.exceptions.SSLError:
print('SSLError', line)
This should have less of a change of matching bad things.
thanks to @stf for the report. I ran tests comparing the original vs these
new patterns, and it was a 100% match. So at least it didn't make things
worse.
Here's the test script:
#!/usr/bin/env python3
import os
import re
old_vcsearch_g = re.compile(r'''.*[Vv]ersionCode[ =]+["']*([0-9]+)["']*''').search
old_vnsearch_g = re.compile(r'.*[Vv]ersionName *=* *(["\'])((?:(?=(\\?))\3.)*?)\1.*').search
old_psearch_g = re.compile(r'.*(packageName|applicationId) *=* *["\']([^"]+)["\'].*').search
new_vcsearch_g = re.compile(r'''.*[Vv]ersionCode\s*=?\s*["']*([0-9]+)["']*''').search
new_vnsearch_g = re.compile(r'''.*[Vv]ersionName\s*=?\s*(["'])((?:(?=(\\?))\3.)*?)\1.*''').search
new_psearch_g = re.compile(r'''.*(packageName|applicationId)\s*=*\s*["']([^"']+)["'].*''').search
old = re.compile(r'.*(packageName|applicationId) *=* *["\']([^"]+)["\'].*').search
new = re.compile(r'''.*(packageName|applicationId)\s*=*\s*["']([^"']+)["'].*''').search
for root, dirs, files in os.walk('build'):
for f in files:
if f.endswith('.gradle'):
with open(os.path.join(root, f)) as fp:
for line in fp:
for old, new in ((old_vcsearch_g, new_vcsearch_g),
(old_vnsearch_g, new_vnsearch_g),
(old_psearch_g, new_psearch_g)):
found_old = old(line)
found_new = new(line)
oldresult = None
newresult = None
if found_old or found_new:
if found_old:
oldresult = found_old.groups()
#print('OLD', oldresult)
if found_new:
newresult = found_new.groups()
#print('NEW', newresult)
if oldresult != newresult:
print('--------------------------------')
print(f, oldresult, newresult)
git-svn will put up the "Reject/Accept" prompt if it encounters a bad HTTPS
certificate. I could find no way to stop it from doing that. So instead,
this checks the HTTPS connection with an HTTP HEAD request first.
Subversion does not verify each commit as strongly as git does, so HTTPS is
really important. Also, there is the possibility of injecting code into
`fdroid checkupdate` calls if plain HTTP is used.
This uses both the env vars and the command line options to ensure
that it works with as many versions of git as possible. Also, git-svn
uses the env vars, but not necessarily the command line options.
This uses /bin/true to pretend that it succesfully got the password.
If password auth is truly required, then it will fail further on down
the line.
We always want to run all utilities non-interactively. By default
subprocess.Popen() inherits stdin descriptor from parent process, i.e.
when fdroid is run from an interactive shell, subprocesses may expect
input from it.
Reading from /dev/null immediately returns EOF, failing any user prompt
and preventing us from hang.
This is a quick and very incomplete addition of '--' to command line calls
to source VCSs like git and hg that could manipulated by malicious
tag/branch names or other vectors.
These were all manually tested by calling the command lines on my own
machine.
The currently included Qt has known security issues and is outdated. This
can now be replaced by downloading and installing the Qt installer using
the sudo= build field. @relan's provisioner system will also replace this
once that's done. There are only two apps that currently use the Qt stuff:
* csd.qtproject.minesweeper
* org.openorienteering.mapper
These were both spamming the output with lots of confusing messages, even
when --verbose was not used. Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.
closes#405
os.rename() only works if source and destination are on the same file
system, shutil.move() works across file systems.
OSError: [Errno 18] Invalid cross-device link: '/builds/eighthave/fdroidclient/app/build/outputs/apk/app-debug.apk' -> '/tmp/tmp966vh75f/tmp.apk'
If we allow SSH, then we'd have to manage known_hosts.
All VCS and submodule URLs should use HTTPS. SSH URLs have security vulns:
https://blogs.msdn.microsoft.com/devops/2017/08/15/git-vulnerability-with-submodules/https://www.theregister.co.uk/2017/08/13/ssh_flaw_in_git_mercurial_svn/
CVE-2017-1000117
I did a manual scan of the setup on jenkins.debian.net to see if I could
find any suspicious URLs. Looks good so far. This is what I used:
find . -type f -print0 |xargs -0 grep -Eo 'ssh[:+][svn/]+...................'
find . -type f -print0 |xargs -0 grep -Eo 'ssh://-[^ "]+'
Also, some ssh://_ URLs in submodules might still work, because of the URL
rewriting in fdbfb4d1. But https://-oProxyCommand=pwnme does not really do
anything, unlike ssh://-oProxyCommand=pwnme
Later revisions might have removed the submodules so we want to keep
going when there are no submodules present.
We still abort when there is an error initializing submodules.
Fixesfdroid/fdroidserver#231
Something is preventing `fdroid build --all` from exiting after a long
run. @bubu, @uniqx and I think it is because of the use of
AsynchronousFileReader, somehow it's thread does not exit. So the
workaround for now is to just try a hard exit instead of waiting for
things to finish cleanly with `sys.exit(0)`.
https://jenkins.debian.net/job/reproducible_fdroid_build_apps/94/console
