These were both spamming the output with lots of confusing messages, even
when --verbose was not used. Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.
closes#405
os.rename() only works if source and destination are on the same file
system, shutil.move() works across file systems.
OSError: [Errno 18] Invalid cross-device link: '/builds/eighthave/fdroidclient/app/build/outputs/apk/app-debug.apk' -> '/tmp/tmp966vh75f/tmp.apk'
If we allow SSH, then we'd have to manage known_hosts.
All VCS and submodule URLs should use HTTPS. SSH URLs have security vulns:
https://blogs.msdn.microsoft.com/devops/2017/08/15/git-vulnerability-with-submodules/https://www.theregister.co.uk/2017/08/13/ssh_flaw_in_git_mercurial_svn/
CVE-2017-1000117
I did a manual scan of the setup on jenkins.debian.net to see if I could
find any suspicious URLs. Looks good so far. This is what I used:
find . -type f -print0 |xargs -0 grep -Eo 'ssh[:+][svn/]+...................'
find . -type f -print0 |xargs -0 grep -Eo 'ssh://-[^ "]+'
Also, some ssh://_ URLs in submodules might still work, because of the URL
rewriting in fdbfb4d1. But https://-oProxyCommand=pwnme does not really do
anything, unlike ssh://-oProxyCommand=pwnme
Later revisions might have removed the submodules so we want to keep
going when there are no submodules present.
We still abort when there is an error initializing submodules.
Fixesfdroid/fdroidserver#231
Something is preventing `fdroid build --all` from exiting after a long
run. @bubu, @uniqx and I think it is because of the use of
AsynchronousFileReader, somehow it's thread does not exit. So the
workaround for now is to just try a hard exit instead of waiting for
things to finish cleanly with `sys.exit(0)`.
https://jenkins.debian.net/job/reproducible_fdroid_build_apps/94/console
We never allow git via SSH or password/key access, and right now, this
causes things to hang forever. This sets things up to fail quickly
with invalid ssh connections.
BatchMode=yes - passphrase/password querying will be disabled.
StrictHostKeyChecking=yes - never automatically prompt, or add host keys to
the ~/.ssh/known_hosts file, and refuse to connect to hosts whose host key
has changed.
If a git fetch/clone/submodule URL points to gitlab, github, bitbucket, etc
and that repo does not exist any more, those services will prompt the user
for a username/password so that the service can check if its a private
repo. Private repos show up the same as non-existent repos. This employs
two techniques for making sure that git never waits at those prompts. It
instead should just fail immediately. The buildserver has been hanging on
these prompts forever, until manually killed. This change will apply to
updates both on the buildserver host, and the buildserver guest vm.
This uses the "insteadOf" git config option to rewrite URLs to always use
HTTPS and then include a fake username/password so that git will use those
in the prompts and fail immediately. This trick has been in use on the
verification server for a long while and has been working well. It has
also been used on jenkins.debian.net in the host.
https://f-droid.org/en/docs/Verification_Server/
It also includes GIT_TERMINAL_PROMPT, which also prevents the bad behavior,
which was added in git 2.3.
https://github.com/blog/1957-git-2-3-has-been-released
Debug logging is not saved in buildserver logs. Also move this line into
build.py's main() to avoid duplication: common.setup_vcs(app) may be
called from parse_metadata() too.
Since this code was not setting the "create system" and "compress type",
Python uses it's defaults. Those will be different than what the Android
tools produces if this is run on UNIX. The Android tools uses the bare
bones "Windows" ZIP format, e.g. no permissions, etc.
For example:
https://verification.f-droid.org/eu.siacs.conversations_234.apk.diffoscope.html
This came about testing on OSX, where there are often multiple versions of
the JDK installed. This was choosing the oldest version. It should
choose the most recent version.
This fixes a problem when there is a broken symlink with the name
AndroidManifest.xml which will then lead to a crash when trying to open
it for trying to remove the debuggable flag.
This fixes all the bugs I could find that prevented fdroid from
handling files with spaces in them. This is more important now that
fdroid supports random media files, and Repomaker
apksigner v0.7 (build-tools 26.0.1), Google made it require that the
AndroidManifest.xml was present in the archive before it verifies the
signature. So this needs to stick with the jarsigner hack for JARs.
os.mknod() fails on OSX with:
Traceback (most recent call last):
File "/Users/travis/build/fdroidtravis/fdroidserver/tests/../fdroid", line 154, in <module>
main()
File "/Users/travis/build/fdroidtravis/fdroidserver/tests/../fdroid", line 130, in main
mod.main()
File "/Users/travis/build/fdroidtravis/fdroidserver/fdroidserver/update.py", line 1768, in main
common.write_to_config(config, 'repo_keyalias', config['repo_keyalias'])
File "/Users/travis/build/fdroidtravis/fdroidserver/fdroidserver/common.py", line 2328, in write_to_config
os.mknod(cfg)
PermissionError: [Errno 1] Operation not permitted
