Package repos come from untrusted sources, in terms of the buildserver. They
should be handled in VMs and containers as much as possible to avoid
vulnerabilities. As far as I could tell, `fdroid update` only has a single
place where it executes any VCS system: if there is .fdroid.yml present in
a package repo, then it will fetch the commit ID using git.
For better security properties, this implements a simple function to just
read the files to get that commit ID. The function that executes git to do
the same thing is relabeled "unsafe". That is used for status JSON
everywhere, but that runs on fdroiddata.git and fdroidserver.git, which are
trusted repos.
The unsafe version is also used in places where git.Repo() is needed for
other things.
Some places in the code that need this, like verify.py, do not have
app and build instances, but do have appid and versionCode. And
fdroidserver/build.py is going away.
There is no longer any reason for these to be intertwined.
This deliberately avoids touching some files as much as possible because
they are super tangled and due to be replaced. Those files are:
* fdroidserver/build.py
* fdroidserver/update.py
# Conflicts:
# tests/testcommon.py
# Conflicts:
# fdroidserver/btlog.py
# fdroidserver/import_subcommand.py
This adds a check for "testOnly" to the existing "debuggable" check, since
they are very similar. We should really be refactoring all the checks into
a more reasonable setup. Since "debuggable" and "testOnly" are both set in
the same place (`<application>` in _AndroidManifest.xml_) and are both set
by the same process (running debug builds), I thought it would be OK to
include both in the same place. Plus it was a one-line change.
This SSH connection is only ever to the Vagrant VM on the same machine.
>> Issue: [B507:ssh_no_host_key_verification] Paramiko call with policy set to automatically trust the unknown host key.
Severity: High Confidence: Medium
CWE: CWE-295 (https://cwe.mitre.org/data/definitions/295.html)
More Info: https://bandit.readthedocs.io/en/1.7.6/plugins/b507_ssh_no_host_key_verification.html
Location: ./fdroidserver/build.py:104:8
103 sshs = paramiko.SSHClient()
104 sshs.set_missing_host_key_policy(paramiko.AutoAddPolicy())
105 sshs.connect(sshinfo['hostname'], username=sshinfo['user'],
Found while compiling io.privatestorage.privatestoragemobile_9:
Traceback (most recent call last):
File "/home/fdroid/fdroidserver/fdroidserver/build.py", line 1053, in main
if trybuild(app, build, build_dir, output_dir, log_dir,
File "/home/fdroid/fdroidserver/fdroidserver/build.py", line 813, in trybuild
build_server(app, build, vcs, build_dir, output_dir, log_dir, force)
File "/home/fdroid/fdroidserver/fdroidserver/build.py", line 253, in build_server
logging.debug("buildserver > " + str(line, 'utf-8').rstrip())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfa in position 8: invalid start byte
The builder should check the `AllowedAPKSigningKeys` at build time, so
that the CI can check if somebody gives a wrong value that doesn't match
a compared RB binary. In the event it fails, it gives useful
information, and in the event it succeeds, it makes it clear that this
build has verification back to the developer's original key.
Also, add tests for this to the test suite.
The build can get stuck when running into the timeout and paramiko not
noticing it. This can be reproduced by setting a small build timeout for
an app. I tried setting set_keepalive() but it did not help. This
forcefully closes the ssh channel to the VM so the fdroid build will
continue. As we destroy the VM in the next step anyhow I think this is
fine.
makebuildserver.config.py is no more, builder/Vagrantfile is now where the
CPU and memory is configured for the buildserver VM. In fact, that was
always the actual place, the makebuildserver.config.py thing was just
confused.
This should have been part of !1222
Make sudo, init prebuild, build and Prepare fields lists and only
concatenate them with '; ' before execution. This allows arbitrary
commands inside the fileds (even && and ';') as we don't need to split
the commands again for rewritemeta.
dfdd2913 the logic to only write the status json upon build but before
the build happens. This makes sure that it is written after a
successful build or exception.
23b0b6bc added logging in case of an FDroidException on the build
server. This broke the log of apps that fail to build.
Prior to 23b0b6bc a failing build in the VM triggered a BuildException
on the server side but the build output was written to the log in the
finally part of build_server(). After 23b0b6bc the finally part writs
the build as well but the BuildException is caught in main() and the
build log is overwritten with the empty exception content. This patch
always adds the build log to the exception so it is written to the log.
Closes: #882
This happened with us.spotco.fennec_dos_2912000 where no build log was
saved in the repo but the wiki has an error text. The error text was:
"Command '['rsync', '--recursive', '--perms', '--links', '--quiet', '--rsh=ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=FATAL -o IdentitiesOnly=yes -o PasswordAuthentication=no -p 2222 -i /home/fbuild/.vagrant.d/boxes/buildserver/0/virtualbox/vagrant_private_key', 'build/srclib/MozFennec', 'vagrant@127.0.0.1:/home/vagrant/build/srclib']' returned non-zero exit status 255."
Which is generated by the rsync subprocess call in build_server(). I
assume that it threw an Exception (not an FdroidException, because the
string "Build completed at" is not in the wiki site) that was caught in
main().
