This commit fixes two bugs with reproducible builds:
* Files added by the buildserver to META-INF (fdroidserverid and buildserverid)
were causing signature verification to always fail when --on-server was used.
Since they are not needed anymore, they are no longer added to APKs.
* When showing a diff between both APK files, `jar xf` did not extract
the full APK properly which was causing useless diffs.
Instead of using jar, python's zipfile library is used instead.
Currently f-droid.org has a lot of pending builds and big delays
between "fdroid build --all" runs. Bump overall build time limit from
12 hours to 36 hours to increase buildserver throughput.
There are all sorts of unfiltered user inputs like tag and branch names in
source repos. If those names are fed into popen calls that use shell=True,
that opens up a wide range of exploits. All core operations should never
use shell=True.
This code has never been used and contains some insecure uses of shell=True
Building Kivy apps should be done with the buildozer=yes method. The
buildozer method should probably be moved to a provisioner once that is in
place.
Only start new builds for 12 hours. This ensures we publish new builds
often enough even on long backlogs.
This could be made configurable at a later point.
ERROR: Could not build app org.fdroid.fdroid due to unknown error: Traceback (most recent call last):
File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 1202, in main
options.onserver, options.refresh):
File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 972, in trybuild
build_server(app, build, vcs, build_dir, output_dir, log_dir, force)
File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 82, in build_server
logging.debug(_('Fetched buildserverid from VM: ') + buildserverid)
TypeError: Can't convert 'bytes' object to str implicitly
We remove the whole "build" directory while cleaning source code tree
because Gradle can leave there files even after "gradle clean". But some
projects (Mozilla Fennec) actually have useful stuff checked into VCS
under the "build" directory.
Remove only those subdirectories that we known for sure are leftovers
from Gradle.
Fixesfdroid/fdroidserver#438.
When `fdroid build` is run using the buildserver, it should fetch the
buildserverid on the first build.
Seems this was really a silly bug in 837fc99d74
Since `fdroid build --all` can run a long time, knowing when that command
was started will be very useful information for figuring out what the build
server is doing.
This needed now because the buildserver is hanging so often, that we are
often going a week or more without any builds published. Perhaps this is
only temporary, or maybe we will want to flush this feature out more as a
standard thing. But we really need it for now to at least get some builds
out on a daily basis.
Since the website deploy is also triggered by this cycle, making the build
finish more often means the website will be published more often.
Fixes bb758d3f, spotted by @bubu:
DEBUG: buildserver > DEBUG: > sudo apt-get -y purge sudo
DEBUG: buildserver > Reading package lists...
DEBUG: buildserver > Building dependency tree...
DEBUG: buildserver > Reading state information...
DEBUG: buildserver > The following packages will be REMOVED:
DEBUG: buildserver > sudo*
DEBUG: buildserver > 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
DEBUG: buildserver > After this operation, 2,391 kB disk space will be freed.
(Reading database ... 68491 files and directories currently installed.)
DEBUG: buildserver > Removing sudo (1.8.10p3-1+deb8u4) ...
DEBUG: buildserver > You have asked that the sudo package be removed,
DEBUG: buildserver > but no root password has been set.
DEBUG: buildserver > Without sudo, you may not be able to gain administrative privileges.
DEBUG: buildserver >
DEBUG: buildserver > If you would prefer to access the root account with su(1)
DEBUG: buildserver > or by logging in directly,
DEBUG: buildserver > you must set a root password with "sudo passwd".
DEBUG: buildserver >
DEBUG: buildserver > If you have arranged other means to access the root account,
DEBUG: buildserver > and you are sure this is what you want,
DEBUG: buildserver > you may bypass this check by setting an environment variable
DEBUG: buildserver > (export SUDO_FORCE_REMOVE=yes).
DEBUG: buildserver >
DEBUG: buildserver > Refusing to remove sudo.
DEBUG: buildserver > dpkg: error processing package sudo (--purge):
DEBUG: buildserver > subprocess installed pre-removal script returned error exit status 1
DEBUG: buildserver > Errors were encountered while processing:
DEBUG: buildserver > sudo
DEBUG: buildserver > E: Sub-process /usr/bin/dpkg returned an error code (1)
Once `sudo` has been used to execute the commands in sudo=, then it should
be removed from the build VM. That prevents any other part of the build
from using sudo. That means that all commands run with `sudo` must be
committed to fdroiddata.git, making them very visible.
closes#424
This rsync hung because of an SSH unknown key prompt. Since this is just
the vm host sshing to the vm guest, it is not essential to check the host
keys.
Save rsync error output and combine that with the command invocation
into an FDroidException which can be logged to the wiki.
This additionally sets -q for rsync to only print errors.
Something is preventing `fdroid build --all` from exiting after a long
run. @bubu, @uniqx and I think it is because of the use of
AsynchronousFileReader, somehow it's thread does not exit. So the
workaround for now is to just try a hard exit instead of waiting for
things to finish cleanly with `sys.exit(0)`.
https://jenkins.debian.net/job/reproducible_fdroid_build_apps/94/console
When running `fdroid build --all` on a buildserver with thousands of apps,
it was frequently hitting the open file limit. This increases the open
file limit based on how many apps are being process. It is doubled to
provide a margin of safety.
There are probably open file leaks which ideally would be fixed, but this
is also useful to make things more resilient to all the random stuff apps
include in their build systems.
Sending many small files with paramiko's sftp implementation is quite
slow. There seems to be no way around this, other projects (ansible)
recommend to use rsync in this case.
Our sourcecode directories sometimes have a LOT of files, it can take
up to an hour to copy all required things inside the buildserver.
Instead we just use rsync with the ssh options we get from vagrant.
For rsync specific options we use:
* sync permissions as before
* sync symlinks (was done partially before)
* don't sync hardlinks and ownership as these don't make for things
coming from a VCS
Closesfdroid/fdroidserver#227.
Debug logging is not saved in buildserver logs. Also move this line into
build.py's main() to avoid duplication: common.setup_vcs(app) may be
called from parse_metadata() too.
An app may not have a top-level build.gradle file, while Gradle still
creates top-level .gradle/ directory. When build.gradle is absent,
fdroidserver will not remove .gradle/, scanner will find binaries in it
and fail the whole build.
Fix this by also checking for settings.gradle file (should always exist
in the top-level directory) in addition to build.gradle.
This allows all the text to be localized via Weblate. This is a quick
overview of all the strings, but there are certainly some that were left
out.
closes#342
There were multiple conventions used in the code, but mostly it was already
using the convention from the docs, so this converts things to using that
convention:
https://docs.python.org/3/library/os.html#os.walk
