When running `fdroid build --all` on a buildserver with thousands of apps,
it was frequently hitting the open file limit. This increases the open
file limit based on how many apps are being process. It is doubled to
provide a margin of safety.
There are probably open file leaks which ideally would be fixed, but this
is also useful to make things more resilient to all the random stuff apps
include in their build systems.
This creates a mirror of a full repo by downloading all files listed in
the index, and the ones that are generated based on that data, e.g. icons
of different resolutions. This could be useful for setting up mirrors of
small repositories, instead of having to learn and manage rsync or
something else for mirroring. This just needs a working repo.
It uses wget in a batch mode with the aim as being as efficient as
possible. wget mirroring over HTTP is always going to be less efficient
than rsync, but it shouldn't be so bad since it uses --continue to check
whether it has already downloaded a file. I suppose it could be extended
to use ETags for a little more efficiency.
I developed this creating a test mirror of f-droid.org, which is now a bit
ironic, since I added a specific check to prevent people from using this
on f-droid.org.
Sending many small files with paramiko's sftp implementation is quite
slow. There seems to be no way around this, other projects (ansible)
recommend to use rsync in this case.
Our sourcecode directories sometimes have a LOT of files, it can take
up to an hour to copy all required things inside the buildserver.
Instead we just use rsync with the ssh options we get from vagrant.
For rsync specific options we use:
* sync permissions as before
* sync symlinks (was done partially before)
* don't sync hardlinks and ownership as these don't make for things
coming from a VCS
Closesfdroid/fdroidserver#227.
This was introduced in cd23bb933c
apparently to work around problems in vagrant 1.3.5.
This seems not to be necessary any more, I have never had a problem with
those sleeps removed.
We never allow git via SSH or password/key access, and right now, this
causes things to hang forever. This sets things up to fail quickly
with invalid ssh connections.
BatchMode=yes - passphrase/password querying will be disabled.
StrictHostKeyChecking=yes - never automatically prompt, or add host keys to
the ~/.ssh/known_hosts file, and refuse to connect to hosts whose host key
has changed.
If a git fetch/clone/submodule URL points to gitlab, github, bitbucket, etc
and that repo does not exist any more, those services will prompt the user
for a username/password so that the service can check if its a private
repo. Private repos show up the same as non-existent repos. This employs
two techniques for making sure that git never waits at those prompts. It
instead should just fail immediately. The buildserver has been hanging on
these prompts forever, until manually killed. This change will apply to
updates both on the buildserver host, and the buildserver guest vm.
This uses the "insteadOf" git config option to rewrite URLs to always use
HTTPS and then include a fake username/password so that git will use those
in the prompts and fail immediately. This trick has been in use on the
verification server for a long while and has been working well. It has
also been used on jenkins.debian.net in the host.
https://f-droid.org/en/docs/Verification_Server/
It also includes GIT_TERMINAL_PROMPT, which also prevents the bad behavior,
which was added in git 2.3.
https://github.com/blog/1957-git-2-3-has-been-released
Debug logging is not saved in buildserver logs. Also move this line into
build.py's main() to avoid duplication: common.setup_vcs(app) may be
called from parse_metadata() too.
Since this code was not setting the "create system" and "compress type",
Python uses it's defaults. Those will be different than what the Android
tools produces if this is run on UNIX. The Android tools uses the bare
bones "Windows" ZIP format, e.g. no permissions, etc.
For example:
https://verification.f-droid.org/eu.siacs.conversations_234.apk.diffoscope.html
This came about testing on OSX, where there are often multiple versions of
the JDK installed. This was choosing the oldest version. It should
choose the most recent version.
This reverts commit 5f5d3ea896.
These new scanner rules for JARs causes most builds on fdroiddata to
break. For example, at least the past 20 releases of
org.fdroid.fdroid. fdroiddata needs to be cleaned up and prepared for
this before we can include 5f5d3ea896
I'm running fdroid scanner on the whole archive now, I'll post the log
to an issue once its done. Ignoring test files would eliminate some
of these build failures, at least for most org.fdroid.fdroid builds.
refs #!325
This fixes a problem when there is a broken symlink with the name
AndroidManifest.xml which will then lead to a crash when trying to open
it for trying to remove the debuggable flag.
Wow, this is hacky and weird, but its been there for a long time. I
don't know if its even relevant anymore, but is isolated so it
shouldn't hurt anything. Plus there is a test for it in
tests/complete-ci-tests, so it seems worth fixing.
Running `fdroid verify` I was seeing FDroidExceptions from verify.py:98
that had a tuple rather than a string.
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 152, in <module>
main()
File "/home/hans/code/fdroid/server/fdroid", line 128, in main
mod.main()
File "/home/hans/code/fdroid/server/fdroidserver/verify.py", line 98, in main
logging.info("...NOT verified - {0}".format(e))
File "/home/hans/code/fdroid/server/fdroidserver/exception.py", line 22, in __str__
ret += "\n==== detail begin ====\n%s\n==== detail end ====" % ''.join(self.detail).strip()
TypeError: sequence item 1: expected str instance, HTTPError found
