GitLab storage provides two mirrors by default:
* https://gitlab.com/user/repo/raw/master/fdroid/repo
* https://user.gitlab.io/repo/fdroid/repo
While the F-Droid client will happily fetch the index*.jar files and
parse them from either of these two mirrors, only the GitLab Pages
mirror will serve files with the correct mime type. Many repos
tend to put index.html files (and associated .css/.js/image files) in
the root of a repository to provide information about that repo.
One example is RepoMaker. The way in which RepoMaker decides the public
URL of a repo, is to take the first mirror in the list. This means that
the URL which RepoMaker directs people to for GitLab storage returns a
.html document in text/plain, which means that it is not rendered.
We could change RepoMaker so that it takes the last mirror, and then it
woruld work. However there is something nice about the first mirror in a
list being the most authoritative (even though the mirror order doesn't
- and perhaps shouldn't have any specific meaning).
ERROR: Could not build app org.fdroid.fdroid due to unknown error: Traceback (most recent call last):
File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 1202, in main
options.onserver, options.refresh):
File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 972, in trybuild
build_server(app, build, vcs, build_dir, output_dir, log_dir, force)
File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 82, in build_server
logging.debug(_('Fetched buildserverid from VM: ') + buildserverid)
TypeError: Can't convert 'bytes' object to str implicitly
We remove the whole "build" directory while cleaning source code tree
because Gradle can leave there files even after "gradle clean". But some
projects (Mozilla Fennec) actually have useful stuff checked into VCS
under the "build" directory.
Remove only those subdirectories that we known for sure are leftovers
from Gradle.
Fixesfdroid/fdroidserver#438.
When `fdroid build` is run using the buildserver, it should fetch the
buildserverid on the first build.
Seems this was really a silly bug in 837fc99d74
Since `fdroid build --all` can run a long time, knowing when that command
was started will be very useful information for figuring out what the build
server is doing.
GitHub only allows an SSH key to be used as a Deploy Key for a single repo.
That means, each nightly build repo on GitHub/Travis must have its own
debug keystore.
This needed now because the buildserver is hanging so often, that we are
often going a week or more without any builds published. Perhaps this is
only temporary, or maybe we will want to flush this feature out more as a
standard thing. But we really need it for now to at least get some builds
out on a daily basis.
Since the website deploy is also triggered by this cycle, making the build
finish more often means the website will be published more often.
Fixes bb758d3f, spotted by @bubu:
DEBUG: buildserver > DEBUG: > sudo apt-get -y purge sudo
DEBUG: buildserver > Reading package lists...
DEBUG: buildserver > Building dependency tree...
DEBUG: buildserver > Reading state information...
DEBUG: buildserver > The following packages will be REMOVED:
DEBUG: buildserver > sudo*
DEBUG: buildserver > 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
DEBUG: buildserver > After this operation, 2,391 kB disk space will be freed.
(Reading database ... 68491 files and directories currently installed.)
DEBUG: buildserver > Removing sudo (1.8.10p3-1+deb8u4) ...
DEBUG: buildserver > You have asked that the sudo package be removed,
DEBUG: buildserver > but no root password has been set.
DEBUG: buildserver > Without sudo, you may not be able to gain administrative privileges.
DEBUG: buildserver >
DEBUG: buildserver > If you would prefer to access the root account with su(1)
DEBUG: buildserver > or by logging in directly,
DEBUG: buildserver > you must set a root password with "sudo passwd".
DEBUG: buildserver >
DEBUG: buildserver > If you have arranged other means to access the root account,
DEBUG: buildserver > and you are sure this is what you want,
DEBUG: buildserver > you may bypass this check by setting an environment variable
DEBUG: buildserver > (export SUDO_FORCE_REMOVE=yes).
DEBUG: buildserver >
DEBUG: buildserver > Refusing to remove sudo.
DEBUG: buildserver > dpkg: error processing package sudo (--purge):
DEBUG: buildserver > subprocess installed pre-removal script returned error exit status 1
DEBUG: buildserver > Errors were encountered while processing:
DEBUG: buildserver > sudo
DEBUG: buildserver > E: Sub-process /usr/bin/dpkg returned an error code (1)
This strips metadata and optimizes the compression of all PNGs copied
from the app's source repo as well as all the icons extracted from the
APKs. There have been exploits delivered via image metadata, and
F-Droid isn't using it all, so its best to just remove it.
This unfortunately uncompresses and recompresses the files. Luckily,
that's a lossless procedure with PNGs, and we might end up with
smaller files. The only tool I could find that strips without
changing the image data is exiftool, but that is written in Perl.
EXIF data can be abused to exploit systems a lot easier than the JPEG image
data can. The F-Droid ecosystem does not use the EXIF data, so keep things
safe and strip it all away. There is a chance that some images might rely
on the rotation to be set by EXIF, but I think having a safe system is more
important.
If needed, only the rotation data could be saved. But that then makes it
hard to tell which images have been stripped. This way, if there is no
EXIF, it has been stripped. And if there is EXIF data, then it is suspect.
https://securityaffairs.co/wordpress/51043/mobile-2/android-cve-2016-3862-flaw.htmlhttps://threatpost.com/google-shuts-down-potentially-massive-android-bug/120393/https://blog.sucuri.net/2013/07/malware-hidden-inside-jpg-exif-headers.html
The big downside of this is that it decompresses and recompresses the
image data. That should be replaced by a technique from jhead,
exiftool, ObscuraCam, etc. that only strips the metadata.
Once `sudo` has been used to execute the commands in sudo=, then it should
be removed from the build VM. That prevents any other part of the build
from using sudo. That means that all commands run with `sudo` must be
committed to fdroiddata.git, making them very visible.
closes#424
The SSH key fingerprint is used in the filename. The base64 used for SSH
key fingerprints includes /. Not all keys will end up having a / in them.
For those that do, this will crash since the ssh key filename ends up being
non-existent dirs:
$ fdroid nightly
Importing keystore /home/mhoffmann/.android/debug.keystore to /tmp/.cqswaeo8/.keystore.p12...
MAC verified OK
writing RSA key
CRITICAL: Unknown exception found!
Traceback (most recent call last):
File "/usr/lib/python3.6/shutil.py", line 544, in move
os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.cqswaeo8/.privkey' -> '/tmp/.cqswaeo8/debug_keystore_PZtS/4Tzk4dpzKiX9AAf1GrhAVi9U7UE1aYEHr6evKo_id_rsa'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mhoffmann/projects/oss/fdroidserver/fdroid", line 156, in <module>
main()
File "/home/mhoffmann/projects/oss/fdroidserver/fdroid", line 132, in main
mod.main()
File "/home/mhoffmann/projects/oss/fdroidserver/fdroidserver/nightly.py", line 284, in main
privkey = _ssh_key_from_debug_keystore()
File "/home/mhoffmann/projects/oss/fdroidserver/fdroidserver/nightly.py", line 73, in _ssh_key_from_debug_keystore
shutil.move(privkey, ssh_private_key_file)
File "/usr/lib/python3.6/shutil.py", line 558, in move
copy_function(src, real_dst)
File "/usr/lib/python3.6/shutil.py", line 257, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib/python3.6/shutil.py", line 121, in copyfile
with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.cqswaeo8/debug_keystore_PZtS/4Tzk4dpzKiX9AAf1GrhAVi9U7UE1aYEHr6evKo_id_rsa'
These were both spamming the output with lots of confusing messages, even
when --verbose was not used. Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.
closes#405
Just give a proper error message rather than this stack trace:
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 156, in <module>
main()
File "/home/hans/code/fdroid/server/fdroid", line 132, in main
mod.main()
File "/export/share/code/fdroid/server/fdroidserver/init.py", line 148, in main
if os.path.isfile(aapt):
File "/usr/lib/python3.5/genericpath.py", line 30, in isfile
st = os.stat(path)
This rsync hung because of an SSH unknown key prompt. Since this is just
the vm host sshing to the vm guest, it is not essential to check the host
keys.
