Merge branch 'buildserver-qemu-kvm' into 'master'

buildserver running in qemu/kvm to support KVM on KVM

jenkins.debian.net runs in QEMU/KVM instances, so in order to run the F-Droid buildserver there, it needs to work inside of a KVM guest.  The best way I found to do that is to create QEMU/KVM instances via KVM's "nested" virtualization support.  This collection of commits enables using QEMU/KVM as the buildserver when `./makebuildserver` detects that it is running inside of KVM.  Otherwise, the old behavior is default: running in VirtualBox.

I have run these tests inside of ubuntu/16.04 on bare metal, which uses VirtualBox, and ubuntu/16.04 KVM guest, which uses QEMU/KVM.  It'll also run on the Guardian Project jenkins box, which is Debian/jessie.

@mvdan @CiaranG @krt

See merge request !168

buildserver/Vagrantfile

@@ -15,10 +15,28 @@ Vagrant.configure("2") do |config|
   config.vm.box = configfile['basebox']
   config.vm.box_url = configfile['baseboxurl']
 
-  config.vm.provider "virtualbox" do |v|
-    v.customize ["modifyvm", :id, "--memory", configfile['memory']]
-    v.customize ["modifyvm", :id, "--cpus", configfile['cpus']]
-    v.customize ["modifyvm", :id, "--hwvirtex", configfile['hwvirtex']]
+  # TODO detect if jessie64.box is libvirt, or `vagrant mutate jessie64 libvirt`
+  if `systemd-detect-virt`.include? "qemu" or configfile["vm_provider"] == "libvirt"
+    # use KVM/QEMU if this is running in KVM/QEMU
+    config.vm.provider :libvirt do |libvirt|
+      libvirt.driver = "kvm"
+      libvirt.host = "localhost"
+      libvirt.uri = "qemu:///system"
+      libvirt.cpus = configfile["cpus"]
+      libvirt.memory = configfile["memory"]
+    end
+    config.vm.synced_folder './', '/vagrant', type: '9p'
+    synced_folder_type = '9p'
+  elsif not configfile.has_key? "vm_provider" or configfile["vm_provider"] == "virtualbox"
+    # default to VirtualBox if not set
+    config.vm.provider "virtualbox" do |v|
+      v.customize ["modifyvm", :id, "--memory", configfile['memory']]
+      v.customize ["modifyvm", :id, "--cpus", configfile['cpus']]
+      v.customize ["modifyvm", :id, "--hwvirtex", configfile['hwvirtex']]
+    end
+    synced_folder_type = 'virtualbox'
+  else
+    abort("No supported VM Provider found, set vm_provider in Vagrantfile.yaml!")
   end
 
   config.vm.boot_timeout = configfile['boot_timeout']
@@ -32,8 +50,12 @@ Vagrant.configure("2") do |config|
   # default does not need a custom mount
   if configfile["cachedir"] != "buildserver/cache"
     config.vm.synced_folder configfile["cachedir"], '/vagrant/cache',
-      owner: 'root', group: 'root', create: true
+      create: true, type: synced_folder_type
   end
+  # Make sure dir exists to mount to, since buildserver/ is
+  # automatically mounted as /vagrant in the guest VM. This is more
+  # necessary with 9p synced folders
+  Dir.mkdir('cache') unless File.exists?('cache')
 
   # cache .deb packages on the host via a mount trick
   if configfile.has_key? "aptcachedir"

buildserver/provision-android-sdk

@@ -23,10 +23,27 @@ if [ ! -x $ANDROID_HOME/tools/android ]; then
     unzip -qq $tools
 fi
 
+# disable the repositories of proprietary stuff
+disabled="
+@version@=1
+@disabled@https\://dl.google.com/android/repository/extras/intel/addon.xml=disabled
+@disabled@https\://dl.google.com/android/repository/glass/addon.xml=disabled
+@disabled@https\://dl.google.com/android/repository/sys-img/android/sys-img.xml=disabled
+@disabled@https\://dl.google.com/android/repository/sys-img/android-tv/sys-img.xml=disabled
+@disabled@https\://dl.google.com/android/repository/sys-img/android-wear/sys-img.xml=disabled
+@disabled@https\://dl.google.com/android/repository/sys-img/google_apis/sys-img.xml=disabled
+"
+test -d ${HOME}/.android || mkdir ${HOME}/.android
+for line in $disabled; do
+    echo $line >> ${HOME}/.android/sites-settings.cfg
+done
+
+
 cd /vagrant/cache
 
 # make links for `android update sdk` to use and delete
-for f in android_*.zip android-[0-9]*.zip platform-[0-9]*.zip build-tools_r*-linux.zip; do
+latestm2=`ls -1 android_m2repository*.zip | sort -n | tail -1`
+for f in $latestm2 android-[0-9]*.zip platform-[0-9]*.zip build-tools_r*-linux.zip; do
     rm -f ${ANDROID_HOME}/temp/$f
     ln -s /vagrant/cache/$f ${ANDROID_HOME}/temp/
 done

buildserver/provision-apt-get-install

@@ -11,6 +11,9 @@ sed -i "s,http://ftp.uk.debian.org/debian/,${debian_mirror},g" /etc/apt/sources.
 printf 'APT::Install-Recommends "0";\nAPT::Install-Suggests "0";\n' \
        > /etc/apt/apt.conf.d/99no-install-recommends
 
+printf 'APT::Acquire::Retries "20";\n' \
+       > /etc/apt/apt.conf.d/99acquire-retries
+
 if grep --quiet jessie /etc/apt/sources.list; then
     echo "deb $debian_mirror jessie-backports main" > /etc/apt/sources.list.d/backports.list
 fi