Currently, if buildjni is not used but ndk is set to an invalid value,
the build would start but $ANDROID_NDK would be empty. This is happening
in VLC, which results in very confusing errors.
If a build uses a ndk= value that is not set up, such as r11/r12 which
we do not have yet, it should error with "NDK version could not be
found". It does with this change.
Pass a date from APK to KnownApks.recordapk()
... if --use-date-from-apks option is used.
Essentially, it just expands influence of `--use-date-from-apks` option to `stats/known_apks.txt`.
See merge request !141
OBB files provide a mechanism to deliver large data blobs that do not need
to be part of the APK. For example, a game's assets do not need to change
often, so they can be shipped as an OBB, then APK updates do not need to
include all those assets for each update.
https://developer.android.com/google/play/expansion-files.html
Note that the apt packages are split into two halves, because it takes
too long (on 64 bit!) to install them all. The sensible fix would be
to simply up the timeout on the package installation section, but this
is completely broken in chef.
The ZIP format has no official encoding :-| so we have to do hacks. The
zipfile devs couldn't even sort this out:
https://bugs.python.org/issue10614closes#167
`fdroid update` should be able to handle any valid filename (hopefully
aapt doesn't barf on them). To handle that, the environment where the
shell commands are run in needs to have a UTF-8 locale set. If LANG is
not set, things default to ASCII and UTF-8 filenames fail.
This also renames test APK with lots of Unicode chars as a test case.
closes#167
This will make `vagrant ssh` and `fdroid build --server` be the same env,
so troubleshooting should be easier. !135 Here's what `man bash` says:
When bash is invoked as an interactive login shell, or as a
non-interactive shell with the --login option, it first reads and
executes commands from the file /etc/profile, if that file
exists. After reading that file, it looks for ~/.bash_profile,
~/.bash_login, and ~/.profile, in that order, and reads and
executes commands from the first one that exists and is readable.
The --noprofile option may be used when the shell is started to
inhibit this behavior. When a login shell exits, bash reads and
executes commands from the file ~/.bash_logout, if it exists.
This stops these errors:
fdroid/fdroidserver/fdroidserver/update.py:744: ResourceWarning: unclosed
file <_io.BufferedReader
name='repo/icons-320/info.guardianproject.urzip.100.png'>
fdroid/fdroidserver/fdroidserver/update.py:721: DeprecationWarning: The
'warn' function is deprecated, use 'warning' instead
This replaces the current default behavior of always forcing the
build_tools version and allows the user to set build-tools forcing in
config.py.
closes#147
By checking first, this prevents a stacktrace when the duplicate metadata
file is not valid. For example, in the tests, the duplicate is just a zero
length file, which was causing a stacktrace.
Switching to a script /etc/profile.d to set up the environment is all
well and good, except that doesn't get run when you directly execute a
command directly via ssh, which means that the buildserver didn't work
at all (at least for anything that used gradle, or relied on the
environment variables.
This fix doesn't look very nice, but it works - it just forces the
appropriate script to run before build.py is executed on the server.
(Side note, I thought we had tests for this, how did it get past them?)
switch SDK/NDK/gradle buildserver provisioning to shell scripts
This converts the SDK, NDK, and gradle chef recipes to vagrant provisioning shell scripts. Those recipes were just shell scripts anyway, forced into that ugly ruby/chef syntax. Hopefully this will make things easier to maintain since simple bash is easier for most devs than ruby/chef. Also, it might speed up the provisioning a little bit since the whole script is sent to the VM then executed, rather than sent line-by-line. Additionally, the SDK components are now installed using `android update sdk` so we do not need to duplicate Google's crazy kludges with that stuff.
See merge request !132
When the metadata changes, different things will be stored about each APK.
So invalidate the cached info parsed from APKs if the cache's metadata
version does not match the metadata version of the currently running tools.
bash provides a standard file location for a script to be run when the
shell starts: /etc/profile.d/ This converts the scattered bits of code for
making ~/.bsenv into a single provisioning script to generate
/etc/profile.d/bsenv.sh, which gets automatically executed when bash starts
.java .gradle and XML files all can use any encoding. Most code is ASCII,
but authors' names, etc. can easily be non-ASCII. UTF-8 is by far the most
common file encoding. While UTF-8 is the default encoding inside the code
in Python 3, it still has to deal with the real world, so the encoding
needs to be explicitly set when reading and writing files. So this switches
fdroidserver to expect UTF-8 instead of ASCII when parsing these files. For
now, this commit means that we only support UTF-8 encoded *.java, pom.xml
or *.gradle files. Ideally, the code would detect the encoding and use the
actual one, but that's a lot more work, and its something that will not
happen often. We can cross that bridge when we come to it.
One approach, which is taken in the commit when possible, is to keep the
data as `bytes`, in which case the encoding doesn't matter.
This also fixes this crash when parsing gradle and maven files with
non-ASCII chars:
ERROR: test_adapt_gradle (__main__.BuildTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/tests/build.TestCase", line 59, in test_adapt_gradle
fdroidserver.build.adapt_gradle(testsdir)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/build.py", line 445, in adapt_gradle
path)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/common.py", line 188, in regsub_file
text = f.read()
File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9460: ordinal not in range(128)
This makes UTF-8 the sole supported encoding for F-Droid's files. This is
mostly codifying the already existing practice for config.py and index.xml.
The other files where always just ASCII before.
* config.py
* metadata/*.txt
* known_apks.txt
* categories.txt
* latestapps.txt
* latestapps.dat
* index.xml
Note: this does not change the read/write encoding of stats files. That is
still ASCII.
This fixes fdroid build com.umang.dashnotifier:18:
ERROR: Could not build app com.umang.dashnotifier due to unknown error: Traceback (most recent call last):
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 1121, in main
options.onserver, options.refresh):
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 950, in trybuild
build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 480, in build_local
extlib_dir, onserver, refresh)
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1335, in prepare_source
srclibpaths.append(getsrclib(lib, srclib_dir, build, preponly=onserver, refresh=refresh))
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1258, in getsrclib
cmd = replace_config_vars(srclib["Prepare"], None)
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1822, in replace_config_vars
cmd = cmd.replace('$$NDK$$', build.ndk_path())
AttributeError: 'NoneType' object has no attribute 'ndk_path'
get_ndk_path was removed long ago when App and Build were introduced.
Now Build.ndk_path() should be used. Trying to use the object as a dict
resulted in a nasty crash:
TypeError: 'Build' object is not subscriptable
This is probably a result from faulty rebasing and not enough test
coverage.
Though the YAML people recommend .yaml for the file extension, in Android
land it seems clear that .yml has won out:
* .travis.yml
* .gitlab-ci.yml
* .circle.yml
* Ansible main.yml
This adds a new method for `fdroid import` that will generate the fdroidserver
metadata based on a local git repo. This new mode generates the metadata in
the new .fdroid.yaml format in the git repo itself. It is intended as a quick
way to get starting building apps using the fdroidserver tools.
This changes the function name to include the format of the metadata file,
and also changes the order of the args to match the parse_*_metadata()
functions.
The default config for .fdroid.* metadata that is included in a git repo is
different than for the standard metadata/ layout because the expectations
are different. In this case, the most common user will be the app
developer working on the latest update of the app on their own machine.
Since you can have a .fdroid.* metadata file included directly in an
app's git repo, it seems annoying to force developers to also include
a blank config.py in order to run `fdroid build` from their git repo.
The next step after this is some kind of global config file for
setting paths and things that need to be customized that would apply
to all git repos, for example, to enable using a buildserver.
This also required moving all of the env setting for FDroidPopen into
a single place so that it is properly setup when using the local
metadata.
The start up sequence of processes that are based on the .fdroid.* metadata
is a bit different, so this ensures that the environment variables get
properly initialized in all cases.
This also creates a single function where the environment is set. Before
it was being set in multiple places across multiple files.
This allows app makers to include a .fdroid.(json|xml|yaml|txt) metadata
file in the root of the git repo of their app, then they can build it using
`fdroid build`. This is useful for developers who want to maintain the
fdroid build recipe themselves, and run the fdroid build process for their
own builds.
If multiple tags point at the same commit, limiting the regex search to
one tag per line would only catch one tag. This broke org.wikipedia's
update check.
