Currently, if buildjni is not used but ndk is set to an invalid value,
the build would start but $ANDROID_NDK would be empty. This is happening
in VLC, which results in very confusing errors.
If a build uses a ndk= value that is not set up, such as r11/r12 which
we do not have yet, it should error with "NDK version could not be
found". It does with this change.
Note that the apt packages are split into two halves, because it takes
too long (on 64 bit!) to install them all. The sensible fix would be
to simply up the timeout on the package installation section, but this
is completely broken in chef.
This will make `vagrant ssh` and `fdroid build --server` be the same env,
so troubleshooting should be easier. !135 Here's what `man bash` says:
When bash is invoked as an interactive login shell, or as a
non-interactive shell with the --login option, it first reads and
executes commands from the file /etc/profile, if that file
exists. After reading that file, it looks for ~/.bash_profile,
~/.bash_login, and ~/.profile, in that order, and reads and
executes commands from the first one that exists and is readable.
The --noprofile option may be used when the shell is started to
inhibit this behavior. When a login shell exits, bash reads and
executes commands from the file ~/.bash_logout, if it exists.
This replaces the current default behavior of always forcing the
build_tools version and allows the user to set build-tools forcing in
config.py.
closes#147
Switching to a script /etc/profile.d to set up the environment is all
well and good, except that doesn't get run when you directly execute a
command directly via ssh, which means that the buildserver didn't work
at all (at least for anything that used gradle, or relied on the
environment variables.
This fix doesn't look very nice, but it works - it just forces the
appropriate script to run before build.py is executed on the server.
(Side note, I thought we had tests for this, how did it get past them?)
bash provides a standard file location for a script to be run when the
shell starts: /etc/profile.d/ This converts the scattered bits of code for
making ~/.bsenv into a single provisioning script to generate
/etc/profile.d/bsenv.sh, which gets automatically executed when bash starts
The default config for .fdroid.* metadata that is included in a git repo is
different than for the standard metadata/ layout because the expectations
are different. In this case, the most common user will be the app
developer working on the latest update of the app on their own machine.
The start up sequence of processes that are based on the .fdroid.* metadata
is a bit different, so this ensures that the environment variables get
properly initialized in all cases.
This also creates a single function where the environment is set. Before
it was being set in multiple places across multiple files.
File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 1119, in main
options.onserver, options.refresh):
File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 954, in trybuild
build_server(app, build, vcs, build_dir, output_dir, force)
File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 403, in build_server
output += get
TypeError: Can't convert 'bytes' object to str implicitly
closes#164https://gitlab.com/fdroid/fdroidserver/issues/164
Fixes a couple errors like:
File "./makebuildserver", line 30, in vagrant
out += line
TypeError: Can't convert 'bytes' object to str implicitly
If universal_newlines=False, the default, then Popen will return bytes if
the newlines in the data do not match the system's newlines. Setting it to
true enables auto-conversion, and then guarantees that the data is always
str.
"If universal_newlines is True, the file objects stdin, stdout and stderr
are opened as text streams in universal newlines mode, as described above
in Frequently Used Arguments, otherwise they are opened as binary streams."
https://docs.python.org/3/library/subprocess.html#subprocess.Popen
The point of removing these was to make sure that gradlew wasn't used,
and that our gradle was used instead. Removing the scripts already
accomplishes this.
Removing gradle/ should be harmless, but some apps like I2P re-use this
directory to also hold other stuff that is actually needed. So be safer
and don't remove it at all.
This avoids the following incorrect error:
fdroidserver.common.FDroidException: Only one local metadata file allowed! Found: .fdroid.keystorepass.txt .fdroid.keypass.txt
Before, `fdroid build` would create some subdirs, then fail when it could
not read the metadata via metadata.read_metadata(). This checks before
whether there is any metadata available, and warns the user accordingly.
1. It is packaged in modern versions of distros, update docs accordingly
2. 1.1 is hopelessly outdated - support 1.4 onwards
3. Check the version properly, and write a better config (previously it
wrote the 1.1 config for 1.1 and 1.3+
4. Ensure that the default share from later versions is not present when
building, it's only required for provisioning.
It is a good idea in theory, especially for extra tasks specified by the
user. But in practice, tasks specified by developers often don't have a
clean counterpart. So this just breaks the build.
This breaks on newer plugin versions. Instead of complicating this logic
even further to support them, upstream devs should instead use gradle
configs that actually work.
If their build fails because of lint, it's theirs to fix, not ours.
This simplifies usage, goes from
build['flag']
to
build.flag
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
While at it, unify "build", "thisbuild", "info", "thisinfo", etc into
just "build".
This simplifies usage, goes from
app['Foo']
to
app.Foo
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
replace deprecated optparse with argparse
squashed and rebased merge request fdroid/fdroidserver!74
following guidelines from:
https://docs.python.org/2/library/argparse.html#upgrading-optparse-code
except, still using option = parse.parse_args() instead of args = ...
- using the following script in folder fdroidserver:
```
for i in *.py; do
sed -i -e 's/optparse/argparse/' \
-e 's/OptionParser/ArgumentParser/' \
-e 's/OptionError/ArgumentError/' \
-e 's/add_option/add_argument/' \
-e 's/(options, args) = parser/options = parser/' \
-e 's/options, args = parser/options = parser/' \
-e 's/Usage: %prog/%(prog)s/' $i;
done
```
- use ArgumentParser argument to replace (option, args) = parser.parse()
call
- use parser.error(msg) instead of raise ArgumentException as suggested
in https://docs.python.org/2/library/argparse.html#exiting-methods
- in fdroid catch ArgumentError instead of OptionError
See merge request !75
following guidelines from:
https://docs.python.org/2/library/argparse.html#upgrading-optparse-code
except, still using option = parse.parse_args() instead of args = ...
- using the following script in folder fdroidserver:
for i in *.py; do
sed -i -e 's/optparse/argparse/' \
-e 's/OptionParser/ArgumentParser/' \
-e 's/OptionError/ArgumentError/' \
-e 's/add_option/add_argument/' \
-e 's/(options, args) = parser/options = parser/' \
-e 's/options, args = parser/options = parser/' \
-e 's/Usage: %prog/%(prog)s/' $i;
done
- use ArgumentParser argument to replace (option, args) = parser.parse()
call
- use parser.error(msg) instead of raise ArgumentException as suggested
in https://docs.python.org/2/library/argparse.html#exiting-methods
- in fdroid catch ArgumentError instead of OptionError
This addresses the discussion in !64https://gitlab.com/fdroid/fdroidserver/merge_requests/64
Sometimes, buildToolsVersion is a kind of gradle macro call, and other
times it is a variable assignment. This regsub pattern now handles both of
those cases.
Python libraries work better in Python than running external commands, and
it also makes the code much more portable. For example, the GNU and BSD
sed commands have different, and sometimes conflicting, flags.
This also reworks the regexp patterns to be more tightly focused, and not
change the same variable name in comments or elsewhere.
