23b0b6bc added logging in case of an FDroidException on the build
server. This broke the log of apps that fail to build.
Prior to 23b0b6bc a failing build in the VM triggered a BuildException
on the server side but the build output was written to the log in the
finally part of build_server(). After 23b0b6bc the finally part writs
the build as well but the BuildException is caught in main() and the
build log is overwritten with the empty exception content. This patch
always adds the build log to the exception so it is written to the log.
Closes: #882
This happened with us.spotco.fennec_dos_2912000 where no build log was
saved in the repo but the wiki has an error text. The error text was:
"Command '['rsync', '--recursive', '--perms', '--links', '--quiet', '--rsh=ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=FATAL -o IdentitiesOnly=yes -o PasswordAuthentication=no -p 2222 -i /home/fbuild/.vagrant.d/boxes/buildserver/0/virtualbox/vagrant_private_key', 'build/srclib/MozFennec', 'vagrant@127.0.0.1:/home/vagrant/build/srclib']' returned non-zero exit status 255."
Which is generated by the rsync subprocess call in build_server(). I
assume that it threw an Exception (not an FdroidException, because the
string "Build completed at" is not in the wiki site) that was caught in
main().
In case the app repository has a broken submodule, checkupdates failed
and did not search for any version updates. Ignoring the error let's us
at least find new version in the main repo (which is probably the right
place anyhow) and thus an improvement.
Regression from cd405cc9.
Parse_androidmanifests() can return 'Unknown' or 'Ignore' if it did not
find a version name. The check_tags() always returned the tag and
checkupdates_app() replaced the version by the tag in the 'Unknown'
case. Since cd405cc9 the tag is the hash and so the version would become
the hash as well. This patch moves the 'Unknown' check directly after
the Parse_androidmanifests().
allowlist and blocklist are much clearer terms with no cultural baggage.
This changes all "whitelist" references to "allowlist", and all "blacklist"
references to "blocklist".
config.yml requires ASCII or UTF-8 encoding because this code does not
auto-detect the file's encoding. That is left up to the YAML library.
YAML allows ASCII, UTF-8, UTF-16, and UTF-32 encodings. Since it is a
good idea to manage config.yml (WITHOUT PASSWORDS!) in git, it makes
sense to use a globally standard encoding.
Android Studio recommends "you use UTF-8 encoding whenever possible",
so this code assumes the files use UTF-8. UTF-8 is also the default
encoding on GNU/Linux and macOS.
https://sites.google.com/a/android.com/tools/knownissues/encoding
Windows will probably default to UTF16, since that's the native
encoding for files. So forcing things to use UTF-8 should help
compatibility.
Windows seems to require this, otherwise this happens:
Traceback (most recent call last):
File "tests/update.TestCase", line 737, in test_translate_per_build_anti_features
apps = fdroidserver.metadata.read_metadata(xref=True)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 813, in read_metadata
app = parse_metadata(metadatapath, appid in check_vcs, refresh)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1023, in parse_metadata
parse_yaml_metadata(mf, app)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1073, in parse_yaml_metadata
yamldata = yaml.safe_load(mf)
File "C:\python37\lib\site-packages\yaml\__init__.py", line 162, in safe_load
return load(stream, SafeLoader)
File "C:\python37\lib\site-packages\yaml\__init__.py", line 112, in load
loader = Loader(stream)
File "C:\python37\lib\site-packages\yaml\loader.py", line 34, in __init__
Reader.__init__(self, stream)
File "C:\python37\lib\site-packages\yaml\reader.py", line 85, in __init__
self.determine_encoding()
File "C:\python37\lib\site-packages\yaml\reader.py", line 124, in determine_encoding
self.update_raw()
File "C:\python37\lib\site-packages\yaml\reader.py", line 178, in update_raw
data = self.stream.read(size)
File "C:\python37\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 37: character maps to <undefined>
gotorevisionx tries to clean up the git repo before checking out a new
revision. In b848b99ba this was changed to reset and clean any submodule
as well. In case upstream has a broken submodule configuration this
could fail and we can't checkout the new revision. As we are doing a
reset and clean after checking out the new revision anyhow, this change
ignores submodule errors before the checkout and only makes sure that
the main repo is reset and clean.
This broke checkupdates for apps where old versions had broken
submodules. It checkout out the old version and got stuck, not able to
checkout any other version.
