Android Studio recommends "you use UTF-8 encoding whenever possible",
so this code assumes the files use UTF-8. UTF-8 is also the default
encoding on GNU/Linux and macOS.
https://sites.google.com/a/android.com/tools/knownissues/encoding
Windows will probably default to UTF16, since that's the native
encoding for files. So forcing things to use UTF-8 should help
compatibility.
We switched to apksigner in 50f0534d but old apks where still verified
with jarsigner (or an old apksigner version). Bumping the
METADATA_VERSION to force a rebuild of apkcache.
Hopefully this resolves de.chagemann.regexcrossword getting the
KnownVuln, DisabledAlgorithm tags.
https://jenkins.debian.net/job/reproducible_fdroid_test/422https://jenkins.debian.net/job/reproducible_fdroid_test/423
```
Traceback (most recent call last):
File "../fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 227, in main
raise e
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 208, in main
mod.main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2316, in main
apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1746, in process_apks
use_date_from_apk, ada, True)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1619, in process_apk
apk = scan_apk(apkfile)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1385, in scan_apk
scan_apk_androguard(apk, apk_file)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1527, in scan_apk_androguard
resource_id = arsc.get_id(apk['packageName'], icon_id)
File "/usr/lib/python3/dist-packages/androguard/core/bytecodes/axml/__init__.py", line 1668, in get_id
self._analyse()
File "/usr/lib/python3/dist-packages/androguard/core/bytecodes/axml/__init__.py", line 1321, in _analyse
ate = self.packages[package_name][nb + 3 + nb_i]
IndexError: list index out of range
```
Implemented as a fallback solution when parsing settings.gradle
and globbing the build dir with "'*', 'src', '*', 'play'" did not
yield results.
Before, the logic would not find triple-t metadata in projects where
settings.gradle is in a subdirectory or 'src' is found at a directory
depth != 2.
Closesfdroid/fdroiddata#2295
If a non-APK is added with the appid/packageName that matches some APKs, it
should through an error.
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 211, in main
mod.main()
File "/home/hans/code/fdroid/server/fdroidserver/update.py", line 2343, in main
index.make(apps, sortedids, apks, repodirs[0], False)
File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 142, in make
fdroid_signing_key_fingerprints)
File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 166, in make_v1
v1_sort_packages(packages, fdroid_signing_key_fingerprints)
File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 292, in v1_sort_packages
packages.sort(key=v1_sort_keys)
File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 288, in v1_sort_keys
.format(apkfilename=package['apkName']))
fdroidserver.exception.FDroidException: at.roteskreuz.stopcorona_8.jobf does not have a valid signature!
```
Traceback (most recent call last):
File "../fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 227, in main
raise e
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 208, in main
mod.main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2340, in main
repoapps = prepare_apps(apps, apks, repodirs[0])
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2176, in prepare_apps
copy_triple_t_store_metadata(apps_with_packages)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1076, in copy_triple_t_store_metadata
_set_author_entry(app, 'authorWebSite', os.path.join(root, f))
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 784, in _set_author_entry
with open(f, errors='replace') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'build/player.efis.cfd/pfd/src/main/play/contact-website.txt'
```
There are so many possible installation paths for Python modules, it has
been very hard to even find and test them all. This adds a fallback option
if the examples dir cannot be found. A repo can work without an icon or
the example config.py.
This removes the fake assumption that the icon can be a full path in the
config.py. While the path was being properly passed through to the index
file, the file was never copied properly into place nor rsynced to the web
server.
When working or testing with large APK collections, oftentimes it can take
hours to scan all the APKs. If there is a failure before the apkcache is
written out, then all that work is lost since no cache is written out. This
moves the final cache writing before writing the index and knownapks to
make it more likely that the apkcache is successfully written.
Since metadata files are now YAML, programs should be able to just read in
the .yml files, then use them with fdroidserver methods without having to
know about the App class. The App class just provides syntactic sugar by
allowing dict keys to be accessed as attributes.
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds". This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().
The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.
```
sed -i \
-e 's/app\.builds/app.get('Builds', \[\])/g' \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
-e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
-e "s/app\['builds'\]/app.get('Builds', [])/g" \
*/*.*
```
When using fdroidserver methods as an API, the full setup might not
have taken place. `app` instances can always just be a dict, the App
class is mostly just a typing shortcut. This is incremental, it only
affects a couple of functions in fdroidserver/update.py.
This makes it possible to use process_apks(), get_cache(), and anything
calling disabled_algorithms_allowed() as an API without having to set
options up beforehand.
latestapps.dat was being used for the old-website, it's not used anymore
Fixes the following crash in production:
```
2020-09-13 19:26:59,000 CRITICAL: Unknown exception found!
Traceback (most recent call last):
File "/home/fbuild/fdroidserver/fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/home/fbuild/fdroidserver/fdroidserver/__main__.py", line 230,
in main
raise e
File "/home/fbuild/fdroidserver/fdroidserver/__main__.py", line 211,
in main
mod.main()
File "/home/fbuild/fdroidserver/fdroidserver/update.py", line 2451, in
main
app = apps[appid]
KeyError: '45b464b398a7d9fac5a186bd3d3d8dc1e6a25f7f9cd48c7462619b1e5fba87c2'
```
