When working or testing with large APK collections, oftentimes it can take
hours to scan all the APKs. If there is a failure before the apkcache is
written out, then all that work is lost since no cache is written out. This
moves the final cache writing before writing the index and knownapks to
make it more likely that the apkcache is successfully written.
Since metadata files are now YAML, programs should be able to just read in
the .yml files, then use them with fdroidserver methods without having to
know about the App class. The App class just provides syntactic sugar by
allowing dict keys to be accessed as attributes.
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds". This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().
The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.
```
sed -i \
-e 's/app\.builds/app.get('Builds', \[\])/g' \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
-e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
-e "s/app\['builds'\]/app.get('Builds', [])/g" \
*/*.*
```
When using fdroidserver methods as an API, the full setup might not
have taken place. `app` instances can always just be a dict, the App
class is mostly just a typing shortcut. This is incremental, it only
affects a couple of functions in fdroidserver/update.py.
This makes it possible to use process_apks(), get_cache(), and anything
calling disabled_algorithms_allowed() as an API without having to set
options up beforehand.
latestapps.dat was being used for the old-website, it's not used anymore
Fixes the following crash in production:
```
2020-09-13 19:26:59,000 CRITICAL: Unknown exception found!
Traceback (most recent call last):
File "/home/fbuild/fdroidserver/fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/home/fbuild/fdroidserver/fdroidserver/__main__.py", line 230,
in main
raise e
File "/home/fbuild/fdroidserver/fdroidserver/__main__.py", line 211,
in main
mod.main()
File "/home/fbuild/fdroidserver/fdroidserver/update.py", line 2451, in
main
app = apps[appid]
KeyError: '45b464b398a7d9fac5a186bd3d3d8dc1e6a25f7f9cd48c7462619b1e5fba87c2'
```
This was accidentally changed in !756 because the functionality was
hidden in `apply_info_from_latest_apk` which is a less than stellar name
for something that also applies infos from app->apk and in this case did
apply info from *oldest* apk->app.
So instead move that into a separate step.
Note: This restores the previous behaviour. There's discussion in #801
on further changes to make the added date also work for repos which
don't keep an archive at all.
This makes update.py:main a lot saner by removing a lot of the
implicit assumptions between the different stages of generating the
repository index.
* mostly unify repo and archive processing, that means the archive is
now actually getting the same treatment regarding i.e. fastlane data.
Previously the archive didn't get considered at all here.
* already filter the list of apps to include in a repo in update.py and
give that prefiltered list to index. This makes sure we actually only
copy fastlane/triple-t/etc. stuff for apps ending up in the index.
This both, can save a lot of time if there are a lot of old /build
dirs lying around and doesn't clutter /repo with things that aren't
referenced from the index.
Closesfdroid/fdroidserver#524
This is a historic detail of the index format, so move it there.
For wiki update and status json there's really no reason why this should
be done in alphabetic app name order. Use the default sort order by
appid.
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs. Now it is a username. So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration. LiberapayID: will not override
Liberapay:. Clients are expected to prefer Liberapay: over LiberapayID:
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links. Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:. Specifying those in the metadata file takes
precedence over the FUNDING.yml. This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata. This lets the
git repo maintain those specific donations links themselves.
https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files
The test file was generated using:
```python
import os, re, yaml
found = dict()
for root, dirs, files in os.walk('.'):
for f in files:
if f == 'FUNDING.yml':
with open(os.path.join(root, f)) as fp:
data = yaml.safe_load(fp)
for k, v in data.items():
if k not in found:
found[k] = set()
if not v:
continue
if isinstance(v, list):
for i in v:
found[k].add(i)
else:
found[k].add(v)
with open('gather-funding-names.yaml', 'w') as fp:
output = dict()
for k, v in found.items():
output[k] = sorted(v)
yaml.dump(output, fp, default_flow_style=False)
```
These entries are hardcoded as a single line in all the app stores, so
newlines should be stripped to get the data simple to use. This is in
contrast with the on-disk format for Fastlane and Triple-T, which includes
a newline in the title.txt and short_description.txt files. I think all
files in those systems are normalized to end in a newline.
6d0b1bbe6fae0909683f2c6a154515bc4bfcb674 didn't handle the
allow_disabled_algorithm case at all, so we add it back.
This additionally fixes a (previously existing) bug where setting
allow_disabled_algorithms to True didn't move apks back from archive to
repo. Introduce a new test for this.
The disabled_algorithm archiving logic is still all over the place so
ideally that needs a future refactor.
We want to keep the stable version (CV) of an app in /repo with highest
priority and only move it to /archive when ArchivePolicy is set to 0.
For this to work we need two changes here:
1) when sorting apks by version code we insert the apk corresponding to
the CVC of the app afterwards with highest priority. So when walking the
list of apks afterwards the CVC apk is always kept first.
2) Instead of the two pass algorithm of moving things back and forth to
the archive we instead figure out where each apk of an app goes in first
before actually moving them into the right place.
Fixes: fdroid/fdroidserver#385
Python PIL is not so tolerant, so bad EXIF causes crashes:
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main
insert_localized_app_metadata(apps)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata
_strip_and_copy_image(os.path.join(root, f), destdir)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image
in_image = Image.open(fp)
File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open
% (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>
Using a filename based on the hash of the contents means that the caching
algorithms for fdroidclient and browsers can safely cache the file forever
using the filename, since this guarantees that the contents will never
change for a given filename.
This does not cover screenshots, only icon.png, featureGraphic.png,
tvBanner.png, and promoGraphic.png.
fdroidserver#689
fdroid-website!453
This was done with much help from @uniqx. This is the first level of
supporting APK Signatures v1, v2, and v3. This is enough to include
APKs with any combo of v1/v2/v3 signatures. For this to work at all,
apksigner and androguard 3.3.3+ must be installed.
closes#399
androguard's v2 signature methods return DER-encoded certificates. The DER-
encoded certificates are parsed from JAR Signature's raw "Signature Block
File".
