There are so many possible installation paths for Python modules, it has
been very hard to even find and test them all. This adds a fallback option
if the examples dir cannot be found. A repo can work without an icon or
the example config.py.
This removes the fake assumption that the icon can be a full path in the
config.py. While the path was being properly passed through to the index
file, the file was never copied properly into place nor rsynced to the web
server.
Before, it would never set current_version_file to the current APK since it
first set current_version_code, then tested against CurrentVersionCode. So
if there is only a single APK and its the CurrentVersionCode, then
current_version_file would not get set.
refs #772
Since metadata files are now YAML, programs should be able to just read in
the .yml files, then use them with fdroidserver methods without having to
know about the App class. The App class just provides syntactic sugar by
allowing dict keys to be accessed as attributes.
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds". This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().
The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.
```
sed -i \
-e 's/app\.builds/app.get('Builds', \[\])/g' \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
-e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
-e "s/app\['builds'\]/app.get('Builds', [])/g" \
*/*.*
```
The key fingerprint should be only hex digits, everything else can be
discarded. That makes it easy to use this function various fingerprint
formats, including the common, human-readable forms spaces between pairs or
quartets.
This was accidentally changed in !756 because the functionality was
hidden in `apply_info_from_latest_apk` which is a less than stellar name
for something that also applies infos from app->apk and in this case did
apply info from *oldest* apk->app.
So instead move that into a separate step.
Note: This restores the previous behaviour. There's discussion in #801
on further changes to make the added date also work for repos which
don't keep an archive at all.
This makes update.py:main a lot saner by removing a lot of the
implicit assumptions between the different stages of generating the
repository index.
* mostly unify repo and archive processing, that means the archive is
now actually getting the same treatment regarding i.e. fastlane data.
Previously the archive didn't get considered at all here.
* already filter the list of apps to include in a repo in update.py and
give that prefiltered list to index. This makes sure we actually only
copy fastlane/triple-t/etc. stuff for apps ending up in the index.
This both, can save a lot of time if there are a lot of old /build
dirs lying around and doesn't clutter /repo with things that aren't
referenced from the index.
Closesfdroid/fdroidserver#524
This is a historic detail of the index format, so move it there.
For wiki update and status json there's really no reason why this should
be done in alphabetic app name order. Use the default sort order by
appid.
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.
Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
It seems now that xml.dom.minidom preserves the order of attributes, rather
than sorting them. We assume alpha-sort, so this manually
This diff in the test suite running on Debian/testing pointed it out:
https://gitlab.com/fdroid/fdroidserver/-/jobs/486970383
```diff
--- /builds/fdroid/fdroidserver/tests/repo/index.xml 2020-04-11 13:36:57.000000000 +0000
+++ repo/index.xml 2020-04-11 13:41:44.000000000 +0000
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<fdroid>
- <repo icon="fdroid-icon.png" name="My First F-Droid Repo Demo" pubkey="308204e1308202c9a003020102020434597643300d06092a864886f70d01010b050030213110300e060355040b1307462d44726f6964310d300b06035504031304736f7661301e170d3136303931333230313930395a170d3434303133303230313930395a30213110300e060355040b1307462d44726f6964310d300b06035504031304736f766130820222300d06092a864886f70d01010105000382020f003082020a028202010086ef94b5aacf2ba4f38c875f4194b44f5644392e3715575d7c92828577e692c352b567172823851c8c72347fbc9d99684cd7ca3e1db3e4cca126382c53f2a5869fb4c19bdec989b2930501af3e758ff40588915fe96b10076ce3346a193a0277d79e83e30fd8657c20e35260dd085aa32eac7c4b85786ffefbf1555cafe2bc928443430cdbba48cfbe701e12ae86e676477932730d4fc7c00af820aef85038a5b4df084cf6470d110dc4c49ea1b749b80b34709d199b3db516b223625c5de4501e861f7d261b3838f8f616aa78831d618d41d25872dc810c9b2087b5a9e146ca95be740316dcdbcb77314e23ab87d4487913b800b1113c0603ea2294188b71d3e49875df097b56f9151211fc6832f9790c5c83d17481f14ad37915fd164f4fd713f6732a15f4245714b84cd665bdbd085660ea33ad7d7095dcc414f09e3903604a40facc2314a115c0045bb50e9df38efb57e1b8e7cc105f340a26eeb46aba0fa6672953eee7f1f92dcb408e561909bbd4bdf4a4948c4d57c467d21aa238c34ba43be050398be963191fa2b49828bc1e4eeed224b40dbe9dc3e570890a71a974a2f4527edb1b07105071755105edcb2af2f269facfb89180903a572a99b46456e80d4a01685a80b233278805f2c876678e731f4ec4f52075aeef6b2b023efbb8a3637ef507c4c37c27e428152ec1817fcba640ad601cb09f72f0fbe2d274a2410203010001a321301f301d0603551d0e04160414c28bf33dd5a9a17338e5b1d1a6edd8c7d141ed0b300d06092a864886f70d01010b0500038202010084e20458b2aafd7fc27146b0986f9324f4260f244920417a77c9bf15e2e2d22d2725bdd8093ec261c3779c3ca03312516506f9410075b90595b41345956d8eb2786fb5994f195611382c2b99dba13381b0100a30bc9e6e47248bf4325e2f6eec9d789216dc7536e753bf1f4be603d9fa2e6f5e192b4eb988b8cdb0bb1e8668a9225426f7d4636479f73ed24ad1d2657c31e63c93d9679b9080171b3bd1bf10a3b92b80bd790fbf62d3644900cd08eae8b9bf9c2567be98dc8cdd2ae19a8d57a3e3e2de899f81f1279f578989e6af906f80c8c2b67651730ee7e568c1af5bcb845b6d685dc55332a9984aeceaea3b7e883447edf1c76b155d95253e39b9710eaa22efa6c81468829702b5dce7126538f3ca70c2f0ad9a5795435fdb1f715f20d60359ef9a9926c7050116e802df651727447848827815f70bd82af3cedd08783156102d2d8ce995c4c43b8e47e91a3e6927f3505a5d395e6bebb84542c570903eeab4382a1c2151f1471c7a06a34dc4d268d8fa72e93bdcd2dccc4302ecac47b9e7e3d8bc9b46d21cd097874a24d529548018dc190ff568c6aa428f0a5eedff1a347730931c74f19277538e49647a4ad7254f4c1ec7d4da12cce9e1fad9607534e66ab40a56b473d9d7e3d563fd03cad2052bad365c5a29f8ae54f09b60dbca3ea768d7767cbe1c133ca08ce725c1c1370f4aab8e5b6e286f52dc0be8d0982b5a" timestamp="1480431575" url="https://MyFirstFDroidRepo.org/fdroid/repo" version="21">
+ <repo name="My First F-Droid Repo Demo" icon="fdroid-icon.png" url="https://MyFirstFDroidRepo.org/fdroid/repo" version="21" timestamp="1480431575" pubkey="308204e1308202c9a003020102020434597643300d06092a864886f70d01010b050030213110300e060355040b1307462d44726f6964310d300b06035504031304736f7661301e170d3136303931333230313930395a170d3434303133303230313930395a30213110300e060355040b1307462d44726f6964310d300b06035504031304736f766130820222300d06092a864886f70d01010105000382020f003082020a028202010086ef94b5aacf2ba4f38c875f4194b44f5644392e3715575d7c92828577e692c352b567172823851c8c72347fbc9d99684cd7ca3e1db3e4cca126382c53f2a5869fb4c19bdec989b2930501af3e758ff40588915fe96b10076ce3346a193a0277d79e83e30fd8657c20e35260dd085aa32eac7c4b85786ffefbf1555cafe2bc928443430cdbba48cfbe701e12ae86e676477932730d4fc7c00af820aef85038a5b4df084cf6470d110dc4c49ea1b749b80b34709d199b3db516b223625c5de4501e861f7d261b3838f8f616aa78831d618d41d25872dc810c9b2087b5a9e146ca95be740316dcdbcb77314e23ab87d4487913b800b1113c0603ea2294188b71d3e49875df097b56f9151211fc6832f9790c5c83d17481f14ad37915fd164f4fd713f6732a15f4245714b84cd665bdbd085660ea33ad7d7095dcc414f09e3903604a40facc2314a115c0045bb50e9df38efb57e1b8e7cc105f340a26eeb46aba0fa6672953eee7f1f92dcb408e561909bbd4bdf4a4948c4d57c467d21aa238c34ba43be050398be963191fa2b49828bc1e4eeed224b40dbe9dc3e570890a71a974a2f4527edb1b07105071755105edcb2af2f269facfb89180903a572a99b46456e80d4a01685a80b233278805f2c876678e731f4ec4f52075aeef6b2b023efbb8a3637ef507c4c37c27e428152ec1817fcba640ad601cb09f72f0fbe2d274a2410203010001a321301f301d0603551d0e04160414c28bf33dd5a9a17338e5b1d1a6edd8c7d141ed0b300d06092a864886f70d01010b0500038202010084e20458b2aafd7fc27146b0986f9324f4260f244920417a77c9bf15e2e2d22d2725bdd8093ec261c3779c3ca03312516506f9410075b90595b41345956d8eb2786fb5994f195611382c2b99dba13381b0100a30bc9e6e47248bf4325e2f6eec9d789216dc7536e753bf1f4be603d9fa2e6f5e192b4eb988b8cdb0bb1e8668a9225426f7d4636479f73ed24ad1d2657c31e63c93d9679b9080171b3bd1bf10a3b92b80bd790fbf62d3644900cd08eae8b9bf9c2567be98dc8cdd2ae19a8d57a3e3e2de899f81f1279f578989e6af906f80c8c2b67651730ee7e568c1af5bcb845b6d685dc55332a9984aeceaea3b7e883447edf1c76b155d95253e39b9710eaa22efa6c81468829702b5dce7126538f3ca70c2f0ad9a5795435fdb1f715f20d60359ef9a9926c7050116e802df651727447848827815f70bd82af3cedd08783156102d2d8ce995c4c43b8e47e91a3e6927f3505a5d395e6bebb84542c570903eeab4382a1c2151f1471c7a06a34dc4d268d8fa72e93bdcd2dccc4302ecac47b9e7e3d8bc9b46d21cd097874a24d529548018dc190ff568c6aa428f0a5eedff1a347730931c74f19277538e49647a4ad7254f4c1ec7d4da12cce9e1fad9607534e66ab40a56b473d9d7e3d563fd03cad2052bad365c5a29f8ae54f09b60dbca3ea768d7767cbe1c133ca08ce725c1c1370f4aab8e5b6e286f52dc0be8d0982b5a">
<description>This is a repository of apps to be used with F-Droid. Applications in this repository are either official binaries built by the original application developers, or are binaries built from source by the admin of f-droid.org using the tools on https://gitlab.com/u/fdroid. </description>
<mirror>http://foobarfoobarfoobar.onion/fdroid/repo</mirror>
<mirror>https://foo.bar/fdroid/repo</mirror>
@@ -94,9 +94,9 @@
<added>2017-12-22</added>
<sig>056c9f1554c40ba59a2103009c82b420</sig>
<permissions>ACCESS_NETWORK_STATE,ACCESS_WIFI_STATE,CHANGE_WIFI_MULTICAST_STATE,INTERNET,READ_EXTERNAL_STORAGE,WRITE_EXTERNAL_STORAGE</permissions>
- <uses-permission maxSdkVersion="18" name="android.permission.READ_EXTERNAL_STORAGE"/>
- <uses-permission maxSdkVersion="18" name="android.permission.WRITE_EXTERNAL_STORAGE"/>
- <uses-permission-sdk-23 maxSdkVersion="27" name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS"/>
+ <uses-permission name="android.permission.READ_EXTERNAL_STORAGE" maxSdkVersion="18"/>
+ <uses-permission name="android.permission.WRITE_EXTERNAL_STORAGE" maxSdkVersion="18"/>
+ <uses-permission-sdk-23 name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" maxSdkVersion="27"/>
</package>
</application>
<application id="fake.ota.update">
@@ -182,9 +182,9 @@
<added>2013-12-31</added>
<sig>eb41d4d6082bb3e81c3d58dbf7fc7332</sig>
<permissions>ACCESS_NETWORK_STATE,ACCESS_WIFI_STATE,BLUETOOTH,BLUETOOTH_ADMIN,CHANGE_NETWORK_STATE,CHANGE_WIFI_MULTICAST_STATE,CHANGE_WIFI_STATE,INTERNET,NFC,RECEIVE_BOOT_COMPLETED</permissions>
- <uses-permission maxSdkVersion="22" name="android.permission.ACCESS_NETWORK_STATE"/>
- <uses-permission maxSdkVersion="18" name="android.permission.BLUETOOTH_ADMIN"/>
- <uses-permission-sdk-23 maxSdkVersion="25" name="android.permission.WRITE_SETTINGS"/>
+ <uses-permission name="android.permission.ACCESS_NETWORK_STATE" maxSdkVersion="22"/>
+ <uses-permission name="android.permission.BLUETOOTH_ADMIN" maxSdkVersion="18"/>
+ <uses-permission-sdk-23 name="android.permission.WRITE_SETTINGS" maxSdkVersion="25"/>
</package>
</application>
<application id="obb.main.twoversions">
```
When downloading a repo index, the downloaded index got written to a
file with `.write()` in a `with` clause. Before the file got actually
written to the disk, it got already passed into the next function,
resulting in a `VerificationException`:
```
JAR signature failed to verify: /tmp/tmppq2r51r0
jarsigner: java.util.zip.ZipException: zip file is empty
```
This behavior got introduced in
869cc114a3.
I've found this bug with help of Repomaker's tests: https://gitlab.com/fdroid/repomaker/merge_requests/215#note_148994053
The mirrors are displayed in fdroidclient in the order they are received,
and there might be some reason why a repo maintainer wants them in a
specific order. The danger is that if the mirrors are set in config.py
using "mirrors = {'foo', bar'}" they will have a randomized order since it
will be in a set. They should be set using [] or () to have a fixed order.
androguard's v2 signature methods return DER-encoded certificates. The DER-
encoded certificates are parsed from JAR Signature's raw "Signature Block
File".
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle. For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed. That is not possible with JSON.
This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.
closes#163
Instead of just crashing, first try to use the versionName as written in the
build metadata, otherwise just let it be blank. A blank versionName will
cause fdroidclient < 1.3 to crash. Blank versionNames are not allowed in
the .txt metadata format, only .yml.
closes#477closes#478
closes fdroidclient#1416
closes fdroidclient#1417
closes fdroidclient#1418
fdroiddata!3061
GitLab storage provides two mirrors by default:
* https://gitlab.com/user/repo/raw/master/fdroid/repo
* https://user.gitlab.io/repo/fdroid/repo
While the F-Droid client will happily fetch the index*.jar files and
parse them from either of these two mirrors, only the GitLab Pages
mirror will serve files with the correct mime type. Many repos
tend to put index.html files (and associated .css/.js/image files) in
the root of a repository to provide information about that repo.
One example is RepoMaker. The way in which RepoMaker decides the public
URL of a repo, is to take the first mirror in the list. This means that
the URL which RepoMaker directs people to for GitLab storage returns a
.html document in text/plain, which means that it is not rendered.
We could change RepoMaker so that it takes the last mirror, and then it
woruld work. However there is something nice about the first mirror in a
list being the most authoritative (even though the mirror order doesn't
- and perhaps shouldn't have any specific meaning).
These were both spamming the output with lots of confusing messages, even
when --verbose was not used. Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.
closes#405
apksigner v0.7 (build-tools 26.0.1), Google made it require that the
AndroidManifest.xml was present in the archive before it verifies the
signature. So this needs to stick with the jarsigner hack for JARs.
This allows all the text to be localized via Weblate. This is a quick
overview of all the strings, but there are certainly some that were left
out.
closes#342
With the new localization support, the name/summary/description in the
metadata file becomes the global override. So most apps are not going to
have those fields present in their metadata file. This fixes the index.xml
generation to fall back to the localized versions of those fields when they
are not set in the metadata field.
https://forum.f-droid.org/t/what-has-happend-to-osmand
This makes `fdroid server update` fail if pushing to one of the git mirrors
fails. This is what happens if the other methods fail, e.g. rsync or S3.
closes#347
gitlab serves raw files from a CDN, so its appropriate to use the raw URL.
@pserwylo @grote and I discussed it and found a reference, but I can't find
that reference now.
Since the client will try the next mirror if one fails, it makes sense to
include both the gitlab raw and gitlab pages URLs to the mirror. The
gitlab pages deploy process is still a bit flaky anyway.
