We want to keep the stable version (CV) of an app in /repo with highest
priority and only move it to /archive when ArchivePolicy is set to 0.
For this to work we need two changes here:
1) when sorting apks by version code we insert the apk corresponding to
the CVC of the app afterwards with highest priority. So when walking the
list of apks afterwards the CVC apk is always kept first.
2) Instead of the two pass algorithm of moving things back and forth to
the archive we instead figure out where each apk of an app goes in first
before actually moving them into the right place.
Fixes: fdroid/fdroidserver#385
Python PIL is not so tolerant, so bad EXIF causes crashes:
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main
insert_localized_app_metadata(apps)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata
_strip_and_copy_image(os.path.join(root, f), destdir)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image
in_image = Image.open(fp)
File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open
% (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>
Using a filename based on the hash of the contents means that the caching
algorithms for fdroidclient and browsers can safely cache the file forever
using the filename, since this guarantees that the contents will never
change for a given filename.
This does not cover screenshots, only icon.png, featureGraphic.png,
tvBanner.png, and promoGraphic.png.
fdroidserver#689
fdroid-website!453
This was done with much help from @uniqx. This is the first level of
supporting APK Signatures v1, v2, and v3. This is enough to include
APKs with any combo of v1/v2/v3 signatures. For this to work at all,
apksigner and androguard 3.3.3+ must be installed.
closes#399
androguard's v2 signature methods return DER-encoded certificates. The DER-
encoded certificates are parsed from JAR Signature's raw "Signature Block
File".
Also, update the documentation comment to be absolutely clear.
The function used to copy images from any locale directory starting with at least two lowercase letters correctly, but only add those to the app metadata that started with two letters plus one big letter or a "-". The same problem did not exist for adding texts to the metadata.
build-tools 28.0.3 added a new field in the end 'compileSdkVersionCodename=', which also accidentally ends with the string 'name='.
The purpose of this regex was to catch the 'packageName' field, which is in ht eaapt ouput the exact ' name=', therefore added whe non-caracter \W prefix match.
sample aapt output (28.0.3):
package: name='com.a.b.app' versionCode='1' versionName='1.0' compileSdkVersion='28' compileSdkVersionCodename='9'
previously, regex was catching second occurence, so '9'
See merge request !582
Fastlane/Triple-T app store metadata files must be in UTF-8. Before this
would crash if they were not. This changes the handling to just replace
the non-UTF8 chars with a �. Here's the stacktrace:
CRITICAL: Unknown exception found!
Traceback (most recent call last):
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 164, in <module>
main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 159, in main
raise e
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 138, in main
mod.main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2010, in main
insert_localized_app_metadata(apps)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 885, in insert_localized_app_metadata
os.path.join(root, f))
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 700, in _set_localized_text_entry
text = fp.read()[:limit]
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 94: invalid start byte
699b3e4c69 got it wrong for targetSdkVersion.
Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for
com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk.
F-Droid never really supported running on android-1 or android-2, so it
seems pointless to debug support for them.
Android Application IDs must be valid Java Package Names. While the build
tools likely validate the Application ID, it is possible to manually create
a malicious APK.
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle. For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed. That is not possible with JSON.
This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.
closes#163
Sometimes androguard returns the XMLNS as entirely empty, which would make
it an invalid APK since normally the 'android' name is mapped to the
'http://schemas.amazon.com/apk/res/android' value. Occasionally, a
different key is used.
closesfdroid/fdroidserver#515
fixes de35f1b05b:
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 164, in <module>
main()
File "/home/hans/code/fdroid/server/fdroid", line 138, in main
mod.main()
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1932, in main
apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1459, in process_apks
use_date_from_apk, ada, True)
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1332, in process_apk
apk = scan_apk(apkfile)
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1051, in scan_apk
scan_apk_androguard(apk, apk_file)
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1220, in scan_apk_androguard
res_id = arsc.get_id(apk['packageName'], res_id)[1]
TypeError: 'NoneType' object is not subscriptable
For example https://f-droid.org/archive/com.abitsinc.andr_5.apk:
$ aapt dump badging archive/com.abitsinc.andr_5.apk |head -1
package: name='com.abitsinc.andr' versionCode='5' versionName='5' platformBuildVersionName='2.3.3'
Instead of just crashing, first try to use the versionName as written in the
build metadata, otherwise just let it be blank. A blank versionName will
cause fdroidclient < 1.3 to crash. Blank versionNames are not allowed in
the .txt metadata format, only .yml.
closes#477closes#478
closes fdroidclient#1416
closes fdroidclient#1417
closes fdroidclient#1418
fdroiddata!3061
