Fastlane/Triple-T app store metadata files must be in UTF-8. Before this
would crash if they were not. This changes the handling to just replace
the non-UTF8 chars with a �. Here's the stacktrace:
CRITICAL: Unknown exception found!
Traceback (most recent call last):
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 164, in <module>
main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 159, in main
raise e
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroid", line 138, in main
mod.main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2010, in main
insert_localized_app_metadata(apps)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 885, in insert_localized_app_metadata
os.path.join(root, f))
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 700, in _set_localized_text_entry
text = fp.read()[:limit]
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 94: invalid start byte
699b3e4c69 got it wrong for targetSdkVersion.
Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for
com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk.
F-Droid never really supported running on android-1 or android-2, so it
seems pointless to debug support for them.
Android Application IDs must be valid Java Package Names. While the build
tools likely validate the Application ID, it is possible to manually create
a malicious APK.
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle. For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed. That is not possible with JSON.
This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.
closes#163
Sometimes androguard returns the XMLNS as entirely empty, which would make
it an invalid APK since normally the 'android' name is mapped to the
'http://schemas.amazon.com/apk/res/android' value. Occasionally, a
different key is used.
closesfdroid/fdroidserver#515
fixes de35f1b05b:
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 164, in <module>
main()
File "/home/hans/code/fdroid/server/fdroid", line 138, in main
mod.main()
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1932, in main
apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1459, in process_apks
use_date_from_apk, ada, True)
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1332, in process_apk
apk = scan_apk(apkfile)
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1051, in scan_apk
scan_apk_androguard(apk, apk_file)
File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1220, in scan_apk_androguard
res_id = arsc.get_id(apk['packageName'], res_id)[1]
TypeError: 'NoneType' object is not subscriptable
For example https://f-droid.org/archive/com.abitsinc.andr_5.apk:
$ aapt dump badging archive/com.abitsinc.andr_5.apk |head -1
package: name='com.abitsinc.andr' versionCode='5' versionName='5' platformBuildVersionName='2.3.3'
Instead of just crashing, first try to use the versionName as written in the
build metadata, otherwise just let it be blank. A blank versionName will
cause fdroidclient < 1.3 to crash. Blank versionNames are not allowed in
the .txt metadata format, only .yml.
closes#477closes#478
closes fdroidclient#1416
closes fdroidclient#1417
closes fdroidclient#1418
fdroiddata!3061
Fixes the following crash:
```
$ fdroid update --create-metadata --rename-apks
WARNING: Using Java's jarsigner, not recommended for verifying APKs! Use apksigner
CRITICAL: Unknown exception found!
Traceback (most recent call last):
File "/home/jonas/miniconda3/bin/fdroid", line 164, in <module>
main()
File "/home/jonas/miniconda3/bin/fdroid", line 138, in main
mod.main()
File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1927, in main
apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1454, in process_apks
use_date_from_apk, ada, True)
File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1327, in process_apk
apk = scan_apk(apkfile)
File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1056, in scan_apk
scan_apk_aapt(apk, apk_file)
File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1184, in scan_apk_aapt
apk['icons_src'] = _get_apk_icons_src(apkfile, icon_name)
File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1089, in _get_apk_icons_src
density_re = re.compile('^res/(.*)/' + icon_name + '\.(png|xml)$')
TypeError: must be str, not NoneType
```
aapt --rename-manifest-package changes the applicationId for an app without
changing the packageName listed in AndroidManifest.xml under
<application android:package="">
repo/ch.swift.willi_417101.apk had a C/Java comment in the
AndroidManifest.xml rather than an XML comment:
<uses-sdk android:minSdkVersion="16" android:targetSdkVersion="26">
</uses-sdk>
// Remove permissions introduced by the appsflyer library
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION">
</uses-permission>
For androguard, @thezero already developed a way to get all the icons after
only extracting the icon name. So this uses that for the aapt-based scans
also, to make them less brittle.
This should fix the problem where `fdroid update` was choosing the XML icon
for apps that include one, like NewPipe.
closesfdroid/fdroid-website#192
