Commit graph

517 commits

Author SHA1 Message Date
Hans-Christoph Steiner
e17815e9f0 handle bad SDK Version values in APKs
Even though it is invalid to have *SdkVersion in AndroidManifest.xml set as
anything but an integer, sometimes people manage to get something in there.
`fdroid update` needs to be able to handle all that.

* https://developer.android.com/guide/topics/manifest/uses-sdk-element#min
* https://gitlab.com/souch/SMSbypass/blob/v0.9/app/src/main/AndroidManifest.xml#L29
* https://gitlab.com/souch/SMSbypass/blob/v0.9/app/src/main/res/values/strings.xml#L27

admin#65
2018-04-26 15:19:41 +02:00
Hans-Christoph Steiner
afe0c6a167 fix scan_apk_androguard when parsing versionName without a dot (e.g. 5)
fixes de35f1b05b:

Traceback (most recent call last):
  File "/home/hans/code/fdroid/server/fdroid", line 164, in <module>
    main()
  File "/home/hans/code/fdroid/server/fdroid", line 138, in main
    mod.main()
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1932, in main
    apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1459, in process_apks
    use_date_from_apk, ada, True)
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1332, in process_apk
    apk = scan_apk(apkfile)
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1051, in scan_apk
    scan_apk_androguard(apk, apk_file)
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1220, in scan_apk_androguard
    res_id = arsc.get_id(apk['packageName'], res_id)[1]
TypeError: 'NoneType' object is not subscriptable

For example https://f-droid.org/archive/com.abitsinc.andr_5.apk:
$ aapt dump badging archive/com.abitsinc.andr_5.apk |head -1
package: name='com.abitsinc.andr' versionCode='5' versionName='5' platformBuildVersionName='2.3.3'
2018-04-18 12:32:28 +02:00
Hans-Christoph Steiner
de35f1b05b update: handle APKs with a blank versionName
Instead of just crashing, first try to use the versionName as written in the
build metadata, otherwise just let it be blank.  A blank versionName will
cause fdroidclient < 1.3 to crash.  Blank versionNames are not allowed in
the .txt metadata format, only .yml.

closes #477
closes #478
closes fdroidclient#1416
closes fdroidclient#1417
closes fdroidclient#1418
fdroiddata!3061
2018-04-17 12:36:49 +02:00
Hans-Christoph Steiner
9bf9159a77 update: use only 7 chars of SHA256 for non-APK version name
#460
2018-04-17 12:36:49 +02:00
Phoenix09
ea89b25fd7 fdroidserver/update.py: APK_LABEL_ICON_PAT icon pattern should not be greedy 2018-04-14 23:30:23 +01:00
Ciaran Gultnieks
bd00b87e4e Filter another invalid mediawiki page title character 2018-04-11 14:32:55 +01:00
Ciaran Gultnieks
31e8605864 Replace deprecated mwclient method
It's functionally the same, just a different name.
2018-04-08 09:37:48 +01:00
Jonas Kalderstam
1f706283ce Fix crash when icon_name is None
Fixes the following crash:

```
$ fdroid update --create-metadata --rename-apks
WARNING: Using Java's jarsigner, not recommended for verifying APKs! Use apksigner
CRITICAL: Unknown exception found!
Traceback (most recent call last):
  File "/home/jonas/miniconda3/bin/fdroid", line 164, in <module>
    main()
  File "/home/jonas/miniconda3/bin/fdroid", line 138, in main
    mod.main()
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1927, in main
    apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1454, in process_apks
    use_date_from_apk, ada, True)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1327, in process_apk
    apk = scan_apk(apkfile)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1056, in scan_apk
    scan_apk_aapt(apk, apk_file)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1184, in scan_apk_aapt
    apk['icons_src'] = _get_apk_icons_src(apkfile, icon_name)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1089, in _get_apk_icons_src
    density_re = re.compile('^res/(.*)/' + icon_name + '\.(png|xml)$')
TypeError: must be str, not NoneType
```
2018-03-27 14:17:08 +02:00
Hans-Christoph Steiner
54242dee32 update: handle AndroidManifest XML namespace named other than 'android'
https://stackoverflow.com/questions/5856719/why-do-we-specify-namespace-in-android-xml-file

https://f-droid.org/repo/org.moire.ultrasonic_60.apk has:
xmlns:a="http://schemas.android.com/apk/res/android"
2018-03-15 15:56:30 +01:00
Hans-Christoph Steiner
b5c1634320 update: handle renameManifestPackage when extracting icons with androguard
aapt --rename-manifest-package changes the applicationId for an app without
changing the packageName listed in AndroidManifest.xml under
<application android:package="">
2018-03-14 18:43:05 +01:00
Hans-Christoph Steiner
493c485755 update: support all official DPIs when extracting icons
https://developer.android.com/training/multiscreen/screendensities.html
2018-03-13 16:47:24 +01:00
Hans-Christoph Steiner
3b09e5ee08 update: do not crash on android-26+ XML icon
For example:
res/drawable-v26/icon.xml

Here's the full range of possibilities, we have a lot of work to do:

$ for f in */*.apk; do unzip -l $f |grep -F icon.| grep -Eo 'res/drawable[^/]*'; done | sort -u
res/drawable
res/drawable-anydpi-v21
res/drawable-anydpi-v26
res/drawable-hdpi
res/drawable-hdpi-v11
res/drawable-hdpi-v12
res/drawable-hdpi-v4
res/drawable-hdpi-v5
res/drawable-hdpi-v6
res/drawable-hdpi-v9
res/drawable-large
res/drawable-large-hdpi-v11
res/drawable-large-hdpi-v4
res/drawable-large-ldpi-v4
res/drawable-large-mdpi
res/drawable-large-mdpi-v11
res/drawable-large-v4
res/drawable-large-xhdpi-v11
res/drawable-large-xhdpi-v4
res/drawable-large-xxhdpi-v11
res/drawable-large-xxhdpi-v4
res/drawable-ldpi
res/drawable-ldpi-v11
res/drawable-ldpi-v4
res/drawable-ldpi-v5
res/drawable-ldpi-v6
res/drawable-ldpi-v9
res/drawable-ldrtl-v17
res/drawable-mdpi
res/drawable-mdpi-v11
res/drawable-mdpi-v12
res/drawable-mdpi-v4
res/drawable-mdpi-v5
res/drawable-mdpi-v6
res/drawable-mdpi-v9
res/drawable-nodpi
res/drawable-nodpi-v4
res/drawable-tvdpi-v4
res/drawable-v11
res/drawable-v14
res/drawable-v21
res/drawable-xhdpi
res/drawable-xhdpi-v11
res/drawable-xhdpi-v12
res/drawable-xhdpi-v4
res/drawable-xhdpi-v9
res/drawable-xlarge-hdpi
res/drawable-xlarge-hdpi-v4
res/drawable-xlarge-mdpi
res/drawable-xlarge-mdpi-v4
res/drawable-xlarge-v4
res/drawable-xlarge-xhdpi-v4
res/drawable-xxhdpi
res/drawable-xxhdpi-v11
res/drawable-xxhdpi-v21
res/drawable-xxhdpi-v4
res/drawable-xxhdpi-v9
res/drawable-xxxhdpi
res/drawable-xxxhdpi-v11
res/drawable-xxxhdpi-v4
2018-03-13 12:13:23 +01:00
Hans-Christoph Steiner
e89478e529 update: do not crash on APKs without icons when using androguard 2018-03-13 11:19:19 +01:00
Hans-Christoph Steiner
2c8008356b update: handle bad XML when using androguard
repo/ch.swift.willi_417101.apk had a C/Java comment in the
AndroidManifest.xml rather than an XML comment:

<uses-sdk android:minSdkVersion="16" android:targetSdkVersion="26">
</uses-sdk>
// Remove permissions introduced by the appsflyer library
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION">
</uses-permission>
2018-03-12 17:55:45 +01:00
Hans-Christoph Steiner
d98d8c2e52 update: do not crash with androguard when working with apps without icons
For example, SpeedoMeterApp.main_1.apk
2018-03-12 17:55:45 +01:00
Hans-Christoph Steiner
70d9633555 build/checkupdates/update: log current fdroiddata commit to wiki 2018-03-05 21:49:09 +01:00
Hans-Christoph Steiner
b2ca49b26c update: make icon extraction less dependent on aapt
For androguard, @thezero already developed a way to get all the icons after
only extracting the icon name.  So this uses that for the aapt-based scans
also, to make them less brittle.

This should fix the problem where `fdroid update` was choosing the XML icon
for apps that include one, like NewPipe.

closes fdroid/fdroid-website#192
2018-02-26 23:43:42 +01:00
Hans-Christoph Steiner
88e24dc4e3 update: switch to improved androguard detection 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
52b3436ff6 make is_apk_and_debuggable() default to using androguard before aapt 2018-02-22 15:08:53 +01:00
Hans-Christoph Steiner
aa4f54bf18 update: include implied permissions when using androguard
`aapt dump badging` includes these when listing uses-permissions:
https://github.com/androguard/androguard/pull/428
2018-02-21 12:34:54 +01:00
Hans-Christoph Steiner
63d4d46291 update: 'features' list only includes required features
The F-Droid index 'features' list is not the same as what is in the
AndroidManifest.xml.  It only includes "required" features, for example.
2018-02-20 17:08:55 +01:00
Hans-Christoph Steiner
008110889a update: fix crash from missing file extension in extracted icon
I missed this in 40fac10ebc, yay tests!
2018-02-20 16:29:52 +01:00
Hans-Christoph Steiner
d1ded7f64d update: use androguard by default if it is available
closes #236
2018-02-15 14:28:48 +01:00
Hans-Christoph Steiner
5281228ea5 update: switch to new androguard v3.1 API based on lxml
apkobject.get_android_manifest_xml() used to return a xml.dom.minidom
object, now it returns an lxml.etree.Element object.
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
699b3e4c69 update: fix min/target/max edge case parsing with androguard
In order to test that aapt defaults minSdkVersion to 3, I ran this script
then compared the output with meld:

cd $ANDROID_HOME/build-tools
for d in *.*; do echo $d; $ANDROID_HOME/build-tools/$d/aapt dump badging /home/hans/code/fdroid/server/tests/repo/com.politedroid_3.apk > /tmp/${d}.txt; done
meld /tmp/17.0.0.txt /tmp/26.0.2.txt /tmp/27.0.3.txt
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
40fac10ebc update: extract and store XML icons
These can then be used by the client.

#344
#392
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
5713b54e0b update: find alternate PNGs for apps that have an XML app icon
Apps can now use an XML icon, but if the app supports older Android
versions, it'll also contain PNG versions of the same icon.  This finds
those PNGs and uses them instead.

#344
closes #392
fdroiddata#913
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
1f99a27a9c Revert "wiki: include per-app link to all related activity on gitlab.com"
This reverts commit a57f17b276.

Turns out this was totally useless, this can be fully handled in
https://f-droid.org/wiki/page/Template:App since it already has the
Application ID there.
2018-02-12 12:15:56 +01:00
Hans-Christoph Steiner
a57f17b276 wiki: include per-app link to all related activity on gitlab.com 2018-01-26 10:18:41 +01:00
Hans-Christoph Steiner
010f1c5029 log installed android sdk versions for update and checkupdates 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
4beb2d52e9 wiki: log update start/stop time and command line 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
5b92820ff3 wiki: fix bug updating Repository Maintenance
site.pages doesn't seem to exist anywhere, site.Pages is used throughout.
2018-01-22 13:49:10 +01:00
Izzy
42ac65e8aa simplifying fix for "cannot identify image file" with XML icons 2018-01-17 16:48:08 +01:00
Izzy
6f5b539a54 fix "cannot identify image file" with XML icons 2018-01-12 22:12:27 +01:00
Hans-Christoph Steiner
2659312a7c update: support working with old versions of PIL/Pillow
Image.close() was added in Pillow 2.4 or so.
2017-12-20 23:46:37 +01:00
Michael Pöhn
c17aeb5231 fix handling unreadable images in update.extract_apk_icons 2017-12-16 22:06:20 +01:00
Hans-Christoph Steiner
42522c23c9 update: do not crash if AndroidManifest.xml in APK has invalid date
This crash actually blocked a Janus exploit APK from being added to the
repo, but crashing isn't really the appropriate way to do that.
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
8f45796ecb update: close unclosed Image instance 2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
387eebc4d6 update: strip all metadata from PNGs
This strips metadata and optimizes the compression of all PNGs copied
from the app's source repo as well as all the icons extracted from the
APKs.  There have been exploits delivered via image metadata, and
F-Droid isn't using it all, so its best to just remove it.

This unfortunately uncompresses and recompresses the files.  Luckily,
that's a lossless procedure with PNGs, and we might end up with
smaller files.  The only tool I could find that strips without
changing the image data is exiftool, but that is written in Perl.
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
67b9514c5a update: strip EXIF data from all JPEGs
EXIF data can be abused to exploit systems a lot easier than the JPEG image
data can.  The F-Droid ecosystem does not use the EXIF data, so keep things
safe and strip it all away.  There is a chance that some images might rely
on the rotation to be set by EXIF, but I think having a safe system is more
important.

If needed, only the rotation data could be saved.  But that then makes it
hard to tell which images have been stripped.  This way, if there is no
EXIF, it has been stripped.  And if there is EXIF data, then it is suspect.

https://securityaffairs.co/wordpress/51043/mobile-2/android-cve-2016-3862-flaw.html
https://threatpost.com/google-shuts-down-potentially-massive-android-bug/120393/
https://blog.sucuri.net/2013/07/malware-hidden-inside-jpg-exif-headers.html

The big downside of this is that it decompresses and recompresses the
image data.  That should be replaced by a technique from jhead,
exiftool, ObscuraCam, etc. that only strips the metadata.
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
bde0558d82 update: reject APKs with invalid file sig, probably Janus exploits
This just checks the first four bytes of the APK file, aka the "file
signature", to make sure it is the ZIP signature and not the DEX signature.
This was checked against the test APK, and I ran it against some known
malware and all of f-droid.org to make sure it works.

All valid ZIP files (therefore APK files) should start with the ZIP
Local File Header of four bytes.

https://www.guardsquare.com/en/blog/new-android-vulnerability-allows-attackers-modify-apps-without-affecting-their-signatures
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
5ce950e748 update: print warnings for all KnownVulns found
Some baby steps towards making the KnownVuln stuff more visible.
2017-12-14 16:57:22 +01:00
Andrea Scarpino
19a83b6219 Add Liberapay support 2017-12-12 11:53:31 +01:00
Hans-Christoph Steiner
5d54446efc update: do not replace Name/Summary from template unless blank
`fdroid nightly` needs this change so it can set the Summary using the
template.
2017-12-07 22:39:32 +01:00
tobiasKaminsky
d5ecb34df7
add flavour to metadata 2017-12-01 15:25:00 +01:00
Hans-Christoph Steiner
d46d9574b4 update: use KnownApks dates to check system clock on offline machines
KnownApks provides a reliable source of a relatively recent date.
2017-11-29 21:06:02 +01:00
Hans-Christoph Steiner
9ade09349b update: improve help strings 2017-11-08 14:05:17 +01:00
Hans-Christoph Steiner
7be46d53c7 sort Anti-Features before writing them to the wiki
This avoids silly updates to the wiki where the sort order of the Anti-
Features is the only change:

−{{AntiFeature|KnownVuln}}
 {{AntiFeature|DisabledAlgorithm}}
+{{AntiFeature|KnownVuln}}

https://f-droid.org/wiki/index.php?title=info.guardianproject.browser&curid=517&diff=189655&oldid=188037
2017-10-27 09:15:22 +02:00
Ciaran Gultnieks
505d2c1325 update bombs out when no icons (closes #402) 2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
ac69d5b17d update: make all strings translatable 2017-10-25 23:01:25 +02:00