Commit graph

610 commits

Author SHA1 Message Date
Jonas Kalderstam
1f706283ce Fix crash when icon_name is None
Fixes the following crash:

```
$ fdroid update --create-metadata --rename-apks
WARNING: Using Java's jarsigner, not recommended for verifying APKs! Use apksigner
CRITICAL: Unknown exception found!
Traceback (most recent call last):
  File "/home/jonas/miniconda3/bin/fdroid", line 164, in <module>
    main()
  File "/home/jonas/miniconda3/bin/fdroid", line 138, in main
    mod.main()
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1927, in main
    apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1454, in process_apks
    use_date_from_apk, ada, True)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1327, in process_apk
    apk = scan_apk(apkfile)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1056, in scan_apk
    scan_apk_aapt(apk, apk_file)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1184, in scan_apk_aapt
    apk['icons_src'] = _get_apk_icons_src(apkfile, icon_name)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1089, in _get_apk_icons_src
    density_re = re.compile('^res/(.*)/' + icon_name + '\.(png|xml)$')
TypeError: must be str, not NoneType
```
2018-03-27 14:17:08 +02:00
Hans-Christoph Steiner
54242dee32 update: handle AndroidManifest XML namespace named other than 'android'
https://stackoverflow.com/questions/5856719/why-do-we-specify-namespace-in-android-xml-file

https://f-droid.org/repo/org.moire.ultrasonic_60.apk has:
xmlns:a="http://schemas.android.com/apk/res/android"
2018-03-15 15:56:30 +01:00
Hans-Christoph Steiner
b5c1634320 update: handle renameManifestPackage when extracting icons with androguard
aapt --rename-manifest-package changes the applicationId for an app without
changing the packageName listed in AndroidManifest.xml under
<application android:package="">
2018-03-14 18:43:05 +01:00
Hans-Christoph Steiner
493c485755 update: support all official DPIs when extracting icons
https://developer.android.com/training/multiscreen/screendensities.html
2018-03-13 16:47:24 +01:00
Hans-Christoph Steiner
3b09e5ee08 update: do not crash on android-26+ XML icon
For example:
res/drawable-v26/icon.xml

Here's the full range of possibilities, we have a lot of work to do:

$ for f in */*.apk; do unzip -l $f |grep -F icon.| grep -Eo 'res/drawable[^/]*'; done | sort -u
res/drawable
res/drawable-anydpi-v21
res/drawable-anydpi-v26
res/drawable-hdpi
res/drawable-hdpi-v11
res/drawable-hdpi-v12
res/drawable-hdpi-v4
res/drawable-hdpi-v5
res/drawable-hdpi-v6
res/drawable-hdpi-v9
res/drawable-large
res/drawable-large-hdpi-v11
res/drawable-large-hdpi-v4
res/drawable-large-ldpi-v4
res/drawable-large-mdpi
res/drawable-large-mdpi-v11
res/drawable-large-v4
res/drawable-large-xhdpi-v11
res/drawable-large-xhdpi-v4
res/drawable-large-xxhdpi-v11
res/drawable-large-xxhdpi-v4
res/drawable-ldpi
res/drawable-ldpi-v11
res/drawable-ldpi-v4
res/drawable-ldpi-v5
res/drawable-ldpi-v6
res/drawable-ldpi-v9
res/drawable-ldrtl-v17
res/drawable-mdpi
res/drawable-mdpi-v11
res/drawable-mdpi-v12
res/drawable-mdpi-v4
res/drawable-mdpi-v5
res/drawable-mdpi-v6
res/drawable-mdpi-v9
res/drawable-nodpi
res/drawable-nodpi-v4
res/drawable-tvdpi-v4
res/drawable-v11
res/drawable-v14
res/drawable-v21
res/drawable-xhdpi
res/drawable-xhdpi-v11
res/drawable-xhdpi-v12
res/drawable-xhdpi-v4
res/drawable-xhdpi-v9
res/drawable-xlarge-hdpi
res/drawable-xlarge-hdpi-v4
res/drawable-xlarge-mdpi
res/drawable-xlarge-mdpi-v4
res/drawable-xlarge-v4
res/drawable-xlarge-xhdpi-v4
res/drawable-xxhdpi
res/drawable-xxhdpi-v11
res/drawable-xxhdpi-v21
res/drawable-xxhdpi-v4
res/drawable-xxhdpi-v9
res/drawable-xxxhdpi
res/drawable-xxxhdpi-v11
res/drawable-xxxhdpi-v4
2018-03-13 12:13:23 +01:00
Hans-Christoph Steiner
e89478e529 update: do not crash on APKs without icons when using androguard 2018-03-13 11:19:19 +01:00
Hans-Christoph Steiner
2c8008356b update: handle bad XML when using androguard
repo/ch.swift.willi_417101.apk had a C/Java comment in the
AndroidManifest.xml rather than an XML comment:

<uses-sdk android:minSdkVersion="16" android:targetSdkVersion="26">
</uses-sdk>
// Remove permissions introduced by the appsflyer library
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION">
</uses-permission>
2018-03-12 17:55:45 +01:00
Hans-Christoph Steiner
d98d8c2e52 update: do not crash with androguard when working with apps without icons
For example, SpeedoMeterApp.main_1.apk
2018-03-12 17:55:45 +01:00
Hans-Christoph Steiner
70d9633555 build/checkupdates/update: log current fdroiddata commit to wiki 2018-03-05 21:49:09 +01:00
Hans-Christoph Steiner
b2ca49b26c update: make icon extraction less dependent on aapt
For androguard, @thezero already developed a way to get all the icons after
only extracting the icon name.  So this uses that for the aapt-based scans
also, to make them less brittle.

This should fix the problem where `fdroid update` was choosing the XML icon
for apps that include one, like NewPipe.

closes fdroid/fdroid-website#192
2018-02-26 23:43:42 +01:00
Hans-Christoph Steiner
88e24dc4e3 update: switch to improved androguard detection 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
52b3436ff6 make is_apk_and_debuggable() default to using androguard before aapt 2018-02-22 15:08:53 +01:00
Hans-Christoph Steiner
aa4f54bf18 update: include implied permissions when using androguard
`aapt dump badging` includes these when listing uses-permissions:
https://github.com/androguard/androguard/pull/428
2018-02-21 12:34:54 +01:00
Hans-Christoph Steiner
63d4d46291 update: 'features' list only includes required features
The F-Droid index 'features' list is not the same as what is in the
AndroidManifest.xml.  It only includes "required" features, for example.
2018-02-20 17:08:55 +01:00
Hans-Christoph Steiner
008110889a update: fix crash from missing file extension in extracted icon
I missed this in 40fac10ebc, yay tests!
2018-02-20 16:29:52 +01:00
Hans-Christoph Steiner
d1ded7f64d update: use androguard by default if it is available
closes #236
2018-02-15 14:28:48 +01:00
Hans-Christoph Steiner
5281228ea5 update: switch to new androguard v3.1 API based on lxml
apkobject.get_android_manifest_xml() used to return a xml.dom.minidom
object, now it returns an lxml.etree.Element object.
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
699b3e4c69 update: fix min/target/max edge case parsing with androguard
In order to test that aapt defaults minSdkVersion to 3, I ran this script
then compared the output with meld:

cd $ANDROID_HOME/build-tools
for d in *.*; do echo $d; $ANDROID_HOME/build-tools/$d/aapt dump badging /home/hans/code/fdroid/server/tests/repo/com.politedroid_3.apk > /tmp/${d}.txt; done
meld /tmp/17.0.0.txt /tmp/26.0.2.txt /tmp/27.0.3.txt
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
40fac10ebc update: extract and store XML icons
These can then be used by the client.

#344
#392
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
5713b54e0b update: find alternate PNGs for apps that have an XML app icon
Apps can now use an XML icon, but if the app supports older Android
versions, it'll also contain PNG versions of the same icon.  This finds
those PNGs and uses them instead.

#344
closes #392
fdroiddata#913
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
1f99a27a9c Revert "wiki: include per-app link to all related activity on gitlab.com"
This reverts commit a57f17b276.

Turns out this was totally useless, this can be fully handled in
https://f-droid.org/wiki/page/Template:App since it already has the
Application ID there.
2018-02-12 12:15:56 +01:00
Hans-Christoph Steiner
a57f17b276 wiki: include per-app link to all related activity on gitlab.com 2018-01-26 10:18:41 +01:00
Hans-Christoph Steiner
010f1c5029 log installed android sdk versions for update and checkupdates 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
4beb2d52e9 wiki: log update start/stop time and command line 2018-01-22 13:49:10 +01:00
Hans-Christoph Steiner
5b92820ff3 wiki: fix bug updating Repository Maintenance
site.pages doesn't seem to exist anywhere, site.Pages is used throughout.
2018-01-22 13:49:10 +01:00
Izzy
42ac65e8aa simplifying fix for "cannot identify image file" with XML icons 2018-01-17 16:48:08 +01:00
Izzy
6f5b539a54 fix "cannot identify image file" with XML icons 2018-01-12 22:12:27 +01:00
Hans-Christoph Steiner
2659312a7c update: support working with old versions of PIL/Pillow
Image.close() was added in Pillow 2.4 or so.
2017-12-20 23:46:37 +01:00
Michael Pöhn
c17aeb5231 fix handling unreadable images in update.extract_apk_icons 2017-12-16 22:06:20 +01:00
Hans-Christoph Steiner
42522c23c9 update: do not crash if AndroidManifest.xml in APK has invalid date
This crash actually blocked a Janus exploit APK from being added to the
repo, but crashing isn't really the appropriate way to do that.
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
8f45796ecb update: close unclosed Image instance 2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
387eebc4d6 update: strip all metadata from PNGs
This strips metadata and optimizes the compression of all PNGs copied
from the app's source repo as well as all the icons extracted from the
APKs.  There have been exploits delivered via image metadata, and
F-Droid isn't using it all, so its best to just remove it.

This unfortunately uncompresses and recompresses the files.  Luckily,
that's a lossless procedure with PNGs, and we might end up with
smaller files.  The only tool I could find that strips without
changing the image data is exiftool, but that is written in Perl.
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
67b9514c5a update: strip EXIF data from all JPEGs
EXIF data can be abused to exploit systems a lot easier than the JPEG image
data can.  The F-Droid ecosystem does not use the EXIF data, so keep things
safe and strip it all away.  There is a chance that some images might rely
on the rotation to be set by EXIF, but I think having a safe system is more
important.

If needed, only the rotation data could be saved.  But that then makes it
hard to tell which images have been stripped.  This way, if there is no
EXIF, it has been stripped.  And if there is EXIF data, then it is suspect.

https://securityaffairs.co/wordpress/51043/mobile-2/android-cve-2016-3862-flaw.html
https://threatpost.com/google-shuts-down-potentially-massive-android-bug/120393/
https://blog.sucuri.net/2013/07/malware-hidden-inside-jpg-exif-headers.html

The big downside of this is that it decompresses and recompresses the
image data.  That should be replaced by a technique from jhead,
exiftool, ObscuraCam, etc. that only strips the metadata.
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
bde0558d82 update: reject APKs with invalid file sig, probably Janus exploits
This just checks the first four bytes of the APK file, aka the "file
signature", to make sure it is the ZIP signature and not the DEX signature.
This was checked against the test APK, and I ran it against some known
malware and all of f-droid.org to make sure it works.

All valid ZIP files (therefore APK files) should start with the ZIP
Local File Header of four bytes.

https://www.guardsquare.com/en/blog/new-android-vulnerability-allows-attackers-modify-apps-without-affecting-their-signatures
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
5ce950e748 update: print warnings for all KnownVulns found
Some baby steps towards making the KnownVuln stuff more visible.
2017-12-14 16:57:22 +01:00
Andrea Scarpino
19a83b6219 Add Liberapay support 2017-12-12 11:53:31 +01:00
Hans-Christoph Steiner
5d54446efc update: do not replace Name/Summary from template unless blank
`fdroid nightly` needs this change so it can set the Summary using the
template.
2017-12-07 22:39:32 +01:00
tobiasKaminsky
d5ecb34df7
add flavour to metadata 2017-12-01 15:25:00 +01:00
Hans-Christoph Steiner
d46d9574b4 update: use KnownApks dates to check system clock on offline machines
KnownApks provides a reliable source of a relatively recent date.
2017-11-29 21:06:02 +01:00
Hans-Christoph Steiner
9ade09349b update: improve help strings 2017-11-08 14:05:17 +01:00
Hans-Christoph Steiner
7be46d53c7 sort Anti-Features before writing them to the wiki
This avoids silly updates to the wiki where the sort order of the Anti-
Features is the only change:

−{{AntiFeature|KnownVuln}}
 {{AntiFeature|DisabledAlgorithm}}
+{{AntiFeature|KnownVuln}}

https://f-droid.org/wiki/index.php?title=info.guardianproject.browser&curid=517&diff=189655&oldid=188037
2017-10-27 09:15:22 +02:00
Ciaran Gultnieks
505d2c1325 update bombs out when no icons (closes #402) 2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
ac69d5b17d update: make all strings translatable 2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
ffc91e301a make _ always be the gettext function, nothing else
This avoids hard bugs where the _() function gets overidden by a str or
something else.
2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
927104a4e3 update: make strings translatable 2017-10-19 23:16:58 +02:00
Hans-Christoph Steiner
460c4185bb update: support Fastlane's official location for screenshots
fastlane/android/metadata/locale/images/*Screenshots

https://github.com/fastlane/fastlane/blob/2.61.0/screengrab/lib/screengrab/runner.rb#L324

closes #357
2017-10-13 14:58:15 +02:00
Hans-Christoph Steiner
a4169484fd update: fix detection of Triple-T screenshots
* https://github.com/westnordost/StreetComplete/issues/489
* https://forum.f-droid.org/t/screenshots-dont-show-up-in-client-using-gradle-play-publisher/881/2

#357
2017-10-13 14:58:15 +02:00
Michael Pöhn
290b9050e0 convert apk timestamp from filesystem to utc before storing 2017-09-26 14:13:08 +02:00
Michael Pöhn
6fc968f7cd choose best apk version vor index v0 2017-09-26 14:11:09 +02:00
Michael Pöhn
6930edf889 sort index-v1; publish now creates and stores a list of signature fingerprints 2017-09-26 14:11:09 +02:00