Commit graph

2970 commits

Author SHA1 Message Date
Jochen Sprickerhof
9d589071f0
Fix ruamel round_trip_dump will be removed
https://sourceforge.net/p/ruamel-yaml/code/ci/4b91cd7e820baf97d8d28b1dcf77979530d43c3c/#diff-1

!932
2021-06-01 14:21:27 +02:00
Hans-Christoph Steiner
92438bbf78 metadata: stop setting up source repo when running lint/rewritemeta
broken in 54e8867a7e
2021-03-09 17:53:03 +01:00
Hans-Christoph Steiner
ff1638c7dc scanner: show error if scan_binary fails to run apkanalyzer 2021-03-09 15:58:04 +01:00
Hans-Christoph Steiner
8b89793d3c common: properly parse version from NDK's source.properties 2021-03-09 15:58:00 +01:00
Hans-Christoph Steiner
2b5093969a update: stop extracting and storing XML icons, they're useless
APKs that only have XML icons in them should include an icon.png in the
metadata or fastlane/triple-t

closes #344
2021-03-09 15:38:19 +01:00
Hans-Christoph Steiner
240139baf9 index: raise error rather than crash on bad repo file
If a non-APK is added with the appid/packageName that matches some APKs, it
should through an error.

Traceback (most recent call last):
  File "/home/hans/code/fdroid/server/fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 211, in main
    mod.main()
  File "/home/hans/code/fdroid/server/fdroidserver/update.py", line 2343, in main
    index.make(apps, sortedids, apks, repodirs[0], False)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 142, in make
    fdroid_signing_key_fingerprints)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 166, in make_v1
    v1_sort_packages(packages, fdroid_signing_key_fingerprints)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 292, in v1_sort_packages
    packages.sort(key=v1_sort_keys)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 288, in v1_sort_keys
    .format(apkfilename=package['apkName']))
fdroidserver.exception.FDroidException: at.roteskreuz.stopcorona_8.jobf does not have a valid signature!
2021-03-09 15:38:19 +01:00
Hans-Christoph Steiner
cfbee12ad2 update: handle large, corrupt, or inaccessible fastlane/triple-t files
```
Traceback (most recent call last):
  File "../fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 227, in main
    raise e
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 208, in main
    mod.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2340, in main
    repoapps = prepare_apps(apps, apks, repodirs[0])
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2176, in prepare_apps
    copy_triple_t_store_metadata(apps_with_packages)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1076, in copy_triple_t_store_metadata
    _set_author_entry(app, 'authorWebSite', os.path.join(root, f))
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 784, in _set_author_entry
    with open(f, errors='replace') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'build/player.efis.cfd/pfd/src/main/play/contact-website.txt'
```
2021-03-09 15:38:19 +01:00
fossdd
152250646e Revert removing Licenses 2021-03-09 15:38:19 +01:00
fossdd
eca7b23fc9 Update SPDX License List 2021-03-09 15:38:19 +01:00
Hans-Christoph Steiner
5ed531ce52 checkupdates: set User-Agent to make gitlab.com happy
closes #863
2021-03-09 14:57:04 +01:00
Jochen Sprickerhof
ba87076d64 Run push_binary_transparency only once 2021-03-09 14:52:45 +01:00
Hans-Christoph Steiner
31c29f2fb3 support official ANDROID_SDK_ROOT env var when running CLI tools
ANDROID_HOME is deprecated:
https://developer.android.com/studio/command-line/variables#android_sdk_root

!816 #547
2021-01-29 16:34:36 +01:00
Hans-Christoph Steiner
63be6f6347 standardize naming in strings and comments on all caps APK 2021-01-29 15:40:59 +01:00
Hans-Christoph Steiner
97d5933a05 change config.py to config.yml everywhere it is needed 2021-01-29 13:24:17 +01:00
Hans-Christoph Steiner
32a0c61010 init: enable apksigner by default if it is found 2021-01-28 22:25:33 +01:00
Hans-Christoph Steiner
197ca7e36f update: warn with --nosign if keystore/repo_pubkey are not present
This should also make it easier to use index.make() as an API function
since this changes whether the instance var exists before checking the
value.
2021-01-28 22:25:33 +01:00
Hans-Christoph Steiner
f527b17715 sync repo examples with defaults and comment out example repo settings
!359
2021-01-24 09:26:11 +01:00
Hans-Christoph Steiner
5ae2703477 init: no error if examples/ dir cannot be found; overhaul repo_icon
There are so many possible installation paths for Python modules, it has
been very hard to even find and test them all.  This adds a fallback option
if the examples dir cannot be found.  A repo can work without an icon or
the example config.py.

This removes the fake assumption that the icon can be a full path in the
config.py.  While the path was being properly passed through to the index
file, the file was never copied properly into place nor rsynced to the web
server.
2021-01-18 14:07:20 +01:00
Hans-Christoph Steiner
94cb19f571 set METADATA_VERSION to 20000 to leave room for 1.1.x changes 2021-01-15 14:33:21 +01:00
Hans-Christoph Steiner
fff59e5197 only use AutoName: in checkupdates
AutoName: is only needed for the commit messages generated by checkupdates,
and it makes the logic for localized names confusing.

closes #654
refs #304
2021-01-14 20:36:09 +01:00
Hans-Christoph Steiner
0f6b638986 common.get_app_display_name() for finding app names 2021-01-14 20:34:15 +01:00
Hans-Christoph Steiner
c7fcfe3bfa update: fix setting current_version_file for make_current_version_link
Before, it would never set current_version_file to the current APK since it
first set current_version_code, then tested against CurrentVersionCode. So
if there is only a single APK and its the CurrentVersionCode, then
current_version_file would not get set.

refs #772
2021-01-14 17:09:44 +01:00
Hans-Christoph Steiner
0a1793ab7f update: write out cache as soon as possible
When working or testing with large APK collections, oftentimes it can take
hours to scan all the APKs.  If there is a failure before the apkcache is
written out, then all that work is lost since no cache is written out. This
moves the final cache writing before writing the index and knownapks to
make it more likely that the apkcache is successfully written.
2021-01-14 17:09:44 +01:00
Hans-Christoph Steiner
d36f71e048 make config optional when using common.set_FDroidPopen_env()
common.set_FDroidPopen_env() is used to call git and other tools that
are not part of the Android SDK nor require Java, so the items that
are being set from the config are optional.  This lets plugins do
quite a bit without ever setting up a config.
2021-01-11 20:58:50 +01:00
mimi89999
01699db376
Add androidx.work:work-gcm to non-free dependencies 2021-01-11 10:44:48 +01:00
Jochen Sprickerhof
a9c380b2b8 Ignore src/test only in path_in_build_dir
Found while debugging Debian autopkgtests.
2021-01-04 21:20:31 +01:00
Izzy
434a23f822
add Google Play Core to non-free/proprietary dependencies 2021-01-04 17:39:57 +01:00
Hans-Christoph Steiner
0b879f18e5 use dict syntax for app instances to allow API usage with plain dicts
Since metadata files are now YAML, programs should be able to just read in
the .yml files, then use them with fdroidserver methods without having to
know about the App class.  The App class just provides syntactic sugar by
allowing dict keys to be accessed as attributes.
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner
e93acf7964 metadata: work with dicts generated by standard YAML parsers
Now that the mismatch between 'builds' and 'Builds' has been fixed, it is
now possible to read metadata/*.yml files with a standard YAML parser like
PyYAML, then output them using `metadata.write_metadata()`.  Other API
functions in fdroidserver should also work in this case.

I did this as I was working on fdroiddata!7860
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner
bf25b4ca03 eliminate app.builds everywhere, it should be app['Builds']
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds".  This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().

The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.

```
sed -i \
 -e 's/app\.builds/app.get('Builds', \[\])/g' \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
 -e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
 -e "s/app\['builds'\]/app.get('Builds', [])/g" \
 */*.*
```
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner
cfec25d33a update: tame androguard debug logs when --verbose is set 2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner
40cd51ed59 build: include commit ID in build log 2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner
fb628c2cb2 include modified and untracked files in status JSON
Ideally, an fdroid repo should be running from a clean git repo, so that
all changes are tracked in git.  This change is useful in seeing which
changes and/or files are not in git.  If there are modified files, the
dirty flag will be set, so this info can help debugging that.
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner
384922118f index: sanitize fingerprint arg, extract_pubkey() returns with spaces
The key fingerprint should be only hex digits, everything else can be
discarded.  That makes it easy to use this function various fingerprint
formats, including the common, human-readable forms spaces between pairs or
quartets.
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner
d9171f11e0 update: improve logging when exiting due to bad APK file
closes #851
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner
ba854cbc0f index: fix GitLab Raw URLs with gitlab.com and recent versions
GitLab seems to be moving to always having "-" as the first path segment in
all the project URLs.  So the URL without a "-" is now a redirect.
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner
54e8867a7e
fix read_metadata in fdroid build
Broken in fdroid/fdroidserver!828
2020-12-09 22:29:51 +01:00
Hans-Christoph Steiner
ac5ed93428 Merge branch 'purge-description-formatting' into 'master'
Purge description formatting

Closes #845 and #678

See merge request fdroid/fdroidserver!828
2020-12-09 18:28:08 +00:00
Jochen Sprickerhof
66d0e6b1f9 find_sdk_tools_cmd always raise exception if not found
test_sdk_exists() succeeds on an empty directory so it does not give any
new information. Contrary, test_sdk_exists() succeeds on an empty
directory, so find_sdk_tools_cmd() returned None even though the tools
where not found, before.
2020-12-09 17:37:02 +01:00
Hans-Christoph Steiner
129438fd30 make read_metadata only parse files actually used by lint/rewritemeta
Now that the description formatting is removed, there is no need to load
all of the app metadata before operating on a single one.  This change
makes lint and rewritemeta only load the metadata for the apps it is actually
operating on.  Before, it would always load all metadata files.  #845

closes #678
2020-12-08 09:37:49 +01:00
Hans-Christoph Steiner
c98b15e72a purge code that modifies the repo descriptions
YAML now provides nice ways of handling blocks of text, including removing
indents and newlines.  Remove the custom hack.
2020-12-08 09:37:49 +01:00
Hans-Christoph Steiner
828d6015ef purge code that modifies the app description, including linkifying
closes #845
2020-12-08 09:37:49 +01:00
Hans-Christoph Steiner
77db3ea709 mirror: if index is verified, save a copy in the local mirror 2020-12-03 21:48:37 +01:00
Hans-Christoph Steiner
2a265cbc0b Merge branch 'build_tools_tests' into 'master'
Build tools tests

See merge request fdroid/fdroidserver!834
2020-11-30 08:39:20 +00:00
Jochen Sprickerhof
25b1f3fe8c Ignore non version number entries in build_tools_path
LooseVersion('debian') stay a string and results in a type error.
2020-11-29 19:04:35 +01:00
Jochen Sprickerhof
4d78d79280 Revert "Ignore non version number entries in build_tools_path"
This reverts commit 5c36f84430.
2020-11-29 19:04:12 +01:00
Jochen Sprickerhof
ce10fcb759 Check if build_tools path exists before listdir 2020-11-29 10:06:39 +01:00
Hans-Christoph Steiner
1c7df94e76 purge unneeded 'build_tools' config option
Back when fdroidserver was built around aapt, that was needed to
guarantee that a compatible version of aapt was used. Now, aapt is
only optionally used for getting the APK ID, so this was just
complicating maintenance.
2020-11-29 10:06:39 +01:00
Hans-Christoph Steiner
1a736c2314 Merge branch 'versionCode_underscore' into 'master'
Support underscore (_) in versionCode

See merge request fdroid/fdroidserver!830
2020-11-27 10:30:44 +00:00
Jochen Sprickerhof
5c36f84430 Ignore non version number entries in build_tools_path 2020-11-26 23:31:20 +01:00