Commit graph

3341 commits

Author SHA1 Message Date
Hans-Christoph Steiner
dbf80ad771
metadata: remove git repo handling from metadata parsing
This is a vestige of implementing builds from a .fdroid.yml file directly
in the app's source repo.  It was never fully complete and seems to not be
used in any apps in fdroiddata.  This makes `fdroid build --all` runs much
faster since it does not need to do any git handling for apps that do not
have any new builds to run.

4e8e29794
948689281
a4e431080
e37be9b06
e775d330c
2021-03-17 13:44:24 +01:00
Hans-Christoph Steiner
78842e9cc2 scanner: should not exit with error when apkanalyzer fails
apkanalyzer produces useful output when it can run, but it does not
support all recent JDK versions, and also some DEX versions, so this
cannot count on it to always produce useful output or even to run
without exiting with an error.

211dd65ff0 was based on false
assumptions that apkanalyzer can always produce output.

fdroiddata!8585
fdroiddata!8584
2021-03-15 10:54:29 +01:00
Hans-Christoph Steiner
5695993521 build: include useful successful build IDs in status JSON, closes #825 2021-03-11 18:52:31 +01:00
Felix C. Stegerman
259ca51001 use vagrantfile instead of 'builder/Vagrantfile' in get_clean_builder() 2021-03-10 09:12:52 +00:00
Hans-Christoph Steiner
8bb2c7d4b1 metadata: stop setting up source repo when running lint/rewritemeta
broken in 54e8867a7e
2021-03-09 17:55:07 +01:00
Hans-Christoph Steiner
211dd65ff0 scanner: show error if scan_binary fails to run apkanalyzer 2021-03-09 15:55:45 +01:00
Hans-Christoph Steiner
649347dcd4 common: properly parse version from NDK's source.properties 2021-03-09 15:55:45 +01:00
Hans-Christoph Steiner
6a875a1821 update: stop extracting and storing XML icons, they're useless
APKs that only have XML icons in them should include an icon.png in the
metadata or fastlane/triple-t

closes #344
2021-03-09 15:40:30 +01:00
Hans-Christoph Steiner
850e244526 index: rename vars to match the dict key 'signer'
'sig' matches the old deprecated entry for the signer fingerprint that
uses the custom fingerprint algorithm.
2021-03-09 15:40:30 +01:00
Hans-Christoph Steiner
544a45c16a index: raise error rather than crash on bad repo file
If a non-APK is added with the appid/packageName that matches some APKs, it
should through an error.

Traceback (most recent call last):
  File "/home/hans/code/fdroid/server/fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 211, in main
    mod.main()
  File "/home/hans/code/fdroid/server/fdroidserver/update.py", line 2343, in main
    index.make(apps, sortedids, apks, repodirs[0], False)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 142, in make
    fdroid_signing_key_fingerprints)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 166, in make_v1
    v1_sort_packages(packages, fdroid_signing_key_fingerprints)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 292, in v1_sort_packages
    packages.sort(key=v1_sort_keys)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 288, in v1_sort_keys
    .format(apkfilename=package['apkName']))
fdroidserver.exception.FDroidException: at.roteskreuz.stopcorona_8.jobf does not have a valid signature!
2021-03-09 15:40:30 +01:00
Hans-Christoph Steiner
b5c941938a update: reuse local_copy_dir to provide auto-copying with --nosign
When using `fdroid update --nosign` in combo with `fdroid signindex`, the
unsigned index files have to be copied to a thumb drive or something in
order to be brought to the offline signing server.  This reuses the config
option local_copy_dir as the destination for those unsigned index files.
2021-03-09 15:40:30 +01:00
Hans-Christoph Steiner
a2f0356d84 build: include all wiki data in the JSON status output 2021-03-09 15:40:30 +01:00
relan
56b84251f6 build: set overall timeout to 72 hours
Some builds are pending for weeks now because of the global time limit.
Give them a chance to be built.
2021-03-08 11:41:30 +03:00
Jochen Sprickerhof
933eea149a Add needsUpdate and noUpdateCheck to update.json
As done in update_wiki.
2021-02-12 11:04:24 +00:00
Jochen Sprickerhof
33bff6b298 Publish unsigned builds
This should make debugging reproducible builds easier.
2021-02-12 10:20:30 +00:00
Hans-Christoph Steiner
0d836751e5
code format cleanup 2021-02-12 10:58:07 +01:00
Hans-Christoph Steiner
007c62065d
pylint flags "_" as bad variable name, its the gettext function
Throughout the fdroidserver code base, "_()" is used for localizing
strings.  So it should not be used as an ignored variable name.  Those
should be called "_ignored".
2021-02-12 10:56:31 +01:00
Hans-Christoph Steiner
525dcb8f98 update: handle large, corrupt, or inaccessible fastlane/triple-t files
```
Traceback (most recent call last):
  File "../fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 227, in main
    raise e
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 208, in main
    mod.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2340, in main
    repoapps = prepare_apps(apps, apks, repodirs[0])
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2176, in prepare_apps
    copy_triple_t_store_metadata(apps_with_packages)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1076, in copy_triple_t_store_metadata
    _set_author_entry(app, 'authorWebSite', os.path.join(root, f))
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 784, in _set_author_entry
    with open(f, errors='replace') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'build/player.efis.cfd/pfd/src/main/play/contact-website.txt'
```
2021-02-11 17:42:13 +01:00
fossdd
bcdefccb31 Revert removing Licenses 2021-02-10 19:10:14 +00:00
fossdd
8371a2e07a Update SPDX License List 2021-02-08 15:14:28 +00:00
Benedikt Brückmann
022d73b3b6 Add HTML/CSS tidy test
and tidy up the content
2021-02-08 12:25:02 +01:00
Benedikt Brückmann
0936051c7b Fix pep8 2021-02-08 12:25:02 +01:00
Benedikt Brückmann
96854be84a Only overwrite index.html/css files which were created by fdroid update
Further changes:
* use real value for icon instead of hard coded value
* Move qrcode.make() to top
* fix identation of css file generation
2021-02-08 12:25:02 +01:00
Benedikt Brückmann
8d51566a70 * Change image name
* add fingerprint
2021-02-08 12:25:02 +01:00
Benedikt Brückmann
4d464051ff Fix pycodestlye errors 2021-02-08 12:25:02 +01:00
Benedikt Brückmann
997f5d267b Properly add css 2021-02-08 12:25:02 +01:00
Benedikt Brückmann
44ee531426 Add QR code and css 2021-02-08 12:25:02 +01:00
Benedikt Brückmann
788333991d move code to own method 2021-02-08 12:25:02 +01:00
Benedikt Brückmann
12682f063d First draft of index.html generation 2021-02-08 12:25:02 +01:00
Hans-Christoph Steiner
fc7f9204b6 checkupdates: set User-Agent to make gitlab.com happy
closes #863
2021-02-02 21:37:53 +01:00
Hans-Christoph Steiner
57017fa373 Merge branch 'transparency_once' into 'master'
Run push_binary_transparency only once

See merge request fdroid/fdroidserver!865
2021-02-02 09:53:52 +00:00
Jochen Sprickerhof
2103569958 Run push_binary_transparency only once 2021-02-02 09:07:28 +01:00
Hans-Christoph Steiner
31c29f2fb3 support official ANDROID_SDK_ROOT env var when running CLI tools
ANDROID_HOME is deprecated:
https://developer.android.com/studio/command-line/variables#android_sdk_root

!816 #547
2021-01-29 16:34:36 +01:00
Hans-Christoph Steiner
63be6f6347 standardize naming in strings and comments on all caps APK 2021-01-29 15:40:59 +01:00
Hans-Christoph Steiner
97d5933a05 change config.py to config.yml everywhere it is needed 2021-01-29 13:24:17 +01:00
Hans-Christoph Steiner
32a0c61010 init: enable apksigner by default if it is found 2021-01-28 22:25:33 +01:00
Hans-Christoph Steiner
197ca7e36f update: warn with --nosign if keystore/repo_pubkey are not present
This should also make it easier to use index.make() as an API function
since this changes whether the instance var exists before checking the
value.
2021-01-28 22:25:33 +01:00
Hans-Christoph Steiner
f527b17715 sync repo examples with defaults and comment out example repo settings
!359
2021-01-24 09:26:11 +01:00
Hans-Christoph Steiner
5ae2703477 init: no error if examples/ dir cannot be found; overhaul repo_icon
There are so many possible installation paths for Python modules, it has
been very hard to even find and test them all.  This adds a fallback option
if the examples dir cannot be found.  A repo can work without an icon or
the example config.py.

This removes the fake assumption that the icon can be a full path in the
config.py.  While the path was being properly passed through to the index
file, the file was never copied properly into place nor rsynced to the web
server.
2021-01-18 14:07:20 +01:00
Hans-Christoph Steiner
94cb19f571 set METADATA_VERSION to 20000 to leave room for 1.1.x changes 2021-01-15 14:33:21 +01:00
Hans-Christoph Steiner
fff59e5197 only use AutoName: in checkupdates
AutoName: is only needed for the commit messages generated by checkupdates,
and it makes the logic for localized names confusing.

closes #654
refs #304
2021-01-14 20:36:09 +01:00
Hans-Christoph Steiner
0f6b638986 common.get_app_display_name() for finding app names 2021-01-14 20:34:15 +01:00
Hans-Christoph Steiner
c7fcfe3bfa update: fix setting current_version_file for make_current_version_link
Before, it would never set current_version_file to the current APK since it
first set current_version_code, then tested against CurrentVersionCode. So
if there is only a single APK and its the CurrentVersionCode, then
current_version_file would not get set.

refs #772
2021-01-14 17:09:44 +01:00
Hans-Christoph Steiner
0a1793ab7f update: write out cache as soon as possible
When working or testing with large APK collections, oftentimes it can take
hours to scan all the APKs.  If there is a failure before the apkcache is
written out, then all that work is lost since no cache is written out. This
moves the final cache writing before writing the index and knownapks to
make it more likely that the apkcache is successfully written.
2021-01-14 17:09:44 +01:00
Hans-Christoph Steiner
d36f71e048 make config optional when using common.set_FDroidPopen_env()
common.set_FDroidPopen_env() is used to call git and other tools that
are not part of the Android SDK nor require Java, so the items that
are being set from the config are optional.  This lets plugins do
quite a bit without ever setting up a config.
2021-01-11 20:58:50 +01:00
mimi89999
01699db376
Add androidx.work:work-gcm to non-free dependencies 2021-01-11 10:44:48 +01:00
Jochen Sprickerhof
a9c380b2b8 Ignore src/test only in path_in_build_dir
Found while debugging Debian autopkgtests.
2021-01-04 21:20:31 +01:00
Izzy
434a23f822
add Google Play Core to non-free/proprietary dependencies 2021-01-04 17:39:57 +01:00
Hans-Christoph Steiner
0b879f18e5 use dict syntax for app instances to allow API usage with plain dicts
Since metadata files are now YAML, programs should be able to just read in
the .yml files, then use them with fdroidserver methods without having to
know about the App class.  The App class just provides syntactic sugar by
allowing dict keys to be accessed as attributes.
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner
e93acf7964 metadata: work with dicts generated by standard YAML parsers
Now that the mismatch between 'builds' and 'Builds' has been fixed, it is
now possible to read metadata/*.yml files with a standard YAML parser like
PyYAML, then output them using `metadata.write_metadata()`.  Other API
functions in fdroidserver should also work in this case.

I did this as I was working on fdroiddata!7860
2020-12-15 08:55:05 +01:00