Commit graph

3677 commits

Author SHA1 Message Date
Michael Pöhn
c954a14139 update.py: fix archiving .log.gz files 2019-10-23 11:04:07 +02:00
Michael Pöhn
7fa3c34e5b update tests for fixed log deployment+changelog 2019-10-15 15:19:18 +02:00
Michael Pöhn
d665106813 fix disappearing build logs when deploying 2019-10-15 14:28:45 +02:00
Hans-Christoph Steiner
6d11da5e13 Merge branch 'triple-t-2' into 'master'
parse Triple-T v2.x layout

See merge request fdroid/fdroidserver!683
2019-10-08 11:15:12 +00:00
Hans-Christoph Steiner
a72904ee23
lint: allow Beerware and XSkat licenses
These licenses were already in our repo.  The are not FSF- or OSI-approved,
but they are included in Debian.  So for now at least, we should maintain
them in our list until we have a clearer picture of how we should make the
list of licenses we support.

* fdroidserver!682
* https://github.com/spdx/license-list-XML/issues/876
* Beerware in Debian:
  https://metadata.ftp-master.debian.org/changelogs//main/libm/libmd/libmd_1.0.1-3_copyright
* XSkat in Debian:
  https://metadata.ftp-master.debian.org/changelogs//main/x/xskat/xskat_4.0-7_copyright
2019-10-07 12:05:37 +02:00
Hans-Christoph Steiner
5904aef5a7
update: parse Triple-T v2.x layout
Thanks to the very nice example from Piwigo, included here under the GPLv3+
https://github.com/Piwigo/Piwigo-Android

https://gitlab.com/fdroid/fdroiddata/merge_requests/5579#note_225834868
2019-10-04 14:11:40 +02:00
Hans-Christoph Steiner
a7dd7a812f
update: strip leading/tailing whitespace on author contact fields
Whitespace is not valid in email addresses, URLs, or phone numbers, and the
triple-t/fastlane file format generally has a newline at the end.
2019-10-04 11:33:39 +02:00
Hans-Christoph Steiner
1ef4f74aff lint: include MIT-CMU, it is so close to FSF/OSI-free MIT
fdroidserver!682

https://github.com/spdx/license-list-data/issues/53

* FSF lists two closely related variants as  X11 or Expat, search for "MIT license" in https://www.gnu.org/licenses/license-list.html
* X11 is considered free: https://directory.fsf.org/wiki/License:X11
* Expat is considered free: https://directory.fsf.org/wiki/License:Expat
* It is included in Debian, so it is DFSG-free: https://metadata.ftp-master.debian.org/changelogs//main/f/flite/flite_2.1-release-3_copyright
* Fedora considers it free https://fedoraproject.org/wiki/Licensing:MIT#CMU_Style
2019-10-04 11:30:24 +02:00
Hans-Christoph Steiner
3801db064a lint: improve HTTPS check
It was missing some domains, so I added another rule.  @IzzySoft pointed
out it was redudnant, so this removes the redudant rule and fixes the
original.

https://gitlab.com/fdroid/fdroidserver/merge_requests/681/diffs#note_225263464
2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner
af4e231f7d lint: enforce HTTPS on GitHub and GitLab pages
The always provide HTTPS, so let's enforce it.
2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner
17320c23f4 lint: don't trip up on projects with 'master' in the name
https://gitlab.com/fdroid/fdroiddata/merge_requests/5557#note_223283359
2019-10-04 11:12:48 +02:00
Michael Pöhn
d5ab303d83 lint: license override config option + require FSF/OSI approved licenses by default 2019-10-02 22:14:09 +02:00
Michael Pöhn
3c9535d64b lint: fsf approved licenses only 2019-10-02 12:15:02 +02:00
Michael Pöhn
0963da9fd3 Merge branch 'bugly' into 'master'
scanner: blacklist Bugly

See merge request fdroid/fdroidserver!680
2019-09-28 07:59:52 +00:00
relan
bd96a17ef2 scanner: blacklist Bugly
The artifact is licensed under "The Bugly Software License, Version 1.0"
[1], but link to the full text [2] is broken. LICENSE file in the source
code repository is empty.

I guess this library is non-free.

[1] https://mvnrepository.com/artifact/com.tencent.bugly/crashreport/2.8.6.0
[2] http://bugly.qq.com/licenses/LICENSE-1.0.txt
[3] https://github.com/BuglyDevTeam/Bugly-Android
2019-09-28 07:58:19 +03:00
Hans-Christoph Steiner
c7048f2c39 update: log errors on bad graphics, and then ignore the file
Python PIL is not so tolerant, so bad EXIF causes crashes:

  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main
    insert_localized_app_metadata(apps)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata
    _strip_and_copy_image(os.path.join(root, f), destdir)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image
    in_image = Image.open(fp)
  File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open
    % (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>
2019-09-26 22:06:21 +02:00
relan
3a7ad650e3 scanner: fix local Debian Maven repo handling
The resulting regex was 'https?://file:///usr/share/maven-repo' causing
scanner error.
2019-08-29 21:32:54 +03:00
Hans-Christoph Steiner
70e7e720b9
update: use graphics filename with hash in index to support caching
Using a filename based on the hash of the contents means that the caching
algorithms for fdroidclient and browsers can safely cache the file forever
using the filename, since this guarantees that the contents will never
change for a given filename.

This does not cover screenshots, only icon.png, featureGraphic.png,
tvBanner.png, and promoGraphic.png.

fdroidserver#689
fdroid-website!453
2019-08-29 08:23:51 +02:00
Hans-Christoph Steiner
508af00e84
update: only copy graphics and screenshots if mtime/size has changed
Instead of copying every time, trust the filesystem to tell us when the
file has changed.
2019-08-29 08:23:47 +02:00
Hans-Christoph Steiner
7090ce63dd Merge branch 'remove-provides-in-yml' into 'master'
Remove `Provides:` in yaml metadata

Closes #676

See merge request fdroid/fdroidserver!654
2019-08-28 13:39:59 +00:00
Michael Pöhn
8e5232076f do not delete yml metadata when raumel not installed 2019-08-27 15:38:40 +02:00
Michael Pöhn
902f1307dc rewritemeta: refactor supported list to module variable SUPPORTED_FORMATS 2019-08-27 15:38:40 +02:00
Michael Pöhn
08acb55897 keep yaml metadata when rewrite failed 2019-08-27 15:38:40 +02:00
Michael Pöhn
39c589fb85 Merge branch 'update-create-template' into 'master'
make metadata template behave well for empty values

Closes #681

See merge request fdroid/fdroidserver!657
2019-07-24 11:02:38 +00:00
Michael Pöhn
be3d498391 checkupdates: check_http fix ignore check 2019-07-23 22:01:18 +02:00
Michael Pöhn
1c4d377793 use logging.warning in checkupdates 2019-07-23 22:01:18 +02:00
Gerhard Olsson
19aa736011 UpdateCheckName is Ignore: Do not parse manifests 2019-07-23 01:02:31 +02:00
Gerhard Olsson
7c4bed0558 UpdateCheckIgnore: Exception if version not matched
Ignoreversions were checked also if version did not match which raised the exception
The 'version check' runs in some unexpected situations like just
to find directories where to find the AutoName.
2019-07-23 00:21:28 +02:00
Michael Pöhn
e2fed09af1 make metadata template behaves well for empty values 2019-07-22 01:34:55 +02:00
Michael Pöhn
dcf3837bcb parse yaml: ignore (and warn) deprecated field: Provides 2019-07-15 15:45:02 +02:00
Michael Pöhn
191363ad55 exclude Provides metadata from yml 2019-07-11 03:35:23 +02:00
Hans-Christoph Steiner
26af94974a Merge branch 'rsync-buildslogs-to-webroot-repo' into 'master'
build: rsync buildlogs to <webroot>/repo

See merge request fdroid/fdroidserver!651
2019-07-10 08:55:04 +00:00
Michael Pöhn
6bd39d7532 Merge branch 'checkupdates-regex' into 'master'
checkupdates: match whole words

See merge request fdroid/fdroidserver!649
2019-07-06 15:31:05 +00:00
Michael Pöhn
2c87b5e6f9 deploy build logs: no timestamps 2019-07-06 16:48:56 +02:00
Michael Pöhn
f30983368c build: rsync buildlogs to <webroot>/repo 2019-07-06 16:48:56 +02:00
Hans-Christoph Steiner
7381a94b47
gitlab-ci: disable bandit fail on standard debug keystore password 2019-06-30 22:48:52 +02:00
relan
781b55f777 checkupdates: match whole words
Match only whole words when looking for versionCode, versionName, etc. in
manifests. A real build.gradle example:

    flutterVersionCode = '1'
    flutterVersionName = '1.0'
    ...
    defaultConfig {
        versionCode 53
        versionName "2.0.3"
    }

Before this change checkupdates was erroneously getting version code and
version name from the first two lines and failing to find a new release.
2019-06-13 19:51:44 +03:00
Michael Pöhn
66105de833 improve litecoin validation + tests 2019-05-28 11:04:43 +02:00
Michael Pöhn
06cec2041d improve bitcoin validation regex + testcases 2019-05-07 22:43:05 +02:00
Hans-Christoph Steiner
5959a395c4 nightly: archive older versions; remove archive if git mirror is full 2019-05-01 21:16:15 +02:00
Hans-Christoph Steiner
869c68dbdb deploy: if git mirror > 1GB after deleting history, delete the archive
git mirrors are meant to be an easy way to host a repo that is zero
maintenance.  They are not meant to be the canonical repo with full,
preserved archive.  This option provides the zero maintenance mode.
2019-05-01 21:16:15 +02:00
Hans-Christoph Steiner
f0460dea6e scanner: allow local Debian Maven repo file:///usr/share/maven-repo
It is now possible to build an app using only things in Debian.  Since the
buildserver will always control the contents of file:///usr/share/maven-repo,
it is the most safe repo there is.
2019-05-01 21:16:15 +02:00
Hans-Christoph Steiner
dd2f9d60f8
publish: fix stupid error in repro-signing and add integration test
stoopid mistake in ea84014f9b reported
by @CiaranG
2019-04-11 14:06:51 +02:00
Licaon_Kter
0ea03ddfaa metadata.py - typo 2019-04-08 16:30:32 +00:00
Michael Pöhn
67731470cc Revert "Merge branch 'write-yaml-overhaul' into 'master'"
This reverts merge request !630
2019-04-01 10:24:00 +00:00
Michael Pöhn
d45431aa9e Merge branch 'write-yaml-overhaul' into 'master'
use pyyaml for writing metadata instead of ruamel

Closes #611

See merge request fdroid/fdroidserver!630
2019-03-30 16:53:11 +00:00
Nico Alt
3798a884a6 Flush file before passing it to next function
When downloading a repo index, the downloaded index got written to a
file with `.write()` in a `with` clause. Before the file got actually
written to the disk, it got already passed into the next function,
resulting in a `VerificationException`:
```
JAR signature failed to verify: /tmp/tmppq2r51r0
jarsigner: java.util.zip.ZipException: zip file is empty
```

This behavior got introduced in
869cc114a3.

I've found this bug with help of Repomaker's tests: https://gitlab.com/fdroid/repomaker/merge_requests/215#note_148994053
2019-03-22 17:15:58 +01:00
Hans-Christoph Steiner
06ef99be6a deploy: show HTTP errors when virustotal error fails.
Otherwise, this was posting mystery JSON parsing errors because it was
trying to parse the HTTP error page as JSON.  This also moves the
virustotal API key to post data so it is not printed out in the stacktrace.
2019-03-22 10:54:28 +01:00
Michael Pöhn
2683b37044 yml metadata write: do not use local functions 2019-03-19 01:01:18 +01:00
Michael Pöhn
63afc0acb5 use pyyaml for writing metadata instead of ruamel 2019-03-18 22:45:35 +01:00