Commit graph

186 commits

Author SHA1 Message Date
Hans-Christoph Steiner
4aba6be8b8
signindex: added simple test case 2022-11-16 20:19:06 +01:00
Hans-Christoph Steiner
6a2e1f0af4
gitlab-ci: use python3-ruamel.yaml to avoid building from source 2022-11-16 20:18:42 +01:00
Hans-Christoph Steiner
f47840542d
gitlab-ci: port ubuntu_bionic_pip to Ubuntu/jammy
This job is here to test using fdroidserver with only depends from pip
and sdkmanager (e.g. not apt-get).  Production is now on bullseye #1038
2022-11-16 20:18:40 +01:00
Hans-Christoph Steiner
89483e42b2
gitlab-ci: run test suite in the buildserver environment
Run the whole test suite in an environment that is like the
buildserver guest VM.  This installs python3-babel because that is
only used by the test suite, and not needed in the buildserver.

Some extra packages are required for this test run that are not
provided by the buildserver since they are not needed there:

 * gnupg-agent for the full signing setup
 * python3-babel for compiling localization files
 * python3-clint for fancy progress bars for users

fdroid/ci-images-base is deprecated and archived.
2022-11-16 20:18:38 +01:00
Hans-Christoph Steiner
d2ef0fee71
nightly: support OpenSSL 3.0 with Paramiko
OpenSSL 3.0 changed the default output format from PKCS#1 to PKCS#8,
which paramiko does not support.

https://www.openssl.org/docs/man3.0/man1/openssl-rsa.html#traditional
https://github.com/paramiko/paramiko/issues/1015
2022-11-15 23:01:35 +01:00
Hans-Christoph Steiner
58dec57439 gitlab-ci: paramiko from Debian to avoid crypto libs from source
Before it was cryptography, now bcrypt requires a Rust compiler:
https://gitlab.com/fdroid/fdroidserver/-/jobs/2929483724
2022-09-06 22:45:46 +02:00
Hans-Christoph Steiner
55b3905951 gitlab-ci: fix servergitmirrors by using Debian packages
this is like:
* ca5ebcb30f
* 972bfe346f
* 23bd62c31c
* !1035
* !1034
* !1029
* !1028
2022-02-03 16:41:57 +01:00
Hans-Christoph Steiner
ca5ebcb30f
gitlab-ci: fix plugin_fetchsrclibs by using Debian packages
* https://gitlab.com/eighthave/fdroidserver/-/jobs/1961701458

Collecting pynacl>=1.0.1 (from paramiko->fdroidserver==2.1a0)
  Downloading https://files.pythonhosted.org/packages/a7/22/27582568be639dfe22ddb3902225f91f2f17ceff88ce80e4db396c8986da/PyNaCl-1.5.0.tar.gz (3.4MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xokvr6uk --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.8.0 wheel "cffi>=1.4.1; python_implementation != 'PyPy'":
  Collecting setuptools>=40.8.0
    Using cached https://files.pythonhosted.org/packages/eb/53/0dd4c7960579da8be13fa9b2c2591643d37f323e3d79f8bc8b1b6c8e6217/setuptools-60.5.0-py3-none-any.whl
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl
  Collecting cffi>=1.4.1
    Downloading https://files.pythonhosted.org/packages/00/9e/92de7e1217ccc3d5f352ba21e52398372525765b2e0c4530e6eb2ba9282a/cffi-1.15.0.tar.gz (484kB)
  Collecting pycparser (from cffi>=1.4.1)
    Downloading https://files.pythonhosted.org/packages/62/d5/5f610ebe421e85889f2e55e33b7f9a6795bd982198517d912eb1c76e1a53/pycparser-2.21-py2.py3-none-any.whl (118kB)
  Building wheels for collected packages: cffi
    Running setup.py bdist_wheel for cffi: started
    Running setup.py bdist_wheel for cffi: finished with status 'error'
    Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-d1knhl7l --python-tag cp37:
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory

        No working compiler found, or bogus compiler options passed to
        the compiler from Python's standard "distutils" module.  See
        the error messages above.  Likely, the problem is not related
        to CFFI but generic to the setup.py of any Python package that
        tries to compile C code.  (Hints: on OS/X 10.8, for errors about
        -mno-fused-madd see http://stackoverflow.com/questions/22313407/
        Otherwise, see https://wiki.python.org/moin/CompLangPython or
        the IRC channel #python on irc.libera.chat.)

        Trying to continue anyway.  If you are trying to install CFFI from
        a build done in a different context, you can ignore this warning.

    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/lock.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/recompiler.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/__init__.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/cparser.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/verifier.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/api.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/commontypes.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/error.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/model.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_embedding.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.7/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/c
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/builds/eighthave/fdroidserver/env/include -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
    Failed building wheel for cffi
    Running setup.py clean for cffi
  Failed to build cffi
  Installing collected packages: setuptools, wheel, pycparser, cffi
    Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ek80c81s/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xokvr6uk --compile --install-headers /builds/eighthave/fdroidserver/env/include/site/python3.7/cffi:
      unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
      unable to execute 'x86_64-linux-gnu-gcc': No such file or directory

          No working compiler found, or bogus compiler options passed to
          the compiler from Python's standard "distutils" module.  See
          the error messages above.  Likely, the problem is not related
          to CFFI but generic to the setup.py of any Python package that
          tries to compile C code.  (Hints: on OS/X 10.8, for errors about
          -mno-fused-madd see http://stackoverflow.com/questions/22313407/
          Otherwise, see https://wiki.python.org/moin/CompLangPython or
          the IRC channel #python on irc.libera.chat.)

          Trying to continue anyway.  If you are trying to install CFFI from
          a build done in a different context, you can ignore this warning.

      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.7
      creating build/lib.linux-x86_64-3.7/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/lock.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/recompiler.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/__init__.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/cparser.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/verifier.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/api.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/commontypes.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/error.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/model.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_embedding.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.7/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-3.7
      creating build/temp.linux-x86_64-3.7/c
      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/builds/eighthave/fdroidserver/env/include -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
      unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
      error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

      ----------------------------------------
  Command "/builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ek80c81s/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xokvr6uk --compile --install-headers /builds/eighthave/fdroidserver/env/include/site/python3.7/cffi" failed with error code 1 in /tmp/pip-install-djg9jc8p/cffi/

  ----------------------------------------
Command "/builds/eighthave/fdroidserver/env/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xokvr6uk --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.8.0 wheel "cffi>=1.4.1; python_implementation != 'PyPy'"" failed with error code 1 in None
2022-01-13 21:22:25 +01:00
Hans-Christoph Steiner
839f5b109a
gitlab-ci: publish buildserver image to Docker Registry 2022-01-13 21:21:26 +01:00
FestplattenSchnitzel
565d9e4994
Format setup.py with Black 2021-12-16 20:34:38 +01:00
FestplattenSchnitzel
23a4135ccd
Compile translation files on install
Solution based on https://stackoverflow.com/questions/40051076/compile-translation-files-when-calling-setup-py-install
2021-12-16 20:13:24 +01:00
Hans-Christoph Steiner
778e600269 gitlab-ci: enable pyjks in lint_mypy job now that it is in backports 2021-12-07 10:24:27 +01:00
Jochen Sprickerhof
6756f842e4 [CI] install python3-matplotlib from apt 2021-11-19 18:17:36 +01:00
Jochen Sprickerhof
46abffa0fa [CI] install python3-pil from apt in servergitmirrors
New version would need to be compiled, otherwise.
2021-10-23 15:02:57 +02:00
Jochen Sprickerhof
b8285ebd0e [CI] Install python3-pil from apt
New version would need to be compiled, otherwise.
2021-10-23 14:33:06 +02:00
Jochen Sprickerhof
972bfe346f Fix servergitmirrors ci 2021-10-14 14:40:31 +02:00
Jochen Sprickerhof
23bd62c31c [CI] fetchsrclibs use apt cffi 2021-10-14 13:55:51 +02:00
Jochen Sprickerhof
60532f0912 Ignore failing Windows CI 2021-10-14 13:40:19 +02:00
FestplattenSchnitzel
ef20e1b329 Add CI job "black" for code format checks
The CI job will fail, if a file needs to be reformatted.
Only files already well formatted will be checked at the moment.
2021-09-25 15:45:37 +00:00
Hans-Christoph Steiner
3b95d3de64
update: AllowedAPKSigningKeys metadata to enforce APK signers
This field lets you specify which signing certificates should be
trusted for APKs in a binary repo.
2021-08-05 16:43:48 +02:00
Hans-Christoph Steiner
7ed75abc2f
gitlab-ci: use CVC for fdroid build job for more reliably runs 2021-07-01 14:45:39 +02:00
Benedikt Brückmann
1e943a22df rewrite docstrings to match numpy style guide 2021-06-25 06:28:47 +02:00
Hans-Christoph Steiner
660520e3e1
gitlab-ci: switch ubuntu_lts_ppa job to keyserver.ubuntu.com
fixes:

Executing: /tmp/apt-key-gpghome.ZwePp5ymqm/gpg.1.sh --keyserver hkp://pool.sks-keyservers.net --recv-key 9AAC253193B65D4DF1D0A13EEC4632C79C5E0151
gpg: keyserver receive failed: No name

https://gitlab.com/fdroid/fdroidserver/-/jobs/1374517971
2021-06-24 21:34:29 +02:00
Hans-Christoph Steiner
fc989d94b3
gitlab-ci: add Windows job 2021-06-18 11:12:27 +02:00
linsui
466c8a44d7 add vscode settings 2021-06-18 06:50:50 +00:00
Hans-Christoph Steiner
ac86a2af89
gitlab-ci: use mypy from Debian/bullseye to prevent random breakage
mypy 0.901 was released and everything broke:
https://gitlab.com/linsui/fdroidserver/-/jobs/1330206567

My point is to reduce the number of false job fails like this one. We
have a lot of checkers running, I really think they need to prove they are
adding value before we invest any time maintaining them. mypy is still
"wait and see" in terms of the adding any value.

!951
2021-06-09 12:35:50 +02:00
Hans-Christoph Steiner
bcd3e89614
gitlab-ci: fix whitespace 2021-06-09 12:25:00 +02:00
Hans-Christoph Steiner
9d44fa7919
gitlab-ci: auto-generate merge request when NDK release found
Following the pattern of the gradle bot, this will check the transparency
log for any new NDK release.  If there are any, it will make a merge
request from @fdroid-bot.
2021-05-25 17:06:30 +02:00
Hans-Christoph Steiner
e4a7726033 gitlab-ci: fix archlinux docker image 2021-05-13 17:20:41 +02:00
Benedikt Brückmann
6791c3a550 Only publish pages on default branch 2021-05-10 14:31:51 +00:00
Benedikt Brückmann
245aeec972 Publish docu to GL Pages 2021-05-10 14:31:51 +00:00
Benedikt Brückmann
bdcd843d75 Publish docu to GL Pages 2021-05-10 14:31:51 +00:00
Benedikt Brückmann
a15581faa9 Draft: set up docs based on docstrings and publish HTML versions 2021-05-10 14:31:51 +00:00
Hans-Christoph Steiner
022d0796d8 gitlab-ci: use latest alpine/bandit to fix CVE flagged by safety 2021-05-10 16:05:59 +02:00
Hans-Christoph Steiner
634864f206
gitlab-ci: document jobs 2021-04-16 09:40:12 +02:00
Hans-Christoph Steiner
5346ea3c19 gitlab-ci: prevent dualing linters: pyflakes vs mypy
* pyflakes says: "typing.Dict' imported but unused"
* mypy says: "Name 'Dict' is not defined"
2021-04-14 23:18:05 +02:00
Hans-Christoph Steiner
9f60516fcb gitlab-ci: fix broken download link for fdroiddata tarball
fdroiddata!8730
fdroiddata!8732
rfp@f49178ea0aed808607f8d6d753285b101e53f0be
2021-04-06 10:41:59 +02:00
Hans-Christoph Steiner
cf1fa84c77
gitlab-ci: rm HTML index gen test from job, fails on stretch
```console
  Downloading https://files.pythonhosted.org/packages/61/cf/0d6598b1ceb463a46b75a716fbb23d2d5ccc5738aba8ddceeec3fa59997f/slimit-0.8.1.zip (88kB)
Collecting tinycss2>=0.4 (from html5print->fdroidserver==2.1a0.dev0)
  Downloading https://files.pythonhosted.org/packages/65/f7/63bf697a7c7257d304269b49f1be3dfe429856889e93963d6f5790d77d82/tinycss2-1.1.0-py3-none-any.whl
tinycss2 requires Python '>=3.6' but the running Python is 3.5.3
```
2021-04-06 10:23:17 +02:00
Benedikt Brückmann
8f0712e89b Add a mypy run to the pipeline 2021-04-02 08:57:56 +00:00
Hans-Christoph Steiner
20c299358d gitlab-ci: modernize ubuntu_lts_ppa job using updates to PPA
The PPA now provides apksigner that does not need binfmt, and defusedxml is
a hard Depends:.
2021-03-24 11:20:00 +01:00
Hans-Christoph Steiner
67e65ce2ea purge complete-ci-tests, those are better covered in .gitlab-ci.yml 2021-03-22 23:36:04 +01:00
Hans-Christoph Steiner
93ca13fa16 run-tests: remove broken test that never ran due to a syntax error 2021-03-22 22:31:29 +01:00
Hans-Christoph Steiner
cc3b88ab30 gradlew-fdroid: use gradle zipballs from local cachedir, if present 2021-03-11 08:40:34 +01:00
Hans-Christoph Steiner
d75ba81be0
gitlab-ci: arch job needs to build python-cryptography 2021-02-09 09:26:07 +01:00
Benedikt Brückmann
b1bde9e407 * Skip tidy check if html5print is not available
* install test dependencies on arch
2021-02-08 12:25:02 +01:00
Hans-Christoph Steiner
c887c14be1 gitlab-ci: use python3-cryptography to avoid building rust sources 2021-02-08 11:40:41 +01:00
Hans-Christoph Steiner
3d536d4515
gitlab-ci: use python3-cryptography to avoid building rust sources 2021-02-08 10:59:48 +01:00
Hans-Christoph Steiner
e4087f1798 gitlab-ci: hide massively verbose messages from sdkmanager 2021-01-28 22:25:33 +01:00
Hans-Christoph Steiner
87477cd910
gitlab-ci: gradle job needs git history to find what changed 2021-01-25 11:05:22 +01:00
Hans-Christoph Steiner
d133b38f22 fdroid-bot creates merge request when new gradle versions are found 2021-01-25 00:26:14 +01:00