Commit graph

1864 commits

Author SHA1 Message Date
Hans-Christoph Steiner
4422dca528 fix py3 bug when running fdroid build with a buildserver
File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 1119, in main
    options.onserver, options.refresh):
  File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 954, in trybuild
    build_server(app, build, vcs, build_dir, output_dir, force)
  File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 403, in build_server
    output += get
TypeError: Can't convert 'bytes' object to str implicitly

closes #164 https://gitlab.com/fdroid/fdroidserver/issues/164
2016-03-15 13:21:19 +01:00
Hans-Christoph Steiner
016fa78111 fix stab at a Cygwin port (aaaahh Windows!)
This gets `fdroid init` and `fdroid readmeta` working on Cygwin.
2016-03-15 11:30:27 +01:00
Ciaran Gultnieks
5d296f8c61 Fix python3 checkupdates crashes 2016-03-14 22:29:46 +00:00
Ciaran Gultnieks
6cecb69bf2 Fix another python3 build crash 2016-03-14 22:23:19 +00:00
Hans-Christoph Steiner
7039d16046 always parse versions as strings, not bytes
Fixes a couple errors like:
  File "./makebuildserver", line 30, in vagrant
    out += line
  TypeError: Can't convert 'bytes' object to str implicitly

If universal_newlines=False, the default, then Popen will return bytes if
the newlines in the data do not match the system's newlines.  Setting it to
true enables auto-conversion, and then guarantees that the data is always
str.

"If universal_newlines is True, the file objects stdin, stdout and stderr
are opened as text streams in universal newlines mode, as described above
in Frequently Used Arguments, otherwise they are opened as binary streams."
https://docs.python.org/3/library/subprocess.html#subprocess.Popen
2016-03-14 12:49:38 +01:00
Daniel Martí
f267a1d7c9 Merge branch 'py3' into 'master'
Python 3

I tried to keep commits separate, so if anything causes trouble, it can be reverted or changed easily.

* pre-commit hooks pass
* all tests pass
* My use of `build`, `checkupdates`, `lint`, `import`, `publish` and `update` work as usual
* 2to3 does not report anything useful anymore (only useless parentheses and list() encapsulation of iterators)
* rewritemeta works exactly as usual

CC @eighthave

See merge request !88
2016-03-11 23:51:56 +00:00
Daniel Martí
a023a235a2 metadata: always open metadata files in UTF-8
For some reason, Python uses whatever encoding the system is using via a
locale. Since CI uses an ascii locale, stuff breaks. I can't find a way
to make all of fdroidserver ignore the locale and just use UTF-8, so for
now force it where it's bothering CI.
2016-03-11 13:27:01 +00:00
Daniel Martí
23a950bd00 Fix issue related to creating fdroidvcs file 2016-03-11 13:27:01 +00:00
Daniel Martí
d98b4d1b83 metadata: slightly speed up post_metadata_parse
Iterating over all the fields and checking which are modified is slower
than just iterating over the modified ones.
2016-03-11 13:27:01 +00:00
Daniel Martí
7f6276889e metadata: Sorting builds twice is not necessary 2016-03-11 13:27:00 +00:00
Daniel Martí
d4f806d757 Undo utf-8 text wrap workaround
This was added to fix unicode support in Python2. No longer needed in
Python3.
2016-03-11 13:27:00 +00:00
Daniel Martí
031bb39961 import: switch to python3 urllib 2016-03-11 13:27:00 +00:00
Daniel Martí
4e28fa78fc Fix python headers in some source files
These were missing the python2 header, so they went under my radar when
doing the switch go python3.
2016-03-11 13:27:00 +00:00
Daniel Martí
a1703f1204 update: fix encoding issues with the xml index 2016-03-11 13:27:00 +00:00
Daniel Martí
24266d48e1 update: get output bytes, not str, from keytool 2016-03-11 13:27:00 +00:00
Daniel Martí
4cb5ff0247 common: Introduce FDroidPopenBytes
* We can now get the output bytes directly if desired
* FDroidPopen remains with str output
* Ouptut is now buffered efficiently with a BytesIO instead of
  concatenating strings
2016-03-11 13:27:00 +00:00
Daniel Martí
63f8718cf5 Replace iterkeys() with keys() 2016-03-10 16:43:37 +00:00
Daniel Martí
8b3c27cb47 update: replace encode('hex') with hexlify 2016-03-10 16:43:37 +00:00
Daniel Martí
b31e20ee8b publish: fix hashing of strings 2016-03-10 16:43:37 +00:00
Daniel Martí
a78341d8e5 init: fix writing into file 2016-03-10 16:43:37 +00:00
Daniel Martí
da258dad57 Replace md5 with hashlib.md5 2016-03-10 16:43:37 +00:00
Daniel Martí
dd8ad7a4fd common: fix str fetching from manifest xml 2016-03-10 16:43:37 +00:00
Daniel Martí
d39bf37ce8 Port urllib and HTMLParser imports to python3 2016-03-10 16:43:37 +00:00
Daniel Martí
fc21dbc667 Replace remaining file() usage 2016-03-10 16:43:37 +00:00
Daniel Martí
5026d4e08b FDroidPopen: return str again
In the future we might want to instead return bytes, but for now the
easiest for the port to python3 is to continue to return str.
2016-03-10 16:43:37 +00:00
Daniel Martí
b73cc8e0b3 Replace itervalues() with values() 2016-03-10 16:43:37 +00:00
Daniel Martí
ce18d1cc85 Port to python3's configparser 2016-03-10 16:43:37 +00:00
Daniel Martí
2493e705f2 lint: remove sets usage 2016-03-10 16:43:37 +00:00
Daniel Martí
ec301b8566 common: update base64 encoding of bytes 2016-03-10 16:43:37 +00:00
Daniel Martí
920ae4692f Port all imports to python3 2016-03-10 16:43:37 +00:00
Daniel Martí
11b8b1ca4f update: replace ord() with bytearray 2016-03-10 16:43:37 +00:00
Daniel Martí
ee9a296b64 Make pre-commit hook pass after python3 switch 2016-03-10 16:43:37 +00:00
Daniel Martí
099b45fcca Remove import workarounds for Python2 2016-03-10 16:43:37 +00:00
Daniel Martí
f1f5836a66 Remove code that worked around python2 unicode 2016-03-10 16:43:37 +00:00
Daniel Martí
e829b0f9e7 Get readmeta and rewritemeta running under python3 2016-03-10 16:43:37 +00:00
Daniel Martí
832b1224b9 Replace basestring with str 2016-03-10 16:43:37 +00:00
Daniel Martí
75419c1f04 Replace iteritems() with items() 2016-03-10 16:43:37 +00:00
Daniel Martí
6819c109fe Replace execfile with open+compile+exec 2016-03-10 16:43:37 +00:00
Daniel Martí
ca5ee87b16 Switch to io.StringIO 2016-03-10 16:43:37 +00:00
Daniel Martí
e3591cb7b8 Fix imports in Python 3
Since common and metadata import each other, we must import via
"fdroidserver.*".
2016-03-10 16:43:37 +00:00
Daniel Martí
99edd64372 Switch all headers to python3 2016-03-10 16:43:36 +00:00
Daniel Martí
2a73b74603 checkupdates: sort tags by date in one go
This greatly speeds up checkupdates when UCM:Tags is used on git
repositories, especially on ones with lots of tags. The old method ran a
command per tag. The new method runs a single command and doesn't
require any sorting.

Test runs show that `fdroid checkupdates org.adaway` is down from ~13s
to ~10s on my laptop.
2016-03-10 11:14:08 +00:00
Daniel Martí
261cbcd3ee build: don't remove wrapper dir gradle/
The point of removing these was to make sure that gradlew wasn't used,
and that our gradle was used instead. Removing the scripts already
accomplishes this.

Removing gradle/ should be harmless, but some apps like I2P re-use this
directory to also hold other stuff that is actually needed. So be safer
and don't remove it at all.
2016-03-08 23:50:14 +00:00
Daniel Martí
ace4834cf7 lint: Also warn about gitlab links missing /issues 2016-03-07 22:30:52 +00:00
Boris Kraut
f4ce1737d9 import: use .git suffix only for repo-url 2016-03-06 11:20:10 +01:00
Daniel Martí
920f11d047 lint: error on fdroid import disable line 2016-03-03 14:19:32 +00:00
Daniel Martí
bf32477433 Merge branch 'p1' into 'master'
Fix pubkey extraction on update

Replacement of !86.

Fix pubkey extraction in case of non-empty _JAVA_OPTIONS. Fixes #133.

I didn't actually run the test suite (it looks like there are some preparations to be done for that), but I checked the commands from `test_fdroid_popen_stderr_redirect` in ipython.

See merge request !103
2016-02-28 13:37:17 +00:00
Daniel Martí
e3f60e2b78 common: make jdk detection more strict
We weren't using ^ and $, so stuff like java-8-openjdk-i386 would match
both the Arch and the Debian regexes. A list with one regex per line in
the same format is also easier to read and maintain.

This might be why java8 isn't being detected properly on our Debian
stable buildserver.
2016-02-23 12:43:58 +00:00
Daniel Martí
3e124c8105 build: don't error on .fdroid* files in fdroiddata
This avoids the following incorrect error:

	fdroidserver.common.FDroidException: Only one local metadata file allowed! Found: .fdroid.keystorepass.txt .fdroid.keypass.txt
2016-02-23 12:03:26 +00:00
Daniel Martí
2e5c96dcfa Merge branch 'scanner-repos' into 'master'
Allow commonsware and gradle plugin repos

This is in a MR because I'm uneasy about adding a semi-random AWS url to the list. But seems like the commonsware guy only publishes jars there, which doesn't make any sense to me.

Any idea @eighthave?

See merge request !102
2016-02-20 07:29:12 +00:00