Import tweaks and tests
This makes the code for `fdroid import` a bit more flexible and adds tests for the first time. It also comments out options in `examples/config.py` that just mirror the defaults to make it clear that they are defaults, and help illustrate other options (this is standard procedure in default config files).
More details in the commit messages.
See merge request !76
replace deprecated optparse with argparse
squashed and rebased merge request fdroid/fdroidserver!74
following guidelines from:
https://docs.python.org/2/library/argparse.html#upgrading-optparse-code
except, still using option = parse.parse_args() instead of args = ...
- using the following script in folder fdroidserver:
```
for i in *.py; do
sed -i -e 's/optparse/argparse/' \
-e 's/OptionParser/ArgumentParser/' \
-e 's/OptionError/ArgumentError/' \
-e 's/add_option/add_argument/' \
-e 's/(options, args) = parser/options = parser/' \
-e 's/options, args = parser/options = parser/' \
-e 's/Usage: %prog/%(prog)s/' $i;
done
```
- use ArgumentParser argument to replace (option, args) = parser.parse()
call
- use parser.error(msg) instead of raise ArgumentException as suggested
in https://docs.python.org/2/library/argparse.html#exiting-methods
- in fdroid catch ArgumentError instead of OptionError
See merge request !75
If for whatever reason the update check results in an older version that we
didn't package, don't "update" to that version if we already packaged newer
versions.
following guidelines from:
https://docs.python.org/2/library/argparse.html#upgrading-optparse-code
except, still using option = parse.parse_args() instead of args = ...
- using the following script in folder fdroidserver:
for i in *.py; do
sed -i -e 's/optparse/argparse/' \
-e 's/OptionParser/ArgumentParser/' \
-e 's/OptionError/ArgumentError/' \
-e 's/add_option/add_argument/' \
-e 's/(options, args) = parser/options = parser/' \
-e 's/options, args = parser/options = parser/' \
-e 's/Usage: %prog/%(prog)s/' $i;
done
- use ArgumentParser argument to replace (option, args) = parser.parse()
call
- use parser.error(msg) instead of raise ArgumentException as suggested
in https://docs.python.org/2/library/argparse.html#exiting-methods
- in fdroid catch ArgumentError instead of OptionError
For a bit repo like f-droid.org, it makes sense to standardize on a single
format for metadata files. This adds support for enforcing a single data
format, or a reduced set of data formats. So f-droid.org would run like
this if it changed to YAML:
accepted_formats = ['txt', 'yaml']
Then once everything was converted to YAML, it could look like this:
accepted_formats = ['yaml']
In order to prevent confusion caused by multiple metadata files for a given
app, fdroid will exit with an error if it finds any app metadata file with
the same package ID as one that has already been parsed.
This puts process of setting up the defaults for the internal dict
that represents a parsed app into a single method that is reused for all
metadata formats.
YAML is a format that is quite similar to the .txt format, but is a
widespread standard that has editing modes in popular editors. It is also
easily parsable in python.
The .pickle for testing is a lightly edited version of the real metadata
for org.videolan.vlc:
* comments were removed
This puts all of the needed post parsing checks on the metadata into a
single method that is used by all parsing methods (.txt, JSON, XML, YAML).
This provides the single place to normalize the internal representation of
the metadata.
It would be good to also change the internal representation to use more Python
bool/int types so that less post parsing is needed for JSON, XML, and YAML.
The SMSSecure test .pickle was changed to account for the use of lstrip()
and rstrip() on all 'script' types.
This also changes the example JSON to use ints for versionCodes
No need for a specific metadata type for 'Requires Root'. Just use 'bool',
then convert on output. This keeps the internal representation consistent
since all boolean-ish values will be Python bools. This in turn makes
parsing metadata in various formats easier. This also makes setting
booleans more tolerant since all these values are valid:
Yes yes No no True true False false
While the current text metadata format is good for human readability and
editability, it is difficult to produce and parse using code. XML is a
widespread standard format for easy automatic parsing and creating, while
having decent human readability.
The .pickle for testing is a lightly edited version of the real metadata
for net.osmand.plus:
* comments were removed
* "NonFreeNet" was added as an AntiFeature
The AntiFeatures metadata is a comma-separated list of tags, like
Categories, so it should also be stored internally as a list. This makes
parsing XML and JSON easier.
The test cases' .pickle files look like they change a lot, but they really
don't, its only the change of default AntiFeatures value from None to []
The only data produced by parsing metadata that was stored as a tuple was
the comments' key/values. Everywhere else, lists are used. This changes
the comments to also only use lists to keep the data consistent internally.
That makes it vastly easier to produce the exact same metadata dict when
parsing .txt, JSON, XML, etc.
fdroidserver currently uses plain python 2.x strings rather than unicode
strings, so the parsed JSON should produce the same format as the parsed
.txt metadata.
While the current text metadata format is good for human readability and
editability, it is difficult to produce and parse using code. JSON is a
widespread standard format for easy automatic parsing and creating, while
having decent human readability.
