This allows all the text to be localized via Weblate. This is a quick
overview of all the strings, but there are certainly some that were left
out.
closes#342
There were multiple conventions used in the code, but mostly it was already
using the convention from the docs, so this converts things to using that
convention:
https://docs.python.org/3/library/os.html#os.walk
This makes `fdroid server update` fail if pushing to one of the git mirrors
fails. This is what happens if the other methods fail, e.g. rsync or S3.
closes#347
git hosts like github, gitlab, bitbucket usually allow 1 gig repos. This
changes the git mirroring behavior to keep the history until the repo hits
1 gig. Keeping history makes updates a lot faster, since the whole repo
does not need to be pushed on each update.
It turns out it is error prone to `git push` to a non-bare git repo. For
the offline signing machine, the git remote needs to be a regular git repo
in a directory on a thumbdrive so that once the thumbdrive is plugged into
an online machine, that git repo can be transferred to the online machine.
This helps with uploading faster when hitting the public API rating
limiting since it will not hit the API if there is already a matching
response on the local filesystem. This also provides a source of data for
things like repomaker to diplay to users.
This restructures the virustotal uploading so that first checks whether
virustotal already has the file, and only if not does it upload it. This
also handles the public API rate limiting, which returns an HTTP 204. This
will now try again until it succeeds, even when rate limited. Instead of
just getting the list of files from the filesystem, this reads the index-v1
which also already has the SHA256 in there. virustotal also uses SHA256 as
a unique ID for files.
libcloud has been flaky for years, but it is also widely available and is
more pythonic. So this adds the option for using `s3cmd sync` if it is
available. `s3cmd sync` acts like rsync for uploading to S3. So this code
emulates the rsync uploading code in `fdroid server update`.
Instead of each time deleting the whole local git-mirror repo, and
recreating it, this just deletes the .git/ dir, then lets the rsync in
_local_sync() handle deleting anything that should no longer be in the repo.
This option was not hooked up at all, and does not make sense as a command
line argument. It should just be a config.py item. In that case, the
presence of config.py marks the current dir as a repo, so there is no
longer a need to test for a dir called repo/ as a safety. This makes the
setup easier, since sync_from_localcopy() now creates repo/ for the user.
Since `fdroid server update` is the place where all uploads to servers
happens, it makes sense to also handle the git push for the binary
transparency log here instead of `fdroid btlog`
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc). This still generates the exact same index.xml as before.
This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats. UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
A Binary Transparency Log is a append only log of all binaries published by
a repo. This is useful for people to find whether the binary they have
matches what F-Droid has published, and also makes it more difficult for
the published history to be changed without notice, or for a server to give
specific users custom malware binaries.
https://www.eff.org/deeplinks/2014/02/open-letter-to-tech-companies
following guidelines from:
https://docs.python.org/2/library/argparse.html#upgrading-optparse-code
except, still using option = parse.parse_args() instead of args = ...
- using the following script in folder fdroidserver:
for i in *.py; do
sed -i -e 's/optparse/argparse/' \
-e 's/OptionParser/ArgumentParser/' \
-e 's/OptionError/ArgumentError/' \
-e 's/add_option/add_argument/' \
-e 's/(options, args) = parser/options = parser/' \
-e 's/options, args = parser/options = parser/' \
-e 's/Usage: %prog/%(prog)s/' $i;
done
- use ArgumentParser argument to replace (option, args) = parser.parse()
call
- use parser.error(msg) instead of raise ArgumentException as suggested
in https://docs.python.org/2/library/argparse.html#exiting-methods
- in fdroid catch ArgumentError instead of OptionError
This keeps memory usage low because it only has to read a chunk at a time
into memory while before it read the whole file into memory before
uploading it.
This also seems to handle setting the permissions ACL better.
For devs that want to build and distribute nightly builds of their apps
using the fdroid tools. The core idea here is to make the fdroidserver
tool suite the single set of tools for all types of builds and releases.
That will hopefully drive more free software developers to make f-droid.org
an core channel for official releases.
For use cases where there is a web server running on the same machine where
the `fdroid update` is being run, allow plain paths in the serverwebroot
list. This is useful for debug repos from build servers, like:
https://dev.guardianproject.info/fdroid
In order to keep things working as much as possible during the update, the
rsync should only delete the obsolete APKs after it has finished uploading
the new APKs.
rsync's --chmod works a bit oddly, it only affects the source files. To
make it set the destintation with the perms set in --chmod, the --perms
flag must also be included.
