This helps with uploading faster when hitting the public API rating
limiting since it will not hit the API if there is already a matching
response on the local filesystem. This also provides a source of data for
things like repomaker to diplay to users.
This restructures the virustotal uploading so that first checks whether
virustotal already has the file, and only if not does it upload it. This
also handles the public API rate limiting, which returns an HTTP 204. This
will now try again until it succeeds, even when rate limited. Instead of
just getting the list of files from the filesystem, this reads the index-v1
which also already has the SHA256 in there. virustotal also uses SHA256 as
a unique ID for files.
libcloud has been flaky for years, but it is also widely available and is
more pythonic. So this adds the option for using `s3cmd sync` if it is
available. `s3cmd sync` acts like rsync for uploading to S3. So this code
emulates the rsync uploading code in `fdroid server update`.
Instead of each time deleting the whole local git-mirror repo, and
recreating it, this just deletes the .git/ dir, then lets the rsync in
_local_sync() handle deleting anything that should no longer be in the repo.
This option was not hooked up at all, and does not make sense as a command
line argument. It should just be a config.py item. In that case, the
presence of config.py marks the current dir as a repo, so there is no
longer a need to test for a dir called repo/ as a safety. This makes the
setup easier, since sync_from_localcopy() now creates repo/ for the user.
Since `fdroid server update` is the place where all uploads to servers
happens, it makes sense to also handle the git push for the binary
transparency log here instead of `fdroid btlog`
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc). This still generates the exact same index.xml as before.
This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats. UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
A Binary Transparency Log is a append only log of all binaries published by
a repo. This is useful for people to find whether the binary they have
matches what F-Droid has published, and also makes it more difficult for
the published history to be changed without notice, or for a server to give
specific users custom malware binaries.
https://www.eff.org/deeplinks/2014/02/open-letter-to-tech-companies
following guidelines from:
https://docs.python.org/2/library/argparse.html#upgrading-optparse-code
except, still using option = parse.parse_args() instead of args = ...
- using the following script in folder fdroidserver:
for i in *.py; do
sed -i -e 's/optparse/argparse/' \
-e 's/OptionParser/ArgumentParser/' \
-e 's/OptionError/ArgumentError/' \
-e 's/add_option/add_argument/' \
-e 's/(options, args) = parser/options = parser/' \
-e 's/options, args = parser/options = parser/' \
-e 's/Usage: %prog/%(prog)s/' $i;
done
- use ArgumentParser argument to replace (option, args) = parser.parse()
call
- use parser.error(msg) instead of raise ArgumentException as suggested
in https://docs.python.org/2/library/argparse.html#exiting-methods
- in fdroid catch ArgumentError instead of OptionError
This keeps memory usage low because it only has to read a chunk at a time
into memory while before it read the whole file into memory before
uploading it.
This also seems to handle setting the permissions ACL better.
For devs that want to build and distribute nightly builds of their apps
using the fdroid tools. The core idea here is to make the fdroidserver
tool suite the single set of tools for all types of builds and releases.
That will hopefully drive more free software developers to make f-droid.org
an core channel for official releases.
For use cases where there is a web server running on the same machine where
the `fdroid update` is being run, allow plain paths in the serverwebroot
list. This is useful for debug repos from build servers, like:
https://dev.guardianproject.info/fdroid
In order to keep things working as much as possible during the update, the
rsync should only delete the obsolete APKs after it has finished uploading
the new APKs.
rsync's --chmod works a bit oddly, it only affects the source files. To
make it set the destintation with the perms set in --chmod, the --perms
flag must also be included.
With FAT filesystems, the user, group, and permissions will not be at all
preserved. With file systems like ext4 that have perms, the umask might
not be set to something that makes sense for the public repo files, which
are meant to be published and therefore readible by all.
If need be, it would be easy enough to add a config option for rsync's
chmod string, to address setups that have specific permissions needs.
fixes#23https://gitlab.com/fdroid/fdroidserver/issues/23
It is easier to handle programming with python rather than subprocess calls
so I replaced the subprocess call to 'ssh' with paramiko. This also makes
fdroid more portable since it no longer relies on the local system having
ssh installed.
It seems that paths for rsync must have a trailing slash in order to sync
rather than make a subdir, i.e. this makes a duplicate subdir:
rsync /tmp/fdroid/repo repo
While this syncs the dirs
rsync /tmp/fdroid/repo/ repo/
If `fdroid server update` is run with config that includes an archive, but
the 'archive' subdir does not exist, create it. This mirrors the code that
is in `fdroid update`. Seems to trivial to move to common.py.
To support a fully offline build/signing machine, there is the "local copy
dir". The repo is generated on the offline machine and then copied to a
local dir where a thumb drive or SD Card is mounted. Then on the online
machine, using `fdroid server update --sync-from-local-copy-dir` allows
the whole server update process to happen in a single command:
0. read config.py on online machine's repo
1. rsync from the local_copy_dir to the current dir
2. copy to serverwebroot, awsbucket, etc.
In `fdroid server update`, the rsync command used --update, which
`man rsync` says: "skip files that are newer on the receiver". That could
cause issues of the public repo getting out of sync with the private,
master repo. --archive is a better sync method since it aims to exactly
reproduce the sending dir to the receiving dir.
This allows a dir to be specified in config.py that `fdroid server update`
will automatically rsync the repo to. The idea is that the path would
point to an SD card on a fully offline machine that serves as the secure
repo signing machine.
This allows the SSH key used to sync with the server to be specified via
the config.py or the command line. I need it for running automated tests
and setups.
rsync uses the modification time and size of the file when deciding whether
to update a file. These are relatively easy to control in malicious code,
so instead make rsync use a full MD5 checksum when decided whether the
index needs to be updated. I suppose we could add an option to use
checksum checking on all files, but since the signed repo already provides
a checksum check, it seems not worth the added load on the process.
Also, renamed 'index' to 'indexxml' to make it clear what is the XML and
what is the JAR.
