With FAT filesystems, the user, group, and permissions will not be at all
preserved. With file systems like ext4 that have perms, the umask might
not be set to something that makes sense for the public repo files, which
are meant to be published and therefore readible by all.
If need be, it would be easy enough to add a config option for rsync's
chmod string, to address setups that have specific permissions needs.
fixes#23https://gitlab.com/fdroid/fdroidserver/issues/23
It is easier to handle programming with python rather than subprocess calls
so I replaced the subprocess call to 'ssh' with paramiko. This also makes
fdroid more portable since it no longer relies on the local system having
ssh installed.
It seems that paths for rsync must have a trailing slash in order to sync
rather than make a subdir, i.e. this makes a duplicate subdir:
rsync /tmp/fdroid/repo repo
While this syncs the dirs
rsync /tmp/fdroid/repo/ repo/
If `fdroid server update` is run with config that includes an archive, but
the 'archive' subdir does not exist, create it. This mirrors the code that
is in `fdroid update`. Seems to trivial to move to common.py.
To support a fully offline build/signing machine, there is the "local copy
dir". The repo is generated on the offline machine and then copied to a
local dir where a thumb drive or SD Card is mounted. Then on the online
machine, using `fdroid server update --sync-from-local-copy-dir` allows
the whole server update process to happen in a single command:
0. read config.py on online machine's repo
1. rsync from the local_copy_dir to the current dir
2. copy to serverwebroot, awsbucket, etc.
In `fdroid server update`, the rsync command used --update, which
`man rsync` says: "skip files that are newer on the receiver". That could
cause issues of the public repo getting out of sync with the private,
master repo. --archive is a better sync method since it aims to exactly
reproduce the sending dir to the receiving dir.
This allows a dir to be specified in config.py that `fdroid server update`
will automatically rsync the repo to. The idea is that the path would
point to an SD card on a fully offline machine that serves as the secure
repo signing machine.
This allows the SSH key used to sync with the server to be specified via
the config.py or the command line. I need it for running automated tests
and setups.
rsync uses the modification time and size of the file when deciding whether
to update a file. These are relatively easy to control in malicious code,
so instead make rsync use a full MD5 checksum when decided whether the
index needs to be updated. I suppose we could add an option to use
checksum checking on all files, but since the signed repo already provides
a checksum check, it seems not worth the added load on the process.
Also, renamed 'index' to 'indexxml' to make it clear what is the XML and
what is the JAR.
If a key 'foo' is set to None, `if config.get('foo'):` will be false while
`if 'foo' in config:` will be true. A None value is not useful here, so
config.get() is the better check.
Thanks to Adam Pritchard for the suggestion.
Since it is possible to check the file size and MD5 hash of the file up on
the AWS S3 bucket, `fdroid server update` can check that a file needs to be
updated before actually deleting and uploading the new file.
fixes#3137https://dev.guardianproject.info/issues/3137
This makes the AWS S3 setup dead simple: just put in a awsbucket name of
your choosing, set the AWS credentials, and it'll do the rest, whether the
bucket exists already or not. S3 buckets are trivial to delete too, in
case of error: `s3cmd rb s3://mybadbucketname`.
apache-libcloud enables uploading to basically any cloud storage service.
This is the first implementation that allows `fdroid server` to push a repo
up to a AWS S3 'bucket'. Supporting other cloud storage services should
mostly be a matter of finding the libcloud "Provider" and setting the
access creditials.
fixes#3137https://dev.guardianproject.info/issues/3137
Right now, ssh+rsync is the only supported server upload type. Things like
cloud storage services are useful storage bins for fdroid repos since they
are often not blocked while specific websites like Google Play are.
Having serverwebroot optional in `fdroid server` means that it can support
multiple methods of hosting, like cloud storage services. `fdroid server`
can also then support multiple repo hosting options at the same time.
For user-generated repos, the default path/URL is .*/fdroid/repo, with
fdroid/ as the root where the 'fdroid' tool operates. This makes for a URL
that is quite unique and easily matched automatically using patterns, like
in fdroidclient.
For those who don't like the standard, they can override the errror from
config.py using nonstandardwebroot = True
This is quite simple, but makes the user experience consistent when setting
up repos: run init then update locally, then run init then update on the
server.
This patch also includes PEP8 formatting fixes
