Follow

Can I use the command line or WebDav?

Yes, you can!

WebDAV URL: https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/

Note:

Cloudstor will authorize a maximum of 3 login events within a given five minute period for a given user account.

We strongly recommend setting a separate app password for your WebDAV connection, however you can use either the same app password that you use to login to the sync client or your sync password. (for more information see: https://support.aarnet.edu.au/hc/en-us/articles/205160658-Why-set-a-different-password-for-WebDAV-and-the-desktop-and-mobile-clients-)

 

Command line:

We recommend Open Source Project rclone https://rclone.org

Documentation here: https://rclone.org/docs/

or

Cyberduck's CLI tool tool https://duck.sh/

Documentation here: https://trac.cyberduck.io/wiki/help/en/howto/cli#Windows

 

rclone

Setup

Rclone requires some setting up in order to communicate with CloudStor.

Note items in bold text. Also please read the questions as this is for v1.52.2 and the option may change in future updates.

C:\temp\rclone-v1.52.2-windows-amd64>rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> Cloudstor
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
2 / Alias for an existing remote
\ "alias"
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Citrix Sharefile
\ "sharefile"
9 / Dropbox
\ "dropbox"
10 / Encrypt/Decrypt a remote
\ "crypt"
11 / FTP Connection
\ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
13 / Google Drive
\ "drive"
14 / Google Photos
\ "google photos"
15 / Hubic
\ "hubic"
16 / In memory object storage system.
\ "memory"
17 / Jottacloud
\ "jottacloud"
18 / Koofr
\ "koofr"
19 / Local Disk
\ "local"
20 / Mail.ru Cloud
\ "mailru"
21 / Mega
\ "mega"
22 / Microsoft Azure Blob Storage
\ "azureblob"
23 / Microsoft OneDrive
\ "onedrive"
24 / OpenDrive
\ "opendrive"
25 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
26 / Pcloud
\ "pcloud"
27 / Put.io
\ "putio"
28 / QingCloud Object Storage
\ "qingstor"
29 / SSH/SFTP Connection
\ "sftp"
30 / Sugarsync
\ "sugarsync"
31 / Tardigrade Decentralized Cloud Storage
\ "tardigrade"
32 / Transparently chunk/split large files
\ "chunker"
33 / Union merges the contents of several upstream fs
\ "union"
34 / Webdav
\ "webdav"
35 / Yandex Disk
\ "yandex"
36 / http Connection
\ "http"
37 / premiumize.me
\ "premiumizeme"
38 / seafile
\ "seafile"
Storage> 34
** See help for webdav backend at: https://rclone.org/webdav/ **

URL of http host to connect to
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Connect to example.com
\ "https://example.com"
url> https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/
Name of the Webdav site/service/software you are using
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Nextcloud
\ "nextcloud"
2 / Owncloud
\ "owncloud"
3 / Sharepoint
\ "sharepoint"
4 / Other site/service or software
\ "other"
vendor> 2
User name
Enter a string value. Press Enter for the default ("").
user> user@institute.edu.au
Password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> y
Enter the password:
password: <<SYNC or APP password>>
Confirm the password:
password: <<SYNC or APP password>>
Bearer token instead of user/pass (eg a Macaroon)
Enter a string value. Press Enter for the default ("").
bearer_token> <<LEAVE BLANK, press enter key>>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> y
Command to run to get a bearer token
Enter a string value. Press Enter for the default ("").
bearer_token_command>
Remote config
--------------------
[Cloudstor]
type = webdav
url = https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/
vendor = owncloud
user = user@institute.edu.au
pass = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name Type
==== ====
Cloudstor webdav

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Parallel upload example:

# rclone copy --progress --transfers 8 ~/testdata/ CloudStor:/test
  Key Value Notes
Parallel streams --transfers 8 Recommend not to go beyond 8 parallel streams
Upload to   CloudStor:/test Upload to “test/”. The CloudStor path is whatever is after CloudStor:/ in the path (CloudStor:/test)
Upload from   ~/testdata/  

Parallel download example:

# rclone copy --progress --transfers 8 CloudStor:/test ~/testdata/ 
  Key Value Notes
Parallel streams --transfers 8 Recommend not to go beyond 8 parallel streams
Download from   CloudStor:/test Download from “test/”. The CloudStor path is whatever is after CloudStor:/ in the path (CloudStor:/test)
Download to   ~/testdata/  

 

File listing example

# rclone ls CloudStor:/test
  Key Value Notes
Username -u user@institute.edu.au  
List ls    
List from   CloudStor:/test List from “test/”. The CloudStor path is whatever is after CloudStor:/ in the path (CloudStor:/test)

 

We also have a wrapper script for rclone:

https://github.com/AARNet/copyToCloudstor 

It features:

  • Automatic retry failed uploads
  • Handle data sets with large number of directories
  • Tuned default values for smooth uploads

This script is recommended for automating data uploads to CloudStor.

 

Duck.sh

Parallel upload example: 

# duck -u user@institute.edu.au --parallel 8 -e compare --upload davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/ ~/testdata/

 

  Key Value Notes
Username -u user@institute.edu.au  
Parallel streams --parallel 8 Recommend not to go beyond 8 parallel streams
Transfer action for existing files -e compare Skips files that match size, modification date or checksum
Upload --upload    
Upload to   davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/ Upload to “test/” . The CloudStor path is whatever is after webdav/ in the url (davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/)
Upload from   ~/testdata/ Must use absolute paths

Parallel download example:

# duck -u user@institute.edu.au --parallel 8 -e compare --download davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/ /tmp/testdata/

 

  Key Value Notes
Username -u user@institute.edu.au  
Parallel streams --parallel 8 Recommend not to go beyond 8 parallel streams
Transfer action for existing files -e compare Skips files that match size, modification date or checksum
Download --download    
Download from   davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/ Download from “test/”. The CloudStor path is whatever is after webdav/ in the url (davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/)
Download to   /tmp/testdata/ Must use absolute paths

File listing example

# duck -u user@institute.edu.au --list davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/

 

  Key Value Notes
Username -u user@institute.edu.au  
List --list    
List from   davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/ List from “test/”. The CloudStor path is whatever is after webdav/ in the url (davs://cloudstor.aarnet.edu.au/plus/remote.php/webdav/test/)