Siacoin - Introducing full-computer backup with Sia through the new Duplicati integration



  • We are excited to announce that our second bounty, an integration between Sia and Duplicati, is now complete! User fuffenz submitted the integration only two days after we posted the bounty, and user alexbarnsley helped test. The Duplicati team has merged the integration into their code, so all Duplicati users can now select Sia as a backup destination!

    Duplicati is full-featured, open source backup software, which allows you to select backup destinations (such as Amazon S3 or Dropbox) and schedule automated backups. It provides a friendly browser-based UI, so even nontechnical users can quickly configure backups.

    Crucially, Duplicati compresses large numbers of files into a small number of zip folders — which are more easily handled by the Sia daemon.

    This is a really important integration, as it allows users to back up any folder in their computers directly to the Sia network. You can download the latest release at their Github. Sia support is currently in the Canary version, starting from v2.0.2.3, and will be included in the next main release.

    duplicati/duplicati
    duplicati - Store securely encrypted backups in the cloud!github.com

    Have questions or want to discuss? Come join our #bounties channel on Discord or hop onto the Duplicati Forum!

    Congratulations to fuffenz and alexbarnsley! The bounty reward will be split between them, with 75% going to fuffenz and 25% to alexbarnsley.

    Getting Started

    You’ll need a few things to get started with the Duplicati integration:

    1. A Sia instance running on your computer or server. Sia-UI or siac both work. For nontechnical users, Sia-UI is recommended, as it provides a friendly user interface.
    2. An unlocked and funded Sia wallet, with contracts already set up. You can follow these guides if you are new to Sia. Initial contract formation can take ~30 minutes, and initial sync of the blockchain can take a few hours on SSDs or a couple days on HDDs.
    3. Duplicati installed and running on the same computer or server. Duplicati may ask you to install some dependencies.

    Configuring a new backup

    First, you’ll want to open Duplicati. It will launch in your browser.

    Click “Add backup” and choose to configure a new backup.

    Name your backup. Then, select no encryption. This is because Sia automatically encrypts everything before uploading. You can encrypt in Duplicati as well, but it would be redundant.

    Choose your backup destination and configure the settings:

    • Server: 127.0.0.1:9980 (This is the default for Sia. You have to type this in manually, or you will get an error.)
    • Folder-path: Choosing a path will create a folder in your Sia instance, and store the backups in that folder. This is not required, but is recommended.
    • Server password: Leave blank for most users. If you are using “authenticate” to start siad, then enter that password.
    • Minimum redundancy: This is the redundancy level that each zip folder needs to reach before Duplicati begins uploading the next folder (Duplicati uploads are unfortunately not parallel at this time). We recommend setting a redundancy level between 1 and 1.5 for fast uploads. I used 1.2. Note: Sia will boost the redundancy automatically until it gets to 2.0-3.0. If you specify a redundancy less than that, it may be a bit more expensive to boost the redundancy. That’s because Duplicati will have deleted the temporary file, so Sia has to use online file repair to download files and upload to new hosts. So, if you prioritize cost over time, you could set the redundancy higher.

    Choose your source data — this is the stuff you’ll be uploading. I’ve chosen to upload my entire Home directory, which is about 11 GB.

    Optional: set a schedule for automated backups.

    Finally, choose the Upload volume size. This dictates how large each zip folder is. For example, if you choose 1 GB upload volume size, and upload 10 GB of compressed data, you will have 10 zip folders (each about 1GB in size). If you choose 500 MB upload volume size, you would have 20 zip folders. And so on.

    Sia does a good job handling large files, but does not perform as well when handling large quantities of files. For that reason, setting an Upload volume size of 500 MB or higher is a good idea. As we improve the way Sia handles large quantities of files, this will not matter.

    You’re done! You can click “Run now” or just wait for the scheduled backup to commence. You’ll see a green progress bar up top when the backup is running.

    And, on the Sia side, you’ll have a new folder filled with your compressed backup files.

    Troubleshooting

    Having trouble getting things working? Follow the below suggestions.

    • Make sure your Sia instance is running, is updated to v1.3.0 or higher, is unlocked, has contracts, and has enough funds in the wallet to renew contracts as needed.
    • Make sure you are running the latest Duplicati build.
    • If the upload is taking too long, you may want to (1) reduce your minimum redundancy specified in the Duplicati config or (2) increase your Upload volume size.
    • Still having issues? Join our #bounties channel Discord or the Duplicati Forum.

    Known Issues

    • Sia currently cannot restore your data from the 29-word seed (this feature will be introduced in a couple months). So, if your hard drive fails completely, your data will still be lost. To create a foolproof backup system, you will have to backup your Sia data folder to another backup service. You can do this through Duplicati as well, setting up an automated backup to Dropbox, Google Drive, or another service. Just back up the folders found at: Windows: %UserProfile%\AppData\Roaming\Sia-UI\sia Linux: ~/.config/Sia-UI/sia/ Mac: ~/Library/Application Support/Sia-UI/sia/
    • Duplicati uploads files one-at-a-time. Sia is optimized for parallel uploads. In order to greatly increase upload speed, Duplicati will need to be improved to allow for parallel uploads. See outstanding bounty: https://github.com/duplicati/duplicati/issues/2026
    • Each Sia client can only upload up to ~5 TB of data. We are working hard to improve this. Our next update, slated for November, should get us up to ~50 TB per client.


Looks like your connection to Cryptocentral was lost, please wait while we try to reconnect.