Maidsafe coin Details+Update





  • SAFE Network Introduction Video:




    Features of the SAFE Network

    The SAFE (Secure Access For Everyone) Network is made up of the unused hard drive space, processing power and data connection of its users. It offers a level of security and privacy not currently available on the existing Internet and turns the tables on companies, putting users in control of their data, rather than trusting it to organisations. A number of features make this possible:


    Login and secure your data with no middle man.






    No need to give your password to anyone, or ask a third party’s permission to access your data. Your information, and access to it, belongs to you!

    Your PIN and Keyword are used to locate your data on the network and your password, which never leaves your machine, is then used to decrypt it locally. This means that no one needs to hold a record of your files, or your login details and there’s no need to ask anyone for permission to access it. This process is called Self-Authentication and enables you to find, unlock and decrypt your own data.



    Self-Encryption: Data which encrypts itself, with itself.





    Files uploaded to the network are broken into pieces, encrypted and distributed across the network. This process is called Self-Encryption.

    When a user uploads (or saves) a file to the network, via one of the SAFE Network apps, the file is automatically broken up into chunks. These chunks are then encrypted (encoded so that only authorised parties can read it), randomised and stored on the computers of other SAFE Network users. These encrypted chunks are completely unreadable and inaccessible to anyone other than the owner.


    Distributed network with opportunistic data caching.




    The SAFE Network is fully decentralised, with files distributed and stored all over the world, on different devices. This allows the network to be robust to attacks, with no central point of weakness.

    The distributed nature of the network provides your data with physical security, meaning that no third party can access or delete it, as can happen with existing centralised solutions. The feature called Opportunistic Caching automatically creates more copies of popular data closer to where it is being requested, so popular websites and other data feeds will actually speed up as they get more visitors, rather than slow down and crash as they do on today’s web.


    Data availability and built-in redundancy.





    The network is programmed to keep duplicate copies of each piece of data at all times. As users turn their computers off, the network makes more copies and stores them on other machines, ensuring that users always have access to their files.

    This constant movement of data (called churn) is a key part of the security that the SAFE Network offers because there is no central point for hackers to target as the data locations keep changing.


    Unneeded duplicates are automatically removed.



    Once a file is uploaded, other users who upload the same file will be automatically referred to the original, limiting the number of copies and reducing the computing resources needed to store it


    Crowd sourced internet:


    Rather than using data centres and servers common within today’s internet, which are prone to data theft and surveillance, the SAFE Network uses advanced P2P technology, that joins together the spare computing capacity of all SAFE users, creating a global network. You can think of SAFE as a crowd sourced internet. It is on this network that everyone’s data and applications reside.

    All you need to join this network is to download the free SAFE software, create your account and tell the network how much space you are going to provide. It’s that simple!


    Safecoin:


    Global distribution without human intervention.

    Safecoins are distributed entirely by the algorithms within the SAFE Network. Human beings are not involved in the process. Only 4.3 billion coins will ever be in circulation at one time and each will have its own unique identity. Safecoins will be recycled when users exchange them for network services, ensuring there is always a supply for users to earn


    Farming: an incentive for a crowd sourced Internet.




    Safecoins are given as an incentive to users for providing their resource to the network. This resource is their: storage space, CPU, bandwidth and online time that enable the encrypted chunks of network data to be stored and retrieved from their computer. The process of providing resource and receiving safecoin in return is called ‘Farming’. Each piece of encrypted network data is stored in a Farmer's ‘Vault’, a data storage and management location on the Farmer's computer which they cannot read, or access. The SAFE network is designed to self-manage these resources


    Resource based economy




    As network data is retrieved from a computer, the network pays safecoin to the user. This payment doesn’t take place every time data is retrieved, instead it is more like a lottery, where the Farmer is rewarded at random. The safecoin is automatically paid by the network into the user’s safecoin wallet address that is tied (cryptographically) to their vault. The amount of safecoin a user can earn is directly linked to how much resource they provide to the network and how often their computer is available to the network (turned on).



    Proof of resource.





    The resource provided by each Farmer is continually checked by the network to make sure that it is still available. It does this by attempting to store a random piece of data. Should the network find that the resource committed to the network is no longer available, it reduces the rank of the vault thereby reducing the earning potential of that specific Vault.


    A new market place.






    Safecoins can be used to access and use network applications, with part of the ‘payment’ going directly to the application developer. Safecoins can also be bought by users unwilling, or unable to provide resource to the network. This creates a market of buyers and sellers so that farmers can sell the coins they earn for another currency. These transactions will take place on an exchange, another SAFE Network application, or directly between individuals.



    Decentralised Applications





    The ecosystem around the SAFE Network is already starting to blossom and a number of exciting apps* are already being built by third party developers using the SAFE APIs.

    Applications that provide access via the Launcher API ensure that you can access all apps with the same PIN, username and password used when you created your SAFE account. This means that you stay in control of your details at all times.

    *The projects listed within the link above are built and maintained by independent parties.


    Team: https://maidsafe.net/company.html

    Download: https://maidsafe.net/alpha.html





  • safe_vault

    safe_launcher

    safe_core

    routing

    self_encryption

    https://github.com/maidsafe

    Links:


    [ref Network] MaidSafe Network website : www.maidsafe.net

    [ref Autonomous] Autonomous Network, David Irvine, Fraser Hutchison, Steve Mucklisch : https://github.com/maidsafe/MaidSafe/wiki/unpublished_papers/AutonomousNetwork.pdf?raw=true

    [ref Routing] MaidSafe Routing github site : https://github.com/maidsafe/MaidSafe-Routing/wiki

    [ref BitCoin] Bitcoin : A Peer-to-Peer Electronic Cash System, Satoshi Nakamoto, https://bitcoin.org/bitcoin.pdf‎

    [ref RUDP] MaidSafe RUDP github site : https://github.com/maidsafe/MaidSafe-RUDP/wiki

    [ref Persona] Vault Documentation : https://github.com/maidsafe/MaidSafe-Vault/wiki/Documentation

    [ref Escrow] The Escrow service for Bitcoin : http://btcrow.com/

    White Paper:https://github.com/maidsafe/Whitepapers/blob/maste...



  • MaidSafe Announces Equity Funding Round on BnktotheFuture

    http://cryptocentral.info/topic/430/maidsafe-annou...






  • RFC 42 – SAFE Launcher API v0.6

    https://github.com/maidsafe/rfcs/blob/master/text/...




    Summary

    Expose low level API from launcher to allow applications to allow thrid party applications to access the building blocks of the SAFE Network.

    Motivation

    Exposing low level apis allow dynamic applications to be built on SAFE Network. Using the StructuredData API third party applications can build and manage their own data structures to fit their needs.

    Detailed design

    Expose low level building blocks via launcher for third party applications, which will allow building their own data structures. Moreover, dynamic applications can be built on using the appendable data api.

    The low level apis use the FFI interface for low level API as detailed in the safe_core low level RFC

    APIs for,

    are exposed from the launcher through the REST interface.

    Permission

    Application must request for LOW_LEVEL_API access permission to invoke the low level api to store or read encrypted data. Since low level apis can be used to create data in the network, it might be possible where the data created by the apps cannot be deleted by the user again to retrieve the lost space. Thus, it makes it important to request user for the permission to access low level apis.

    Unauthorised access

    Unauthorised access is granted for reading public data using low level api by default.

    Handle Id

    The low level apis return Handle-Id corresponding to the data type that is being worked with. For example, the structured data will return a DataIdentifier-handle-Id using this Handle-Id, the operations can be performed on the structured data.

    It becomes the applications responsibility to drop the handle after the usage.

    Limitation

    Ability to work with versioned data is exposed only from low level APIs. Launcher's NFS API doesn't support versioning yet.

    Drawbacks

    If the handle id is not cleaned up properly it can lead to memory leak. safe_core can purge if the number of handles goes above a certain threshold.

    Alternatives

    Nil

    Unresolved questions

    Nil



  • MaidSafe Dev Update – October 11, 2016

    Last week we released the second dev tutorial15. Since we are trying to make a new release roughly every two weeks, you can expect Dev Updates between releases to be a bit more quiet

    In addition to working on the next dev tutorial, the front-end team is discussing improvements related to app authorization, permissions and data sharing between applications.

    We are also thinking about how SAFE Launcher can be scaled to various platforms (e.g. mobile, embedded, etc.). For example, SAFE Launcher currently needs to run in the background and continuously proxy HTTP requests to the SAFE Network, but on some mobile platforms that wouldn't necessarily be possible. It would be preferable if mobile apps could communicate directly with the SAFE Network.

    There are multiple approaches we are looking at, and we're planning to have a detailed discussion about potential changes to SAFE Launcher this week. We'll write a summary of the discussion and post it on the Dev Forum. The ideas we'll present will still be open to debate by the community and we’d love to hear your thoughts and feedback once we post that topic

    SAFE Launcher

    APIs that were integrated late into safe_core are being updated in launcher. The API for handling versioned structured data is being updated and tested from launcher.

    The test module for the APIs is being updated to work with electron-mocha6 and once this switch is done, the CI system should be able to run the test suite again.

    As said in the introduction, this week has been more related to discussions on how the launcher should be shaped for the future and how the APIs should be exposed.

    SAFE Core

    We have started the async overhaul this week and are using (relatively much more recent) rust-async-paradigm crates tokio-core1 and futures-rs1. That is the direction Rust community seems to be moving and lot of work is going into them. We have created a separate dev branch to allow master to continue to be usable with present Launcher etc., as it is a big and a breaking change. A lot of work has gone into that already in 2 days and there is steady progress.

    Routing & Vault

    This week we have another member joining the Routing team: Diggory Hardy18

    With 2 new members in Routing and Andreas getting back, we are now looking at our next milestone: user running vaults from home. This is our number one priority.

    The big issue we need to address for vaults from home is being able to prune out vaults that are so low resourced they kill the network. Along with RFC 37 – Disjoint Groups2 (which is already underway), to address the issue of low-powered nodes, we are looking at RFC 45 – Node Ageing9.

    We are also discussing what other changes we need to make before being able to start testing vaults from home again. For example, data chains (which will provide data republish) is not a priority for now, this will come after we are able to test vaults from home.

    Dev Outreach

    With @lightyear starting full time in October, we've started planning our dev outreach strategy. We are currently researching events and have submitted applications to the ones we think most suitable. In comparison with previous years, we are going to be attending more events than ever through 2017 as this is a key part of our strategy in pushing engagement on the back of the alpha and early tutorial releases. We will keep you posted about the events we’ll take part in, of course.

    Each event varies and we will use a combination of talks and workshops where possible. While talks are great for introducing concepts, we are excited by the workshop format, enabling Ben and other members of the team to bring app developers up the learning curve.

    We see dev outreach as a key part of our strategy moving forward and growing the developer outreach team during 2017 will be one of our priorities.

    In addition to attending conferences, we will also be seeding and working with members of the community interested in presenting SAFE at meetups, or run their own developer events and workshops. We will support this activity as much as possible. If you are interested in doing any of these, please get in touch with @lightyear or me (@frabrunelle).

    If you have any suggestions for conferences you think would help us spread the word in the developer community, please post them here1.

    Bitcoin Wednesday, Amsterdam

    We are going to be speaking at Bitcoin Wednesday in Amsterdam on the 7th of December so if you guys can make it through for that event we would love to see you there. We will keep you posted about the details.

    Chaos Communication Congress, Hamburg

    Ben is also planning to be at the Chaos Communication Congress this year2 (December 27th – 30th, 2016 in Hamburg). We are still waiting to hear back whether he got accepted to give a talk there, but he is planning on going either way. And he was wondering who else was planning on coming and whether anyone would be interested in organising an Assembly with him4.

    SAFE Browser

    Last week we created a new Dev Forum topic6 to give feedback on SAFE Browser. @joshuef also created a topic with a general todolist of what he’s working on: Safe Browser Dev Updates (@v0.3.2)2.

    There's an updated release of the Safe Browser to 0.3.2.

    This features a more robust 'safemode' toggle, including a navbar icon showing that you are on the safe network exclusively. This can be toggled by clicking the icon, or going into the File menu. This will reload all tabs upon toggle, blocking/enabling insecure (non safe) content.

    Funding

    MaidSafe’s funding round20 on BnkToTheFuture ends in about 14 hours. £1,246,869 have been raised through 226 backers so far. Thanks again to everyone who invested, supported and helped out with this round!

    Bug reporting

    If you know which component a bug is associated with, feel free to report it on GitHub in the corresponding repo. If you’re not sure where to open an issue, you can simply create a new topic on this forum and add the #bug tag. @lightyear and I are subscribed to the #bug tag, so we will automatically receive a notification whenever someone adds the #bug tag to any topic. We’ll take care of opening an issue on GitHub if necessary.

    If an issue is specifically related to the SAFE API, then it makes sense to use the SAFE Dev Forum. But for issues related to end-user applications (e.g. SAFE Launcher, SAFE Demo App, SAFE Browser, etc.), it's fine to use the SAFE Network Forum. Most users already have an account here

    Also, to make the process of opening an issue on GitHub easier, we will start adding Issue Templates3 to most of our repos.



  • Top 8 Reasons to Invest in Maidsafe Shares

    1) ALL SAFE APPS THAT DO NOT ADOPT THE GPLv3 LICENSE OR USE THE SAFE APIs MUST PAY MAIDSAFE 1% OF THEIR REVENUE ALWAYS. I can't stress how huge this is. Once SAFE's network-effect takes hold, giant companies will be forking over crazy amounts to keep their source code private.

    2) MaidSafe is positioned to receive a large part of the 5% of ALL generated SafeCoins, that are set aside for core devs, for the forseeable futur (until other groups and programmers start helping in this area). That is a huge income. (They may also be among the largest farming groups early on when farming is most profitable)

    3) MaidSafe is currently among--if not The--top SAFE application dev teams in the world, as they created the world's first SAFE email system within a week or two. They could dominate the app scene with ease--especially early on, when SafeCoin sees its giant rises--if they so choose. Don't forget 10% of ALL generated SafeCoins go straight to the app devs.

    4) These stocks are LIMITED, AND MORE MAY NOT BE OFFERED AGAIN! They allow the possibility for future MaidSafe IPO with huge gains, or possible MaidSafe acquisition for giant gains from current tiny price point (of barely $1M USD), or large possible fiat/ SafeCoin dividends paid regularly.

    5) MaidSafe is a hyper-efficient company, as their systems scale infinitely, disrupt companies with billions in profits (and billions in equipment expenses), but has no infrastructure costs of its own. That sets the stage for insane profitability.

    6) Even just the current Alpha release is enough to be licensed to large data centers for profit, as it would eliminate data losses during server crashes. So I know we all aren't happy it isn't 100% finished being fully decentralized yet, but don't overlook the technical marvel of what a networking achievement it is, with its vaults and routing.

    7) Finally, last place is the original USB sticks straight from MaidSafe, and live SafeCoins sent straight to you at launch, both of which could become extremely valuable. Also an amount of free BTC every day from BNK (so the $1k is offset by all this).

    8) oh &..............Con$$$ulting. Period.

    So anyone who believes that SAFE will be successful at all would be foolish to not invest anything into the company that is in the best position to profit hugely from it!!

    (not legal advice etc) (also I made this shortened version of @fergish 's podcast to focus on the parts that talk about profitability. It's the one I keep referencing above.)

    Register & Invest!



  • Safe Browser Dev Updates (@v0.3.6-2)

    0.3.5

    Download:
    https://github.com/joshuef/beaker/releases/tag/v0.3.5

    New pre-release is available here: https://github.com/joshuef/beaker/releases/tag/v0.3.6-0 with updated error handling and more promisey base APIs

    Todo

    Here's the general todolist, vaguely in the order in which I'll attack them:

    SAFE Mode Toggle button/indicator. So you know you're safe.
    Straighten out Appveyor deployfor naming and to only have one link.
    - Linux/Windows safe: protocol integration, so safe: links will (optionally) load the browser
    Investigate localhost development... what can we do here to facilitate this
    - Add launcher detection (helper in safe-js), and appropriate notification
    - Safe network sync for browser data.
    - Add 'You can access the resternet' type screen when in SAFE mode. Instead of ugly broken screen.
    - Setup some helpful default 'favourites' for safe (suggestions welcome, I'd probably start with email [as per @polpolrene's suggestion] and a site list )
    - Investigate beaker shortcut issues.
    - (almost ) update package naming to match maidsafe standards
    - rename windows unpack folder
    - ensure production environment used for builds & remove BeakerDev menu

    safejs specifics

    improve safejs promise error rejection for DNS/NFS/Auth
    - improve safejs promise error rejection for new APIs
    - improve safejs test coverage
    improve safejs response parsing
    improve safejs handling of streaming APIs and html5 streaming




  • Maidsafe-EASIEST API TUTORIAL website:




    It's this new SAFE API Tutorial Website

    Logical extension from APP ZERO. I want it to be extremely approachable, and eventually have tutorials and walkthrough code for anyone creating web apps with safe-js. (the site will ultimately be SAFE-hosted, once the examples work, so that people can click the boxes to make the commands actually execute with launcher)

    Auth was covered by APP ZERO, but it looks like creating the basic folders is the logical next step, and a bit confused by the example code both on the MaidSafe website (bare-js? didn't work) and safe-js gh readme.

    tl;dr NEED HELP WITH CREATING DIRECTORY IN SAFE-JS so I'll tag @joshuef @DavidMtl (did you get your SimpleSafe working?) @happybeing and anyone else can help, submit PR's etc (site repo, & code repo)



  • New authentication flow + Granular permission control

    Today, we are presenting the RFCs related to the new authentication flow that was first explained in the topic about the future of SAFE Launcher.

    New authentication flow

    The current authentication and permissions handling flow relies on SAFE Launcher, a stateful intermediate between any particular app and the network. While this allows for fine-grained control over the access flow and user setup, it has the drawback of requiring the process to always run, keep a local state and proxy all network requests. This is particularly cumbersome on embedded devices and mobile, where we cannot provide this pattern reliably.

    RFC 46 – New Auth Flow proposes to replace SAFE Launcher with SAFE Authenticator (a stateless alternative).

    This RFC outlines a new process to give applications authorised access to act on the SAFE Network with the user's credentials. In particular, it is designed for the mobile and embedded use case in mind.

    Highlights

    At the core stands a rethinking of the data flow and authentication responsibility. Rather than requiring every app to go through the launcher to use the user's credentials, in this approach, the user creates "sub-credentials" per app and assigns them to the app. The app can then access the network with these credentials independently from any other process running. We call these credentials appKeys (as they are a public-private-key-pair attached to a user's maidsafe keys), which an app can interact with the network directly with.

    Apps will have to directly interact with the network. We will restructure safe_core to provide all needed facility for that, as well as the proposed authentication flow against the authenticator. That means that each App has to bundle their own safe_core and link to that library. The library will expose all necessary functionality also through FFI to allow for binding in other environments than rust.

    The root container is the main entry point for the user and most apps to interact with. It is a locally encrypted container stored at a random location on the network known only to the user, that generally only the authenticator has write access to. It reference will be stored in the user’s session packet on account creation. Keys starting with an underscore (_) are reserved for internal usage by the authenticator, while the authenticator may also allow the creation of other keys later.

    When creating a user account the authenticator will create the following minimal set of containers within the root container, each one with its own random network address:
    • _apps/net.maidsafe.authenticator/
    • _documents
    • _downloads
    • _music
    • _pictures
    • _videos
    • _public
    • _publicNames

    It is recommended that apps should only request the rights read and insert - which we will refer to as the BASIC-access. To promote the usage of this, APIs should provide a short-hand for this. Furthermore, the Authenticator may require a multi-step authorization from the user if it encounters an app that asks for more permissions than these and may allow the user to disable specific permissions asked for.

    Discussion topic for RFC 46 – New Auth Flow1


    Granular permission control

    Existing data types don't support granular access control which is required for mobile platforms where access to the network through an intermediary like SAFE Launcher is complicated: each app would have to receive full permissions to modify and fetch data on a user's behalf and private keys to sign it. This compromises security and allows malicious apps to get full access to the user's data.
    To mitigate that, MutableData provides fine-grained access control and more narrow-scoped data operations. This data type is an evolved version of AppendableData. It combines StructuredData and AppendableData features and improves the network efficiency, saves bandwidth, and provides a standardised approach for apps to fetch and mutate data in the network.

    RFC 47 – MutableData proposes to combine StructuredData and AppendableData into a new single data type, MutableData, similar to HashMap.

    <code>pub <span class="hljs-keyword">struct</span> MutableData {
        <span class="hljs-comment">/// Network address</span>
        name: XorName,
        <span class="hljs-comment">/// Type tag</span>
        tag: u64,
        <span class="hljs-comment">// ---- owner and vault access only ----</span>
        <span class="hljs-comment">/// Maps an arbitrary key to a (version, data) tuple value</span>
        data: BTreeMap<Vec<u8>, Value>,
        <span class="hljs-comment">/// Maps an application key to a list of allowed or forbidden actions</span>
        permissions: BTreeMap<User, BTreeSet<Permission>>,
        <span class="hljs-comment">/// Version should be increased for every change in MutableData fields</span>
        <span class="hljs-comment">/// except for data</span>
        version: u64,
        <span class="hljs-comment">/// Contains a set of owners which are allowed to mutate permissions.</span>
        <span class="hljs-comment">/// Currently limited to one owner to disallow multisig.</span>
        owners: BTreeSet<sign::PublicKey>,
    }</code>

    The current features of Appendable Data will be covered by Mutable Data, but rather than just append, Mutable Data will support many discrete actions (e.g. “List entries”, “List keys”, "List values”, “Get value”, “Insert”, “Update”, “Delete”, etc.). These discrete actions will mutate specific key-value pairs directly on the network, rather than having to replace the entire content. Consistency and permissions will be enforced by the vaults. Mutable Data will also allow to create multiple type tags (just like Structured Data).

    The reason for this change is that we want to provide granular permission control over data changes within the network (rather than on the client side). Many things, that so far happened in the launcher (so we can develop faster) but should be done by the network (like stronger enforcement of permissions), will finally move into vaults.

    Secondly, we propose to introduce a new paradigm on the data type itself to handle permissions. Expanding from the owner-or-public-model, we propose to have field on the data type (called “permissions”) which maps signing keys to a list of actions that a key is allowed to perform on a particular data object. Every discrete action is checked by the vaults against the permissions assigned to that key and will only succeed when the required permissions are present.

    Highlights

    The proposed data type defines discrete actions on data that doesn't require a user to replace or fetch the entire structure contents. This is enabled by functionality similar to this of the HashMap data structure or a key-value database such as Redis.

    Permissions can be modified granularly by issuing actions similar to those used to update data. Only owners and apps having the ManagePermission permission must be allowed to issue such requests.

    Pending writes must support granular changes in data or permissions. Ownership transfer and encrypt key changes also must be represented by separate pending writes.

    Limits

    The MutableData data type imposes the following limits:

    • Maximum size for a serialised MutableData structure MUST be 1 MiB;
    • Maximum entries per MutableData MUST be 100;
    • Not more than 5 simultaneous mutation requests are allowed for different keys;
    • Only 1 mutation request is allowed for a single key.

    These numbers were picked as a starting point. They will be profiled and updated via testnets as the performance of the network improves.

    Discussion topic for RFC 47 – MutableData1


    Authorize apps on behalf of the owner to mutate data

    If the requests made by the permitted apps need to be charged these must all go through the Maid-Managers and they should know about the owner that an app is associated with so that they can charge the correct account.

    RFC 48 – Authorise apps on behalf of the owner to mutate data details how apps route their mutation requests to Data-Managers and how revocation works.

    Highlights

    We require all mutations to go via the owner's Maid-Managers. Maid-Managers will keep a list of sign::PublicKey which is the representation of the apps the owner has allowed to mutate the network on their behalf. Maid-Managers will then forward the request to the appropriate Data-Managers. The type MutableData thus does not need to store any signatures and Data-Managers will use the group authority to trust the requester of the mutation. Then they will use the permissions field to allow or disallow the mutation itself.

    Ownership transfers are a special case. The apps are not allowed to do this operation. They will need to ask the Authenticator to do this. Maid-Managers shall enforce that change_mdata_owner only comes from the owner. Data-Managers shall enforce that current owner of the data matches the incoming Maid-Manager group.

    Once the owner decides to revoke permission of an app to mutate network on their behalf, they will go to the Maid-Managers and delete sign_pk for the app. Hence the app can no longer perform any mutating operations on behalf of the owner now as the owner's Maid-Managers will reject it not finding it in the list.

    Discussion topic for RFC 48 – Authorise apps on behalf of the owner to mutate data



  • MaidSafe Dev Update - February 2, 2017 - Test 12

    This will enable some users to run vaults from home. This test doesn't contain the new client API (with mutable data). It's purely for testing the features related to the routing
    update (mainly disjoint sections and resource proof). With such a long
    period between testnets and a considerable upgrade to the codebase, this
    test is essential for us to confirm that the network is improving and
    that we are on track for the remaining features, which required this
    change. The next target for us in terms of features is data republish
    and increased security (node ageing and data chains are large parts of
    the current in house discussions).




    As part of the resource proof paradigm, a spot check for bandwidth is carried out when you start a node. This may extend to CPU at a later stage (this is already included as part of the resource_proof4 library). Your vault will connect to the members of the network's section that it is trying to join, and will try to upload some verifiable and unique data to each of them within five minutes. If successful, the section will accept your node and it will start routing and handling requests.

    Note that each section will only profile one node at a time, so if many users are starting vaults at the same time, many of them will need to retry multiple times until they are accepted as a candidate and start the bandwidth check.

    With the Node Ageing RFC4, the bandwidth parameters will adjust in real-time based on the network requirements. At the moment, we are using "magic numbers" that we have selected through testing to do these spot checks. We have made these joining node resource checks extremely aggressive to begin with. A large percentage of the community vaults (maybe 50%) may not be able to join Test 12. This affects only the vaults though, clients will not be affected by these spot checks.

    Right now, the upload bandwidth requirement is ~6 Mbps. In the long run, we totally intend to make use of every node we can (e.g. for particular specialized tasks). This is just step 1 of the process.

    This test represents a huge refactor and we are very keen to confirm that there are no regression or surprises as we move forward towards Alpha 2.

    SAFE Vault v0.13.0

    You can run only 1 Vault node per LAN – running more than 1 will result in this message:

    More than 1 routing node found on LAN. Currently this is not supported

    Download

    Changelog

    • Migrate to routing 0.28.0.
    • Use a single event loop for routing and safe_vault.
    • Fix issues with account creation and data requests.

    SAFE Launcher v0.10.1

    o connect to Test 12, you need to use SAFE Launcher v0.10.1.

    Each client account is limited to 500 PUTs. A PUT is a request to store a single chunk on the network (a file may contain many chunks). The maximum chunk size is 1 MB.

    Please be aware that we might need to restart this test network, wiping all data that is stored on it. We will announce this in a Dev Update (when possible) prior to taking it down.

    Download

    Documentation

    Changelog

    • The release binaries uses safe_core v0.22.2 to connect with the Test-12.
    • Config file updated to connect with Test-12

    SAFE Demo App v0.6.2


    safe-demo-app.png2244x1624 194 KB

    The latest version of SAFE Demo App (v0.6.2) should continue to work fine.

    When uploading files using SAFE Demo App, the maximum file size is 25 MB per file.

    Download

    Documentation

    SAFE Browser v0.4.3


    safe-browser.png3824x2310 238 KB

    The latest version of SAFE Browser (v0.4.3) should continue to work fine.

    Download

    Other apps

    Apps that were working with TEST 111 should continue to work fine with TEST 12.

    Support

    If you need help with anything related to SAFE Vault, SAFE Launcher or SAFE Demo App, please use the #support category of this forum.

    Where should I report issues?

    If you know which component a bug is associated with, feel free to report it on GitHub in the corresponding repo.

    SAFE Authenticator & API

    Team leader: Krishna

    We made good progress this week resolving the issues we had with the FFI integration. The authenticator plugin1 is now integrated with the safe_browser3 and tested. The initial integration looks good and the tests we have carried have exposed a few bugs and corner cases to be handled.

    Also, the Node.js API (for app developers) has seen some considerable progress. We now have the majority of the APIs tested manually. The File-related APIs have some integration issues and we are looking into those at the moment. Along with fixing the known issues, we are also expanding the test suite and examples for the Node.js API.

    You can take a look at the safe_app_nodejs5 repository to get a better understanding of how the Node.js API can be used. You can only try it on Linux and macOS at the moment. For Windows we need to finalise the build process. We are hoping to continue the steady progress in the upcoming week and announce a stable API for the next update. It's worth mentioning also that we started making plans for the documentation. For the Node.js API, we will likely use documentation.js2 (which uses the JSDoc syntax2).

    SAFE Client Libs & Crust

    Team leader: Spandan

    Carl completed the Mio2 task to provide and test iOS and Android compatibility. We are looking to update Crust to the latest version of Mio. We are currently debugging some bugs/changes in behavior with the latest version of Mio on Windows. We think the bugs might be in Mio, but this is not confirmed yet. Once this is sorted out, we should be good to merge Crust's port to the latest Mio. Carl is again helping out with those bugs.

    Right now, Crust only supports TCP. We are going to work with Carl to support other protocols (e.g. uTP) and improve hole punching as well as some security features we require for "man in the middle" attack avoidance.

    The front-end integration with safe_client_libs is the current workload of the rest of our team and this is being done in parallel with Krishna's team to complete any remaining issues with the new APIs for developers.

    Routing & Vault

    Team leader: Andreas1

    We fixed the remaining issues for the bandwidth resource proof and the feature is now ready for a first test with the community.

    We are also having lots of discussions and meetings about our medium/long-term plans, both technical and organisational.

    Team leaders

    The team leaders (Krishna, Spandan and Andreas) are currently in Troon, Scotland for the next 2 weeks. This is the first time that we have the department heads, managers and local dev team all in one place! We're using the opportunity to discuss: the company roadmap, community engagement (of course ), documentation (including the website), as well as some ongoing partnership opportunities, including our workshop with Glasgow University2 which is due to commence in early March. Being the gracious hosts that we are, we will of course show them the local sights, which in Ayrshire boils down to the interior of one of the many local pubs. 

    Developer outreach

    Nikita (@nbaksalyara local DEF CON group's mini-conference4.

    Also, he recently got accepted in the Mozilla Tech Speakers3 training program. He hopes it will give him more experience for presenting at large events. 



  • MaidSafe Dev Update - February 23, 2017

    So this week had a few surprises in store for us, some good and some not so much in terms of Test-12c. Firstly, we completed the routing patches (that we mentioned in the last dev update) on Friday and early this week, we also had the routing soak test run about 1000 iterations successfully. These are tests that use the mock crust feature to simulate different types of network setups and specific cases we'd like to test. This passed all the way through and was great news.

    We then moved on to testing the production vaults in droplets with internal tests. This was where the not so great news occurred. We hit a few issues very early on that we patched, however, we continued to see a few more issues through the week. It's worth noting that each issue that needs to get debugged and patched takes a bit of time since this involves collecting about 1800 log files and spotting the source of the issue, (fun times . The Routing team went at it hard (so hard that we've made Andreas and Diggory sick) to try and get these resolved ASAP. This was further complicated with the new Crust version we were using in the live network, and involved us running two internal tests to confirm Crust wasn't the variable here (it certainly wasn't, so looking good from that side, more good news from Crust coming up).

    Now for some more good news, we've currently had an internal test network running since yesterday afternoon that has gone through about 2100 churn events. That translates to about ~3500 nodes joining and leaving the network. The internal tests also test some particular corner cases such as lots of merges/splits happening in the network at the same time as nodes joining/leaving. As of writing this dev update, that network is retaining data while the invariant is holding, this is certainly great news. All is not completely done yet though. We have one other issue we've spotted that can occur when a candidate (new node going through resource proof check) is dropped by a section after it splits which in an edge case prevents the split section from ever re-connecting to that candidate. The bug meant that the section did not get rid of the candidate properly after the split which prevented them from even re-establishing the connection to that candidate (who by that time might have been promoted to a full node). This is certainly an issue and @qi_ma is patching this today. We've not run into this edge case in the network within the last 24 hours.

    Once we have the patch for this in place, we intend to keep repeating the internal tests until we can confirm a bug-free network before progressing to the most elaborate test of user run vaults (Test-12c) From the results this week we're hoping that we're not far away, but please do take that with a grain of salt as this situation is very fluid as you can imagine.

    With all this happening at one side this week, we've got some really good news to share from some of the other teams, specifically the progress being made in protocol support from Crust for uTP, and various hole punching tests that we've tried in combination (all this will increase the number of people able to join the network and not just rely on TCP). With frontend APIs getting more reliable, @ustulation and team are also starting their work in Vault to support — yep, you guessed it! — mutable data This will enable the test networks to move from the SD/AD data types, introduce the new data type MD which will then facilitate the Authenticator paradigm in place of Launcher, which would get deprecated at that stage. Just to note this isn't a patch job that we expect to be done with in a week, but we're certainly in motion there to get vaults to support this data type. We've also got the frontend guys going hard at their applications and docs which we intend to share more with you this week. Please check the individual module sections for more information in all these aspects.

    As you already know, we have some of the team in Asia which is great, but as a small team it does hurt dev progress a little back home. So this balance is a fine one that we choose to make, even though it does add a lot of time pressure. We feel sure though that getting the SAFE name out and solidifying relationships with our Asian partners will benefit the SAFE Network and all that is stands for in the long run.

    So in summary, it's been a pretty hectic week all around, got a few people ill (looks like they'll survive just fine), got some of us in Asia spreading the SAFE Network concepts, patching some network bugs from internal tests and progressing along with the MD implementation and Crust updates for protocol support and doc updates of frontend APIs.

    SAFE Authenticator & API

    Last week, @happybeing (again we must thank him for his amazing contributions to this project) had hosted the API documentation for safe_app_nodejs for easier access. We have now published the documentation17 for safe_app_nodejs' master branch, which we will keep up-to-date with the latest changes.

    We're currently improving the build process for the Node.js library and also the safe_browser. The library will be downloading the native dependency at run time and won't have to install Rust and build locally to get the native libraries. @lightyeardependency downloader8 which is a postinstall tool for safe_app_nodejs to download the native library for the current architecture and the platform being installed on.

    @lightyear@shankar have been working on packaging the example app4 – which depends on safe_app_nodejs – using Electron Forge5. This is tested on macOS and we are currently setting it up for use on Windows. We hope to have it integrated soon and test the same across all three platforms. Minor feature updates to the example app have been implemented by @shankar.

    Once we have these tested across platforms, it will make the installation of the safe_app_nodejs library seamless. The browser build process will also be simplified with this WIP PR3 getting merged.

    @bochaco@bochaco is working on identifying the helper functions which can be moved to the Rust FFI layer so that the integration with FFI will be easier across languages.

    SAFE Client Libs & Crust

    Quite a few good things are planned for Crust and Rust networking in general. Carl1 having solved the issues with the internals of Mio, is planning to work on an async uTP support on top of Mio. We had given him a few pointers to where the protocol was itself, which is implemented in C++ as libutp, and had asked if he could come up with a list of TODOs. He has given us a rough idea of what API he plans to expose in this new crate. We were recently discussing on having a coded multiplexer in the uTP listener logic that functions on just one socket vs spawning multiple sockets per accept. The general consensus is to spawn a new socket per accept instead of using multiplexers. Although multiplexer would have helped us keep number of used descriptors under control, they would have a significant overhead in terms of CPU cycles and could potentially affect performance. Once we have uTP support, we will certainly be much better off in terms of reaching peers. UDP hole punching trumps TCP by a good factor and uTP wraps a UDP socket with reliability and congestion control layers added on top of it.

    @nbaksalyar in the meantime has been writing more test cases for Crust and looking into the challenges of setting up a virtual network within a machine to test external reachability and other parts of code, which necessitates peers on non-loopback addresses thus posing challenge. While there are ways on *NIX platforms, Windows as usual needs more looking into and out of the box support seems bleak there. If he does figure it out, it might prove to be highly useful for tests in Crust because they would be closer to the real world and will help us iron out deficiencies if at all observed.

    @adam started working on the implementation of MutableData in vaults. We already have this supported in safe_clients_libs via mock vault. Based on the churn handling in the vault library which isn't catered for in the mock, we might have to optimise or revisit certain components of this implementation. MaidManagers should be easier to deal with than DataManagers just now, and that's where we have started. Every successful mutation request now would deduct account balance (previously POST was free and went straight to the DataManagers). Also the final cap is based on total number of mutations, not just PUTs as previously was the case. See this post for more info.

    @ustulation had intermittently done some NAT-traversal testing with different kinds of routers including Symmetric NATs and observed some expectedly unfriendly router behavior. NAT traversal is a cat and mouse game in which we need to find out and try and trick through the weird behavior of routers. All the observations until now are documented in detail with workarounds for successfully reaching peers in this repository13. This is meant to be an independent pluggable crate for robust p2p communication. This is a work-in-progress and current design (and other) details (which are aimed to explain all observations and techniques thoroughly) can be found in the crate docs5. Though still in early stages, a somewhat working chat-engine is already written and tested here9.

    Routing & Vault

    As mentioned in the introduction, this week has mainly involved us patching and updating the routing library based on internal tests. Some of the changes that have been applied this week include:

    • #1357 adds a test with an increased level of churn.
    • #13631 makes the log messages a bit more concise.
    • #13702 fixes problems that occur if a candidate (a joining node doing the resource proof) joins a section that is currently merging with another one.
    • #13714 addresses another section merge bug.

    Developer outreach

    This week's Tech Conference in Jakarta was a great success! See this blog post29 by @nicklambert for more info.

    Also, Ben (@lightyearBob Conf Berlin7 tomorrow. His talk is titled "Privacy-first data structures on SAFE4". We hope it's as good as the excellent talk he gave at FOSDEM 201711 



  • MaidSafe Dev Update - April 13, 2017

    Everything that we had as objectives for Alpha 2 has been met with Test 164. But based on what we've seen from Test 16's limitations, these limitations were the main things that were abused in various ways (e.g. nodes bypassing the relocation process, clients spamming the network with GET requests, nodes having a lot of tunnels). As @dirvinethis post13, these are all things that are currently just spot checks and that were put in and highlighted as limitations that would be resolved down the road with features like node ageing and data chains. So features that were meant to be for Alpha 3 are limitations right now, because we haven't implemented them yet, but those are exactly the things that would solve the kind of issues we experienced in Test 16.

    We have discussed the idea of helping set up a community testnet, it's a double-edged sword though. Test 16 did not show any bugs and, barring the attacker's participation, was fine and would be Alpha 2. We are not sure what can be achieved even with a community testnet right now. We know vaults from home work, we know the limitations and we know devs really want to use the new SAFE API with Mutable Data.

    With this in mind, and instead of delaying Alpha 2 and taking features from Alpha 3 and rushing them into implementation now, we're now going to swap the objectives of Alpha 2 and Alpha 3. We have done the donkey work here, mostly, but need to spend a few short weeks at least soak testing and finalising the mutable data integration to a point where we are happy to release the next testnet. We will try to speed through this with half the team while the other half are already on the routing & vault updates we need for Alpha 3. Alpha 2 will also affect the Alpha 3 code but it's pretty clear where and we can easily cope with any conflicts there. However, to ensure we do not knee-jerk this, we will measure the results carefully and with luck we should do this internal testing over the next few weeks and of course update you all as we progress.

    Alpha 2

    • Mutable Data and authenticator pattern (with the new SAFE API)
    • SAFE Browser DOM APIs and Node.js SDK along with example apps, tutorials and documentation

    Alpha 3

    • Vaults from home (more people should be able to run more nodes and on more platforms)
    • Node Ageing
    • Data Chains, Part 1

    This means that the upcoming testnets will focus on Mutable Data. Like Alpha 1, these will be client-only networks (MaidSafe will run all the vaults). We hope these testnets will trigger a lot of discussion and feedback from the community in terms of how to use the new APIs, how to migrate from SAFE Launcher to SAFE Authenticator, what are things that might need to be addressed or done better, documentation, supporting different platforms, etc. It should be an exciting period!

    Alpha 3 is going to have everything that the vaults had with Test 16 (what we currently have now), plus the missing features without the hacks. We can add temporary hacks here and there, but this is not something we feel is good in the long term of the project and there is not much we would gain out of doing this right now. Instead, we're going to go ahead and implement the actual features that are planned for Alpha 3 in terms of data chains, node ageing2, nodes key as name5, etc. But we're not going to wait for that to be complete before allowing app devs to start using the new SAFE API with Mutable Data, which is why the next testnets will focus on Mutable Data. Mutable Data support will be added to the master branch of routing & safe_vault and, in parallel, we'll implement the features for Alpha 3 in development branches.

    Once the Mutable Data objectives are complete and at a stage where we consider the API stable enough for a "version 1", and if we are not done with node ageing and the first part of data chains (which might be likely), then we will call that the new Alpha 2. Once we are done with the original Alpha 3 objectives (as far as vaults are concerned), we will release Alpha 3 and community members will be able to run vaults from home again. Alpha 2 and 3 may appear very close together in this case.

    Beyond Alpha 3

    • Data Chains, Part 2
    • Data republish and network restarts
    • A security audit of the network
    • Test safecoin
    • Real-time network upgrades
    • Network validated upgrades

    In between alpha networks, there will be many testnets. Each alpha will focus on a few major components. Major features will be added one by one in testnets. Once all those things are in place, there may be other alpha networks, or we may move to beta.

    Upcoming Dev Forum discussions

    We are going to have quite a lot of technical discussions on the SAFE Dev Forum4 over the next few weeks. This will involve discussions from the Mutable Data side and also from the back-end network side (e.g. data chain integration into vaults).

    Here are some of the topics you can expect to see on the Dev Forum in the next few days/weeks:

    • Data Chains (Part 1) design proposal
    • Vault chunk store optimisations (as pointed out by @mavthis topic7, which has lead to @qi_ma100 entries limit8 of Mutable Data
    • Operation recovery for Mutable Data from the network. So a transaction-based process (for some operations initially) to allow rollback in case of failed operations midway (this one is likely going to take a while longer as the guys are still in the early stages of working out the details)

    These will all be open discusions, but at the same time, we will need to consider what is going to work well for the project in the long term and decisions will then need to get made accordingly. To be clear though we will still have ongoing and possibly more blunt debates in house, these are critical to achieve the best outcomes but not always good for public consumption as there would be too many ways that people who would wish to cause harm could take this kind of frank discussion out of context. Any conclusions though would immediately be made public as you would expect and deserve.

    SAFE Authenticator & API

    We finished porting the Markdown editor web application and email application to use the new APIs, but we haven't merged the code to the safe_examples1 repository yet because of an API issue related to Mutable Data. The test cases for Mutable Data are passing, however the API fails when invoked from Electron. We created a CLI sample app to test the Mutable Data APIs and can confirm that the APIs are working fine from the Node.js environment. We are looking into this issue and will have it resolved soon. Once the issue is fixed, we will be able to get the updated examples merged to the safe_examples repository.

    @Kumar tested a few approaches for Java JAR packaging and distribution. He is also trying to get a simple FFI sample tested on Android. JNR6 seems to be a problem on Android. If JNR is not going to work we will switch to JNI4. We would like to have a working sample before starting to implement the actual code for safe_app_java. Ideally, it would be easier if we could use the same source code for both Android and Java desktop apps.

    SAFE Client Libs & Crust

    We are looking into viable solutions to prevent GET and MUTATION spam on the Network. GET spam can be caused by either registered or unregistered accounts, where a (malicious) client makes repeated GET requests of large chunks of Mutable Data (Immutable Data is cached so the benefit to the malicious client is not that huge) and makes the Network do work for them with relatively no effort on their part. It can also be GETs to non-existent data, which will result in failure response but still cause the Network to process such requests. MUTATION spam, while not as obvious as GET spam, can also cause the Network to do a lot of work. In this case, frequent bound-to-fail mutations can be done. As an example, consider a PUT request for a chunk of data (largest allowable size) that already exists on the Network. It will travel to the MaidManagers who will deduct account balance, forward it to the DataManagers who will complain that data exists, MaidManagers will undo the balance deduction on getting failure from DataManagers and will finally relay the failure back to the client. Thus, currently, without spending any balance it is possible to make the Network do a lot of work (thus a spam attack vector). Also, invalid mutations can be done which get rejected at the MaidManagers themselves (e.g. mutations by someone who has run out of account balance or by an unregistered client). Though not as severe as the previous example, they will still cause some level of spam.

    The problem and some of the solutions are being discussed here9. We have some more ideas to eliminate these issues and we'll continue to discuss the merits of each before putting out a proposal, but it seems a pretty solvable issue.

    Then there is the cap of 100 entries for MutableData entries. In the current paradigm we do not actually delete entries but merely blank out the value of the key-value part, and this still occupies space and counts towards the capacity limit (as it needs to be refreshed in churn etc. and treated just like other data). We are also thinking on how to solve this and once we have a proposal we will share it with you.

    Apart from that, there were some bug fixes (e.g. this one2) in safe_client_libs in the dev branch as the front-end team continues to test against it.

    Routing & Vault

    We added section versions to routing that increment whenever a section splits or merges. This simplifies some message flows and already allowed us to get rid of some types of messages, just because it makes it trivial to determine whether the information in a message is more current than the one we already have, or whether it is outdated and can be ignored. It is also a first step towards data chains, which will produce an independently verifiable record of the network's and the data's history.

    In parallel, we have been working on the next step in that direction (Data Chains, Part 1), which will provide a log of all section changes, but not yet data changes. I.e. it will not only version sections, but also produce cryptographic proofs for them, and it will not just include splits and merges, but also additions and removals of nodes.

    We are finally taking the last few steps towards deterministic tests2.

    @qi_mathis topic5, we can hopefully analyse the findings and pick a suitable solution to move this forward.

    Finally, in some other news, Diggory (@dhardy) is off from next week. We'd like to thank him for the time he spent at MaidSafe, he's been of great help in the Routing Design team and in the production cycles as well. We are close to closing some additional positions in the Engineering teams so again there will be more new faces to get to know 



  • MaidSafe (MAID) Dev Update

    we are publishing different proposals related to the first part of data chains:

    Data Chains - Option A

    Data Chains - General rules in an eventually consistent network

    Data Chains - Option B

    Please note that we have not concluded on any of these proposals. They are all still being considered.

    Also, we'd like to share an idea that we are considering for the next testnet with MutableData. Currently, every time we start a new testnet, all user data is lost and people need to create a new account and start from scratch (e.g. you need to reupload your data and recreate your SAFE websites). The real solution will be to implement data republish and network restarts via data chains, but since we are currently running client-only networks (where MaidSafe manages all the vaults), we are thinking that it could be a good idea to implement a temporary solution to make user data more persistent. So if a network is taken offline because we are adding new APIs and vaults need to be updated, we wouldn't have to lose all the user data. This means that app developers won't have to reupload all their apps every time we start a new testnet  It also means that we'll have to start caring more about backward compatibility (e.g. deprecating APIs, migrating from one version of the API to another version, etc.).

    There are a few simple approaches to achieve this, while none of them would be the ideal long term solution, they should let us get the client tests going and allow network updates without much of a hassle in terms of data loss during restarts. Most approaches involve allowing nodes to retain their previous ID and restart to the same area (address space) of the network. In the long term with node ageing, while nodes will still try and do the same, they will get relocated again at this point. For now however, we're considering letting nodes just remain where they were which will then get their chunk stores relevant again and data available to continue operations. It involves keeping the chunk store persistent and also flushing MaidManager accounts which are currently not flushed to disk. Doing the last two parts could be achieved again in multiple ways such as using a custom network RPC to trigger and create a named backup which can later be restored to or go for a simpler option of just having the single backup by not dropping chunk_store on termination/startup. Each approach comes with its own set of advantages and we've not finalised an approach just yet.

    SAFE Authenticator & API The safe_browser's DOM APIs were further simplified for usage. Previously, for every API call, the auth token had to be passed as an argument to the API function. Now, we are internally mapping the handle with the corresponding client, so we can remove this unnecessary argument. APIs to free the handles are exposed as part of the DOM API. Also, the for-each function was not working consistently, but this is now fixed. A new version of the SAFE Browser with the updated APIs and fixes was built with the mock network and published for the devs to try. @bochaco also documented the DOM APIs and we have hosted the same on GitHub Pages for easier reference.

    The safe_app_java APIs are being wired up and a few test cases of ImmutableData are failing when run as a complete test suite. We have
    wired up most of the APIs and only the MutableData API is pending. We are hoping to resolve the issues and have the Java API ready as soon as possible.

    @shankar is also getting the UX/UI of the authenticator improved based on the designs18 @shona.

    SAFE Client Libs & Crust

    SAFE Client Libs have been updated to use the latest library versions (including Serde 1.0) and now we have replaced SHA-2 everywhere with a more efficient hashing algorithm, SHA-33. Some changes are being made to the dev branch of safe_client_libs to improve the public API consistency further: now we have a function to generate cryptographic nonces (generate_nonce), which makes the crypto module available to apps more complete. Further, working closely with the Authenticator team, we have closed the gap by fixing inconsistencies in some of the modules: for example, mdata_info didn't have a proper function to deallocate it, and now there is mdata_info_free.

    After updating safe_client_libs master to use the latest version of the serialization library Serde, we ran into some issues with the existing invitation-based account creation scheme. It was causing some weird out-of-memory errors, so we have changed the stored account information to be more strongly defined and put it into a common module in routing that soon will be used in safe_vault too.

    The stress test from the master branch of safe_client_libs has been ported to the dev branch to include support of MutableData. This will help us to test the robustness and performance of the MutableData implementation in safe_vault. Then, based on the results, we can see where improvements are needed.

    Routing & Vault

    While the design and simulation efforts are still ongoing, we decided to make public some of the documents and proposals today and invite you to participate in the discussion:

    Why data chains?

    There are a lot of decisions the network needs to make as a whole: Which nodes are allowed to join, and in which section? Which nodes are responsible for any given part of the namespace, and in particular for which data chunks? What chunks are currently stored on the network and what is their content?

    Eventually, after having exchanged a few messages, the nodes always need to reach an agreement on all of those questions, so that they don't send contradictory responses to clients, and so that they agree on which sets of nodes are responsible to make the next bunch of decisions. In other words, both the data itself and the information about the network's own structure are supposed to be eventually consistent: the nodes must always, although possibly after some delay, reach a consensus on them.

    The goal of data chains is to be a historical log and a cryptographically verifiable representation of that consensus, that can be persisted and exchanged as a proof among nodes.

    As a first step, we are only aiming to include the information about the network's own structure (which nodes are members of the network, what is their name, and which sections and groups are they in). Even without data, that will already have considerable benefits:

    • With a more well-defined notion of a proof that a section consists of a particular list of members, we can at last enable the final piece of the puzzle for securely relaying messages2: Each message will contain the required blocks that allow the recipient to validate an uninterrupted chain of signatures back to the sender.
    • Clients and nodes joining the network will be able to verify that they are connected to the real network without having to trust their proxy node.

    Events in the network

    Since the Disjoint Sections RFC1, the network is subdivided into sections defined by a prefix, a finite sequence of ones and zeros. The section with a given prefix consists of exactly those nodes whose names in binary begin with that prefix. At first, the network has only one section. Once enough nodes have joined, it splits into sections 0 and 1. When section 1 has enough nodes, it splits into 10 and 11, and later into 100, 101, 110 and 111, and so on. If a section loses too many nodes, it merges, effectively undoing a split.

    Nodes that join the network are assigned a small interval, and must pick a name within that. They are then challenged by the section they are about to join, to prove that they satisfy some minimum bandwidth (and later CPU) requirements.

    So the main events that are represented by the blocks of the chain are:

    1. A node has passed the challenge and joined the network.
    2. A node has disconnected and left the network.
    3. A section has split into two.
    4. Two sections have merged into one.

    The problem is that the existing nodes in the network don't always agree on which of these events have happened yet or should happen:

    1. A joining node can pass the challenges it got from different section members at different points in time. And it could even only pass some of them but not all.
    2. A node could leave because of connectivity problems, and disconnect from its peers one by one, with several minutes in between.
    3. A node could believe that its section should not split because it will soon have to merge with another section that lost too many nodes. Another node could have received an update on that other section earlier or later, and thus come to a different conclusion.
    4. And similarly for merges: I might think that we, section 01, need to merge with section 00, because 00 has lost too many nodes, but you might already know that 00 has in the meantime regained a node and we don't need to merge at all.

    So our data chain implementation will need to start out from those different subjective views of the network, before arriving at an agreed "truth", usually by making the nodes vote: "I think this node should join our section.", "I think that node has disconnected.", "We should split!", etc.

    Different approaches to a solution

    Initially, most of the team's time went into the Data Chains - Option A87 plan: We are trying to adapt established consensus algorithms like PBFT6 to our use case. Each section of the network would run an instance of that algorithm, and special care needs to be taken whenever the section changes, because a node joined or left or the section splits into two, or merges with another one.

    Whenever an event happens, e.g. a node goes offline, the other nodes exchange several rounds of messages to reach an agreement that this event should define the next block in the section's chain, and then commit that block. Apart from merges and splits, where blocks have two successors or predecessors, this produces a serially ordered history of each section: for any two blocks in the same section, it is well-defined which of them was agreed first. That makes this approach relatively easy to reason about, although the consensus algorithms themselves are rather complex. Other drawbacks are:

    • If an attacker controls a third of a section's nodes, they can break the section. (This is a general result that affects a large class of consensus algorithms.)
    • PBFT is leader-based: One node in the section plays a special role, and attacking that node (e.g. DoS) can delay progress in the section until it has decided on a new leader — which could immediately be attacked again!

    We looked into Honey Badger8 as an alternative leaderless consensus algorithm, but it looks like it cannot be adapted to our use case. We are continuing to evaluate other consensus algorithms and search for better fits.

    However, these drawbacks led to us taking the leaderless patterns further and considering approaches that avoid doing a complex consensus algorithm before committing. That means they don't necessarily always reach consensus on which event happens "first": e.g. there might remain an ambiguity (decision branch) about whether node A left first or node B left first. Nodes can process and action events concurrently and eventually reach a single state again on which they agree (both nodes A and B have left), that might be what we need from them to achieve our requirements here.

    Data Chains - General rules in an eventually consistent network34 is a discussion of implicit ordering, and explores a set of rules to reorder a vector of blocks to account for the fact that different nodes could have received the blocks (or rather the votes for them) in a different order.

    Data Chains - Option B33 proposes a mechanism that computes the validity of blocks from a set of votes that is independent of the votes' order.



  • MaidSafe Dev Update 

    Last week
    we talked about potentially adding a temporary data persistence
    solution to client-only networks (testnets where MaidSafe manages all
    the Vaults). The advantage of implementing this feature is that the
    front-end team and app developers would be able to start gaining
    experience with versioned APIs, versioned data, backward compatibility
    and deprecating APIs before data republish gets implemented via data
    chains. But based on what we heard from the community, we're going to
    drop any plans to support this immediately. We can revisit scoping
    options for this if we have too many short testnet iterations causing
    data wipes, since this probably wouldn't be a great experience for app
    developers.

    Developers who want to try out the new SAFE API can use the SAFE Browser binaries built with mock routing. See also the documentation for the Node.js AP and the DOM API7 as well as the following example applications:


    SAFE Authenticator & API

    We updated the safe_app_nodejs API based on the recent changes in the
    dev branch of safe_client_libs. The changes in safe_client_libs address
    a few inconsistencies and include a couple of API changes as mentioned
    in thispull request @bochaco also expanded the example snippets in the DOM API documentation. A few minor bugs were identified while creating simple code snippets and have been fixed. We'll update the API docs at the same time we release new safe_browser binaries, but devs who want to see the latest DOM API documentation can build it locally via the beaker-plugin-safe-app

    repo.

    @kumar has been away for this week and, as a result, the progress with the Java API has been slower. We had cross-platform issues while running the test cases. The issue was with the FFI callbacks being garbage collected. The issue is fixed now and the tests pass across several platforms. We will be wiring up the MutableData and NFS API next week.

    Based on the community's request, we will be releasing signed binaries
    from now on. We are planning to release these for the browser and also
    for the web hosting example application. We hope to provide signed
    binaries early next week which would include all the latest changes.


    SAFE Client Libs & Crust

    We are close to done with updating the master branch of safe_vault to latest routing (in this pull request

    including changes like Serde 1.0 and SHA-3. This is currently the
    single remaining library that needs these updates, good news and nice to
    see the end of that road.

    With help from the front-end team, we continue to polish
    safe_client_libs and apply a few more fixes and tidy up some of the
    code. Lately we discovered some inconsistencies in NFS functions
    available to user apps: they don't work as expected (e.g., not updating
    last modification time) and this behaviour is not documented properly.
    So currently we are in the process of reworking the set of NFS functions
    in safe_app to match the API available to Rust developers in safe_core::nfs

    @bochaco
    has discovered several bugs in the DOM API that are possibly rooted in
    safe_client_libs and we are looking through these cases, we're also
    trying to find a more efficient way to debug such problems in the
    future.

    Routing & Vault

    The simulation5 for option B22 is making good progress with most of the routing team working on that this week. The initial results covering adding, removing, splitting and merging look promising. We are making a few small improvements to make the simulation more accurate, and also slightly extending the option B proposal itself, specifically:

    • We need to cope with the fact that we can have several "contradictory" valid, current blocks affects safe_vault, and how data can be handled given that kind of uncertainty.
    • The document assumes that we have a mechanism that guarantees that eventually, every node receives every vote that concerns it. That will also need to be designed in detail.

    We also applied a few tweaks to the rules to make them less ambiguous and to address some scenarios we came up with independently of the simulation. The forum post22 has been updated.

    In routing master, we fixed a potential problem3 where two parallel merges could cause nodes to go out of sync.




  • MaidSafe Dev Update 


    SAFE Browser v0.2.1 and Web Hosting Manager v0.1.0 built against mock routing are available for download. The API docs have been updated (Node.js API and DOM API)

    In addition, we are also working on a Getting started or User walkthrough content and we are happy to share the initial draft of the same

    This content will be improved in the weeks to follow and we will eventually take this to the dev website.

    On another note, we received a package from @beekeeper who sent three jars of honey all the way to Troon. This sweet (see what we did there) gesture is very much appreciated and the fact that the honey came from Ron's very own hives makes it all the more special! Now, back to the serious stuff...

    SAFE Authenticator & API

    The safe_app_nodejs and DOM APIs are now up-to-date with the dev branch of safe_client_libs. A few issues with the authenticator and the DOM APIs were fixed. We also identified that the network status callback is not propagated in the DOM API, and so we will be implementing this at the earliest. We are testing the APIs with small code snippets which you can find in the Getting started draft41 mentioned in the introduction.




      @Kumar has resumed work and he has wired up the MutableData API for
    safe_app_java. We are expecting to wind this week with a feature
    complete safe_app_java API and start with the test cases for MutableData
    and NFS emulations next week

    SAFE Client Libs & Vault

    This week's major change has been the merge of the large pull request5 for the safe_vault repository. This hits an important milestone of bringing support for MutableData to vaults, which were the only missing component required for transition to the new apps authorisation paradigm. We continue to test these features and we're running soak tests to catch any remaining bugs and inconsistencies, as we want to make sure they won't ruin the user experience. After these tests are done, we're going to run an internal network and test it in a real-world environment.

    The mutable-data branch in vaults also needs a few minor tweaks to catch up on changes that were made in the master branch, such as invitation-based accounts registration, so that's also what we'll be focused on for the rest of this week.



    Upcoming changes in routing will also affect tests in vaults. Improved message delivery randomisation in mock-crust allows to run soak tests more efficiently and it already helped us in catching some rare bugs in vaults.



    In parallel, we're starting to investigate a possible new feature for SAFE Client Libs to support transactional mutations. We are considering it as a solution to the operation recovery problem. It might help in guaranteeing that sequences of operations are executed in full (or not executed at all), without an option to leave data in an incorrect state.



    Routing

    We keep extending the simulation and added tools to produce graphs that visualise how each section reaches consensus on its member list and prefix. With its help, we made a few more tweaks to the rules of Option B to reduce the number of messages that need to be sent.

    This graph shows the number of messages in flight (queue size) and
    sent as well as the total number of nodes in each step of the
    simulation. The y-axis is the index of the current step, i.e. time flows
    from the left to the right

    The spikes show the casting of votes by nodes in cases of churn, and subsequently trying to agree on valid blocks (representing the outcome of that event) eventually. Each spike roughly corresponds to a churn event, i.e. a node joining or leaving or a section splitting or merging.



    To look at the development of the simulated network in more detail, we can also produce a graph that shows each valid block.

    This one actually represents the same run of the simulation, but allows
    you to retrace in detail how the network gained nodes, split into six
    sections, lost nodes and merged back into one again. Time flows roughly
    from the top to the bottom, although two blocks of different sections
    (e.g. prefix 10 and 11) with the same version (e.g. 63, in this example) didn't necessarily become valid at the same time.

    It also illustrates a central difference between option B and option A: In option B (simulated here) you occasionally see two or three "contradictory" blocks with the same version and prefix which both became valid, i.e. the nodes didn't strictly decide, for example, whether node 1 left before or after node 2: they declared both sequences of events to be valid. This slight "uncertainty" yields a simpler message flow, fewer rounds of message exchanges and faster accumulation of valid blocks.



    In routing master, we are further extending the mock Crust tests in
    order to cover even more possible sequences in which messages could be
    handled. In the current implementation, for example, the tests don't
    simulate situations where two nodes exchange several messages back and
    forth while a message from a third node is in flight. If we make the
    test suite general enough to take all kinds of extreme message delay
    scenarios into account, we hope to find potential problems much sooner
    in the future



  • MaidSafe Dev Update - June 15, 2017

    The soak tests that we ran last week had very positive results. We've now started running an internal testnet with Mutable Data and we're trying out the authenticator and the example apps against this internal testnet.

    There are still some things we need to implement before releasing a public testnet. While this will be a client-only network, tests still involve churn to deal with special cases where a few droplets may go offline and these are cases where churn is expected to handle the data requirements as we'd expect. This also allows us to add more nodes to the network if there is a requirement to add more usable space to the network. Initial tests have identified a few bugs that did not occur with mock network tests which the front-end and client team are working on right now while the back-end team are also addressing some of the spam prevention requirements in parallel.

    After 18 months of working with MaidSafe, Andreas (@AndreasF) has decided to move on. We discussed him leaving, at least for a while, he is more than welcome back in the future and the door is open. We would like to thank him for working under such pressure and achieving so much in doing so. Heading up the routing design team, Andreas has been a committed and well respected member of the company. As you know he was very much involved in the recent changes to the Routing library and we all wish him very well. One of the benefits of the guys working so closely together is that a number of our engineers are very familiar with that part of the system and its future design direction.

    Andreas spent a lot of time with Spandan assessing potential new
    recruits and scaling the team continues to be a focus. We are working
    with a number of recruitment agencies and are starting to see a larger
    than ever before volume of CVs. While the front-end team has taken on a
    couple of new developers in the last few months, and we are close to
    confirming more, the network team has not grown much and we are making
    changes to improve that situation with a view to speeding up development
    and spreading the huge workload much more evenly. We recently dropped
    the necessity of requiring each candidate to have Rust experience, and
    are also looking for system level devs (C, C++ and Rust) who also have
    experience working with other P2P systems, or network security, or other
    distributed technologies, such as Hadoop, Redis or Cassandra. This
    means we are looking for system level devs experienced in Rust, or
    interested in learning it

    SAFE Authenticator & API

    We tested the front-end APIs and applications against an internal test network. We came across a few issues which were not present with the mock network. The entire team has been focusing on fixing these issues and we are now almost done fixing them. However, applications don't work as expected when revoked and reauthorised. We also found a gap in creating the unregistered client and it was wired up in safe-core. The unregistered client related changes were integrated with authenticator and the Node.js API. In general, it is expected that an unregistered client should be created without even needing an account in the network. At present, it is required to have an account to create an unregistered client. This issue will be addressed in the next few days.

    Other than the functional issues, we have few a UI updates that we will be working on as we continue to test and fix the issues.

    The changes that were made in the client libs must be integrated in the Java API. The test cases are being wired at present for the existing integrated APIs.

    SAFE Client Libs & Vault

    We keep catching and fixing bugs in SAFE Client Libs and we're adding missing functions found by the front-end team. As described in the section above, we've been working on adding missing functions to propagate Crust bootstrap configurations to unregistered clients. We've also fixed account registration and login functions that weren't following the calling convention that we use for all other functions exported from Rust.

    Currently we're working together with the front-end team to correct the app revocation and reauthentication flow and we're adding more automated tests to make sure that these features work as intended.

    There are some more bugs in mock-routing that were discovered by @shankar,
    concerning handling of MutableData addressing and storage. The current
    mock-routing implementation assumes that only one MutableData instance
    can be stored with a given XorName, while in the real network that is
    not the case: apps can store multiple MutableDatas with
    various type tags under the same XorName. Another issue that we've found
    is that network status is not propagated to the front-end properly and
    it is not properly tested on the Rust side. So this is what we'll be
    addressing for the next couple of days.



    For spam prevention3 we are trying to see what numbers are good enough a compromise. We have
    also identified more areas that can potentially be patched and some of
    them are 


    already underway. That for e.g. addresses the problem that a client can send RPCs which
    it shouldn't (but does it because it's malicious). Soon we will be able
    to come up with a list of approaches and then we'll present that
    information in this Dev Forum topic

    On the vaults side, we've added tests and checks to forbid duplicate message IDs3
    for client requests. This is required because if clients send two (or
    more) request using the same message ID, the account balance checks
    could be bypassed as the balance would have been charged only once.
    We've also added some new tests and increased the test coverage.

    Routing

    The mutable data branch has finally been merged into master! That's a huge change, although not quite as big as the one for safe_vault is going to be. We are now internally running test networks and adding the last few pieces to the implementation.

    Apart from minor tweaks to Option B, we are now working on extending the simulation in two directions:

    • It used to assume for simplicity, and to first only test the convergence aspect of the proposal, that every piece of information — every vote — is sent to every node in the network. We need to work out, test, and simulate the details of who actually needs to send the votes to whom, and how to recover from cases where messages failed to arrive.
    • The simulation is too memory-hungry to create a large number of nodes, but there are ways to deduplicate some of the information shared by the nodes (votes and blocks that are held by the simulated nodes) without changing the semantics. And of course the first point will also help: if less information is exchanged, the code will run faster and with less RAM.

    And finally, a new option to mock crypto primitives in tests speeds up the test suite more than 4-fold!





  • MaidSafe Dev Update - June 22, 2017

    We are continuing to progress towards our goal of releasing a public testnet with Mutable Data. The internal testnet we ran last week showed a few issues and we've been working on fixing these issues during the last few days. We had hoped to release the public testnet this week, but these bugs would mean we would have to take development time out to support issues we know of and then soak test various possible attack vectors that might be applicable for client-only test networks. Patches for a few of these are already in place, with more to get merged soon to master and tested thoroughly to confirm they satisfy the spam prevention requirements. We realise everyone is very keen that we release the new testnet and then Alpha 2 as soon as possible and we will.

    Some of these issues had to do with file synchronisation. Only after multiple apps (each of which is now a different process) were run together with the authenticator did we notice that things put into the common file used by mock-routing/vaults were not properly picked up by other processes, etc. leading to some test failures. These are now resolved. Another point is the (current temporary solution) delete handling which leaves the entries behind, merely blanking out the values. Some parts of code incorrectly assumed that only the absence would mean the entry was not present when in fact we should have treated the presence with a blanked out value similarly. This is sometimes an annoying paradigm which is accentuated as apps are revoked and reauthenticated when in each revocation the previous entries need to be re-encrypted for protection. Since we cannot erase, we need to leave the previous entries blanked out and insert the new ciphertext, making the number of entries and size of the mutable data grow even without any app having done anything. We are soon going to address this by providing a way to actually delete and differentiate the types that don't allow strict deletion with a different range of type tags. We are dwelling on this though and it is not a confirmed approach yet.

    Recruitment

    As you will know from previous updates, recruitment is a focus for us. We are all acutely aware of development timescales and doing as much as we can to shorten them. With regard to recruiting network engineers, we have just made an offer to a developer and have several more at various stages in our process, which is a combination of a technical interview, a coding challenge, and a cultural fit interview. We are currently screening through about 20 CVs per week so hopefully we will be able to grow the team with really good quality engineers. We're also delighted to announce that @joshuef@Krishna_Kumar's team from next week. As many will know, Josh was the creator of the SAFE Browser and it is great to have a developer of his calibre on board. We have also hired 2 additional team members for the operations team, an admin/finance manager and an office manager to help support the team as we move forward with launch preparations and increased relationships.

    Upcoming meetups

    The SAFE Network San Francisco meetup groupis being revived by @hunterlestera meetup about IoT devices for the SAFE Networkon July 11.

    We're grateful to Mozilla SF for graciously allowing us to use their lovely community events room to host this meetup.

    Thank you to core Rust engineer, Brian Anderson (@brson), for volunteering to help organise and oversee our event.

    This meetup will be heavy on fun experimentation, making, building, and very light, if any, on presentations. The initial focus will be on building IoT devices as inexpensively as possible to interface with the SAFE Network.

    The SAFE Network London meetup group is organising a meetup on July 5:

    The first Project SAFE gathering of 2017 will combine an update on SAFE protocol, demos and a social gathering. Awaiting venue being confirmed (RISE have move location)


    See the event page4 for more information.


    SAFE Authenticator & API We are continuing to test the APIs and the applications with the actual network. Testing with the actual network has exposed a few gaps in the mock implementation and we are addressing the issues as we continue to test. We did make some great progress in terms of addressing the issues that we had come across and we hope only a few are left to complete. Issues, especially with revocation and reauthorisation, have multiple scenarios to test and we are making steady progress. While most of the team is working on fixing these issues and testing, @hunterlester has been working on a tool which can serve as a playground for the DOM APIs. Most of the requests from developers were related to the DOM API and we thought of putting together a tool which can help in learning the APIs. @hunterlester has created a small video teaser on how it is shaping up. The Java API is also catching up with the changes in the FFI APIs and the test cases are being updated. @Kumar is hoping to wrap the APIs soon.

    The Java API is also catching up with the changes in the FFI APIs and the test cases are being updated. @Kumar is hoping to wrap the APIs soon.

    SAFE Client Libs & Vault

    After more than half a year of development, the mutable-data branch of SAFE Client Libs1 has been merged into master, along with utility repositories and crates such as self_encryption. This means that from now on we won't be maintaining older versions of these crates and we'll focus only on MutableData, which is upstream now.

    This week, a lot of time was spent on debugging and resolving multiple issues in safe_client_libs, including those described in the previous update. First, we've found some synchronisation issues with the mock-vault
    implementation: while it's been working fine when confined to a single
    process, the front-end team started to get some weird bugs trying to run
    several processes using the same vault at once (e.g. the browser with
    the authenticator and an app). In this case, changes made in one process
    weren't available to another. Now we've made a change2 to synchronise a vault state in memory with the file system on each read operation.

    Besides that, we've fixed multiple errors concerning handling of app revocation (caused by incorrect handling of removed entries), added graceful handling of low balance errors in mock-vault, and fixed incorrect network connection status propagation.

    Vaults have seen some improvements in limiting concurrent mutations of a
    single MutableData instance. Previously, during the edge case of
    approaching the limit of allowed entries or size, one could send the
    maximum number of allowed mutations and vaults would have to honour all
    those requests due the artifact of unknown order of accumulation and
    eventual consistency and the limit could be slightly exceeded. This
    wasn't noted down as a big problem (as there was a limit to the number
    of mutation requests that could be handled at a time) but it existed and
    was known. We have changed the paradigm for checks (due to the nature
    of MutableData which allows for independent entries) by taking into
    account the data not only in the chunk store but also pending writes in
    the cache and now it should not be possible to go any bit over the max
    allowed limit. The change has been applied.


    Routing

    There was nothing blocking clients to send disallowed RPCs to the network. Now we have put checks to disconnect from any client that tries to send non-allowed RPCs right at the proxy. This will make sure that we reduce the effort the network spends on a misbehaving client and in future PRs we will also try and penalise such an action if detected. To prevent client spamming the network, we have introduced a concept of rate-limiter. We start with defining what is the total bandwidth that can be handled non-disruptively by the network on behalf of the clients (bandwidth usage due to churn/data-relocation etc. are over and above this). These numbers will likely be tweaked based on test results and actual network capability. Then we divide that by the total number of potential proxies we plan the testnet for, thus getting the maximum throughput allowed for clients per proxy. Then, depending on the total number of connected clients, we distribute this bandwidth evenly among them for fair usage policy. The Rate Limiter ensures that no client goes above the maximum throughput it is allowed (throttling). There is an upper cap of 10 clients per proxy.

    Next, we only allow one client per IP at the proxy layer, which establishes the physical network connection to clients. This further prevents a malicious client from restarting every time with a new unregistered client and bypass the system. We are also working on a ban for a period if malicious activity is detected, thus the client cannot just restart and login from the same IP frequently. It can however go to another proxy if needed until it exhausts all of them after which it has to compulsorily wait for the ban to expire with at least one of the proxies before it can bootstrap to the network again. Of course there can be an attack if someone chooses to spend resources on getting many different IPs to DDoS everything, but that is not the attack vector we are catering to currently (at least not in the upcoming testnets where the network size is limited to a fraction of what we'd expect it to be when users are also running vaults and contributing to the various network personas).



Looks like your connection to Cryptocentral was lost, please wait while we try to reconnect.