Maidsafe coin Details+Update
MaidSafe Dev Update - Test 18
Today, we are pleased to be releasing Test 18 for all forum users who are at least trust level 1
This is a new network, so trying to access data from Test 17 won’t work (you need to create a new account). We are planning to take Test 17 offline this weekend (both the internal test and Test 17 itself).
Here are the main changes compared to Test 17
- The storage limit has been doubled from 500 operations (store and amend) to 1000 per account.
- Add new FFI function to get the account info from the network (number of available/used mutations)
- Implement operations recovery for account creation, app authentication and app revocation.
- Modify the client Rate Limiter paradigm to not put an upper threshold on number of clients with unique IPs to bootstrap off a proxy.
It’s also now much easier to run your own local network (because of the
new developer configuration options that have been added to Routing,
Crust and Vault). See this Dev Forum topic for more info.
Earlier this week, we started to document how to use the mock, local and
actual network (for developers who want to build SAFE Network
applications and websites). See this Dev Forum topic to read our first draft.
For Test 18, we are intending to have some front-end updates next week
to introduce an option for apps to request access to arbitrary Mutable
Data objects (see this JIRA task for more details). The network itself shouldn’t require an update (unless something unforeseen occurs of course).
Based on the feedback from the community, we are redesigning the UI for Web Hosting Manager app. You can see the new design in this Dev Forum topic. Please do share your feedback on the latest design.
SAFE Browser v0.4.0
To connect to Test 18, you need to use SAFE Browser v0.4.0.
Accounts can be created using the authenticator bundled with the SAFE Browser. To open it, click on the Authenticator icon in the toolbar of the browser.
To obtain an invitation token, trust level 1 forum users can use this website20, which can also be accessed via the SAFE Browser itself when creating an account.
Each client account is limited to 1000 operations. It’s worth noting that this number is just a temporary placeholder and is shared between all the apps on a given account.
Please be aware that we might need to restart this test network, wiping all data that is stored on it. We will announce this in a dev update (when possible) prior to taking it down.
- Freeing of safe_app object handles. Object handles created by the applications will be automatically freed on closing the tab or even on refresh.
- Toggle safe browsing. Allow browsing of clearnet sites by toggling from
File > SAFE Browsing Enabled
- Index file to be read automatically. Providing the
<service>.<publicid>as the URL loads the
index.htmlfile of the service.
- The number of store/modify operations completed for the account is displayed in the Authenticator.
- Enhancements to the NFS API are now being exposed to the DOM layer for web applications.
safe:protocol URLs will be opened in the SAFE Browser when opened from external links.
localhost:protocol is now supported. In order to open a URL like
localhost:<port>, the URL must be converted to
<anything>can be any characters but not empty, for example,
- Several UI enhancements and minor fixes have been introduced, like ordering the application list, reducing the header size, loading overlay has rounded edges.
- Issues with the building process for mock and real network have been resolved.
- Compatible with Test 18.
- History and bookmarks are not saved in the network right now.
- Public IDs created via examples are not exchangeable between the apps.
- This is the error reported with the Access Denied / Requested Entry not found error. This issue is related to the permissions of the service/mail mutable data. The root folder with the default folders are designed for easier sharing and collaboration between apps. Applications can request access to these while the authenticator manages the app access levels. In the case of public IDs, the public ID is added as a key into the _publicNames container. A corresponding MutableData is created by the app and the permissions for the same are set by the app. This MutableData is then added as the value to the public ID entry. Now when the second app tries to add a service to the MutableData created by the first app, it doesn’t have the permissions to do so as this was not created by the authenticator. To resolve this, we are going to introduce an option for apps to request access to arbitrary Mutable Data objects (see this JIRA task9 for more details).
Web Hosting Manager v0.1.2
When uploading files using Web Hosting Manager, the maximum file size is 25 MB per file.
- Present more user-friendly error messages.
- Long public ID and service name are truncated and shown on the UI.
- Internal use of the NFS API has been adapted to recent changes in
safe_client_libsmaster branch. The NFS API was using ImmutableData to store the files and now the API is updated to save DataMap pointers. For each file, a pointer to the data map is stored in the MutableData entry representing that file. The actual data map is stored in the network either encrypted or plaintext depending on the privacy level specified.
- Window resizing is now disabled.
- Compatible with Test 18.
SAFE Mail Tutorial v0.2.2
- Compatible with Test 18.
SAFE Web API Playground
A new tool called SAFE Web API Playground is now being formally maintained in MaidSafe’s safe_examples repository6, which allows any developer to learn and experiment with the DOM API in a friendly and interactive way.
It is meant to help people aiming at creating websites/webapps that need to access the SAFE Network DOM API, which is available on the global
windowobject in the SAFE Browser.
Instead of having to upload your site to the network every time you want to test some code, you can simply use this tool to experiment and get a deeper understanding of how each API works.
In order to use it, follow the instructions found in its
README, and either upload it as a web app using the Web Hosting Manager application or just run it locally and access it from the SAFE Browser using the now supported
localhostprotocol (see above for details).
It is also available on the test network at
safe://webapi.learn/, however, it is much faster when accessed locally.
Example HTML files
examples.htmlfile has been published with the safe_examples release. This file showcases some basic use cases of the DOM API, like authorising an application, creating MutableData objects, as well as reading and iterating over MutableData entries. You can upload this file using the web_hosting_manager application and publish it as a website, and then run any of the three examples available from the SAFE Browser.
Another file called
leaking.htmlhas been also published with the safe_examples release. This file can be used for testing the browser capability of freeing safe_app object handles. If you click on one of the example applications it will create a client and authorise the app but it won’t free them, causing a leak in clients. After running any of these examples more than twice you will start getting a connection error due to the limit in the number of clients being reached. Refreshing the page will automatically free all leaked objects allowing you to run any of the examples again.
If you need help with anything related to SAFE Browser, Web Hosting Manager or SAFE Mail Tutorial, please use the #support category of this forum.
Where should I report issues?
If you know which component a bug is associated with, feel free to report it on GitHub in the corresponding repo.
- Report issues with SAFE Browser5
- Report issues with SAFE Authenticator2
- Report issues with SAFE App DOM API1
- Report issues with Web Hosting Manager3
- Report issues with SAFE Mail Tutorial3
- Report issues with SAFE Web API Playground3
- Report issues with SAFE App Node.js API
SAFE Authenticator & API
- Upgrade safe_client_libs to latest in master branch.
- Support the new NFS API from safe_client_libs. The safe_app_nodejs library was internally using ImmutableData to store the files, but this is now being handled directly by safe_client_libs.
- Allow the
logPathfunction in the API to receive a filename parameter to automatically generate the log path for it. The path will be the same as the path of the safe_core lib log file.
appTokenparameter in the DOM API functions was renamed to
appHandleto make it more consistent with what it really is.
SAFE Client Libs
The pull request for operations recovery in safe_authenticator has been merged now, but this road doesn’t end here and we are planning on several upcoming improvements and fixes for this feature - all of them required to be sure that we have covered most of the known edge cases and this feature works as intended. One of the possible limitations is the concurrent app revocation: for instance, if a user wants to revoke apps on their PC and a mobile device at the same time, it’s guaranteed by the design that nothing wrong should happen, but we want to verify it and we’ll be improving our test suite. Also, we already have a plan to improve, optimise, and simplify some things related to how apps authentication and revocation work, so that’s what we’ll be working on for the rest of this week. However, as these changes are mostly just nice-to-haves and the main functionality is implemented & tested, we’re releasing new versions of the SAFE Client Libs crates with what we already have.
The team also made a few additional changes. @[email protected] has finished his work on the new feature to share Mutable Data objects between apps through the Authenticator, and his pull request will be merged soon. @adam has updated the repository to use the recent stable Rust version (1.19).
We’ve also been involved in debugging an interesting bug that was
reported by the front-end team as we weren’t sure of its origin (on the
Rust side or on the front-end). As it turned out, it was neither: we’ve
traced it down to node-ffi (the library that we use to communicate
thread while waiting for a callback to return a result. This comes into
a conflict with our rule about non-blocking front-end callback
functions and this behaviour caused SAFE Browser freezing in certain
circumstances. We’ve tracked down & fixed these particular cases but
the problem still remains and soon we’ll be discussing how to fix it
Routing & Vault
The Local Network support via dev options is now fully integrated and tested in both Routing and Vault. There were a couple of fixes regarding rate-limiter. In this commit1, we make sure to send enough information to identify and clear the message from the acknowledgement manager in case we get the rate-limit exceeded error. When this error happens, the whole message is forgotten by Routing and the upper libraries are informed, which then have the responsibility to handle it the way they want. Safe_core for instance is going to retry the operation after a wait so that it doesn’t keep getting the same error back. This PR changes GETs to PUTs (mutations) as all it is testing is whether the request reaches the correct number of nodes at the correct destination. GETs are charged for the worst case as we don’t know in advance how much traffic it is going to cause. Changing to mutations allows the test clients to invoke the RPC pertinent to the test without getting an interference from the rate-limiter.
For vaults, we now get the group size and quorum from Routing where it can even be tweaked now that Routing has a config file. The rest of the changes2 in vaults were to integrate with the Routing changes.
MaidSafe Dev Update - Authenticator & API, SAFE Client Libs & More
Last week, we released Test 18. The focus during this week has been on getting all the issues that need to get completed for Alpha 2 (including those reported by the community) added to JIRA and making sure there are no loose ends.
With Test 17, we had the rate limiter quite relaxed and a lot of traffic was allowed. In Test 18, it is kind of the opposite since there are many more reattempts and retries that need to be happening from the clients. We are now working on striking a good balance between these two extremes and we’ll be updating the rate limiter to improve its performance and give a better user experience in terms of data traffic. With these changes, a lot more future requests should go through in the first attempt.
Advisory and Endorsement Requests
Members of the team, especially David, are getting quite a lot of direct approaches, either through lengthy DMs and LinkedIn messages, from people looking for advice on other projects, or to endorse ICOs. While we would like to be in a position to help, our focus on network development does not allow us time to read these messages in many cases, let alone respond. Please do not be offended by this, it doesn’t mean that we don’t see an importance in what you’re doing, we are simply very busy and your time is likely to be better spent engaging with other potential advisors.
SAFE Authenticator & API
Issues that were raised in Test 18 have been gathered and organised in JIRA. The new IPC for sharing MutableData is being integrated this week. Integrating the IPC functions in safe_app_nodejs and authenticator is almost completed. This feature has been the priority for us this week and @[email protected] are getting this integrated and tested end-to-end with the applications.
We have also fixed a few issues this week. The native modules were not being loaded by safe_browser on a few Windows machines which did not have the 2015 Visual C++ redistributable. When the user tries to log in, the authenticator kept spinning and it was misleading as if it was not connecting to the network. We are now handling this case and the proper error message is presented. safe_browser issue #99 is also resolved.
When the browser is trying to load a page even if the requested page/resource is not available, the loader in the browser keeps on spinning. This issue is now resolved in the master branch of beaker-plugin-safe-app.
Based on feedback from the dev community, the
getHomeContainerAPI is being renamed to
This API is used to fetch the application’s own/root container. The DOM
APIs have also been updated with this change. Please do keep a note of
this change in case you are trying to build the browser or your apps
with the latest code in the master branch of safe_app_nodejs and
beaker-plugin-safe-app. Also, @hunterlestersafe_browser issue #1003 soon.
The Web Hosting Manager app updates the progress bar only after the file is uploaded to the network. If a big file is uploaded, the progress bar takes a long time to update, leaving the user guessing whether the application is hung or still working. This will need to use the NFS API to write as a stream instead of passing the entire content at once. safe_app_nodejs already exposes APIs for the same. The Web Hosting Manager app must be refactored to use the API to write as streams and update the progress bar as chunks of data are being uploaded. @hunterlester should be able to resolve this issue this week.
@joshuefpull request for storing the history and bookmarks in the network. The feature looks
stable after a couple of testing iterations and we’re expecting it to be
merged soon. @joshuef
is looking into the DOM APIs to enable returning the error code along
with the error message. Right now, only the error message is returned as
beaker browser does not allow the APIs to return complex objects. @joshuef is trying to implement this feature for returning errors.
SAFE Client Libs
The back-end team has been working hard on fixing bugs reported by
the front-end team and the community, and improving the features that
were released last week. We’ve started with refactoring the account
packet structure and improving
safe_core, the essential
parts of the SAFE Client Libs. The account packet is the data structure
that holds the user account details, keys, and some extra information,
such as pointers to the root directories - and that is the part that we
wanted to improve because it remained almost intact since the SAFE
Launcher days, while now we have a much more complicated design, with
apps and the Authenticator. So now we’ve got rid of unnecessary
indirections (the pointer to the access container is now stored as a
part of the account packet) and inefficiencies (the root directory has
been removed and the containers info is now stored directly in the
access container). Along the way, we’ve reduced the number of network
requests and parallelised some operations (which previously introduced
useless synchronisation delays), so overall these changes should make
the UI more snappy. They’re mostly completed and likely to be merged
before the end of this week. For more details, you can see this JIRA task
In parallel, @[email protected] have been working on improving the new API for sharing Mutable Data. The basic implementation has been completed and merged and now we’re working on implementing useful comments and suggestions from @bochaco
and the front-end team. First, we’ve decided to clearly define the key
for the metadata entry of shared Mutable Data objects because allowing
an app to provide its own metadata key could lead to security issues.
For example, if a password manager app sets the metadata entry to say
something like “this data must not be shared with other apps”, another
malicious app requesting shared access could point to some other entry
to be looked up for metadata, effectively bypassing the caution message
and making a user confused about the intent of the sharing request.
Second, the permissions part of the request has been simplified: now we
assume that an app wants to request only
Allowpermissions, delegating more fine-grained permission settings (involving blacklists and
Deny) to apps having the
ManagePermissionsprivilege. All these changes go in this pull request and we’re currently reviewing them and perfecting the details.
Next, we’ll be focusing on optimising apps revocation. We’ve started with this pull request,
which fixes some flaws and edge cases, such as attempts to authorise an
application during an ongoing app revocation. This is not allowed now,
as authorisation might interfere with re-encryption of revoked
containers. Another issue could happen if an app tried to update or
insert new data into a container which failed to be re-encrypted - in
some circumstances it used an incorrect set of keys and mangled the
data. Tomorrow we’ll continue addressing similar minor (but crucial)
On top of that, there was a lot of smaller improvements. @marcinconfirming and fixing bugs
found by the front-end team, such as a case when subsequent app
re-authorisation didn’t result in the creation of app’s own container
(if it wasn’t created on the first request). @marcinmore NFS tests and verified that a user can still log into his account and browse the network even if he’s exhausted his account balance. He’s also removed a legacy module, public_id, which remained from the SAFE Launcher and wasn’t actually used by the front-end. @canndrewupdated the Routing dependency
to the latest version, removing the necessity of retrying requests if a
request rate limit has been exceeded - now that this is the Routing’s
responsibility, we’ve considerably simplified this part of the code in
safe_core. Finally, @adamhas implemented
an environment variable switch that disables the on-disk vault storage
for tests. This change should speed up both front-end and back-end tests a bit.
Routing & Vault
As stated in the introduction, the rate-limiter for Test 18 is stricter compared to the previous test network. Message resend logic has also not been too efficient as Routing breaks the user messages from upper libraries into user message parts, each of which can be a maximum of 20 KiB. So if there was a user message which was 200 KiB, Routing would have broken it into 10 chunks of 20 KiB each. If a few of them were rejected, we’d currently be sending all parts again from the Client libs. The user library would then have to resend the entire message again after a pre-defined waiting period. To optimise this, we’ve modified Routing to handle the message resend at that layer directly. With the new changes, Routing will now selectively re-transmit the user message parts for which the error occurred. This encapsulates the message resend logic away from the upper libs. Routing also optimises route traffic by allowing the upper libs to indicate a maximum wait period after which message relay via further routes will not be attempted. This not only applies to rate limiter errors but also other parts such as the Ack-Manager.
There have been a few patches that have been brought in to master to address certain bootstrap bugs from Test 18 and the Rate limiter has also been refined this last week to allow clients to scale their data traffic capacity based on concurrent client connections active at the proxy at a given time. The PR addressing the same is currently waiting to be reviewed and merged2. Rate limiter throughput calculations have also been updated to reflect the actual data throughput relayed by the proxy nodes to the clients.
MaidSafe Dev Update - SAFE Authenticator & API , Routing & Vault & More
Being at the epicentre of the software world and with a huge number of developers, San Francisco is one of many areas where we would like to see more engagement, and these events represent some early steps as we start a more focussed effort on spreading the word about the SAFE Network. We are now working to maximise the effectiveness of these events and have incorporated both internal feedback and feedback from members of the community. Over the next few events, we’ll be improving the way in which the network is presented, as well as improving the AV so that the content can be made available for those unable to make it to events.
This week the team have and continue to work through a number of JIRA tasks that either fix issues raised by the community in previous tests, or that add features. This might potentially continue into the following week too. Lots of details about that below. Backend code is more or less at a frozen state in terms of required features for Alpha 2 (Authenticator) and frontend doesn’t look too far away from the same too.
We have also spent more time planning development in recent days to spread some of the engineering responsibilities around the team as we have a couple of guys either out at the moment (Adam is on vacation and Bart is relocating to Hong Kong for a few months) or, as in @Krishna_Kumar’s case, preparing to get married.
Congratulations from all the team here Krishna, we wish you and your soon to be wife (Rennee) a long and very happy future together, although she is potentially in for a tough time as she competes for attention with your laptop which may have to be surgically removed for the big day
Community-driven content Continuing in the marketing vein, we would like to point out for those who have missed it, the effort being put in by a few long-standing members of the community including @polpolrene, @Krekc, @JPL and others who are starting a collaborative effort to pull together a new SAFE Network white paper/primer that brings together the most up to date content in one place. This comes on the back of a survey conducted by @JPL that gauges the SAFE communities attitudes toward the way the existing information is presented.
David also took to his own personal blog a couple of times during this past week to write about The Impossible Network and to clarify the differences between the SAFE Network and alternative data/storage networks that incorporate blockchain technology. The fact that the SAFE Network completely removes intermediaries and humans from the management of our data through the network’s algorithms is a huge differentiator and the benefits of that is something we will be focussing more on in future marketing communications. We are also now starting to recruit 2 digital marketers to be based at our HQ in Scotland to help promote the network and drive engagement with the technology and the company.
SAFE Authenticator & API
The focus for this week has been to complete the tasks that were added to JIRA. The new IPC for requesting access to arbitrary MutableData objects has been integrated with the authenticator and also with both example applications (Web Hosting Manager and SAFE Mail Tutorial). We have tested the initial integration with the apps and authenticator. An application can set the Metadata for the MutableData in case it wants to share it with other applications as explained in this comment. We don’t force the devs to set the metadata. If the app is not setting the metadata and another application wants to access the same, then there is no useful information to be presented to the user for aiding the decision making on whether to grant access or not. Hence, we decided to provide an option to look up for the applications that have been granted access for that MutableData. With this feature, the user can see the list of applications that have been granted access before and then decide on whether or not to grant access to the requesting application. The API to get the list of apps that are granted permission for the MutableData is being actively developed by the core team and we will integrate the same with the authenticator.
Another major improvement on the browser this week was to enable the error codes being passed back from the APIs on failure cases. We were not able to pass complex objects across the DOM APIs and thus we were only passing the error message previously. @joshuef has resolved this issue and now we can pass the error codes back to the DOM API layer.
The API changes in
safe_app_nodejsrelated to the new IPC for MutableData are also exposed from the DOM APIs. The DOM APIs are now up to date and in sync with the Node.js APIs.
The upload experience of files in the Web Hosting Manager app is improved. The progress bar used to update only when the file upload was completed and when a large file was uploaded, the progress bar would stay at zero percentage until the file is completely uploaded. The user gets an impression that the app has hung, while it was still uploading the file behind the scenes. Now we have fixed it by using the NFS API to write in smaller chunks to the network and the progress bar reflects the status immediately.
We have made good progress this week and the pending task list looks much smaller compared to where we were last week
SAFE Client Libs
We’ve been continuing to work on tasks that were mentioned in last week’s update particularly improvements on the authentication side in the
safe_corepart of the libraries. We have finished with the refactoring and optimisation of the account packet structure
and the account registration flow, which is now only the Authenticator’s responsibility (
is concerned only about filling and putting the account packet to the
network). These changes helped us to simplify and decouple multiple
modules in the libraries
@adamhave been merged
last week. After some discussions with the front-end team, we have
concluded that the mutable data sharing flow needs to be changed a bit:
previously, we didn’t support sharing of Mutable Data objects without
metadata because it wasn’t clear what we should show in the UI in this
case - a user couldn’t tell what the exact purpose of the requested
Mutable Data is just by looking at its name and a type tag. Now, we
decided to implement a new function in the Authenticator API to return a
list of all apps that have access to a certain Mutable Data - this way a
user can infer its purpose by looking at the list of apps that have
permissions to access it. It’s outlined in more detail in the JIRA task.
Routing & Vault
The final PR for rate limiter6z which brings in the feature of soft capacity is now merged. We patched a bug in Routing which showed up in the soak tests after many iterations. It’s when two nodes want to connect to each other, prepare connection info request and send it to each other but are blocked from connecting to each other by the mock-crust test. The code in peer manager prepares connection info as a response to both connection info request and connection info response. Thus after the nodes realised that they could not direct connect and started to tunnel to each other which also failed for the first tunnel selected as this was also a part of the test’s simulation logic, one of the nodes got a connection info response from the other. This made its peer manager change state from searching-for-tunnel to preparing-connection-info and hand it over to
node.rs. Unfortunately, this operation in
node.rswas a no-op when called from
handle_connection_info_response. Thus there was a disconnect between peer manager, which thought Routing was asking Crust to prepare connection info, and
node.rswhich ignored this and never contacted Crust. To fix this, peer manager is now patched to not change its state while searching for tunnel if it gets a connection info response. Eventually, other potential tunnel nodes would be tried and the connection would succeed. This fix is now merged. Soak tests with the latest changeset have been running for a few days now and are looking good.
A Vault PR to integrate with the latest Routing is now up and waiting to be reviewed. It has the necessary refactor mandated by changes in Routing which no longer raises proxy rate limit exceeded errors (as it deals with them internally), the details of which were stated in the previous dev update.
Maidsafe - Safe Tube release
I just finished the last bit of code on Safe Tube, githubTo access the website go to safe://safetube.jam
and paste the code from here
into the browser developer’s console (right-click on website, click on
inspect element and click on the console tab) or you can access it using
a local webserver. The videos have to be loaded instead of streamed so
it will take some time for the website to fully load. Feel free to give
some spare maidsafecoins to this address:
If you have any issues or requested features please leave them here.
This is part of Joseph's safe websites project. I will now release a seperate topic for each completed project as it will be easier to keep track of bugs and requested features.
Guys, I'm too old and lazy to figure it out. For example, there http://www.fintech.finance/01-... is a survey for investors - it's better to burn tokens after buyback or release them back. Which way is better?