Stop storing your users' data!

... and switch to open protocols in the process.

  1. Thanks for coming to this talk
  2. Come to Decentralize all the things after this one!
  3. Going to show a proposed solution for decentralizing data storage on the Web

About me

Srsly, you should join us at Hacker Beach!!1!

http://hackerbeach.org

  1. Every year in January

Content

  1. Unhosted

    Separating user data from apps

  2. remoteStorage

    An open protocol for per-user storage

  3. remoteStorage.js

    A JS library for integrating remoteStorage in apps

  1. First, let me give you some context and history of the idea

Part 1: Unhosted

unhosted
  1. Who has heard of Unhosted before?

The Web

It works!

  1. I'm using the term Web here, but it's also about the Internet itself
  2. Why does it work?! Why can I type http://jsunconf.eu and see a nicely designed web page coming from a random other computer somewhere on the planet?
  3. The answer, of course: protocols and open standards
  4. TCP, DNS, HTTP, SSL, TLS, URI, HTML, CSS and countless others
  5. Common languages we agree upon; agnostic of the technology/implementation!
  6. Student in dorm room can program Web server
  7. Anarchist co-op can run an open-source email server sending emails to the CEO of a corporation using a proprietary one.

The open Web is a unique ecosystem because no one controls or owns it.

Andreas Gal (CTO, Mozilla)

  1. Andreas Gal just became CTO of Mozilla, after that other person
  2. In the technology sector, the Web is unique
  3. No licensing fees, no royalties, no certification, no membership required
  4. Recently there was a conversation at W3C about how to download ISO specs. Not so easy.

Also the Web

  1. But this is also the Web
  2. When it comes to your data, it's not so open anymore
  3. Both we as users and we as developers have to jump through hoops to move data between Web applications
  4. And it's locked in to one application, give or take API access
  5. (actually bad example picture, because many social apps, think readlater, task managers, your photo library, etc)
  6. Why is there so much lock-in? What do all these have in common?
  7. No common protocols or standards (for user data)
  8. Use Web protocols and standards, but not for the data itself

Protocol/standards development process can be messy and complicated

  1. Standards process can look like this from the outside
  2. And it often is messy and long-winded
  3. Because it's mostly based on consensus
  4. Anyone can jump on a public mailing list and add arguments to the discussion (please do!)
  1. And it can actually be fun to watch
  2. Especially when it comes to EME a.k.a. DRM in HTML5

But it works (mostly)

  1. But it works (mostly)
  2. And in the end it's how the Web is made (awesome)
  3. So that everyone can participate and compete

Thanks, timbl!

  1. Thanks, timbl!
  2. And of course thanks to everybody else who created the Web and underlying technology
  3. This brings us to the beginnings of Unhosted
  4. The Unhosted idea was actually inspired by a timbl article from 2009

5 years ago...

  1. It was title "socially aware cloud storage"
  2. In that article he decsribes exactly the problem of data tied to applications
  3. And lines out ideas for possible solutions

Key idea

  1. Key point: separate applications from data

Fuck yeah!

  1. A bunch of hackers read this article and thought "Fuck Yeah"

But let's take it a step further!

Because the Web Platform has become pretty awesome

  1. But Web technologies have come pretty far, and we have many more things to build upon
  2. Also, not all data is social, and almost all of our data lives in silos now
  1. HTML5 yo

New possibilities

  1. The future is in the browser – on the client side!
  2. We can build a lot of UI apps completely without servers now
  3. So the hackers had the idea for an even better architecture
  4. Let me explain to you what it looks like

Hosted architecture

  1. Your average Web application / SaaS

Unhosted architecture

  1. All of the app's source loaded into browser; data storage connected separately
  2. Same architecture for general "no-backend" apps.
  3. E.g. Firebase etc.
  4. The difference to all other no-backend tech is that the Data cylinder would not be controlled by the user.
  5. Feel free to ignore RS etc, ditch unhosted concept, but build stuff on the Web Platform anyway. It's awesome.

So, 3 years ago

Unhosted project is started by Michiel de Jong with the aim to develop a protocol and reference implementation for this.

unhosted

Initial sponsoring by NLnet Foundation

nlnet

  1. Michiel had initial idea, is main spec author
  1. NLnet is funding network/Internet technologies (e.g. DNSsec, GPL v3, URI spec) and open source
  2. E.g. hood.ie also received a grant for their plugin architecture
  3. Send in short proposal, up to 30K, no questions asked.
  4. Next deadline May 1! Apply!

Skipping more history...

Fast forward to today

Example of a proprietary Unhosted app

Fargo.io is an outliner app that stores OPML files in your Dropbox

fargo
  1. Dave Winer (of RSS fame) created an app using Dropbox from the client
  2. The very app I used to prepare this talk

Even cooler: let the user choose!

Laverna is a note taking application that supports both Dropbox and remoteStorage

laverna
  1. This is what the architecture enabled developers to do

Part 2: remoteStorage

remoteStorage
  1. Let's talk about the only open protocol for this, though
  2. Water
  3. Check time

remoteStorage is a protocol of protocols

Instead of re-inventing the wheel, it creatively combines existing protocols/standards.

  1. Internet Draft
  2. Long-term goal is to take it to RFC
  3. New version every 6 months for now
  4. NOT AN RFC, looks official, but is not
  5. 3 steps

Step 1: Discovery

How to tell an app where my storage is?

  1. Not necessary with a central data store like e.g. Dropbox
  2. But crucial for letting a user choose any storage server on the Web

Webfinger

webfinger
  1. RFC 7033
  2. Can be used to discover information about people or other entities on the Internet

user@host

e.g. basti@5apps.com

  1. Using a user address in the form of user@host

Retrieve storage info

GET https://5apps.com/.well-known/webfinger?resource=acct:basti@5apps.com

  1. .well-known is actually another standard (RFC 5785)
  2. http://tools.ietf.org/html/rfc5785 Defining Well-Known Uniform Resource Identifiers (URIs)
  3. Also, format is a standard called JRD, pronounced Jared, JSON Resource Descriptor (delivered as application/jrd+json)
  4. JRD was introduced in https://tools.ietf.org/html/rfc6415 (Web Host Metadata)
  5. Also means you can put this on your own server

Step 2: Authorization

How to allow an app to access my data?

  1. And access only parts of it?
  2. And maybe with read-only permission?

OAuth 2.0

oauth
  1. RFC 6749
  2. Allow access to resources
  3. OpenID Connect using Webfinger + OAuth as well. Might have read our spec. :)
  4. Ripple will use this e.g. for their payment network

Request access to user data

GET https://5apps.com/rs/oauth/basti?redirect_uri=https://webmarks.5apps.com&scope=bookmarks:rw&client_id=https://webmarks.5apps.com&response_type=token
5apps auth
  1. Request read or read-write access to data categories (which are base directories)
  2. Currently only "Implicit grant flow" supported

Revoke anytime

5apps auth revoke

Step 3: Storage

How can an app read, write and delete data?

REST API (with CORS)

HTTP verbs: GET, PUT, DELETE

A few rules and required formatting for e.g. directory listings

  1. Important to keep it simple!
  2. Stay agnostic of technology
  3. Make it easy to implement on the server-side

Create/update

PUT /basti/testing/testfile.txt HTTP/1.1
Authorization: Bearer 1a2b3c4d5e
Content-Type: text/plain

Ohai, Hamburg!
HTTP/1.1 200 OK
Status: 200 OK
Content-Length: 0
Date: Fri, 25 Apr 2014 14:27:23 GMT
...

Read file

GET /basti/testing/testfile.txt HTTP/1.1
Authorization: Bearer 1a2b3c4d5e
HTTP/1.1 200 OK
Content-Length: 14
Content-Type: text/plain
Etag: "1C6NMyI94YVNaYqXlOJCfW"
...


Ohai, Hamburg!

Read directory

GET /basti/testing/ HTTP/1.1
Authorization: Bearer 1a2b3c4d5e
  1. Pretty much the only custom thing regarding our HTTP usage
  2. Contains a JSON-LD context (moar standards!)
  1. JSON-LD is a suite of W3C specs (went Rec not too long ago)
  2. Used to make sense of linked data on the Web

Delete file

DELETE /basti/testing/testfile.txt HTTP/1.1
Authorization: Bearer 1a2b3c4d5e
HTTP/1.1 204 No Content
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, PUT, DELETE
Date: Fri, 25 Apr 2014 15:03:00 GMT
...
  1. For the people who don't know it: this is what CORS headers look like
  2. Very simple spec, basically just headers like these and OPTIONS requests

Part 3: remoteStorage.js

remoteStorage
  1. Reference JavaScript client library
  2. Integrate remoteStorage in JS apps
  3. Water
  4. Check time

Quick facts

  1. Most libs I've seen neglect private mode, can break badly in a number of ways
  2. We could use some help from experienced JS folks

Integrate remoteStorage in JavaScript apps

remoteStorage connect widget
  1. Usually via this connect widget (but not required)
  2. This could use some serious design love (esp. mobile)
  3. Contribute and get your work in front of a ton of people!

Demo time!

This better be working, or I look like an idiot.

How do I use it?

When in doubt, consult API docs and/or ask on IRC (#remotestorage on Freenode)!

Configure and connect

  1. This gives your remoteStorage instance access to the data and lets users connect their storage

Option 1: Use existing data module

Option 1: Use existing data module

Option 2: Create your own module or extend one

... and if it makes sense, share it!

  1. One module per category/base directory
  2. Makes it possible for different apps to use the same data
  3. E.g. enter tasks in one app, track time on them in another one

Defining a module

Gives us:

Defining data types

Use JSON Schema to define object types (plus validations for free)

  1. JSON Schema is pretty awesome on its own! You should check it out in any case.

Storing objects

Use the defined data type to store objects

  1. This is where you can do much more complicated stuff, e.g. create thumbnails (insert canvas, resize, etc.)

Linked Data

remoteStorage.js automatically adds a JSON-LD context:

Unfortunately links not pointing to spec/schema yet. (Wanna help?)

  1. This is what ends up on the server and your the local cache
  2. Handled in the background
  3. Unfortunately not linking to anything yet :(

Sry, kind of beta

Common data modules are the least developed part of the project. Could use your help!

Not mandatory, but super useful.

  1. The data modules that is, not BaseClient
  2. E.g. shared functionality, moving to ES6 modules, etc.

Handling incoming changes

Coming soon

  1. Not sure you want to use a service with Condie Rice on the board, but...

Where to go from here?

  1. So that's it for my talk.
  2. Where to go from here?

Links

  1. Check out the protocol
  2. Check out the library
  3. Give feedback, report issues (tell us what's bad)
  4. Get hacking
  5. Get designing (e.g. widget, in front of a lot of people)
  6. Help improve docs (every character counts)
  7. Use protocol as base for your own archticture/product
  8. Wanna earn some (not much) money? (best for e.g. student hackers)
  9. Let's go for a beer (or a coffee or sth), core team in Berlin during next 2 weeks
  10. Unhosted Unconference

Thanks!

Sebastian Kippe – @skddc

/