Real-time, asynchronous    collaboration on JSON

Tomek Wytrębowicz

@tomalecpl

tomalec

JS❤ON

- simple,

- lightweight,

- flexible,

- composable,

- cross-platform

JS❤ON

 

- simple,

- lightweight,

- flexible,

- composable,

- cross-platform

communication problems

- traffic,

- altering nested nodes,

- blinking UI,

- documenting constraints,

- concurrent changes,

- freezing UI,

- ...

but...

TL;DR

bower install PuppetJs

Pattern for remote JSON data synchronization.


That uses JSON-Patch over HTTP or WebSocket,

with optional Operational Transformations.

Standards

 Magic  Math

 Magic

JSON

 - JS object literal

 - Array

 - "string"

 - Number

 - Boolean

 - null

IETF RFC 4627

ECMA-404

{
    "nestedObject": {},
    "array": [],
    "string": "foo",
    "number": 1,
    "boolean": true,
    "null": null
}

JSON

few implementations listed at
json.org

JSON

- data

- state

- view-model

JSON

- data

- state

- view-model

Needs to be changed often

- trafic?

- readability?

- constraints?

{
    "status": false,
    "error": "email_not_found",
    "loginProvider": false,
    "message": "E-mail wasn't found. First time here? Welcome, signing up is fast ;)",
    "redirect": "",
    "userBar": "",
    "userData": {
        "userId": 0,
        "firstName": "",
        "lastName": "",
        "email": "",
        "city": "",
        "postalCode": "",
        "address": "",
        "country": ""
    }
}
{
    "status": false,
    "error": "wrong_login_info",
    "loginProvider": false,
    "message": "Incorrect username or password!",
    "redirect": "",
    "userBar": "",
    "userData": {
        "userId": 0,
        "firstName": "",
        "lastName": "",
        "email": "",
        "city": "",
        "postalCode": "",
        "address": "",
        "country": ""
    }
}

Clean & lightweight changes

JSON Patch

{
    "status": false,
    "error": "email_not_found",
    "loginProvider": false,
    "message": "E-mail wasn't found. First time here? Welcome, signing up is fast ;)",
    "redirect": "",
    "userBar": "",
    "userData": {
        "userId": 0,
        "firstName": "",
        "lastName": "",
        "email": "",
        "city": "",
        "postalCode": "",
        "address": "",
        "country": ""
    }
}
{
    "status": false,
    "error": "wrong_login_info",
    "loginProvider": false,
    "message": "Incorrect username or password!",
    "redirect": "",
    "userBar": "",
    "userData": {
        "userId": 0,
        "firstName": "",
        "lastName": "",
        "email": "",
        "city": "",
        "postalCode": "",
        "address": "",
        "country": ""
    }
}
[
    {"op": "replace", "path": "/error", "value": "wrong_login_info"},
    {"op": "replace", "path": "/message", "value": "Incorrect username or password!"}
]

RFC 6902

application/json-patch+json

JSON Patch

{"op": "add", "path": "/where/to/add", "value": "new value"}

add

{"op": "replace", "path": "/what/to/replace", "value": 1}

replace

{"op": "remove", "path": "/what/to/remove"}

remove

paths are JSON Pointers - RFC 6901

{"op": "test", "path": "/this/node/should/be/equal", "value": 7}

test

{"op": "move", "from": "/from/where/", "path": "/to/where"}

move

{"op": "copy", "from": "/from/where/", "path": "/to/where"}

copy

JSON Patch

 trafic?

 readability?

 constraints?

 awareness of each node/widget/component API?

- lightweight

- self-explanatory

- validation system built in

- shared standard on altering document directly

"op": "test"

✓

✓

✓

✓

 

JSON Patch is a JSON ❤

So your environment already can work with it

"Can work" - but who will apply it?

bower install fast-json-patch

 npm

jsonpatch.com

22 implementations

10 languages (incl. C#, Ruby, Python, Java, Go,..)

jsonpatch.apply(myObject, receivedPatch)

Duplex?

jsonpatch.observe(myObject, function(patches){...});

Move Forward

fast-json-patch  handles more than 1,200,000 ops/sec

- tiny packages,

- efficient applications,

- HTTP Patch,

- WebSocket

Now, it starts to look like real-time collaboration

but...

Constraints?

Sequential order

not that much, isn't it?

but hey!

JavaScript

Networking

Computing

Human interaction

peers changes

Async!

Async communication

{
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Keynote",
            "done": false
        },
        {
            "title": "Real-time JSON collabo",
            "done": false
        }
    ]
}
{
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Real-time JSON collabo",
            "done": false
        }
    ]
}
[{ "op": "remove", "path": "/talks/0"}]
[{ "op": "replace", "path": "/talks/0/done", "value": true}]

2.

1.

{
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Keynote",
            "done": false
        },
        {
            "title": "Real-time JSON collabo",
            "done": false
        }
    ]
}
[{ "op": "remove", "path": "/talks/0"}]
[{ "op": "replace", "path": "/talks/0/done", "value": true}]
{
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Real-time JSON collabo",
            "done": true
        }
    ]
}

Async communication

2.

1.

{
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Keynote",
            "done": false
        },
        {
            "title": "Real-time JSON collabo",
            "done": false
        }
    ]
}

Locking UI, after request was send, until server responds?

Versioned JSON Patch

versionedjsonpatch.org

{
    "version": 1,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Keynote",
            "done": false
        },
        {
            "title": "Real-time JSON collabo",
            "done": false
        }
    ]
}
[
    { "op": "test", "path": "/version", "value": 2},
    { "op": "replace", "path": "/version", "value": 3}
    { "op": "remove", "path": "/done/0"}
]
[
    { "op": "test", "path": "/version", "value": 1},
    { "op": "replace", "path": "/version", "value": 2},
    { "op": "replace", "path": "/talks/0/done", "value": true}
]

both peers should agree where to store the version. 

 

/__meta/document_version

This could be as well

1.

2.

Versioned JSON Patch

versionedjsonpatch.org

{
    "version": 1,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Keynote",
            "done": false
        },
        {
            "title": "Real-time JSON collabo",
            "done": false
        }
    ]
}
{
    "version": 2,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Keynote",
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "done": false
        }
    ]
}
{
    "version": 3,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Real-time JSON collabo",
            "done": false
        }
    ]
}
[
    { "op": "test", "path": "/version", "value": 2},
    { "op": "replace", "path": "/version", "value": 3}
    { "op": "remove", "path": "/talks/0"}
]
[
    { "op": "test", "path": "/version", "value": 1},
    { "op": "replace", "path": "/version", "value": 2},
    { "op": "replace", "path": "/talks/0/done", "value": true}
]

✓

✓

2.

1.

Do I really have to bother about this?

bower install json-patch-queue

// create queue
var myQueue = new JSONPatchQueue("/path_to_version", jsonpatch);
// to compose versioned JSON Patch, to be send somewhere?
var versionedPatchToBeSent = myQueue.send(regularpatch);
// to apply/queue received versioned  JSON Patch
myQueue.receive(myObject, reveivedVersionedPatch);

your JSON does not have to be modified

myObject.path_to_version === undefined; // true

Constraints?

Don't interrupt me, while I'm talking!

We cannot apply changes simultaneously.

Distributed changes

Track 1 - Alice

Track 2 - Bob

{
    "version": 1,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}
{"op": "test", "path": "/version", "value": 1},
{"op": "replace", "path": "/version", "value": 2},
{"op": "replace", "path": "/talks/1/started", "value": true}
{"op": "test", "path": "/version", "value": 1},
{"op": "replace", "path": "/version", "value": 2},
{"op": "remove", "path": "/talks/0"}

A

B

Version does not match

Lightnings didn't started!

Should we wait for something?

Or ingore or block such behavior?

Track 1 - Alice

Track 2 - Bob

✓

{"op": "test", "path": "/version", "value": 1},
{"op": "replace", "path": "/version", "value": 2},
{"op": "replace", "path": "/talks/1/started", "value": true}
{"op": "test", "path": "/version", "value": 1},
{"op": "replace", "path": "/version", "value": 2},
{"op": "remove", "path": "/talks/0"}

A

B

{"op": "test", "path": "/version", "value": 1},
{"op": "replace", "path": "/version", "value": 2},
{"op": "replace", "path": "/talks/1/started", "value": true}
{"op": "test", "path": "/version", "value": 1},
{"op": "replace", "path": "/version", "value": 2},
{"op": "remove", "path": "/talks/0"}
{
    "version": 1,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}
{
    "version": 1,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}

✓

{
    "version": 2,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": true,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}
{
    "version": 2,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}

B

A

Hey! It's 2015! server push, distributed operations, WebSockets.
Collaboration, we've said!

Multiple Versioned JSON PATCH

local version

enfroces sequential queue

remote version(s)

specifies point in time when peers were in sync, base for conflict resolutions

versionedjsonpatch.org

Just add a version for each peer.

Track 1 - Alice

Track 2 - Bob

{
    "alice_version": 0,
    "bob_version": 0,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}
{"op": "test", "path": "/bob_version", "value": 0},
{"op": "test", "path": "/alice_version", "value": 0},
{"op": "replace", "path": "/alice_version", "value": 1},
{"op": "replace", "path": "/talks/1/started", "value": true}
{"op": "test", "path": "/alice_version", "value": 0},
{"op": "test", "path": "/bob_version", "value": 0},
{"op": "replace", "path": "/bob_version", "value": 1},
{"op": "remove", "path": "/talks/0"}

That's first Bob's change, based on Alice's initial version

Here is first Alice change, based on Bob's initial version 

Multiple Versioned JSON PATCH

A

B

Track 1 - Alice

Track 2 - Bob

{
    "alice_version": 0,
    "bob_version": 0,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}
{
    "alice_version": 0,
    "bob_version": 0,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}

✓

✓

{"op": "test", "path": "/bob_version", "value": 0},
{"op": "test", "path": "/alice_version", "value": 0},
{"op": "replace", "path": "/alice_version", "value": 1},
{"op": "replace", "path": "/talks/1/started", "value": true}
{"op": "test", "path": "/alice_version", "value": 0},
{"op": "test", "path": "/bob_version", "value": 0},
{"op": "replace", "path": "/bob_version", "value": 1},
{"op": "remove", "path": "/talks/0"}

A

B

{"op": "test", "path": "/alice_version", "value": 0},
{"op": "test", "path": "/bob_version", "value": 0},
{"op": "replace", "path": "/bob_version", "value": 1},
{"op": "remove", "path": "/talks/0"}

B

{"op": "test", "path": "/bob_version", "value": 0},
{"op": "test", "path": "/alice_version", "value": 0},
{"op": "replace", "path": "/alice_version", "value": 1},
{"op": "replace", "path": "/talks/1/started", "value": true}

A

- nothing related to this change

Hmm.. version does not match, but Alice and Bob know what they changed since that time.

we may apply it  ☃

- I was changing same array ☹

that's the conflict that we can resolve

{"op": "test", "path": "/bob_version", "value": 1},
{"op": "test", "path": "/alice_version", "value": 0},
{"op": "replace", "path": "/alice_version", "value": 1},
{"op": "replace", "path": "/talks/0/started", "value": true}
{
    "alice_version": 1,
    "bob_version": 0,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": true,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}
{
    "alice_version": 0,
    "bob_version": 1,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}

Multiple Versioned JSON PATCH

bower install json-patch-queue

// create queue
var myQueue = new JSONPatchQueue(["/local_version", "/remote_version"], jsonpatch);
// to compose versioned JSON Patch, to be send somewhere?
var versionedPatchToBeSent = myQueue.send(regularpatch);
// to apply/queue received versioned  JSON Patch
myQueue.receive(myObject, reveivedVersionedPatch);

your JSON does not have to be modified

myObject.path_to_version === undefined; // true

But how to resolve those conflicts?

How to transform operations?

Magic

voo-doo?

merging, rebasing?

inverting?

AI?

Operational Transormation (OT)

well researched algorithmic problem

https://en.wikipedia.org/wiki/Operational_transformation

Google/Apache Wave

OT

Algebra of transformations

A

{"op": "replace", "path": "/talks/1/started", "value": true}
{"op": "remove", "path": "/talks/0"}

B

B∘A

{"op": "remove", "path": "/talks/0"}

A∘B

{"op": "replace", "path": "/talks/0/started", "value": true}

keep it

shift index

OT

Track 1 - Alice

Track 2 - Bob

{
    "alice_version": n,
    "bob_version": 0,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Fast and fpsious",
            "started": true,
            "done": true
        },
        {
            "title": "Real-time JSON collabo",
            "started": true,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}
{
    "alice_version": m,
    "bob_version": x,
    "title": "WebCamp Zagreb 2015",
    "talks": [
        {
            "title": "Real-time JSON collabo",
            "started": false,
            "done": false
        },
        {
            "title": "Lightning Talks",
            "started": false,
            "done": false
        }
    ]
}
\begin{array}{c} A^1_0\\ \vdots\\ A^n_0\end{array}
A01⋮A0n\begin{array}{c} A^1_0\\ \vdots\\ A^n_0\end{array}
\begin{array}{c} B^1\\ \vdots\\ B^x_m\end{array}
B1⋮Bmx\begin{array}{c} B^1\\ \vdots\\ B^x_m\end{array}
B_m^x
BmxB_m^x
B_m^x \circ A_0^{m+1} \circ \ldots \circ A_0^n
Bmx∘A0m+1∘…∘A0nB_m^x \circ A_0^{m+1} \circ \ldots \circ A_0^n
\text{plus, can clear } A_0^1, \ldots, A_0^m \text{ from pending list}
plus, can clear A01,…,A0m from pending list\text{plus, can clear } A_0^1, \ldots, A_0^m \text{ from pending list}

OT

6^6 \cdot ? = \text{a lot!}
66⋅?=a lot!6^6 \cdot ? = \text{a lot!}

combinations of operations

werid path relations

copy ≈ replace

x ∘ test =  x

move ≈ add + remove

luckily

You don't have to do it by yourself

bower install json-patch-ot-agent json-patch-ot

// create queue
var myQueue = new new JSONPatchOTAgent(
                JSONPatchOT.transform, 
                ['/localVersion', '/remoteVersion'], 
                jsonpatch
             );
// to compose versioned JSON Patch, to be send somewhere?
var versionedPatchToBeSent = myQueue.send(regularpatch);
// to apply/queue/transform received versioned JSON Patch
myQueue.receive(myObject, reveivedVersionedPatch);

your JSON does not have to be modified

myObject.path_to_version === undefined; // true

So far, works on production, use by many clients for more than a year now without any vital issue or change.

❤ math

Putting all together

JSON Patch,

JSON Patch Queue,

Multiple Versioned JSON Patch,

OT,

…

Puppet

that's it.

change it here or there

bower install PuppetJs

// Defines a connection to a remote PATCH server, 
// gives an object that is persistent between browser and server
var puppet = new Puppet();
// ..
// use puppet.obj
puppet.obj.someProperty = "new value";

Puppet

- WS/HTTP

- specific remote(s):

  - client-server,

  - p2p / n2n

- SPA/not

- OT/not

- ...

Puppet

"There is a custom element for that"

bower install puppet-client

<puppet-client ref="nodeToBind"></puppet-client>

which you can be easily bind to any native/polymer/angular app, via DOM API

What now?

asynchronous,

distributed,

concurrent,

real-time,

easy,

consistent!

collaboration.

implementations:

JS - PuppetJs, & co.
C# - Starcounter

eventually-

UI with server-driven view-models, plus SPA

example: Puppet team use case

HTTP Patch text/html

HTTP Patch application/json-patch+json

No glue-code,

no REST API parsers,

just any template binding (HTML-JS) + puppet binding (client-server)

Maintenance

Composability

everything is a nested JSON tree

Security

business logic on server

Imagine there's no delay,

It's easy if you try

{"op": "test", "path": "/lyrics/JohnLennon/Imagine/version", "value": "final"}
{"op": "replace", "path": "/lyrics/JohnLennon/Imagine/lines/0", "value": "Imagine there's no delay"}

Thanks!

Tomek Wytrębowicz

@tomalecpl

tomalec

Specs:

JSON Patch - https://tools.ietf.org/html/rfc6902

JSON Pointer - https://tools.ietf.org/html/rfc6901

Proposed conventions:

Versioned JSON Patch - https://github.com/tomalec/Versioned-JSON-Patch

Libraries used:

Fast  JSON Patch - https://github.com/Starcounter-Jack/JSON-Patch

JSON Patch Queue - https://github.com/PuppetJs/JSON-Patch-Queue

JSON Patch OT Agent - https://github.com/PuppetJs/JSON-Patch-OT-agent

JSON Patch OT - https://github.com/PuppetJs/JSON-Patch-OT

PuppetJs - https://github.com/PuppetJs/PuppetJs
<puppet-client> -  https://github.com/PuppetJs/puppet-client

Visualization

 http://tomalec.github.io/PuppetJs-operational-transformation/visualization.html​

Implementations:

JS browser, (complate node.js support in progres) https://github.com/PuppetJs/puppetjs

C# http://starcounter.io/

App examples

http://starcounter.io/sample-apps/