Nicholas Kariniemi

Talk: JavaScript and Self Deception

Thu, 24 May 2018 00:00:00 +0000

JavaScript and Self-Deception: Why TypeScript is not enough

JavaScript is often derided for callback hell and for quirks around equality, number handling, type coercions and the “this” keyword. This is largely a straw man argument. Developers work around these shortcomings through conventions, tools, and libraries and problems caused by these kinds of quirks come up infrequently. This talk instead aims to explore the best ways that I know how to write JavaScript- using good conventions, using functional libraries, using types, using promises and immutable data- and motivate PureScript as a way of pushing the limits and bringing development in the JavaScript ecosystem to the next level.

Presented at the Reaktor Breakpoint 2018 conference on May 24, 2018.

Fear, trust and PureScript

Thu, 08 Mar 2018 00:00:00 +0000

In my previous post I argue that JavaScript is by default a low trust environment for programming, that ideas that build trust like optional typing, functional transformations, and immutability have severe trade-offs in JavaScript, and that choosing one of these ideas often means rejecting another. Despite that, JavaScript is a better environment than most popular languages for building trust, and the broader argument applies to most popular languages, typed or not.

In PureScript you get types, immutability, and functional programming “for free”– the trade-offs aren’t as steep. This is largely true of a few other languages as well, but let’s see what our discussion of fear and trust from the previous post looks like in PureScript from the same two perspectives: understanding the shape of data and changing data.

Read the full post on the Reaktor blog.

Fear, trust and JavaScript

Sun, 04 Mar 2018 00:00:00 +0000

As developers, we want to reduce fear of our code failing and increase trust in our code working well. Many developers working with JavaScript borrow useful ideas from functional programming and strongly-typed programming languages to reduce fear by transferring trust from developers to tools and the code. Ideas like optional types, functional transformations, and immutability can all help to write better JavaScript. When pulling these ideas together in JavaScript, however, they come with severe trade-offs, work together poorly, and ultimately fail in the goal of effectively transferring trust from developers to code and tools.

Read the full post on the Reaktor blog.

Hacker News (2018-10-27)

Refactoring 30000 lines of JS with types

Tue, 24 Jan 2017 00:00:00 +0000

30000 lines of client-side JavaScript. No tests. Two difficult TV deployment platforms with poor tooling. Strong dependencies on poorly documented external APIs. The task: add support for a third TV platform to the two supported platforms and switch to a new backend with a different API. How can we do this without breaking things?

One approach is to add tests incrementally- end to end tests to avoid breaking functionality and unit tests to aid with refactoring. But this was mostly user interface code on TV platforms with poor and varied tooling. Automated user interface testing on each platform would be challenging if possible at all. Manual testing on each platform was time consuming due to slow and inconsistent deployment tools.

Some parts of the application could be tested and this was a part of how we made this change. But the more interesting idea we tried was to refactor with types.

Read the full post on the Reaktor blog.

Wrapping JavaScript for PureScript

Mon, 05 Sep 2016 00:00:00 +0000

How do I wrap a JavaScript API to use from PureScript? I had a chance to think about this while wrapping the Screeps game API so I could play Screeps using PureScript.

Screeps is an MMO strategy game for programmers. You play by writing a JavaScript AI to control your minions. Everyone’s code runs 24/7 across hundreds of rooms in the same world.

PureScript is a small strongly typed language that compiles to JavaScript.

As a target for PureScript, the Screeps JavaScript API has some challenges. It makes heavy use of prototypes, a class hierarchy, and implicit effects. There’s some implicit mutable state. Performance is important, because the game caps the amount of CPU and memory you can use. But I gave a shot at wrapping it and I’m pretty happy with the result.

How do I…

As a relative newcomer to PureScript and strongly typed functional programming languages, I ran into a number of puzzles in wrapping the Screeps API. Here is what I ran into and how I ended up solving it.

How do I call a JavaScript function from PureScript?

Write a foreign import type declaration for how you want to call the function and a corresponding JavaScript function that actually calls the function (and handles currying). Borrowing an example from the PureScript Wiki:

PureScript:

foreign import joinPath :: FilePath -> FilePath -> FilePath

JavaScript:

exports.joinPath = function(start) {
  return function(end) {
    return require('path').join(start, end);
  }
}

In practice, though, writing these curried JavaScript functions is tedious and error-prone so you’ll want to use helper functions. The purescript-functions library provides some basic wrappers to allow you to do this instead:

PureScript:

foreign import joinPathImpl :: Fn2 FilePath FilePath FilePath

joinPath :: FilePath -> FilePath -> FilePath
joinPath = runFn2 joinPathImpl

JavaScript:

exports.joinPathImpl = require('path').join;

How do I wrap functions that use `this`?

The Screeps API is makes heavy use of object and singleton methods. For example, to make a creep attack a target you call

creep.attack(target)

PureScript doesn’t have classes or the concept of this. I represent this in PureScript as a function that takes in the object as the first argument. Calling the method on the object is handled with a little FFI helper function.

PureScript:

getActiveBodyparts :: Creep -> BodyPartType -> Int
getActiveBodyparts = runThisFn1 "getActiveBodyparts"

foreign import runThisFn1 :: forall this a b. String -> this -> a -> b

JavaScript:

exports.runThisFn1 = function(key){
  return function(self){
    return function(a){
      return self[key](a);
    }
  }
}

How do I wrap constant values?

The Screeps API has a large number of constants. I wanted to expose these to PureScript in a type-safe way without much runtime overhead. The solution I used was to wrap constants in newtypes.

PureScript:

newtype ReturnCode = ReturnCode Int
foreign import ok :: ReturnCode
foreign import err_not_owner :: ReturnCode
foreign import err_no_path :: ReturnCode

JavaScript:

exports.ok = OK;
exports.err_not_owner = ERR_NOT_OWNER;
exports.err_no_path = ERR_NO_PATH;

For those unfamiliar with newtypes, it provides a type alias that at runtime works the same as the underlying type. In theory this gives us the benefits of both worlds: compile-time type safety and run-time performance. If we never expose the constructor for the newtype, the only way for a user to obtain values e.g. of type ReturnCode is through the ways we provide. This allows the use of only the constant values that we have defined.

The type could also be defined with a foreign import:

foreign import data ReturnCode :: *

This might be nicer for some cases, but makes it harder to write, say, type class implementations because you can’t rely on generics.

How do I wrap a class hierarchy?

Screeps has a class hierarchy a few classes deep:

RoomObject
-> Creep
-> Structure
  -> OwnedStructure
    -> StructureContainer
    -> StructureSpawn
    -> ...etc

PureScript doesn’t have classes, prototypes, or inheritance. In particular, wrapping this in PureScript involves at least two challenges:

How can functions be defined once to work with any objects of a given class?
How can appropriately-typed sub-class instances be obtained when needed?

The solution to 2) is discussed elsewhere. My initial solution for 1) used a type class hierarchy like the following:

class IRoomObject a
class (IRoomObject a) <= IStructure a
class (IStructure a) <= IOwnedStructure a

foreign import data RoomObject :: *
instance iRoomObjectRoomObject :: IRoomObject RoomObject

foreign import data Structure :: *
instance iRoomObjectStructure :: IRoomObject Structure
instance iStructureStructure :: IStructure Structure

Generic functions can then be written that require any member of a certain type class:

room :: forall a. (IRoomObject a) => a -> Room
room = unsafeField "room"

This was less than ideal to use, though. Type classes aren’t very composable, they say, and in particular I was having difficulty defining ADTs that used type class instances. Type classes and ADTs don’t seem to mix well. You can’t, for instance, put a type class constraint on an ADT.

I ended up borrowing a trick from wxHaskell using phantom types. Instead of using type classes, I defined the class hierarchy using ADTs:

foreign import data RawOwnedStructure :: * -> *
foreign import data RawRoomObject :: * -> *
foreign import data RawStructure :: * -> *

foreign import data RawContainer :: *

type RoomObject a = RawRoomObject a
type Structure a = RoomObject (RawStructure a)
type OwnedStructure a = Structure (RawOwnedStructure a)

type Container = Structure RawContainer
type Controller = OwnedStructure RawController

The types the user uses are defined as type synonyms on the left, e.g. RoomObject or Container. Parent classes contain a phantom type a which carries the class hierarchy information. On the right side we have the “raw” types which are not used directly by users but are seen in type errors.

The room function from above ends up like this:

room :: forall a. RoomObject a -> Room
room = unsafeField "room"

This is simpler–no type class constraint– and works more nicely with ADTs and other constructs. How it works was a little tricky to me at first. Suppose we pass in a Container to our room function, which expects a RoomObject a. Container is a type synonym. If we expand out the type synonyms it ends up looking as follows:

  Container
= Structure RawContainer
= RoomObject (RawStructure RawContainer)
= RawRoomObject (RawStructure RawContainer)

The actual type defined by Container is

RawRoomObject (RawStructure RawContainer)

Notice how this contains the class hierarchy. So a function taking in RoomObject a will accept a Container because, looking at the expansion above, Container is equivalent to

RoomObject (RawStructure RawContainer) = RoomObject a

Pretty neat. Also, all of these types are defined only at compile-time and disappear at run time. In the end the JS code just passes in objects like it always does, but with strong compile-time guarantees about correct objects being passed within our code.

How do I parse function return values into the correct types?

The Screeps API has a number of functions that take in a constant specifying a type and return an object of that type. For example, room.find(FIND_CREEPS) finds all creeps in a room, room.find(FIND_FLAGS) finds all flags, and so forth.

In JavaScript this is not a problem, because you either assume the correct type or verify the type otherwise e.g. from a “type” field. For PureScript we would like to have this function return the appropriate type e.g. Creep or Flag.

Phantom types to the rescue. It turns out that we can tag our FIND_* constants with a phantom type and use this type tag to return an object of the correct type.

newtype FindType a = FindType Int

foreign import find_creeps :: FindType Creep
foreign import find_flags :: FindType Flag
foreign import find_construction_sites :: FindType ConstructionSite
foreign import find_my_spawns :: FindType Spawn

find :: forall a. Room -> FindType a -> Array a
find = runThisFn1 "find"

Some constants are not associated with the most specific type, though. For example room.find(FIND_STRUCTURES) returns all structures in a room, which may be one of any number of more specific structure types. We can’t know at compile time what types it is returning. In these cases I just return Structure Unit and then provide functions for safe casting to more specific sub-types.

toContainer :: forall a. Structure a -> Maybe Container
toContainer = unsafeCast structure_container

unsafeCast :: forall a b. StructureType -> Structure a -> Maybe b
unsafeCast t struc
  | structureType struc == t = Just $ unsafeCoerce struc
  | otherwise = Nothing

This checks the type of the object from the structureType field, which contains the string name of the type, and then uses unsafeCoerce to force it to the correct type. This works similarly to TypeScript’s user defined type guards. I’m not entirely happy with this solution but it works.

How do I handle functions that return null or undefined?

In JavaScript anything can be null or undefined and there’s no way of knowing. In PureScript nullable values are represented explicitly using Maybe types. We can parse values to Maybe values using a little FFI helper that just checks for null or undefined and returns the appropriate Maybe value.

PureScript:

findClosestByPath :: forall a. RoomPosition -> FindType a -> Maybe a
findClosestByPath pos findType = toMaybe $ runThisFn1 "findClosestByPath" pos findType

toMaybe :: forall a. NullOrUndefined a -> Maybe a
toMaybe n = runFn3 toMaybeImpl n Nothing Just

foreign import toMaybeImpl :: forall a m. Fn3 (NullOrUndefined a) m (a -> m) m

JavaScript:

exports.toMaybeImpl = function(val, nothing, just){
    if(val === null || val === undefined){
        return nothing;
    } else {
        return just(val);
    }
}

How do I handle functions that throw exceptions?

JavaScript functions can throw exceptions willy nilly. In PureScript it’s more common to use an Either type, for instance, to return either the value we want or an error. So we catch the exception and return an Either value:

PureScript:

findClosestByPath :: forall a. RoomPosition -> FindContext a -> Either Error (Maybe a)
findClosestByPath pos ctx = errorToEither \_ ->
  toMaybe $ runThisFn1 "findClosestByPath" pos (unwrapContext ctx)
  
errorToEither :: forall a. (Unit -> a) -> Either Error a
errorToEither fun = errorToEitherImpl fun Left Right

foreign import errorToEitherImpl :: forall a.
  (Unit -> a) ->
  (Error -> Either Error a) ->
  (a -> Either Error a) ->
  Either Error a

JavaScript:

exports.errorToEitherImpl = function(fun){
  return function(left){
    return function(right){
      try {
        return right(fun());
      } catch(e){
        return left(e);
      }
    }
  }
}

This admittedly gets a bit ugly in some cases such as the above example, where it turns out the Screeps API will either throw an exception, or return a value which is sometimes null or undefined. But that’s how the API is, and we keep our type guarantees.

It turns out we can make this easier by using functions from purescript-exceptions. try takes an effectful function that throws an exception and turns it into a function that catches exceptions and returns either the exception or the value you wanted. As we’ve handled all of the side effects, we can make it a pure (non-effectful) function again with runPure.

findClosestByPath :: forall a. RoomPosition -> FindContext a -> Either Error (Maybe a)
findClosestByPath pos ctx = runPure (try closestByPath)
  where closestByPath = toMaybe <$> runThisEffFn1 "findClosestByPath" pos (unwrapContext ctx)

How do I pass in arguments to overloaded functions?

For example, a number of Screeps functions take in either x and y coordinates or a RoomPosition (containing x and y coordinates) or a RoomObject (which exists at a certain x and y position).

PureScript doesn’t have function overloading, but you can handle it fairly neatly using ADTs as in the following:

data TargetPosition a =
  TargetPt Int Int |
  TargetObj (RoomObject a) |
  TargetPos RoomPosition
  
getDirectionTo :: forall a. RoomPosition -> TargetPosition a -> Direction
getDirectionTo pos (TargetPt x' y') = runThisFn2 "getDirectionTo" pos x' y'
getDirectionTo pos (TargetPos otherPos) = runThisFn1 "getDirectionTo" pos otherPos
getDirectionTo pos (TargetObj obj) = runThisFn1 "getDirectionTo" pos obj

Of course, this requires the caller to wrap the argument in the appropriate type constructor e.g. getDirectionTo pos (TargetObj obj).

How do I pass in a large number of optional parameters?

Some Screeps functions take in a JavaScript object which contains a number of optional fields. In JavaScript this works neatly because you just pass in an object with the fields you want. In PureScript it’s not as clean because optional fields have to be represented explicitly e.g. as Maybe values.

One seemingly common pattern for handling this in Haskell is to provide a default options object that has all of the options with their defaults. Users can then modify the object to include the options that they want with a relatively short syntax.

The tweak I made in my library is to make all of the options Maybe types and to only include the Just values in the final options object. The reason for this is that the way to indicate that an option is not used in the Screeps API is to exclude it. Including a default value, even undefined, may not be the same semantically.

moveTo' :: forall a e. Creep -> TargetPosition a -> MoveOptions -> Eff (cmd :: CMD, memory :: MEMORY | e) ReturnCode
moveTo' creep (TargetPt x y) opts = runThisEffFn3 "moveTo" creep x y (selectMaybes opts)

type MoveOptions = PathOptions
  ( reusePath :: Maybe Int
  , serializeMemory :: Maybe Boolean
  , noPathFinding :: Maybe Boolean )

type PathOptions o =
  { ignoreCreeps :: Maybe Boolean
  , ignoreDestructibleStructures :: Maybe Boolean
  , ignoreRoads :: Maybe Boolean
  , ignore :: Maybe (Array RoomPosition)
  , avoid :: Maybe (Array RoomPosition)
  , maxOps :: Maybe Int
  , heuristicWeight :: Maybe Number
  , serialize :: Maybe Boolean
  , maxRooms :: Maybe Int
  | o }

moveOpts :: MoveOptions
moveOpts =
  { ignoreCreeps: Nothing
  , ignoreDestructibleStructures: Nothing
  , ignoreRoads: Nothing
  , ignore: Nothing
  , avoid: Nothing
  , maxOps: Nothing
  , heuristicWeight: Nothing
  , serialize: Nothing
  , maxRooms: Nothing
  , reusePath: Nothing
  , serializeMemory: Nothing
  , noPathFinding: Nothing }

selectMaybes :: forall a. a -> JsObject
selectMaybes obj = unsafePartial $ selectMaybesImpl isJust fromJust obj

foreign import selectMaybesImpl :: forall a. (Maybe a -> Boolean) -> (Maybe a -> a) -> a -> JsObject
foreign import data JsObject :: *

JavaScript:

exports.selectMaybesImpl = function(isJust){
    return function(fromJust){
        return function(obj){
            var newObj = {};
            for(var key in obj){
                if(obj.hasOwnProperty(key) && isJust(obj[key])){
                    newObj[key] = fromJust(obj[key]);
                }
            }
            return newObj;
        }
    }
}

How do I store memory in a type-safe way?

Screeps allow you to store 2 MB of your own memory. The default way this is handled is that you store objects on the Memory object that are serializable to JSON. When your code is loaded on each tick, the memory gets deserialized. This deserialization also counts toward your CPU usage.

You can also handle memory at a lower level using RawMemory, which allows you to write your own serialization/deserialization in place of the default JSON.stringify and JSON.parse.

In PureScript we want all of our loaded data to be typed, of course, so we need easy ways to store and load typed data.

The approach I took is to use purescript-argonaut to allow you to store any data for which you can write DecodeJson and EncodeJson instances. You can use generics to avoid writing most of the boilerplate and you end up with something like the following to do encoding and decoding:

derive instance genericCreepState :: Generic CreepState
instance decodeJsonCreepState :: DecodeJson CreepState where
  decodeJson = gDecodeJson
instance encodeJsonCreepState :: EncodeJson CreepState where
  encodeJson = gEncodeJson

One interesting API quirk I ran into is that it turns out some functions implicitly modify the memory. This is not necessarily directly represented in the Screeps API, but in my PureScript wrapper those functions are tagged with the MEMORY effect to warn of this. This brings me to the next item.

How do I handle effectful code?

Screeps has a number of commands that have implicit side effects. When you give a creep a command, the command is implicitly executed at the end of the game tick. Some functions implicitly or explicitly modify stored memory. Others are time-dependent.

In PureScript implicit side effects are made explicit to make the code easier to think about and refactor. The classic Haskell example is that if a function can launch nukes as a side effect, we would like to know about it.

Well, Screeps has a function for launching nukes: launchNuke. We save ourselves from accidentally launching nukes by tagging it with an effect:

launchNuke :: forall e. Nuker -> RoomPosition -> Eff (cmd :: CMD | e) ReturnCode
launchNuke = runThisEffFn1 "launchNuke"

Returning an effect forces all callers to recognize the fact that this function has side effects. It makes it clearer what functions actually do and makes it easier to separate the simpler side-effect free functions from the more difficult side-effectful functions.

In the end I defined four effects:

-- | Execute a Screeps command before the next tick as a side effect e.g. to move a creep.
foreign import data CMD :: !

-- | Get or set mutable memory
foreign import data MEMORY :: !

-- | Global scope is cleared periodically, so values depending on global variables
-- | like Game and Memory need to be fetched dynamically. This effect enforces this.
foreign import data TICK :: !

-- | Time-dependent functions have different output depending on when they are called.
foreign import data TIME :: !

How do I get performance?

Code performance in Screeps is important. You are allowed a certain CPU/processing cap and if you exceed the cap your code is killed. The cap rises according to your level.

PureScript doesn’t have a runtime and compilation mostly includes only the bits of libraries that you use. Overall performance using this PureScript library is good enough and I suspect depends more on the user’s code than the library itself. Anecdotally my AI code performs close to its CPU cap (e.g. 40/40 CPU used for 2 rooms) where others seem to be able to perform much under the CPU cap (e.g. 10-20 CPU for 2 rooms). This is probably because the code is mostly a pure function with side effects at the edges. More performant code might spend more time in the Eff monad’s do notation. But it’s not something I’ve looked into much yet.

There are a few things I do in the library and in my code to try to eke out a little more performance:

Use type aliases or newtype wrapping to only use types at compile time.
Use pattern-matching where possible in place of e.g. chained Maybe values. In Haskell I suspect (Just a) <|> somethingElse wouldn’t execute somethingElse, but in PureScript you need to handle the laziness yourself. Pattern-matching maps directly to performant nested if statements.
Avoid inlining functions.
Use lazily-evaluated values via purescript-lazy where necessary.

Conclusions

Writing Screeps code in PureScript is fun. Tick-based gameplay maps nicely to modeling your AI as a pure function from memory and game state to new memory and commands to execute, with side effects executed only at the edges. Type-safety saves you from breaking everything in late night refactoring or frantic live-coding updates to save your base. Though to be fair it didn’t quite save my base from annihilation. Twice.

Wrapping JavaScript in PureScript requires more thought than writing TypeScript or Flow definitions for JavaScript. To do a proper job of it you have to grok the code you are wrapping. But it also allows you to create better compile-time guarantees about how your code works, which translates to more reliable and refactorable code.

Compared to compile-to-js languages like ClojureScript or GHCJS, it’s likely easier to get performant PureScript code because PureScript doesn’t have a runtime. The low-level tools like mutability and JavaScript hacks are also there to be used if necessary.

The Screeps PureScript library is here. Go forth and screep, my friend.

Edit 2016-9-13: Added comments about purescript-exceptions and defining constants with foreign import data.

Why CSP matters II: How do I know sync works?

Thu, 25 Feb 2016 00:00:00 +0000

How does one know if their synchronization algorithm works? I have a real-time synced grocery list app. In the last post I attempted to scrape by on synchronization by implementing differential sync based on an intelligent-sounding paper. In the end my implementation had subtle bugs and was difficult to debug. I hoped to find a better approach to the problem using the CSP language.

One challenge is just to make the problem understandable. If the CSP language can represent the problem clearly and succinctly, it already makes it easier to manipulate the algorithm, discover problem cases, and implement it properly. But can CSP actually prove that synchronization works? And perhaps more importantly, is it worth the effort?

CSP and CSP_M

CSP has a long and rich history in research and there are doubtless countless uses and resources of which I haven’t the foggiest. The books I found that I liked were Roscoe’s “The Theory and Practice of Concurrency” and Steve Schneider’s “Concurrent and Real Time Systems: the CSP approach”, in addition to The CSP Book from Hoare himself. But the important thing for me came from Roscoe’s book, which introduced me to the CSP_M language.

CSP is a mathematical language for representing processes that talk to each other. It allows you to model problems and then make assertions about those models. Modeling can be useful, as it helps you think about the problem, and assertions are grand. If you can make it work. Eventually. But to me this felt difficult to apply and not very well scalable to more complicated problems.

What CSP looks like

But CSP_M, as it turns out, is just a programming language. It’s a modern programming language, inspired by languages like Miranda and Haskell. But don’t be too worried if Haskell doesn’t inspire. It is a much stripped down and focused version of this.

So instead of writing intricate mathematical algorithms, we can just write programs and debug and test them. Better still, the CSP_M “IDE”, the FDR3 CSP refinement checker, comes with a REPL. You can write algorithms like you write code, load it into the REPL, explore it visually, and tweak it and see if it still compiles and passes assertions. Let’s try it out.

A simple algorithm: syncing broccolis with CSP_M

The goal is to keep grocery list items in sync between different clients, connected to a central database via one or more servers. Here is what the algorithm looks like, using broccoli:

Broccoli algorithm

The rest of this section walks through representing the broccoli algorithm using CSP_M. You can follow along by downloading FDR3 and my complete CSP_M program and opening the program using FDR3 (“fdr3 sync.csp”). Useful FDR3 commands to start out: :help to see what commands are available and :reload to reload the current file after changes have been made.

Client sync process

NUM_CLIENTS = 2
NUM_DB_STATES = 5
CLIENTS = {0..NUM_CLIENTS-1}
TIMES = {0..NUM_DB_STATES-1}

channel up, down, render:CLIENTS.TIMES

CLIENT(i, t) = 
   up!i!t -> CLIENT'(i, t)
[] CLIENT'(i, t)

CLIENT'(i, t) = 
   down!i?server_t
   -> render!i!server_t
   -> CLIENT(i, server_t)

For the client, I define two different processes which essentially behave as one process: CLIENT and CLIENT’. The CLIENT process either uploads a diff to the server when the user performs an action or behaves just like the CLIENT’ process. The CLIENT’ process takes a diff from the server, renders it, and then behaves like the CLIENT process. The t variable represents the client’s state and is incremented each time data is saved to the database. The client process has three events it can engage in: up, down, and render. Each event has a client number associated with it to identify the client and a state representing the database “time”.

How do we know if we implemented the process correctly? You can interactively explore the states of the process using :probe CLIENT(0, 0), which gives you a dialog like this:

The probe dialog shows the possible actions our process can engage in at any given state. It’s easy to run through the diagram, try out different scenarios, and assure oneself that the process works how it was supposed to.

Server sync process

SERVER(i, client_t) = 
   up!i?server_t
   -> save!i
   -> saved!i?new_server_t
   -> down!i!new_server_t
   -> SERVER(i, new_server_t)
[] report_queue?j:diff(CLIENTS,{i})?new_server_t
   -> if new_server_t == client_t
      then SERVER(i, client_t)
      else down!i!client_t!new_server_t
        -> SERVER(i, new_server_t)

The server sync process does two things. First, it takes in input from the client via the up event, saves it to the database, and sends any remaining changes back down to the client. Second, it reads in changes from other clients via (our representation of) the Datomic report queue and sends them down to the client, if they haven’t already been handled.

Database process

DB(t) =
   save?i
   -> saved!i!next_t(t)
   -> DB(next_t(t))

REPORTQUEUE(i) =
   saved?j:diff(CLIENTS,{i})?t
   -> REPORTQUEUE'(i, j, t)
REPORTQUEUE'(i, j, t) =
   saved?j':diff(CLIENTS,{i})?new_t
   -> REPORTQUEUE'(i, j', new_t)
[] report_queue!j!t -> REPORTQUEUE(i)

The database process DB is very simple. It takes a new state to save from the server sync process i and saves it, returning the id t representing the database after being saved. The report queue process REPORTQUEUE takes these saved events and puts them in a sliding buffer of size 1 for other server sync processes to consume. This is how changes are passed from one client to another. It is a sliding buffer of size 1 because we only care about the latest state of the database. If we don’t have time to process an event in between it doesn’t matter - we just diff with the latest state we have available and send that down to the client.

Process wiring

Finally, everything is wired together in the end.

SERVER_WITH_REPORTS(i, t0) = (SERVER(i, t0) [|{| report_queue |}|] REPORTQUEUE(i))
CONN(i, t0) = CLIENT(i, t0) [|{| up.i, down.i |}|] SERVER_WITH_REPORTS(i, t0)
CONNS(t0) = [| productions(saved) |] i:CLIENTS @ CONN(i, t0)
SYSTEM = DB(0) [|{| save, saved |}|] CONNS

All processes start with an initial state t=0. The CLIENT process communicates with the SERVER process with the up and down events. The server process in turn communicates with the REPORTQUEUE process via the report_queue event. Each client/server connection pair, represented by the CONN process, receives saved events synchronously from the database. Finally, all of the connections communicate with the database via the save and saved events.

Putting it all together

As I left out a few details, here is the full example:

NUM_CLIENTS = 2
NUM_DB_STATES = 10
CLIENTS = {0..NUM_CLIENTS-1}
TIMES = {0..NUM_DB_STATES-1}

channel up, down, render, saved, report_queue:CLIENTS.TIMES
channel save:CLIENTS

next_t(t) = (t + 1) % NUM_DB_STATES

CLIENT(i, t) = 
   up!i!t -> CLIENT'(i, t)
[] CLIENT'(i, t)

CLIENT'(i, t) = 
   down!i?server_t
   -> render!i!server_t
   -> CLIENT(i, server_t)

SERVER(i, client_t) = 
   up!i?server_t
   -> save!i
   -> saved!i?new_server_t
   -> down!i!new_server_t
   -> SERVER(i, new_server_t)
[] report_queue?j:diff(CLIENTS,{i})?new_server_t
   -> if new_server_t == client_t
      then SERVER(i, client_t)
      else down!i!new_server_t
        -> SERVER(i, new_server_t)

DB(t) =
   save?i
   -> saved!i!next_t(t)
   -> DB(next_t(t))

REPORTQUEUE(i) =
   saved?j:diff(CLIENTS,{i})?t
   -> REPORTQUEUE'(i, j, t)
REPORTQUEUE'(i, j, t) =
   saved?j':diff(CLIENTS,{i})?new_t
   -> REPORTQUEUE'(i, j', new_t)
[] report_queue!j!t -> REPORTQUEUE(i)

SERVER_WITH_REPORTS(i, t0) = (SERVER(i, t0) [|{| report_queue |}|] REPORTQUEUE(i))
CONN(i, t0) = CLIENT(i, t0) [|{| up.i, down.i |}|] SERVER_WITH_REPORTS(i, t0)
CONNS(t0) = [| productions(saved) |] i:CLIENTS @ CONN(i, t0)
SYSTEM = DB(0) [|{| save, saved |}|] CONNS(0)

Sync algorithm in CSP

So now we have a model representing our implementation. This is not the simplest model we could have come up with, nor is it the most complex. There are a lot of details that we left out. Client-server connections are not actually synchronous, even though we represent them as so. There is a certain amount of buffering in the system that is not modeled. We consider the diffing and patching algorithm only in very vague terms, passing around a single variable t to represent the database state. We don’t represent the possibility of lost packets or of the client disconnecting. We don’t represent users making changes as a separate event, instead tying it in directly to the moment the changes are sent to the server via the up event.

This is a simplified representation of the problem but already a useful one. Our algorithm is concise and unambiguous. We can visualize the algorithm and iteratively exploring the states of the algorithm using FDR3’s :probe SYSTEM.

But beyond that, this allows us to prove things about our algorithm.

Proving synchronization works: Sync one broccoli

What would we like to say about our algorithm using this model? Well, we want to prove that syncing “works”. What does it mean for syncing to “work”? Let’s start simple. Syncing works if I enter a broccoli on my phone and, at some point later, my wife sees broccoli on her phone. No other users or vegetables are involved.

Syncing a single broccoli

How can we verify this using our CSP_M model?

1) Allow only one change from one client

First, we want to restrict our implementation to allow only a single change from a single user. We do this by defining a processes that perform according to this specification and then forcing our implementation to synchronize with these processes on the relevant events.

Our first process restricts the implementation to only a single change:

MaxInputs(0) = SKIP
MaxInputs(n) = up?i?t -> MaxInputs(n-1)

The MaxInputs process performs n changes from any client i and then stops. If we set n=1 then it only allows one change. The question mark in up?i?t indicates that we are taking in i and t as inputs, which in this case we ignore. The SKIP event represents successful termination of the process.

A second process specifies that the input must come from a specific client:

OnlyClient(i) = up!i?t -> OnlyClient(i)

The OnlyClient process takes in inputs infinitely, but only from client i. The important syntactical difference here is in up!i?t. The exclamation point indicates that we are forcing i to be the specified client i i.e. only the one client is allowed to make changes.

To force our algorithm to behave according to these constraints, we tell it to synchronize with our constraint processes on the up event:

OneInputFromClientZero = (OnlyClient(0) [|{| up |}|] MaxInputs(1)) [|{| up |}|] SYSTEM

The system can only engage in up events when both of our other processes are ready to engage in up events. Thus the system is only allowed to take in a single input from client 0 and then cannot take any more inputs. Our constraints only affect the inputs, so it can still freely do whatever else that it wants, like upload to the server or save changes to the database. To verify that our constrained process works properly you can explore it with :probe OneInputFromClientZero.

2) Assert that the change makes it to the other client

Now, we want to show that a change on client 0 will make it to client 1. What does this look like from the outside i.e. what sequence of events would we expect to see if this works? We would expect to see an input/upload event from client 0 followed by a render event from client 1: <up.0.0, render.1.1>. In between a bunch of other events happen that we don’t care about right now.

Here is a process that does that:

SyncOneInput = up.0.0 -> render.1.1 -> STOP

The SyncOneInput process is our specification. This is how we hope our system will behave. We hope it will take input from client 0, render the state on client 1, and stop.

Finally, we prove that our system fulfills this specification by asserting that it does:

assert SyncOneInput [FD= OneInputFromClientZero 
  \diff(Events, union(productions(up.0), {render.1.1}))

What we assert here is that our constrained implementation OneInputFromClientZero must behave according to our specification SyncOneInput. If our specification takes in one input and sends it up to the server, our implementation must take in one input and send it up to the server. If our specification then, at some point after this, renders a new state on the other client, our implementation must also at some point render a new state on the other client.¹

We can run the assertion through FDR3 and see that it works. Syncing one broccoli works. This is like a simple end to end test for our algorithm. But let’s make it more interesting.

Proving synchronization works: Sync many vegetables

This app isn’t very useful if it only sends one broccoli from one person to another. What if I throw in tomatoes and my wife throws in milk and our friends who are visiting want zucchini and butternut squash?

Syncing many vegetables from many people

Instead of only allowing one client to make a single change, let’s allow any client to make any number of changes, up to maximum number of changes. They stop adding things at some point. Then we verify that each client properly renders the final changes. This is essentially a definition for eventual consistency.

First, we need to allow a larger number of inputs n instead of just one input. We can use our MaxInputs process from above:

MaxInputSystem(n) = SYSTEM [|{| up |}|] MaxInputs(n)

Next, we need to define what we actually want to see in our specification. If the system is restricted to n inputs, what we would like to see after all of the inputs have been entered is for each of the connected clients to render the final state. For example, if 9 inputs are allowed, we allow 9 inputs from arbitrary clients and then expect to see all of the clients to render the state after 9 changes i.e. engage in the render.i.9 event. Conceptually this is still a very simple specification, though the syntax gets more involved:

sequences({}) = {<>}
sequences(a) = {<z>^z' | z <- a, z' <- sequences(diff(a, {z}))}

renderAll(sequence, t) = ; i:sequence @ render!i.t -> SKIP

SyncAll(n) = |~| i:CLIENTS @ up!i!0 -> SyncAll'(n, n-1)
SyncAll'(n, 0) = |~| renderSeq:sequences(CLIENTS) @ renderAll(renderSeq, n); STOP
SyncAll'(n, m) = |~| i:CLIENTS, t:TIMES @ up!i!t -> SyncAll'(n, m-1)

Without going into the syntax, this allows input nondeterministically in any order from any client and then requires them all to render the final state in nondeterministic order.

The final assertion can be run with FDR3 and it passes. We can even change the number of clients from two to four and see that it still passes.

assert SyncAll(9) [FD= MaxInputSystem(9)
  \diff(Events, union(productions(up), {render.i.9 | i <- CLIENTS}))

What have we proved here? A similar end-to-end test might work as follows. We pull up four browsers. We randomly make n changes on a random window, stop for some period, and then verify that all of the browsers end up in the same state. This is a useful test. It shows that the system, including the parts outside of the algorithm, works in this specific situation. It means you probably didn’t screw up anything majorly.

But ultimately that test is also very limited. It randomly tests one single possible timing of events out of a huge space of possible timings of events. The exact behavior of the system, and whether it works or not, may depend on very specific timings. The test only tests one of these timings. The next time it runs it tests a different timing of events. Even if it fails, we have no good way of repeating the test. We just know that our system is “broken”. Somehow.

Our CSP assertion is much more powerful than an end-to-end test. Instead of testing one specific situation, it tests every single combination of timings that can occur with our system and says that in all cases our system will end up in sync. If the assertion passes, we know that the algorithm should work for any timing of events. If the assertion fails, it provides a single clear example of a sequence of events that causes the assertion to fail.

Conclusion: Does my sync algorithm work?

Our “eventual consistency” assertion says that given n connected clients, if we allow them to arbitrarily make k changes, eventually all clients will render those changes and stop changing. In this case I tested it with 4 clients and 9 changes, but this can be arbitrarily adjusted to any number of clients or changes as long as computing power suffices. On my computer two clients take a moment to verify, three take 36 seconds, and four take over twenty minutes. The parameters can be adjusted until you’re sure that the algorithm actually does work.

Note some of the things that this assertion does not say. It does not make any claims about timing. In theory synchronization could take a huge (finite) number of steps. Nor does not make any claims about the diffing/patching algorithm itself. We just assume it works.

The second, important part of verifying that the sync algorithm works is to make sure that the model actually represents our app in the important ways. It doesn’t help to write one model and implement another. It’s also easy to overlook real world aspects of the system like buffering or disconnections. Some of these could be handled simply enough outside of the model, as long as our model assumptions hold. Others might need to be included into the model.

Because the Clojure core.async library is based on CSP, it is a straightforward process to translate our model into Clojure. My final sync algorithm as it currently stands looks as follows.

Client sync process (source)

(defn sync-client! [initial-state to-server ui-state-buffer diffs full-syncs connected ui-state]
  (go (loop [client-state initial-state
             server-state initial-state
             awaiting-ack? false]
        (let [channels (if awaiting-ack?
                         [diffs full-syncs connected]
                         [diffs full-syncs connected ui-state-buffer])
              [event ch] (a/alts! channels)]
          (when DEBUG (println event))
          (when-not (nil? event)
            (condp = ch
              full-syncs (let [{:keys [full-state]} event]
                           (reset! ui-state full-state)
                           (when DEBUG (println "Full sync, new ui state tag:" (:tag @ui-state)))
                           (recur full-state full-state false))
              ui-state-buffer (let [new-ui-state @ui-state]
                                (if (state/state= server-state new-ui-state)
                                  (recur server-state server-state false)
                                  (do
                                    (when DEBUG (println "Changes, current ui state tag:" (:tag new-ui-state)))
                                    (>! to-server (event/diff-msg server-state new-ui-state))
                                    (recur new-ui-state server-state true))))
              diffs (let [{:keys [diff]} event]
                      (if (= (:shadow-tag diff) (:tag server-state))
                        ;; Our state is based on what they think it's based on
                        (let [;; Update server state we are based on
                              new-server-state (diff/patch-state client-state diff)
                              ;; Apply changes directly to UI
                              new-client-state (swap! ui-state diff/patch-state diff)]
                          (when DEBUG (println "Applied diff, new ui tag:" (:tag new-client-state)))
                          (when DEBUG (println "Applied diff, new server tag:" (:tag new-server-state)))
                          ;; If there are any diffs to reconcile, they will come back through input buffer
                          (recur new-client-state new-server-state false))

                        ;; State mismatch, do full sync
                        (do (>! to-server (event/full-sync-request))
                            (recur client-state server-state true))))
              connected
              ;; Need to make sure we are in sync, send diff
              (do
                (when DEBUG (println "Reconnected, sending diff"))
                (>! to-server (event/diff-msg server-state @ui-state))
                (recur client-state server-state true))

              (throw "Bug: Received a sync event on an unknown channel")))))))

(defn start-sync! [to-server new-ui-states diffs full-syncs connected ui-state]
  (let [ui-state-buffer (chan (a/sliding-buffer 1))]
    (a/pipe new-ui-states ui-state-buffer)
    (go (<! connected)
        (>! to-server (event/full-sync-request))
        (let [full-sync-event (<! full-syncs)
              initial-state (:full-state full-sync-event)]
          (reset! ui-state initial-state)
          (sync-client! initial-state to-server ui-state-buffer diffs full-syncs connected ui-state)))))

Server sync process (source)

(defn start-sync! [list-name to-client diffs full-sync-reqs db-conn report-queue]
  (let [id (rand-id)]
    (go (loop [client-tag nil
               awaiting-state? true]
          (let [channels (if awaiting-state? [full-sync-reqs diffs] [full-sync-reqs diffs report-queue])
                [event ch] (a/alts! channels)]
            (when-not (nil? event)
              (condp = ch
                diffs
                (let [{:keys [diff shadow-tag tag]} event
                      client-shadow-db (d/as-of (d/db db-conn) shadow-tag)
                      client-shadow-state (db/get-current-db-state client-shadow-db list-name)
                      a (debug-print (str id " " "Got diff from client: " shadow-tag " -> " tag))
                      {:keys [db-after]} (db/patch-state! db-conn list-name diff)
                      new-tag (d/basis-t db-after)
                      new-state (assoc (db/get-current-db-state db-after list-name) :tag new-tag)
                      new-shadow (assoc (diff/patch-state client-shadow-state diff) :tag tag)
                      return-diff (event/diff-msg new-shadow new-state)]
                  (debug-print (str id " " "Send diff to client : " tag " -> " new-tag))
                  (>! to-client return-diff)
                  (recur new-tag false))

                full-sync-reqs
                (let [current-db (d/db db-conn)
                      current-tag (d/basis-t current-db)
                      current-state (assoc (db/get-current-db-state current-db list-name) :tag current-tag)]
                  (debug-print (str id " " "Full sync client to : " current-tag))
                  (>! to-client (event/full-sync current-state))
                  (recur current-tag false))

                report-queue
                (let [tx-report event
                      new-db-state (:db-after tx-report)
                      new-tag (d/basis-t new-db-state)]
                  (if (>= client-tag new-tag)
                    ;; Already up to date, do nothing
                    (do (debug-print (str id " " "Got report " new-tag " but client already up-to-date at " new-tag))
                        (recur client-tag false))

                    ;; Changes, send them down
                    (let [new-state (assoc (db/get-current-db-state new-db-state list-name) :tag new-tag)
                          client-db (d/as-of (d/db db-conn) client-tag)
                          client-state (assoc (db/get-current-db-state client-db list-name) :tag client-tag)]
                      (debug-print (str id " " "Got report, send diff to client: " client-tag " -> " new-tag))
                      (>! to-client (event/diff-msg client-state new-state))
                      (recur new-tag false))))

                (throw (Throwable. "Bug: Received an event on unknown channel")))))))))

I think it works. I have a few niggling doubts about the interaction between buffering and the diffing algorithm. But if it doesn’t work, I have a solid way to make it better. It might not work because the implementation doesn’t match the model, in which case I update one or the other. Or it might be that the model itself is flawed. If the model is flawed, I should be able to write an assertion that fails and then change the model until it does not fail. All of these changes build on each other. The model can be iteratively improved without the need to start over from the beginning.

Try it out here. Pull up two browsers on the same URL and see if they sync. Use it for your family grocery run. If it doesn’t work, let me know. Or better yet, show how it doesn’t work based on the CSP algorithm, or by showing how the implementation doesn’t match the algorithm.

Why CSP matters

The point here isn’t that my CSP representation is a perfect model of my app and that my app must work in all situations because the model works. The point is that this CSP approach makes writing complicated concurrent or distributed systems easier. It is a relatively lightweight model checking approach that falls more in the category of advanced type systems than formal verification. We don’t need to manually manipulate math to see that our algorithm works. We just write a program in a funny (but modern) language, write tests in the form of assertions, and have our model checker brute force it to see if our tests pass or fail.

Porting the resulting CSP model to Clojure is relatively straightforward because Clojure’s core.async library is based on CSP. It would be even more straightforward for a CSP implementation in a language closer to CSP_M like Haskell’s CHP. The matching of models means that CSP can be used to solve complex problems using core.async in a way that would be more difficult using another approach like functional reactive programming or actors. It’s not that you couldn’t do it, but impedance mismatch makes CSP modeling less useful. Translation between languages may take more work and it’s more likely that the program doesn’t actually match the model.

This is not a silver bullet. It’s easy for models to grow large enough to make them time consuming to check, for instance. The practicality of this approach for my day-to-day problems is an open question. But the promise is tantalizing. Once I had sufficient understanding of how CSP and CSP_M work, writing a model and a specification was relatively straightforward. It feels more approachable and practical than writing distributed algorithms in CSP by hand, without sacrificing the power of the approach. With specifications that don’t care about most of the implementation details, algorithms can be refactored without fear of breaking basic premises. Instead of relying on an ad hoc implementation of an algorithm that someone on the Internet wrote, trusting that they knew what they were doing because they used fancy words and worked at Google, we can modify and verify the algorithm ourselves. Or we can throw it out and write our own algorithm entirely.

This is why CSP matters.

This post is duplicated on both my blog and the Reaktor blog.

Footnotes

What we are actually specifying is much more precise than my hand-waving explanation. We are saying that our specification process failures/divergences refines our implementation process. For further detail I direct you to the Roscoe book, which provides precise mathematical definitions. ↩

Why CSP matters I: Keeping things in sync

Tue, 23 Feb 2016 00:00:00 +0000

The core.async library is a well known library in the Clojure community for managing asynchronous programming. It is based on CSP or Communicating Sequential Processes, originally introduced by Tony Hoare in a 1978 paper. The fact that core.async is based in CSP is oft-mentioned in core.async introductions. But why should we care?

CSP as a mathematical modeling language has a long and rich history in academic research. But as a developer trying to Get Things Done, does CSP matter? Once a library written in CSP exists, is there any reason for users to care about the underlying CSP concepts? And are there practical reasons that a library based on the CSP language is useful in ways that, say, a functional reactive programming or actors library is not?

Rich Hickey, in the blog post introducing core.async, hints that there are. He talks about building upon the work of CSP and its derivatives and states that “CSP proper is amenable to certain kinds of automated correctness analysis”. Edsger Dijkstra, in the introduction to Tony Hoare’s CSP book, describes the CSP approach in glowing terms as showing the way to “what computing science could– or even should–be”.

Tickled by the possibility of upgrading my mental tools, I looked through Hoare’s CSP book. The book is well written. It presents a mathematical language for describing communicating processes, along with low level examples of how different forms of communication could be modeled.

Reading through the book, I felt as though there was some profound understanding to be had in the CSP language, but the understanding itself eluded me. The examples were very low level. It seemed like a fine theoretical abstraction that was too difficult and time consuming to apply to practical problems. I put the book back on my mental shelf, forgot about the promise of CSP, and went to the store to solve a more practical problem: getting groceries.

A practical problem: Keeping groceries in sync

Getting groceries sucks. Everyone has to do it, thinking about food takes mental energy, and the process is error prone and sometimes stressful. Most people fall back on simple strategies to make grocery shopping work. They get the things they always get, or they painstakingly write out and maintain paper grocery lists.

There are other solutions in this problem space, but my idea was to write a real-time synced grocery list web app. You add items to the list from recipes and everyone sees them immediately. You can split up in the grocery store with your buddies or wife and check off items as you find them. Everyone sees the whole time what is left to get. You all get out of the store quicker.

How do I keep these items in sync? That’s simple. Every time something changes, we send the change to everyone else. As every software developer knows*, the network is fast and reliable and available so the changes will make it through immediately and everyone will stay in sync.

This poor man’s solution actually worked remarkably well. Until it didn’t. My child would press something on my phone while I was weighing broccoli and I would lose my Internet connection and end up getting twice as many tomatoes as I needed, because my wife had picked them up while I was disconnected. The blue cheese would disappear from the list under mysterious circumstances. Mistakes were made. The dinner table suffered. Test users (my wife and two year old) didn’t appreciate my explanations about the complexities and prohibitive costs of proper synchronization algorithms for a small unfunded product development team.

Sync issues: duplicate items

Worst of all is the uncertainty. You’re never quite sure if everything is still on the list, or if items have been mysteriously “lost”, or if items that you check off have already been checked off by someone else. This is clearly not a proper solution for a software developer (nor a software developer’s wife).

Sync issues: missing items

I hunted around for better solutions to my sync problems. I could make everything synchronous, but that would be slow and only work online. Operational transformations developed for text-based collaboration sounded promising but probably more complex than necessary. Finally I settled on differential synchronization, described in a 2009 paper by Neil Frasier of Google.

Learning by implementing papers: Differential sync

Differential sync had a number of aspects that felt like a good fit for my use case. Everyone can edit all the time without blocking others. Local states naturally converge in an “eventually consistent” sort of manner. Finally and perhaps most importantly, someone had written a coherent paper that explained how to implement it.

How differential sync works

The basic approach in the client-server setup works as follows. Client and server start out in the same state. Both client and server maintain their own state and a “shadow” state representing the last synced state from the other party. When changes are made on a client, a diff is computed between the new state and the shadow state. The diff is tagged with the client and server versions that it is based on and sent off to the server. The server verifies the version numbers, patches the changes onto its own state and its shadow client state, and then compares the two. If there are differences, i.e. someone else made changes on the server in the meantime, the two new states are diffed and the diff is sent back to the client. Repeat ad infinitum. Go ahead and read the paper, it’s actually quite well written.

Differential synchronization guaranteed delivery method (source)

This is a simple idea on the face of it, but as always, the devil is in the implementation details.

How my implementation works

I implemented this as a Clojure and ClojureScript web app. You can try it out here ¹ and read the source here. The front end uses Om for the UI and is connected via WebSocket to a Clojure backend, which persists changes to a Datomic database. Synchronization is handled by a client-side core.async process (go block) communicating with a server-side core.async process. To update the UI when the state changes, it passes the new state to a Clojure atom and through Om/React magic the UI is re-rendered efficiently. To capture all UI state changes, it uses a nifty feature of Om that allows one to observe all UI data changes. We pass these via a core.async channel to our our client synchronization process.

App architecture

The diffing and patching algorithm works as follows. Both client and server start out in the same state containing a list of our grocery items, which I refer to as “grubs”:

{:grubs { "id1" {:text "2 cans cherry tomatoes" :completed false}
          "id2" {:text "cream" :completed false}
          "id3" {:text "4 T. red pesto" :completed false}
          "id4" {:text "1 yellow onion" :completed false}
          "id5" {:text "2 T. brown sugar" :completed false}
          "id6" {:text "1 garlic bulb" :completed false}
          "id7" {:text "cottage cheese" :completed false}}}

When one client makes some changes

{:grubs { "id1" {:text "2 cans cherry tomatoes" :completed true} ;; completed
          ;; removed "cream"
          "id3" {:text "4 T. red pesto" :completed true}         ;; completed
          "id4" {:text "1 yellow onion" :completed false}
          "id5" {:text "2 T. brown sugar" :completed false}
          "id6" {:text "2 garlic bulbs" :completed false}        ;; edited
          "id7" {:text "cottage cheese" :completed false}
          "id8" {:text "milk" :completed false}}}                ;; added

we compute the diff of these changes and send it on to the server:

{:grubs {:+ {"id1" {:completed true}
             "id3" {:completed true}
             "id6" {:text "2 garlic bulbs"}
             "id8" {:text "milk" :completed false}}
         :- #{"id2"}}}

The diff tells which items were added or modified under the “:+” map and the IDs of items that were removed under the “:-“ set. To update the server state, we then run through the diff and merge in the changes to the server state. Merges happen on a “last edit wins” basis.

How Clojure makes this nice

I want to stop for a moment to point out how elegantly the whole process works using Clojure. Diffing and patching is handled in a straightforward and relatively efficient manner using Clojure’s immutable data structures. UI updates are just a render call with the latest state. By using core.async for communication between our sync processes, our sync processes don’t actually care that the communication is going over the network. They take in messages and send out messages and all that matters is the type and timing of messages. This also makes it straightforward to test. We connect the processes directly to each other, spoof messages as necessary, and verify that the processes shoot out the correct messages.

Finally, storing the data in a Datomic database, which I had not used before, has been pleasant and has interesting properties. Datomic is an immutable data store. It works similarly to Clojure’s data structures in preserving the history of changes while using structure sharing for performance.

Having the entire history of changes instantly makes the grocery app much more valuable. It allows me to do things like look over my entire history of grocery shopping and see how often I cook burgers in a month. The app could suggest items to add to grocery lists based on items I have previously purchased. It could make recommendations based on users who have similar tastes.

This together with the fact that changes are synced immediately also has other interesting implications both from technical and privacy perspectives. You can derive a lot of personal information from this kind of data. You could analyze the timing of when items are completed to organize the grocery items by store section. You could find out when people shop for groceries, when they think about groceries, what they eat, what time zone they live in, how their tastes have changed, or how many times they change their mind about broccoli and end up getting a premade meal instead. Immutable data unlocks compelling possibilities but can also have serious privacy implications.

Another interesting property of using Datomic as a data store is from the synchronization angle. In a normal synchronization setting you would have to store a certain amount of history and sync clients from this history. If the client state is older than the history, then you would have to do a full sync, losing all changes from the client. With Datomic we have the entire history of changes. We can do a sync from any historical state of the database. In practice you would probably want to do a full sync for older client states anyway, but the idea is compelling.

If I had to implement this project in JavaScript, the complexity may have killed me well before this point. One way to do it would be to basically reinvent Clojure in JavaScript: use ImmutableJS or Mori for immutable data structures, use BaconJS streams to communicate events, separate React rendering and app data and use a single event loop in order to render and capture changes, and use event sourcing or the like to keep a history of database events. This would be a much more difficult project to tackle. Using Clojure makes it possible to tackle complex problems that would otherwise be too much trouble.

Why sync was still difficult

Even using Clojure, I ran into my own challenges with implementing the differential sync algorithm. The paper sometimes lacked detail. My use case was not identical to the paper’s use cases. My own architectural decisions did not always agree with the paper’s approach. As my implementation strayed from the strait and narrow path drawn by the paper, it became more and more uncertain whether the implementation worked.

In particular, I ran into the following challenges:

The paper assumes a traditional polling-based HTTP request model. The paper’s algorithm only allows one packet in flight at a time and otherwise seems to imply a traditional HTTP request based model. I had wanted to use WebSockets in order to receive changes from other clients as quickly as possible. It is unclear whether these two thoughts are compatible.
The paper assumes the server will keep the shadow state for a client even if the client is not connected. The paper does not describe the process of initialization, but assumes the client and server are in a “connected” state throughout (although packets may be lost). This means the server must maintain a representation of the client state even when the client is disconnected. This feels like an unreasonable restriction.
Code complexity. This synchronization requires careful coordination between sync processes and the UI and database. Refactoring core.async processes can also be more difficult than normal refactoring because go threads are implicit and used through macros (which have different scoping rules).

My implementation seemed to work, but after all was said and done I was left with a nagging feeling of uncertainty. The uncertainty that had plagued me with my initial ad hoc approach to synchronization was creeping back. Grabbing pepperoni slices from the grocery store freezer I was plagued by the thought that I was forgetting something. I checked and double-checked that blue cheese had made it onto the list. I asked my wife to show me her own list at times to see if my list looked the same. The lists did match. Except sometimes, inexplicably, they didn’t.

How can I know that my sync algorithm works?

I tried different tactics to overcome this uncertainty. One of the basic problems is that the problem was too large to easily think about. Error cases occurred in very distinct scenarios, and it was too difficult to know if changing one aspect of the algorithm would affect a completely different scenario.

I tried finding ways to represent the problem in a more conceptualizable manner. I drew diagrams and wrote out possible synchronization scenarios on paper. I thought about using statecharts or other diagramming languages to represent the problem more clearly.

I wrote test cases to verify that different aspects of the synchronization algorithm worked correctly and to verify that the big picture worked if you hammered on it. But testing concurrent code is fundamentally broken. You can write test cases. You can pull up many browsers and have them all hammer on the app and randomly disconnect and connect and see that the browsers end up in the same state. This helps you feel more confident, but ultimately it only tests a small part of the problem space. Concurrent errors may depend on very specific timings of events. How do you know if synchronization works in all cases?

In short I had failed. I didn’t feel more confident about my implementation of differential sync than I had felt about the ad hoc approach. I blamed Neil Frasier for not thinking his algorithm through completely. I blamed my implementation, which had grown in complexity and was difficult and time consuming to verify. I blamed core.async, which had failed me in my moment of need. But the thought nagged me that there had to be a better way to handle this problem. I needed to be able to write my own algorithm. I needed to upgrade my mental tools.

And I remembered the promise of CSP.

Continued in part II

This post is duplicated on both my blog and the Reaktor blog.

Footnotes

Go ahead and use the app if you like. I try not to break things but this is a hobby project so there are no guarantees. Different URLs correspond to different lists, which are synced independently. ↩

PureScript on Android

Tue, 13 Oct 2015 00:00:00 +0000

I’ve been playing around with PureScript on Android via React Native.

Here’s what a TodoMVC-style app looks like:

And here’s what the guts of the code look like:

render :: forall props eff. Render props AppState eff
render ctx = do
  (AppState state) <- readState ctx
  return $ 
    view [(style "container")] [
      text [style "title"] "todos",
      view [style "newTodoContainer"] [
        textInput [style "newTodo", 
                   P.value state.newTodo,
                   P.placeholder "What needs to be done?",
                   N.onChangeText \newTodo -> transformState ctx (updateNewTodo newTodo),
                   N.onSubmitEditing \_ -> transformState ctx addTodo]],
      listView [style "todoList",
                N.renderRow $ todoRow ctx,
                N.renderSeparator todoSeparator,
                N.renderHeader $ view [style "separator"] [],
                N.dataSource state.dataSource],
      view [style "bottomBar"] [
        view [style "filters"] [
           filterButton ctx state.filter All, 
           filterButton ctx state.filter Active,
           filterButton ctx state.filter Completed],
        text [style "clearCompleted", 
              N.onPress \_ -> transformState ctx clearCompleted] 
             "Clear completed"]]
        
main = do
  log "Running app"
  registerComponent "PureScriptSampleApp" component
  where
    component = createClass $ spec initialState render
    dataSource = listViewDataSource initialTodos
    initialState = updateDataSource $ AppState { nextId: 18, 
                                                 newTodo: "", 
                                                 todos: initialTodos, 
                                                 dataSource: dataSource, 
                                                 filter: All }

Check out the example source code or the React Native wrapper code.

This wrapper code builds on Phil Freeman’s purescript-react low-level wrapper for React, adding support for various elements used in Android. It is a work in progress. Functional afficionados will raise their eyebrows. Android gurus will throw their hands up in disgust. But it is PureScript. It is Android. And it is awesome.

The state of Clojure on Android

Thu, 23 Apr 2015 00:00:00 +0000

Note that this post was written in April of 2015, based on experiments run on relatively current Android devices of the time. The situation may be different now.

Clojure on Android suffers from the slow startup times of the Clojure runtime. The Lean Clojure compiler projects promise fast startup times and performance at the cost of dynamism and complexity. Does it work?

How do you know if anything works? You test it. You set up some experiment that you think models the problem. You make your change, run the experiment with and without the change, and see what happens. You draw conclusions and quibble about whether you tested what you thought you tested and whether the results mean anything.

Here’s what I benchmarked and what I think it means. But first a bit of background.

What is Lean Clojure?

The idea of Lean Clojure is to remove some of the dynamism of Clojure for performance. Over the summer of 2014 two projects were developed related to this. Alexander Yakushev developed the Skummet lean Clojure compiler based on the standard Clojure compiler. Reid McKenzie worked on the Oxcart compiler based on the Clojure in Clojure compiler. The Oxcart compiler compiles a rather limited subset of Clojure so it is not presented here.

The Skummet compiler works something like the following.

Clojure namespaces set up a dynamic mapping between symbols, vars, and functions. This mapping looks basically like this:

At run time Clojure calls functions like this, in decompiled JVM bytecode:

RT.var("clojure.core", "cons").getRawRoot().invoke(args);

Skummet changes this by dropping out the middle men, the symbols and vars, to get something like this:

Invoking functions gets a lot simpler:

clojure.core$cons.invoke(args);

Skummet does other things related to metadata and workarounds to preserve dynamism when necessary, but I believe this is the meat of it.

It’s fairly obvious why this would be expected to improve Clojure execution speed. Clojure is mostly functions and this simplifies almost every function execution. This should speed things up by reducing overhead and making it easier for the virtual machine to optimize by inlining and doing whatever other black magic virtual machines do.

It’s perhaps less immediately obvious why this is expected to improve startup times. When the Clojure runtime loads a program, it loads all of the namespaces that are used in the program right at the start. When a namespace is loaded, it sets up the mapping from symbols to vars to functions. It also sets metadata on those vars. This isn’t a complicated process, but you need to create your var objects, create your metadata objects, and assign them to the appropriate places. It takes some time. Skummet cuts out a lot of this work, and so should reduce startup times.

That’s the theory. But does it work?

Does Lean Clojure work?

I took five benchmarks from the Computer Language Benchmarks Game, two little benchmarks of my own, and ran the benchmarks on the Nexus 5 and Nexus 7 on both Dalvik and ART. Here are the results:

The same benchmarks presented with only startup times by test environment:

Each benchmark opens an Android activity and performs some task. The hello benchmark just prints “Hello world”. The dependencies benchmark does something trivial with two library dependencies, RxClojure (or RxJava) and Transit. The others execute algorithms specified in the Computer Language Benchmarks Game.

The programs are written in Java and Clojure and compiled using the Java, Clojure, and Skummet compilers. Each benchmark is executed thirty times and the results are averaged.

What might we conclude from this?

Clojure on Android apps start slowly (2+ seconds minimum)

Well, duh, you’re thinking. The benchmarks give a bit of the scope of the problem, though. On Android Dalvik, Clojure apps take a minimum of nearly two seconds to start on the Nexus 5 phone and 2.5 seconds on the Nexus 7 tablet. The Nexus 5 is a relatively new phone and probably faster than most phones on the market, so the general case for performance is likely to be worse.

The dependencies benchmark performs a trivial task with two library dependencies and has startup times exceeding 2.5 and 3.5 seconds for the same device setups. This suggests a fairly fast scaling up of startup times. Actual apps would likely have more dependencies and code and take significantly longer.

ART helps (1.5+ seconds minimum)

The latest version of Android, Lollipop, uses the new ART virtual machine to execute apps in place of Dalvik. This improved startup times in these benchmarks by about about 20-30% on average.

Lean Clojure helps even more (0.7+ seconds minimum)

Lean Clojure cuts Clojure on Android startup times in half across the board, dropping them from around 1.5 to 2.5 seconds to around 0.7 to 1 seconds. Run time performance seems to be on par with standard Clojure, though these benchmarks are poor tests of performance.

What does this mean?

Lean Clojure works. Skummet cuts Clojure on Android startup times in half in these benchmarks. But it’s not good enough.

A half a second is about the minimum amount of time needed to execute Clojure code on Android on Skummet. Times for actual programs are likely to be significantly higher. This might not be a problem for apps that are loaded once and run for a long period, but for many types of development this is just too long.

If the lean Clojure project were continued, it seems likely it would bring this down to an acceptable range. Dependency shaking and inlining functions could make a large difference. Tools like ProGuard could make additional improvements. There are also other possible directions for Clojure on Android like ClojureScript plus third party frameworks such as Titanium or the upcoming Facebook React Native.

But Android is still waiting for it’s Swift.

More details

The rather verbose version of this post is in my thesis.

Solving Clojure Boot Time

Wed, 19 Mar 2014 00:00:00 +0000

Clojure programs start slowly because they load the clojure.core namespace before doing anything useful. Loading the clojure.core namespace loads the Java class files for all of the functions in clojure.core and sets up Vars to point to new class instances corresponding to each function.

This is slow. Simple desktop Clojure programs start about 35x more slowly than their Java counterparts. Clojure Android apps start as little as 6x more slowly than their Java counterparts, but the base start time is much higher so the problem is worse.

In the very best case a Clojure on Android app currently starts in about 1.7 seconds. More normal cases likely take at least 3-5 seconds. Jakob Nielsen in his book Usability Engineering suggests a few rules of thumb for UI response time. A user perceives as much as 0.1 seconds as instantaneous. Delays of more than 0.2 seconds are noticeable and delays of more than one second should have some indication to the user that something is happening. Ten seconds or more and you will lose your user’s attention completely.

Currently Clojure on Android apps fall in the “noticeably slow” range. This is also true for command line tools like Leiningen, even on reasonably fast computers. Reducing startup delay to 0.5 seconds on Android would get us into the range of noticeable delay not requiring a loading screen. This means we’re looking for about a 4-6x improvement in boot times.

This amount of speedup will probably require dramatic changes to the Clojure runtime. About half of the boot time on the desktop is due to loading the Java classes themselves. This suggests that the Clojure bootstrap process is just doing too much. We need to do dramatically less work during the Clojure runtime boot process.

So here are a few ideas for how to improve Clojure start time. These are mostly suggested by other people and I have added my own thoughts regarding each of them. I break them out broadly into three categories: do less work, do it faster, or do it later.

Do Less Work

What work are we doing in the bootstrap process that is unnecessary? How can we eliminate this work to speed up our bootstrap process?

1. Stripped, application-specific runtime

Most of the work of bootstrapping Clojure programs relates to setting up dynamic variables for Clojure functions. Many of these functions will never be used, but dynamic features like runtime evaluation and compilation require access to all functions in the Clojure language.

But what if we decide that we don’t need runtime evaluation or compilation, at least for production? Then there’s no sense in keeping around all of these extra functions that are never used. This idea is to generate a specific stripped production runtime for the application being developed. Instead of including the normal Clojure runtime/compiler with your app, the app would use this new stripped runtime. The runtime would differ from the normal runtime in the following ways:

All core functions that are not used are stripped out from the Clojure runtime and therefore not loaded in core initialization.
Runtime compilation and evaluation functions are removed.
Unused metadata is also removed.

This is a fairly easy idea to test. Take a few Clojure programs, see what clojure.core functions they rely on, strip the remaining functions from clojure.core source, recompile the Clojure runtime, and run the program using the new runtime.

Using this approach I was able to cut the run time for a Hello World program in half. This could possibly be reduced even further. This is a great improvement, but it is essentially the best case scenario. This best case scenario is not good enough. Most programs will use a much larger fraction of clojure.core and will see much smaller speedups.

Do It Faster

How can we do the same bootstrap work faster? In what ways can we optimize this process?

2. Optimized runtime loading

This idea I vagely label “optimize runtime loading”. Currently the Clojure runtime (by which I mostly mean the clojure.core namespace) is compiled in almost the exact same way as any other Clojure code. It is also run in the same way as any other namespace. While this is elegant from a language development perspective, is there some way we could optimize clojure.core loading so that it works faster?

Every time a Clojure program is run it does the exact same work in setting up a Namespace object for clojure.core. Is there some way we can do this work once, save the result to a single binary file, and then load the file at one time? I don’t know. It seems like there could be a way but I don’t know what it is.

Otherwise maybe there are shortcuts we could take for compiling clojure.core that would not work in the general case for Clojure code. I don’t know what those shortcuts might be, but I think they exist.

Do It Later

Do we need to do all of this work on boot? Why can’t we defer setup work until we actually need it?

3. Modularized clojure.core

This idea is, on the face of it, quite simple. Break clojure.core into smaller namespaces. Overall Clojure 1.5.1 has 591 publicly available functions or variables via ns-publics. Is this a lot? I don’t know. Direct comparisons with non-functional languages is a bit difficult. Scala and Java would appear to have fewer classes and methods in their core libraries and Ruby appears to have more functions, but I cannot say for sure.

The Clojure core namespace does provide a lot of functionality: bitwise operations, regular expressions, type inspection, concurrency operations, multifunctions, Java interop, operations for the core data structures, structs, transients, namespaces, vars, hierarchies, protocols, metadata, compilation and evaluation. Probably a number of these could be broken into separate namespaces.

But here’s the challenge. Here’s my chart of the function interdependencies in clojure.core:

Each of those nodes is a function in clojure.core. The lines indicate which other functions in clojure.core the function relies on.

Now tell me how that breaks down cleanly into separate namespaces.

There’s one cluster on the bottom left that relies solely on defn. That could maybe be pulled out. On the other hand, I believe those functions work mostly by deferring to clojure.lang.RT, so pulling them out might not help much.

If there are too many interdependencies in clojure.core, then breaking it down further would make loading time worse rather than better. If a function depends on several functions in other namespaces then loading multiple namespaces takes longer than loading one.

4. Lazy initialization of Clojure functions

This is an idea presented by galdolber, who has worked on a Clojure compiler for Objective C. Here’s my understanding of how this idea might work.

Clojure core bootstrapping sets up a Var to point to every function in the clojure.core namespace. In Java bytecode the setup work looks like this:

Var const__cons = RT.var("clojure.core", "cons");
AFn const__consMeta = RT.map(metaForCons);

const__cons.setMeta(const__consMeta);
const__cons.bindRoot(new core.cons());

The end result looks kind of like this:

The Namespace object for clojure.core is set up with a mapping of Symbols to Vars corresponding to functions. Each Var in turn points to its own metadata and an instance of the Java class file which implements the Var functionality.

When you call a function, it is done like this (in pseudo-Java bytecode):

// (cons args)
RT.var("clojure.core", "cons").getRawRoot().invoke(args);

This first fetches the core namespace and then fetches the Var in the namespace corresponding to cons. From the Var we can then grab the current value with getRawRoot and invoke it.

But why do we need to set up everything beforehand when a namespace is loaded? Why can’t we just load and set up functions as we need them? galdolber’s idea would move the Vars into their corresponding class instances and initialize them only when they are first used. This also sounds fairly similar to Laurent petit’s comment on my other post. I picture the result something like this:

We keep Clojure’s dynamic binding features, as we still use mutable Namespaces and Vars, while pushing the time to load and initialize functions to the first time they are used.

This could be a good idea. It could also cause hiccups in performance. Right now all loading penalties are paid cleanly when a namespace is loaded. Under this idea if you use a function that has dependencies across clojure.core you may pay the loading time penalty at an awkward point.

This could also be combined with the stripped runtime idea. We throw out unneeded functions and load the functions we do need lazily. It could work. But is it enough?

In the Objective C case galdolber quotes a speedup from 2.3 seconds to 900 milliseconds. This is about a 2.6x speedup. Stripping our runtime could give a little more. A similar speedup in Android would drop our fastest startup times from about 2 seconds to 770 milliseconds. The average case could be twice that. This is good, but based on those numbers it’s not good enough. Prove me wrong, galdolber.

Do Less Work, Faster, and Later

The problem with many of the above ideas is that they don’t seem to go far enough. We need dramatic speed improvements. This probably requires dramatic changes to the way Clojure works.

5. Lean JVM runtime

This idea makes the most dramatic changes but also has the most compelling potential. This is also what Daniel Solano Gómez has been proposing all along and echoes some of what mikera has been saying. It has taken me several months to understand what is going on well enough to appreciate this idea.

Clojure dynamism

Clojure is a dynamic language. Two key features that Clojure supports as a dynamic language are dynamic binding of vars and namespaces and dynamic incremental compilation and evaluation.

Dynamic binding is in some ways a surprising feature for a language that touts its immutability. All variables defined with def and functions defined with defn are implemented using mutable Vars. The Vars are stored within mutable Namespaces. Either can be changed at any time. This means that every function call requires at least two levels of indirection: first get the Namespace, then the Var, then the function itself, and finally call the function. Refer to the diagram further up.

The second important feature is dynamic incremental compilation and evaluation. Clojure is a very flexible language. Almost anything that can be done at compile time can also be done at run time. It includes built in evaluation ^[1] and compilation. Among other things, this enables the power of the Read Eval Print Loop (REPL). Combined with dynamic binding of namespaces and namespace variables, this gives a programmer a substantial amount of power to change their program while it is running.

In development, both of these features are very useful. Dynamic binding allows you to redefine functions for testing and development and gives more tools to work with external libraries that you do not control. Runtime evaluation and compilation combined with dynamic binding enable the short feedback loops of a powerful REPL development environment.

But how important are these features in production? Dynamic compilation seems to be used hardly at all in production. Use of dynamic binding in production is usually discouraged. In the case of Android development, dynamic recompilation is not even possible in the same way. When using a Clojure library from a Java environment it seems unlikely these features would be useful. In both of these last cases fast startup time and runtime performance seem more important.

The Idea

The premises of a lean JVM runtime are the following:

The program is Ahead of Time (AOT) compiled.
After compilation the program does not need to be redefined.
Run time performance is the main priority.

Based on these premises, the following changes would be made to the Clojure compiler and runtime:

Dynamic binding is removed. Namespaces compile to classes. Namespace variables and functions compile to static fields or methods. The Var is probably removed entirely.
Dynamic compilation and evaluation are removed. Functions that rely on runtime compilation and evaluation such as eval, compile, load, and so forth are eliminated.

How will this improve Clojure startup times?

When variables compile to static methods and fields there are fewer classes to load and they can be loaded lazily. This means there is a lot less to load at boot and boot times should be dramatically shorter.
Immutable functions and namespace variables can be directly referenced. This removes the two levels of indirection from fetching mutable namespaces and fetching mutable vars. For boot time or when loading classes, this means there is a lot less work to do.

How else could this improve Clojure performance?

Removing Var and Namespace indirection should significantly improve runtime performance.
Static compilation enables dead code elimination. The size of packaged Clojure apps could be reduced significantly.
Static compilation enables other compiler optimizations that are not possible under a dynamic, “nothing is certain” environment.

But is it still Clojure?

In theory the default Clojure compiler and runtime could be used for development. The same REPL development story would work, tests could redefine variables, and any other dynamic features would be available.

At run time, however, it would be a different story. A large laundry list of features would be dropped from the Clojure runtime. The lean runtime would be fast but lack the elegant symmetry and power of near equivalent run time and compile time features. It could also limit some of the interop with existing libraries which rely on dynamic rebinding of variables.

In many ways this would be similar to ClojureScript. ClojureScript has a subset of Clojure functionality. Among other differences, it lacks runtime evaluation and compilation, vars, and much of the runtime access to variables. This lean runtime would provide a similar subset of Clojure functionality, while taking it further in some respects.

It also looks like Common Lisp has corresponding features that allow the developer to specify what is compiled for speed and what is interpreted for flexibility. Some helpful Common Lisp guru could tell if these features are relevant. Scala would be another source for inspiration, as it generates static code much different than Clojure’s code.

Summary

The Clojure bootstrapping process needs to do much less work for Clojure programs to start in a reasonable time for library or Android applications. I have presented here five different ideas for achieving faster start times in Clojure:

Stripped, application-specific runtime
Optimized runtime loading
Modularized clojure.core
Lazy initialization of Clojure functions
Lean JVM runtime

The only idea that I feel goes far enough to achieve reasonable performance on Android is the lean JVM runtime idea. It is probably the most difficult of these ideas to implement but it is the idea I will try to move forward with.

There are lots of unanswered questions here. The implementation details of the lean JVM runtime are still vague. What do you think? Am I missing important details? Am I in completely over my head? Are there other good ideas out there? What would you like to see?

[1] Clojure is cited as a purely compiled language in a number of places. But then why does this look so much like interpretation? Is Java bytecode involved in some way that I can’t see right now? Update: as per Alex Miller’s comment, yes, it is actually compiled.

Edit 2014-03-19: removed all references to interpretation as per Alex Miller’s correction.

</small>

Nicholas Kariniemi

Talk: JavaScript and Self Deception

Fear, trust and PureScript

Fear, trust and JavaScript

Refactoring 30000 lines of JS with types

Wrapping JavaScript for PureScript

How do I…

How do I call a JavaScript function from PureScript?

How do I wrap functions that use this?

How do I wrap constant values?

How do I wrap a class hierarchy?

How do I parse function return values into the correct types?

How do I handle functions that return null or undefined?

How do I handle functions that throw exceptions?

How do I pass in arguments to overloaded functions?

How do I pass in a large number of optional parameters?

How do I store memory in a type-safe way?

How do I handle effectful code?

How do I get performance?

Conclusions

Why CSP matters II: How do I know sync works?

CSP and CSPM

A simple algorithm: syncing broccolis with CSPM

Client sync process

Server sync process

Database process

Process wiring

Putting it all together

Proving synchronization works: Sync one broccoli

1) Allow only one change from one client

2) Assert that the change makes it to the other client

Proving synchronization works: Sync many vegetables

Conclusion: Does my sync algorithm work?

Why CSP matters

Footnotes

Why CSP matters I: Keeping things in sync

A practical problem: Keeping groceries in sync

Learning by implementing papers: Differential sync

How differential sync works

How my implementation works

How Clojure makes this nice

Why sync was still difficult

How can I know that my sync algorithm works?

Footnotes

PureScript on Android

The state of Clojure on Android

What is Lean Clojure?

Does Lean Clojure work?

What does this mean?

More details

Solving Clojure Boot Time

Do Less Work

1. Stripped, application-specific runtime

Do It Faster

2. Optimized runtime loading

Do It Later

3. Modularized clojure.core

4. Lazy initialization of Clojure functions

Do Less Work, Faster, and Later

5. Lean JVM runtime

Clojure dynamism

The Idea

How will this improve Clojure startup times?

How else could this improve Clojure performance?

But is it still Clojure?

Summary

How do I wrap functions that use `this`?

CSP and CSP_M

A simple algorithm: syncing broccolis with CSP_M