Tuesday, December 3, 2013

Structuring Single Page Applications

Knockout js, Require js, Templating

1. I came up with a custom solution to a problem we were having at my job. We needed to be able to add components to the screen and then reuse that screen real estate later. Therefore we wanted to dispose of old components and prevent memory leaks. In the past I had experience with Backbone js and two frameworks written for it, Chaplin and Marionette. I found Marionette's solution to be the best, it had the idea of a Region. A Region represents an area of the screen usually an element. It controls opening and closing a view inside it's area. This allows you to composite the screen up into Regions, also taking care of making sure a parent Region closes any child Regions it knows about.

2. When initially designing the application we wanted to be able to split up the javascript files to make for easy development. In order to make a quicker load in production we wanted to be able to recombine and minify those files into one. Require js allows us to do all of these things easily.

3. Finally we wanted to split up our knockout templates into separate files, I used handlebars, and would use a "View" to apply a knockout viewmodel to a rendered template inside an element. Something like this ko.applyBindings(new BlahViewModel(), this.el);

Final thoughts on this approach and combination of technologies. I found knockout js to be extremely productive and I wasted less time doing simple things by using knockout's expressive templates and great bindings. Dependency tracking proved extremely useful and very versatile as well. Using require js and templating (handlebars in my case) to split up the javascript files and the markup/knockout stuff made the application much more maintainable. Falcon js seems to combine alot of the things I did by hand but I don't see any mention of the idea of Regions. http://stoodder.github.io/falconjs/#view

Backbone js, Marionette js, Require js

Look to my previous articles for a detailed explanation about these technologies. All in all has many of the benefits as the knockout stack above but not nearly as productive as knockout. Knockout has great support for binding html components to javascript viewmodels which saves LOTS of time.

Tuesday, November 19, 2013

Infinite scroll performance with lots of elements

The problem when using infinite scroll with LOTS of elements is the browser gets bogged down.

This guy eloquently gives a great explanation of a solution.

We had to deal with a similar problem on FoldingText. As the document grew larger, more line elements and associated span elements were created. The browser engine just seemed to choke, and so a better solution needed to be found.
Here's what we did, may or may not be useful for your purposes:
Visualize the entire page as a long document, and the browser viewport as the lens for a specific part of the long document. You really only have to show the part within the lens.

So the first part is to calculate the visible view port. (This depends on how your elements are placed, absolute / fixed / default)

var top = document.scrollTop;
var width = window.innerWidth;
var height = window.innerHeight;
Some more resources to find a more cross-browser based viewport:

Second, you need a data structure to know which elements are visible in that area

We already had a balanced binary search tree in place for text editing, so we extended it to manage line heights too, so this part for us was relatively easy. I don't think you'll need a complex data structure for managing your element heights; a simple array or object might do fine. Just make sure you can query heights and dimensions easily on it. Now, how would you get the height data for all your elements. A very simple (but computationally expensive for large amounts of elements!)
var boundingRect = element.getBoundingClientRect()
I'm talking in terms of pure javascript, but if you're using jQuery $.offset$.position, and methods listed here would be quite helpful.
Again, using a data structure is important only as a cache, but if you want, you could do it on the fly (though as I've stated these operations are expensive). Also, beware of changing css styles and calling these methods. These functions force redraw, so you'll see a performance issue.

Lastly, just replace the elements offscreen with a single, say <div> element with calculated height

  • Now, you have heights for all the elements stored in your Data structure, query all the elements that lie before the visible viewport.
  • Create a <div> with css height set (in pixels) to the sum of the element heights
  • Mark it with a class name so that you know its a filler div
  • Remove all the elements from the dom that this div covers
  • insert this newly created div instead
Repeat for elements that lie after the visible viewport.
Look for scroll and resize events. On each scroll, you will need to go back to your data structure, remove the filler divs, create elements that were previously removed from screen, and accordingly add new filler divs.
:) It's a long, complex method, but for large documents it increased our performance by a large margin.


I'm not sure I explained it properly, but the gist of this method is:
  • Know the vertical dimensions of your elements
  • Know the scrolled view port
  • Represent all off-screen elements with a single div (height equal to the sum of all element heights it covers for)
  • You will need two divs in total at any given time, one for elements above the visible viewport, one for elements below.
  • Keep track of the view port by listening for scroll and resize events. Recreate the divs and visible elements accordingly

Great Linkedin demonstation of the problem and solution.

Mobile devices have less memory and CPU power compared to Desktop computers. If you render a very long list in the HTML, you run the risk of crashing the device. This makes it challenging to build large, interactive HTML5 apps for mobile devices. Native technologies provide UITableViewController to build long, infinite scrolling lists. UITableView contains reusable UITableViewCells which are optimized for memory, performance and responsiveness . For HTML5, we did not have any solution. So we set out to build one!

Technique #1: unloading images

Technique #2: hiding pages

Technique #3: removing pages

Technique #4: avoid scaling and box-shadow

Technique #5: minimizing DOM nodes


Grid of elements with variable heights and a sorted order

You have a grid of elements each with a fixed width and a variable height. Example. Images that have fixed width and a variable height because you want to preserve the aspect ratio.

The problem you want to align them into a grid but simple floating them leaves large gaps in vertical space between elements.

Great further explanation of the problem, taken from link below.

"To my knowledge, there's no way to fix this problem with pure CSS (that works in all common browsers):
  • Floats don't work.
  • display: inline-block doesn't work.
  • position: relative with position: absolute requires manual pixel tuning. If you're using a server-side language, and you're working with images (or something with predictable height), you can handle the pixel tuning "automatically" with server-side code.
Instead, use jQuery Masonry."



1. This sorts the elements by height.

            arrange.elements.push(new Array($(this).height(), $(this).clone()));
 for(var i = 0; i < arrange.elements.length; i ++)
          if(arrange.elements[i][0] > current)
              current = arrange.elements[i][0];
              elem = arrange.elements[i][1];
              index = i;

This is pretty slow by the way.

2. Use jquery masonry or isotope



3. Absolute positioning

Jquery Waterfall Plugin

4. Solution we settled on after playing with all of the above techniques.

We organize the elements into their respective sections one for each column, this way we can both insert sorted data into them and preserve reasonable vertical spacing.

So for n columns you would have markup like the following.

<section class="mediaItems overview" data-bind="foreach: columns">
<div class="column-div" data-bind="unveil: $root.everyNthMediaItems($index())">
<img data-bind=" attr: { src: url}" />

We precompute the items that will be in each column to make it faster loading on the dom aka (everyNthMediaItems). That way we start with sorted data and then run everyNthMediaItems to put that data into columns thereby preserving the sorting and keeping vertical spacing correct.


Thursday, October 17, 2013

Make your JQuery plugin loadable with Require.JS

A simple and easy way to make your JQuery plugin work in an AMD and NON-AMD context is the  following snippet. See how it cleverly uses the factory.

(function (factory) {
if (typeof define === 'function' && define.amd) {
    // AMD. Register as an anonymous module.
    define(['jquery'], factory);
} else {
    // Browser globals
}(function ($) {

    $.fn.jqueryPlugin = function () {
        // Put your plugin code here


JQuery Architecture

I found the following code example on JQuery's architecture and found it very helpful. One thing to note is the great use of ensuring JQuery is always called with a new instance. Relevant snippet is the following:
if (!(this instanceof foo))
      return new foo(arg);

(function() {
  var foo = function(arg) { // core constructor
    // ensure to use the `new` operator
    if (!(this instanceof foo))
      return new foo(arg);
    // store an argument for this example
    this.myArg = arg;

  // create `fn` alias to `prototype` property
  foo.fn = foo.prototype = {
    init: function () {/*...*/}

  // expose the library
  window.foo = foo;

// Extension:

foo.fn.myPlugin = function () {
  return this; // return `this` for chainability

foo("bar").myPlugin(); // alerts "bar"

Thursday, September 26, 2013

Knockout.js Gotchas

  1. Knockout's observableArray dependency tracking breaks if you bind with a call to sort,reverse, etc.
    1. Therefore don't do things like data-bind="foreach items.sort()", you instead bind normally with sortedItems() and then create a computed property sortedItems that sorts items. Be careful though this technique will sort items in place and not just the new computed property sortedItems.
    2. http://stackoverflow.com/questions/18729643/ko-observablearray-update-breaks-when-binding-with-sort

Friday, August 30, 2013

In Search of an Understandable Consensus Algorithm with CoreOS



In Search of an Understandable Consensus Algorithm

JQuery UI slider with more than 2 handles

$('#slider').slider({min: 1, max: 100, values: [0, 3, 15, 25],
    slide: function(event, ui) {
        var index = $(ui.handle).siblings('a').andSelf().index(ui.handle);
        var values = $(this).slider('values');
        return (index == 0 || ui.value > values[index - 1]) &&
            (index == values.length - 1 || ui.value < values[index + 1]);
    stop: function( event, ui ) {
        alert("VAL 1=" + ui.values[0] + "/n" + "VAL 2=" + ui.values[1] + "/n" +"VAL 3=" + ui.values[2] + "/n" +"VAL 4=" + ui.values[3] + "/n");

Monday, July 29, 2013

Awesome Ruby Stack with Vagrant

http://www.vagrantup.com/ + vmware = Provisioning of your development environment

https://puppetlabs.com/ - automate installation of needed programs

https://github.com/capistrano/capistrano for deploying ruby code

Ruby Servers

Friday, July 26, 2013

Backbone JS lessons learned ... Ongoing

  1. Two major frameworks to consider, Chaplin and Marionette help give you guidelines when building applications in Backbone. After using both I like Marionette the most especially for the way it manages the screen with Regions and Layouts.
  2. When developing widgets in Backbone you will most likely want to the sync the Model when the view changes and the reverse when the model is updated. Think this through very carefully I have seen many bugs in this type of logic.
    1. One bug occurred when a developer built widgets that would get themselves into a circular event firing loop. He unknowingly did this by listening to the change event on the element, the issue arose because this same view listened to model change events. The model would be fetched and fire a change event which would update the view. Then the change event on the element would fire causing the model to update and so on ... Initially he solved this by checking if the value on the model was different than the value on the element. This got around the issue, but the issue could be avoided by not listening to change events on the element. Instead we should listen to user events on the element like onBlur, ...
  3. Decide early on how you are going to handle the loading part of the application, will widgets simply listen to change events and render when the data comes in, sort of having them pop up when available. Another approach is explicitly showing a loading sign while all the ajax calls are fetching models and collections.
  4. If you are going to need to relate your models and collections you will most likely pick between backbone relational and backbone associations. Be aware that if you pick backbone relational you will only be allow one model with the same id per type. In order to fix this you will likely have to call YourModelType.findOrCreate throughout your code instead of simply instantiating a new model. TODO I need to look into how backbone associations handles this problem.
  5. Routing across your application should also be well thought out. Are you going to need to programmatically navigate from one page of your application to another. Also think about your handling of where your javascript is deployed, will you need to know the context path in order to route properly?

Saturday, July 20, 2013

Functors, Applicatives, and Monads
  • Functor
    • Functors apply a function to a wrapped value:
    • Easy example to remember it by:  getPostTitle <$> (findPost 1)
    • A value can be in a context. Now when you apply a function to this value, you’ll get different results depending on the context
    • Functor is any data type that defines howfmap applies to it. 
    • The Maybe data type defines two related contexts: we’ll see how function application is different when something is a Just a versus a Nothing
    • When a value is wrapped in a context, you can’t apply a normal function to it:This is where fmap comes in
  • Applicatives
    • Applicatives apply a wrapped function to a wrapped value:
    • Easy example to remember it by: Just (+5) <*> (Just 3)
    • Applicatives allow you to apply a function that takes two arguments to two wrapped values?
  • Monads
    • Monads apply a function that returns a wrapped value to a wrapped value.
    •  > Just 20 >>= half >>= half >>= half


Friday, July 19, 2013

Temporarily ignore file checked in to git

So, to temporarily ignore changes in a certain file, run:
git update-index --assume-unchanged <file>
Then when you want to track changes again:
git update-index --no-assume-unchanged <file>


Tuesday, July 16, 2013

Software Engineering Points

I have tried to distill some knowledge that has come from building a few green field projects. These are the three points that have come about from this experience.
  • Focus on building up a language of "higher concepts" that are relevant to your system.
    • As your system progresses into later stages of development the hope is that you will be able to begin to spend more time applying the "higher concepts" you built in earlier stages.
  • Naming, Modularization, and Code Organization should not be under estimated.
    • Giving modules a well defined purpose and exporting a public interface is crucial. While it may not affect how the the code works it allows you to think at a higher level and keep less information in your brain at once. KISS
  • The eventual goal is to build software that is "product ready", in other words you want to be able to release the software and leave it alone.
    • If you have to hire a big IT team to support your software maybe you can improve your code to be more self sufficient.

Git pick out individual commits

The following approach was easy and straightforward to migrate some code on a branch that had diverged greatly from the current one. I was able to grab a specific commit out.

git format-patch -1 b50788b
git am 0001-First-pass-at-rake-task.patch


Tuesday, July 9, 2013

A Resilient RPC Infrastructure with Thrift Finagle ServerSets Zookeeper

One use case for ZooKeeper within Twitter is service discovery. Finagle services register themselves in ZooKeeper using our ServerSet library, see finagle-serversets. This allows clients to simply say they’d like to communicate with “the production cluster for service a in data centre b” and the ServerSet implementation will ensure an up-to-date host list is available. Whenever new capacity is added the client will automatically be aware and will start load balancing across all servers.


Great tutorial on building a Finagle Search Service, http://twitter.github.io/scala_school/searchbird.html

Another interesting article on unique id generation, https://github.com/twitter/snowflake/

Distributed databases, http://en.wikipedia.org/wiki/Gizzard_(scala_framework)



Thursday, May 23, 2013

Spring, Camel consumer shutdown hack

import java.lang.reflect.Field;
import java.util.List;

import org.apache.camel.Consumer;
import org.apache.camel.spring.remoting.CamelServiceExporter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.SmartLifecycle;

import amqp.spring.camel.component.SpringAMQPConsumer;

 * Bean to help shutdown Camel beans that do not have a doStop() method that does what's needed to shut them down
public class AmqpConsumerShutdownBean implements SmartLifecycle {
@Autowired(required = false)
List<CamelServiceExporter> camelExports;
private boolean running = false;
public AmqpConsumerShutdownBean() {

public void start() {
this.running = true;

public void stop() {
if (this.camelExports == null) {
for (CamelServiceExporter bean : this.camelExports) {
Field f = null;
try {
f = bean.getClass().getDeclaredField("consumer");
Consumer consumer = (Consumer)f.get(bean);
if (consumer instanceof SpringAMQPConsumer) {
((SpringAMQPConsumer) consumer).doShutdown();
f.set(bean, null);
} catch (Exception e) {
} finally {
if (f != null) {
this.running = false;

public boolean isRunning() {
return this.running;

public int getPhase() {
return 1;

public boolean isAutoStartup() {
return true;

public void stop(Runnable callback) {
if (this.isRunning()) {