Google’s golang for president

Being a hungry geek I can’t help myself from innovating myself, and so I read blogs here and there on the current state of software and architecture. But I didn’t really have any alarm bells going off the last couple of months when I came across Google’s Go language. I think it was just a classic example of my assumptions getting in the way (knowing Docker was built using Go, I figured it was some new low level generic language). But now that I finally started studying it, it appeals to me more and more.

You see, after having advocated Node.js for some years now, and seeing the architectural shift towards frontend middleware becoming a reality, I never really looked for anything better or more suited for that. And that is exactly where Go fits in. It’s such an elegant solution to the need of scalable applications that handle concurrency and parallelism gracefully. It’s still a functional language, but at the same time it’s blocking! It’s kinda weird that I am excited about that, since I have been addicted to events for the last years, and have a hard time shedding that skin. But I have seen the complexity of large scale applications that are built upon callbacks and promises, and it secretly made me wish for something simpler. Something that did not make us do custom code-(re)structuring all the time. But the flexibility just kept me in love and favor it above anything else.

And now I found Go, and Rust, but that is another story that might not have a happy ending.

Event store with Node.js and AWS

It’s been a while since I posted anything here, but a lot has happened on the front. I will give a quick update about the things that have interested me since then

In 2013 I created my first auto scalable event store architecture for a huge client in Node.js, involving custom web servers receiving events from different endpoints in different formats, meta-tagging them and then injecting them into amazon queues, with processors on the other end enriching and transforming the events for storage in AWS DynamoDB. Post processors would be run periodically to store aggregates in S3. It was required to auto-scale to handle 200.000 events per second. (Yes, you read that right). I created a stateless architecture with the code for all the roles (server, processor, post-processor etc), built into one repo, which would be tarred and deployed onto S3 by our Bamboo server, to allow new nodes to be bootstrapped with that. The node itself was already booted by puppet with a role to perform, and thus knew it’s role to play. For hot upates and rollbacks we’d tell a Saltstack master to update a certain range of nodes, which would then pull the wanted source from the S3 registry again and update themselves without downtime. Pretty nifty, but rather proprietary.

The company I worked for used Puppet for configuration management, but also for app deployment, which I thought was the wrong approach. Puppet is imo not designed for realtime deployment, but rather for booting and maintaining vm’s from config. That is how I came across Saltstack’s powerful realtime command capabilities, and decided to script our deployment process to be controlled by Saltstack. I actually haven’t updated on that front in a long time, but I saw it fit the bill for our needs and I was so bold to build it into our POC.

Too bad we hadn’t learned about Google’s Go language back then, otherwise I would have scratched myself behind the ears and probably opted for that, instead of Node.js for our highly concurrent applications.

Node.js is the future of web dev happening now

Having been a javascript fan since I started working with it a long time ago, I immediately fell in love with Node.js. Having discarded Jaxer earlier as too proprietary, even though it offered a solid mechanism for code reuse on the client, I have now adopted the uber active Node community.

You see, me and my partner decided to build a new and exciting community site/app (of which I have to keep the details secret for now). So when I started designing our new web 3.0 app with accompanying mobile app, I thought about and thoroughly  investigated the possible frameworks out there. We decide to build a single page application that would work in all javascript clients, and would gracefully degrade to server roundtrips when javascript was not available (but also to enable deep linking for SEO). I quickly decided to discard most PHP frameworks for our MVC setup. Not only because of the fact that those were mostly too bloated or complex, but also because it implied having to recode a lot of functionality for the client. Of course I also favor the stateful and event driven possibilities of javascript, so I made the paradigm shift and chose the Node javascript stack.

With all the Node modules out there it is finally possible to create a full MVC framework operating on the server as well as in the client, in the form of backbone.js with the help of bones (a Node backbone server implementation offering code reuse in the client).
And with the help of HTML5’s new pushState, we don’t have to worry about breaking Google’s need for deep linking. (Those looking for an example with gracefully degrading URL’s, take a look at the jquery address plugin.)
More goodies come in the form of the browserify module, enabling us to optimally pack all our resources for client side usage, even all our templates and other static files!

Some more info for those interested:

With regards to storage, we decided to go for Cassandra, since we expect a lot of writes, and a lot of scaling. In the mean time I am hoping somebody will come up with a nice abstraction on top of the new Cassandra CQL language, since there are already some Node modules out there working with it.

Taking sessions into account, I am currently favouring Redis, which also has a nice pubsub layer. But I haven’t investigated that path fully yet.

In the mean time I am working on an iphone demo in Appcelerator’s Titanium. Too bad it doesn’t support all the functionality we need on Android as well.

That’s all for now. I have to try and curb my enthusiasm, as all this goodness may come at the expense of my sanity due to sleep deprivation.