Backbone everywhere

I finally put my newly built Node.js MVC stack on github! You can download it here: backbone-everywhere.

What’s so special about it? Here’s my list of exciting features:

  • Pages are rendered on the Node.js server by Backbone and jQuery.
  • All script resources are bundled by browserify and fileify,  uglified by uglify, and gzipped by connect-gzip for fast loading and deployment on other possible javascript environments.
  • The entire Backbone MVC stack works on the server, and is loaded in javascript enabled browsers to take over from there.
  • The app state is reflected in the url by means of HTML5’s pushState, or using hash notation when not supported.
  • The same app state is regained for non-javascript browsers pulling full requests from the server, so no worries about SEO!
  • All client / server communication is handled by (ajax is sooo 2009) and subscribed clients are updated with published models.
  • A JSON-RPC service listening to ‘/api’ requests, with an easy to build on service layer. Handy for non-web contexts such as mobile devices.
  • All data is persisted in Redis through an adaptation of mine of backbone-redis, enabling indexing, sorting and  foreign key lookups.

For me this is a whole new approach at engineering web applications, but I think I’ve managed to get a grip on it.
Not only that, it gave me a great impulse to reconnect with the pioneers of tomorrow. Because what I have done was build on top of the stuff from people with great vision.
Big shout out to the open source community, and the people willing and wanting to share. The sum of it’s parts will eventually overcome the current patent trolling paradigm.

What are you waiting for? Dig in!

Node.js is the future of web dev happening now

Having been a javascript fan since I started working with it a long time ago, I immediately fell in love with Node.js. Having discarded Jaxer earlier as too proprietary, even though it offered a solid mechanism for code reuse on the client, I have now adopted the uber active Node community.

You see, me and my partner decided to build a new and exciting community site/app (of which I have to keep the details secret for now). So when I started designing our new web 3.0 app with accompanying mobile app, I thought about and thoroughly  investigated the possible frameworks out there. We decide to build a single page application that would work in all javascript clients, and would gracefully degrade to server roundtrips when javascript was not available (but also to enable deep linking for SEO). I quickly decided to discard most PHP frameworks for our MVC setup. Not only because of the fact that those were mostly too bloated or complex, but also because it implied having to recode a lot of functionality for the client. Of course I also favor the stateful and event driven possibilities of javascript, so I made the paradigm shift and chose the Node javascript stack.

With all the Node modules out there it is finally possible to create a full MVC framework operating on the server as well as in the client, in the form of backbone.js with the help of bones (a Node backbone server implementation offering code reuse in the client).
And with the help of HTML5’s new pushState, we don’t have to worry about breaking Google’s need for deep linking. (Those looking for an example with gracefully degrading URL’s, take a look at the jquery address plugin.)
More goodies come in the form of the browserify module, enabling us to optimally pack all our resources for client side usage, even all our templates and other static files!

Some more info for those interested:

With regards to storage, we decided to go for Cassandra, since we expect a lot of writes, and a lot of scaling. In the mean time I am hoping somebody will come up with a nice abstraction on top of the new Cassandra CQL language, since there are already some Node modules out there working with it.

Taking sessions into account, I am currently favouring Redis, which also has a nice pubsub layer. But I haven’t investigated that path fully yet.

In the mean time I am working on an iphone demo in Appcelerator’s Titanium. Too bad it doesn’t support all the functionality we need on Android as well.

That’s all for now. I have to try and curb my enthusiasm, as all this goodness may come at the expense of my sanity due to sleep deprivation.

My love for Paypal just died a little.

After having put some days of work into creating my friends book site (, I decided to keep it simple and implement Paypal’s “Add to cart” buttons.
Wrong choice.

You see, I was under the impression that Paypal’s shopping cart would be able to handle shipping cost based on weight. But no, it isn’t. Well, only if you’re a merchant residing in the US.

What? I thought computers and the internet would help bring an end to discrimination and suffering.

What could be the reason for this? I am sure it’s not a technical problem, because that little and simple piece of code is agnostic of location. And it isn’t money, because they would probably make even more if they didn’t piss off merchants outside the US.
Maybe they just don’t care anymore, having all that money, and getting (re)tired of it all.
Or, could it be some political hidden agenda? These things pop up everywhere these days.
I am probably missing something here.

Taking PHP development to the next level

A lot of my enthusiasm for programming comes from my never ending interest in it’s evolution. I played with computers since I was 7, and already knew at age 13 that I wanted to become active one day in the ‘automating’ industry. I’ve always regarded programming as a trade or tool to automate repetitive tasks, but also to offload (error prone and inconsistent) people tasks to fool proof automated systems. I just wanted to make stuff easier and better, and found a way to do it.
I truly started my programming career in 1999 as a Java programmer in the corporate world, learning a lot about real world software problems. It made me realize a lot of things.
But even in the year 2000 internet applications were not widely recognized as “the way to go”. My employer, one of the best amongst Dutch knowledge engineering services, did not pay full attention to it yet.
So I left. I simply had to find out more about web technology, which I regarded as the future of software development.
I carefully chose to follow the ‘open source’ path (as opposed to the closed MS world), and divided the territory into “corporate vs fun and dynamic”. I ended up in PHP land.

But soon I found out about the drawbacks of PHP development. It’s a loose typed language, with a scattered user base, supporting all kinds of immature frameworks and libraries, for all the wrong reasons. Simply stated: PHP land was short on inhabitants with extensive knowledge of software development in general. After a while though, PHP was recognized for its RAD abilities, and more and more people with skills came into the game. This lead to a myriad in yet more frameworks and libraries, but now it was all about documentation, participation, and a growing user base. Because solutions were being reviewed, tested and explained, quality started to drift to the surface.

But how does one recognize quality in this field?
Since I have never studied software development on a university or such, I will always regard myself as a noob on matters that I have no experience with. I have learned everything by reading books, and from material on the internet. But because I was aways so insecure about my knowledge, and truly interested in what I was doing, I never stopped reading. That’s how I discovered that it’s most beneficial to start learning the industries’ best practices and ‘patterns’, before formulating your own suspicions and concepts regarding programming solutions. Building onwards from that premise I feel I have finally grasped an overall intuition for what matters most, and which choices should prevail others. Of course there is no holy grail of software design, but given a business goal, context and limiting factors, one surely can sketch a fruitful path for future development.
Do I bore you already?
The point is that I think that higher level principles should be considered first, when making choices in software development. Therefore I was always drawn to development platforms that obey such principles, even though some of their subsolutions are plain awful.
Steep learning curves and incomplete documentation were arguments very low on my choice break down list. As I have a brain, and a debugger, I’d rather work with beautiful flexible software than some simplifying framework caving me in and making me jump hoops to do the in-ordinary.
(Please don’t respond with PHP framework ‘showdown’ arguments, as I think Sheldmandu has a nice take on that, even though I don’t follow his simplistic view on ‘inheritance is key’.)

That’s why, about 4 years ago, I chose to go with the Zend Framework, being the most mature and offering so much of what I thought was important. It still forced me to do a lot of work by hand, and a lot of their code base seemed rather inconsistent. I think mainly because they let it grow too quickly without proper reviewing user contributed code base additions. And as a result of that, I think a lot of their solutions are simply not well thought out (like forms and data access).
But I loved their coding style, and the way they used naming conventions to get stuff rolling.

Then, about two years ago, I stumbled upon FLOW3, a new PHP framework built by the developers of TYPO3. In my opinion it will be the most advanced and productive PHP framework in the near future, but that has to be seen as it needs to become widely adopted. It has implemented a lot of the best practices from the Java scene (annotations, AOP, dependency injection). Which I think is a good thing, as that scene has been dealing with the largest chunk in complex software development for a long while.
I concluded that it needed a lot more time to come of age.

Around that same time I started using Magento for my own webstore (project was killed over a refuted business deal), and after that I worked freelance for 2 years with it’s complicated, yet powerful CMS abilities. A lot of 3rd party extensions built by novice Magento developers were indirectly compromising a lot of the projects I worked on (in the form of performance degrading or bug introducing plugins), thanks to Magento being too flexible and not understood well enough.
But Magento made me realise something:
What’s the use of a framework or library, if it doesn’t offer concrete reusable solutions to real world problems? In other words, why could I not find a development platform crafted on top of these high quality frameworks and libraries I so much admired?
Magento at least offered me out of the box configurable layouts, views, themes, multi site functionality….bla bla, etc etc.

Now that I wish to work on more versatile projects, preferably using out of the box, proven solutions, I have to check my facts again. I don’t want to go back and reinvent so many wheels. What are my options? What CMS can I use to not have to repeat myself too much for every similar project?
Let me write down some of my requirements:

  • An intelligent caching layer, like Magento’s, with it’s hole-punching capabilities.
  • An annotation-based domain persistence layer, using POPO’s (plain old php objects), decoupling data storage and making persistence trivial. (Doctrine 2 might fit these needs.)
  • An easy to setup service layer or domain API, enabling easier development of client-server architectures. Javascript, apps and flash are my main clients these days.
  • Straightforward creation of sites, templates and content, facilitating a multi disciplinary approach and separation of concerns.
  • Inheritance based themeing, allowing a stack of theme overrides, like Magento’s upcoming themeing system.
  • Plugin based architecture, preferably with a controlling mechanism warning developers about slow and impeding functionality.
  • Many great working plugins fitting my business needs!
  • Possibly some elaborate publishing mechanism, allowing for the entire application (files AND data please) to be pushed onwards in it’s development cycle: DEV / TEST / USER ACCEPTANCE / PRODUCTION
  • A large user base and developer involvement ensuring stable releases.

(Please feel free to add to this list)

Lately I found that FLOW3 is approaching it’s first stable release (possibly september 2011). But not only that, the release of TYPO3 v5 will be based upon FLOW3. So I quite hope TYPO3 to be the CMS I am looking for.

Group Messaging needs to grow up.

I recently developed the need for a real time group messaging system that allows the members to subscribe to topics and tags. Posting messages in such topics or including such tags would then result in a push notification to the subscribers mobile devices.
The admin of the group could create a taxonomy for the groups context, and members could contribute to the taxonomy, based on an approval system.
This allows for subscribers to carefully decide themselves what messages they want to receive, instead of somebody else deciding for them that they are interested in.

You see, because I hang out with a loosely organised group of artists, freelancers and randomly creative individuals, we need a tool to be able to mobilize each other, but only the people interested in the joint venture at that moment. So we want to create a list of topics/tags and locations to subscribe to. We want to keep informed of only what we subscribed to. The group is just too large to blindly spam everybody with every message.
And since we are often in the pre-organising fase, we simply want to be able to tweet a message including one of these tags to elicit interest and motivation to those possibly wanting to cooperate.
Not only that, sometimes we simply want to mobilize a group of individuals to meet at a certain location. Last minute spontaneous action.

Well, after doing some research I found some great and promising startups in the group messaging industry, as that industry is taking off like storm.
I really like the irc-like Convore, and posted my feature request there, but no answer yet. Too bad none of the tools I looked at (Beluga, GroupMe, Kik) offer what we need. Sure, there are some forum-like tools that send you an email when somebody posts in a thread, but none of these tools allows for subscribing to a taxonomy. Please correct me if I’m wrong.
I think most of these tools are truly helpful after the fact of organising structures. When all the roles are defined, the players are known, and from there it’s just a matter of communication and collaboration.
Not what we need.

So, because I am a programmer, and therefor lazy by nature, I want to appeal to the group messaging industry first. By asking them to incorporate such a mechanism in their platform, so I don’t have to build it myself.


PAS OP: NavAudio zijn criminelen die je geld niet teruggeven.

Ik ben niet de enige die gedupeerd werd door de firma NavAudio.

Had ik maar goed onderzoek gedaan naar de (blijkbaar altijd slechte) service van NavAudio, er zijn genoeg posts:

Dynavin voor BMW E39 bijna net zo slecht als de firma NavAudio die hem levert.

Ok, ik vond dat mijn BMW E39 uit 2000 wel een onboard GPS systeem moest hebben, en dan het liefst met geintegreerde iphone aansluiting etc. Ik had wat forums afgestruind, om te stuitten op de enige twee betaalbare alternatieven: de Dynavin of de Hualingan. Beide zijn chinese lookalikes van de originele BMW unit met video touchscreen. Helaas had ik blijkbaar niet goed genoeg gezocht naar problemen met die units, terwijl ik meer dan genoeg lovende verhalen aantrof. Ik ga nu niet helemaal in op de voor en nadelen, want eigenlijk zijn ze die aandacht niet waard vind ik nu.
Maar goed, ik had mijn Dynavin besteld bij de firma NavAudio, kreeg mijn unit opgestuurd, en vond helaas de volgende problemen bij  het in gebruik nemen:

  • iPhone (zowel 3GS als 4) aansluiting werkte gewoon niet. Deze werd wel gedetecteerd, maar was niet te bedienen en gaf geen geluid. Ik had alle (overigens zeer simpele) kabelaansluitingen gecontroleerd, en moest concluderen dat deze iPhones niet ondersteund werden.
  • Bluetooth verbinding om handsfree te bellen was zeer slecht. Ik moest het geluid helemaal op de maximum stand 40 zetten, en de auto aan de kant, om iemand net te kunnen verstaan.
  • Geluid van de radio was veel te hard uitgestuurd. Op stand 10 was dit al oorverdovend hard.
  • Geluid van de subsystemen waren niet goed op elkaar afgesteld: stel je praat met iemand op volumestand 40, je hangt op, en vervolgens knallen je boxen kapot omdat het systeem terugspringt naar radio met volumestand 40 !!

Ik was duidelijk niet tevreden en stuurde het product terug binnen de 7 dagen zichtgarantie (volgens de wet Koop of Afstand), met duidelijke feedback waarom, en kreeg ik de volgende reactie:

“Helaas, maar er komen wel diverse updates voor.”

Alsof ik daar op moet wachten na voor bijna 700 euro te hebben neergelegd. Nee, dank je wel zei ik, en vroeg mijn geld terug. Maar dat zou ik na 30 dagen terug krijgen. Allemaal volgens de wet. Prima.

Maar na 5 weken had ik nog steeds mijn geld niet terug, en ging ik er achteraan bellen. Ondertussen had de firma NavAudio ook nog eens een nieuwe verpakte unit op mijn adres laten afleveren. Toen begon bij mij al het vermoeden dat deze firma niet graag geld teruggeeft aan zijn gedupeerde klanten. En ja hoor, ze wisten van niets en zeiden niets ontvangen te hebben. Ze wisten nota bene ook niet eens dat ze een nieuw pakket hadden opgestuurd!? Ik moest maar bewijzen dat ik iets had teruggestuurd, en daarmee was voor hen de kous af.
Gelukkig had ik natuurlijk nog het verzendbewijs van de TNT post, en kon ik laten zien dat het eerste pakketje op tijd was ontvangen door hen.

Maar toen kreeg ik ook nog eens te horen dat ze 5% van de aanschafprijs in zouden houden, terwijl dit volgens de wet verboden is. De wet Koop op Afstand zegt dat een consument binnen 7 dagen het product mag terugsturen met volledige restitutie van het aankoopbedrag, en zonder opgaaf van reden. Daar bovenop leverde het geleverde product niet wat er beloofd werd, en heb ik daarom ook recht op alle in rekening gebrachte, en door mij gemaakte verzendkosten.

Uiteindelijk ben ik dus gedupeerd voor 5% plus 3 maal verzendkosten, want de firma NavAudio weigerde gewoonweg te betalen. Ik zal jullie niet letterlijk al hun email reacties oplepelen, maar ik kan wel zeggen dat ze niet zo fris waren, en duidelijk naar voren kwam dat ik te maken had met onbetrouwbare figuren met geen gevoel voor eerlijk zakendoen. De firma NavAudio deed gewoonweg denken aan een stel kampers die je een kapotte auto verkopen. Alleen wordt dat niet duidelijk middels een internet aankoop. Ik heb gelukkig nog nooit ergens voor betaald en het niet gekregen, maar ik moet blijkbaar toch een betere neus kweken voor dit soort stinkende zaakjes.

Nu kan me die 55,35 euro gestolen worden, want ik vind het veel belangrijker om een toekomstige geinteresseerde te behoeden om ooit maar iets te kopen van de firma NavAudio. En al helemaal niet zo’n slecht product als deze Dynavin!
Toch vind ik ook dat het recht zijn loop moet hebben, en heb ik besloten contact op te nemen met mijn advocaat om een zaak aan te spannen tegen de firma NavAudio. Natuurlijk heb ik alle bewijzen om deze zaak te winnen, en ook dat laatste beetje geld terug te krijgen. En tegelijkertijd doet het me stiekem een genoegen om die sujetten op hun nummer te zetten.

Wordt vervolgd.

SimpleHelix won’t refund my money

Last week I talked to SimpleHelix support (who were very responsive btw) over their live chat app, to get some definitive info over their “semi VPS” hosting deal for Magento.

So we talked through a lot, and I went with the deal they had: 50% off for the first year if paid upfront. I paid about $425,- with paypal. But once I had server access I found out that they don’t offer in-memory caching on the semi VPS, which I definately need for Magento. Upgrading to a full VPS was too costly ($75 vs $160 p/m), plus it would also still lack the in-memory caching, so I decided to stick with my NL based host (IZI Services, very nice company btw).

That’s when I asked for my money back, since they offer a 30 day money back guarantee. But guess what, they simply said no! Because I paid with paypal!? What? Why? Where should I have read that? And why wasn’t that communicated to me before?

Can you believe that? In this day and age? Take money for a service which does not meet your needs? Man, I thought customer was king in the US, but I guess they need mouths to feed overthere now :p

I asked them a couple of times to help me out, because they couldn’t deliver what I need, but to no avail. I finally had to create a dispute with paypal, which is currently waiting for a settlement.

My advice: Don’t bother with SimpleHelix if you need a decent Magento setup.

Magento’s dataHasChangedFor(isDirty)

I just built 4 hours of code on top of isDirty, when I noticed it isn’t even working! I wanted to test if something has changed in the ‘customer_save_before’ event, so I could do something first. But Magento seems to simply not care if an object’s main properties or relations are dirty. It just saves whole object trees, wether something changed or not. AAAAAARGH! Yet another performance penalty, straight from the hip.

Isn’t it basic stuff to have your value container flag properties and itself as dirty? Magento started on isDirty and flagDirty, but didn’t pull through and left it unfinished. Instead they implemented a dummed down method called “hasDataChangedFor”, which doesn’t allow for testing if the object itself is dirty or not (because one of its properties is). I notice that it’s sparsely used, but not for testing whole objects before they’re saved.

Now I have to create an unnecessary bloated mechanism that goes through each property in getData, and see if it is found in oldData, and then test if it’s changed with hasDataChangedFor. AAW!

I hope the Varien guys pick up on this and start using the dirty pattern, because it’s simply too much to carelessly save whole object trees and make costly database queries when it’s not even needed.

Maybe I am too much of an angry guy right now, instead of a quiet and thorough investigator, so I might have overlooked something here. I hope so. Please tell me if I’m wrong 🙂

UPDATE: it’s even worse than I thought, because in both customer_address_save_before/after, _origData is not populated, so I have no clue wether something about the object has changed or not. Bummer! Does that mean that I have to load up the old objects from the database first? So much overhead, I am getting more even depressed.

Magento rocks!

I had to find the right solution for our online Chronos Coffee store, and preferred to find a SaaS (Software as a Service) application which would be flexible enough and also affordable. I looked at all the big players and also the smaller ones offering shared solutions, but none of them were flexible enough for what I envisioned: A scalable, controllable application with a plugin architecture to custom fit our needs. So I decided I’d give Magento a try again. I had fiddled around with it a year ago, but decided it was too infantile to be used in a production environment (and a web store always is). After having done in depth research, I now think it is actually very mature. It has a very large community and is widely deployed all over the net. Whats was even more interesting to me is the fact that it is built on top of Zend Framework, which I specialize in and am an advocate of.
So I arranged for a private virtual server to get it up and running, and must say I am really impressed. Compared to other web solutions out there, I find my install of Magento to be very snappy and responsive. The backend is rather sluggish, but that’s not such a big deal.
I have put the website online already, because we want to get out of the sandbox Google has put us in, and get a search rank asap. Don’t look at the design tho, because we have used a standard template, and still have to create product images. Here’s a link: