baby steps

[caption id="attachment_714" align="alignright" width="200" caption="Today I conquer steps, tomorrow France!"][/caption]

Last night I had the unadulterated fun of updating a client's blog. After handing off the reigns everything was going smoothly until some unknown combination of plug-ins was causing the mobile version of the site to be served to actual browsers. I was called in to triage the situation, nothing too dire, just two plug-ins with some configuration problems, once I told them to play nice, all was fine. It was then that I noticed there were 15 plug-in updates waiting and the whole WordPress install needed to be updated. So I took the time to take a backup and update everything, nothing too serious (for anyone that doesn't use WordPress, updating is a one-click affair now) just the day to day stuff that needs to be done to keep the lights on.

This got me thinking about upgrading and Google's decision to discontinue support for IE6. Some people are crying foul, saying that IE6 should still be supported, I'm decidedly of the I-hope-IE6-dies-in-a-fire camp. Lest this become a let's bash on IE6 rant, which it so easily could, fucking broken box-model CSS1 piece of shit, let's get back on track. The people that are upset about this are not technical luddites clinging to the familiar. Digg performed a user survey to figure out what they should do about IE6 and part of that was asking why people used IE6. Of IE6 users 70% responded that they either lacked administrative rights needed to upgrade IE6 or were told by someone at work that they could not upgrade. For anyone that has ever worked in Corporate America this should be no surprise to you. The nagging question though is why?

Why would the IT department, home of nerds always hell bent on trying out the latest and greatest, demand that you use a 9 year old piece of software? There are a number of reasons people point to.

1. Mission Critical application is only compatible with IE6


2. IE6 comes installed on all the machines


3. IE6 is maintained by Microsoft, less of a headache for us


4. Licensing / Partnership / Legal-mumbo-jumbo

If you fall into #4, I pity you. #3 is laughable because although the updates might be bundled with Windows Updates you aren't saving yourself any administrative costs by putting the most targeted web browser for malware on all your machines. #2 is true, but if you are big enough for an IT department you probably have some way to push software to machines (or you could, and this is a shocking idea, trust your employees to choose a browser that fits their needs). I really, really, really want to talk about #1 though, because that is the point of this whole post.

I've worked at a number of places that used some sort of Web Application that was deemed critical that only worked right in IE6. This was then used as a justification for never moving past IE6, the timecard application won't work or the spline flanger won't have those cool IE6 exclusive filters. I understand that no one wants to expend resources on a "working" application pulling it into the future. But what is the longterm idea here, are you just going to use IE6 forever? This prevailing consensus seems to be ultimately self-defeating. Google recognizes this and as they try to push what can be done with the web they find themselves expending far too much time and energy on a nearly decade old piece of technology.

The nice part about fixing an application on IE6 is that you avoid some cost in updating it to make it work with other browsers like IE7. You could have spent a week or two adding in some CSS fixes and some markup changes and bingo bango IE7 now works like a charm. Instead you decide to use Corporate Fiat to force the continued use of IE6. Well now IE9 is looming around the corner and instead of paying a week here or a week there you now have a mountain of technical debt to pay down to get it to work.

Although IE6 is a convenient example, we can see this happen in all parts of the software development world. This program compiles when linked to an older version of the library but not this new one, I don't have the time to figure it out, well just link in the old one. This is all well and fine to meet the deadline but you have just accrued some technical debt. Later when that library gets some great new speed-up or killer new feature, there you will be plodding along with version 0.91 because you can't get it to work with the new hawtness. The problem is that these debts are not additive, they compound on one another and make what would be a simple conversion to 0.92 a nearly impossible conversion to 3.76.

Keep your tools up to date, keep an eye on what's up and coming, you don't have to be the first to jump on the latest and greatest but when a new version has proven itself and become the new standard move to it. Technology doesn't stand still, if you don't keep up with the latest stable releases you will find yourself in a tight spot trying to do an overnight re-engineer of your product. Take the easy baby steps from version to version so you don't find yourself having to take a technological leap down the road.

toolmaker

[caption id="attachment_535" align="alignright" width="262" caption="Pictured Above: Programming"][/caption]

For about a year I had the unique experience of working at Abercrombie & Fitch's world headquarters in New Albany, Ohio as a Java Developer. It was a really interesting place to work, very stylish, loud music all the time, complex problems of incredible scale, and some of the best developers I've ever worked with. I was lucky enough to get placed into one of the most successful development teams, Planning. This team had developed a tool that was hugely popular and well regarded, and I was coming on board to maintain and improve this tool. The beautiful people and board-short dress code was just icing on the cake.

As a software developer this is a fairly basic scenario, there is some task that is complicated, business users want it to be less complicated, they give me some money, I make them some program, and everyone is happy. The odd thing though is that while programmers are completely used to making tools for others, they often forget that they can leverage this amazing skill for their own benefit. I distinctly remember a conversation I had with one of my colleagues there, Steve. It centered around the idea that most developers are happy to just get their assigned tasks done, but there are some that are toolmakers, and being a toolmaker is more valuable to a company than being a developer.

Fast forward to after my stint at A&F, I was on a new project and given the task of translating 80,000 lines of PL/SQL into T-SQL. The idea was to use...um...find and replace...or....something...and...um...get this to work. With such a fine technical specification and a mountain of PL/SQL staring at me, Steve's words bubbled up to my brain, "Make a tool." I had a time window of a some number of weeks to do the translation, it was a bit of a gamble, but I decided to spend a week writing a tool.

After remembering how C# worked after a year in the trenches with Java I had a program that took in the PL/SQL parsed each line and translated it to T-SQL. The task was made easier by the fact that I didn't have to write a generalized parser for PL/SQL as the 80,000 lines were written in a fairly consistent style, which greatly simpified the translation tool. After running the tool a few times, fixing some bugs, I had finished the task in a fraction of the projected time. Triumphantly, I went to my PM to show how I had just saved weeks of manual laborious work. "Great, now here is your next task...."

[caption id="attachment_538" align="alignnone" width="150" caption="Expected Reaction"][/caption]

[caption id="attachment_539" align="alignnone" width="150" caption="Actual Reaction"][/caption]

So I didn't get the praise I was expecting, that's fine, I'm not 4, I don't need a cookie. I continued on in my tasks quietly pleased with saving myself weeks of tedious work. Then the day came when the database requirements changed, the hundred or so CREATE TABLE statements would need to be altered. The task wheel was spun, and since only my name was on it at the time (thus defeating the need for the wheel) I was the lucky person to go through and change various aspects of each table.

I thought to myself, "Great, there goes a day or two down the grunt-work hole." Then the part of my brain that I put in charge of remembering important things woke up and slapped me. "You haven't altered the SQL yet, just alter your tool to do this too and re-translate, dumbass." After pondering the suggestion, and possibly more importantly the insult, leveled at me by my brain, I opened up my translation tool. A few keystrokes later the change to the translation mappings was made and I fired off the program. Bits flew, gears cranked, and out came the exact result I was looking for, the tool had just saved me another day or two.

Ultimately the project was a big hit, we delivered on-time and on-budget for what was a somewhat aggressive schedule. The tool saved us a good amount of time, let us get the application bootstrapped and running, which allowed us to start iterating and get things working. Over the course of that project I created tools here or there that ended up saving me, and some thankful members of my team, time and effort.

The lesson here is that you should look around you and see if you can make some tools. Tools can be scary though, they are a bit of an upfront cost, and if it doesn't work out then you've wasted even more time. With time and experience it becomes easier to recognize when it's a good time to make a tool and when it's a waste. Here are some good guidelines for tool making success.

• The best candidates are tasks with high amounts of repetition and little logic.


• Check to see if a similar tool already exists, don't reinvent the wheel.


• YAGNI (You Ain't Gonna Need It). You should be intimately familiar with the problem you are going to solve, solve exactly that problem, you can scale it out later. I didn't bother writing a general purpose PL/SQL to T-SQL translator, I only had one PL/SQL file to translate, if it worked on that then it worked on 100% of its input.


• If it would take less time to do the boring thing manually than to build a tool, just suck it up and do the boring manual task


• Share your tools, even if you don't think they will be useful to anyone else on your team, sometimes a tool can provide a surprising benefit to others


• KISS (Keep It Simple, Stupid). This is no time to become a highfalutin architect. Tools are normally quick and dirty one-offs, normal concerns like efficiency, user-friendliness, etc. can be brushed aside. If the tool is getting used often and by many people you can refactor it to make it more efficient or user-friendly, but don't pay that cost upfront

Developing software often requires a high degree of focus on the problem at hand, I wrote the other day about one approach to problem solving, simplifying the problem. This strategy can be coupled with toolmaking to dramatic effect. Once you realize the simpler actions you can automate those and write a tool to perform the task for you. The next time you are facing some mountainous problem, consider the toolmaking route.

impedance

[caption id="attachment_456" align="alignleft" width="300" caption="you can find really cool images searching for impedance mismatch, this one has lazers (with a z)"][/caption]

There is a threat to good software, one that threatens the most promising of software projects. This threat is impedance mismatch. I use impedence mismatch in the sense that Object-relational impedance mismatch uses the term, but make no mistake, I'm not talking about the difficulties of writing an ORM. What I wish to discuss is far more insidious, it is death by a thousand paper cuts, it can spell ruin for any project. Impedance mismatch is born out of the confusing first steps of a project, it can also arise from multiple developers working over time on a large project or from internal disagreements over the optimal solution.

The major problem is that when we go about designing some new piece of software, some whiz-bang solution, we do not fully understand the problem domain. Any good development team will spend days filling up whiteboards with the solution that models the problem domain, but anyone with experience will tell you that once you get into the guts of a problem you realize that things need to be changed. This is fine, in fact this is a sign that you are solving the actual problem and not the ideal one you began with, don't worry about changing your solution to meet the realities of the domain. The problem though is making these changes first class citizens and not some weird third class monster thrown into the mix.

Have you ever been on a project and heard a conversation like this?

New Developer: What is X?
Grizzled Veteran: X is what we refer to internally as Y
New Developer: Why do we call it Y internally but X on the UI
Grizzled Veteran: Time constraints... budgets.... other misc. bullshit
New Developer: *mental note: X is called Y sometimes*

BAMMO! There's some nasty impedance mismatch there. The problem is not so much in what we display to the user, or what our object properties are called, or what the database columns are named, the issue is that we have made our mental model convoluted. At the heart of all these systems we build as software developers is some sort of mental model, at runtime this is expressed in system as an object graph, for persistence it is expressed as a collection of tables and relationships, but all of these things are just a representation of some mental model.

[caption id="attachment_455" align="alignright" width="300" caption="artist depiction: working with an elegant mental model"][/caption]

When our mental model is internally consistent and simple and elegant, it makes life easy and happy and unicorns shoot out of our monitors and everything is sunshine and puppies. When your mental model gets convoluted it can be an awful and bewildering experience. The problem comes from an increased overhead, instead of the mapping from one representation to the next being clear-cut, the developer has to carry around a hashmap in their skull mapping various representations of their mental model. The difficulty of carrying out even trivial tasks is increased as the mapping from X -> Y -> Z -> Q -> P takes place.

[caption id="attachment_459" align="alignleft" width="200" caption="impedance mismatch vs. coupling: battle to the death"][/caption]

So let's just name everything the same and go home, problem solved. Oh, if it could only be so easy, the problem is that on the other end from impedance mismatch is a devilish little troll named coupling. Coupling is the amount that two pieces of software are intertwined, they could be tightly-coupled wherein changing one requires changing the other, or they can be loosely-coupled all the way to the point of being considered decoupled or orthogonal. Orthogonal pieces are nice to work with because you can change one and know that you won't have side-effects on other pieces. Orthogonality goes by some other names: encapsulation, modularity, and decoupled, it is generally considered a Good Thing© and I make no argument against it.

We now have these two seemingly opposing forces, we want to decrease impedance mismatch without introducing coupling, what should we do? There are many ways to deal with this issue, let's look at a few.

Adapters are nice little pieces of code (also called wrappers), that allow two different things to look the same. Let's assume we want to write some code to manage users, users have a name, an address, and an age. The developer of the class decides that this object can be pretty simple.

[php]
class User {
public $name;
public $address;
public $age;
}
[/php]

The database developer decides that he loves structured data and that this should be broken out into all kinds of columns and tables.

[sql]
table user (
user_id int autoincrement primary key,
first_name varchar,
last_name varchar,
addr_street varchar,
addr_number int,
addr_city int foreign key,
addr_zipcode int,
age int
)

table city (
city_id int autoincrement primary key,
city_name varchar,
state_id int foreign key,
)

table state (
state_id int autoincrement primary key,
state_name varchar,
state_abbrev varchar
)
[/sql]

We could create an adapter class that bridges the difference, for brevity's sake I won't include the code for it, but it would basically translate strings into database fields, as the application developer I get to imagine the database as mirroring my object, even though it doesn't.

ORM - ORM's are like automatic adapters. They are not without their faults and some implementations are better than others, but they mask the complexity of marshaling data and reduce the impedance mismatch.

Convention over Configuration - This was made famous by Ruby on Rails. There is some strong convention that you follow that allows for a consistent mapping between domains, an example would be that tables are always named the pluralized version of the class name.

There is no silver bullet to solving the problem of impedance mismatch, but a combination of the above can help limit it. The best thing you can do is be aware of the problem, don't introduce impedance where it is unnecessary, and never be afraid to ruthlessly refactor out impedance mismatch when you find it. Keeping impedance mismatch low increases developer productivity, lowers the bar for bringing on new people, and makes unicorns leap out of your monitor farting rainbows.

plain stupid

[caption id="attachment_409" align="alignright" width="136" caption="typical dumbass"][/caption]

Are you a dumbass? It's an important question to ask yourself every once and a while, especially if you want people to put their trust in you. This has been brought up because of a service called RockYou that recently exposed 32,603,388 plaintext passwords. Then it got even worse when it was confirmed that RockYou was also storing 3rd party passwords, again in plaintext.

Passwords are a tricky thing to store, RockYou clearly never got the memo. I've been at this whole "writing software thing" for a while and I've gone through my own personal journey of discovery about vulnerabilities and best practices. Let's walk through that now

• Plaintext - This is the first stage, the one that RockYou never left apparently. It is the simplest to implement, take the password, shove it in the DB. Need to check against the password, if(input == password) and done! If you don't see the problem with storing your passwords plaintext, go to the nearest hard surface and slam your head into it over and over again, you no longer need cognitive functions.


• MD5 Hashes - This is generally the second stage of password discovery. Many languages have a nice built-in function, php does. Store that MD5 hash and you get that warm fuzzy feeling in your tummy that you aren't storing things in plaintext. The problem now is that MD5 has well known weaknesses that can be exploited. Rainbow tables provide an attack vector and reverse look-up tools opens up a ton of exposure.


• Salty passwords.... delicious - Your users are probably going to pick awful passwords, even if you have a password policy its probably going to end up "password1" You can't rely on them to provide strong passwords, so you can do the next best thing, make it strong for them. Static salting is the act of adding predictable data to a password before hashing. Maybe you append the string "Matt~is~the~best~programmer~7829" before hashing the string. This will defeat most reverse lookup tools (as very few people have bothered hashing "Matt~is~the~best~programmer~7829password1" and putting it into a database of reverse lookups) The problem is you are still open to dictionary and rainbow table attacks. If your database is compromised there is a chance that your source could be compromised as well, and then that string sitting in some configuration file is the key that unlocks everyone's password.


• Dynamic Salt... deliciously nerdy - Instead of having a static string you can create some dynamic salt. When a user want to store a password, generate some dynamic salt, combine it with the password, hash, mangle, and store. Now every hash is a puzzle, security in layers.


• Non-trivial calculation times - The problem with MD5 is that its too quick. Use a more complex hashing algorithm like SHA-1 or SHA-256, perform data mangling and multiple encryption sweeps so that turning a plaintext password into a hash you store takes some non-trivial amount of computation time. What's the point of this? It may add 1 second to saving and comparing hashes for you which is normally negligible in terms of user experience. The advantage comes from defending against a brute-force attack where the attacker has to generate millions of hashes. Will Bond does an excellent job outlining this in the rational of the fCryptography class in Flourish.


• Timeouts - The normal user will have to enter a password once maybe twice if they fat-finger it. Using a quadratic-growth timeout system can help prevent brute-force attacks, or at least slow them way down. A brute-force attack relies on the ability to quickly cycle through thousands if not millions of passwords in an attempt to find the correct one. Let's take a look at how quadratic-growth timeouts defeat this system. Each attempt has a timeout twice as much as the timeout that preceded it, we will start with a timeout value of 1 second.
  
  
  
  • 1st Brute-Force attempt fails server let's the person retry, but they must wait 1 second.
  
  
  • 2nd attempt fails, retry in 2 seconds.
  
  
  • 3rd attempt fails, retry in 4 seconds.
  
  
  • 4th attempt fails, retry in 8 seconds.
  
  
  • 5th attempt fails, retry in 16 seconds.
  
  
  • ...
  
  
  • 10th attempt fails, retry in 512 seconds. (8 and a halfish minutes).
  
  
  • ...
  
  
  • 20th attempt fails, retry in 524288 seconds (8738 minutes or 145 hours or 6 days)
  
  
  
  Most users will never experience more that 3 seconds of timeout, a cracker would have to wait 35,702,051 years to attempt 50 passwords.
  

Storing passwords correctly and defending against brute-force attacks is non-trivial. It's one of those things that people a lot smarter than you or I have spent a lot of time thinking about. In a former life I was a mathematician (sometimes I fancy that I still am), I have had the pleasure of picking up a book on formal cryptography, getting about 2 chapters in, and setting it down with a headache. There are a lot of great libraries out there to do this stuff for you if you are too lazy to take the time to do it right yourself.

For php there is Will Bond's fCryptography, for perl there is Crypt (I'm not a perl person so anyone is more than welcome to provide a better alternative), for any language there is normally a high quality cryptography library. If you are not using them, then you are simply being lazy.

Of course you could side-step the whole issue and let someone else do your dirty work. Its all a symptom of a problem that Mozilla Labs is trying to solve, the concept of identity on the web. I think the future will be the browser being the trusted agent instead of third-party websites, but until then we have to be responsible software developers.

web development today

[caption id="attachment_402" align="alignright" width="128" caption="the internet is so shiny these days"][/caption]

I've been playing with my new iMac all weekend and am getting it set-up to start doing development. I'm a web developer by trade, but a systems developer by education. First let me give you some quick background

• I went to BGSU where we were taught mainly C++


• At University I worked for ResTech developing and maintaining an Intranet written in PHP


• My current side-project is prosper a database abstraction layer written in PHP


• After graduation I went to work for HMB




• I've worked on 2 projects that were ASP.net


• I slung Java for Abercrombie & Fitch for just over a year, doing some web frontend and middleware


• Current project is doing some Java middleware

So my experiences are mostly C-style languages, but I have dabbled from time to time in other languages, ruby, python, haskell, erlang, sql (much more than I would like to with prosper), etc. Being on a new platform though makes you wonder about what makes it tick, I don't particularly have to wonder though as I read Apple's excellent documentation (PDF warning) a few years ago. My subconscious has been mulling over Objective-C development, which looks somewhat interesting, but it has brought up an old friend I was interested in a while ago Objective-J.

A while back I had heard about Cappuccino and the language that powers it, Objective-J. I remember thinking at first how crazy the whole thing was, someone didn't want to bother learning JavaScript so they found a way to make Objective-C compile down to JavaScript. This concern was assuaged some when I read this arstechnica article and realized that anyone skilled enough to write a compiler in JavaScript had to have some vague understanding of its workings. The design decisions behind making Objective-J were sound, they didn't just think my love was ugly.

I've been giving a new look at some of these new web development toolkits, platforms, libraries, etc. They promise to bring desktop quality applications to the web leveraging existing technology. And there are a few I would like to bring to everyone's attention.

[caption id="attachment_396" align="alignright" width="150" caption="Cappuccino"][/caption]


• Cappuccino and Objective-J - Its hard to talk about one with the other. Objective-J is a brand new language for client-side scripting in the web. It compiles down to cross platform JavaScript on the client-side and is used to power cool stuff like 280slides and EnStore. One of the best things about Cappuccino and Objective-J is that they remove the burden of implementation. Is the display made up of input elements, divs, one big canvas? Don't worry about it, that's Objective-J's job to figure out the best way to display things in a given browser. Cappuccino provides a great framework to write web applications in, anyone familiar with Cocoa will feel at home. The biggest downside I can find for Objective-J is its verbosity, but maybe Atlas will help with that


[caption id="attachment_395" align="alignright" width="150" caption="SproutCore"][/caption]


• SproutCore - SproutCore got a ton of press when Apple's MobileMe decided to use it as a foundation for their relaunch. A nod from a big player like Apple can help set aside fears that it isn't ready for primetime. Right now SproutCore is gearing up for their 1.0 release, its still in beta but they've been making great strides as of late. SproutCore is maturing quickly and has some core goals I can get behind: Be easy. Be consistent. Be opinionated. Stay small(ish). When working with SproutCore you will write code in JavaScript and also use a host of command-line tools (à la Rails).


[caption id="attachment_397" align="alignright" width="100" caption="GWT 2.0"][/caption]


• Google Web Toolkit - GWT has a lot of good stuff going for it. It's made by Google (they might know a thing or two about the web), it just hit version 2.0, its the juice powering stuff like GMail and the new Google Wave. With GWT you write your code in Java which then get compiled into fast and cross-browser JavaScript. Ray Cromwell has an amazing write-up of all the new cool stuff in the 2.0 release.

All of these frameworks allow you to create a fully functioning web application without mucking with the guts of the web (HTML, CSS, JavaScript). This is a double-edged sword though.

The up side is that if you already know Objective-C or Java then your learning curve is drastically reduced. As technology changes the library can output the latest and greatest HTML without you having to change any code (hopefully). The Google Wave team has said, you can create much more complex applications leveraging the power of these frameworks.

The down side is that you are spending time and energy learning a framework and not web development. If you learn SproutCore backwards and forwards that doesn't transfer to Objective-J or GWT. The other downside of this is it introduces some magic, if you don't understand the foundations of HTML and JavaScript it can be far more difficult or impossible to fix errors. A corollary of that last point is that applications look similar, Objective-J apps look like Objective-J apps, GWT looks like GWT, it doesn't have to but when the default styling is sufficient or, as is the case with many of these frameworks, actually looks nice, its easy to just keep the default.

Web development keeps moving forward, this internet thing is going to be big. As we move more data to the cloud, as Operating Systems become more Web Centric, and as more and more of our lives our spent looking at the web, these development frameworks will become more and more important. It's an interesting time to take a look at some new takes on web development