learn autoit

[caption id="attachment_655" align="alignright" width="300" caption="Bezels and shadows and reflections, oh my"][/caption]

If you use windows at work or at play this post is for you, macheads and linux people I'm sorry, this post probably won't help you much. I have had the joy of doing documentation work for the last 2-3 months at work. Gnarly, ugly documentation that serves little purpose other than making someone feel happy that there are reams and reams of documentation. At some point, probably around the 5th document, I made the realization that out of 20 pages it was mostly boilerplate, with about 8 things peppered throughout that changed. I decided that it would be easiest to make a template document and then just do a find and replace on those eight symbols [[DOCNUM]] [[DOCNAME]] [[DOCTRANSACTION]] etc.

Productivity shot up, I could now stub out a document in a few minutes instead of the hour or so that it used to take to manually hunt through and change things. All was well and after a day or two of find / replacing I had stubbed out the 80 or so documents. Then came the meeting where it was decided that 2 new sections would be needed and that 8 sections could be removed as being redundant. This victory for common sense was also a giant headache, as I didn't really look forward to restubbing 80 documents. There had to be an easier way, a better way, I remembered a screen driver tool I had some exposure to at a past engagement, AutoIt.

After reading through the documentation and goofing around with it for a while yesterday I now have a fully functional script that will automatically stub out all of the documentation with the click of a button. A task that used to be an error-prone, manually intensive process now requires no intervention. We can restub on demand now, so changing the template is no problem.

The Good

• Saves a ton of time


• Removes the human aspect from an existing process


• Centralizes specific knowledge


• Easy to write and test

The new script saves a ton of time, I can regenerate all documentation in about 10 minutes, so I click a button and go walk around the office bugging people. AutoIt simply takes a known working process and steps in to be the person driving it, I didn't have to take time dreaming up a brand new workflow, I could focus on just getting it done. The script is now the system of record for the specific things that change from document to document, which is nice for trying to determine at a glance what makes an HCR1 different from a ARC6 command, without digging through 40 pages of documentation. AutoIt also utilizes a stripped down version of SciTE with auto-completion and built in support for compiling and running, which makes writing and testing the script a breeze.

The Bad

• Ugly syntax (like VBScript and PHP had a bastard child)


• Oddly organized documentation and variably helpful community


• Inherently brittle scripts


• Still slower than I'd like


• Foreground processing

AutoIt has an ugly syntax (think VBScript), but it has all the pieces parts to make a nice script, functions and variables. The documentation takes a little getting used to, there is plenty in there, but it could be organized better. AutoIt depends on things like the window title and absolute paths, so it is inherently brittle, I doubt this script would run unaltered on someone else's machine. This could just be me being a n00b at AutoIt but I followed the practices laid out in the tutorials and the documentation. The other bad part about AutoIt is that it drives your screen, it simulates you pressing buttons and mousing about, so the script is slow and you can't interact with the machine while its running or you will probably mess everything up.

Alternatives

After proclaiming my victory over the documentation monster, I got some replies from colleagues asking why I didn't just make a powershell or ruby program or java program or something. I could have cracked open OpenWriter or something and attempted to build some massive program that could create .docx files, but that would have taken a ton of time. The AutoIt solution was incredibly quick to produce, took about 2 hours of playing around with it. There were a bunch of side benefits that the alternatives wouldn't have had. The template file is just a plain ordinary .docx file with all kinds of crazy formatting and images, instead of trying to figure out how to reproduce this in code, I can use Word to do the heavy lifting for me. This allows business users to change the template and we can rapidly restub and get back up and running.

Conclusion

You should go and learn AutoIt or AppleScript or whatever GUI driver program is available for your platform. It is not the tool for every job, but it's perfect for some jobs. Being the person on your team that can take some boring, error-prone, laborious task and turn it into a turn-crank task is a great thing. You can save your team time, get back to doing actual work more quickly, and make a name for yourself as a problem solver.

ipad

[caption id="attachment_649" align="alignright" width="300" caption="nerd law also required I use a picture with Spock in it"][/caption]

As required by nerd law, here is my mandatory post on the iPad. I've got lots of nerd friend and the general feeling I get from them is a sigh and a "meh." The term "underwhelmed" is being thrown around a lot, reinvigorating the debate as to whether or not there is a perfect amount of "whelmed" one can be so as to be neither under- nor over- whelmed. Here is my take on the new shiny.

Let's not forget our history. When the iPhone came out people far and wide declared that it will fail, is failing, already failed. Yet despite such serious problems as no cut-and-paste and a weird virtual keyboard, somehow it won Time's gadget of the year award, and a huge population of happy iPhone users. People to this day don't quite get it, I have an Android phone (one of the first G1s) and I love it, but there is no denying the sexiness and sleekness of an iPhone. If you don't understand the big deal play with one for a week, its just a completely polished user experience.

That is the strength of Apple's products, the experience. Its easy to find plenty of small, reliable mp3 players that store tons of music, people buy iPods because they are seamless and easy. I was asked recently what made OSX superior to Win7, I didn't have an answer, I work with both and I enjoy OSX much more but have no real good bullet point for why. I was looking for some big reason, oh the dock, oh it's pretty, oh the finder, but couldn't think of any quantitative reason why they were better. When it came down to it, there are just a million little reasons that all add up to a great experience.

To each their own though, I don't want to start a Windows vs. Mac flamewar, there are plenty out there if you feel the need just type "I hate Windows" or "I hate Mac" into google if you must. The point I want to make is that Apple thinks about their products from a full end-to-end experience, and that's where something like an iPad could beat the pants off of a netbook. Make no mistake about this either, that is where they are targeting, Jobs said it in the keynote and the pricepoint confirms it. iPad is not meant to compete with MacBooks or Dell D830s (the computer I'm currently typing on) they are meant to compete with Dell Inspiron Minis and eee PCs

There is a niche market out there and I think Apple has the marketing and business savvy necessary to corner it. With all the hype it would have been difficult to not be underwhelmed. It doesn't cure cancer or come with a free pony, boohoo!

Now I'm going to take my life into my own hands here and make a prediction (watch out!) The iPad will succeed, it will have a spreading pattern to certain niches until it picks up a critical mass and people who don't really need them start to buy them. First the early-adopter and mac-fanboys will start buying them up, then there will be some app that endears it to some group of people in a fierce way. Maybe a text-book app or medical records app, whatever it is there will be some population that is in love with the iPad. They will start to infect their friends and family with iPad fever and it will become hip and cool, then it will become annoying, and then it will just be fact, 30 jillion people have iPads and you won't be able to walk into a Starbucks without someone reading Hipster monthly on one.

I'm excited about the iPad, it doesn't have everything I always wanted, but it will definitely move the ball forward. I personally think the very slightly modified iPhone OS was a mistake, I think there are lots of mistakes and things that I wished were included (like that pony I really wanted). But I'm going to have to reserve judgment until I ca go down to the Best Buy and screw around with one.

objective measure

[caption id="attachment_642" align="alignright" width="300" caption="this ruler is deep and spiritual, that\'s why it was photographed in black and white"][/caption]

I've found myself in an odd situation as of late regarding my work, I am a programmer by trade and education but haven't done any professional programming for 2-3 months now. It's not that I'm being lazy and don't want to program, I would jump at the chance to fire up Eclipse right now and start writing mountains of verbose Java code, it's that I'm not allowed to program. "Not allowed?" I hear the shock and dismay in your voice, so let me explain.

My role on this project is to design middle-ware but because of the way the business is set up I'm not actually allowed to program, the programming is all outsourced. Instead of actual programming I get to engage in some weird form of meta-programming where I turn a program into long flowing sentences in a Word document that take about 3 times the effort to write than the program itself so that it can be shipped off and someone else can program it. It is an odd situation indeed, but I am no business man, this project is important, and so I shall continue in this weird meta-programming.

This is not my first dance at the outsourcing-prom and with the economy in shambles I'm certain it won't be my last. I have yet to work on a project where the supposed benefits of outsourcing have actually materialized. I did some research (typed something into Google) to attempt to find out how many outsourced projects fail and saw figures between 25-40%. These seem fairly reasonable to me, and with some statistical hand-waving I will use these as "good enough" figures for the rest of this post. This leads to an interesting question though, if this strategy fails so often why do companies continue to outsource?

I would like to posit that a major reason for the continued use of outsourcing is the gap between subjective and objective measure. Subjective measures come from within, they are personal opinions, things like "Frank is a great worker!" or "Jim is so lazy" are subjective measures. Objective measures are fact-based measures things like "Frank worked 48 hours this week" or "Jim only produced 1 class file this week."

Often, like in the examples above, our objective measures inform our subjective measures. The problem with software development is that there are no good objective measures. As much as we like to think of what we do as a science or even an industry, it is still a craft. In the above scenario we are led to believe that Frank is a "great worker!" because he worked 48 hours this week and that Jim is "so lazy" because he only produced 1 class file. If Frank worked 48 hours doing simple things and just racked up the billable hours is he really "working" harder than Jim who wrote a full web server in one week (monolithic in design so it was only one class). Getting to the heart of the value that someone produces is a difficult process and is by its nature subjective.

Here in lies the problem, software development can really only be measured subjectively, which is hard. Objective measures are much easier though, and so those are often used as a proxy for doing the difficult work of subjective measures. There have been many attempts to make sure people are performing up to par, lines of code, commits, bugs, hours worked, etc. None of these are really ideal, together they can begin to form a picture of the subjective nature of the situation, but they are not nearly enough.

Objective measures are also very seductive because you can perform calculations on them, they are easy to compare, they are easy to play out what if scenarios with. No one balks at answering the question which child is taller, but ask them which one they love more and watch them squirm. Objective measures are great because you can throw them in excel and analyze them, this is also a weakness. It's easy to draw up a spreadsheet with your in-house assets pitted against outsourced assets and see that you can save a bajillion dollars by outsourcing, but that misses the bigger picture. What are the subjective downsides?

These subjective downsides end up failing some 40% of outsourced projects, communication, quality control, and responsiveness are hard things to chart and graph, but can easily cause a project to run off the rails. What is needed is either some way to make subjective measure easier or some fantastic new unit of objective measure. I would propose ihumanables per hour be the unit if anyone can come up with some really nifty objective system, but I don't see that happening anytime soon, so all that's left is making subjective measure easier, a tall order indeed.

At the heart of making subjective measure easier is really what a lot of the agile methodology is about, stand up meetings, scrums, kanban boards, they are all meant to produce a tight feedback loop so that the subjective nature of the situation is constantly being polled. I'm not going to sit here and tell you all to start running agile projects, although they can be a good bit of fun, but if you are managing people you need to get in the dirt with them. Progress reports and weekly status meetings are meant to supplement not substitute actually working with and understanding what the people under you are doing.

Remember the next time you are making a decision based off of objective measure, if you are "going by the numbers" that there are very real subjective side-effects that you should account for and be aware of. And if someone develops a really killer objective measure for software development, remember ihumanables per hour ;)

when in rome

[caption id="attachment_624" align="alignright" width="300" caption="Trust me, you want to do what the Romans do, I mean... have you ever built a Collisium?"][/caption]

Ah the cliché of it all, "When in Rome." Although it would be nice to have a post about being in or traveling to Rome, sadly this is not the case. Instead, this post is about following convention and how that can help you learn.

I've been getting into ruby as of late, after learning Smalltalk, Objective-C, and LISP (to varying degrees), ruby hits a nice sweet spot of stealing some of the best bits from each. I was recently approached by a friend who does primarily .Net programming about the best way to go about learning ruby. He had decided that, at least in the beginning, he should use the NetBeans IDE. This seems harmless enough, even logical for someone who spends their entire day in Visual Studio to want the comforting guidance of the IDE. It is also the exact wrong approach.

As he progressed past simple "Hello World" examples and onto his first rails project he found himself battling with the IDE. Odd error messages, the IDE struggling to provide auto-completion on the dynamic models, but most importantly a complete lack of resources. The ruby and rails world to a large degree lives and breathes on the command line, right now if you go to my home computer you will find 4 terminal windows arranged with a mysql shell, script/console, script/server, and one for git. If something goes wrong and I get a weird error, I do what all great programmers do, copy and paste that sucker into Google. I normally find somewhere around a bajillion hits with solutions, huzzah!

My friend on the other hand had trouble doing the simple things through the IDE, like installing gems. I'm certain the IDE was capable of it, and that someone better versed in it would be able to help him out, but the Google-scape was barren to his questioning. All I could say to answer his questions were, "well I'm not sure how NetBeans does it, but on the command line just type gem install [gem]" (Also I can speak in teletype, I just do a robot voice.)

Despite the difficulties, my friend clung to the belief that the IDE was the way to go for beginners. I finally realized the perfect analogy to drive home the point, I asked, "Would you ever advise someone new to .Net to learn it by using the command line tools?" It's a perfectly valid question, I'm sure plenty of people who hang out in vim all day and don't mind doing things like ./configure && make && sudo make install (thanks Jeremiah for pointing out my n00bishness) would feel much more at home on the command line.

I am not free of fault on this either, when I attempted to learn git for the first time, I used TortoiseGit. I was very comfortable with TortoiseSVN and thought it would ease the transition. It sure did, I was able to treat git exactly like svn, and completely miss the point! Once I moved to the command line (and watched some screencasts and read some articles) I felt much more at home with git, and even began to understand why I would want to use it over svn. I had stopped trying to make it something it isn't and embraced the thing that it is.

The point here is that when you learn something new, it's new, don't bring along your technological baggage. If the community all rallies around some IDE (.Net, I'm looking at you) then for flying spaghetti monster's sake, use that IDE. If they rally around the command line (ruby, I'm looking at you) then by the hammer of Thor, use the command line. If they rally around some weird virtual environment machine (smalltalk, I'm looking at you) then have fun and realize you are never going to be able to use that at work (just poking fun, I <3 smalltalk).

The point here is that learning something new is often times uncomfortable, you have to feel like a tourist for a while. When in Rome though, do as the Romans do. Embrace the methodologies, naming conventions, and tools that are the community standard. Give it a few weeks, if you still find yourself hating it, then maybe that particular technology isn't for you, it's ok, we can't all love everything. Life is about trying new things and figuring out what makes you want to get out of bed in the morning, for some people it will be .Net, for some ruby, for some COBOL (just kidding, no one likes COBOL).

You never do yourself any favors by half-way learning something, because you will either hate it because it was poorly shoehorned into an inappropriate paradigm, or you will learn to love it. "That second thing doesn't sound too bad" (I can sense your thoughts), and at first blush its not, until you want to start participating with the community by using others' code and sharing your code. Then you will have to unlearn all the bad practices you've adopted and have a difficult transition into the community, attempting to erase months of muscle memory. Save yourself the difficult task of trying to unlearn something and simply embrace the technology in full to begin with. It's a steeper curve, but the payout is the depth of understanding.

api

[caption id="attachment_618" align="alignright" width="142" caption="oh creator of hot pockets, we praise thee!"][/caption]

I remember being a young lad preparing myself for university I was given a gift from my mother, "C++ for dummies." The vote of confidence on my status as a "dummy" aside, I read the book with great interest. There was an analogy the author used to explain the idea of classes and what functions they should expose, I'm going to shamelessly steal it (paraphrasing as I don't have the book with me).

Imagine your son comes up to you and says he wants to make some nachos. You tell him that its fine by you, just go cook them in the microwave, and there is nothing controversial about this statement. Microwaves are actually boxes full of high energy radiation being produced by cavity magnetrons or waveguides, to the end user, the microwave is just a magic warming box. It exposes a very simple interface, some buttons that allow you to enter the amount of time you want to cook something.

This is the essence of an API (Application Programming Interface), wrapping up something complex and possibly dangerous in something safe and easy to interact with. When building code that you intend other people to use someday, it is the part that is most important part, and the part that is easiest to overlook. The problem is that we are often too close to something and too concerned with our use case. If you want to design code for others to use, it requires significant time and effort, and even then you probably still won't get it right.

Prosper is still undergoing active development, I'm currently agonizing over how I want to expose various execution modes. The solution, no matter what I pick, is trivial to implement, but the api is the most important part. A great api exposes a consistent concept, something that is easily grasped and allows the end user of the api to declare what they want to do without having to worry about how its going to get done. Since good programmers write good code and great programmers steal great code, I've modeled the api for prosper extensively off of jQuery. And why not, let's take a look at two different APIs, the browser dom api and jquery.

[javascript]
//Let's barber pole a list by coloring every other element blue
var list = document.getElementById('the_list');
var highlight = false;
for(var i = 0; i < list.children.length; i++) {
if(highlight) {
list.children[i].style['backgroundColor'] = '#FF0000';
}
highlight = !highlight;
}
[/javascript]

Fairly straightforward implementation, but it concerns itself heavily with the "how" of what its doing. Manually traversing to pick the elements it wants, eww.

[javascript]
//Same thing using jquery
$("ul#the_list li:odd").css("background-color", "#FF0000");
[/javascript]

jQuery is definitely magic, but this code is great because it let's you focus on the "what" of what you are doing. How does jQuery go about selecting the right elements and all that? I don't care, and the great thing is I don't have to care, and if in the next version of jQuery they find a way to do it faster, I win without having to do anything.

Writing a great api is difficult, you have to divorce yourself from the concrete problem you are solving and look at it in the abstract. Put yourself into the shoes of someone trying to figure out how the api works, and then ask the questions they are going to ask.

• Why do I need to build up this context thing and pass it in?


• How come there isn't a sensible default for these arguments?


• What dumbass made this thing?

Answer those questions, and keep working at it, strive for elegance and consistency, because then it will be easy for people to learn and use. If your code is easy to learn and use, people are going to want to use it more, and they are going to want to tell their friends about it. Then you can get some lucrative ad campaigns with Nike because of the boffo library you write in FoxPro.

There is a more subtle lesson in all of this though. Any code you write is exposing an api to someone else. "But only I am ever going to use this code!" I hear the naysayers warming up their keyboards for the comments section. This may be true now, but the six-months-from-now-you is going to look back at the you-of-today and wonder what the hell he was thinking.

Get in the habit of making your code easy to use, and expose a nice api. This will endear you to your fellow programmers and help make maintenance easy. Strive to be that guy on the team that writes functions and classes people want to use. Make life easier for your fellow developers and even if they don't return the favor, maybe they will take you out for a beer. People notice over time that your code is the best to work with, they internalize it, and they start to think of you as a great programmer. That's a pretty great api to expose to the world.

hustle

[caption id="attachment_613" align="alignright" width="299" caption="That time was so much more glamorous"][/caption]

A realization has been dawning on me as of late, one that I've always known, but that is easy to forget, easy to misplace, easy to neglect. The best way to learn anything is to do it, to struggle through, to forge on, to fight and gnash teeth and curse at. There is no knowledge as highly regarded as that which you have to work for.

I'm not even particularly certain how this all came about, it seems to have been a culmination of things. I didn't intend to but somehow I ended up putting on my plate a bunch of new stuff to learn around the same time. Git, ruby, rails, blogging, prosper, hosting my own domain, and testing (Testing being a decidedly late addition). This blog was the genesis of a lot of it, that and just being curious.

I had my first post back on October 2nd of last year, as of writing this 112 days ago. At that point I didn't really know anything about the 7 topics above, I started the blog using Google's Blogger platform, it was an easy onramp.

After a month or so, I was encourage by some friends to move the blog to my own domain, which you are now at. I also kicked off the prosper project around the same time as starting this blog. I played around with ruby and a crazy friend convinced me to try out rails. I moved prosper to GitHub and began learning git. Now prosper supports 19 backends and has gained over 100 unit tests in the last few days.

Here's the kicker though, I don't know what I'm doing. I never really did, I just started doing stuff, and have been running ever since. There have been days when I've gotten 5,000+ hits on this site, several days with several hundreds of hits and I've been steadily increasing the daily reader count. I don't know how I did that, I just kept writing, I liked it, I promoted it over twitter and hacker news if I thought I had written something insightful, and I guess that is working. The same thing with prosper, I saw a need for something that didn't exist, I had a vague idea of how to make it, and I started coding. I've rewritten and refactored it piece by piece to the point that it supports all kinds of things I never thought it would. It's due for another architectural change after I finish these unit tests.

If there is one key thing I could convey to anyone reading this is to hustle. You will never be prepared for the things you are capable of doing. You will achieve your greatest accomplishments not by building up a grand framework of skill and then deftly creating something glorious, but by starting small and persevering in making it better and better. It is never an easy road and you will gain a grand framework of skills, but you have to push your boundaries to grow.

This came together for me last night, I was working on a rather tricky bit of rails and broke something. I hadn't gone too far into it and so I typed rake db:rollback and git reset head --hard and was back to a working application. I stopped myself for a second and thought about what I had just done, how improbable it all was from where I was just a month ago. I then thought about what I was doing in rails, and thought about how earlier in the day I was wrapping up prosper functionality in unit tests and finding regressions, and how I would have to write some cucumber tests for what I was doing, and I realized that 112 days ago I didn't have the vaguest idea of any of this.

I would love to put a triumphant "I'm just so damned smart and talented and handsome" paragraph here, but that's not the case. I just steeped myself in this stuff, I worked in git daily, I read about it, watched screencasts, I bought agile web development in rails, I got design patterns in ruby, I hustled. And you can do it too, take the first step today.

The first step that you should take is to invest yourself in something non-trivially. Want to learn rails, then go buy agile web development in rails, want to learn github, move an active project out there, want to learn linux, reformat your machine so that's all you have. You have to invest yourself, it plays a trick on your brain that makes it want to not waste that "investment" by quitting. If you can burn your boats (Hernán Cortés reference) all the better. I had no choice but to learn git or else I couldn't keep working on my project, and as a side bonus I got the joy (and frustration) of working in git everyday.

Get out there and hustle, learn something new, do something that scares you, reach beyond your grasp.

unlicense'd

[caption id="attachment_606" align="alignright" width="300" caption="Freedom has never kicked so much ass"][/caption]

I wrote yesterday about the importance of open source software. I got one comment saying that prosper would be awesome if it were in the public domain. Here is the comment from Arto Bendiken

Prosper looks *sweet*. It’d be awesome if Prosper indeed were in the public domain. I think such a move would make for a decisive selling point over the numerous competing PHP libraries, setting Prosper clearly apart from the herd and allowing it to be easily embedded into any new emerging PHP web frameworks.

It’s worth mentioning that a key reason that SQLite, which is in the public domain, has prospered (no pun intended) has been the fact that it can be so easily (both technically and legally speaking) embedded into any other software. The end results of technical excellence combined with singularly liberal licensing speak for themselves: http://www.sqlite.org/famous.html

I think a lot of these same benefits could come into play also in such a potentially key higher-level web app infrastructure piece as Prosper. It certainly seems that you have the technical side of things well in hand, already, so here’s hoping for a bright and prosperous future.

Well I completely agree with Arto and want to clear up any confusion about prosper and its license. Prosper was original hosted on Google Code, which requires you to choose a license, at the time I selected the MIT/X11 license based off of my belief that it was the most free. A little further down the road I saw the WTFPL - Do What The Fuck You Want To Public License and thought about switching to that. There is a bluntness to the ideal of that license that appealed to me.

The problem with the WTFPL is of course its name, I want prosper to power lots of things and that means taking into account a wider audience. While hackers and rock star programmers can easily adopt something with WTFPL, I didn't want to be responsible for some corporation turning their back on an otherwise useful library because they were squeamish about a word in the license.

[caption id="attachment_607" align="alignright" width="220" caption="I always thought people just hated the letter C... then I found out this means no copyright"][/caption]

Then I found this post on Hacker News (ironically by the same Arto Bendiken) and it lead me to the Unlicense. The Unlicense is exactly what I was looking for and simple to adopt. If you go to the GitHub repo you will see that the project is now officially Unlicense'd. The 0.8 release will be the first official Unlicense'd release, and so will every release going forward. I believe in free and open source software and so I'm publicly declaring now that prosper will always be free, open source, and unencumbered. The mechanism that I will use to ensure this going forward will be Unlicense.

What's the big deal, plenty of things are open source, why is this any different? The difference is that I'm giving this away fully, without any restrictions. Want to build a cool new library that uses prosper as the low-level database abstraction layer, go for it. Want to build a proprietary ORM that utilizes prosper, go for it. Want to repackage prosper and try to sell it (as underhanded as that is), go for it. Want to smother prosper with jam and eat it, if you can find a way to, go for it. Want to.... enough already, you can do whatever you like.

It is easy to talk about things like freedom and support them in the abstract, it becomes more difficult when you have something concrete. People could very well take prosper and take it in directions I don't care to see it go, but that is life, that is creation, that is the essence of open source. It is my goal to see prosper provide a solid foundation for building new and better tools and frameworks in php. I want as many people as possible to use it, and so this was a logical choice.

The next time you have some useful chunk of code lying around and you want to give it to the public domain for the good of the public, think about using Unlicense. I can't say it any better than they can so I will just quote their section about why you should use Unlicense. "Because you have more important things to do than enriching lawyers or imposing petty restrictions on users of your code."

open source

[caption id="attachment_602" align="alignleft" width="300" caption="insert stupid star wars pun here"][/caption]

I've been getting into the guts of the SimpleTest trying to make it bend to my will. So far I've been pleased with the framework, I've been able to set up a testing structure that works for me. To understand the testing structure better you need to know a little about prosper's structure.

Prosper uses a classic adapter pattern to handle the different backends it supports, so it has a directory structure similar to the following

• Query.php


• Adapters/




• BaseAdapter.php


• PreparedAdapter.php


• MySqlAdapter.php


• etc...

Since I wanted the unit tests structure to make some sense I decided that there should be one point of entry that I could run to test every adapter and then entry points for each adapter individually. I ended up with a test structure like so.

• All Tests (Suite)




• MySQL Tests (Suite)




• Select Statements


• Insert Statements


• etc...




• Oracle Tests (Suite)




• Select Statements


• Insert Statements


• etc...




• etc...

[caption id="attachment_597" align="alignright" width="300" caption="All my tests pass but its just so boring"][/caption]

This structure is working nicely and now I get to have the fun of writing hundreds and hundreds of unit tests, it really puts into perspective the shear size and scope that prosper has grown into. The next thing though was that I wanted beautiful results. If I was going to have to look at these things all day, I wanted them to caress my eyeballs and make them feel all warm and fuzzy.

SimpleTest (like all PHP code) is open source and even has a way to extend the reporters, although its easier said than done. I fired up my favorite php editor and got down to work rewriting the HtmlReporter class so that the results were more to my liking. It was not the easiest task, but there is some ok documentation about it and the names were all fairly sensible. I hacked away at it, changing a little bit here or there and then finally decided to just scrap what they had wrote and redo most of it. It took some time to get everything squared away but with some inspiration ...cough...blatent theft...cough... from QUnit, I was able to make the following.

[caption id="attachment_599" align="aligncenter" width="300" caption="such hotness! unit tests have never looked so good"][/caption]

This is why I love open source so much, I wasn't stuck wishing that this thing looked better, I didn't have to resort to some sort of greasemonkey magic, I was able to fundamentally alter simpletest by changing the code. This is also why prosper is released under the MIT License, although recently I've been thinking about moving it to the Unlicense. I want people to be able to fix bugs, play around and change things, and ultimately make prosper a better tool.

Right now there are some Prosper specific hacks going on in the new reporter I wrote and its only been tested in Firefox, since its what I develop in when I'm on windows. I plan on cleaning up the code a bit and then submitting a patch to SimpleTest containing my new QHtmlReporter. This is part of being a good open source citizen, you can reap the benefits but then you should also try to pay it forward or back.

Open source software is empowering, not only in what it can do, but how it empowers the users to make them better. I'm going to continue writing up unit tests and with this awesome new display, my eyes will be happy to watch the results scroll by.

unit test

[caption id="attachment_583" align="alignright" width="300" caption="How much time did I spend looking at this instead of typing tutor in elementary school"][/caption]

I've read the blog posts, heard the speeches, seen the tweets about unit testing. At a conceptual level I understand, you should unit test because then when you make changes you can find regressions easier. I've even heard the TDD people beating their drum about how I should never write code without tests. I get it, it sounds really good, and here is the dirty secret, I don't do it.

For those of you who have continued reading don't worry this isn't another volley in the war between TDDers and non-TDDers, this is about why I couldn't wrap my head around testing, and how I finally did, and where I am now. I hope that if you are in the same place I was, you agree that the idea of testing is really good, but can't figure out how to incorporate it yet, that this post might help you. I'm also going to take you down memory lane a little bit, but there is some good knowledge about a cool PHP testing framework in here, you can skip to it, if you like.

I grew up in a little house just outside of Cleveland, Ohio with an incredibly warm and caring family. There were 4 boys, my mom, and my dad. There was never much money around, but I never realized that this was such a bad thing, I had what I needed and a little something more, a computer donated to us by my Uncle Marty. The computer was a magical thing at first, I was 5 or 6 when it arrived in our house, an old black and green with less RAM than the Rockstar energy drink I'm sipping on right now. You could put in 5.25in floppies and play Dungeons & Dragons, Hitchhiker's Guide to the Galaxy, or some weird Submarine Game. You could do something else though too, if you put in a blank disc you could make your own game. My uncle showed me the basics of BASIC and my school library had a book filled with BASIC code that could make egg drop games and even a star trek rip off. I was hooked.

[caption id="attachment_584" align="alignleft" width="300" caption="Oh those halcyon days, you were all I needed dosshell"][/caption]

As I became a slightly older lad, still around 8 or 9, we had another family donation. This time my Uncle Bob had upgraded their family computer, and so we got the old one. This one was a marvel though, it had colors and could even go online. It had windows 3.0, although it was really just this odd thing we fired up from time to time to play in, we spent most of our time in dos shell or on the command line. I wondered if this computer had BASIC, and I found QBasic.

QBasic was were I really started programming, although the current programmer in me wants to throw up reading that sentence. I was no longer just copying code out of a book and making changes here or there, I was coming up with new ideas and figuring out as I went along. The built-in help system coupled with fairly readable error messages and a tightly integrated run time made learning easy and accessible to me. I would code stuff up and give it a go, I didn't realize that I was running an interpreter or had any idea what a compiler was, I would just develop and try it out. It was my REPL, although much more manual.

I moved from there for a short time to Visual Basic, and then at BGSU to C and C++. The C languages seemed so onerous to me at first, there was this lengthy compile step, I had to declare everything, there was odd punctuation floating about. But I got used to it, and became fairly proficient in it. Towards the end of my time at BGSU I took a job with the University fixing computers for RCC and from that job joined a team of PHP programmers.

Something felt familiar and comfortable about PHP development, I would just type some stuff and refresh the browser, immediate feedback. That's when I realized how my brain was set up, wired from the radiation of that old green and black monitor, I wanted to develop in quick insanely small iterations. Change this or that and test it out. Toss in an echo here or there to see the state and make decisions based off of that. It was always more structured than I'm making it out to be here, but the structure was there to support this notion of Rapid Application Development.

To this day I still find myself more comfortable building something new by getting the simplest thing that could possibly work (Minimum Viable Product) up and running and then doing iterations over it again and again, polishing it up. When I coupled this strategy with really aggressive refactoring I was able to produce some cool stuff, complex reporting architectures for clients, constricted language parsers, bytecode compilers, and to a large degree prosper.

[caption id="attachment_586" align="alignright" width="113" caption="Thank you creepily fake referee man for illustrating my point"][/caption]

Testing though seemed like a great idea, TDD seems like a nice way to write code, tell me what you are going to do, then do it. Makes sense. I tried to do some unit testing but had some false starts. I tried to do it with some Java code, wrote up a jUnit test, ran it with an ant task and it told me everything was cool beans. Did a few more and then couldn't figure out what the point was, I was testing things I knew had to work, if variable assignment stops working in the JVM then my unit tests aren't going to make a difference. Then I worked on 2 small projects that were Test Driven and they both hampered development so much that the project either failed or I quit altogether from it. My innocence had been dashed and now the thought of TDD left a sour taste in my mouth.

I had secretly hoped that like pair programming, scrums, and kanban boards this would be another good idea from agile that would have its time and then quietly be swept aside as impractical (of course I'm not saying these things are impractical, just that none of the enterprise shops I've worked at cared about them at all). After CodeMash I realized that these things weren't going away, so I redoubled my efforts to learn them and that brings us to today. (That was a long and weird trip if you read through it instead of just clicking the "skip that stuff" link, thanks!)

SimpleTest

It's all about tools. I've tried various testing frameworks in Java and .Net but never found something that fit in with my workflow. I sat down at the Oracle of All Knowledge (Google) and started looking for a unit testing framework for PHP. Then I found SimpleTest, the name immediately drew me in. It was exactly what I was looking for, something simple I could use to write unit tests with. I went through the tutorials and figured it all out and it was exactly what I wanted, I could write up my tests and then just point my browser at http://whatever/tests/all.php and it would generate a cool webpage telling me exactly how my tests are doing. I was converted.

I'm making some fairly big architectural changes to the internals of Prosper for the next release (0.8) and have decided that part of the effort is going to be wrapping stuff up in unit tests, lots and lots and lots of unit tests. This is my first go on unit testing so I'm certain I will get some things wrong, but I'm hoping to learn a lot, and you can play at home and tell me what an idiot I'm being.

At the end of the day I'm beginning to move from, tests are cool but I don't test, to tests are cool and I'm learning how to do it. I've been down this road before with git and ruby and rails, its frustrating but worth it. SimpleTest is really cool, but I'm still finding my way through it, I'm writing a new output printer inspired by John Resig's QUnit and will release the code for that soon. Once I have a workflow and test directory structure I like I plan on producing a series of posts. You can find them because I plan on tagging them all with "Dr. SimpleTest or: How I Learned to Stop Worrying and Love the Test"

codemash review

[caption id="attachment_577" align="alignright" width="300" caption="Colonel Whiskers and I have both missed you dearly"][/caption]

You may have noticed that I have not posted anything since last Tuesday, gasp! Fear not, although various shadow organizations are still after me, I was not captured. I had the awesome opportunity to go to CodeMash, and spent most of last week in the beautiful Kalahari Resort in Sandusky, Ohio.

CodeMash, for those of you not in the know, is a pretty cool event. Instead of focusing on one technology (.Net or Ruby on Rails for instance), there are tons of sessions spanning the technology spectrum. Want to learn about ruby, there is something for you, F#, got it covered, Clojure, for goodness sake they even did a lisp. The great thing about this style of conference is that you can pick and choose whatever interests you, nibbling on concepts here and there to get a bit of everything, or really chowing down on one topic.

The other cool thing is that the company I work for HMB is forward thinking enough to not only sponsor CodeMash, but to send me there for free! Then the big topper came last month when they asked me to present on prosper.

Well now it is all over and I can give you a rundown of how things went.

The Good

• Kalahari - The Resort is beautiful, the rooms are huge, the staff is excellent. It is a really cool place to spend some time in, they served us some great food, and the facilities were excellent in their range, rooms small enough for a handful of people to collaborate in, all the way up to grand dining halls to house the 800 or so attendees.


• Keynotes - Mary Poppendieck gave a great talk on Lean Software Development, Hank Janssen talked about how Microsoft is moving into the Open Source world, and Andy Hunt, co-founder of Pragmatic Bookshelf and author of the Pragmatic Programmer, gave an amusing talk on the Mother of all Bugs, our brains.


• Prosper Talk - After weeks of preparing and fretting over my presentation I think it went well, there were some technical difficulties, but nothing that could stop the incredible power of Prosper!


• Session Leaders - The people leading the sessions don't get paid, they do it out of love of software development, and it shows.

The Bad

• Superficiality - Sessions are only an hour long, this means that no matter how great the sessions are they can't really get into the guts of anything in any kind of depth.


• Beginner Overlap - I went to a lot of Ruby sessions, as that is my current love, and even as a n00b it became tiring to hear the 5th explanation of duck-typing for the day.


• Kalahari is way too big - I'm just kidding about this one, but it was honestly about a 2 mile hike from the convention floor to my room.

What's Hot

• Ruby - Ruby is blowing up, not just rails anymore. Rails did a great job of introducing the world to the quirky little language that could. Now innovations are being made inside the Ruby community and people are starting to port that out to the rest of the software world. (See the next point)


• TDD and BDD - Test Driven Development and even moreso Behavior Driven Development are becoming huge and are here to stay for the Agile practitioners among us. A big driving force for BDD is Cucumber.


• Cloud and NoSQL - I'm lumping these two together because they are both technologies at scale. Virtual infrastructure is becoming a big business, be it Amazon's EC2, Microsoft's Azure, or Rail's Heroku. Applications running at huge scales are also turning to NoSQL storage solutions.

The Point

CodeMash is a chance to meet really cool people how are excited about technology. It's a chance to get exposed to a ton of cool technology. It is a chance to try a bunch of cool stuff and see what you like. Sessions are only an hour long so at worst you end up wasting 60 minutes, at best you get fired up on something wonderful. You get to meet a lot of cool people, here some great talks, and best of all you are at a gorgeous indoor water park that is 84 degrees when its freezing out in the harsh wasteland of Ohio.

who cares?

[caption id="attachment_573" align="alignright" width="300" caption="I shot who in the what now?"][/caption]

It's one of my favorite questions to ask during a technical discussion. Who cares? Some others I like to throw out our, why should I care? why does it matter? and I don't care! Am I some sort of raging lunatic completely out of my mind and trying to pick a fight with everyone I meet? Yes, but that's not why I ask these questions. I ask these questions because as great as the big picture is we sometimes have to drop complexities to get stuff done.

When you have a big flow diagram or giant UML Class Hierarchy it can be fun to think about the abstract way that things interact and how you can pass state around and whatnot. This definitely is useful for getting a feel for a new system, but then there comes the worst part of any software developer's job, actually getting things done. You see that's what we get paid for, actually putting our fingers on keys, typing stuff out, compiling it, and shipping a product. Part of this is doing the upfront design work and getting all of our t's crossed and i's dotted, but just as important is actually getting something working.

I talked about this before in my post about simplifying things. This simple question, "who cares?" (you can be more diplomatic about it if you like) is a great tool for getting to the heart of a problem. I like the simplicity and straight-forwardness of the question, it normally catches people off guard and challenges their assumptions.

Frank: So we need to pass this data into the layer
Bob: Who cares?
Frank (taken aback): I care, I care deeply about this data
Bob: But why do you care about it being in that layer, its just going to hand it off to this layer, and that layer doesn't care at all about it
Frank: I guess you are right, we don't need it there

BAM!!! Problem simplified! For anyone that's ever taken any education classes you may recognize this as a rather blunt application of Socratic Method. The point is confrontation, but not between the two people discussing something, between each person and their preconceived notions about a problem or situation.

In High School I took a chemistry class, it is one of the few classes that I really remember in any detail from so long ago, the teacher taught the entire semester in Socratic Method. He would question us, we would experiment, we would reason out why we observed what we observed, he prodded us, and we cherished our knowledge. Anyone can tell you water is composed of 2 hydrogen atoms and 1 oxygen atom, but when you had to work for it, when you had to defend your knowledge of it from scrutiny, it became concrete, it became your discovery.

Applying this to software development is useful, we can get locked into our preconceived notions, and direct, blunt, sometimes even rude questions can spur you to really consider what you are talking about. Software is incredibly fluid, sometimes an idea or concept can become out-dated in a matter of months or a matter of hours. The biggest ones are the most difficult to challenge, because they often have the most dependent code around them, but this makes them the most important to constantly consider.

Keeping your conceptual design, implementation, and solution in lock step with a problem domain often involves some amount of change, that change can be hard to come by though. By challenging our standing beliefs we can find new avenues for exploration or strengthen the existing techniques and gain a better understanding of them.

grok

[caption id="attachment_565" align="alignright" width="282" caption="necessary part of grokking something"][/caption]

Computer nerds should be familiar with the term grok while the rest of the populace has no need for such a silly word. You may hear it down at your favorite nerdery, "I don't quite grok this" or "Once I grokked it, I flew through the project." What does this word mean though and where did it come from.

Grok for the non-nerd means, "to understand thoroughly and intuitively." Although this is true it belies what it truly means to grok something. For that we need to realize the origin of this fantastic geek speak, this term is straight out of Robert A. Heinlein's Stranger in a Strange Land. He defined it as follows

Grok means to understand so thoroughly that the observer becomes a part of the observed—to merge, blend, intermarry, lose identity in group experience. It means almost everything that we mean by religion, philosophy, and science—and it means as little to us (because of our Earthly assumptions) as color means to a blind man.

The great thing about grokking something is that it changes your world view, it fundamentally shifts the way that you see, perceive, and understand the world. The bad thing about grokking something is that it is incredibly difficult to get to that point. Let's take a look at a fantastic chart I've whipped up to help us understand this better.

[caption id="attachment_566" align="alignleft" width="300" caption="the graph of grokitude"][/caption]

In this graph we have time and effort along the horizontal and understanding along the vertical. We can spend a great deal of time after "mastering" a subject before we get the epiphany and truly grok something. To really understand a subject on the level that a computer nerd would consider grokking it takes not only a full in-depth knowledge of the function of something, but an understanding of the design, an internalization of the principals, and a familiarity that borders on oneness.

This is the reward for hundreds or thousands of man-hours spent toiling in something, the day when the clouds part and you truly understand something, this has an amazing non-linear effect on your understanding and your productivity. Once you grok something, be it a framework or a library or a concept, your ability to use that and your productivity in it increase in a non-linear fashion.

Although it is important to have a wide breadth of knowledge we must remember the importance of having a few areas of incredible depths of knowledge. These areas will permeate everything else you do, so try to choose wisely. The subjects that you truly grok are the ones that will effect your big picture outlook.

Here are my suggestions for some fundamental things to learn, and learn to a point that it fundamentally shifts the way you look at the world.

• A Pure Object Oriented Langauge - I would suggest smalltalk for purity, but ruby makes a fine substitute and has more application.


• A Functional Language - For this look no further than Haskell, and Learn you a Haskell for Great Good


• A Set Language - Take the time to learn SQL, its a tool most of us use everyday and very few of us take the time to learn properly


• A Markup Language - HTML + CSS, these are the embodiment of separating data and display, learn them intimately.

Learning these, but more importantly grokking them will color the way you approach problems and solutions. Having a depth of knowledge in a key variety of tools allows you to quickly master new ones, better understand problem domains, and more completely express your thoughts in the abstract.

limits

[caption id="attachment_558" align="alignright" width="300" caption="its beauty is only matched by its cliffy-ness... or something"][/caption]

I had to make a hard decision today, it was one that I had known I would have to make at some point, and frankly I was just putting it off. I've written about burnout before and I'm no stranger to pushing myself harder than I should. But I like to think that because I'm aware of the danger and am actively working to avoid burnout that I can push myself, and still maintain some sanity.

Part of maintaining sanity is knowing one's limits. I have a full-time job, prosper, and until today 2 side projects. Today I resigned from one of my side projects and I released any code or concepts I had worked on to the other developer. It was not something that I particularly looked forward to doing, but I had to prioritize. I had to dispassionately measure the various aspects of my life and make a cut before I burned out and lost everything.

Now I'm left with a strange feeling, almost ashamed that I let down the other party, but at the same time a sense of relief has come over me. After I made sure that I didn't burn down any professional bridges, another wave of relief. Looking forward in my calendar and seeing all the time I've got to work on Prosper and my other side project, a wave of excitement. The problem was that I was spread too thin, at my limit, I was pushing ahead 110% but not moving anything forward. Now though I feel back in control, pushing ahead and making things happen again.

I've given this new attitude a shot in the arm over the last few days. For the last few days I've been knocking out tasks that have been long outstanding and non-trivial. It's easy to trick yourself into thinking you are being productive by banging out a few quick trivial tasks and checking some stuff off the todo list, but picking long standing non-trivial tasks is important.

My girlfriend's father's computer was the first such task. It's hard drive died, and I've had it for far too long. What should have been a simple diagnostic-swap-reimage was bedeviled by problems. I didn't have the right keyboard to perform the diagnostic (all my usb keyboards were read by the bios but not the ubcd prompt), then the replacement drive I was provided was the wrong type, and the recovery disks were all oddly labeled. The task itself though is not that difficult, get the right drive, swap, get the right disk, click click type, wait 4 hours for reimaging. The giant tower sitting there day after day had become a constant and depressing embodiment of my inability to move tasks forward. Yesterday though I finally hooked it all up, found the right disk (on the third try) and reimaged it. Everything works fine now and the problem is solved.

For the presentation I'm giving at CodeMash I plan on doing a live demo, I needed to pin this completely down to the last detail and do all the boring set up work so the presentation can be all whiz-bang cool. After an hour or two hacking around this is now done. My minimal PowerPoint presentation is well on its way to being complete. Getting these non-trivial roadblocks out of my way is freeing up my mind to think of other things, to begin figuring out how to move other things forward.

My suggestion for anyone feeling overwhelmed is twofold. Be realistic, are you taking on too much, I was, it happens to the best of us. It's never a great feeling to admit that you've overextended yourself, but it's much better than just constantly being overextended. Are there tasks that are constantly at the back of your mind, nagging todo's that you keep putting off, well just tackle them. It's not fun, but when you put them off they start growing larger and larger in your mind's eye. The live demo only took around an hour to code through, even though in my mind it was important and therefore a monumental task.

Pick out one of those tasks that's been on your todo list for a while, knock it out, just having it done is worth the work.

codemash

[caption id="attachment_553" align="alignright" width="300" caption="I want you to come to Matt\'s CodeMash Session"][/caption]

Disclaimer: This post is shameless, and I mean really, really shameless, self-promotion.

If you have ever been on this site before you may recognize the name prosper. Prosper is my attempt at creating something of value for the PHP community. For anyone is following along at GitHub, you may have noticed that there hasn't been much work done in the last few weeks, what's the deal?! Well, don't fret, prosper is still under active development but it has been frozen for a few reasons.

1. It was the holidays, and I was doing this weird thing where you interact with humans


2. I've been seeking, and receiving, feedback from some key users


3. I will be giving a live demo of Prosper and have been working on that demo more than the library as of late

"A Live Demo?" you say bedeviled by the notion. Don't worry you don't all need to cram into my apartment to see some technological wizardry, all you need to do is head on over to CodeMash, then at 3:00 on Thursday head to room E. The awesome company that I work for, HMB is sponsoring CodeMash and have selected me to give a presentation on this crazy open source project they've heard I wrote.

I've been working on a very minimal PowerPoint, because I know people can read so there is no point in me reading to them. At the end of the presentation though will be a live demo building up a simple database backed PHP application for managing all the fond memories of CodeMash, a quote board. In case you are unaware, building an application in front of people is incredibly nerve-wracking, which is why for the next 5 days I will be building this application over and over while talking to absolutely no one, to the confusion of my poor neighbors.

Live demos have a fantastic way of just going all wrong, as Bill will demonstrate

I'm not trying to poke fun at Mr. Gates (for once), giving a presentation in front of people, with new software is never a fun experience, and bad things© do happen. I will be practicing and practicing and practicing, and after that I will probably be practicing, for this presentation to make it the best it can be.

In other prosper news, I have set up http://prosper-lib.com to be the new home of prosper (currently just a forward to here, although the update is still working its way through GoDaddy's intertubes so you may get a parking page for a while). I am working with my good friend Ashley Kruso to get a new logo for prosper. Once I have the new logo I will begin building a new web home for prosper that will be all kinds of pretty. This is all in the run-up to Prosper 1.0, which is a goal for this year, but I haven't quite nailed down when.

It's all steam ahead on Prosper, thanks to the contributions and support of many people. When I started this project a few months ago I would not have been able to guess what it would turn into, so thanks to everyone that has helped or is going to help. By month's end there should be a 0.8 release that will have all kinds of new fun stuff including deferred result sets and lots of bug fixes from my friend who was kind enough to evaluate Prosper, Ian.

toolmaker

[caption id="attachment_535" align="alignright" width="262" caption="Pictured Above: Programming"][/caption]

For about a year I had the unique experience of working at Abercrombie & Fitch's world headquarters in New Albany, Ohio as a Java Developer. It was a really interesting place to work, very stylish, loud music all the time, complex problems of incredible scale, and some of the best developers I've ever worked with. I was lucky enough to get placed into one of the most successful development teams, Planning. This team had developed a tool that was hugely popular and well regarded, and I was coming on board to maintain and improve this tool. The beautiful people and board-short dress code was just icing on the cake.

As a software developer this is a fairly basic scenario, there is some task that is complicated, business users want it to be less complicated, they give me some money, I make them some program, and everyone is happy. The odd thing though is that while programmers are completely used to making tools for others, they often forget that they can leverage this amazing skill for their own benefit. I distinctly remember a conversation I had with one of my colleagues there, Steve. It centered around the idea that most developers are happy to just get their assigned tasks done, but there are some that are toolmakers, and being a toolmaker is more valuable to a company than being a developer.

Fast forward to after my stint at A&F, I was on a new project and given the task of translating 80,000 lines of PL/SQL into T-SQL. The idea was to use...um...find and replace...or....something...and...um...get this to work. With such a fine technical specification and a mountain of PL/SQL staring at me, Steve's words bubbled up to my brain, "Make a tool." I had a time window of a some number of weeks to do the translation, it was a bit of a gamble, but I decided to spend a week writing a tool.

After remembering how C# worked after a year in the trenches with Java I had a program that took in the PL/SQL parsed each line and translated it to T-SQL. The task was made easier by the fact that I didn't have to write a generalized parser for PL/SQL as the 80,000 lines were written in a fairly consistent style, which greatly simpified the translation tool. After running the tool a few times, fixing some bugs, I had finished the task in a fraction of the projected time. Triumphantly, I went to my PM to show how I had just saved weeks of manual laborious work. "Great, now here is your next task...."

[caption id="attachment_538" align="alignnone" width="150" caption="Expected Reaction"][/caption]

[caption id="attachment_539" align="alignnone" width="150" caption="Actual Reaction"][/caption]

So I didn't get the praise I was expecting, that's fine, I'm not 4, I don't need a cookie. I continued on in my tasks quietly pleased with saving myself weeks of tedious work. Then the day came when the database requirements changed, the hundred or so CREATE TABLE statements would need to be altered. The task wheel was spun, and since only my name was on it at the time (thus defeating the need for the wheel) I was the lucky person to go through and change various aspects of each table.

I thought to myself, "Great, there goes a day or two down the grunt-work hole." Then the part of my brain that I put in charge of remembering important things woke up and slapped me. "You haven't altered the SQL yet, just alter your tool to do this too and re-translate, dumbass." After pondering the suggestion, and possibly more importantly the insult, leveled at me by my brain, I opened up my translation tool. A few keystrokes later the change to the translation mappings was made and I fired off the program. Bits flew, gears cranked, and out came the exact result I was looking for, the tool had just saved me another day or two.

Ultimately the project was a big hit, we delivered on-time and on-budget for what was a somewhat aggressive schedule. The tool saved us a good amount of time, let us get the application bootstrapped and running, which allowed us to start iterating and get things working. Over the course of that project I created tools here or there that ended up saving me, and some thankful members of my team, time and effort.

The lesson here is that you should look around you and see if you can make some tools. Tools can be scary though, they are a bit of an upfront cost, and if it doesn't work out then you've wasted even more time. With time and experience it becomes easier to recognize when it's a good time to make a tool and when it's a waste. Here are some good guidelines for tool making success.

• The best candidates are tasks with high amounts of repetition and little logic.


• Check to see if a similar tool already exists, don't reinvent the wheel.


• YAGNI (You Ain't Gonna Need It). You should be intimately familiar with the problem you are going to solve, solve exactly that problem, you can scale it out later. I didn't bother writing a general purpose PL/SQL to T-SQL translator, I only had one PL/SQL file to translate, if it worked on that then it worked on 100% of its input.


• If it would take less time to do the boring thing manually than to build a tool, just suck it up and do the boring manual task


• Share your tools, even if you don't think they will be useful to anyone else on your team, sometimes a tool can provide a surprising benefit to others


• KISS (Keep It Simple, Stupid). This is no time to become a highfalutin architect. Tools are normally quick and dirty one-offs, normal concerns like efficiency, user-friendliness, etc. can be brushed aside. If the tool is getting used often and by many people you can refactor it to make it more efficient or user-friendly, but don't pay that cost upfront

Developing software often requires a high degree of focus on the problem at hand, I wrote the other day about one approach to problem solving, simplifying the problem. This strategy can be coupled with toolmaking to dramatic effect. Once you realize the simpler actions you can automate those and write a tool to perform the task for you. The next time you are facing some mountainous problem, consider the toolmaking route.

experimentation

[caption id="attachment_523" align="alignright" width="300" caption="Like most computer related things this type of experimentation will have less blond women"][/caption]

I love my mom, she is a kind, intelligent, funny person. She has worked 2 jobs since I was just a little kid and she still does. For most of my childhood she worked in what we refer to in this country as the "Service Industry." This is a nice euphemism for mind-numbing tedium, the fact that she put in 14-16 hour days to give me the chance to sit here and blog and make good money consulting is a testament to her devotion as a mother. I remember growing up she worked at a Gas Station and at Arby's, sometimes bringing home some tasty chicken sandwiches or Apple Turnovers.

Now my mom works retail and in an office. She gets to work in a nice office and do office work which means that she now has to professionally use a computer. Being the "computer" son I get the calls when there is some task that falls outside the scope of her knowledge, and after all she's sacrificed for me, I'm ecstatic that I get to help her.

The things she needs help with our never very complicated, sometimes its an unfamiliarity with an FTP Client or looking for a way to open up and print multiple files quickly and easily. She normally has figured out or been told some way to get the job done, but it's so slow and tedious that there must be a better way. Normally after a few minutes of explanation we figure out a way that she can do her job faster and easier, and everyone wins.

In my job as a software developer I often have to think about the user, and its an interesting relationship. On the one hand if there were no users then I would make no money and that would be sad, on the other hand the users often end up taking a simple and elegant piece of code and twisting it into a horrible abomination so they can understand it better. The mental images of a slack-jaw fool and a well-meaning bug filer compete in my head. Now though I have a new user mental model, my mom.

I know that she is hardworking, intelligent, and does know how to use a computer for daily tasks. Yet, even with these great user attributes, she still doesn't discover things that I do, and I wonder why.

[caption id="attachment_525" align="alignleft" width="300" caption="Yea that\'s assembly not basic, go complain about it in the comments."][/caption]

Perhaps its a generational difference, I can't actually remember a time when I didn't have a computer. I remember distinctly getting the old green and black computer with 5.25in disks that I could play submarine hunting games on from my Uncle Marty. One of the first things I did with this new toy was to get a book from the library full of BASIC games and start typing them here, that's right I've been programming since I was about 6 years old. The thing that no one ever told me was that I could break the computer, to me it was a toy, and the worst thing that could happen is I mess up my disk and have to try again. As I moved up to more and more powerful machines I've always kept that same mindset, the computer is there to accept my commands, there is almost nothing I can do to it that can't be undone.

My mom on the other hand is pre-occupied with the idea that she might break the computer. This is a major difference between computer people and users. Watch a power user trying to figure something out, they will experiment see what happens when they click this or that, issue this command, or right-click in this box over here. A normal user on the other hand normally just uses the program in the way that they've been instructed to, instead of exploring the interface.

In a perfect world everyone would read the user manual, in real life though people don't. Even if people did, it's hard to learn reading, it's easy to learn doing. The power user learns through trial-and-error how something works, once they feel comfortable with the bulk of the application they may turn to the manual to figure out some of the trickier bits. The normal user will never get to a level of comfort to attempt to master the tricky bits. As the power user explores they feel validated and gain a sense of control, giving them a desire to assert their control over the rest of the system. The normal user is simply performing a rote task, the tool is not there to be explored and mastered, it merely performs some function.

Its the difference between becoming a magician and learning a card trick. I can learn a card trick and practice it everyday, becoming incredibly good at the card trick, this in no way makes me capable of sawing a woman in half (and having her live).

Trying to get users to experiment involves a few things

• Everything your application does should be easily and apparently reversible.


• Experimentation should be rewarded, never punished


• The Application must cope well with the unexpected, no nasty error messages


• Context Sensitive Help

[caption id="attachment_527" align="alignright" width="300" caption="TextMate is full of win"][/caption]

Case in point: I've recently got my sexy new iMac with the excellent TextMate for coding. It is one of the best experiences I have ever had, but it's not because it is easy. TextMate encourages you to explore and experiment, defining a function, def :tab: will automatically write the end and position you to write the methods. The Bundles menu is quick and accessible, the bundles make it easy to figure out what to do and the reward is instant. All of this culminates to making the software seem usable and easy, even though its very complex.

Turning users into explorers is no small task, but the payoff is great. The more a user feels in control, the more they explore, the more they explore the more in control they feel, its a positive feedback loop. The more in control a user feels the more they associate your project with being easy to use and generally a good piece of software.

simplify

[caption id="attachment_515" align="alignright" width="293" caption="simplicity is sometimes surprisingly complex"][/caption]

I love abstractions, most people do, they allow us to take complex things and wrap them in simple things and everyone is happy. The thing to realize is that there are degrees of complexity and different goals of abstractions.

Programming is one of those things where more often than not our abstractions are really helpful. The problem becomes one of perceived complexity vs. actual complexity. A good example is network protocols.

Network protocols are these complex beasts that let us talk across networks with some degree of certainty. Its really quite the marvel of engineering prowess, man has taken the various blinkenlights and cables and a big ole' pile of Cisco and now you can watch a video of a cat playing keyboard whenever you like. Network protocols were designed to simplify the process of two programs on physically distant machines sharing information.

When you first try to learn a complex abstraction it can seem daunting, UDP, TCP/IP, sockets and fifos and netcats oh my! Sometimes it can seem overwhelming, but take a step back and realize that no matter how complicated a piece of software is, its just moving around bytes.

Perl has a philosophy that at the end of the day, everything is strings. That's why its the glue that holds together the interwebs, because it's just a really good string manipulation language. At the end of the day that's all the internet is, strings floating around, normally between angled brackets.

Like most of the things I write I'm almost certain I had a point... oh there it is. When I started writing prosper it seemed like an impossible mountain to climb. I wanted to write a library that would bridge the gaps between all supported database backends, to have it be a sort of universal data access object.

[caption id="attachment_516" align="alignleft" width="354" caption="This was much easier to think about than the actual problem"][/caption]

I had a few false starts on my project until I took a step back, I simplified my problem domain, and created an architecture that worked. Instead of focusing on the end result of prosper I thought about it in a more abstract way, prosper was a mapper from function calls to string snippets. This isn't actually what prosper does today, prosper is in fact much more complex now than I would have ever imagined when I first started out.

The obstacle that stood in my way the most when starting out on prosper was being overwhelmed and unsure. The problem domain I was trying to tackle was too big, even when I tried solving it i ended up painting myself into a corner by focusing too much on how one specific backend acted. I needed a simpler problem that was more easily solvable, but one that would in the end help me reach my actual goal.

As much as I wish I could take credit for this idea, it's not mine, it's something called Minimum Viable Product. I wanted something that at its core would produce cross-platform SQL, so the first iteration was to make something that did exactly that. If you go on GitHub and pull one of the earlier commits you will see that that's exactly what I built. Over time I have added the ability to execute those strings, connection and transaction management, standardized results, prepared statement support, and much more.

I couldn't have accomplished any of this though if I hadn't taken a step back from the problem domain and looked at the simpler problem at the core of what I was trying to do. I stumbled in the beginning, I've refactored again and again, over the break my good friend Ian took a look at the library and found a whole slew of problems, and I thank him for it. Prosper isn't perfect yet, but it is moving toward being something really worthwhile, and to begin moving it required that I have something, anything, to start with.

So here is my suggestion for you for the new year. If you have some project that you've wanted to do but couldn't figure out how to start, step back from the problem domain and try to find the core of the problem and solve it with the simplest thing that could possibly work. Don't worry about the first iteration, or the first 20 iterations, you have to just get started. Once it is started you can iterate and refactor and maybe throw everything away and start again. No matter what the outcome of the first iteration it will be worth it, the best case scenario is that you can use the first iteration as a core moving forward, the worst case is that you learn a bunch about the problem domain and have to try again.

It's the start of a new year, resolve to start your dream project, because that's the hardest part. Once you get rolling momentum has a fantastic way of carrying you through.