who cares?

[caption id="attachment_573" align="alignright" width="300" caption="I shot who in the what now?"][/caption]

It's one of my favorite questions to ask during a technical discussion. Who cares? Some others I like to throw out our, why should I care? why does it matter? and I don't care! Am I some sort of raging lunatic completely out of my mind and trying to pick a fight with everyone I meet? Yes, but that's not why I ask these questions. I ask these questions because as great as the big picture is we sometimes have to drop complexities to get stuff done.

When you have a big flow diagram or giant UML Class Hierarchy it can be fun to think about the abstract way that things interact and how you can pass state around and whatnot. This definitely is useful for getting a feel for a new system, but then there comes the worst part of any software developer's job, actually getting things done. You see that's what we get paid for, actually putting our fingers on keys, typing stuff out, compiling it, and shipping a product. Part of this is doing the upfront design work and getting all of our t's crossed and i's dotted, but just as important is actually getting something working.

I talked about this before in my post about simplifying things. This simple question, "who cares?" (you can be more diplomatic about it if you like) is a great tool for getting to the heart of a problem. I like the simplicity and straight-forwardness of the question, it normally catches people off guard and challenges their assumptions.

Frank: So we need to pass this data into the layer
Bob: Who cares?
Frank (taken aback): I care, I care deeply about this data
Bob: But why do you care about it being in that layer, its just going to hand it off to this layer, and that layer doesn't care at all about it
Frank: I guess you are right, we don't need it there

BAM!!! Problem simplified! For anyone that's ever taken any education classes you may recognize this as a rather blunt application of Socratic Method. The point is confrontation, but not between the two people discussing something, between each person and their preconceived notions about a problem or situation.

In High School I took a chemistry class, it is one of the few classes that I really remember in any detail from so long ago, the teacher taught the entire semester in Socratic Method. He would question us, we would experiment, we would reason out why we observed what we observed, he prodded us, and we cherished our knowledge. Anyone can tell you water is composed of 2 hydrogen atoms and 1 oxygen atom, but when you had to work for it, when you had to defend your knowledge of it from scrutiny, it became concrete, it became your discovery.

Applying this to software development is useful, we can get locked into our preconceived notions, and direct, blunt, sometimes even rude questions can spur you to really consider what you are talking about. Software is incredibly fluid, sometimes an idea or concept can become out-dated in a matter of months or a matter of hours. The biggest ones are the most difficult to challenge, because they often have the most dependent code around them, but this makes them the most important to constantly consider.

Keeping your conceptual design, implementation, and solution in lock step with a problem domain often involves some amount of change, that change can be hard to come by though. By challenging our standing beliefs we can find new avenues for exploration or strengthen the existing techniques and gain a better understanding of them.

grok

[caption id="attachment_565" align="alignright" width="282" caption="necessary part of grokking something"][/caption]

Computer nerds should be familiar with the term grok while the rest of the populace has no need for such a silly word. You may hear it down at your favorite nerdery, "I don't quite grok this" or "Once I grokked it, I flew through the project." What does this word mean though and where did it come from.

Grok for the non-nerd means, "to understand thoroughly and intuitively." Although this is true it belies what it truly means to grok something. For that we need to realize the origin of this fantastic geek speak, this term is straight out of Robert A. Heinlein's Stranger in a Strange Land. He defined it as follows

Grok means to understand so thoroughly that the observer becomes a part of the observed—to merge, blend, intermarry, lose identity in group experience. It means almost everything that we mean by religion, philosophy, and science—and it means as little to us (because of our Earthly assumptions) as color means to a blind man.

The great thing about grokking something is that it changes your world view, it fundamentally shifts the way that you see, perceive, and understand the world. The bad thing about grokking something is that it is incredibly difficult to get to that point. Let's take a look at a fantastic chart I've whipped up to help us understand this better.

[caption id="attachment_566" align="alignleft" width="300" caption="the graph of grokitude"][/caption]

In this graph we have time and effort along the horizontal and understanding along the vertical. We can spend a great deal of time after "mastering" a subject before we get the epiphany and truly grok something. To really understand a subject on the level that a computer nerd would consider grokking it takes not only a full in-depth knowledge of the function of something, but an understanding of the design, an internalization of the principals, and a familiarity that borders on oneness.

This is the reward for hundreds or thousands of man-hours spent toiling in something, the day when the clouds part and you truly understand something, this has an amazing non-linear effect on your understanding and your productivity. Once you grok something, be it a framework or a library or a concept, your ability to use that and your productivity in it increase in a non-linear fashion.

Although it is important to have a wide breadth of knowledge we must remember the importance of having a few areas of incredible depths of knowledge. These areas will permeate everything else you do, so try to choose wisely. The subjects that you truly grok are the ones that will effect your big picture outlook.

Here are my suggestions for some fundamental things to learn, and learn to a point that it fundamentally shifts the way you look at the world.

• A Pure Object Oriented Langauge - I would suggest smalltalk for purity, but ruby makes a fine substitute and has more application.


• A Functional Language - For this look no further than Haskell, and Learn you a Haskell for Great Good


• A Set Language - Take the time to learn SQL, its a tool most of us use everyday and very few of us take the time to learn properly


• A Markup Language - HTML + CSS, these are the embodiment of separating data and display, learn them intimately.

Learning these, but more importantly grokking them will color the way you approach problems and solutions. Having a depth of knowledge in a key variety of tools allows you to quickly master new ones, better understand problem domains, and more completely express your thoughts in the abstract.

limits

[caption id="attachment_558" align="alignright" width="300" caption="its beauty is only matched by its cliffy-ness... or something"][/caption]

I had to make a hard decision today, it was one that I had known I would have to make at some point, and frankly I was just putting it off. I've written about burnout before and I'm no stranger to pushing myself harder than I should. But I like to think that because I'm aware of the danger and am actively working to avoid burnout that I can push myself, and still maintain some sanity.

Part of maintaining sanity is knowing one's limits. I have a full-time job, prosper, and until today 2 side projects. Today I resigned from one of my side projects and I released any code or concepts I had worked on to the other developer. It was not something that I particularly looked forward to doing, but I had to prioritize. I had to dispassionately measure the various aspects of my life and make a cut before I burned out and lost everything.

Now I'm left with a strange feeling, almost ashamed that I let down the other party, but at the same time a sense of relief has come over me. After I made sure that I didn't burn down any professional bridges, another wave of relief. Looking forward in my calendar and seeing all the time I've got to work on Prosper and my other side project, a wave of excitement. The problem was that I was spread too thin, at my limit, I was pushing ahead 110% but not moving anything forward. Now though I feel back in control, pushing ahead and making things happen again.

I've given this new attitude a shot in the arm over the last few days. For the last few days I've been knocking out tasks that have been long outstanding and non-trivial. It's easy to trick yourself into thinking you are being productive by banging out a few quick trivial tasks and checking some stuff off the todo list, but picking long standing non-trivial tasks is important.

My girlfriend's father's computer was the first such task. It's hard drive died, and I've had it for far too long. What should have been a simple diagnostic-swap-reimage was bedeviled by problems. I didn't have the right keyboard to perform the diagnostic (all my usb keyboards were read by the bios but not the ubcd prompt), then the replacement drive I was provided was the wrong type, and the recovery disks were all oddly labeled. The task itself though is not that difficult, get the right drive, swap, get the right disk, click click type, wait 4 hours for reimaging. The giant tower sitting there day after day had become a constant and depressing embodiment of my inability to move tasks forward. Yesterday though I finally hooked it all up, found the right disk (on the third try) and reimaged it. Everything works fine now and the problem is solved.

For the presentation I'm giving at CodeMash I plan on doing a live demo, I needed to pin this completely down to the last detail and do all the boring set up work so the presentation can be all whiz-bang cool. After an hour or two hacking around this is now done. My minimal PowerPoint presentation is well on its way to being complete. Getting these non-trivial roadblocks out of my way is freeing up my mind to think of other things, to begin figuring out how to move other things forward.

My suggestion for anyone feeling overwhelmed is twofold. Be realistic, are you taking on too much, I was, it happens to the best of us. It's never a great feeling to admit that you've overextended yourself, but it's much better than just constantly being overextended. Are there tasks that are constantly at the back of your mind, nagging todo's that you keep putting off, well just tackle them. It's not fun, but when you put them off they start growing larger and larger in your mind's eye. The live demo only took around an hour to code through, even though in my mind it was important and therefore a monumental task.

Pick out one of those tasks that's been on your todo list for a while, knock it out, just having it done is worth the work.

codemash

[caption id="attachment_553" align="alignright" width="300" caption="I want you to come to Matt\'s CodeMash Session"][/caption]

Disclaimer: This post is shameless, and I mean really, really shameless, self-promotion.

If you have ever been on this site before you may recognize the name prosper. Prosper is my attempt at creating something of value for the PHP community. For anyone is following along at GitHub, you may have noticed that there hasn't been much work done in the last few weeks, what's the deal?! Well, don't fret, prosper is still under active development but it has been frozen for a few reasons.

1. It was the holidays, and I was doing this weird thing where you interact with humans


2. I've been seeking, and receiving, feedback from some key users


3. I will be giving a live demo of Prosper and have been working on that demo more than the library as of late

"A Live Demo?" you say bedeviled by the notion. Don't worry you don't all need to cram into my apartment to see some technological wizardry, all you need to do is head on over to CodeMash, then at 3:00 on Thursday head to room E. The awesome company that I work for, HMB is sponsoring CodeMash and have selected me to give a presentation on this crazy open source project they've heard I wrote.

I've been working on a very minimal PowerPoint, because I know people can read so there is no point in me reading to them. At the end of the presentation though will be a live demo building up a simple database backed PHP application for managing all the fond memories of CodeMash, a quote board. In case you are unaware, building an application in front of people is incredibly nerve-wracking, which is why for the next 5 days I will be building this application over and over while talking to absolutely no one, to the confusion of my poor neighbors.

Live demos have a fantastic way of just going all wrong, as Bill will demonstrate

I'm not trying to poke fun at Mr. Gates (for once), giving a presentation in front of people, with new software is never a fun experience, and bad things© do happen. I will be practicing and practicing and practicing, and after that I will probably be practicing, for this presentation to make it the best it can be.

In other prosper news, I have set up http://prosper-lib.com to be the new home of prosper (currently just a forward to here, although the update is still working its way through GoDaddy's intertubes so you may get a parking page for a while). I am working with my good friend Ashley Kruso to get a new logo for prosper. Once I have the new logo I will begin building a new web home for prosper that will be all kinds of pretty. This is all in the run-up to Prosper 1.0, which is a goal for this year, but I haven't quite nailed down when.

It's all steam ahead on Prosper, thanks to the contributions and support of many people. When I started this project a few months ago I would not have been able to guess what it would turn into, so thanks to everyone that has helped or is going to help. By month's end there should be a 0.8 release that will have all kinds of new fun stuff including deferred result sets and lots of bug fixes from my friend who was kind enough to evaluate Prosper, Ian.

toolmaker

[caption id="attachment_535" align="alignright" width="262" caption="Pictured Above: Programming"][/caption]

For about a year I had the unique experience of working at Abercrombie & Fitch's world headquarters in New Albany, Ohio as a Java Developer. It was a really interesting place to work, very stylish, loud music all the time, complex problems of incredible scale, and some of the best developers I've ever worked with. I was lucky enough to get placed into one of the most successful development teams, Planning. This team had developed a tool that was hugely popular and well regarded, and I was coming on board to maintain and improve this tool. The beautiful people and board-short dress code was just icing on the cake.

As a software developer this is a fairly basic scenario, there is some task that is complicated, business users want it to be less complicated, they give me some money, I make them some program, and everyone is happy. The odd thing though is that while programmers are completely used to making tools for others, they often forget that they can leverage this amazing skill for their own benefit. I distinctly remember a conversation I had with one of my colleagues there, Steve. It centered around the idea that most developers are happy to just get their assigned tasks done, but there are some that are toolmakers, and being a toolmaker is more valuable to a company than being a developer.

Fast forward to after my stint at A&F, I was on a new project and given the task of translating 80,000 lines of PL/SQL into T-SQL. The idea was to use...um...find and replace...or....something...and...um...get this to work. With such a fine technical specification and a mountain of PL/SQL staring at me, Steve's words bubbled up to my brain, "Make a tool." I had a time window of a some number of weeks to do the translation, it was a bit of a gamble, but I decided to spend a week writing a tool.

After remembering how C# worked after a year in the trenches with Java I had a program that took in the PL/SQL parsed each line and translated it to T-SQL. The task was made easier by the fact that I didn't have to write a generalized parser for PL/SQL as the 80,000 lines were written in a fairly consistent style, which greatly simpified the translation tool. After running the tool a few times, fixing some bugs, I had finished the task in a fraction of the projected time. Triumphantly, I went to my PM to show how I had just saved weeks of manual laborious work. "Great, now here is your next task...."

[caption id="attachment_538" align="alignnone" width="150" caption="Expected Reaction"][/caption]

[caption id="attachment_539" align="alignnone" width="150" caption="Actual Reaction"][/caption]

So I didn't get the praise I was expecting, that's fine, I'm not 4, I don't need a cookie. I continued on in my tasks quietly pleased with saving myself weeks of tedious work. Then the day came when the database requirements changed, the hundred or so CREATE TABLE statements would need to be altered. The task wheel was spun, and since only my name was on it at the time (thus defeating the need for the wheel) I was the lucky person to go through and change various aspects of each table.

I thought to myself, "Great, there goes a day or two down the grunt-work hole." Then the part of my brain that I put in charge of remembering important things woke up and slapped me. "You haven't altered the SQL yet, just alter your tool to do this too and re-translate, dumbass." After pondering the suggestion, and possibly more importantly the insult, leveled at me by my brain, I opened up my translation tool. A few keystrokes later the change to the translation mappings was made and I fired off the program. Bits flew, gears cranked, and out came the exact result I was looking for, the tool had just saved me another day or two.

Ultimately the project was a big hit, we delivered on-time and on-budget for what was a somewhat aggressive schedule. The tool saved us a good amount of time, let us get the application bootstrapped and running, which allowed us to start iterating and get things working. Over the course of that project I created tools here or there that ended up saving me, and some thankful members of my team, time and effort.

The lesson here is that you should look around you and see if you can make some tools. Tools can be scary though, they are a bit of an upfront cost, and if it doesn't work out then you've wasted even more time. With time and experience it becomes easier to recognize when it's a good time to make a tool and when it's a waste. Here are some good guidelines for tool making success.

• The best candidates are tasks with high amounts of repetition and little logic.


• Check to see if a similar tool already exists, don't reinvent the wheel.


• YAGNI (You Ain't Gonna Need It). You should be intimately familiar with the problem you are going to solve, solve exactly that problem, you can scale it out later. I didn't bother writing a general purpose PL/SQL to T-SQL translator, I only had one PL/SQL file to translate, if it worked on that then it worked on 100% of its input.


• If it would take less time to do the boring thing manually than to build a tool, just suck it up and do the boring manual task


• Share your tools, even if you don't think they will be useful to anyone else on your team, sometimes a tool can provide a surprising benefit to others


• KISS (Keep It Simple, Stupid). This is no time to become a highfalutin architect. Tools are normally quick and dirty one-offs, normal concerns like efficiency, user-friendliness, etc. can be brushed aside. If the tool is getting used often and by many people you can refactor it to make it more efficient or user-friendly, but don't pay that cost upfront

Developing software often requires a high degree of focus on the problem at hand, I wrote the other day about one approach to problem solving, simplifying the problem. This strategy can be coupled with toolmaking to dramatic effect. Once you realize the simpler actions you can automate those and write a tool to perform the task for you. The next time you are facing some mountainous problem, consider the toolmaking route.

experimentation

[caption id="attachment_523" align="alignright" width="300" caption="Like most computer related things this type of experimentation will have less blond women"][/caption]

I love my mom, she is a kind, intelligent, funny person. She has worked 2 jobs since I was just a little kid and she still does. For most of my childhood she worked in what we refer to in this country as the "Service Industry." This is a nice euphemism for mind-numbing tedium, the fact that she put in 14-16 hour days to give me the chance to sit here and blog and make good money consulting is a testament to her devotion as a mother. I remember growing up she worked at a Gas Station and at Arby's, sometimes bringing home some tasty chicken sandwiches or Apple Turnovers.

Now my mom works retail and in an office. She gets to work in a nice office and do office work which means that she now has to professionally use a computer. Being the "computer" son I get the calls when there is some task that falls outside the scope of her knowledge, and after all she's sacrificed for me, I'm ecstatic that I get to help her.

The things she needs help with our never very complicated, sometimes its an unfamiliarity with an FTP Client or looking for a way to open up and print multiple files quickly and easily. She normally has figured out or been told some way to get the job done, but it's so slow and tedious that there must be a better way. Normally after a few minutes of explanation we figure out a way that she can do her job faster and easier, and everyone wins.

In my job as a software developer I often have to think about the user, and its an interesting relationship. On the one hand if there were no users then I would make no money and that would be sad, on the other hand the users often end up taking a simple and elegant piece of code and twisting it into a horrible abomination so they can understand it better. The mental images of a slack-jaw fool and a well-meaning bug filer compete in my head. Now though I have a new user mental model, my mom.

I know that she is hardworking, intelligent, and does know how to use a computer for daily tasks. Yet, even with these great user attributes, she still doesn't discover things that I do, and I wonder why.

[caption id="attachment_525" align="alignleft" width="300" caption="Yea that\'s assembly not basic, go complain about it in the comments."][/caption]

Perhaps its a generational difference, I can't actually remember a time when I didn't have a computer. I remember distinctly getting the old green and black computer with 5.25in disks that I could play submarine hunting games on from my Uncle Marty. One of the first things I did with this new toy was to get a book from the library full of BASIC games and start typing them here, that's right I've been programming since I was about 6 years old. The thing that no one ever told me was that I could break the computer, to me it was a toy, and the worst thing that could happen is I mess up my disk and have to try again. As I moved up to more and more powerful machines I've always kept that same mindset, the computer is there to accept my commands, there is almost nothing I can do to it that can't be undone.

My mom on the other hand is pre-occupied with the idea that she might break the computer. This is a major difference between computer people and users. Watch a power user trying to figure something out, they will experiment see what happens when they click this or that, issue this command, or right-click in this box over here. A normal user on the other hand normally just uses the program in the way that they've been instructed to, instead of exploring the interface.

In a perfect world everyone would read the user manual, in real life though people don't. Even if people did, it's hard to learn reading, it's easy to learn doing. The power user learns through trial-and-error how something works, once they feel comfortable with the bulk of the application they may turn to the manual to figure out some of the trickier bits. The normal user will never get to a level of comfort to attempt to master the tricky bits. As the power user explores they feel validated and gain a sense of control, giving them a desire to assert their control over the rest of the system. The normal user is simply performing a rote task, the tool is not there to be explored and mastered, it merely performs some function.

Its the difference between becoming a magician and learning a card trick. I can learn a card trick and practice it everyday, becoming incredibly good at the card trick, this in no way makes me capable of sawing a woman in half (and having her live).

Trying to get users to experiment involves a few things

• Everything your application does should be easily and apparently reversible.


• Experimentation should be rewarded, never punished


• The Application must cope well with the unexpected, no nasty error messages


• Context Sensitive Help

[caption id="attachment_527" align="alignright" width="300" caption="TextMate is full of win"][/caption]

Case in point: I've recently got my sexy new iMac with the excellent TextMate for coding. It is one of the best experiences I have ever had, but it's not because it is easy. TextMate encourages you to explore and experiment, defining a function, def :tab: will automatically write the end and position you to write the methods. The Bundles menu is quick and accessible, the bundles make it easy to figure out what to do and the reward is instant. All of this culminates to making the software seem usable and easy, even though its very complex.

Turning users into explorers is no small task, but the payoff is great. The more a user feels in control, the more they explore, the more they explore the more in control they feel, its a positive feedback loop. The more in control a user feels the more they associate your project with being easy to use and generally a good piece of software.

simplify

[caption id="attachment_515" align="alignright" width="293" caption="simplicity is sometimes surprisingly complex"][/caption]

I love abstractions, most people do, they allow us to take complex things and wrap them in simple things and everyone is happy. The thing to realize is that there are degrees of complexity and different goals of abstractions.

Programming is one of those things where more often than not our abstractions are really helpful. The problem becomes one of perceived complexity vs. actual complexity. A good example is network protocols.

Network protocols are these complex beasts that let us talk across networks with some degree of certainty. Its really quite the marvel of engineering prowess, man has taken the various blinkenlights and cables and a big ole' pile of Cisco and now you can watch a video of a cat playing keyboard whenever you like. Network protocols were designed to simplify the process of two programs on physically distant machines sharing information.

When you first try to learn a complex abstraction it can seem daunting, UDP, TCP/IP, sockets and fifos and netcats oh my! Sometimes it can seem overwhelming, but take a step back and realize that no matter how complicated a piece of software is, its just moving around bytes.

Perl has a philosophy that at the end of the day, everything is strings. That's why its the glue that holds together the interwebs, because it's just a really good string manipulation language. At the end of the day that's all the internet is, strings floating around, normally between angled brackets.

Like most of the things I write I'm almost certain I had a point... oh there it is. When I started writing prosper it seemed like an impossible mountain to climb. I wanted to write a library that would bridge the gaps between all supported database backends, to have it be a sort of universal data access object.

[caption id="attachment_516" align="alignleft" width="354" caption="This was much easier to think about than the actual problem"][/caption]

I had a few false starts on my project until I took a step back, I simplified my problem domain, and created an architecture that worked. Instead of focusing on the end result of prosper I thought about it in a more abstract way, prosper was a mapper from function calls to string snippets. This isn't actually what prosper does today, prosper is in fact much more complex now than I would have ever imagined when I first started out.

The obstacle that stood in my way the most when starting out on prosper was being overwhelmed and unsure. The problem domain I was trying to tackle was too big, even when I tried solving it i ended up painting myself into a corner by focusing too much on how one specific backend acted. I needed a simpler problem that was more easily solvable, but one that would in the end help me reach my actual goal.

As much as I wish I could take credit for this idea, it's not mine, it's something called Minimum Viable Product. I wanted something that at its core would produce cross-platform SQL, so the first iteration was to make something that did exactly that. If you go on GitHub and pull one of the earlier commits you will see that that's exactly what I built. Over time I have added the ability to execute those strings, connection and transaction management, standardized results, prepared statement support, and much more.

I couldn't have accomplished any of this though if I hadn't taken a step back from the problem domain and looked at the simpler problem at the core of what I was trying to do. I stumbled in the beginning, I've refactored again and again, over the break my good friend Ian took a look at the library and found a whole slew of problems, and I thank him for it. Prosper isn't perfect yet, but it is moving toward being something really worthwhile, and to begin moving it required that I have something, anything, to start with.

So here is my suggestion for you for the new year. If you have some project that you've wanted to do but couldn't figure out how to start, step back from the problem domain and try to find the core of the problem and solve it with the simplest thing that could possibly work. Don't worry about the first iteration, or the first 20 iterations, you have to just get started. Once it is started you can iterate and refactor and maybe throw everything away and start again. No matter what the outcome of the first iteration it will be worth it, the best case scenario is that you can use the first iteration as a core moving forward, the worst case is that you learn a bunch about the problem domain and have to try again.

It's the start of a new year, resolve to start your dream project, because that's the hardest part. Once you get rolling momentum has a fantastic way of carrying you through.