time

[caption id="attachment_708" align="alignright" width="300" caption="Finally was able to get the clock to show me its good side, dirty girl"][/caption]

Don't worry this won't be a rant about mortality or getting things done or any of the philosophy that has been dominating this blog as of late. This is back to basics, a discussion about software and a particularly tricky aspect of it, time. Not time as in, scheduling and managing time, but something far more fundamental representing time. It is an insidiously tricky problem and one that can be quite difficult to wrap your head around.

The problem comes from how we think about time as people living normal lives. "I'll meet you at 3 o'clock" is a rather dull and completely normal type of phrase to say to someone. As two normal people living normal lives, this simple phrase "3 o'clock" is plenty to convey when they should meet. This is because there exists a great deal of unspoken context between the two parties, if we are meeting for a business meeting I clearly mean 3:00pm not 3:00am. If we both work in the same building I probably mean relative to the our shared timezone, 3:00pm EST not 3:00pm GMT. There is a world of shared unspoken context that makes human-human time discussions easy and natural.

Computers are really stupid though, they need everything spelled out. If you were trying to store time you might take a naive approach at first and just store the string "3:00" maybe if you are really thinking it out you would store "3:00pm EST." This method soon starts showing its weaknesses as its hard to compare times, or perform operations on them. How many hours are between 2:00am EST and 5:30pm CST? There is a nasty problem to try to solve unless you have some sort of way to represent times in the abstract.

In steps a number of formats to represent time. There is the venerable Unix Timestamp which is the number of seconds from Jan. 1, 1970 as of my current writing it stands at 1265738039, but feel free to check for yourself. Then there are numerous proprietary formats like Microsoft's, Oracle's, etc. These all allow you to represent an exact moment of time in a portable abstract way with no dependence on the cavalcade of context us fleshy humans share.

Well problem solved, just bust out your favorite abstract representation and you are done. Not so fast, there are many other considerations to take into account when dealing with time. There are of course the tricky problems of Daylight's Saving Time, leap years, and the like. Imagine you are trying to add an event to a calendar system everyday at 5:00pm EST, you think you could just add it to today and then just add 24 hours and create a new event. DST hits your algorithm over the head at some point and everything is off an hour, oh no! Also now you have a ton of data to represent one basic fact, something happens everyday at 5:00pm EST. Its only one fact, you should need one record, not an infinite number of 5:00pm EST records. This hints at the next difficulty of time.

Humans think about repeatable events (sometimes with complex and insane rules) as commonplace and easy. This thing happens on the third Thursday of every month, unless of course Monday was a holiday and then it gets shifted to Friday. The problem with time and dates and repeating events is that human beings erected a ton of complex processing rules before they realized we were going to try and digitize them. These are difficult to represent and difficult to get right.

At first the task of representing arbitrary points and spans of time seems fairly straightforward, but it is a complex and nuanced task, like most things the devil is in the details. Before you go off half-cocked building up your own representation, take a look at some established formats, like Unix Timestamps, RFC5545, and ISO 8601.

experience as a force multiplier

[caption id="attachment_701" align="alignleft" width="300" caption="Oh DS cat you lovable scamp, don\'t use up all my healing potions"][/caption]

Over the weekend I experienced two things, one rather normal for a "computer guy" the other less in my wheelhouse. The first thing was that my girlfriend wanted me to help her register a web domain for her dad for his birthday. I have done this twice before, once for the website you are currently viewing and one other time for http://prosper-lib.com (which just redirects back here for the time being). I remember the first time I registered a domain and carefully read all the forms, making measured decisions, and hesitantly moved from step to step unsure of myself. It took an hour or two and then a few more to set up the hosting and get ftp access, set up wordpress and get everything up and running. In total it took a few hours. I didn't have such a luxury of time, she needed me to get the site up and running, preferably with a custom "Coming Soon" page before lunch, it was 11:00 so I only had around an hour. I clicked over to the site, click click click type type type, and it was up and running. I looked over at the clock to see how badly I had overshot the envelope, 11:15... wait what?

The second experience I had this weekend was tearing up some carpet at my new place. I have undertaken tearing up 850 sq ft of carpet to install new wood flooring in my house. The first night I was able to remove carpet from two rooms in just under 4 hours. It was my first time, and I went slowly following the instructions and making sure that I did everything just right. The second night I worked with an experienced floor guy, we finished taking out the mats, staples, tack strips and carpets from 5 rooms. Over the weekend there was one room left to do, I was able to knock it out, carpet, mats, tack strips, the whole nine yards in about an hour.

After these two events it solidified a point in my head, experience is the best force multiplier. In the first case it was my own hard won experience (not that there is really anything that difficult about registering a website). After performing the task a few times I gained an experience that multiplied my force by an order of magnitude, what once took a few hours now only took a few minutes. In the second case I was able to leverage someone else's experience to augment my own. I learned countless tricks to getting the job done quickly and efficiently, but more importantly I gained a huge amount of confidence. Having someone that already had mastered the task work with me and confirm that I knew what I was doing gave me all the confidence in the world to go in on Saturday and tear up the last room lickety split.

There were two take away points from these experiences. When doing something new seek out someone more experienced than you to pair with, even if its just temporary. Fully participate with them, ask questions, and do not let them do everything for you. This is the best time to fail, you will get immediate feedback from the more experienced party about how you are failing, why you are failing, and how to avoid failure next time. This will give you a foothold to quickly learn by doing. The second point to take away is to experience lots of things, even if you suspect you will fail. Life is full of failures, but those failures impart experience which is the best force multiplier to get to success.

I would be unable to count the number of times I've ruled out some great idea because it took me down some unknown avenue, and I was afraid of failing. I have no experience charging people money, I don't know how to start up a business, I don't know how to make a website. These are all things I've said to myself at one point or another, the thing I realize now is that no one is born knowing how to do these things, there is only one way to get experience, try new things. Don't be afraid to fail, work hard to succeed, and know that no matter what happens the experience is valuable and worth it.

persist

[caption id="attachment_696" align="alignright" width="300" caption="measure twice, turn once"][/caption]

I just witnessed an interesting even unfold in front of my current client's building. Today in beautiful Columbus, Ohio we are experiencing one hell of a winter storm, lots of fun. A semi truck decided that our parking lot was a great place to turn around in, he just apparently forgot to measure. CRASH BOOM and the lamppost has changed from its normal 90° to a more precarious 78ish°. After striking the lamppost and ramming a concrete barrier the semi was fairly well wedged, but the driver continued revving and the engine struggled and moaned. This persistence paid off and after about 10 minutes of maneuvering he was free. Now to call the insurance companies.

This got me thinking about the nature of difficult situations (self-imposed or otherwise). I also had a breakthrough largely due to persistence today, so happy coincidence. After months of working on documentation, being told that I would never be allowed to program the sacred system, the tides have turned (thanks to an excessively high quote) and the coach has called me up, and I'm ready to play under the bright lights.

I've just completed printing out 400 pages of documentation (my weekend is going to be so much fun!) and I will prepare myself for the difficult task ahead. My persistence has paid off, but the bigger realization is that persistence in general pays off.

To borrow one of my favorite quotes from Scrubs, "People are bastard-coated bastards with bastard filling." People don't ever want to think that you can succeed, and they are going to tell you no over and over again. There are two reasons for people to shut down your dreams:

1. They have tried and failed, the task is impossible, your success would undermine them. If you succeed where they have failed then it isn't an impossible task, they just weren't good enough to do it.


2. The have achieved it, but only by their virtue of their amazing talents and pure force of will. You don't have what it takes!

This is why it's vitally important to ask people for advice, sage wisdom, realistic assessments, but always keep in mind that very few people actually want you to succeed, especially if its in something they can't or already have. Don't worry though, these aren't bad people, it's human nature, and when you succeed you will act the same way. Success through hardship and against the odds plays a fun trick on the human brain, it makes you believe you pulled off the impossible. This feels amazing and so you will do anything in to hold on to it, and you don't want some dirty unwashed masses pulling off the same feat and diminishing it.

The big lesson here though is persistence, the most successful people in the world are those that were willing to have the door slammed in their faces 1000 times before making that sale. This is what you need to do, steel yourself for the struggle, prepare to be emotionally drained, and realize that this is what living is. Life is a stream of people telling you no, with the occasional yes. What separates the successful from the mediocre is that the successful are willing to wade through all the noes to get to the sweet succulent yes.

I've been paying my dues on this project for 3 months, and finally I get to go back to what I love the most, coding. There are a hundred stories of people trying and failing and trying and failing and trying and failing to finally succeed. It is definitely the path less taken, but it will make all the difference.

finite

[caption id="attachment_689" align="alignright" width="300" caption="all that eggnog caught up to him... the cocaine addiction didn\'t help either."][/caption]

Here's something unpleasant to think about, you are finite. Think about it deeply, drink it in, sit with it, feel like you've come to understand it, then wake up in a cold sweat thinking about it some more. There is only one certainty in this life, it will end. It gets worse, your time, your effort, your experiences, they are all finite. It's one of those big huge things in life that is too inconvenient to think about, so we don't. This is the core reason why most religions exist, coming to terms with one's own limited nature is terrifying. I've always been an Atheist, intimately connected to this notion that one day the unique thing that is me will be no more, just as for billions of years before I popped into existence I wasn't yet. It's something I tell myself I have come to terms with, but it is a daunting challenge and one that must be revisited from time to time.

As an aside: although I'm more than happy to discuss my personal spiritual beliefs I attempt to keep such charged matters off of this blog, if you would like to chat about it just post a short comment indicating so and I will contact you. I am an Atheist but I respect others' belief systems and don't intend to offend anyone.

I was driving home the other day and an interesting story came up on the radio. The main point being that as we age we sense that time is moving faster and faster. This is the spark that has led me to revisit one of the least comfortable aspects of our shared humanity. I've been pondering it for the last few days, thinking about my very real finiteness and trying to evaluate if I'm acting like a man with an expiration date. It was a mixed evaluation, in some ways I am, in others I am not, not a terrible fate by any stretch.

In my personal life I feel like I'm running full steam ahead. It might be the last week or so of 14 hour days working on getting my new house ready to move into. (After years of snacking and typing my body isn't as suited to the tasks of tearing up carpets and fixing walls as it once was). I've set real goals in terms of building skills, building relationships, and creating code I can be proud of, and I'm exceeding my expectations. I finally feel like my wheels have caught solid ground and aren't just spinning anymore.

There are still some areas I'm not happy with, I'm taking stock and taking action to address these. On the whole though, I feel like I'm spending my limited time here the right way for now. There are a thousand and one clichés about this subject: live each day like it could be your last, what would you do if you knew you would die tomorrow, etc. The problem is that they all take an outlook that is far too limited. No one could live each day like it could be your last, that's fucking stupid. If I knew I was going to die I'd spend all my money and live with complete abandon, you can't do that everyday of your life. It is the routine of life that makes this so difficult.

Life is finite but there is also a whole big pile of it. We like to think that we can wait and do something next week, month, year, etc. This is where all that "I always thought we'd have more time together" thing comes from. Want to open a bakery, learn to play the drums, fight a bear, there's no time like the present. If you are unhappy, you have to fix that. Other people that aren't you are probably going to tell you that you are foolish for trying to fight a bear, well those people can waste their lives while you're out punching Yogi in the mouth.

Don't let anything stand in your way, take it personally because nothing could be more personal. This is your precious finite life and its ticking away. If you are unhappy or feel like it's being wasted, do something about it. Nothing could be more important. It's not comfortable, and it can be frightening, but take stock of your life today and the whole time keep in mind, I only have so many days left, is this how I want to spend them.

bureaucracy

[caption id="attachment_677" align="alignright" width="300" caption="Wait until he meets the Undersecretary for Reduction Planning and Appropriation\'s new Assistant"][/caption]

Bureaucracy, long the scourge of people who want to git-r-dun. There are some people out there that hate bureaucracy with a passion I can only really muster up for the pending zombie apocalypse. Make no mistake about it, I dislike bureaucracy, I think it is a wasteful but sometimes necessary evil. I've found myself ensnared in a bureaucratic nightmare for the last few months so I thought I would jot down some observations on bureaucracy.

Zombie Bureaucracy

(Wow lots of zombies in this post so far) A Zombie Bureaucracy is a bureaucracy that had some reason (however flimsy) for existing in the past but no longer needs to exist. Not unlike a zombie this bureaucracy manages to live long past its usefulness shambling into the future pointless and frustrating. This is caused by the ease that more bureaucracy can be created, at the stroke of an email someone can dream up a committee or process, but the difficulty in disbanding bureaucracy. It is disproportionately difficult to reduce bureaucracy because it looks like work, and people don't like having their work taken away, it reduces their sense of job security.

Dev: Why do I need 3 developers to sign off on every commit?
Mgr: Because 4 years ago two of the lead developers got into a pissing war over code formatting
Dev: Why didn't they just work it out?
Mgr: Because they had huge egos and management was too scared to fire either one
Dev: Is this still going on?
Mgr: No, Frank left 3.5 years ago because he didn't want to put up with Mark anymore
Dev: So why do I still need 3 developers to sign off on every commit?
Mgr: Because 4 years ago.... I guess it doesn't make much sense anymore.... That's the way it is!

Just like misery, bureaucracy love company

Bureaucracy is an attempt to control something, to take organic chaotic processes and make them orderly. The problem that often arises is that bureaucracy is its own organic chaotic system, that then requires more bureaucracy ad infinitum. If you've ever been in a meeting where all you decided was the schedule of meetings, congratulations, you are in the Matryoshka doll hell of bureaucracy.

Meetings

I loathe meetings, a bunch of people talking about the work they could be doing if they weren't in meetings all the damn time. Meetings are seductive, they sound and feel and look like work, but they are occupational masturbation. No one has ever brought a product to market because they were able to make it to 10 meetings a day. Meetings have almost no value, sometimes they are necessary, but not nearly as common as corporate culture would have you believe.

And the rest...

[caption id="attachment_683" align="alignleft" width="289" caption="Forgot to file the A34C-Bh amendment releasing liability for tongue trauma"][/caption]

I could go on and on, but I will cut the rant short and get to the point. Bureaucracy at its core is about trust, or more importantly the lack thereof. I don't have my girlfriend fill out the A34C-B form (Confirmation of Confection Purchase Agreement) before running to the store to ensure that she understands that I would like her to pick me up some Hot Tamales. I trust her at her word, there is no need for such ridiculous formalities.

When an organization has enshrined itself in a monument of bureaucracy what it is really saying is, "We've been burned before and now we don't trust you." We don't trust you to deploy your code correctly, we don't trust you to design things properly, we don't trust you to do X, so let's get a bunch of people together to review it. This is fine in small doses, frankly I want people to review my code and my designs, I make mistakes like everyone else. But when taken to the extreme it takes a toll on your motivation, on your creativity, and on any preconceptions that you knew what you were doing. Bureaucracy is the surest way to crush your workforce into a homogeneous mix, you will catch the awful at the expense of the great.

stupid ideas

[caption id="attachment_671" align="alignleft" width="263" caption="No caption needed"][/caption]

I want to be honest with you for a minute, I have lots of stupid ideas. I have them all day, in the shower, while eating a sandwich, while writing my blog posts, stupid ideas come streaming into my head. I used to just ignore them, "No one needs a peanut powered lawnmower" I would tell myself. The interesting turn was when I started writing these stupid ideas down.

Some stupid ideas will always remain stupid ideas, some though will change over time. That dumb idea might actually just be so brilliant it looks stupid. It might be so big and scary that you've convinced yourself it's stupid. It might not be a bad idea at all, after some careful reflection. It might even be a wonderful idea.

Why do we throw away our stupid ideas? Is it because we don't have time for them, bah, that can't be it, you are currently wasting precious time reading the stupid things I have to say. Is it because we are so swamped with brilliant ideas that we have to prioritize, probably not, or else you'd be sipping margaritas next to your perpetual motion machine. No, the answer is much simpler, its because we are afraid or failing. Stupid ideas are failure babies, a dumb idea grows up to be a hairy ugly failure, and we have learned in the school of life that failures are bad, so we avoid them.

Have you ever talked to a 4 year old? I have on various occasions, interesting little buggers those 4 year olds. They will tell you every little thought that bounces into their overly energetic little brains, no matter how wrong or foolish. They don't mind failing over and over again, they will insist the moon is made of mashed potatoes and not feel bad at all when you tell them the truth. And most importantly they learn a ton of information.

Now don't take this the wrong way, I don't think you should just go around like some half-cocked lunatic spouting off every little thing that pops into your head. There was a time for that and if you go into the board room now insisting the moon is made of mashed potatoes you will be roundly mocked and rightfully so. What I am advocating is the preservation and analysis of your stupid ideas. Let's take a look at some stupid ideas quickly to see why this is useful.

• Idea #1: A blanket that has arm holes and you wear it backwards or something... yea like a backwards robe! Pretty fucking stupid, but better known as the Snuggie, which has become a huge success and sold tons.


• Idea #2: Instead of executing the code on this machine, let's build another machine that doesn't actually exist and have that fake machine live on the real machine and have code execute on the fake machine.... What? What are you rambling about Jenkins, get out of my office! Jenkins just invented the Virtual Machine architecture that power things like Java's JVM and .Net's CLR.


• Idea #3: Wouldn't it be awesome if you could tape together 4 iPhones and have like a really big iPhone. I'm sorry, did you just smoke a bunch of peyote?! That's dumb.... Or its the new iPad.

You get the picture I'm trying to paint here. A lot of the amazing things we take for granted and depend upon everyday, things that seem like great ideas now, all had a moment in time when they were some far-fetched pie-in-the-sky dream. Most ideas start off as stupid ideas. Stupid ideas are important and valuable, not all of them, but there will be some diamonds in those dirt clods. You just have to be smart enough to sift through them.

My advice to you is to jot down those stupid ideas you normally throw by the wayside. Revisit them every once and a while, re-evaluate their "stupidness." Sometimes you will find a diamond in the rough, sometimes a stupid idea can be the spark needed to have a brilliant idea, and sometimes you will just get a good laugh out of the stupid things you think up.

open format

[caption id="attachment_663" align="alignright" width="300" caption="We\'ll just lock your data up in here, don\'t worry we\'ll open it up later if you need it. This is what a closed format sounds like"][/caption]

Back in the days when a computer had 64k of RAM and high-capacity meant you had a double-sided floppy disc, there were these funny little things called binary file formats. We still have them today, they are the "pattern" that a program will write and read to and from disc to save and load a file into a program. Some are open, like image file formats, and some are closed, like Microsoft's binary blob formats for Office. As the world has advanced and storage has become dirt cheap people started looking for an easier way, and whenever there is a problem that we don't have a tool for yet, we reached for XML.

XML is actually not a bad fit for this problem domain, its a little on the ugly side, but that's ok, very few people crack open a raw file to read it. The people that are cracking open files in a text editor to peer inside are probably tech-savvy enough to read a little XML. The big bonus of switching to XML, like ODF or DOCX, is that there are very mature XML parsers and writers for almost every programming language. That means that a determined programmer can grab a copy of your format spec, her favorite XML parser, and write a program that operates on your file format. This is the essence of that oh-so-marketing-speak word synergy.

Now I would love to take credit for this awesome idea, but if you've read The Art of Unix Programming you will recognize this as nothing more than an extension of the Rule of Composition. Now that you've read the Rule of Composition (and you should make the time to read the rest of the Art of Unix Programming, even if you never plan on programming anything for Unix, the lessons within are just in general great to know), you will recognize the inherent advantage of having parsable file formats. Now that I have cunningly converted you over to my way of thinking, what format should you use?

XML

XML (eXtensible Markup Language) is the old standby, it is reliable, standardized by the W3C, well-supported, and there are tons of tools around for playing with XML. XML is "human-readable" for those of you unfamiliar with it here is an example.

[xml]
<book>
<title>Example Book</title>
<pages>
<page>
<![CDATA[
...page stuff here...
]]>
</page>
<page>
...you get the idea...
</page>
</pages>
</book>
[/xml]

Its a bit tag-heavy and that means a slightly large file size, this isn't a huge concern since storage is so cheap, but you should be aware of it. XML has a neat feature called DTD (Document Type Definition), armed with this most parsers will be able to tell right away if a document is well formed. XML is big and clunky but a fine choice for many applications, especially if you need typing information.

YAML

YAML (YAML Ain't Markup Language) is the format of choice for the ruby community. YAML is well supported by most mainstream programming languages, it is a more lightweight choice than XML. Here is the same thing as above in YAML.

[ruby]
book: Example Book
pages:
- page: >
...here goes my page data...
- page: >
...you get the idea....
[/ruby]

YAML uses the structure of the text to indicate the structure of the data. Ending tags are dropped and indentation becomes more important. YAML looks simplistic at first but has a wide-array of functionality hiding below the simple hello world examples. References, hashes, arrays, and much more are possible with YAML. The specification allows you to make concise documents that contain an astounding amount of information.

JSON

JSON (JavaScript Object Notation) is a lightweight way to represent data structures. JSON excels by being incredibly simple to learn and use. There is native support for it in JavaScript which makes it ideal for use in AJAX (which would then technically by called AJAJ), and there are JSON parser available in most mainstream languages. Here is the example from above in JSON.

[javascript]
{title: "Example Book", pages: [ page: "...page stuff goes here...", page: "...you get the idea..." ] };
[/javascript]

Just like in JavaScript everything in JSON is a Hash or Array. JSON is a simple typeless data recording system, perfect for dynamic languages. JSON is a proper subset of YAML 1.2 so most YAML parsers can also parse JSON. JSON's incredibly lightweight nature lends itself for being used when sending data over a wire or when space is at a premium.

BSON

BSON (Binary JavaScript Object Notation) is a binary form of JSON. It is meant to be more space efficient than JSON but maintain the ease of parsing. It is described as "A General-Purpose Data Format" and was devised as the serialization medium for the MongoDB project. It is an interesting format to look at if space is at a premium and there are already parsers for C, C++, PHP, Python, and Ruby.

---

File formats no longer need to be gnarled, tangled messes. We have new standards that allow us to freely share data created in one program to be analyzed, manipulated, and turned into something new in another program. Being able to freely interoperate is the future of computing, it is the driving force behind web standardizations, micro formats, and open file formats. The next time you need to persist data to the file system, resist the urge to roll your own serialization scheme, and check out one of the technologies presented above. The world will thank you.