Can you hear "it"? "It"'s a beautiful thing.

What's "it" you're wondering?

Why, it is innovation.

I ran across a social experiment this week. In it, Jonathan has shared his Starbucks card. No kidding. You can download a picture of it, and buy yourself a coffee if there's money on it. If you're feeling generous, you can put money on it. He shares instructions on how to do both.

The interesting part… he's created an API for this experiment.

I can't help but wonder if the API will help him understand the parameters of his experiment. With the API others can innovate on his idea, perhaps spark something that he didn't think of. These sparks might help him fine-tune his project to make it even more useful. They might even spark a whole community of activity around his concept.

Can you do this in your enterprise? Can you release a set of services, whoops - there I went and confused API with services again. Silly me. Can you release an API that let's your community extend your idea as a way of listening to what the community wants?

With the API craze that started sometime in the last 24 months, it might be easy to forget that API's are not new. Nor are the things we do with them.

There's been a lot of discussion about understanding your API. What's your business model? Who are your users? I know there are some grey hairs (and some blond-highlights) wondering what the commotion is about.

It's all about integration.

And, when it comes to integration you need an API.

I remember building networks that needed multiple network cards. You know, one for each network protocol (remember Novell Netware or Banyan Vines?). Back then, there was plenty of time to figure out the "application integration stuff" because what was happening at lower layers of the OSI model gave developers the time to react.

Then along came Cisco and others, and they consolidated the network. Yeah, there was a time there was something called a BRouter - a router that also bridged traffic.

(Forgive me, I think the heat-induced lethargy is causing me to be a bit nostalgic... reminding me of when I was young enough that the heat didn't slow me down.)

Anwyays, networks were much harder to build. You actually had to know where your users were, keep them as local as possible to the servers and the apps they were using. VPN's crushed that complexity, adding some more of their own.

The good news was, with VPN's any user could be on any network, but access their apps as if they were local. The bad news was that with VPN's any user could be on any network, but access their apps as if they were local.

It was easy to know - if a user called from the 5th floor, you'd go check out the 5th floor network. Now how are you supposed to know where that user is on your network. (Reminds me of the time a customer lost a server - they knew it was down because it stopped responding, but they couldn't find it. It took about 2-3 hours to find it. It was in a closet, which is also why it failed. There wasn't enough ventilation.)

Fast forward a bit more.

Now there are widespread standards all the way up the stack. Take a WSDL and within minutes someone without any knowledge of the target system at all, but with a reasonable development tool could be integrating systems.

Just like with the users in my VPN example... this creates some great opportunity only we've moved up the stack, so instead of talking about users, we're talking about data. Now, the good news (and the bad news) is that data can be anywhere.

What data? Well, any data. Let's say - customer data. Well, what makes up customer data?

Address. Phone number. You know, customer data.

What does it look like? How is it stored (in each an every system you're working with)?

I know what you're thinking... and let me tell you - Yeah, it does matter. And, it can be complicated.

Let me give you an example.

I upgraded to OS X Lion (for my Mac) this week. I went into iTunes, clicked "buy", and Apple asked me to validate my identity before allowing the purchase. It brought up a panel that asked for my credit card info, name, address, and phone number.

Simple, right?

I was on support for 15 minutes, frustrated, when I figured out the solution.

I logged into American Express, looked up my credit card information, and saw that on my American Express account, my apartment was listed simply as #37F, in a separate apartment field.

Now, I happen to know that the USPS standard for writing addresses is to put the apartment number on the first line, no comma separating. Something like '40 Harrison Street APT 37F'.

When I matched what I put into iTunes to what American Express showed on my account profile, bingo-bammo, it went right through.

Are you doing the integration dance? This isn't some leading-edge SMS-map-real-time-mobile-startup. This is Apple and American Express.

What's going on?

They use different data models. Neither particularly adhering to the Post Office's standard. Remember, it's not just Apple and American Express. It's Visa. Discover. And, so on. And, it's Spotify, Google, Amazon and every other service.

So what do you do? I mean, besides make your users hate you?

Manage your API properly. Deal with the trust, visibility AND data issues in a systemic way. API providers and consumers (both internal- and external-consumers) will have the ability to manage your data model, easily map that data model to the API's that make up your virtual integration infrastructure, and manage your user experience in a way consistent with your brand and user-experience goals.

Yeah. Bet that title got your attention.

I'll be gone for the next 10 days and didn't want you to forget me.

My soapbox about complex integration being ported from the enterprise to the cloud being a huge opportunity for the right solution had me thinking.

I wonder if a lot of the people developing "apps" using APIs even realize what governance is, and why it's critical. At a super high level, governance is the ability to articulate what to expect from an API, the formalization of that expectation into an SLA (service level agreement), and the careful monitoring of the actual API to make sure it continues to meet the SLA as defined.

Of course, one API may have many SLAs based on usage, and all that.

So, where does online dating come in as a benchmark?

I had this idea for turning Facebook into an online dating site to solve a huge problem with online dating. Let's think about the problem with the traditional online dating model.

A person goes online and tells some random website a whole bunch of personal details. Without any confirmation or reference at all, their profile is then published for other people to view. People seem surprised at all the exaggerations that are exhibited in personal profiles.

Like money, dating is something people don't seem to have such realistic views of when it comes to their situations. Makes sense too... if I met you for real, I wouldn't start the conversation by saying "Hi, I'm a compulsive neurotic who will make your life miserable." I'd more likely say something like "I'm great at my job because of my attention to detail" if I even let it come up at all in conversation.

While that's sorta funny... there are other more discrete ways to define people that aren't so "fuzzy". Like, "I'm not married" or "sure, this picture was just taken last month".

Married. Current photo. These are bits of meta data about the person that could be easily confirmed, though we're all concerned about big brother and (US) law doesn't make it easy for companies to be so intrusive in our life.

We're so embarrassed by online dating, that we keep it separate and only let other daters see dating profiles. So, we can exaggerate because our "regular" (non-datable) friends won't see what we're saying or hold us accountable.

From a technical perspective, the same analogy means that sometimes the technical infrastructure isn't there to constantly prove that what I'm saying about my API is true - even if it's a discrete measure. Like uptime. Or, expected performance.

And, that's where Facebook, or rather the community comes into play.

Imagine building a dating site layered on top of Facebook.

All of a sudden, when I say I'm 6'1" and my mother says "in heels" (that's funny because I'm a boy) someone looking at me as a prospective date will know my real height. If I say I'm not married, I'd have a hard time when I got home that day from work. And, if I thought I could create another profile, I'd need to figure out a way to get a reasonable # of friends so I look like a real person, with all the interactions on my facebook page that entails.

The community could hold people accountable.

Back to APIs.

However, if you'd like better governance of API providers, a way to let the community hold companies to a higher ethical standard of how they present their services, wouldn't it be better if the community had a place to share their experiences of the real level/quality of service provided by an API vendor so you could choose the right API for your situation? This community should be in the open, available for prospective customers, but validated by existing ones. Then, we'd have something that more reflects reality and everyone would prosper as a result.

It's absurd... and I'm shocked by such raw behavior.

Of course, maybe I'm old-fashioned? I've been feeling my age a bit lately, so it could be.

Maybe, I'm just too hard. I've seen a lot. Did you know that I once created a song for an IT department, so that they'd remember how IP multicast worked? Yeah, that was pre-internet. Well, truth-be-told, pre-internet-as-we-know-it.

And, that's the rub.

I know things. I've seen things... NO! Don't ask. I won't tell.

What I will say is, be careful.

When you look under the covers, stick your nose into places... you'll never look at a computer application the same way. Never speak to a customer service person without wondering. Never. Book. An. Airline. Ticket. Online. Without. Wondering.

Have they used a vendor's native API for integration without creating a layer of abstraction to prevent vendor lock-in?!?!

There, I said it.

Best practice numero uno. Number one. ABOVE ALL THINGS.

Don't code to a vendor's API because you're stuck with whatever they decide to do with it. You've got their methods, and methodologies (what I like to call "integration philosophy") baked right into your application. And, if you diverge from that vendor, philosophically or commercially, you're screwed.

This is extra important when you have people in your organization using external API's. Do you even know who they are? That's problem #1. Make sure to build in a gateway layer, or intermediary layer, whatever you want to call it. The important thing is to decouple your consumers from those API's so that you can operate independently from the vendor, their methodologies and their roadmap, and switch more easily between vendors should you need to.

Good luck, and code safely.

My thoughts this morning are on chaos. Complexity.

Integration.

What's it all about?

Integration.

The single biggest challenge with integration?

Choice. And, the rapid pace of change.

I know, pick one.

I can't.

There are too many options.

Reading an article on Bloomberg News this morning about Groupon's IPO this morning, I saw that Groupon has 482 imitators. Groupon launched in November 2008, roughly 30 months ago. That means, since launch, roughly 16 competitors a month have entered the mix. Keep in mind there are about 20-22 work days a month. That's close enough to 1 a day as to be really scary.

How do you make a good choice with so many options. Modern thinking on happiness says we can't. Too many choices are bad for happiness.

I think they're bad for business too.

And, when you combine all that choice with the pace of change you have situation where people can't keep up. It's like drinking from a firehose every day just to stay even. And, for many people this "integration" thing isn't even their full time job.

What do you think happens when people are presented with tons of choice, and a constantly changing landscape?

They don't choose.

That's right. They choose not to make a choice. And that's bad for your business.

Transparency is a market imperative, and of course, we each have our own interests. Some of us want transparent performance, others security. Some want transparency into how underlying algorithms are calculated, some only care about the response. In any case, transparency is "free market lubrication".

When it comes to integration, a highly complex collection of technologies to solve some business purpose, transparency into what's actually happening, and what can be expected to happen in the future, is critical to long-term success of the API market.

“Service” continues to be the most overloaded term in the IT industry.  This has led to a lot of confusion when discussing approaches, architectures, and designs.  The term Web Service did help to provide some clarity when discussing Service Oriented Architecture (SOA).  However, over time (and with Gartner’s pushing) that term has become very closely associated with the specific protocols SOAP, WSDL, and UDDI which are clearly not the only options for providing web-based services.  That is unfortunate because I think it was a naturally agnostic term for communicating the concept of electronic services that could be accessed over the Internet. 

Gartner came up with their declarative statement because customers were seeking guidance on whether they were implementing the right services to adopt SOA.  However, as everyone has since realized, the keys to success had very little to do with protocols and much more to do with the design, support, and management of services.  Regardless of what you call them, and how they are accessed over the network, the keys to success remain the same and unfortunately many of the same mistakes continue to be made. 

The recent trend is to use an extension of the term API (Application Programmable Interface) to classify these services.  While the use of API is valid technically, it can also a departure from the original usage which was associated with local software libraries from operating systems that were used to develop desktop applications.  For this reason, API is sometimes prefixed with “Web” or “Open” to further classify this new generation of API’s.  While I think either of these terms could stick, the real question is whether we can address the issues of versioning, service levels, and security that have been consistently problematic across web-based services regardless of what they are called.

When I started working with electronic services 11+ years ago, I saw the amazing potential to have all sorts of capabilities and information programmatically accessible to all types of developers and organizations across the globe.   SOAP was just starting to emerge as one standard, but many people were already starting to do lots of things via plain XML and some XML-RPC.  We had a wealth of knowledge coming out of RMI, DCOM, and other distributed messaging platforms and it really looked like the industry was going to take a huge leap forward.

I have struggled over the last few years to figure out just why we haven’t come anywhere close to that potential destination.  It is easy to pick on SOAP as the big disappointment, but that would be the wrong focus as SOAP was never intended to be a solution to anything all by itself.  Any industry specification (good or bad) requires tooling, best practices, and broad adoption to have any hope to succeed.  The truth is that SOAP was probably never better than it was Day 1.  It went from a fairly simple concept of flexible message exchange to one that incorporated many different interest and opinions to turn it into the next RMI or DCOM.  Meanwhile, all the vendors were looking to one-up each other, adding value that also happened to make it more and more difficult to achieve the promise of interoperability.  Therefore even the most basic premise of SOAP has been compromised.

Many think that a different protocol (REST, ODATA, you name it) will solve the problem.  Only those that do not learn from history could possibly believe that.  In fact, if we really solve this problem, I don’t believe the protocol of choice will matter one bit.  At the end of the day, we simply need a way to discover, evaluate, shape, compose, monitor, and trust services if we are to realize the potential of a global, interconnected, programmable platform. 

Many organizations and analysts talk about reuse as a key determination of success for architecture and application designs, especially when services are involved.  As a provider, you certainly want to build components that are reusable and we’ve been striving to attain that objective for quite some time through the use of objects, data models, and UI controls (to name a few).  Investments are often made in tools, patterns, and training to help maximize the amount of reuse with varying degrees of success.  However, striving and pushing for reuse from that provider perspective is only half the picture… and actually the lesser half.

I have seen many organizations agonize over the design of a single service, wanting to ensure that they are creating a reusable capability for the rest of the organization, if not beyond.  The reality is that the provider has limited knowledge and control over whether this will actually happen.  Since the actual manifestation of reuse comes once something needs to be built or modified, why don’t we shift the burden of reuse to the designers and developers of applications?

There are several different approaches in the industry for designing a new application, but very few have incorporated into their approach the evaluation of “other people’s stuff(ops)”.  It is all too easy for an application designer to either overlook that step completely or give it only casual consideration believing that “ops” isn’t good enough.  In my experience, most of the reuse in the enterprise comes from scenarios where the provider and consumer are the same person or at least report to the same manager.  This is not unique to services as even with OOD, architects are quick to determine that objects designed by other architects are not appropriate enough to extend, much less reuse as-is.

As the industry moves to a more decentralized model to provide user experiences, processing systems, and data, there are more and more opportunities to leverage “ops”.   If we don’t take advantage of these opportunities, we will end up with every developer having their own instance of like capabilities which simply leads to decentralized silos instead of a decentralized ecosystem.

Thus, human nature is certainly one barrier to reuse in the real world, but it is only one of three.  The others are logistical and technical in nature.  In my next post, I’ll talk more about these additional barriers to reuse and what OpusGrid is doing to remove those obstacles.  In the meantime, don’t let yourself or your organization be fooled into thinking you are maximizing reuse when you aren’t even using “other people’s stuff”.

Over the last twelve years, I have been working with enterprise customers and colleagues to leverage the power of services to enable a higher level of agility.  During my travels and interactions I came to realize that there are many barriers to agility, only some of which were technical in nature.  The best practices have to be followed, the tools have to be used, and different constituents have to communicate.   Even with all of this going in your favor, agility is still far from certain.  When all is said and done, change is the reason applications, systems, and even organizations are not agile.  Whether you are developing software or a building, changing a component after the fact is a painful and expensive, if not prohibitive proposition.  Many people have built a house to later lament not putting in a gas stove or a 3rd car garage.  Many architects have designed an application based on US customers to later need to support international addresses.  We have all faced situations where we would love to get a “do over” so that we could deal with the internal desire or external push for change.

Almost five years ago, I started an initiative that allowed software developers to get a “do over” when providing and consuming SOAP services.  This resulted in the recently-abandoned software solution from Microsoft known as the Managed Services Engine.  I learned a great deal from designing that solution and working first hand with dozens of customers that were enticed by the opportunity to leverage service virtualization.  It allowed them to not only get a do-over, but to allow forward-thinking companies to accommodate change proactively through dynamic versioning, routing, and policy.  That solution was designed to address SOA scenarios ranging from the basic to the very advanced that only a handful of enterprise customers would truly appreciate.

In the last couple of years, it has become very apparent that SOAP will not be the de facto standard for intra-application communication.  What is also equally apparent is that no communication standard will be any more agile than anything we have tried so far.  Some protocols make it easier for certain stakeholders by shifting the problem around, but the fundamental problem still remains:  change causes havoc.  It doesn’t matter if you are building an enterprise business application, a Facebook app, or an iPad app.  The more powerful your app, the more it uses services… and the more it depends on those services.

OpusGrid is the next step in my pursuit to help make software and services inherently more consumable and agile.  I am fortunate enough to once again have some great colleagues to help go make this happen.  Over the next several weeks we’ll be unveiling some great capabilities and along the way I’ll be sharing my thoughts on how our journey as well as how the industry is reacting to the ongoing cycle of consolidation and decentralization.  Its going to be a fast-paced ride but its also going to be a lot of fun!