The Secret Recipe for Middleware Success

RecipeIt seems like just a few minutes ago, I was talking about the recipe for success, right?

The most successful companies in the Application Integration and Middleware (AIM) space have done what large software consulting firms have done. They go towards the money. Consulting costs money. Middleware requires consulting. The Fortune 500 requires middleware. They have money. It’s a win-win, right?

What this strategy has led to is an industry that fights over the top 5000 companies world wide and ignores the bottom 9,995,000 companies that can’t afford to have their own IT department, let alone hire a team of integration ninjas to come in and build a custom middleware solution to connect their WooCommerce, QuickBooks and Salesforce accounts.

The really expensive bit in the current middleware market is that huge up-front fee. Because middleware deployment is commonly modeled after software consulting and application deployment, middleware consultants use a traditional software development approach. This generally leads to custom deployments and customizations. This approach is not lean.

In the book, Blue Ocean Strategy, by W. Chan Kim and Renée Mauborgne, the authors suggest an approach to rising above the fray (red ocean) and differentiating your product offering so much that you create an uncontested market space. In many of the examples in the book, a new market was defined because the innovating firms lacked the financial resources to compete head-on with the established players. That the challengers would be the most innovative may seem counter-intuitive, after all, the big companies have all the revenue and capacity for R&D, so let’s discuss a little further.

Any very large company will pursue very large clients because their overhead makes pursuing the small fry cost-prohibitive. Any time you see a monopoly or near-monopoly serving a market, understand that there will be sales that can be made in that market to companies that the largest competitors won’t touch because they are too small. Monopolies get to be monopolies by serving everyone. To serve everyone, they have to grow to enormous size. Growing to an enormous size is inefficient in terms of revenue-per-worker. Overhead costs will mount to the point that only the largest customers with the highest margins look attractive. There will always be smaller companies with a lighter cost structure that will find smaller customers attractive. This is what I call my “Breadcrumb” theory. Not complicated at all.

Ahh… The free market. It’s a beautiful thing.

So, back to middleware. Middleware can be really, really expensive. So, if you sell hand made rugs over Etsy, and you use Xero accounting and you want to track your customers using Zoho CRM you may be forced to do a lot of manual integration. That is you may find yourself spending hours cutting and pasting and reconciling bank statements with customer orders to analyze customer buying habits. If you work in a company where the transactional cost to process an invoice is several hundred dollars, it may make sense to spend a few million dollars on integration automation. Chances are, though, if you sell handmade rugs, you can only afford a few dollars to alleviate this drudgery.

Since leaving Cat, I have been on the lookout for companies with technology I can adopt in my Integration Competency Center as a Service (ICCaaS). There have been several middleware vendors who have broken into this space of integrating the systems of microenterprises. SnapLogic, Mulesoft and Zapier are good examples of companies which are modular in nature, priced reasonably and require little hand-holding. The problem with these and other current technologies is they lack automated interoperability.

Automated interoperability is a fancy schmancy buzzphrase, so I’ll explain it like not everyone has an advanced computer science degree and has spent the last 30 years developing middleware.

If you think of data like water, then middleware is plumbing. That’s why I call myself a data plumber. I help people control the flow of their data. If middleware is plumbing, then the simplest conduit is comprised of a connector or spigot joined with a pipe or hose and another connector that serves as an outlet like a sprinkler.

For data, let’s say you have an accounting system like Microsoft Dynamics. You want your website sales from Shopify to automatically adjust your accounts receivable, inventory and general ledger. You’ll need a connector that has access to your Shopify account, another connector that can log in to your Dynamics system and some kind of pipe like the internet (hopefully a secure connection) to ferry your data back and forth. Two connectors and a pipe. In the data plumbing biz this is called a microservices-based connection.

The problem with the AIM industry today is that every vendor wants to sell an end-to-end solution. If Interwebalytics Incorporated, a company whose name I just made up on the spot (apologies to any real company of that name), has the best darn Dynamics connector in the world, but they don’t have a Shopify connector, you are out of luck. If Manacle Software Industrials, LLC (another company name I made up just now, I also have a really cool logo for them) is made up of ex-Shopify gurus and they have an awesome middleware connector for Shopify, but they don’t have a clue about Microsoft software and they don’t want to learn, again you are out of luck.

The only way a customer is guaranteed to get the best-in-class connectors and pipes for their middleware solution is if they can assemble them from components available from multiple vendors. This is what I mean by automated interoperability. If connectors and pipes from different vendors could work together seamlessly, that, ladies and gentlemen is the secret recipe for middleware success.

Update on the Verge of Launch

541921main_atlasvcloseup425xI’m in a bit of a quandary. When I left Caterpillar three and a half years ago, I had one thing in mind. Start a business that addresses a huge pain. The recipe for success, right? My huge pain was technology-based. The recipe for success, right? This pain existed in a very large industry. The recipe for success, right?

So my first order of business was solve the problem. And this is the problem. Integrating computer programs together is really expensive. Not just “Not on sale, today” expensive. Not just “Customizing your own car” expensive. But maddeningly-inefficient-beyond-belief expensive.

When a person wants to move data between programs, they “Copy” or “Cut” and then they “Paste”. This is called (derisively by many in the software industry) manual integration. Most of the time, using the clipboard goes well. If you happen to be cutting from a spreadsheet and pasting to a word processor, though there are many different ways this can be done, the results are close to what you expect.

When a person wants to move data in a different direction, say from a word processor to a spreadsheet, or from a graphics editor to a database, things don’t go as smooth as expected. This is because applications have data models. They are built around thinking of data in a specific way.

A word processor thinks about the data you type into it as a letter to someone, or maybe a book or a pamphlet. A spreadsheet thinks about data as information you want to perform a variety of analytic exercises with. A word processor then thinks that when you want to move information from another application you want to make it “visual”. You want to show other people “hey, look what I’m doing with this spreadsheet” or “look what I just drew”.

But a spreadsheet has no idea what to make of that 12 page essay with mixed fonts, styles and graphics that you just pasted into it. Analyze what? So pasting something the wrong way tends to produce unpredictable results, simply because applications don’t think the same way about data.

Now, imagine we are no longer talking about applications like e-mail, spreadsheets and drawing programs. Let’s start talking about the big applications that drive industry. Banking systems, healthcare systems, factory systems, human resource systems and engineering systems are good examples of what I mean by applications. These systems are just as driven by their data models, but the data models for enterprise-level systems are hundreds of times as complex as a pixel editor.

So people accept that integrating enterprise systems is complicated and therefore expensive. But it doesn’t have to be. Once the solution has been developed to connect a human resources system with an accounting system, programmers don’t have to pay more to duplicate the solution. They can just install the same solution between the same two applications at another company.

But they don’t. Current practices in the Application Integration and Middleware (AIM) market are to start a business engagement at a new company with a three step process.

  1. Roll up your sleeves.
  2. Pretend like you’ve never seen use cases like this before.
  3. Begin customizing the tar out of everything the company wants to use middleware for.

Now, a lot of industry practitioners are going to say this is balderdash. They’ve never once rolled up their sleeves. I get that. It’s a bit of an embarrassment to hint that the AIM industry loves to customize data models. As a matter of fact, it might be a revelation to some. You see, we don’t do this consciously. We look at customer requirements and our creative juices start flowing.

No two companies behave the same way. So, their use cases will always be different. My recommendation, as a customer, has always been to help us find ways we are similar to other companies with similar use cases. Please. Pretty please, with the least expensive sweetener we can place on top. You see, it isn’t middleware that’s expensive. It’s customization. And if your middleware consultant isn’t showing you ways you can change your own business processes and data models to more accurately reflect industry standards, you can save a lot of money by switching to a consultant who will.

I left Caterpillar because I thought Caterpillar was just doing it wrong. I thought we had no clue what we were doing and we were letting bad actors from the AIM industry steer us into the most expensive way of integrating applications together. After I left, I started talking to people. I started researching integration hairballs and found out the AIM industry does this to everyone. And it’s for a very simple reason. The AIM industry has grown, from its infancy in the 1980s, into an industry just like the automotive or software or hardware industries.

But it has nothing in common with those industries. It needs a totally different business model. And that’s the recipe for success.