Technology only exists thanks to innovation. If nobody was pushing the boundaries with fresh ideas, then technology, and the people who depend on it, would have died out with the neanderthals. But while many people might believe the big tech vendors are the ones responsible for driving most of the innovation in computing today, that assumption couldn’t be further from the truth.

What’s accelerating innovation in tech right now, and what will further drive it in the future, is the thousands and thousands of developers taking part in hundreds of different open-source projects. Some of these projects are on a very small scale, while others are massive, supported by developers from all over the globe.

Even better, companies that for decades closely guarded their own closed-source technology like it was some kind of Holy Grail are not only adopting open-source tech in droves, they’re also actively engaging in the development process as well. People are quickly realizing that while open-source offers plenty of value, there’s much more to be had when they participate in the projects that matter to them. As a result, open-source is booming, and there’s plentiful research to back that up.

Take this 2007 research by Dirk Riehle, which illustrates how the number of open-source projects skyrocketed during the last decade. While that research is dated, adoption has actually increased dramatically since Riehle’s research was published, as more recent Black Duck data indicates.

Of course most readers know this already. But what they might not know is that open-source is now the primary driving force behind all new technologies that matter – Big Data, cloud and mobile – and it’s driving innovation at a rate we’ve never seen before.

“Open source software tends to innovate faster than proprietary because a successful open source project will draw contributions from large numbers of developers and users,” said Bob Wiederhold, CEO of open-source NoSQL firm Couchbase, Inc. “The accumulation of this community participation accelerates delivery of the key features and ecosystem that enterprises need. The result is that open source usually delivers better, faster products than competitive, proprietary alternatives.”

It’s a belief that’s shared by Cloudera Inc. co-founder Mike Olson, who stated that “No dominant platform-level software infrastructure has emerged in the last 10 years in closed-source, proprietary form.” While it’s debatable whether Apple’s iOS meets the definition of “dominant platform-level software infrastructure,” Olson’s statement is mostly true.

The real innovation in tech is now happening in full view of anyone who cares to look. And that will have big consequences as the biggest vendors face up to a reality in which the innovation is out of their control.

Fueling innovation

That open-source will only accelerate innovation was a theory posited by Wikibon’s Jeff Kelly during a recent presentation at #BigDataSV (see clip below). Kelly talked about how consolidation is inevitable in the Big Data/Hadoop space (something we’ve already seen, with Teradata Corp. snapping up companies like Hadapt, Inc. and Think Big Analytics, Inc.). But unlike traditional markets, consolidation won’t stop the pace of innovation.

Red Hat CEO Jim Whitehurst nailed it years ago when he correctly asserted that much of the closed-source enterprise IT world is a “extraordinary waste”.

“The vast majority of software written today is written in enterprise and not for resale,” Whitehurst said. “And the vast majority of that is never actually used. The waste in IT software development is extraordinary….”

Companies zealously guarded closed-source software in the belief that it was a source of competitive advantage. But that strategy tends to quickly kill off the kind of innovation we see in open source.

“So often we’ve seen startups create some great new, innovative technology,” Kelly said. “Then the IBMs and Oracles of the world acquire the best of these start-ups, ramp up marketing and start driving real revenue but the innovation stops.”

The closed-source approach also tends to cause vendors to become more rigid and less specialized, as Redmonk, LLC analyst Stephen O’Grady has noted:

“The mainstream technology industry has, in recent years, eschewed specialization. Virtual appliances, each running a version of the operating system customized for an application or purpose, have entirely failed to dent the sales of general purpose alternatives such as RHEL or Windows. For better than twenty years, the answer to any application data persistence requirement has meant one thing: a relational database. If you were talking about enterprise application development, you were talking about Java. And so on.”

Somewhere along the line companies like Facebook, Google and Netflix realized vendors could no longer meet their needs, and were forced to begin building their own software instead. In effect, the customers have become vendors themselves.

This new breed of company develops software under the assumption that there’s far more to be gained by opening their inventions to the wider world. It’s no longer the vendors, but rather the users of open-source technology that are driving its development – users like Facebook (which created the Cassandra NoSQL database), Amazon and Netflix, among others.

“Since these companies don’t gain their competitive advantage from the technology itself, but rather in how they use the technology, they open source the technology to the world,” Kelly said. “Now the Facebooks of the world also have a community of developers helping make the technology they use internally become that much better.”

Vendors stagnate

One reason open source took off in the first place is that enterprises often have very specialized requirements, and that means the software they use has to be customized for their purposes. As such, many enterprises see open-source software as cheaper and arguably easier to work with than proprietary alternatives because have access to a broader range of experts to solve their problems.

If we think about the most visible technologies today, the vast majority were originally built by a company and have since been made open-source: Android, Cassandra, Hadoop, Linux and MongoDB to name a few. They were developed by user companies because nothing in the commercial space satisfied their needs.

These companies weren’t in the business of selling software. Rather than taking on the expense of enhancing and maintaining their creations, they let a community grow around them. Hadoop was created by Yahoo! Inc. for example, but its development is now led by Cloudera and Hortonworks Inc. Cassandra was built by Facebook, but DataStax, Inc. is its biggest backer now.

A dog-eat-dog world

All well and good, but the only danger with this trend is that innovation is happening so fast that few technologies will be able to keep up, Tech Republic’s Matt Asay argues. Asay says that open-source has essentially become a dog-eat-dog world, and that companies and vendors alike need to be aware that “no dominant platform-level software infrastructure that emerges in open source form may last 10 years without being disrupted by other open-source software.”

With vendors no longer driving innovation, experts believe the Facebooks of the world will be forced to keep inventing newer technologies to solve new problems, and those could easily end up displacing today’s most widely-used solutions. The result is an ever-faster pace of change, faster obsolescence and more difficulty for users trying to keep up.

“The open source model will continue to drive innovation across every aspect of technology as a growing number of enterprises from every industry adopt a new infrastructure to support their Web, mobile and IoT applications,” said Couchbase’s Wiederhold. “In technology areas that are under-served by existing open-source projects, new open-source projects will be started and companies will form around them.”

Which means enterprises will need to be careful about which open-source technologies they choose to back, especially if they intend to participate in development. Numerous open-source projects may be formed to tackle what is fundamentally the same problem, which means that not all of them will survive.

That’s what happened in the NoSQL market, where numerous open source-projects were started in the 2006-2010, said Wiederhold. However, as market leaders like Couchbase, DataStax and MongoDB gained traction, many of those projects fell by the wayside. “The ones that get the most traction will attain critical mass and thrive,” Wiederhold said. “The ones that don’t will flame out relatively quickly.”

Open source eats open source, and so the cycle of innovation continues.

Image credits: Ioxso via Pixabay.com; Nicooo76 via Pixabay.com