Is Schemaless the Next Big Thing?

Schemaless structured information modeling is a growing success story in Internet IT and it’s just now beginning to show signs of life in institutional IT. Is schemaless The Next Big Thing? Is it a fundamental advance or just a fad? Is it a tendency, a trend, or a tsunami? Is it a movement, a mirage, or a market opportunity? These are important questions for investors as well as implementers and these questions will be considered here by means of comparative innovation analysis.

Schemaless is the only workable way to publish text documents so we’re all schemaless experts but schemas have dominated data publishing for decades. Schemaless can be used for data publishing as well as document publishing and now the big open question is whether and when schemaless publishing will displace schemas for structured information modeling.

Schemaless and schemas are two fundamentally different technologies for structured digital information modeling and in many ways they’re antithetical. The most important differences between these technologies involve publishing rather than programming and it’s these publishing issues that will determine which technology is dominant in future IT modeling practice. In many ways schemas are the superior technology from a best practice programming perspective but schemaless has many major publishing advantages including simplicity and scalability which will surely be decisive in the Internet Age.

The printing press displaced parchment, telephony displaced telegraphy, and television displaced radio. The history of technology clearly shows us that superior publishing technologies always displace prevailing ones and schemaless is far superior to schemaless as an Internet publishing technology.

Most IT modeling has long been done with schemas but schemaless modeling is a mature alternative that finally enables mass-scale Internet open data sharing and searching systems as well as disruptive new application architectures that overcome many longstanding limitations of enterprise IT systems. Schemaless is just now beginning to challenge the longstanding dominance of schemas and the accelerating shift to schemaless is being driven by a combination of rapidly mounting economic and engineering forces. The competitive landscape of the IT industry will be substantially altered as the shift to schemaless becomes a switch to schemaless and while this switch may not yet be imminent it’s clearly inevitable. The switch to schemaless is likely to start sometime in the next few years but it will surely begin well before the end of this decade.

Schemaless is a method of systematic structured digital information modeling that’s best understood as an alternative to schema-structured modeling. Most IT modeling done today uses regular, repetitive schema structures and these have been the basis for information modeling in data processing since the 1960s. Schemaless information modeling is older than schemas and computer scientists tell us that schemaless is obsolete yet it’s still widely used in specialized applications where schemas provide inadequate modeling strength or scalability such as design automation.

Schemaless information modeling is hardly new, we all do schemaless information modeling all the time since it’s been the sole basis for practical information modeling since the dawn of human civilization. All practical language modeling is schemaless and the native, natural, nomological meaning structures of practical information modeling have been well understood since at least the 19th century. The classical elements of composition including rigor, rationale, and rhetoric completely explain all the meaning structures of schemaless information modeling and we all routinely master these elements in middle school.

Schemaless structured information modeling is just a very minor variation on the familiar ways that we all routinely model information with documents and so has the potential for widespread popularization. Moreover schemaless information modeling is easy to search and share at Internet scale while this is not possible with schema-structured information which can only be modeled by expert data engineers. For these reasons schemaless is the ideal choice for the democratization Internet data which will make it easy for everyone to author and analyze all kinds of data thus transforming Internet publishing in much the same way as democratization of Internet documents built on innovations such as blogs and wikis.

Systematic schemaless information modeling has been widely practiced in the practical publishing world for at least a century as a tool for ensuring information quality, as a way of doing systematic translations, and as a way of building practical writing skills. Schemaless digital information modeling dates back to the earliest days of digital computing and it’s still prevalent in assembly language programming. Schemaless data modeling has been widely used in design automation and defense automation for more than thirty years where it provides far more modeling strength and scalability that alternative schema-structured methods. The digital circuits in your cell phone are designed with schemaless circuit models and schemaless design automation systems created most of the models used to make and maintain your car.

Schemas are an entirely artificial approach to information modeling that always requires lots of smarts as well as highly specialized skills. Schemas began in the 1960s as a convenient way to move punch card information models from automated tabulating equipment to the first generation of scalable mainframe computers. Schemas are best suited to traditional data processing models represented as highly regular rectangular tables and these are ideal for many high volume applications such as inventory management and financial account journals. Schemas always trivialize any information modeled by stripping away all the referential richness that gives documents their strong meaning but schemaless information models easily retain all this richness and so rivals the editorial modeling strength of documents.

Today were seeing a substantial shift from schemas to schemaless in Internet IT as well as a much smaller shift in institutional IT. In both cases this shift is a technological transition from the mature prior art of schemas but shifts of this sort never happen without good reason.  Style wars have always been endemic in all IT specialties and these are always feuds about coding fads and fashions but the shift to schemaless is driven by major strategic technology advantage so it’s very much like the shift from gaslight to electric light in the 19th century.

In the case of Internet IT the major advantage lies in website scalability and the shift to schemaless becomes an urgent necessity once the intrinsic limitations of schema-structured information modeling and management have been reached. In this way the shift to schemaless is much like the transition from piston engines to jet engines made in the commercial aviation industry starting in the 1960s.

The next big schemaless opportunity in the IT industry is freely open data sharing and searching systems. Planetary scale document sharing and searching has long been a huge success story based on the open publishing technologies of the World Wide Web. So far all attempts to duplicate this success for data have failed due mainly to the scalability limitations of schemas but schemaless data publishing easily overcomes these limitations. Unlimited schemaless data publishing scalability provides the Internet industry with opportunities to build on its growing schemaless experience to pioneer big public data search engines, to finally launch the long-delayed Internet of Things, and to combine the complementary capabilities of individual websites by means of mass federation systems.

In the case of institutional IT there’s not yet an urgent need to switch to schemaless information modeling but schemaless provides the only remaining workable path to progress past the dead end of today’s fully mature data processing practice. Enterprise software progress has long been stalled due to the maturity of schema-structured IT modeling and this explains the long-term trend towards consolidation and commoditization in the enterprise software industry. Here the shift to schemaless most resembles the transition from tubes to transistors in the electronics industry beginning in the early 1960s. Transistors initially gained a foothold by enabling product innovations beyond the mature capabilities of tubes and over time the gradual increase of major and minor transistor product innovations rendered tubes obsolete.

In just this same way schemaless enterprise software promises to provide huge economic and engineering advantages in enterprise IT all the way to eventual perfection but progress will be incremental and we’re just starting to climb the learning curve today. It’s not at all clear how long it will take to climb this curve because there are strong precedents supporting predictions ranging from a few years to a decade or more. In any event this curve will surely be climbed so schemaless enterprise software will eventually render all current enterprise software technologies obsolete so schemas are surely doomed to follow vacuum tubes into technological obsolescence.

Transistors were pioneered by Bell Labs for commercial purposes but their rapid technological development was mostly driven by strategic defense systems while initial commercialization was driven mainly by innovative consumer electronics and computer hardware products. Today there is no sponsored innovation for schemaless IT innovation in government, industry, or academia but at the same time there are no big breakthrough innovation challenges. Much important software innovation of all kinds now results from side projects and these occasionally become the basis for great pioneering products. Schemaless solutions are fairly simple to build and these will surely increase steadily over time as success builds on success while sanctioned solutions will steadily increase as institutional IT managers build a better understanding of schemaless success factors.

Schemaless enterprise IT involves very different sets of tradeoffs than the sort done with schemas. Enterprise software products are packages of modular programs based on fixed schema models that provide a very narrow range of configuration and each package is a closed world system that acts as an isolated island of information so sophisticated techniques such as data warehousing are required to build bridges among these islands making it possible to systematically share and search information across packages. Schema modeling projects are costly and complex requiring weeks or months of expert effort while equivalent schemaless modeling projects can be done by everyone in hours or days. Schemaless information is easy to search and share across diverse sources and topics and systems of schemaless solutions are easily federated eliminating the longstanding islands of information problem.

Catalytic factors sometimes rapidly drive new strategic technology innovations to dominance where there aren’t any big innovation barriers as in the case of schemaless enterprise software. Aggressive U.S. federal government support and sponsorship of World Wide Web projects during the 1990s is one example of just this sort of catalysis. In just this same way new schemaless solutions might rapidly displace established enterprise software solutions and disrupt the enterprise software industry given the right set of strong catalytic factors. Asian governments have often provided just these sorts of catalytic factors in order to capture complacent markets for their nascent technology industries and in just this way an rapidly industrializing nation such as Vietnam or India might easily capture a major share of the enterprise software market in the space of a very few years.

All things considered schemaless information modeling is best understood as a mature technology that must eventually displace schema-structured information modeling in both the Internet and institutional IT segments. In the Internet segment schemaless is now becoming standard practice for larger websites and it’s becoming a standard offering in cloud computing infrastructure stacks. Schemaless is a small but growing trend in institutional IT where it’s seen as an advantageous alternative for an increasing range of specialized applications. There is no schemaless enterprise software industry yet and there is nearly no impact on enterprise software vendors so far but this is an unstable situation so a relatively rapid transition to widespread deployment of schemaless enterprise software is a very real possibility.