Why does XML suck?

XML sucks. Don’t get me wrong. All kinds of really valuable and important systems use XML to perform vital functions. But performing a vital function does not keep something from sucking. Lots of people think Windows sucks, but it performs a vital function, and lots of people use it because of that. In fact, performing a vital function is what keeps sucky things around. If they perform a vital function, they are hard to get rid of, despite the fact that they suck.

Not that that prevents people from trying to work around them.

Scott Able recently tweeted

If we have markup, why do we need markdown?

Markdown has been making major strides recently, particularly in the Web and developer communities. What does it have that XML doesn’t
have? Simple: It’s easy to write.

XML sucks for writing. Lets look at why, and how it came to be that way.

XML is verbose

First and most obviously, XML is verbose. If you write in raw XML you are constantly having to type opening and closing tags, and even if your editor helps you, you still have to think about tags all the time, even when just typing ordinary text structures like paragraphs and lists.

And when you read, all of those tags get in the way of easily scanning or reading the text. Of course, the people you are writing for are not expected to read the raw XML, but as a writer, you have to read what you wrote.

Of course, you could use a specialized XML editor that hides the tags, but that is another tool to learn and use and add to your workflow, and that sucks. Plus, they don’t completely solve the problem, as we shall see.

Markdown is much less verbose. You can type a markdown document with not much more effort than typing plain text, and you can read it easily in source form.

But there are other less obvious, but ultimately more vexing problems with XML.

XML makes whitespace meaningless

How do you indicate the structure of a document in print or on screen? Partly you do it with font choices and the like, but first and foremost, you do it with whitespace. Whitespace naturally and visually delineates the parts of a document. One paragraph is separated from another by whitespace.

Markdown uses whitespace to delineate structure just the way a normal document does.

XML? Not so much.

In fact, in XML, most whitespace is simply discarded. You have to make special provisions to keep significant whitespace in things like code samples. To create a paragraph, you actually have to create an element (element <p> in some languages, <para> in others). We may be so used to this now we hardly think about it, but really, why the overhead when we could just hit return?

In ordinary printed text, we use indentation to indicate a code sample or a quotation. In markdown, you do the same. In XML? Nope. More elements.

This failure to use whitespace to mean what whitespace means in ordinary documents is a major contributor to the verbosity of XML markup. It is why we need so many elements for ordinary text structures and why we need end tags for everything.

Why does XML ignore whitespace? Well, XML is a descendant of SGML. SGML stands for Standard Generalized Markup Language. The Generalized part means that it was intended for describing all kinds of markup for all kinds of purposes. That includes markup for things that might not have paragraphs or code samples or quotations. That means that whitespace could not mean anything specific in SGML’s “reference concrete syntax” — its default way of writing tags and attributes.

However, SGML has several facilities that you could use to make whitespace and other characters mean something specific in a particular tagging language. Using these features, you could make a markup language that looked like Markdown and still parsed as an SGML document. In other words, in SGML, you could create a markup language that was easy to write. (Admittedly, these features were so fiendishly difficult to use that they had no chance of catching on.)

XML did away with all these features. XML discarded all the things in SGML that made it possible to create markup you could actually read or write.

Why? Because that was not what XML was designed for. XML was designed as a data transport layer for the Web. It was supposed to replace HTML and perform the function now performed by JSON. It was for machines to talk to machines, not for writers to write in. Writers were to write either in SGML or in WYSIWYG editors.

So, XML ended up with no way to make whitespace meaningful. Which sucks, because in actual writing, whitespace is the basic building block of structure. Hitting return to create a new paragraph is an ingrained behavior in all writers, and XML breaks it. That sucks.

XML has no semantics

The reason that whitespace is meaningless in XML is actually far broader than just whitespace. XML markup has no semantics. That is, nothing in XML syntax itself has any meaning relative to the content or data it is marking up. It only has meaning in terms of XML’s own abstract data model of elements and attributes. It is entirely up to each specific XML tagging language to define what its elements and attributes mean. XML does not predefine anything for them.

This is deliberate. XML was designed to be semantic-free, and for good reason. XML is what we call a meta-language. It is a language for describing other languages. Thus XHTML, DITA, DocBook, RDFa, RAML, SVG, XSLT, WSDL, XProc, S1000D, and BeerXML, along with all languages in this list on Wikipedia, and thousands more, are all markup languages with specific individual semantics, all of which are defined in XML. If any part of the XML syntax itself had inherent semantics, all those languages would have to share them, even if it were not appropriate for them. So XML is semantic free.

But that means that you have to specify absolutely all of your semantics yourself by defining elements and attributes for everything. This is laborious, and tends to produce quite verbose data formats. The design goals section of the XML specification actually states: “Terseness in XML markup is of minimal importance.”

For any application for which terseness is a useful property, this sucks. Human readable and writable markup is one of those applications.

The lack of semantics in XML itself also creates a processing overhead. You have to parse the XML syntax first to pull out the elements of the specific tagging language and then run separate application code to interpret the semantics of that language. Along with its lack of terseness, this is what led to XML being rejected in favor of JSON by many web developers. JSON is “Java Script Object Notation”. It has the inherent semantics of Java Script objects right in its syntax, which means it is trivial to parse into memory and access directly with application code.

Being semantic free is an important property of XML in its quest to be highly general. But it makes it suck for things like
writing documents or swapping data between web servers and Java Script programs.

XML hides structure

Of course, if you have a graphical XML editor, you get back the ability to hit enter for a new paragraph. At least, sort of.

It’s really not that simple. Since XML has no semantics, there is no way for the editor to know that Enter means paragraph, or even which tag means paragraph. It can know this for specific well known languages like Docbook, of course. But even then, there are problems.

When the writer hits enter, do they mean “end this paragraph” or do they mean “start a new paragraph”. Typically, they mean both. But suppose there are other elements allowed inside a paragraph, like a list? Maybe the writer wants to create a list that is logically inside the current paragraph. It will still be whitespace separated in the output, of course, but if it is logically inside the paragraph, then we can’t assume Enter means “end this paragraph”. It may mean “start a list inside this paragraph”.

And because there are a number of elements that could come after a paragraph, beside a new paragraph, we can’t assume that Enter means “start a new paragraph” either.

Which means that when you hit Enter, the editor has to ask what you want each time. In some editors, therefore, hitting Enter brings up a list of choices. Usually there is a default choice, which is usually a new paragraph, which you can choose by hitting Enter again. That’s not so bad. But then again you can’t always hit enter twice without looking because sometimes the thing you want next isn’t the default. You have to stop typing and look. And that sucks.

It gets worse when you try to edit something. Let’s suppose you want to insert something between a list item at the end of
one section and the title of the next section of a document.

On your screen you will see white space between the list item and the section title. In that white space there are the starts and ends of several elements.

• The end of the paragraphs inside the list item
• The end of the list item
• The end of the list
• The end of the paragraph that contains the list (if it is inside a paragraph).
• The end of the section
• The start of the new section
• The start of the title of the new section

But these starts and ends of structures are all invisible to you because they are not part of the WYSIWYG display. Depending on what element you want to insert, you have to get your cursor into the right bit of on-screen whitespace that represents the element you want to be in.

To add an item to the list, for instance, you need to get your cursor inside the list. To add a new paragraph to the section, you need to get your cursor after the end of the paragraph and before the end of the section. But you can’t see any of these locations. All you can see is a block of whitespace. The editor will have aids to help you find the right place, of course, but it is all terribly cumbersome and time wasting for what should be an operation you perform virtually unconsciously.

This is far from being the most difficult instance of this, and the result is that most writers writing in XML regularly switch back and forth between the graphical view and the raw tags view of the document they are working on, in order to find the place they need to be to make the edits they want. And that sucks.

Cutting and pasting content in the graphical view of an XML editor can be even more of a nightmare. You can’t see exactly which set of tags you have picked up, so you can’t tell what is legal to paste in what location, or what damage you might be doing to the current structure by removing the chunk you cut. Again, the editor will help all it can to repair the damage, but, again, what should be a simple and intuitive operation becomes an exercise in negotiation and troubleshooting.

And this is just the problem of dealing with simple text structures. What if you have significant semantic markup in your document? These elements and attributes are not visible in the graphical view because they are not intended to be visible in the published document, but to inform downstream processes. Particularly maddening are XML attributes, which are use for information that is intended to be “out-of-band”, that is, not presented to the reader. Editing and entering attributes in a graphical view invariably involves multiple keystrokes invoking dialog boxes, choices from lists, and typing into dialog fields. It sucks.

In short, you can’t just sit at the keyboard and type your document, and that sucks, no matter how good your XML editor may be. There are some very good XML editors, and you are likely to need one if you have to write a lot of XML, but none of them can take all the suck out of XML itself.

In Markdown, by contrast, you can just sit down and type your document.

To create structure, you need to see structure

The key thing here is that to create structure, you have to be able to see the structure. We have all used those free-form HTML editors in blogging platforms and Web CMSs and we know that the HTML they produce is a horrendous unstructured mess.

Using Markdown is a huge improvement because the author can see the structure and therefore won’t mess it up. And because Markdown uses whitespace to mean what it natively means, there are few end tags to worry about, and therefore to misplace, so it is physically impossible to make as much mess as you can with HTML tags.

XML editors, while much stricter, still don’t let you see the structure, but instead of letting you make a mess, they continually scold you with obscure error messages about structure errors, which are hard to fix because you can’t see the structure. Or they simply prevent you from typing what you want to type until you get your cursor in the right bit of whitespace. (Actually, how much mess you can make vs. how much scolding you get is more a function of the specific language than then editor. But it is always one thing or the other.)

The alternative is to look at the raw XML tags, but while the structure is technically visible in raw XML view, it is often impenetrable because of the verbosity of the tags, and because your most basic structural clue — whitespace — is either not used at all or is used to show the hierarchy of the tags, not the significant structures of the document.

In short, XML hides document structure no matter how you view it, and that really sucks.

The price for generality is suckiness

This is always the problem with generality. The things you have to do to a system or format to make it more general inevitably make it more complex and more verbose. It can’t incorporate all the shortcuts that would seem natural in a system built for a single purpose because they would be inappropriate, awkward, or limiting for other purposes. If it attempts to provide a mechanism for building your own shortcuts (as SGML did) then it becomes even more complicated.

Inevitably, people will create special purpose alternatives for individual applications (like JSON or Markdown) or attempt to simplify the general system (like XML did to SGML).

The upside of the more general system, at least if it has any traction, is that there are tools and people who support it. Sometimes the presence of these resources makes up for the costs of its verbosity and complexity.

In other cases, however, the level of support does not make up for the suckiness. Then new formats are proposed and developed and support systems develop around them, as they have with JSON and Markdown.

Elliot Kimber has an excellent presentation called Why is DITA so Hard. DITA is hard, he acknowledges, but he maintains it is not DITA’s fault. DITA is hard because the problem is hard. The presentation lays out the full extent of the configuration management and publishing problems associated with technical communication (and large scale corporate publishing more generally) and shows how DITA seeks to be a general solution to those difficult problems, which makes DITA itself difficult.

The thing is, of course, that no one organization actually has the complete technical communication problem, which is the union of all the problems that individual organizations face. DITA is hard because the entire problem is hard. But no one has the entire problem. Many organizations don’t have anything like the whole problem. So why do they need a tool that is as hard as the whole problem?

Part of the answer is that if they have a significant part of the whole problem, it is less expensive to adopt a tool that solves the whole problem — even if it is more difficult to use — because of the support base, the tools, and the accumulated knowledge. (This depends, of course, on whether there are other tools available that are closer to the subset of the problem that they have.)

Partitioning the problem space

There is another approach to solving hard problems, other than building one tool to solve them all, and that is to partition the problem space into smaller problems that are easier to solve, and which have solutions that are easier to use. That is what many organizations have done, and it is often the right economic choice to make.

In fact, all organizations partition the problem of creating content to one extent or another. There is no organization that is going to create all of its content in DITA or in any form of XML. It would be cost prohibitive to do so, and people across the organization would rebel against the difficulty and complexity and the general suckiness of trying to do everyday communication tasks with these tools.

The price of partitioning is fragmentation

This is not to say that all is wine and roses with Markdown. Markdown is a million miles from being a satisfactory substitute for XML in all the vital functions that XML is used for. XML is a metalanguage suitable for a wide range of uses. Markdown is a tagging language suitable for writing simple web pages, and not much else.

XML is semantics-free. Markdown is all semantics. Every piece of Markdown syntax means one specific thing for all Markdown documents. If you want a structure that Markdown does not support, there is no way to add it, no way to express it at all, except by embedding a chunk of HTML. This sucks either way.

And if you also want to create a nicely formatted paper document from your Markdown content, you will find it really does not have the ability to express the structures you need for that. Which also sucks.

As a result, lots of people have built their own versions of Markdown by modifying the Markdown parser (or building their own).

Other people have built alternative lightweight markup languages for different purposes. In fact, there are lots of them, some much older than Markdown. For instance, if you are interested in a language you can use to produce a printed book, you could look at AsciiDoc, which was based on DocBook semantics but with non-XML markup.

Most of these simply describe document structures, some supporting a wider range than others. Some of them, like Github-flavored Markdown or WikiMarkup have been adapted to support specific features of the systems they belong to such as wikiwords or GitHub issue numbers. Most express structure through a combination of whitespace and punctuation characters, unlike the named elements that express structure in an XML document.

Still others are designed for highly specialized purposes with very subject-specific semantics. Examples of these include JavaDoc markup, which is specialized for describing Java APIs and designed to be written inside of comments in Java source code. These work brilliantly for what they do, but they tend to result in API documentation being produced by an entirely different system than the rest of the docs, and without any connection between them. Which sucks.

Seeking a middle way

So, there are problems with both the generalized and the lightweight approaches. They both have aspects of them that suck, and in both cases, the suckiness is not the result of a specific design flaw that could be fixed, it is a consequence of a key feature of the approach itself.

Is there a middle way? I think there could be. We should not expect it to be a panacea that cures all ills. Whatever it is, it will have its own pockets of suckiness. The key will be to get enough things right that it meets the needs of a wide enough group of people at a price they are willing to pay. The only way to get there is going to be to try things and see what sticks.

I believe that a key component of such a middle way will be a structured extensible markup language that doesn’t suck for writing.

That language needs to have the simplicity and clarity of a lightweight markup language combined with the ability to add and enforce custom semantics through a schema or a similar mechanism.

It needs to be a markup language because we need structure for a wider range of applicability and automation and we have seen that WYSIWYG editors suck, in one way or another, for creating structure.

It needs to be extensible, because we need a language that can be used for more than one purpose, though we certainly don’t need it to be as general as SGML or XML. Something suitable for writing the kinds of documents that have paragraphs and code samples and quotations and lists will cover a wide range of needs.

Such a language does not have to be semantics-free. It can extend from a base set of semantics, rather than starting from scratch each time.

It needs to make structure clear and visible without it interfering with easily reading and writing the document.

Getting the balance right is going to be the trick. Particularly, how structure is expressed, and what types of out-of-band data are supported and how they are expressed, is going to be key to making it useful while not sucking to work in.

Some lightweight markup languages, like reStructuredText and AsciiDoc, already have extensibility features, though they rely more on programming than on a schema language for creating extensions.

Personally, I find both of them to rely too much on exotic use of punctuation for my taste. And that is very much the point. We are going to need something that appeals to a broad taste. Markdown seems to be more appealing than either AsciiDoc or reStructuredText, despite being less powerful than either.

I’m experimenting with my own approach to this problem with a  project I call SAM (Semantic Authoring Markdown). This post was written in SAM. I’ll be talking more about it in my series on structured writing on TechWhirl, and in my forthcoming book on Structured Writing from XML Press. Both the series and the book are being written in SAM.

SAM encapsulates my own particular views on how markup languages should be created and structured. It makes less use of exotic punctuation — and more use of whitespace — than other lightweight languages. I think it is more readable and writable than any other lightweight markup language, including Markdown, but that’s me. It remains to be seen what others will think. The language is a work in progress, but you can take a look at it in its current state on GitHub.

Unlike most other lightweight languages, SAM is not intended to be an entirely separate tool chain independent of XML. XML may suck as a format to write in, but it has a rich tool set and support network and SAM is designed to take full advantage of it by outputting XML rather than creating output formats directly. I think that is important. The middle way should not throw the baby out with the bathwater.

SPFE will support SAM.

Will SAM hit a sweet spot? The only way to find out is to put it out there and see what happens.

I know not everyone will agree, but we need to find something that sucks less than XML. The growing popularity of Markdown and other lightweight markup languages tells us that much.