I feel somewhat responsible for putting James Forrester on the hot seat concerning [this article][Visual Editor post] as I was the one who asked him to take out time from his busy schedule to explain some of the challenges faced by the team concerning the project and manage expectations somewhat concerning the release.

I hope it is clear from the post that the [VisualEditor project][] has to overcome many “firsts” to become a reality. Some of the ones mentioned include:

1. The criteria to support 290 languages is beyond the scope of support of existing software.
2. The VisualEditor UI needs to be programmable in such a manner that “free form” HTML editing cannot be permitted unless those edits can be synced with an internal, client-side “data model.”
3. The VisualEditor needs to leave room for extensibility in all fronts to adapt to the extensible nature of current WikiText capabilities like transclusions and templates.
4. A **two-way** representation needs to be created out of wikitext to HTML *back to wikitext*. This cannot be emphasized enough since before the [parsoid project][], it was actually unclear if this was even possible.
5. In both the parser and in all points of the VisualEditor, edits must be made in a manner that only manipulates the areas intended by the editor so was to not introduce “dirty diffs”.
6. All of this must be done in a forward thinking transactional manner to allow things like real-time collaboration, micro-edits, and the ability to walk through actions.

[Visual Editor post]: http://blog.wikimedia.org/2012/12/07/inventing-as-we-go-building-a-visual-editor-for-mediawiki/ “Inventing as we go: building a visual editor for MediaWiki—Wikimedia”
[VisualEditor project]: http://www.mediawiki.org/wiki/VisualEditor “VisualEditor—MediaWiki”
[parsoid project]: http://www.mediawiki.org/wiki/Parsoid “Parsoid—MediaWiki”Right now, [WordPress][] uses [TinyMCE][] as it’s visual editor. Currently, if you switched from visual to source mode and back, the HTML gets completely reformatted. On [MediaWiki][], which powers [Wikipedia][], this would introduce “dirty diffs” with by anyone who edits visually and saves. Looking at differences are how people maintaining the quality of Wikipedia ensure that bad changes don’t go through. If dirty diffs were introduced with a lot of syntactical changes that don’t make any semantic difference, it still would make patrolling these pages an insurmountable task to the current community.

Also, most extensions to WordPress involve adding [short tags][shortcode api] which are not visibly rendered when in visual mode. Right now, editing Wikipedia involves a large context switch from the rendered page to the mass of wikitext. This means a visual editor needs to render (and make editable) even those shortcuts to reduce the opportunity for [change blindness][].

As James mentioned, currently 80% of articles round-trip without any differences, an additional 18% with syntactic differences only, the final 2% with both syntactic and semantical differences. (The difference here is that syntactic changes do not affect the way the page is rendered in HTML, while semantic changes do.) Since the VisualEditor keeps any changed in the HTML in synchronization with the underlying data model, only the areas affected need merged back in. This means that the parser would only introduce syntactical changes in areas that have actually been edited by the person and that semantic changes would not be merged back in, since those areas are locked out to be edited visually anyway.

An example concerning future thinking: imagine if someone were to move a block of text from one area to another and then modify a sentence in said block subtlety in order to change its very meaning. Currently, the “diff” in the edit would look like a huge block of text were moved and the small change could be maliciously inserted. The way the VisualEditor works those are recorded as separate transactions. If UI could be presented and a model could store it on the server, a patroller could actually break into the diff and see it as two separate edits: one to move the text, and the other to modify it.

### Answering some questions posed ###

> According to the time table First English Wikipedia deployments and iteration should start now, by July 2013 Visual Editor should be enabled by default for (almost) every wikipedia/mediawiki instance. Any updates?

The planned release for the VisualEditor on English Wikipedia will be this [Tuesday, December 11th][ve release], in line with the [2012-13 goals document][2012-13 goals]. The purpose of the article was to note that challenges — some foreseen, others not — mitigate the scope of the release in order to hit this deadline. In particular the project will be:

* opt-in only. This means you must be logged-in, and turn the visual editor on to edit visually.
* be missing many important features: categories, templates, tables, references
* have some small rendering errors when editing (those areas shouldn’t be available)
* take longer to load a page for editing, or, in some cases, not allow editing at all (the new parser, because it lacks a cache and is not yet optimized, is slower than the current one)
* does not support IE or Opera.

As for the July 2013 deadline. As deadlines are further out it is harder to be optimistic about this. Given that goal was written in May of this year, it is difficult to predict with certainty at the moment. Some of this will be a function of the parsoid fixing rendering bugs and porting to C++ for speed (both faster language, and the ability to integrate directly with mediawiki instead of through the API). Most of it will be related to challenges in the visual editor:

* overcoming browser weaknesses (for instance, IE does not currently allow editable areas to be closed out from editing within an area that is editable which is the reason it is currently supported)
* how many bugs are found in the current visual editor
* the time it takes to extend support for complex content (references, templates, images, etc)

As with all things engineering, especially in an area as untested as the VisualEditor and the Parsoid, it is hard to make accurate estimations.

> The 2011/2012 targets were, “Develop Visual Editor. First opt-in user-facing production usage by December 2011, and first small wiki default deployment by June 2012.”

In 2011-12, the targets were developed before the challenges James mentioned were even realized. This was before James and I were working at the Foundation, but it’s actually because the 2012-13 goals were developed with some understanding of these challenges that it looks like the team been able to hit the deliverables so far. 🙂

> You say that this is a project undertaken in conjunction with Wikia. What part or proportion of the work is done by Wikia? What are the financial arrangements between Wikia and Wikimedia Foundation for that development work? How many staff are assigned to this in Wikia and in WMF?

The part of the work done by the Wikia team is currently the “rendering, selection and input (ve.ce)” area of the “module stack” architecture in James’s article. This consists of two full time engineers, provided to the foundation by Wikia.

There are currently no financial arrangements between Wikia and the Wikimedia Foundation for development work. Instead there is a commitment by them to provide those engineers for such “core work.” Recently, the VisualEditor has undergone a major refactoring in which all parts of this system is extensible through an API. This means in the near future, it is possible that Wikia will devote engineering resources on areas that are important to them, but not to us —an example might be extending the VE to allow YouTube video embeds. That would be taken by engineers outside of the core team.

The staff directly assigned to the VisualEditor on our end consists of four full time engineers and one part time engineer on the rest of the visual editor stack: Toolbar and inspectors (ve.ui) and Linear model and transaction systems (ve.dm) and three full time engineers on the parser and load/save API (parsoid), plus one full time product manager. Note: one of the full time engineering positions has not been filled as of this moment. I believe this sum (7+2 FTEs + 1 PTE + 1 PM) make this easily the largest single project resourced in the Foundation.

Note I have not counted the tremendous support from Operations and Platform that is provided, nor the work volunteered by other members of other teams at the WMF (I believe three engineers within Features alone have helped out at various times), nor the work done by our volunteer development community, including one summer Google Summer of Code student who has made the first forays into the Collaboration Server.

One small thing to note. This project has concentrated most of our best front-end engineering talent at the Foundation, as well as two of the top engineers at Wikia, furthermore the Parsoid in particular takes a unique engineering talent not typically found in the web world. From time to time, that talent needs to be made available to various other areas of our respective organizations to pay down technical debt, so occasional hiccups in effort have (and will continue) to occur. (As an example of this: one of my engineers once prefaced a request by saying, “I know this messes with the ‘dream team’ you’ve have in the Visual Editor, but…”)

> I don’t like that Wikipedia has such a close relationship with Wikia.

These concerns are noted. Right now, our goals and Wikia’s align with respect to the VisualEditor so it makes sense to work together. Since all the software produced by the teams are open-sourced and freely available, the open-source licensed nature of the project are designed to keep this relationship from being corrupted.

For instance, our goals and Kaltura’s and Google’s align with respect to WebM support in the TimedMediaHandler. Should we look the gift-horse in the mouth, simply because they might be commercial or commercially-driven entities?

Last year, around this time, the Wikia team working on things like FCKedit started experimenting in [contentEditable][]. As browser support for the contentEditable attribute increased, the work they did progressed further and faster than foundation work on the it’s own edit surface. It made sense to combine our efforts.

This combination is unlikely to be a problem in the short term. In the longer term if differences in our goals and Wikia’s appear where the likely solution is in the ui.ce part of the VisualEditor, we might have to devote resources to fix them on our end. This is not true for InternetExplorer support, but I can imagine this being the case because on Opera or some mobile browsers, as the Foundation mandate is much broader than that of Wikia’s.

> I would be interested to know how it’ll deal with Wikipedia’s existing microformats.

The engineers could speak directly to this, but I believe since [Microformats][wikipedia:microformats] are already in HTML, these are preserved by the parsoid. Right now, where the microformats are usually used (infoboxes) are not currently editable by the VisualEditor. This will have to be dealt with when the VE is extended to support it.

> I was disappointed, again, that there was no discussion of creating a Table namespace, as exists with some other wikis (I saw one demonstrated years ago).

Again, the engineers should speak directly to this. Tables are a challenge because the templates that create them do not generate self-closing nodes (see the parsed diagram: the tokens are generated into a HTML5DOM). However, the parsoid seems to support them already even if VisualEditor does not.

I am not familiar with the Table namespace, but I believe the idea was to abstract tables for transclusion in its own namespace. If such a thing existed, then it would make the parsoid much easier to have written, but not affect the challenges already presented to the VisualEditor. Since the parsoid already has overcome this hurdle, I don’t believe it makes a difference at this point.

[WordPress][]: http://www.wordpress.org/ “WordPress—Blog Tool, Publishing Platform, and CMS”
[TinyMCE]: http://www.tinymce.com “TinyMCE”
[Wikipedia]: http://en.wikipedia.org “Wikipedia—The Free Encyclopedia”
[MediaWiki]: http://mediawiki.org “MediaWiki”
[shortcode api]: http://codex.wordpress.org/Shortcode_API “The Shortcode API—WordPress codex”
[change blindness]: http://en.wikipedia.org/wiki/Change_blindness “Change blindness—Wikipedia”
[parsoid testing]: http://parsoid.wmflabs.org:8001 “Parsoid testing results”
[2012-13 goals]: https://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Visual_Editor “Wikimedia Engineering/2012-13 Goals—MediaWiki”
[ve release]: https://docs.google.com/presentation/d/1bsP3zJB-K4KSmMABD-BwJT9d9tMbvo5OujZBpWR94VM/edit?pli=1#slide=id.p28 “VisualEditor-Parsoid: slides for Metrics 2012-12-06”
[contentEditable]: http://www.whatwg.org/specs/web-apps/current-work/multipage/editing.html#contenteditable “Making document regions editable: The contenteditable content attribute—HTML5 standard”
[wikipedia:microformats]: http://en.wikipedia.org/wiki/Wikipedia:Microformats “Wikipedia:Microformats—Wikipedia”