For a language as widely used as JavaScript, any realistic change proposal must strongly consider backward compatibility. For this reason, some of the following proposals might not be particularly realistic, but nevertheless I think that a) it's worthwhile to consider what might change if backward compatibility wasn't a concern, and b) in the long run, all of these changes would improve the ease of use and predictability of how regular expressions work in JavaScript.

Remove RegExp.prototype.lastIndex and replace it with an argument for start position

Actual proposal: Deprecate RegExp.prototype.lastIndex and add a "pos" argument to the RegExp.prototype.exec/test methods

JavaScript's lastIndex property serves too many purposes at once:

It lets users manually specify where to start a regex search

You could claim this is not lastIndex's intended purpose, but it's nevertheless an important use since there's no alternative feature that allows this. lastIndex is not very good at this task, though. You need to compile your regex with the /g flag to let lastIndex be used this way; and even then, it only specifies the starting position for the regexp.exec/test methods. It cannot be used to set the start position for the string.match/replace/search/split methods.

It indicates the position where the last match ended

Even though you could derive the match end position by adding the match index and length, this use of lastIndex serves as a convenient and commonly used compliment to the index property on match arrays returned by exec. Like always, using lastIndex like this works only for regexes compiled with /g.

It's used to track the position where the next search should start

This comes into play, e.g., when using a regex to iterate over all matches in a string. However, the fact that lastIndex is actually set to the end position of the last match rather than the position where the next search should start (unlike equivalents in other programming languages) causes a problem after zero-length matches, which are easily possible with regexes like /\w*/g or /^/mg. Hence, you're forced to manually increment lastIndex in such cases. I've posted about this issue in more detail before (see: An IE lastIndex Bug with Zero-Length Regex Matches), as has Jan Goyvaerts (Watch Out for Zero-Length Matches).

Unfortunately, lastIndex's versatility results in it not working ideally for any specific use. I think lastIndex is misplaced anyway; if you need to store a search's ending (or next-start) position, it should be a property of the target string and not the regular expression. Here are three reasons this would work better:

• It would let you use the same regex with multiple strings, without losing track of the next search position within each one.
• It would allow using multiple regexes with the same string and having each one pick up from where the last one left off.
• If you search two strings with the same regex, you're probably not expecting the search within the second string to start from an arbitrary position just because a match was found in the first string.

In fact, Perl uses this approach of storing next-search positions with strings to great effect, and adds various features around it.

So that's my case for lastIndex being misplaced, but I go one further in that I don't think lastIndex should be included in JavaScript at all. Perl's tactic works well for Perl (especially when considered as a complete package), but some other languages (including Python) let you provide a search-start position as an argument when calling regex methods, which I think is an approach that is more natural and easier for developers to understand and use. I'd therefore fix lastIndex by getting rid of it completely. Regex methods and regex-using string methods would use internal search position trackers that are not observable by the user, and the exec and test methods would get a second argument (called pos, for position) that specifies where to start their search. It might be convenient to also give the String methods search, match, replace, and split their own pos arguments, but that is not as important and the functionality it would provide is not currently possible via lastIndex anyway.

Following are examples of how some common uses of lastIndex could be rewritten if these changes were made:

Start search from position 5, using lastIndex (the staus quo):

var regexGlobal = /\w+/g,
    result;

regexGlobal.lastIndex = 5;
result = regexGlobal.test(str);
// must reset lastIndex or future tests will continue from the
// match-end position (defensive coding)
regexGlobal.lastIndex = 0;

var regexNonglobal = /\w+/;

regexNonglobal.lastIndex = 5;
// no go - lastIndex will be ignored. instead, you have to do this
result = regexNonglobal.test(str.slice(5));

Start search from position 5, using pos:

var regex = /\w+/, // flag /g doesn't matter
    result = regex.test(str, 5);

Match iteration, using lastIndex:

var regex = /\w*/g,
    matches = [],
    match;

// the /g flag is required for this regex. if your code was provided a non-
// global regex, you'd need to recompile it with /g, and if it already had /g,
// you'd need to reset its lastIndex to 0 before entering the loop

while (match = regex.exec(str)) {
    matches.push(match);
    // avoid an infinite loop on zero-length matches
    if (regex.lastIndex == match.index) {
        regex.lastIndex++;
    }
}

Match iteration, using pos:

var regex = /\w*/, // flag /g doesn't matter
    pos = 0,
    matches = [],
    match;

while (match = regex.exec(str, pos)) {
    matches.push(match);
    pos = match.index + (match[0].length || 1);
}

Of course, you could easily add your own sugar to further simplify match iteration, or JavaScript could add a method dedicated to this purpose similar to Ruby's scan (although JavaScript already sort of has this via the use of replacement functions with string.replace).

To reiterate, I'm describing what I would do if backward compatibility was irrelevant. I don't think it would be a good idea to add a pos argument to the exec and test methods unless the lastIndex property was deprecated or removed, due to the functionality overlap. If a pos argument existed, people would expect pos to be 0 when it's not specified. Having lastIndex around to sometimes screw up this expectation would be confusing and probably lead to latent bugs. Hence, if lastIndex was deprecated in favor of pos, it should be a means toward the end of removing lastIndex altogether.

Remove String.prototype.match's nonglobal operating mode

Actual proposal: Deprecate String.prototype.match and add a new matchAll method

String.prototype.match currently works very differently depending on whether the /g (global) flag has been set on the provided regex:

• For regexes with /g: If no matches are found, null is returned; otherwise an array of simple matches is returned.
• For regexes without /g: The match method operates as an alias of regexp.exec. If a match is not found, null is returned; otherwise you get an array containing the (single) match in key zero, with any backreferences stored in the array's subsequent keys. The array is also assigned special index and input properties.

The match method's nonglobal mode is confusing and unnecessary. The reason it's unnecessary is obvious: If you want the functionality of exec, just use it (no need for an alias). It's confusing because, as described above, the match method's two modes return very different results. The difference is not merely whether you get one match or all matches—you get a completely different kind of result. And since the result is an array in either case, you have to know the status of the regex's global property to know which type of array you're dealing with.

I'd change string.match by making it always return an array containing all matches in the target string. I'd also make it return an empty array, rather than null, when no matches are found (an idea that comes from Dean Edwards's base2 library). If you want the first match only or you need backreferences and extra match details, that's what regexp.exec is for.

Unfortunately, if you want to consider this change as a realistic proposal, it would require some kind of language version- or mode-based switching of the match method's behavior (unlikely to happen, I would think). So, instead of that, I'd recommend deprecating the match method altogether in favor of a new method (perhaps RegExp.prototype.matchAll) with the changes prescribed above.

Get rid of /g and RegExp.prototype.global

Actual proposal: Deprecate /g and RegExp.prototype.global, and add a boolean replaceAll argument to String.prototype.replace

If the last two proposals were implemented and therefore regexp.lastIndex and string.match were things of the past (or string.match no longer sometimes served as an alias of regexp.exec), the only method where /g would still have any impact is string.replace. Additionally, although /g follows prior art from Perl, etc., it doesn't really make sense to have something that is not an attribute of a regex stored as a regex flag. Really, /g is more of a statement about how you want methods to apply their own functionality, and it's not uncommon to want to use the same pattern with and without /g (currently you'd have to construct two different regexes to do so). If it was up to me, I'd get rid of the /g flag and its corresponding global property, and instead simply give the string.replace method an additional argument that indicates whether you want to replace the first match only (the default handling) or all matches. This could be done with either a replaceAll boolean or, for greater readability, a scope string that accepts values 'one' and 'all'. This new argument would have the additional benefit of allowing replace-all functionality with nonregex searches.

Note that SpiderMonkey already has a proprietary third string.replace argument ("flags") that this proposal would conflict with. I doubt this conflict would cause much heartburn, but in any case, a new replaceAll argument would provide the same functionality that SpiderMonkey's flags argument is most useful for (that is, allowing global replacements with nonregex searches).

Change the behavior of backreferences to nonparticipating groups

Actual proposal: Make backreferences to nonparticipating groups fail to match

I'll keep this brief since David "liorean" Andersson and I have previously argued for this on ES-Discuss and elsewhere. David posted about this in detail on his blog (see: ECMAScript 3 Regular Expressions: A specification that doesn't make sense), and I've previously touched on it here (ECMAScript 3 Regular Expressions are Defective by Design). On several occasions, Brendan Eich has also stated that he'd like to see this changed. The short explanation of this behavior is that, in JavaScript, backreferences to capturing groups that have not (yet) participated in a match always succeed (i.e., they match the empty string), whereas the opposite is true in all other regex flavors: they fail to match and therefore cause the regex engine to backtrack or fail. JavaScript's behavior means that /(a|(b))\2c/.test("ac") returns true. The (negative) implications of this reach quite far when pushing the boundaries of regular expressions.

I think everyone agrees that changing to the traditional backreferencing behavior would be an improvement—it provides far more intuitive handling, compatibility with other regex flavors, and great potential for creative use (e.g., see my post on Mimicking Conditionals). The bigger question is whether it would be safe, in light of backward compatibility. I think it would be, since I imagine that more or less no one uses the unintuitive JavaScript behavior intentionally. The JavaScript behavior amounts to automatically adding a ? quantifier after backreferences to nonparticipating groups, which is what people already do explicitly if they actually want backreferences to nonzero-length subpatterns to be optional. Also note that Safari 3.0 and earlier did not follow the spec on this point and used the more intuitive behavior, although that has changed in more recent versions (notably, this change was due to a write up on my blog rather than reports of real-world errors).

Finally, it's probably worth noting that .NET's ECMAScript regex mode (enabled via the RegexOptions.ECMAScript flag) indeed switches .NET to ECMAScript's unconventional backreferencing behavior.

Make \d \D \w \W \b \B support Unicode (like \s \S . ^ $, which already do)

Actual proposal: Add a /u flag (and corresponding RegExp.prototype.unicode property) that changes the meaning of \d, \w, \b, and related tokens

Unicode-aware digit and word character matching is not an existing JavaScript capability (short of constructing character class monstrosities that are hundreds or thousands of characters long), and since JavaScript lacks lookbehind you can't reproduce a Unicode-aware word boundary. You could therefore say this proposal is outside the stated scope of this post, but I'm including it here because I consider this more of a fix than a new feature.

According to current JavaScript standards, \s, \S, ., ^, and $ use Unicode-based interpretations of whitespace and newline, whereas \d, \D, \w, \W, \b, and \B use ASCII-only interpretations of digit, word character, and word boundary (e.g., /na\b/.test("naïve") unfortunately returns true). See my post on JavaScript, Regex, and Unicode for further details. Adding Unicode support to these tokens would cause unexpected behavior for thousands of websites, but it could be implemented safely via a new /u flag (inspired by Python's re.U or re.UNICODE flag) and a corresponding RegExp.prototype.unicode property. Since it's actually fairly common to not want these tokens to be Unicode enabled in particular regex patterns, a new flag that activates Unicode support would offer the best of both worlds.

Change the behavior of backreference resetting during subpattern repetition

Actual proposal: Never reset backreference values during a match

Like the last backreferencing issue, this too was covered by David Andersson in his post ECMAScript 3 Regular Expressions: A specification that doesn't make sense. The issue here involves the value remembered by capturing groups nested within a quantified, outer group (e.g., /((a)|(b))*/). According to traditional behavior, the value remembered by a capturing group within a quantified grouping is whatever the group matched the last time it participated in the match. So, the value of $1 after /(?:(a)|(b))*/ is used to match "ab" would be "a". However, according to ES3/ES5, the value of backreferences to nested groupings is reset/erased after the outer grouping is repeated. Hence, /(?:(a)|(b))*/ would still match "ab", but after the match is complete $1 would reference a nonparticipating capturing group, which in JavaScript would match an empty string within the regex itself, and be returned as undefined in, e.g., the array returned by the regexp.exec.

My case for change is that current JavaScript behavior breaks from the norm in other regex flavors, does not lend itself to various types of creative patterns (see one example in my post on Capturing Multiple, Optional HTML Attribute Values), and in my opinion is far less intuitive than the more common, alternative regex behavior.

I believe this behavior is safe to change for two reasons. First, this is generally an edge case issue for all but hardcore regex wizards, and I'd be surprised to find regexes that rely on JavaScript's version of this behavior. Second, and more importantly, Internet Explorer does not implement this rule and follows the more traditional behavior.

Add an /s flag, already

Actual proposal: Add an /s flag (and corresponding RegExp.prototype.dotall property) that changes dot to match all characters including newlines

I'll sneak this one in as a change/fix rather than a new feature since it's not exactly difficult to use [\s\S] in place of a dot when you want the behavior of /s. I presume the /s flag has been excluded thus far to save novices from themselves and limit the damage of runaway backtracking, but what ends up happening is that people write horrifically inefficient patterns like (.|\r|\n)* instead.

Regex searches in JavaScript are seldom line-based, and it's therefore more common to want dot to include newlines than to match anything-but-newlines (although both modes are useful). It makes good sense to keep the default meaning of dot (no newlines) since it is shared by other regex flavors and required for backward compatibility, but adding support for the /s flag is overdue. A boolean indicating whether this flag was set should show up on regexes as a property named either singleline (the unfortunate name from Perl, .NET, etc.) or the more descriptive dotall (used in Java, Python, PCRE, etc.).

Personal preferences

Following are a few changes that would suit my preferences, although I don't think most people would consider them significant issues:

• Allow regex literals to use unescaped forward slashes within character clases (e.g., /[/]/). This was already included in the abandoned ES4 change proposals.
• Allow an unescaped ] as the first character in character classes (e.g., []] or [^]]). This is allowed in probably every other regex flavor, but creates an empty class followed by a literal ] in JavaScript. I'd like to imagine that no one uses empty classes intentionally, since they don't work consistently cross-browser and there are widely-used/common-sense alternatives ((?!) instead of [], and [\s\S] instead of [^]). Unfortunately, adherence to this JavaScript quirk is tested in Acid3 (test 89), which is likely enough to kill requests for this backward-incompatible but reasonable change.
• Change the $& token used in replacement strings to $0. It just makes sense. (Equivalents in other replacement text flavors for comparison: Perl: $&; Java: $0; .NET: $0, $&; PHP: $0, \0; Ruby: \0, \&; Python: \g<0>.)
• Get rid of the special meaning of [\b]. Within character classes, the metasequence \b matches a backspace character (equivalent to \x08). This is a worthless convenience since no one cares about matching backspace characters, and it's confusing given that \b matches a word boundary when used outside of character classes. Even though this would break from regex tradition (which I'd usually advocate following), I think that \b should have no special meaning inside character classes and simply match a literal b.

Fixed in ES3: Remove octal character references

ECMAScript 3 removed octal character references from regular expression syntax, although \0 was kept as a convenient exception that allows easily matching a NUL character. However, browsers have generally kept full octal support around for backward compatibility. Octals are very confusing in regular expressions since their syntax overlaps with backreferences and an extra leading zero is allowed outside of character classes. Consider the following regexes:

• /a\1/: \1 is an octal.
• /(a)\1/: \1 is a backreference.
• /(a)[\1]/: \1 is an octal.
• /(a)\1\2/: \1 is a backreference; \2 is an octal.
• /(a)\01\001[\01\001]/: All occurences of \01 and \001 are octals. However, according to the ES3+ specs, the numbers after each \0 should be treated (barring nonstandard extensions) as literal characters, completely changing what this regex matches. (Edit-2012: Actually, a close reading of the spec shows that any 0-9 following \0 should cause a SyntaxError.)
• /(a)\0001[\0001]/: The \0001 outside the character class is an octal; but inside, the octal ends at the third zero (i.e., the character class matches character index zero or "1"). This regex is therefore equivalent to /(a)\x01[\x00\x31]/; although, as mentioned just above, adherence to ES3 would change the meaning.
• /(a)\00001[\00001]/: Outside the character class, the octal ends at the fourth zero and is followed by a literal "1". Inside, the octal ends at the third zero and is followed by a literal "01". And once again, ES3's exclusion of octals and inclusion of \0 could change the meaning.
• /\1(a)/: Given that, in JavaScript, backreferences to capturing groups that have not (yet) participated match the empty string, does this regex match "a" (i.e., \1 is treated as a backreference since a corresponding capturing group appears in the regex) or does it match "\x01a" (i.e., the \1 is treated as an octal since it appears before its corresponding group)? Unsurprisingly, browsers disagree.
• /(\2(a)){2}/: Now things get really hairy. Does this regex match "aa", "aaa", "\x02aaa", "2aaa", "\x02a\x02a", or "2a2a"? All of these options seem plausible, and browsers disagree on the correct choice.

There are other issues to worry about, too, like whether octal escapes go up to \377 (\xFF, 8-bit) or \777 (\u01FF, 9-bit); but in any case, octals in regular expressions are a confusing cluster-cuss. Even though ECMAScript has already cleaned up this mess by removing support for octals, browsers have not followed suit. I wish they would, because unlike browser makers, I don't have to worry about this bit of legacy (I never use octals in regular expressions, and neither should you).

Fixed in ES5: Don't cache regex literals

According to ES3 rules, regex literals did not create a new regex object if a literal with the same pattern/flag combination was already used in the same script or function (this did not apply to regexes created by the RegExp constructor). A common side effect of this was that regex literals using the /g flag did not have their lastIndex property reset in some cases where most developers would expect it. Several browsers didn't follow the spec on this unintuitive behavior, but Firefox did, and as a result it became the second most duplicated JavaScript bug report for Mozilla. Fortunately, ES5 got rid of this rule, and now regex literals must be recompiled every time they're encountered (this change is coming in Firefox 3.7).

———
So there you have it. I've outlined what I think the JavaScript RegExp API got wrong. Do you agree with all of these proposals, or would you if you didn't have to worry about backward compatibility? Are there better ways than what I've proposed to fix the issues discussed here? Got any other gripes with existing JavaScript regex features? I'm eager to hear feedback about this.

Since I've been focusing on the negative in this post, I'll note that I find working with regular expressions in JavaScript to be a generally pleasant experience. There's a hell of a lot that JavaScript got right.

First of all, if you're not already familiar with how to use exec to iterate over a string, you're missing out on some very powerful functionality. Here's the basic construct:

var	regex = /.../g,
	subject = "test",
	match = regex.exec(subject);

while (match != null) {
	// matched text: match[0]
	// match start: match.index
	// match end: regex.lastIndex
	// capturing group n: match[n]

	...

	match = regex.exec(subject);
}

When the exec method is called for a regex that uses the /g (global) modifier, it searches from the point in the subject string specified by the regex's lastIndex property (which is initially zero, so it searches from the beginning of the string). If the exec method finds a match, it updates the regex's lastIndex property to the character index at the end of the match, and returns an array containing the matched text and any captured subexpressions. If there is no match from the point in the string where the search started, lastIndex is reset to zero, and null is returned.

You can tighten up the above code by moving the exec method call into the while loop's condition, like so:

var	regex = /.../g,
	subject = "test",
	match;

while (match = regex.exec(subject)) {
	...
}

This cleaner version works essentially the same as before. As soon as exec can't find any further matches and therefore returns null, the loop ends. However, there are a couple cross-browser issues to be aware of with either version of this code. One is that if the regex contains capturing groups which do not participate in the match, some values in the returned array could be either undefined or an empty string. I've previously discussed that issue in depth in a post about what I called non-participating capturing groups.

Another issue (the topic of this post) occurs when your regex matches an empty string. There are many reasons why you might allow a regex to do that, but if you can't think of any, consider cases where you're accepting regexes from an outside source. Here's a simple example of such a regex:

var	regex = /^/gm,
	subject = "A\nB\nC",
	match,
	endPositions = [];

while (match = regex.exec(subject)) {
	endPositions.push(regex.lastIndex);
}

You might expect the endPositions array to be set to [0,2,4], since those are the character positions for the beginning of the string and just after each newline character. Thanks to the /m modifier, those are the positions where the regex will match; and since the regex matches empty strings, regex.lastIndex should be the same as match.index. However, Internet Explorer (tested with v5.5–7) sets endPositions to [1,3,5]. Other browsers will go into an infinite loop until you short-circuit the code.

So what's going on here? Remember that every time exec runs, it attempts to match within the subject string starting at the position specified by the lastIndex property of the regex. Since our regex matches a zero-length string, lastIndex remains exactly where we started the search. Therefore, every time through the loop our regex will match at the same position—the start of the string. Internet Explorer tries to be helpful and avoid this situation by automatically incrementing lastIndex when a zero-length string is matched. That might seem like a good idea (in fact, I've seen people adamantly argue that is a bug that Firefox does not do the same), but it means that in Internet Explorer the lastIndex property cannot be relied on to accurately determine the ending position of a match.

We can correct this situation cross-browser with the following code:

var	regex = /^/gm,
	subject = "A\nB\nC",
	match,
	endPositions = [];

while (match = regex.exec(subject)) {
	var zeroLengthMatch = !match[0].length;
	// Fix IE's incorrect lastIndex
	if (zeroLengthMatch && regex.lastIndex > match.index)
		regex.lastIndex--;

	endPositions.push(regex.lastIndex);

	// Avoid an infinite loop with zero-length matches
	if (zeroLengthMatch)
		regex.lastIndex++;
}

You can see an example of the above code in the cross-browser split method I posted a while back. Keep in mind that none of the extra code here is needed if your regex cannot possibly match an empty string.

Another way to deal with this issue is to use String.prototype.replace to iterate over the subject string. The replace method moves forward automatically after zero-length matches, avoiding this issue altogether. Unfortunately, in the three biggest browsers (IE, Firefox, Safari), replace doesn't seem to deal with the lastIndex property except to reset it to zero. Opera gets it right (according to my reading of the spec) and updates lastIndex along the way. Given the current situation, you can't rely on lastIndex in your code when iterating over a string using replace, but you can still easily derive the value for the end of each match. Here's an example:

var	regex = /^/gm,
	subject = "A\nB\nC",
	endPositions = [];

subject.replace(regex, function (match) {
	// Not using a named argument for the index since capturing
	// groups can change its position in the list of arguments
	var	index = arguments[arguments.length - 2],
		lastIndex = index + match.length;

	endPositions.push(lastIndex);
});

That's perhaps less lucid than before (since we're not actually replacing anything), but there you have it… two cross-browser ways to get around a little-known issue that could otherwise cause tricky, latent bugs in your code.

/(?=a?b)ab/.test("ab");
// Should return true, but IE 5.5 – 8b1 return false

/(?=a?b)ab/.test("abc");
// Correctly returns true (even in IE), although the
// added "c" does not take part in the match

I've been aware of this bug for a couple years, thanks to a blog post by Michael Ash that describes the bug with a password-complexity regex. However, the bug description there is incomplete and subtly incorrect, as shown by the above, reduced test case. To be honest, although the errant behavior is predictable, it's a bit tricky to describe because I haven't yet figured out exactly what's happening internally. I'd recommend playing with variations of the above code to get a better understanding of the problem.

Fortunately, since the bug is predictable, it's usually possible to work around. For example, you can avoid the bug with the password regex in Michael's post (/^(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{8,15}$/) by writing it as /^(?=.{8,15}$)(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).*/ (the .{8,15}$ lookahead must come first here). The important thing is to be aware of the issue, because it can easily introduce latent and difficult to diagnose bugs into your code. Just remember that it shows up with variable-length lookahead. If you're using such patterns, test the hell out of them in IE.

According to ECMA-262 3rd Edition, \s, \S, ., ^, and $ use Unicode-based interpretations of whitespace and newline, while \d, \D, \w, \W, \b, and \B use ASCII-only interpretations of digit, word character, and word boundary (e.g. /a\b/.test("naïve") returns true). Actual browser implementations often differ on these points. For example, Firefox 2 considers \d and \D to be Unicode-aware, while Firefox 3 fixes this bug — making \d equivalent to [0-9] as with most other browsers.

Here again are the affected tokens, along with their definitions:

• \d — Digits.
• \s — Whitespace.
• \w — Word characters.
• \D — All except digits.
• \S — All except whitespace.
• \W — All except word characters.
• . — All except newlines.
• ^ (with /m) — The positions at the beginning of the string and just after newlines.
• $ (with /m) — The positions at the end of the string and just before newlines.
• \b — Word boundary positions.
• \B — Not word boundary positions.

All of the above are standard in Perl-derivative regex flavors. However, the meaning of the terms digit, whitespace, word character, word boundary, and newline depend on the regex flavor, character set, and platform you're using, so here are the official JavaScript meanings as they apply to regexes:

• Digit — The characters 0-9 only.
• Whitespace — Tab, line feed, vertical tab, form feed, carriage return, space, no-break space, line separator, paragraph separator, and "any other Unicode 'space separator'".
• Word character — The characters A-Z, a-z, 0-9, and _ only.
• Word boundary — The position between a word character and non-word character.
• Newline — The line feed, carriage return, line separator, and paragraph separator characters.

Here again are the newline characters, with their character codes:

• \u000a — Line feed — \n
• \u000d — Carriage return — \r
• \u2028 — Line separator
• \u2029 — Paragraph separator

Note that ECMAScript 4 proposals indicate that the C1/Unicode NEL "next line" control character (\u0085) will be recognized as an additional newline character in that standard. Also note that although CRLF (a carriage return followed by a line feed) is treated as a single newline sequence in most contexts, /\r^$\n/m.test("\r\n") returns true.

As for whitespace, ECMA-262 3rd Edition uses an interpretation based on Unicode's Basic Multilingual Plane, from version 2.1 or later of the Unicode standard. Following are the characters which should be matched by \s according to ECMA-262 3rd Edition and Unicode 5.1:

• \u0009 — Tab — \t
• \u000a — Line feed — \n — (newline character)
• \u000b — Vertical tab — \v
• \u000c — Form feed — \f
• \u000d — Carriage return — \r — (newline character)
• \u0020 — Space
• \u00a0 — No-break space
• \u1680 — Ogham space mark
• \u180e — Mongolian vowel separator
• \u2000 — En quad
• \u2001 — Em quad
• \u2002 — En space
• \u2003 — Em space
• \u2004 — Three-per-em space
• \u2005 — Four-per-em space
• \u2006 — Six-per-em space
• \u2007 — Figure space
• \u2008 — Punctuation space
• \u2009 — Thin space
• \u200a — Hair space
• \u2028 — Line separator — (newline character)
• \u2029 — Paragraph separator — (newline character)
• \u202f — Narrow no-break space
• \u205f — Medium mathematical space
• \u3000 — Ideographic space

To test which characters or positions are matched by all of the tokens mentioned here in your browser, see JavaScript Regex and Unicode Tests. Note that Firefox 2.0.0.11, IE 7, and Safari 3.0.3 beta all get some of the tests wrong.

In some browsers (most notably, Firefox), although innerHTML is generally much faster than DOM methods, it spends a disproportionate amount of time clearing out existing elements vs. creating new ones. Knowing this, we can combine the speed of destroying elements by removing their parent using the standard DOM methods with creating new elements using innerHTML. (This technique is something I discovered during the development of RegexPal, and is one of its two main performance optimizations. The other is one-shot markup generation for match highlighting, which avoids needing to loop over matches or reference them individually.)

The code:

function replaceHtml(el, html) {
	var oldEl = typeof el === "string" ? document.getElementById(el) : el;
	/*@cc_on // Pure innerHTML is slightly faster in IE
		oldEl.innerHTML = html;
		return oldEl;
	@*/
	var newEl = oldEl.cloneNode(false);
	newEl.innerHTML = html;
	oldEl.parentNode.replaceChild(newEl, oldEl);
	/* Since we just removed the old element from the DOM, return a reference
	to the new element, which can be used to restore variable references. */
	return newEl;
};

You can use the above as el = replaceHtml(el, newHtml) instead of el.innerHTML = newHtml.

innerHTML is already pretty fast...is this really warranted?

That depends on how many elements you're overwriting. In RegexPal, every keydown event potentially triggers the destruction and creation of thousands of elements (in order to make the syntax and match highlighting work). In such cases, the above approach has enormous positive impact. Even something as simple as el.innerHTML += str or el.innerHTML = "" could be a performance disaster if the element you're updating happens to have a few thousand children.

I've created a page which allows you to easily test the performance difference of innerHTML and my replaceHtml function with various numbers of elements. Make sure to try it out in a few browsers for comparison. Following are a couple examples of typical results from Firefox 2.0.0.6 on my system:

1000 elements...
innerHTML (destroy only): 156ms
innerHTML (create only): 15ms
innerHTML (destroy & create): 172ms
replaceHtml (destroy only): 0ms (faster)
replaceHtml (create only): 15ms (~ same speed)
replaceHtml (destroy & create): 15ms (11.5x faster)

15000 elements...
innerHTML (destroy only): 14703ms
innerHTML (create only): 250ms
innerHTML (destroy & create): 14922ms
replaceHtml (destroy only): 31ms (474.3x faster)
replaceHtml (create only): 250ms (~ same speed)
replaceHtml (destroy & create): 297ms (50.2x faster)

I think the numbers speak for themselves. Comparable performance improvements can also be seen in Safari. In Opera, replaceHtml is still typically faster than innerHTML, but by a narrower margin. In IE, simple use of innerHTML is typically faster than mixing it with DOM methods, but not by nearly the same kinds of margins as you can see above. Nevertheless, IE's conditional compilation feature is used to avoid the relatively minor performance penalty, by just using innerHTML with that browser.