parseUri 1.2: Split URLs in JavaScript

I've just updated parseUri. If you haven't seen the older version, parseUri is a function which splits any well-formed URI into its parts, all of which are optional. Its combination of accuracy, flexibility, and brevity is unrivaled.

Highlights:

  • Comprehensively splits URIs, including splitting the query string into key/value pairs. (Enhanced)
  • Two parsing modes: loose and strict. (New)
  • Easy to use (returns an object, so you can do, e.g., parseUri(uri).anchor).
  • Offers convenient, pre-concatenated components (path = directory and file; authority = userInfo, host, and port; etc.)
  • Change the default names of URI parts without editing the function, by updating parseUri.options.key. (New)
  • Exceptionally lightweight (1 KB before minification or gzipping).
  • Released under the MIT License.

Details:

Older versions of this function used what's now called loose parsing mode (which is still the default in this version). Loose mode deviates slightly from the official generic URI spec (RFC 3986), but by doing so allows the function to split URIs in a way that most end users would expect intuitively. However, the finer details of loose mode preclude it from properly handling relative paths which do not start from root (e.g., "../file.html" or "dir/file.html"). On the other hand, strict mode attempts to split URIs according to RFC 3986. Specifically, in loose mode, directories don't need to end with a slash (e.g., the "dir" in "/dir?query" is treated as a directory rather than a file name), and the URI can start with an authority without being preceded by "//" (which means that the "yahoo.com" in "yahoo.com/search/" is treated as the host, rather than part of the directory path).

Since I've assumed that most developers will consistently want to use one mode or the other, the parsing mode is not specified as an argument when running parseUri, but rather as a property of the parseUri function itself. Simply run the following line of code to switch to strict mode:

parseUri.options.strictMode = true;

From that point forward, parseUri will work in strict mode (until you turn it back off).

The code:

// parseUri 1.2.2
// (c) Steven Levithan <stevenlevithan.com>
// MIT License

function parseUri (str) {
	var	o   = parseUri.options,
		m   = o.parser[o.strictMode ? "strict" : "loose"].exec(str),
		uri = {},
		i   = 14;

	while (i--) uri[o.key[i]] = m[i] || "";

	uri[o.q.name] = {};
	uri[o.key[12]].replace(o.q.parser, function ($0, $1, $2) {
		if ($1) uri[o.q.name][$1] = $2;
	});

	return uri;
};

parseUri.options = {
	strictMode: false,
	key: ["source","protocol","authority","userInfo","user","password","host","port","relative","path","directory","file","query","anchor"],
	q:   {
		name:   "queryKey",
		parser: /(?:^|&)([^&=]*)=?([^&]*)/g
	},
	parser: {
		strict: /^(?:([^:\/?#]+):)?(?:\/\/((?:(([^:@]*)(?::([^:@]*))?)?@)?([^:\/?#]*)(?::(\d*))?))?((((?:[^?#\/]*\/)*)([^?#]*))(?:\?([^#]*))?(?:#(.*))?)/,
		loose:  /^(?:(?![^:@]+:[^:@\/]*@)([^:\/?#.]+):)?(?:\/\/)?((?:(([^:@]*)(?::([^:@]*))?)?@)?([^:\/?#]*)(?::(\d*))?)(((\/(?:[^?#](?![^?#\/]*\.[^?#\/.]+(?:[?#]|$)))*\/?)?([^?#\/]*))(?:\?([^#]*))?(?:#(.*))?)/
	}
};

You can download it here.

parseUri has no dependencies, and has been tested in IE 5.5–7, Firefox 2.0.0.4, Opera 9.21, Safari 3.0.1 beta for Windows, and Swift 0.2.

Levels of JavaScript Regex Knowledge

  1. N00b
    • Thinks "regular expressions" is open mic night at a poetry bar.
    • Uses \w, \d, \s, and other shorthand classes purely by accident if at all.
    • Painfully misuses * and especially .*.
    • Puts words in character classes.
    • Uses | in character classes for alternation.
    • Hasn't heard of the exec method.
    • Copies and pastes poorly written regexes from the web.
  2. Trained n00b
    • Uses regexes where methods like slice or indexOf would do.
    • Uses g, i, and m modifiers needlessly.
    • Uses [^\w] instead of \W.
    • Doesn't know why using [\w\d_] gives away their n00bness.
    • Tries to remove HTML tags with replace(/<.*?>/g,"").
    • Escapes all punctuation\!
  3. User
    • Knows when to use regexes, and when to use string methods.
    • Toys with lookahead.
    • Uses regexes in conditionals.
    • Starts to understand why HTML tags are hard to match with regexes.
    • Knows to use (?:…) when a backreference or capture isn't needed.
    • Can read a relatively simple regex and explain its function.
    • Knows their way around the use of replace callback functions.
  4. Haxz0r
    • Uses lookahead with impunity.
    • Sighs at the unavailability of lookbehind and other features from more powerful regex libraries.
    • Knows what $`, $', and $& mean in a replacement string.
    • Knows the difference between string literal and regex metacharacters, and how this impacts the RegExp constructor.
    • Generally knows whether a greedy or lazy quantifier is more appropriate, even when it doesn't change what the regex matches.
    • Has a basic sense of how to avoid regex efficiency problems.
    • Knows how to iterate over strings using the exec method and a while loop.
    • Knows that properties of the global RegExp object and the compile method are deprecated.
  5. Guru
    • Understands the significance of manually modifying a regex object's lastIndex property and when this can be useful within a loop.
    • Can explain how any given regex will or won't work.
    • No longer experiences the excitement of writing complex regexes that work on the first try, since regex behavior has become predictable and obvious.
    • Is immune to catastrophic backtracking, and can easily (and accurately) determine if a nested quantifier is safe.
    • Knows of numerous cross-browser regex syntax and behavior differences.
    • Knows offhand the section number of ECMA-262 3rd Edition that covers regexes.
    • Understands the difference between capturing group nonparticipation vs participating but capturing an empty string, and the behavior differences this can lead to.
    • Has a preference for particular backreference rules related to capturing group participation and quantified alternation, or is at least aware of the implementation inconsistencies.
    • Often knows which browser will run a given regex fastest before testing, based on known internal optimizations and weaknesses.
    • Thinks that writing recursive regexes is easy, so long as there is an upper bound to recursion depth.
  6. Wizard
    • Works on a regex engine.
    • Has patched the engine from time to time.
  7. God
    • Can add features to the engine at a whim.
    • Also created all life on earth using a constructor function.

(Heavily adapted and JavaScriptized from 7 Stages of a [Perl] Regex User.)

JavaScript split Bugs: Fixed!

The String.prototype.split method is very handy, so it's a shame that if you use a regular expression as its delimiter, the results can be so wildly different cross-browser that odds are you've just introduced bugs into your code (unless you know precisely what kind of data you're working with and are able to avoid the issues). Here's one example of other people venting about the problems. Following are the inconsistencies cross-browser when using regexes with split:

  • Internet Explorer excludes almost all empty values from the resulting array (e.g., when two delimiters appear next to each other in the data, or when a delimiter appears at the start or end of the data). This doesn't make any sense to me, since IE does include empty values when using a string as the delimiter.
  • Internet Explorer and Safari do not splice the values of capturing parentheses into the returned array (this functionality can be useful with simple parsers, etc.)
  • Firefox does not splice undefined values into the returned array as the result of non-participating capturing groups.
  • Internet Explorer, Firefox, and Safari have various additional edge-case bugs where they do not follow the split specification (which is actually quite complex).

The situation is so bad that I've simply avoided using regex-based splitting in the past.

That ends now. wink

The following script provides a fast, uniform cross-browser implementation of String.prototype.split, and attempts to precisely follow the relevant spec (ECMA-262 v3 §15.5.4.14, pp.103,104).

I've also created a fairly quick and dirty page where you can test the result of more than 50 usages of JavaScript's split method, and quickly compare your browser's results with the correct implementation. On the test page, the pink lines in the third column highlight incorrect results from the native split method. The rightmost column shows the results of the below script. It's all green in every browser I've tested (IE 5.5 – 7, Firefox 2.0.0.4, Opera 9.21, Safari 3.0.1 beta, and Swift 0.2).

Run the tests in your browser.

Here's the script:

/*!
 * Cross-Browser Split 1.1.1
 * Copyright 2007-2012 Steven Levithan <stevenlevithan.com>
 * Available under the MIT License
 * ECMAScript compliant, uniform cross-browser split method
 */

/**
 * Splits a string into an array of strings using a regex or string separator. Matches of the
 * separator are not included in the result array. However, if `separator` is a regex that contains
 * capturing groups, backreferences are spliced into the result each time `separator` is matched.
 * Fixes browser bugs compared to the native `String.prototype.split` and can be used reliably
 * cross-browser.
 * @param {String} str String to split.
 * @param {RegExp|String} separator Regex or string to use for separating the string.
 * @param {Number} [limit] Maximum number of items to include in the result array.
 * @returns {Array} Array of substrings.
 * @example
 *
 * // Basic use
 * split('a b c d', ' ');
 * // -> ['a', 'b', 'c', 'd']
 *
 * // With limit
 * split('a b c d', ' ', 2);
 * // -> ['a', 'b']
 *
 * // Backreferences in result array
 * split('..word1 word2..', /([a-z]+)(\d+)/i);
 * // -> ['..', 'word', '1', ' ', 'word', '2', '..']
 */
var split;

// Avoid running twice; that would break the `nativeSplit` reference
split = split || function (undef) {

    var nativeSplit = String.prototype.split,
        compliantExecNpcg = /()??/.exec("")[1] === undef, // NPCG: nonparticipating capturing group
        self;

    self = function (str, separator, limit) {
        // If `separator` is not a regex, use `nativeSplit`
        if (Object.prototype.toString.call(separator) !== "[object RegExp]") {
            return nativeSplit.call(str, separator, limit);
        }
        var output = [],
            flags = (separator.ignoreCase ? "i" : "") +
                    (separator.multiline  ? "m" : "") +
                    (separator.extended   ? "x" : "") + // Proposed for ES6
                    (separator.sticky     ? "y" : ""), // Firefox 3+
            lastLastIndex = 0,
            // Make `global` and avoid `lastIndex` issues by working with a copy
            separator = new RegExp(separator.source, flags + "g"),
            separator2, match, lastIndex, lastLength;
        str += ""; // Type-convert
        if (!compliantExecNpcg) {
            // Doesn't need flags gy, but they don't hurt
            separator2 = new RegExp("^" + separator.source + "$(?!\\s)", flags);
        }
        /* Values for `limit`, per the spec:
         * If undefined: 4294967295 // Math.pow(2, 32) - 1
         * If 0, Infinity, or NaN: 0
         * If positive number: limit = Math.floor(limit); if (limit > 4294967295) limit -= 4294967296;
         * If negative number: 4294967296 - Math.floor(Math.abs(limit))
         * If other: Type-convert, then use the above rules
         */
        limit = limit === undef ?
            -1 >>> 0 : // Math.pow(2, 32) - 1
            limit >>> 0; // ToUint32(limit)
        while (match = separator.exec(str)) {
            // `separator.lastIndex` is not reliable cross-browser
            lastIndex = match.index + match[0].length;
            if (lastIndex > lastLastIndex) {
                output.push(str.slice(lastLastIndex, match.index));
                // Fix browsers whose `exec` methods don't consistently return `undefined` for
                // nonparticipating capturing groups
                if (!compliantExecNpcg && match.length > 1) {
                    match[0].replace(separator2, function () {
                        for (var i = 1; i < arguments.length - 2; i++) {
                            if (arguments[i] === undef) {
                                match[i] = undef;
                            }
                        }
                    });
                }
                if (match.length > 1 && match.index < str.length) {
                    Array.prototype.push.apply(output, match.slice(1));
                }
                lastLength = match[0].length;
                lastLastIndex = lastIndex;
                if (output.length >= limit) {
                    break;
                }
            }
            if (separator.lastIndex === match.index) {
                separator.lastIndex++; // Avoid an infinite loop
            }
        }
        if (lastLastIndex === str.length) {
            if (lastLength || !separator.test("")) {
                output.push("");
            }
        } else {
            output.push(str.slice(lastLastIndex));
        }
        return output.length > limit ? output.slice(0, limit) : output;
    };

    // For convenience
    String.prototype.split = function (separator, limit) {
        return self(this, separator, limit);
    };

    return self;

}();

Download it.

Please let me know if you find any problems. Thanks!

Update: This script has become part of my XRegExp library, which includes many other JavaScript regular expression cross-browser compatibility fixes.

Mimicking Lookbehind in JavaScript

Unlike lookaheads, JavaScript doesn't support regex lookbehind syntax. That's unfortunate, but I'm not content with just resigning to that fact. Following are three ways I've come up with to mimic lookbehinds in JavaScript.

For those not familar with the concept of lookbehinds, they are zero-width assertions which, like the more specific \b, ^, and $ metacharacters, don't actually consume anything — they just match a position within text. This can be a very powerful concept. Read this first if you need more details.

Mimicking lookbehind with the replace method and optional capturing groups

This first approach is not much like a real lookbehind, but it might be "good enough" in some simple cases. Here are a few examples:

// Mimic leading, positive lookbehind like replace(/(?<=es)t/g, 'x')
var output = 'testt'.replace(/(es)?t/g, function($0, $1){
	return $1 ? $1 + 'x' : $0;
});
// output: tesxt

// Mimic leading, negative lookbehind like replace(/(?<!es)t/g, 'x')
var output = 'testt'.replace(/(es)?t/g, function($0, $1){
	return $1 ? $0 : 'x';
});
// output: xestx

// Mimic inner, positive lookbehind like replace(/\w(?<=s)t/g, 'x')
var output = 'testt'.replace(/(?:(s)|\w)t/g, function($0, $1){
	return $1 ? 'x' : $0;
});
// output: text

Unfortunately, there are many cases where lookbehinds can't be mimicked using this construct. Here's one example:

// Trying to mimic positive lookbehind, but this doesn't work
var output = 'ttttt'.replace(/(t)?t/g, function($0, $1){
	return $1 ? $1 + 'x' : $0;
});
// output: txtxt
// desired output: txxxx

The problem is that the regexes are relying on actually consuming the characters which should be within zero-width lookbehind assertions, then simply putting back the match unviolated (an effective no-op) if the backreferences contain or don't contain a value. Since the actual matching process here doesn't work anything like real lookbehinds, this only works in a limited number of scenarios. Additionally, it only works with the replace method, since other regex-related methods don't offer a mechanism to dynamically "undo" matches. However, since you can run arbitrary code in the replacement function, it does offer a limited degree of flexibility.

Mimicking lookbehind through reversal

The next approach uses lookaheads to mimic lookbehinds, and relies on manually reversing the data and writing your regex backwards. You'll also need to write the replacement value backwards if using this with the replace method, flip the match index if using this with the search method, etc. If that sounds a bit confusing, it is. I'll show an example in a second, but first we need a way to reverse our test string, since JavaScript doesn't provide this capability natively.

String.prototype.reverse = function () {
	return this.split('').reverse().join('');
};

Now let's try to pull this off:

// Mimicking lookbehind like (?<=es)t
var output = 'testt'.reverse().replace(/t(?=se)/g, 'x').reverse();
// output: tesxt

That actually works quite nicely, and allows mimicking both positive and negative lookbehind. However, writing a more complex regex with all nodes reversed can get a bit confusing, and since lookahead is used to mimic lookbehind, you can't mix what you intend as real lookaheads in the same pattern.

Note that reversing a string and applying regexes with reversed nodes can actually open up entirely new ways to approach a pattern, and in a few cases might make your code faster, even with the overhead of reversing the data. I'll have to save the efficiency discussion for another day, but before moving on to the third lookbehind-mimicking approach, here's one example of a new pattern approach made possible through reversal.

In my last post, I used the following code to add commas every three digits from the right for all numbers which are not preceded by a dot, letter, or underscore:

String.prototype.commafy = function () {
	return this.replace(/(^|[^\w.])(\d{4,})/g, function($0, $1, $2) {
		return $1 + $2.replace(/\d(?=(?:\d\d\d)+(?!\d))/g, '$&,');
	});
}

Here's an alternative implementation:

String.prototype.commafy = function() {
	return this.
		reverse().
		replace(/\d\d\d(?=\d)(?!\d*[a-z._])/gi, '$&,').
		reverse();
};

I'll leave the analysis for your free time.

Finally, we come to the third lookbehind-mimicking approach:

Mimicking lookbehind using a while loop and regexp.lastIndex

This last approach has the following advantages:

  • It's easier to use (no need to reverse your data and regex nodes).
  • It allows lookahead and lookbehind to be used together.
  • It allows you to more easily automate the mimicking process.

However, the trade off is that, in order to avoid interfering with standard regex backtracking, this approach only allows you to use lookbehinds (positive or negative) at the very start and/or end of your regexes. Fortunately, it's quite common to want to use a lookbehind at the start of a regex.

If you're not already familiar with the exec method available for RegExp objects, make sure to read about it at the Mozilla Developer Center before continuing. In particular, look at the examples which use exec within a while loop.

Here's a quick implementation of this approach, in which we'll actually toy with the regex engine's bump-along mechanism to get it to work as we want:

var data = 'ttttt',
	regex = /t/g,
	replacement = 'x',
	match,
	lastLastIndex = 0,
	output = '';

regex.x = {
	gRegex: /t/g,
	startLb: {
		regex: /t$/,
		type: true
	}
};

function lookbehind (data, regex, match) {
	return (
		(regex.x.startLb ? (regex.x.startLb.regex.test(data.substring(0, match.index)) === regex.x.startLb.type) : true) &&
		(regex.x.endLb ? (regex.x.endLb.regex.test(data.substring(0, regex.x.gRegex.lastIndex)) === regex.x.endLb.type) : true)
	);
}

while (match = regex.x.gRegex.exec(data)) {
	/* If the match is preceded/not by start lookbehind, and the end of the match is preceded/not by end lookbehind */
	if (lookbehind(data, regex, match)) {
		/* replacement can be a function */
		output += data.substring(lastLastIndex, match.index) + match[0].replace(regex, replacement);
		if(!regex.global){
			lastLastIndex = regex.gRegex.lastIndex;
			break;
		}
	/* If the inner pattern matched, but the leading or trailing lookbehind failed */
	} else {
		output += match[0].charAt(0);
		/* Set the regex to try again one character after the failed position, rather than at the end of the last match */
		regex.x.gRegex.lastIndex = match.index + 1;
	}
	lastLastIndex = regex.x.gRegex.lastIndex;
}
output += data.substring(lastLastIndex);

// output: txxxx

That's a fair bit of code, but it's quite powerful. It accounts for using both a leading and trailing lookbehind, and allows using a function for the replacement value. Also, this could relatively easily be made into a function which accepts a string for the regex using normal lookbehind syntax (e.g., "(?<=x)x(?<!x)"), then splits it into the various parts in needs before applying it.

Notes:

  • regex.x.gRegex should be an exact copy of regex, with the difference that it must use the g flag whether or not regex does (in order for the exec method to interact with the while loop as we need it to).
  • regex.x.startLb.type and regex.x.endLb.type use true for "positive," and false for "negative."
  • regex.x.startLb.regex and regex.x.endLb.regex are the patterns you want to use for the lookbehinds, but they must contain a trailing $. The dollar sign in this case does not mean end of the data, but rather end of the data segment they will be tested against.

If you're wondering why there hasn't been any discussion of fixed- vs. variable-length lookbehinds, that's because none of these approaches have any such limitations. They support full, variable-length lookbehind, which no regex engines I know of other than .NET and JGsoft (used by products like RegexBuddy) are capable of.

In conclusion, if you take advantage of all of the above approaches, regex lookbehind syntax can be mimicked in JavaScript in the vast majority of cases. Make sure to take advantage of the comment button if you have feedback about any of this stuff.

Update 2012-04: See my followup blog post, JavaScript Regex Lookbehind Redux, where I've posted a collection of short functions that make it much easier to simulate leading lookbehind.

Commafy Numbers

I've never used the few scripts I've seen that add commas to numbers because usually I want to apply the functionality to entire blocks of text. Having to pull out numbers, add commas, then put them back becomes a needlessly complex task without a method which can just do this in one shot. So, here's my attempt at this (if JavaScript regexes supported lookbehind, it could be even shorter):

String.prototype.commafy = function () {
	return this.replace(/(^|[^\w.])(\d{4,})/g, function($0, $1, $2) {
		return $1 + $2.replace(/\d(?=(?:\d\d\d)+(?!\d))/g, "$&,");
	});
}

Number.prototype.commafy = function () {
	return String(this).commafy();
}

Here are a couple examples of how this can be used:

(1000).commafy();
// Output: 1,000

var data = '1\n' +
	'10\n' +
	'100\n' +
	'1000\n' +
	'10000\n' +
	'100000\n' +
	'1000000\n' +
	'12345678901234567890\n' +
	'1000.99\n' +
	'1000.9999\n' +
	'.9999\n' +
	'-1000\n' +
	'$1000\n' +
	'"1000"\n' +
	'1000MHz\n' +
	'Z1000';

data.commafy();
/* Output:
1
10
100
1,000
10,000
100,000
1,000,000
12,345,678,901,234,567,890
1,000.99
1,000.9999
.9999
-1,000
$1,000
"1,000"
1,000MHz
Z1000
*/

Note that it adds commas to numbers followed by non-numeric characters, but avoids adding commas to numbers immediately preceded by a dot (decimal point), letter, or underscore. And as shown, this can be applied to individual numbers or entire blocks of text.

This is a decent example of where regular expressions can help to shorten and simplify code even in places you may not initially think to use them.


Edit: I've included an alternative implementation of the above code in my post Mimicking Lookbehind in JavaScript, which temporarily reverses the string to allow a simplified approach.