JavaScript Regex Lookbehind Redux

Five years ago I posted Mimicking Lookbehind in JavaScript on this blog, wherein I detailed several ways to emulate positive and negative lookbehind in JavaScript. My approaches back then were all fairly rough, and it was complicated to properly customize any of them to work with a given pattern. Plus, they were only designed to simulate lookbehind in a regex-based replacement.

To make it much easier to use lookbehind, I recently posted a collection of short functions on GitHub. They use XRegExp v2, so you should check that out, too.

Here's the code:

// Simulating infinite-length leading lookbehind in JavaScript. Uses XRegExp.
// Captures within lookbehind are not included in match results. Lazy
// repetition in lookbehind may lead to unexpected results.

(function (XRegExp) {

    function prepareLb(lb) {
        // Allow mode modifier before lookbehind
        var parts = /^((?:\(\?[\w$]+\))?)\(\?<([=!])([\s\S]*)\)$/.exec(lb);
        return {
            // $(?!\s) allows use of (?m) in lookbehind
            lb: XRegExp(parts ? parts[1] + "(?:" + parts[3] + ")$(?!\\s)" : lb),
            // Positive or negative lookbehind. Use positive if no lookbehind group
            type: parts ? parts[2] === "=" : !parts
        };
    }

    XRegExp.execLb = function (str, lb, regex) {
        var pos = 0, match, leftContext;
        lb = prepareLb(lb);
        while (match = XRegExp.exec(str, regex, pos)) {
            leftContext = str.slice(0, match.index);
            if (lb.type === lb.lb.test(leftContext)) {
                return match;
            }
            pos = match.index + 1;
        }
        return null;
    };

    XRegExp.testLb = function (str, lb, regex) {
        return !!XRegExp.execLb(str, lb, regex);
    };

    XRegExp.searchLb = function (str, lb, regex) {
        var match = XRegExp.execLb(str, lb, regex);
        return match ? match.index : -1;
    };

    XRegExp.matchAllLb = function (str, lb, regex) {
        var matches = [], pos = 0, match, leftContext;
        lb = prepareLb(lb);
        while (match = XRegExp.exec(str, regex, pos)) {
            leftContext = str.slice(0, match.index);
            if (lb.type === lb.lb.test(leftContext)) {
                matches.push(match[0]);
                pos = match.index + (match[0].length || 1);
            } else {
                pos = match.index + 1;
            }
        }
        return matches;
    };

    XRegExp.replaceLb = function (str, lb, regex, replacement) {
        var output = "", pos = 0, lastEnd = 0, match, leftContext;
        lb = prepareLb(lb);
        while (match = XRegExp.exec(str, regex, pos)) {
            leftContext = str.slice(0, match.index);
            if (lb.type === lb.lb.test(leftContext)) {
                // Doesn't work correctly if lookahead in regex looks outside of the match
                output += str.slice(lastEnd, match.index) + XRegExp.replace(match[0], regex, replacement);
                lastEnd = match.index + match[0].length;
                if (!regex.global) {
                    break;
                }
                pos = match.index + (match[0].length || 1);
            } else {
                pos = match.index + 1;
            }
        }
        return output + str.slice(lastEnd);
    };

}(XRegExp));

That's less than 0.5 KB after minification and gzipping. It provides a collection of functions that make it simple to emulate leading lookbehind:

  • XRegExp.execLb
  • XRegExp.testLb
  • XRegExp.searchLb
  • XRegExp.matchAllLb
  • XRegExp.replaceLb

Each of these functions takes three arguments: the string to search, the lookbehind pattern as a string (can use XRegExp syntax extensions), and the main regex. XRegExp.replaceLb takes a fourth argument for the replacement value, which can be a string or function.

Usage examples follow:

XRegExp.execLb("Fluffy cat", "(?i)(?<=fluffy\\W+)", XRegExp("(?i)(?<first>c)at"));
// -> ["cat", "c"]
// Result has named backref: result.first -> "c"

XRegExp.execLb("Fluffy cat", "(?i)(?<!fluffy\\W+)", /cat/i);
// -> null

XRegExp.testLb("Fluffy cat", "(?i)(?<=fluffy\\W+)", /cat/i);
// -> true

XRegExp.testLb("Fluffy cat", "(?i)(?<!fluffy\\W+)", /cat/i);
// -> false

XRegExp.searchLb("Catwoman's fluffy cat", "(?i)(?<=fluffy\\W+)", /cat/i);
// -> 18

XRegExp.searchLb("Catwoman's fluffy cat", "(?i)(?<!fluffy\\W+)", /cat/i);
// -> 0

XRegExp.matchAllLb("Catwoman's cats are fluffy cats", "(?i)(?<=fluffy\\W+)", /cat\w*/i);
// -> ["cats"]

XRegExp.matchAllLb("Catwoman's cats are fluffy cats", "(?i)(?<!fluffy\\W+)", /cat\w*/i);
// -> ["Catwoman", "cats"]

XRegExp.replaceLb("Catwoman's fluffy cat is a cat", "(?i)(?<=fluffy\\W+)", /cat/ig, "dog");
// -> "Catwoman's fluffy dog is a cat"

XRegExp.replaceLb("Catwoman's fluffy cat is a cat", "(?i)(?<!fluffy\\W+)", /cat/ig, "dog");
// -> "dogwoman's fluffy cat is a dog"

XRegExp.replaceLb("Catwoman's fluffy cat is a cat", "(?i)(?<!fluffy\\W+)", /cat/ig, function ($0) {
    var first = $0.charAt(0);
    return first === first.toUpperCase() ? "Dog" : "dog";
});
// -> "Dogwoman's fluffy cat is a dog"

Easy peasy lemon squeezy. 🙂

Creating Grammatical Regexes Using XRegExp.build

Recently, I've added three new addons for XRegExp v2.0 (currently in release candidate stage on GitHub):

  • XRegExp.build — Lets you build regexes using named subpatterns. Inspired by Lea Verou's RegExp.create.
  • XRegExp Prototype Methods — Adds a collection of methods to be inherited by XRegExp regexes: apply, call, forEach, globalize, xexec, and xtest. These also work for native RegExps copied by XRegExp.
  • XRegExp Unicode Properties — Includes the remaining nine properties (beyond what's already available in other XRegExp addons) required for Level-1 Unicode support: Alphabetic, Uppercase, Lowercase, White_Space, Noncharacter_Code_Point, Default_Ignorable_Code_Point, Any, ASCII, and Assigned.

Jumping right into some code, the following demonstrates how the new XRegExp.build addon can be used to create a grammatical pattern for matching real numbers:

// Approach 1: Make all of the subpatterns reusable

var lib = {
    digit:             /[0-9]/,
    exponentIndicator: /[Ee]/,
    digitSeparator:    /[_,]/,
    sign:              /[+-]/,
    point:             /[.]/
};
lib.preexponent = XRegExp.build('(?xn)\
    {{sign}} ?              \
    (?= {{digit}}           \
      | {{point}}           \
    )                       \
    ( {{digit}} {1,3}       \
      ( {{digitSeparator}} ?\
        {{digit}} {3}       \
      ) *                   \
    ) ?                     \
    ( {{point}}             \
      {{digit}} +           \
    ) ?                     ',
    lib
);
lib.exponent = XRegExp.build('(?x)\
    {{exponentIndicator}}\
    {{sign}} ?           \
    {{digit}} +          ',
    lib
);
lib.real = XRegExp.build('(?x)\
    ^              \
    {{preexponent}}\
    {{exponent}} ? \
    $              ',
    lib
);

// Approach 2: No need to reuse the subpatterns. {{sign}} and {{digit}} are
// defined twice, but that can be avoided by defining them before constructing
// the main pattern (see Approach 1).

var real = XRegExp.build('(?x)\
    ^              \
    {{preexponent}}\
    {{exponent}} ? \
    $              ',
    {
        preexponent: XRegExp.build('(?xn)\
            {{sign}} ?              \
            (?= {{digit}}           \
              | {{point}}           \
            )                       \
            ( {{digit}} {1,3}       \
              ( {{digitSeparator}} ?\
                {{digit}} {3}       \
              ) *                   \
            ) ?                     \
            ( {{point}}             \
              {{digit}} +           \
            ) ?                     ',
            {
                sign:           /[+-]/,
                digit:          /[0-9]/,
                digitSeparator: /[_,]/,
                point:          /[.]/
            }
        ),
        exponent: XRegExp.build('(?x)\
            {{exponentIndicator}}\
            {{sign}} ?           \
            {{digit}} +          ',
            {
                sign:              /[+-]/,
                digit:             /[0-9]/,
                exponentIndicator: /[Ee]/
            }
        )
    }
);

The real and lib.real regexes created by the above code are identical. Here are a few examples of strings they match:

  • -1
  • 1,000
  • 10_000_000
  • 1,111.1111
  • 01.0
  • .1
  • 1e2
  • +1.1e-2

And here are a few examples of strings they don't match:

  • ,100
  • 10,00
  • 1,0000
  • 1.
  • 1.1,111
  • 1k

Grammatical patterns like this are easier to read, write, and maintain, and look more like a BNF than the typical line-noisy regular expressions that some people have come to hate.

Note that the {{…}} syntax shown here works only for regexes created by XRegExp.build. Named subpatterns can be provided as strings or regex objects (strings are passed to the XRegExp constructor). The provided patterns are automatically wrapped in (?:…) so they can be quantified as a unit and don't interfere with the surrounding pattern in unexpected ways. A leading ^ and trailing unescaped $ are stripped from subpatterns if both are present, which allows embedding independently useful anchored patterns. Flags can be provided via XRegExp.build's optional third (flags) argument. Native flags used by provided subpatterns are ignored in favor of the flags argument. Backreferences in the outer pattern and provided subpatterns are automatically renumbered to work correctly within the larger combined pattern. The syntax ({{name}}) works as shorthand for named capture via (?<name>{{name}}). The {{…}} syntax can be escaped with a backslash.

Play around with the above details a bit, and I think you'll find that XRegExp.build works intuitively and handles any edge cases you throw at it.

Feel free to share how you might alter the above regexes. And make sure to check out the fancy new XRegExp v2.0 and its upgraded addons at GitHub.

Ideas for Regular Expressions Cookbook Second Edition

I'm happy to report that work is underway by Jan Goyvaerts and me on the second edition of Regular Expressions Cookbook. In the new edition, we'll be fixing all known errata, improving existing content, updating everything to support the latest versions of the book's eight covered programming languages, and sprinkling several new recipes into existing chapters.

We'll also be adding a new chapter, tentatively titled "Source Code and Log Files". This new chapter will aim to assist programmers and system admins with common tasks, using regex-based solutions. The recipe list for this chapter remains to be determined, but it will include things like how to find strings and comments in various programming languages, and how to find 404 records within Apache HTTP Server logs.

Do you have an idea for a recipe that you'd like to see added? Suggestions are particularly welcome for this new chapter, but ideas for the rest of the book are also welcome.

New language editions

The first edition of Regular Expressions Cookbook has now been published in eight languages. You can get your favorite language version of it from the following sites:

XRegExp Updates

A few days ago, I posted a long-overdue XRegExp bug fix release (version 1.5.1). This was mainly to address an IE issue that a number of people have written to me and blogged about. Specifically, RegExp.prototype.exec no longer throws an error in IE when it is simultaneously provided a nonstring argument and called on a regex with a capturing group that matches an empty string. That's an edge case of an edge case, but it was causing XRegExp to conflict with jQuery 1.7.1 (oops). You can see the full list of changes in the changelog.

But wait, there's more… XRegExp's Unicode plugin has been updated to support Unicode 6.1 (released January 2012), rather than Unicode 5.2. I've also added a new test suite with 265 tests so far, and more on the way.

More substantial changes to XRegExp are planned and coming soon. Follow the brand new XRegExp repository on GitHub to keep up to date or to fork it and help shape the future of this one-of-a-kind JavaScript library. 🙂

Regex Syntax Highlighter

Do you regularly post regular expressions online? Have you seen the regex syntax highlighting in RegexPal, RegexBuddy, or on my blog (example), and wanted to apply it to your own websites? Prompted by blog reader Mark McDonnell, I've extracted the regex syntax highlighting engine built into RegexPal and made it into its own library, unimaginatively named JavaScript Regex Syntax Highlighter. When combined with the provided CSS, this 1.6 KB self-contained JavaScript file can be used, for instance, to automatically apply regex syntax highlighting to any HTML element with the "regex" class. You can see an example of doing just that on my quick and dirty test page.

Highlighting example: <table\b[^>]*>(?:(?=([^<]+))\1|<(?!table\b[^>]*>))*?</table>

Although the library is simple (there's just one function to call), the syntax highlighting is pretty advanced and handles all valid JavaScript regex syntax and errors (with errors highlighted in red). An example of its advanced highlighting support is that it knows, based on the context, whether \10 is backreference 10, backreference 1 followed by a literal zero, octal character index 10, or something else altogether due to its position in the surrounding pattern. Speaking of octal escapes (which are de facto browser extensions; not part of the spec.), they are correctly highlighted according to their subtle differences inside and outside character classes (outside of character classes only, octals can include a fourth digit if the leading digit is a zero). As far as I'm aware, this is the first JavaScript library for highlighting regex syntax, with or without the level of completeness included here. For people who might feel inclined to use or improve upon my work, I've made the licensing as permissive as possible to avoid getting in your way. RegexPal is already open source under the GNU LGPL 3.0 License, but this new library is released under the MIT License. If you plan to customize or help upgrade this code, note that it could probably use a bit of an overhaul (it's ripped from RegexPal with minimal modification), and might require an overhaul if you want to cleanly add support for additional regex flavors. Another nifty feature I plan to eventually add is explanatory title attributes for each element in the returned HTML, which might be particularly helpful for deciphering any highlighted errors or warnings. Let me know if this library is useful for you, or if there are any other features you'd like to see added or changed. Thanks! Link: JavaScript Regex Syntax Highlighter.