Pomsky | Pomsky Blog

Pomsky 0.12

Mon, 10 Nov 2025 10:00:00 GMT

I’m happy to announce version 0.12 of Pomsky, the next-level regular expression language! Pomsky makes writing correct and maintainable regular expressions a breeze. Pomsky expressions are converted into regexes, which can be used with many different regex engines.

If you’re not familiar with Pomsky, here is a quick summary of how it compares to regular expressions.

What’s new?

This release comes packed with new features and improvements. Here are the highlights:

The RE2 flavor, used for Go’s regexp package, is now supported
Intersection of character sets
Improved Unicode support with Script Extensions
A convenient pomsky test subcommand for running unit tests
Powerful optimizations for repetitions, character sets, and alternatives
New IDE capabilities for the VSCode extension
New installers, including an npm package

You might have noticed that Pomsky has a new logo. I’ve also updated the website by switching from Hugo to Starlight, as the Hugo theme we were using was no longer being maintained.

This release took longer than usual because of a few unplanned delays. The last version was released almost two years ago! I had planned to cut a release by the end of 2024, but sometimes things don’t go as planned. The wait is finally over, so let’s take a look at the most exciting new features in this release!

RE2 Support

RE2 is a fast regex engine from Google. Unlike backtracking regex engines such as PCRE2, it is based on finite automata, so it has better worst-case performance. Pomsky now offers a RE2 flavor, which is also compatible with Go’s regexp package. Because the RE2 flavor doesn’t support advanced features such as lookbehind and backreferences, Pomsky produces an error when you try to use them:

> pomsky -f re2 "<< 'test'"
error P0301(compat):
  × Unsupported feature `lookahead/behind` in the `RE2` regex flavor
   ╭────
 1 │ << 'test'
   · ────┬────
   ·     ╰── error occurred here
   ╰────

With RE2, Pomsky now supports 8 regex flavors, covering most mainstream programming languages:

PCRE (PHP, R, Erlang, …)
JavaScript (TypeScript, Dart, …)
Java (JVM languages)
Python
.NET (C#)
Ruby
Rust
RE2 (Go)

Character Set Intersection

Several regex engines¹ support intersecting character sets:

[\p{Thai}&&\p{Nd}]

The above matches a codepoint that is in the ‘Thai’ script and in the ‘Nd’ (decimal number) category. Pomsky now has an & operator to express this:

[Thai] & [Nd]

Some regex engines also support subtraction. Pomsky doesn’t offer this feature, but it can be easily emulated using negation:

[Thai] & ![Nd]   # negating one character set subtracts it

Note that not all flavors support intersection. However, if both character sets are negated, they are merged by applying De Morgan’s first law:

![Thai] & ![Nd]  # is turned into...
![Thai Nd]

Unicode Script Extensions

Most software has to be able to handle text in different languages and writing systems. This is why I’ve always considered good Unicode support to be one of Pomsky’s strongest features. For example, Pomsky polyfills \w in JavaScript, which is surprisingly not Unicode aware, even with the unicode flag enabled.

Pomsky also makes it easy to match a code point in a particular Unicode script: For example, [Syriac] matches all Syriac characters – at least in theory. But Unicode scripts cannot overlap, so code points that would belong in multiple scripts are assigned to the Common or Inherited script instead.

Script Extensions solve this problem, which Pomsky now supports:

# matches all codepoints with a Syriac script extension
[scx:Syriac]

Because code points can have multiple scripts in their ‘Script Extensions’ property, this is more accurate.

Script Extensions are currently only supported in the PCRE, JavaScript and Rust flavors, but hopefully more regex engines will add support in the future.

In addition to scx:, you can also use the gc:, sc:, and blk: prefixes to match a general category, script, or block. These prefixes are optional, but adding them might help with readability:

[Letter]     # old
[gc:Letter]  # new

[InBasic_Latin]   # old
[blk:Basic_Latin] # new

Note that the In prefix of Unicode blocks is omitted when using the blk: prefix. It is recommended to use blk: because we will deprecate the In prefix in the future.

Test subcommand

Pomsky 0.11 added the test construct and a --test flag to run unit tests during compilation. This was a big step towards making Pomsky expressions more correct and maintainable. However, there was no easy way to test all Pomsky expressions in a project during a CI workflow. This has now been addressed with the new pomsky test command:

> pomsky test --path examples/ -e pcre2
testing examples/modes.pomsky ... ok
testing examples/email.pomsky ... ok
testing examples/repetitions.pomsky ... ok
testing examples/version.pomsky ... ok
testing examples/special.pomsky ... ok
testing examples/groups.pomsky ... ok
testing examples/strings.pomsky ... ok
testing examples/capt_groups.pomsky ... ok

test result: ok, 8 files tested in 1.41ms

pomsky test recursively iterates through the given directory, taking .ignore and .gitignore files into account, and tests all files ending with .pomsky. If at least one file doesn’t compile or contains a failing test, the program exits with an error, so it’s easy to include it in your CI pipeline.

Pomsky previously only supported pcre2 for running tests. In this release, we added rust as another option. We want to add more regex engines for unit tests, but this is proving tricky, so it didn’t make it into this release.

Optimizations

Pomsky lets you to refactor parts of an expression into variables to improve readability and follow the DRY principle. For example:

let digit = ['0'-'9'];
let hex_digit = digit | ['a'-'f'];

But this comes with a trade-off: Sometimes the produced regex is less efficient and slower in some regex engines. Optimizations help with this, so you can write readable, DRY code without worrying too much about performance.

In this release, Pomsky gained some important optimizations:

Character ranges are merged if they are adjacent or overlap. For instance, 'a' | ['b'-'d' 'c'-'f'] becomes just [a-f].
Alternations are merged if possible: "case" | "char" | "const" | "continue" compiles to c(?:ase|har|on(?:st|tinue)).

Merging common prefixes can help performance in backtracking regex engines. Note that optimizations are applied after resolving variables, which is important for the example above.

Editor improvements

A key feature of any computer language is its IDE integration. For this release, we’ve added a few new features to the VSCode extension, including:

Go to definition
Find usages
Rename variable

There’s just one caveat: for these features to work, the file has to be syntactically valid. This is because these actions work on the abstract syntax tree (AST), and we currently can’t get the AST if parsing fails.

The next step is therefore to make the parser recoverable, so that it can produce an AST even in the presence of syntax errors. This will also help to improve autocompletion, which currently doesn’t use the AST.

We’ve also added inlay hints to show the index of unnamed capturing groups, meaning you no longer need to count them when writing a replacement pattern.

New installers

For this release, we adopted the wonderful cargo-dist for distributing the pomsky binary, so we could provide more installation options: In addition to the Windows, Linux and macOS binaries, we now have

Shell and PowerShell scripts to download and install Pomsky
An npm package, so you can install Pomsky with
```
npm install -g @pomsky-lang/cli
```
or run it directly with
```
npx @pomsky-lang/cli
```
An msi for installing and uninstalling Pomsky on Windows

As before, you can also get Pomsky from the AUR with yay -S pomsky-bin, from crates.io with cargo install pomsky-bin, or from Homebrew.

Breaking change: Fixing hygiene

Pomsky variables work much like macros in Lisp and Rust: When Pomsky encounters a variable, it is substituted with its content. Variables are hygienic, which means they are properly scoped when substituted.

However, we discovered that modifiers are not hygienic. For example:

let variable = .*;

(enable lazy; variable)

Is the repetition in line 1 lazy or not? In Pomsky 0.11, it was lazy, which is counterintuitive because the enable lazy; statement is not in scope where the repetition appears, only where the variable is used.

Unfortunately, fixing this required a breaking change. We think that the impact will be minimal, but to be sure, please check if you are relying on this behavior anywhere.

Other changes

See the full list of changes in the changelog.

Support us!

We have a lot of exciting plans to make Pomsky a success. To realize them, we need your help! But when I say ‘we’, that’s mostly just me, @Aloso, working on Pomsky in my spare time. I’m looking for contributors to implement new features, tooling, and integrations. If you’d like to help (or have questions, or just want to chat), drop by our Discord channel. Also, if you’re using Pomsky, I’d like to hear about it!

Consider sponsoring me to help making my open-source work financially sustainable. Thank you ❤️

Character set intersection is supported in the JavaScript, Java, Ruby, and Rust flavors. Note that JavaScript requires the v flag for this. ↩

Pomsky 0.11

Thu, 09 Nov 2023 10:00:00 GMT

I am excited to announce version 0.11 of Pomsky, the next level regular expression language! Pomsky makes it easier than ever to write correct and maintainable regular expressions. Pomsky expressions are transpiled to regexes and can be used with many regex engines.

What’s new?

This release comes with some exciting new features:

Unit tests to make sure your Pomsky expressions are correct
Negation of variables makes code re-use easier
New word boundaries < and >, matching at the start and end of a word, respectively
Recursion to match recursive grammars in PCRE and Ruby
Much improved documentation

If you’re unfamiliar with Pomsky, here is a summary of how Pomsky compares to regexes.

Let’s look at the most exciting new features in this release!

Unit tests

If you have non-trivial regular expressions in your code base, tests are essential to prevent regressions when modifying them. The tests also serve as documentation by showing what strings the expressions should match. Pomsky now makes it easier than ever to test regular expressions:

test {
  # unit tests go here!
  match '0.11.0';
  match '2023.3.7';
  match '1.0.0-beta+exp.sha.5114f85';
  match '1.0.0-rc.2';

  # these are invalid:
  reject '1.0';
  reject '1.2.3.4';
  reject '1.7_01';
  reject '^1.7.1';
}

let num = [ascii_digit]+;
let ident = [ascii_alnum '-']+;
let idents = ident ('.' ident)*;

# https://semver.org/
:major(num) '.' :minor(num) '.' :patch(num)
  ('-' :prerelease(idents))?
  ('+' :buildmeta(idents))?

Unit tests can also compare capturing groups:

test {
  match '1.0.0-beta+exp.sha.5114f85' as {
    major: '1',
    minor: '0',
    patch: '0',
    prerelease: 'beta',
    buildmeta: 'exp.sha.5114f85',
  };
}

You can run tests with the --test option.

The VS Code extension and the online playground were updated to execute the tests as you type, so you immediately see when a test fails – ideal for test-driven development. In VS Code this must be enabled with the pomsky.runTests setting.

Note that for now, tests are always run with the PCRE2 regex engine (except in the online playground, which uses the browser’s regex engine). PCRE2 was chosen because it has the most features and is easy to statically link, but proper support for other regex engines is planned as well.

Negating all the things

If you ever needed to negate a variable that refers to a character set, this is now possible:

let d = [digit '_'];

d !d   # output: [\d_][^\d_]

Until now, Pomsky’s syntax allowed the ! only in front of the tokens [, <<, >>, and %. This restriction has now been lifted.

New word boundaries

Pomsky now supports < and > to match the start and end of a word, respectively. You could use the word boundary % for this, but < and > are more specialized since they can only match the start or end of a word, but not both. It might not matter most of the time, but using < and > may be considered more idiomatic:

<'test'>

This matches the word “test”, but not in “detest” or “testament”.

Recursion

When targeting PCRE or Ruby, you can now use recursion. It is pretty straightforward:

test {
  match '3+4*5';
  match '2+(4/(-2))*15';
}

# a nested mathematical expression
'-'? ([digit]+ | '(' recursion ')') (['+-*/'] recursion)*

This recursively matches the whole expression whenever the recursion keyword is encountered.

Improved documentation

For this release, the web docs were overhauled.

Until now, the only documentation was the language tour, so it tried to accommodate everyone. This didn’t work, as it was too advanced for beginners, too wordy for people already familiar in regular expressions, and too imprecise as a specification. The solution is to provide different materials for different people: for

A quick reference for people who already know regular expressions and just want to learn Pomsky’s syntax as quickly as possible
A language tour for to beginners to learn the fundamentals
A complete reference to document everything precisely for contributors or users with a very specific question

I hope you enjoy the new docs. I always appreciate feedback!

See the full list of changes in the changelog.

Roadmap update

As the year is coming to an end and most of the things I planned for 2023 are done, I am evaluating which issues I should tackle next. The following plan is quite ambitious, so let’s see how much I will manage to implement next year!

Language changes

There are many features waiting to be implemented: Modifiers (like ignore_case), intersection and subtraction of character sets, subroutine calls, conditionals, etc.

Other than that, the biggest focus next year will be on optimizations, so you can write expressions however you want without worrying too much about performance. Pomsky should feel more like a high-level language that lets you focus on the “business logic” instead of regex engine internals.

Another focus area will be diagnostics: Pomsky should detect when an expression has poor worst-case performance and could be vulnerable to ReDoS (regex denial-of-service) attacks. It should also detect more patterns that are illegal in the targeted regex engine, e.g. variable-length lookbehind in PCRE and Python.

VS Code improvements

The VS Code extension is lacking some features one has come to expect from their IDE: Go to definition, renaming symbols, and finding usages.

To implement these, we need to parse Pomsky into a syntax tree, which is challenging because the existing parser is written in Rust, but the language server is written in TypeScript. Instead of rewriting the parser in TypeScript, one possibility is to use tree-sitter, a parser framework designed for IDEs.

Tooling

To make migrating to Pomsky easier, I want to make a Regex to Pomsky converter. We also need a formatting tool (like prettier or rustfmt) for Pomsky. These tools will be integrated in the CLI, the VS Code extension, and the online playground.

Writing the Regex to Pomsky converter will be particularly challenging, since it requires a configurable parser that can correctly parse all regex flavors supported by Pomsky.

Support us!

Pomsky is a very small community, and most of the work is still done by me. To make the continued development of Pomsky sustainable, I am looking for contributors. There are lots of opportunities to help; you can drop by our Discord channel if you need guidance.

Also, consider sponsoring me to support my open source work!

Pomsky 0.10 released with new tools

Wed, 22 Mar 2023 00:00:00 GMT

Just last week, Pomsky celebrated its first birthday! Today, I’m announcing version 0.10, along with a VSCode extension and a JavaScript plugin. This allows you to write regular expressions with Pomsky, and include them as RegExp with a bundler like Vite or Webpack.

What is Pomsky?

Pomsky is a modern syntax for regular expressions, transpiled to regexes compatible with JavaScript, Java, PCRE, Rust, Ruby, Python, or .NET. It aims to be more readable, portable, and powerful than traditional regexes. The biggest difference is that Pomsky is whitespace-insensitive, and string literals are quoted. For example, the regex \[.+\] could be written as '[' .+ ']'. Here’s a larger example that shows Pomsky’s strengths:

# variables allow code re-use
let octet = range '0'-'255';

# ranges can match multi-digit numbers
let subnetMask = '/' range '0'-'32';

# concise syntax for named capturing groups (ipv4, mask)
:ipv4(octet ('.' octet){3}) :mask(subnetMask)?

This is transpiled to the following regex:

(?<ipv4>(?:0|1[0-9]{0,2}|2(?:[0-4][0-9]?|5[0-5]?|[6-9])?|[3-9][0-9]?)(?:\.(?:0|1[0-9]{0,2}|2(?:[0-4][0-9]?|5[0-5]?|[6-9])?|[3-9][0-9]?)){3})(?<mask>/(?:0|[1-2][0-9]?|3[0-2]?|[4-9]))?

What’s new

Version 0.10 includes a number of language changes and new tools.

VSCode extension

Pomsky now has a VSCode extension, with syntax highlighting, auto-completion, and diagnostics (errors and warnings). It also has a preview panel, which you can open with the icon in the top right corner or from the right-click menu:

To use the VSCode extension, you currently need to have Pomsky installed locally. You can download it from the releases page. If you want to try Pomsky without installing anything, check out the playground!

Note that we don’t have an official file extension yet. You can vote for one here.

JavaScript plugin

Thanks to @Kyza, we now have a JavaScript plugin that allows you to easily import Pomsky expressions in your JavaScript projects using Vite, Webpack, Rollup, ESBuild or ESM. Once you followed the installation instructions, you can use it like this:

import ipv4 from './ipv4.pom'

function isValidIpV4(string) {
  return ipv4().test(string)
}

How does it work? The plugin looks for imports with the file extension .pom or .pomsky and transforms them to export a function that returns the corresponding RegExp. This is done at build time, so it doesn’t affect website performance. I would like to thank @Kyza for creating this wonderful plugin.

Disabling Unicode

One annoying thing about regular expressions in JavaScript is that \w and \b do not respect Unicode, even if the u flag (short for unicode) is set! Pomsky polyfills Unicode support for [word], but not for % (a word boundary) for several reasons¹.

To address this issue, Pomsky now lets you opt out of Unicode support. Now word boundaries are forbidden in the JavaScript flavor, but can be used if you disable Unicode support:

disable unicode;
% 'Pomsky' %

This makes sense when you know that your input contains only ASCII characters. It also has the advantage that some shorthands, such as [word], can be optimized better: [word] normally matches over 800 Unicode ranges, but when Unicode is disabled, it’s just [a-zA-Z0-9_], which is more performant.

Reserved keywords

U was reserved to make the syntax for codepoints (e.g. U+FF) unambiguous. Now codepoints can be parsed even if they contain spaces (e.g. U + FF), although this isn’t usually recommended.

Also, test has been reserved as a keyword to support unit testing in the future. Pomsky will be able to run unit tests during compilation to make sure that the regex matches certain strings:

test {
  match '127.0.0.1';
  match '80.255.255.254/24';
  reject '400.1.2.3';
}

# your Pomsky expression here...

These tests not only give you immediate feedback if the expression is incorrect, they also document the code so that a reader can see examples of what the expression does or does not match. Unit tests will probably land in Pomsky 0.11, but the syntax is not fully fleshed out yet. Let me know what you think!

You can find the full list of changes in the changelog.

A glimpse into the crystal ball

In January, I published my roadmap for this year, and some things are already done! The VSCode extension is not feature complete, but it is already very useful. Most of the language improvements I envisioned in January have also been implemented. The testing infrastructure has also made progress: All regex flavors are now tested in CI and compiled in our fuzzer.

As for the future, it can be hard to predict what will happen. For example, I didn’t expect someone to create a JavaScript plugin, which was a pleasant surprise! I also underestimated my own velocity, so some of the things I put in the “not planned” section could be completed this year.

If there is something you would like to work on, any help is appreciated! Pomsky has some good first issues. For example, if you’re a native English speaker, we could use your help to overhaul the documentation.

If you like Pomsky, consider sponsoring me. You can also give Pomsky a star and join our Discord server.

Cheers,
Ludwig

[word] is polyfilled as [\p{Alphabetic}\p{M}\p{Nd}\p{Pc}]. Word boundaries (%) aren’t polyfilled for two reasons: First, the produced regex would be very large: (?:(?<![\p{Alphabetic}\p{M}\p{Nd}\p{Pc}])(?=[\p{Alphabetic}\p{M}\p{Nd}\p{Pc}])|(?<=[\p{Alphabetic}\p{M}\p{Nd}\p{Pc}])(?![\p{Alphabetic}\p{M}\p{Nd}\p{Pc}])) Secondly, the polyfill requires lookbehind, which isn’t universally supported. Notably, lookbehind doesn’t work in Safari. ↩

Pomsky 0.9 and our roadmap

Sun, 15 Jan 2023 00:00:00 GMT

What is Pomsky?

Pomsky is a portable, modern syntax for regular expressions. It has powerful features, such as variables, and a much more readable syntax. Check out the language tour to quickly get familiar with Pomsky, or the examples to see some real Pomsky expressions.

Pomsky is not a regex engine. Instead, a Pomsky expression is transpiled to a normal RegExp, compatible with many RegExp engines, including JavaScript, Java, PCRE, Ruby, Python, Rust and .NET.

What’s new

With this release, we’re celebrating 1000 stars on GitHub! This is an important milestone, so I would like to take this opportunity to thank everyone who has supported me along the way: My siblings who always cheer me up, my colleague who gave me legal advice, everyone from the Rust community who provided useful feedback to me, the people who sent me pull requests, reported bugs, proposed new features, or reported vulnerabilities, and DALL·E for designing our logo. Thank you!

Pomsky has a Discord server now. Stop by if you have a question, want to help, or just want to chat with the maintainers (currently only me) or other users!

This is a small release focusing on maintenance and fixing bugs. The only user-facing change is that warnings and errors now include a code, such as P0100. Ideally, the CLI should provide all the necessary information to fix the problem, but if you ever have to search for an error, this code might make it easier to find relevant results.

The other new feature is that Pomsky can now return all output as JSON with the --json flag, which will be important for tools and IDE integrations. Someone is currently working on an IntelliJ plugin, which they plan to release in a few weeks. It already offers many nice features, and I’m excited to announce it once the initial version is ready! If you’re using VS Code or another editor, please be patient. In the meantime, you can always use the online playground, which offers syntax highlighting, auto-completion and inline diagnostics.

If you want to contribute to Pomsky, we now have a contributor’s guide to help you get off the ground. It includes an architectural overview and explains how Pomsky is tested. If something is missing, please reach out.

More testing improvements

Last release, I improved the test suite and fuzzer by compiling generated regexes to check if they are syntactically valid. This proved to be very effective and uncovered a bunch of bugs.

But I only compiled regexes in the Rust and PCRE flavors, because Pomsky can easily include these regex engines as Rust crates. Testing other variants is more difficult, but I was determined to get it working. To be really confident that a regex is valid, you have to test it; reading the documentation on regex engines only gets you so far, and in many cases the documentation is not as detailed as I had hoped.

How did I approach this problem? First, I wrote small JavaScript, Java and Python programs to compile a regex and return whether it is valid. Now all I had to do was run these programs from Rust. But here’s the problem: Starting a JavaScript program with Node.js takes about 100ms, but when fuzzing, I want to compile thousands of expressions per second. Starting a process every time I want to verify a regex is not feasible.

My solution was to spawn long-running processes that communicate via stdin and stdout: Whenever the Node.js program receives an expression via stdin, it tries to compile it and writes either “success!” or the error message to stdout. For Java and Python, we do the same. This works surprisingly well, and is fast enough for our fuzzer. For Ruby, we don’t have to spawn a process: I realized that Ruby uses the Oniguruma library, which is also available as Rust crate. This only leaves .NET, which still isn’t sufficiently tested. I first have to figure out how to compile C# on Linux, and make it work in our GitHub action on Windows, Ubuntu and macOS.

When I started fuzzing with regex testing for Java, JavaScript, Python, Ruby, Rust and PCRE, it took only a few seconds for the fuzzer to spit out the first error. In total, it found 12 bugs, 10 of which are fixed in this release. Fixing the other bugs will require more work.

You can find the complete changelog here.

Roadmap

It’s a misconception that the more work you pour into a project, the less there is to do. A project is not a to-do list where you can do one task after another until you reach the end. In reality, the amount of work never decreases: The more code you write, the more code you have to maintain. Furthermore, as Pomsky becomes more popular, there are more people requesting features for diverse use cases.

Right now, my capacity for feature development is very limited, both by my amount of spare time and by my mental health. Unless more people step up to help, I have to make tough decisions about which tasks take priority.

Here’s what I’m going to work on in the next months:

Testing: I’m committed to invest more work into automated testing to eliminate bugs.
Optimizations: There are some low-hanging fruit to make Pomsky’s output smaller and more efficient.
Language improvements: I have a few ideas how to make Pomsky more expressive. I’m also considering some breaking syntax changes to make Pomsky more ergonomic and consistent, but these ideas are not fleshed out yet.
VS Code extension: Considering the popularity of VS Code, this is a must. I hope I’ll be able to re-use some code from the Pomsky playground for this.

The following features are not on my to-do list right now:

Regex to Pomsky converter
Auto-formatter
Linter
Babel or Vite plugins
Python bindings
Adding more flavors
Publishing the CLI on other platforms

All of these features would be great, but I probably won’t have the time to implement them myself anytime soon. If you want to help, all contributions are welcome and appreciated! Just let me know what you want to work on 😉

If you like Pomsky, consider sponsoring me.

Cheers,
Ludwig

Pomsky 0.8 released

Thu, 22 Dec 2022 12:00:00 GMT

The new logo: A pomeranian husky wearing orange glasses, created with the help of DALL·E 2

What is Pomsky?

Pomsky is not a regex engine. Instead, a Pomsky expression is transpiled to a normal RegExp, compatible with many RegExp engines, including JavaScript, Java, PCRE, Ruby, Python, Rust and .NET.

Announcement

Hi! I’m Ludwig, and I created Pomsky in my spare time. I am pleased to announce that you can now sponsor me for my work! It’s been a lot of fun to develop Pomsky, but as Pomsky is being used by more and more people, I will need more time for maintenance and improving the developer experience. Donations will help me to invest the time needed. If you can spare a few bucks or convince your employer to sponsor me, that would go a long way to make Pomsky sustainable.

All sponsors will receive monthly updates on what I am working on, as I want to be transparent about how your money is used. Find out more on my GitHub sponsors page.

Summary of the changes

This release focuses on stability, but also brings a few new features:

Inline regexes landed, allowing you to use regex features not yet supported by Pomsky
Pomsky now supports the dot, matching any code point
An optimization pass that tries to make the generated regex smaller
Many bugs were fixed

Inline regexes

Inline regexes are an escape hatch to include an arbitrary regex, at the cost of fewer compile-time guarantees. They are added to the output without any escaping; you could compare this to inline assembly in higher-level languages like C or Rust. Inline regexes allow you to produce hand-optimized code, or use regex features that aren’t properly supported yet. For example:

let recurse = regex '(?R)';  # inline regex for using recursion

let number = '-'? [digit]+;
let operator = '+' | '-' | '*' | '/';
let expr = number | '(' recurse ')';

expr (operator expr)*  # matches an arbitrarily nested term

Note that recursion using the (?R) syntax only works in the PCRE flavor.

Inline regexes are very powerful, but use them sparingly: Pomsky doesn’t parse their content, so it cannot warn you from mistakes. Pomsky usually guarantees that the regexes it generates are valid, but this guarantee may not hold if you’re using inline regexes.

The dot

You’re probably wondering: The dot is one of the most important regex features, so why was it missing until now? I have been hesitant to add it because it’s so easy to forget that the dot does not match line breaks by default. Also, the dot is not Unicode-aware in all regex variants, and all regex engines treat line breaks slightly differently. That’s why I initially didn’t include the dot, and recommended to use Codepoint (abbreviated as C) instead.

However, Codepoint won’t work in ERE (POSIX-extended regular expressions), which we want to support eventually. Also, I decided that best practices such as avoiding the dot are best enforced by a linter rather than the compiler, so specific lints can be disabled when necessary. Pomsky doesn’t have a linter yet, but it’s still on my todo list.

Optimizations

Pomsky now does an optimization pass right before code generation. Before, optimizations were intermingled with parsing and code generation, which led to some problems: Parsing wasn’t lossless (extra parentheses would get lost), and the code was hard to maintain and iterate on. The new optimization pass solves these problems while significantly improving the generated code in some cases:

Subsequent repetitions are now merged whenever possible. For example, ('x'{1,9})? becomes x{0,9}.
Duplicated items in a character set are removed. For example, ['hello world'] becomes [helo wrd].
Empty groups like (() '') are removed, even when followed by a repetition.

More optimizations are planned: For example, we want to make it possible to combine alternatives with a common prefix, like 'make' | 'maple' | 'maker'. Ideally, this would produce the output ma(?:ker??|ple), which is much faster in more mainstream¹ regex engines that use backtracking instead of a DFA.

Testing improvements and bugfixes

For this release we improved our testing and fuzzing infrastructure. We have a test harness with several hundred hand-written test cases. For each test case, we check if compiling it produces the expected output, or fails with the expected error if the input is malformed. We also have a fuzzer, which generates test cases and systematically mutates them to discover new code paths. While the test harness verifies that the output is correct, the fuzzer can only verify that the program doesn’t crash, since we can’t manually check millions of generated test cases.

So to increase our confidence in Pomsky’s correctness, we started to compile regexes produced by Pomksy with the respective regex engine. For now, we only do this in the Rust and PCRE flavors, but we plan to compile more regex flavors in the future. This still doesn’t guarantee that the output is correct, but at least it checks if the output is syntactically valid. With this change, the fuzzer was able to find a few bugs during code generation.

We also started measuring test coverage and uploading it to coveralls.io. Looking at the reports helped us identify parts that needed more tests, and improving our test coverage. Unfortunately, the data isn’t as reliable as we’d hoped: Coveralls highlights many struct definitions and enum variants as “untested”, which is inaccurate. Measuring test coverage is still a net win, just don’t read too much into the numbers 😉

Auditable binaries

The binaries published on GitHub and the AUR are now built with cargo audit, so we can be sure there are no known vulnerabilites when we publish a new version. Additionally, the list of dependencies is embedded in the binaries, so you can check with cargo audit bin <PATH_TO_BINARY> if any vulnerabilities were found since then. cargo-audit is a tool that scans libraries for advisories reported in the rustsec advisory DB.

CLI changes

You can now disable all warnings with -W0, compatibility warnings with -Wcompat=0, and deprecation warnings with -Wdeprecated=0. Needless to say that doing this is discouraged, except in edge cases. Warnings exist for a reason!

You can specify which features should be allowed with --allowed-features. For example, with --allowed-features boundaries,dot,lazy-mode,named-groups,ranges,references,variables, only the specified language features are available. Without this argument, all features can be used.

The help messages have been extended. pomsky -h now gives a short summary, while pomsky --help provides more detailed information.

You might remember that I replaced clap with a more lightweight argument parsing library. The new library we’re using doesn’t support help generation, so I wrote a macro to generate help messages that can contain formatting and color. It is general enough that I might publish it as a separate crate, stay tuned!

Happy Hanukkah! Merry Christmas! Happy Holidays!

Ludwig

An earlier version of this post said that “traditional” regex engines use backtracking. That’s not quite correct; backtracking only became popular with Perl. All regex flavors supported by Pomsky use backtracking, except for Rust, which builds a lazy DFA. ↩

Pomsky 0.7 released

Sat, 10 Sep 2022 00:00:00 GMT

I just released Pomsky 0.7 with a new, much faster parser.

What is Pomsky?

Pomsky is not a regex engine. Instead, a Pomsky expression is transpiled to a normal RegExp, compatible with many RegExp engines, including JavaScript, Java, PCRE, Ruby, Python, Rust and .NET.

Summary of the changes

Because I didn’t publish a blog post when I released version 0.6, this post is a bit longer. Here’s a summary:

The parser was rewritten and is now much faster

Pomsky is now published as WASM module to npm

Some syntax additions make writing Pomsky expressions more fun

Diagnostics were improved again. Most importantly, Pomsky now detects typos and suggests the correct spelling!

The binary size has been significantly decreased

The new parser

The biggest change in this release is that the parser was rewritten from scratch. It previously used nom parser combinators, but over time it became evident that nom isn’t well suited for my use case. It made me jump through hoops to get decent error messages, and performance was less than ideal. I considered optimizing other parts of the compiler, but it turns out that parsing is the only performance bottleneck in Pomsky; all other compiler passes take only a small fraction of the time required for parsing.

The new parser doesn’t use any libraries. It has all the flexibility I need, and turned out to be much faster. I haven’t even started to optimize it, and it’s already outperforming the previous parser by up to 500% according to my benchmarks.

So how did the code change? It became slightly longer, but in exchange, control flow is much more explicit, so it’s easier to reason about the code.

During the rewrite, I was glad that Pomsky has a variety of integration tests that cover both valid and invalid inputs. Once the code compiled again, I ran the test suite to identify bugs. When all the bugs I found were fixed and all 200 tests were green, I had much more confidence that my code was correct.

The WASM module

Pomsky is now published to npm. Getting it to work turned out to be much more difficult than anticipated, but I finally found a good solution.

The problem was that the WASM module, compiled with wasm-pack, didn’t work when imported with NPM and served with Vite in development mode. It did work with Vite in release mode and with Webpack, but my particular configuration seemed jinxed. As I’m not ready to enter the world of pain that is Webpack again, I spent lots of hours researching the problem.

What is Vite?

In case you’re not familiar, Vite is a modern JavaScript bundler. It is much faster than Webpack, requires less configuration, and many things (including TypeScript and SCSS support) work out of the box.

I actually found two problems:

wasm-pack emits JavaScript modules added with wasm_bindgen in a folder called snippets, but this folder isn’t included in the package.json file, so it is ignored when publishing the module to NPM.
Vite does some optimizations even in development mode. The optimized files are put in the node_modules/.vite/deps folder. This broke the dynamic import of the WASM file, because a script in this folder looked for the WASM file in the same folder, but the WASM file wasn’t there.

The solution to the first problem was to manually add the snippets folder to package.json before publishing. It’s not ideal having to edit an auto-generated file, but at least it works.

The solution to the second problem was harder to find: Vite allows disabling optimizations for a particular module with the optimizeDeps.exclude key.

The NPM module is now used on the playground.

Syntax changes

In the last two releases, there were a few syntax changes:

You can now use the ^ and $ anchors instead of Start and End. These symbols are well-known, not only because of regular expressions, but also because vim uses them. This should make your Pomsky expressions more concise.
Pomsky now supports atomic groups!
```
# this is guaranteed to never backtrack
% atomic('integer' | 'insert' | 'in') %
```
Atomic groups can be more performant than regular groups. Note that atomic groups are only available in the Java, PCRE, Ruby and .NET flavors.
Alternations now allow leading pipes, so you can format your expressions more beautifully:
```
| 'Lorem'
| :group(
    | 'ipsum'
    | 'dolor'
    | 'sit'
  )
| 'amet'
```
The [codepoint] and [cp] syntax has been deprecated. Use the built-in Codepoint/C variables instead. This is part of an effort to make the syntax more uniform and intuitive.
[.] has also been deprecated, but currently only issues a warning.

I’m still undecided if this syntax should be removed entirely, or if the dot should be available, even though its behavior can be surprising and Codepoint is usually better.

Improving diagnostics

Good error messages are dear to my heart. The last two releases had several improvements to diagnostics that I want to highlight here:

Pomsky now detects typos. When you misspell a character class, variable name or capturing group, Pomsky will suggest the correct spelling.
Many regex syntax diagnostics were added. Pomsky now recognizes most regex syntax and suggests the equivalent Pomsky syntax.

For example, trying to compile (?<grp> "test") will produce an error with this help message:

Named capturing groups use the :name(...) syntax. Try :grp(...) instead
Pomsky is now able to report multiple errors at once. More changes to the parser are required to make use of this effectively, but in some places it already works.

Reducing the binary size

I put some work into reducing the binary size. The first step was to strip debug symbols. I previously didn’t strip them because I wanted to have useful backtraces in case of a panic. However, I’m now confident enough in Pomsky’s stability that I think we don’t need them. And if you find a panic, it should be reproducible with the same input.

Stripping the binary decreased the CLI’s binary size on Linux from 3.24 MiB to 1.39 MiB.

The parser rewrite had little impact on the CLI’s binary size: It reduced the binary size from 1.39 MiB to 1.34 MiB. However, the impact was more significant for the WASM module: It went from 224 KiB to 177 KiB.

The next step was to replace the clap crate with the much smaller lexopt. This reduced the CLI’s binary size from 1.34 MiB to 894 KiB. Now that here isn’t any low-hanging fruit left, I’m satisfied with the result.

That’s it for today!

Cheers,
Ludwig

Introducing: Pomsky (formerly Rulex)

Mon, 04 Jul 2022 00:00:00 GMT

This year in February, I started a new project, which I named Rulex: A language that is transpiled to regular expressions. A month later, I released the first version of Rulex.

A lot has happened since then, so I wanted to give an update. First, the summary:

Rulex got noticed: It trended on Hacker News and now has over 700 stars on GitHub 🚀

I published my first two security advisories for Rulex on GitHub

I was forced to rename Rulex to Pomsky, because a lawyer claimed that my project violates a registered trademark

Pomsky now has a website with a feature-rich online playground

Pomsky is still in an alpha stage, but it has seen many new features, improvements and bugfixes since its first release.

If you’re using Pomsky, or want to use it, I’d like to hear from you!

What is Pomsky all about?

Regular expressions are a very widespread language; it has implementations in just about every programming language. They’re also very powerful and can come in handy in a variety of situations. Yet, many people dread writing them. Furthermore, decyphering a regex beyond a certain length and complexity is very difficult due to its dense and ambiguous syntax.

Pomsky attempts to solve all the issues that regexes suffer from. Writing Pomsky expressions should be fun and not turn into a maintenance burden. I’ll explain how Pomsky achieves that, but note that Pomsky is not a regex engine. Pomsky is just an alternative syntax that you can use with your regex engine of choice. This has the advantage that Pomsky can be used with all the existing programming languages and APIs that use regexes.

Free-spacing mode

If you want to see what Pomsky looks like, I encourage you to check out the examples on the home page.

The biggest problem with regular expressions is that you have to write everything in a single line without any white space. However, some regex engines have a free-spacing mode, enabled with (?x). In free-spacing mode, spaces and line breaks are ignored, and you can add comments. This is really helpful to make regexes more readable.

Pomsky goes one step further: In Pomsky, free-spacing is the default, and it’s supported for all regex engines, even JavaScript.

No backslash escapes

In theory, the syntax of regexes is quite simple. However, there is one thing that regularly trips people up: There’s a myriad of characters that need to be escaped, and forgetting to escape a character leads to subtle bugs. That’s why Pomsky doesn’t have character escapes. Instead, strings are written in quotes, and you can decide if you prefer double or single quotes.

Compatibility and Unicode support

Even though regex engines such as Java, PCRE or EcmaScript use a similar syntax, there are enough differences that writing portable regexes is difficult. If you want to use the same regex in different engines, you can’t use any of the more advanced features, such as named backreferences or lookbehind, because they either aren’t universally supported, or their behavior or syntax differs slightly between engines.

Pomsky can’t solve this problem entirely. For example, when a regex engine doesn’t support lookbehind, there’s nothing we can do about it. However, Pomsky is able to polyfill some features, so they can be used in regex engines where they aren’t natively supported. For example, Pomsky polyfills named backreferences and proper Unicode support in JavaScript. When using an unsupported feature that Pomsky can’t polyfill, it will show an error message, so you don’t waste time trying it out.

A modern, reimagined syntax

Over the years, a lot of features have been added to regular expressions since they were invented in 1951. However, syntax has been limited by the characters that hold a special meaning in regexes: .+*?^$()[]{}|\. So it’s no surprise that many syntax constructs start with a backslash or with (?. For example, in PCRE, named backreferences use the syntax \k<name>, and comments look like (?# comment).

Pomsky’s syntax is designed with newer features in mind. Furthermore, Pomsky prefers keywords over sigils except for the most commonly used constructs. This makes the syntax much easier to read.

Variables

In Pomsky, you can assign part of an expression to a variable and re-use it as often as you want. This can make more complex expressions much simpler. You can follow the best practice “don’t repeat yourself” (DRY), and document your code in the process by using descriptive variable names.

There is just one limitation: Variables can’t recursively reference themselves. That would cause an infinite loop, since variables are inlined by Pomsky, therefore it is forbidden.

Pomsky’s growth in popularity

After Rulex was posted on Hacker News, it got 200 new stars on GitHub on a single day. Since then, there’s been steady growth, and now it has over 700 stars.

This was really useful for me, because I received hundreds of comments on Reddit and Hacker News by people criticizing, praising and showing alternatives for Rulex. I couldn’t respond to all of them, but reading them gave me a better idea of what people want. It was very encouraging to see other people defending my project against critics. I was also positively surprised that the discussion was quite civil and productive 🙂

My first security advisories

A consequence of the growing popularity was that people started trying out Rulex, and someone even set up fuzzing for it. In case you’re unfamiliar with the concept: A fuzzer is a test harness that tests a program by feeding it with random input, and then systematically altering the input to explore different code paths.

It didn’t take long before the fuzzer found a panic, which is a (usually unrecoverable) error in Rust. After a bit more time, it found another panic and a stack overflow.

After the issues were fixed and the fuzzer had run for hours without bringing up any more panics, I published a patch and disclosed two security advisories a few days later. These have “low severity”, because there’s little chance that they could be exploited. An exploit would have required that a web service accepted and parsed untrusted rulex expressions, which is strongly discouraged by the way!

The good thing is that we are now much more confident in the correctness of the parser. After this incident, the integration test suite has also been improved and extended.

I want to thank Evan Richter for setting up the fuzzer, disclosing the panics to me, and guiding me through the security advisory process.

Renaming Rulex

As announced in my previous post, I had to rename Rulex to Pomsky. Read the announcement to find out the reasons, and why I chose the name “Pomsky”.

If you use the rulex library or the rulex-macro crate, instructions for migrating are in the changelog.

Note that the new AUR package hasn’t been released yet.

The website

Considerable work went into improving the landing page, the documentation and the online playground of Pomsky. When I published the first version, there was an mdbook with documentation. I really like mdbook for its ease of use, but I wanted more customizability, so I rewrote it with Hugo.

The first version of the playground has also since been rewritten in React. It now uses monaco editor, the editor powering VS Code, and supports several IDE features such as intelligent autocompletions and highlighting errors while typing. The playground also supports matching on text now, similar to regexr, although not as advanced.

Where we are now

Although Pomsky is still in an alpha stage, I want to highlight how far we’ve come. In version 0.3.0, two significant new features were introduced: Variables and number ranges. These make Pomsky much more expressive than regular expressions. 0.3.0 also featured relative references. Version 0.4.2 deprecated <%, %>, [codepoint], [cp] and [.] in favor of built-in variables. Several versions also contained bug fixes and improved diagnostics.

These diagnostics are something I’m really proud of:

What’s so special about this? Pomsky recogizes a lot of regular expression syntax, so it can show a more useful help message.

But I think we can do better: I’m going to write a tool that converts any regex to a Pomsky expression. This will make it much easier to migrate to Pomsky.

I also have plans to publish a VS Code extension and a babel plugin, to make the development process with Pomsky even smoother. If you have any more suggestions, I’d like to hear them!

Users, where are you?

To inform decisions about the future development of rulex, it would be useful to know how many people are using Pomsky, and how. I would also like to know from people who want to use Pomsky, but can’t for some reason.

However, we currently don’t have that data. Therefore, I’m asking you, if you’re using Pomsky, or would like to, please write a comment in this issue, or send me an email.

Cheers,
Ludwig

Renaming Rulex

Fri, 01 Jul 2022 13:06:32 GMT

Hello world. This is not what I expected my first blog post to be, but here we are.

I was recently contacted by a lawyer on behalf of Rulex, Inc., who claimed that I’m not allowed to use the name “Rulex” for my project, because they own a registered trademark for the name. This is disappointing because coming up with a good name that isn’t already used is hard. Furthermore, their trademark is registered for software using Artificial Intelligence, so I’m pretty sure that I’m allowed to use it for a project that is unrelated to AI.

However, I don’t want to risk a lawsuit, because that would entail a lot of work, time and stress. Therefore, it is with a heavy heart that I have decided to comply with the demands of Rulex, Inc. and rename my project.

What is the new name?

While I was brainstorming new names, I was reminded of pug. For those who don’t know it or weren’t aware, pug was once called jade. In 2016, they were forced to change the name, because some company has a trademark for the word “jade” referring to software. This gave me the idea to also name my project, like pug, after a dog breed.

The breed I settled on is the Pomsky, a hybrid of a Siberian Husky and a Pomeranian. I think this is a great name for several reasons. First, it’s short, only 1 letter longer than rulex. There is no company or registered trademark named “Pomsky”. And the best part, our project now has a really cute mascot 😀

It’s also nice that Pomsky rhymes with Noam Chomsky, a famous linguist. Chomsky influenced the field of linguistics like no other, and he has also worked with formal languages and formal grammars. He came up with Chomsky’s hierarchy, which categorizes formal languages by the strictness of their grammar:

Regular grammars
Context-free grammars
Context-sensitive grammars
Unrestricted

Regular grammars are the most strict, and can always be parsed with a regular expression. And this makes Chomsky’s work relevant for rulex — errrr, Pomsky!

The roadmap

Renaming rulex to Pomsky is quite a bit of work, and can be tracked here. The name appears in a lot of different places, and during the migration we have to be careful not to break anyone’s workflow or invalidate the old website URLs. The rulex-rs.github.io subdomain will redirect to the new website. Furthermore, crates on crates.io are immutable and can’t be renamed, so we have to release new crates using the new name, and we have to add disclaimers to the existing crates that users have to switch to the new ones.

If you have questions about the renaming process, you can ask them in this issue.

Have a great day!

Cheers,
Ludwig