[RFC] Named macro capture groups

[tgross35]

This RFC proposes optional names for repetition groups in macros:

macro_rules! foo {
    ( $group1( $a:ident ),+ ) => {
        $group1( println!("{}", $a); )+
    }
}

Rendered

Small Pre-RFC: https://internals.rust-lang.org/t/pre-rfc-named-capture-groups-for-macros/20883

[RalfJung]

Aren't there still some subtle questions around nesting one has to worry about? Specifically, the RFC doesn't seem to say anything about nested cases like ( $group1:( $a:ident [ $group2:( $b:ident ),+ ] ),+ ). Presumably I will only be allowed to use $group2 inside a $group1(...) repetition? But there can be arbitrarily many other repetitions "in between" -- group2 will always refer to the repetition instance in the "current" group1?

[davidhewitt]

Although I cannot remember the exact use-case, I remember wanting something like this to name an optional trailing comma so that I could forward it on to another macro as part of the expansion. 👍

[tgross35]

Aren't there still some subtle questions around nesting one has to worry about? Specifically, the RFC doesn't seem to say anything about nested cases like ( $group1:( $a:ident [ $group2:( $b:ident ),+ ] ),+ ). Presumably I will only be allowed to use $group2 inside a $group1(...) repetition? But there can be arbitrarily many other repetitions "in between" -- group2 will always refer to the repetition instance in the "current" group1?

Great point, the scope part is tricky - I think what you said is correct. It is probably best if I just add an appendix of examples that do and don't work with reasoning, especially given your comments on Zulip about the existing macro mental model not being very clear (I completely agree).

[RalfJung]

A bunch of examples would be great, but they are not substitute for also describing the general rules that they all follow from.

[kennytm]

when you write this I thought you are defining a metavariable $group1 which matches a comma-separated list of $a:ident, and using $group1 will expand to foo,bar,baz,etc.

but no the $group1 is just a label 🤔

the rationale mentioned using : is to be "to be more consistent with existing fragment specifiers" but IMO similarity with fragment specifiers is exactly why this RFC should not use the current syntax.

i prefer getting rid of : (my main source of confusion) and use a label syntax like

macro_rules! make_functions {
    ( 
        names: [ $'names $($name:ident),+ ],
//               ^~~~~~~~~
        $'greetings $( greeting: $greeting:literal, )?
    ) => {
        $'names $(
            fn $name() {
                println!("function {} called", stringify!($name));
                $'greetings $(println!("{}", $greeting) )?
            }
        )+
    }
}


macro_rules! innermost1 {
    ( $'outer_rep $($a:ident: $'inner_rep $($b:literal),* );+ ) => {
        [$'outer_rep $( $'inner_rep $( ${index($'outer_rep)}, )* )+]
    };
}

[petrochenkov]

Metavariables matching some parts of the input are an interesting idea in general.
If we also introduce "match exactly once repetitions", then we'll also be able to capture just arbitrary token streams.
It would be nice to also be able to bind a name to "repeated exactly one" groups aka just artibtrary substreams in the input.

// macro LHS, capture "exactly once" repetition
$my_tokens:(a b $var c d)①

// macro RHS, reemit `a b value_for_var c d`
$my_tokens

[tgross35]

Thanks for the ideas - I updated the default syntax to be $group1(...) (no colon) and added the label syntax as a possibility.

It would be nice to also be able to bind a name to "repeated exactly one" groups aka just artibtrary substreams in the input.

Is there a standard kleene for a single capture, or could we get by with no kleene at all? I could probably include a proposal for single captures in this RFC, to be accepted as part of it or rejected separately.

[RalfJung]

In regexes, a "capture exactly once" group is just (...).

[tgross35]

After writing some more examples, it seems like "capture once" and "emit the entire capture" would be pretty useful, so I just included them as part of the proposal. Especially for code that just needs to be validated and passed elsewhere, or matching and reemitting optional keywords/tts like mut, & or pub(crate) (just allowing capture groups that don't contain metavars is enough for that, but $pub_crate in the expansion is nicer to read than $pub_crate(pub(crate))).

[petrochenkov]

One advantage of the reemitting is that it will use spans from the input, instead of spans from the macro body.

One example I recently observed in rust-lang/rust#119412 is the mir! macro in standard library.
It will match on something like let $expr = $expr; and then reemit $expr = $expr; as an assignment expression.
If = is emitted manually from the macro it will get the macro span and may mess up the combined span of the assignment expression.
If = is taken from the input, then the expression will get a precise span from the input, which is good for any kind of DSLs.

[c410-f3r]

cc @markbt

[nikomatsakis]

Skimmed this: I feel positive on this direction.

[joshtriplett]

I love this specific point, and have wanted this in the past.

[joshtriplett]

One potential source of ambiguity:

How would you express "I want to expand the entire named group, followed by some tokens in parentheses"?

I have a suggestion for how to solve that: ${$group}(tokens).

[joshtriplett]

🎉 This is a great sign that this design is a good idea.

[joshtriplett]

@tgross35 I love this approach. This seems cleaner, nicely orthogonal, and would fit very well with macro fragment fields.

[joshtriplett]

I can see the argument for either $group:(...) or $group(...). The former seems nicely analogous to existing macro fragments ($x:ident $g:(...) defines two captures with their types), while the latter has less syntax, still parses unambiguously, and has the advantage that the usage of a group matches the input syntax.

[steffahn]

Syntax like e.g. $identifier( /* … */ ), is already valid and useful in macro expansions.

macro_rules! call_many_fuctions {
    ([$($($function_name:ident),+$(,)?)?]($arg:expr)) => {
        [
            $($(
                $function_name($arg),
            )+)?
        ]
    }
}
fn foo(x: i32) -> i32 { x }
fn bar(x: i32) -> i32 { x + 1 }
fn baz(x: i32) -> i32 { x * x }
fn qux(x: i32) -> i32 { x / 2 }

fn main() {
    println!("{:?}", call_many_fuctions!([foo, bar, baz, qux](100)));
}

(this concrete example doesn’t follow it up with a *, + or ?, but these aren’t forbidden either, of course)

If this RFC proposes to make the “parsing” of the macro RHS depend on the “type” of the fragment, I think that’s a clear possible disadvantage.

Especially with the (IMO very useful) idea that $group on its own can already stand in for an expansion of the whole group’s contents, so expanding some identifier $function_name – or expression, pattern, etc… – doesn’t work different from expanding some group $group – it seems desirable (to me) if $function_name($arg), *etc… doesn’t have a severely different meaning from $group($arg), *etc… either.

[kennytm]

the design that $group does not expand is primarily why i suggested in #3649 (comment) that it should use a separate syntax $'group.

[tgross35]

These are great points. I wish we could use the same syntax for metavars and groups, but the $'group syntax does get us out of the problems here, so I'll use that for the implementation and leave an unresolved question.

[bjorn3]

What if you want to use () after the group in the expansion? That would be interpreted as replacing the contents of the capture, right? Maybe it could be written as $...group or something like that?

[PoignardAzur]

As mentioned by other reviewers, the expansion-side macro is already valid.

I'd like to suggest an alternative:

macro_rules! foo {
    ( $group1( $a:ident ),+ ) => {

        ${for a in group1} (
            println!("{}", $a);
        )
    }
}

This alternative is more verbose, but it's also much more explicit about what's going on.

[PoignardAzur]

Or for something closer to existing syntax:

        ${for(group1)} (
            println!("{}", $a);
        )+

[PoignardAzur]

Something I like about everything else in the RFC is that capture groups are mostly distinct from their contents, and mostly serve as explicit scope. Being able to do $capture_group kind of loses that.

I think you should use an explicit syntax for emitting capture groups, maybe a metavariable like ${paste(capture_group)}:

macro_rules! rename_fn {
    (
        $newname:ident;
        $pub(pub)? fn $oldname:ident( $args( $mut(mut)? $arg:ident: $ty:ty )* );
    ) => {
        $pub fn $newname( ${paste(args)} );
    }
}

[chrisbouchard]

If exactly-once groups don't have a kleene operator to terminate the group matcher, how would a macro author define a rule that matches an exactly-once group followed by one of the tokens *, +, or ??

For example, in the following (admittedly kind of silly) example, I want to specify a multiplication-like syntax for my macro. But I think the * would be attached to the group instead, silently turning it into a zero-or-more group.

macro_rules! example1 {
    ( $g1($a:ident[$b:expr]) * $g2($c:ident[$d:expr]) ) => { $g1 $g2 }
}

example1!(x[1] * y[2]);  // ERROR: Something something unexpected `*`...

Further, a kleene-looking token could “reach back” across an intermediate token, interpreting it as a separator. I think this would be extra surprising, because an author may edit a macro definition and unintentionally change the the nature of the group at a distance.

For example, the following example would specify an exactly-once group.

macro_rules! example2 {
    ( $g($a:ident $b:ident) x y + ) => { $g }
}

example2!(foo bar x y +);  // Ok

But if I later edited it to remove the y token, it would silently become a one-or-more group with x as the separator.

macro_rules! example2 {
    ( $g($a:ident $b:ident) x + ) => { $g }
}

example2!(foo bar x +);  // ERROR: Something about `+` is not an ident and/or unexpected end of input...

For this reason, I don't think it would be a satisfying solution to just add an escape syntax for *, +, and ?. It would work, but it would still lead to gotchas like this where the macro author would have to

• Notice that their macro has broken, since the updated rule is still valid even if it doesn't match what they meant, and simple invocations may still work;
• Then reason out what went wrong from possibly-unhelpful error messages, before eventually slapping their head and adding an escape.

I don't think the compiler could help much here, because again these are all valid matchers. The compiler wouldn't know the author wanted an exactly-once group.

I think it would be better to introduce a new “kleene” operator for this new group type. (Scare quotes since Kleene didn't actually define a regular language operator like that, but he wasn't concerned with saving sub-matches anyway.) There isn't any symbol that I love for the job—maybe ! or =?

[chrisbouchard]

I do think there would be some backwards compatibility concerns with my suggestion if we allowed unnamed group matchers to use the new exactly-once operator. For example, consider the following macro definition.

macro_rules! example {
    ( $( $a:ident )!* ) => { $( $a ) }
}

Today, the rule matches zero or more idents with ! as the separator. If we made ! the new exactly-once operator, then I think this rule would instead match exactly one ident followed by *.

Maybe we only allow exactly-once groups with the new named syntax? (That might actually be a good idea regardless what syntax we use, since exactly-once groups are kind of useless without named groups.) But then would it be too surprising if !* meant different things for named and unnamed groups?

[chrisbouchard]

I realized my example in the previous comment wasn't quite right, because the group expansion in the transcriber needs an operator too. My example should have been written as follows, with a * in the transcriber.

macro_rules! example {
    ( $( $a:ident )!* ) => { $( $a )* }
}

Then I think expanding an exactly-once group using a zero-or-more operator should be an error. Allowing a new exactly-once operator with unnamed groups would still a backwards-compatibility concern, but it wouldn't be a silent backwards compatibility concern. (Nope. See below.)

---

I'm actually a bit unclear here, though. The reference for Macros by Example says the following about repetitions:

1. A metavariable must appear in exactly the same number, kind, and nesting order of repetitions in the transcriber as it did in the matcher. So for the matcher $( $i:ident ),*, the transcribers => { $i }, => { $( $( $i)* )* }, and => { $( $i )+ } are all illegal, …

— https://doc.rust-lang.org/reference/macros-by-example.html#r-macro.decl.repetition.fragment

However, it doesn't seem like this is currently enforced. (Playground example mixing * with +, and another mixing * with ?.) I don't know whether that's just a bug, or whether the reference is out of date.

Edit: Bleh, it's not a bug, but it's not exactly allowed either. It was reported in (at least) rust-lang/rust#103256 and the answer is there's an allow-by-default lint meta_variable_misuse that would catch it, but they don't want to make it warn- or deny-by-default because there are too many false positives.

---

I guess this is all kind of moot if we just don't allow exactly-once unnamed groups. And maybe for named groups, at least, things will be unambiguous enough to let us enforce that Kleene operators must match between matcher and transcriber.

[chrisbouchard]

I commented elsewhere about concerns with the proposed syntax for exactly-once groups. But with the current proposal, I think the grammar node would need the trailing MacroRepSep-MacroRepOp sequence to be optional.

Suggested change

	> `$` ( IDENTIFIER_OR_KEYWORD except crate | RAW_IDENTIFIER | _ )<sup>?</sup> ( _MacroMatch<sup>+</sup>_ ) _MacroRepSep_<sup>?</sup> _MacroRepOp_
	> `$` ( IDENTIFIER_OR_KEYWORD except crate | RAW_IDENTIFIER | _ )<sup>?</sup> `(` _MacroMatch<sup>+</sup>_ `)` ( _MacroRepSep_<sup>?</sup> _MacroRepOp_ )<sup>?</sup>

(I also added missing backticks around the literal parentheses tokens that delimit the group body.)

[chrisbouchard]

Suggested change

	> `$` ( _MacroMatch<sup>+</sup>_ ) _MacroRepSep_<sup>?</sup> _MacroRepOp_
	> `$` `(` _MacroMatch<sup>+</sup>_ `)` _MacroRepSep_<sup>?</sup> _MacroRepOp_

[traviscross]

We talked about this in the lang design meeting today. We were enthusiastic about this and in general agreement on the major design points.

Given that, @tgross35 is planning to pursue landing an implementation as a lang experiment. The implementation experience should help us to answer the remaining unresolved questions on some details such that this RFC can be updated to represent the likely-final user-facing behavior. With that, we can accept this RFC, and then hopefully stabilize it soon thereafter.

Thanks to @tgross35 for joining us in the meeting and for putting forward this well motivated and well written RFC.

[juntyr]

When reading, I felt the args repetition was missing a separator, e.g. ,?

Suggested change

	$pub(pub)? fn $oldname:ident( $args( $mut(mut)? $arg:ident: $ty:ty )* );
	$pub(pub)? fn $oldname:ident( $args( $mut(mut)? $arg:ident: $ty:ty ),* );