feat: Redesign the architecture of the plugin system

[nfrankel]

Issue description

Current situation

At the moment, you add a plugin in the request flow by referencing with its name and setting the configuration. The plugin is effectively a singleton.

This approach has some downsides:

• You can use a plugin only once in the request flow. Users cannot add the same plugin multiple times with different configurations.
• For maintainers, it means that they need to duplicate code and reference it with different names and priorities. For example, the ext-plugin-pre-req and the ext-plugin-post-req do the same - run arbitrary Lua code.
• Users cannot change the priority of a plugin, it's hard-coded (see feat: As a User, I want to be able to configure the priority of each plugin so I don't need to change the source code #6504)
• Plugin developers need to handle every possible configuration case (see request help: How to current limit both in minutes and day by using plugin limit-count #6197)

Proposal

In Object-Oriented speak, make Plugin a class and introduce PluginInstance "objects" (or something similarly named):

• Plugin defines the code
• PluginInstance references the Plugin and associates it with a priority and a configuration

Decoupling the code from the configuration/priority would bring much more freedom for users.

[liangliang4ward]

yes! Some time I want use a plugin to config diffent properties.
just like #6527

[tokers]

@nfrankel Could you give a configuration snippet about how to configuring plugins when using the new arch?

Also, we need to evaluate the cost for the re-design, such as how to refactor plugins so that plugin codes are reentrant.

[nfrankel]

@tokers Would something like that be possible?

curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
  "plugins": [
    "plugin": {
      "id": "ext-plugin-req",
      "priority": 500,
      "conf": [{
        "name": "ext-plugin-A",
        "value": "{\"enable\":\"feature\"}"}
      ]},
    },
    "plugin": {
      "id": "ext-plugin-req",
      "priority": -500,
      "conf": [{
        "name": "ext-plugin-B",
        "value": "{\"disable\":\"feature\"}"}
      ]},
    }
  ]
}'

[tokers]

Got it.

But since we use the array to orchestrate plugins, I think we can remove the priority field.

[nfrankel]

But since we use the array to orchestrate plugins, I think we can remove the priority field.

I don't think so. AFAIK, plugins can be set on different levels (Route or Service). Hence, you need a way to merge their orders - be explicit about it. Does it make sense or am I wrong?

[tokers]

But since we use the array to orchestrate plugins, I think we can remove the priority field.

I don't think so. AFAIK, plugins can be set on different levels (Route or Service). Hence, you need a way to merge their orders - be explicit about it. Does it make sense or am I wrong?

IMHO that's confusing, the intuitive order is Route > Service, if we let users decide the order, it's error-prone.

[nfrankel]

Imagine the user has defined a Service with plugin A and priority 5. Now, they define a route with plugin B and C.

How would you compute the order based on the array?

[tokers]

Imagine the user has defined a Service with plugin A and priority 5. Now, they define a route with plugin B and C.

How would you compute the order based on the array?

That's tricky. We may think about another way to allow a plugin can be run multiple times but still use the object to store their config. I'm wondering if there are some circumstances that people want to run the same plugin in a single route but not continuously (e.g. run plugin A, plugin B, plugin A).

[tokers]

Imagine the user has defined a Service with plugin A and priority 5. Now, they define a route with plugin B and C.
How would you compute the order based on the array?

That's tricky. We may think about another way to allow a plugin can be run multiple times but still use the object to store their config. I'm wondering if there are some circumstances that people want to run the same plugin in a single route but not continuously (e.g. run plugin A, plugin B, plugin A).

If not, what about the following way:

{
    "uri": "/api/*",
    "host": "foo.com",
   "plugins": {
       "limit-count": [
          {
             "time_window": 60,
             "count": 10,
             "key": "remote_addr"
          },
          {
             "time_window": 1,
             "count": 5,
             "key": "route_id"
          }
       ] 
    }
}

[liangliang4ward]

Imagine the user has defined a Service with plugin A and priority 5. Now, they define a route with plugin B and C.
How would you compute the order based on the array?

That's tricky. We may think about another way to allow a plugin can be run multiple times but still use the object to store their config. I'm wondering if there are some circumstances that people want to run the same plugin in a single route but not continuously (e.g. run plugin A, plugin B, plugin A).

If not, what about the following way:

{
    "uri": "/api/*",
    "host": "foo.com",
   "plugins": {
       "limit-count": [
          {
             "time_window": 60,
             "count": 10,
             "key": "remote_addr"
          },
          {
             "time_window": 1,
             "count": 5,
             "key": "route_id"
          }
       ] 
    }
}

Is it possible to support more flexible configuration methods? For example, hamc-auth supports multiple configurations and uses the or mode

[tzssangglass]

hamc-auth supports multiple configurations and uses the or mode

I think it's not that hmac-auth itself supports or, but that the plugin orchestration system supports or for the same plugin.

[liangliang4ward]

I think it's not that hmac-auth itself supports or, but that the plugin orchestration system supports or for the same plugin.

yes. but may be not the same plugin. perhaps the same type.

[membphis]

hi @nfrankel , I create a new example follow your way. I think it looks good now.

$ curl -i https://my.host/apisix/admin/plugin_configs/1  -H 'X-API-KEY: abc' -X PUT -d '{
    "desc": "blah",
    "plugins": [
        {
            "id": "limit-count",
            "priority": 500,
            "conf": [
                {
                    "time_window": 60,
                    "count": 10,
                    "key": "remote_addr"
                },
                {
                    "time_window": 1,
                    "count": 5,
                    "key": "route_id"
                }
            ]
        },
        {
            "id": "ip-restriction",
            "priority": 200,
            "conf": [
                {
                    "whitelist": [
                        "127.0.0.1",
                        "113.74.26.106/24"
                    ]
                }
            ]
        }
    ]
}'

[nfrankel]

@membphis LGTM 🙂