Skip to content

Commit b3e1c80

Browse files
JinXingYoungJinXingYoung
committed
Proposal: Adapt Multiple Type of Database for Harbor
Update support-multiDB.md Basic structure add png feat: db proposal initial verion (#2) Update the figure Update adapt_multiple_type_of_database.md (#3) * Update adapt_multiple_type_of_database.md Some update * Update adapt_multiple_type_of_database.md Update authors Update adapt_multiple_type_of_database.md Signed-off-by: JinXingYoung <[email protected]>
1 parent 29b42cd commit b3e1c80

File tree

7 files changed

+399
-0
lines changed

7 files changed

+399
-0
lines changed

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Workgroup is a virtual team of aggregating the efforts of all the interested par
1919
|[Multi Architecture](https://github.com/goharbor/community/tree/master/workgroups/wg-multiarch)|The purpose of this working group is to enable Harbor to support multiple architectures, such as x86 arm.|ZhiPeng Yu@Alauda ([yuzp1996](https://github.com/yuzp1996))| Steven Zou@VMware ([steven-zou](https://github.com/steven-zou)) |[8 Members](https://github.com/goharbor/community/tree/master/workgroups/wg-multiarch#members)|
2020
|[Performance](https://github.com/goharbor/community/tree/master/workgroups/wg-performance)|The purpose of this work group is to improve the performance and scalability of harbor in the use case of large data volumes.|ChenYu Zhang@Alauda ([chlins](https://github.com/chlins))| Steven Zou@VMware ([steven-zou](https://github.com/steven-zou))/ Daniel Morinigo@Alauda ([danielfbm](https://github.com/danielfbm)) |[11 Members](https://github.com/goharbor/community/tree/master/workgroups/wg-performance#members)|
2121
|[Image Acceleration](https://github.com/goharbor/community/tree/master/workgroups/wg-image-accel)|The objective of this working group is to aggregate efforts to discuss, design and finally integrate accelerated container image format support in Harbor.|Bo Liu@AlibabaCloud ([liubogithub](https://github.com/liubogithub))|Steven Zou@VMware ([steven-zou](https://github.com/steven-zou))/Wang Yan@VMWare ([wy65701436](https://github.com/wy65701436))|[7 members](https://github.com/goharbor/community/tree/master/workgroups/wg-image-accel#members)|
22+
|[Databases](https://github.com/goharbor/community/tree/master/workgroups/wg-databases)|The objective of this working group is to aggregate efforts to support different databases in Harbor in addition to PostgreSQL.|Yvonne@Tencent[@JinXingYoung](https://github.com/JinXingYoung)|Yiyang Huang@ByteDance[@hyy0322](https://github.com/hyy0322)|[3 members](https://github.com/goharbor/community/tree/master/workgroups/wg-databases#members)|
2223

2324
## Structure
2425

‎proposals/adapt_multiple_type_of_database.md

Lines changed: 382 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,382 @@
1+
# Proposal: `Adapt Multiple Type of Database for Harbor`
2+
3+
Author:
4+
5+
- Yvonne [@JinXingYoung](https://github.com/JinXingYoung)
6+
- Yiyang Huang [@hyy0322](https://github.com/hyy0322)
7+
- Minglu [@ConnieHan2019]([email protected])
8+
9+
Links:
10+
11+
- Previous Discussion: [goharbor/harbor#6534](https://github.com/goharbor/harbor/issues/6534)
12+
- Related PR from Others Before: [goharbor/harbor#14265](https://github.com/goharbor/harbor/pull/14265)
13+
14+
## Abstract
15+
16+
Propose to support other databases in Harbor, the first step is to support MySQL/MariaDB. This proposal introduces an abstract DAO layer in Harbor, different database can have their drivers to implement the interface. So that Harbor can adapt to other databases as long as a well tested database driver provided.
17+
18+
19+
## Background
20+
21+
As previous discussion([goharbor/harbor#6534](https://github.com/goharbor/harbor/issues/6534)) shown, there are certain amount of users(especially in China) lack the experiences of maintaining PostgreSQL for HA, disaster recovery, etc. And meanwhile they are more familiar with other databases such as MySQL/MariaDB. They prefer to use MariaDB/MySQL instead of PostgreSQL to keep their production environments stable.
22+
23+
As we all know that Harbor used MySQL before. But scanner clair use PostgreSQL as database. In order to keep consistency with clair and reduce maintenance difficulties. Harbor unified database using PostgreSQL.
24+
25+
Since Harbor v2.0 use trivy as default scanner instead of clair，there's no strongly requirement to use PostgreSQL anymore. Therefore, it is possible to adapt different kind of database now.
26+
27+
## Proposal
28+
29+
Support other databases in Harbor other than PostgreSQL.
30+
31+
### Goals
32+
33+
- Keep using PostgreSQL as default database. The Implementation will be compatible with current version of Harbor.
34+
- Abstract DAO layer for different type of databases.
35+
- Support MariaDB(10.5.9), MySQL(8.0) by implementing corresponding drivers and resolving sql compatibility.
36+
- Provide migration tool or guide for users to migrate data from PostgreSQL to MariaDB/MySQL.
37+
38+
### Non-Goals
39+
40+
- Support other type of database, such as MongoDB, Oracle.
41+
- Implement Mariadb/MySQL operator for internal database case.
42+
43+
## Implementation
44+
45+
### Overview
46+
47+
![overview.png](images/multidb/overview.png)
48+
49+
![initdb.png](images/multidb/initdb.png)
50+
51+
![dbdriver.png](images/multidb/dbdriver.png)
52+
53+
### Component Detail
54+
55+
**Common**
56+
57+
Add mysql config settings: src/lib/config/metadata/metadatalist.go
58+
```go
59+
var (
60+
// ConfigList - All configure items used in harbor
61+
// Steps to onboard a new setting
62+
// 1. Add configure item in metadatalist.go
63+
// 2. Get/Set config settings by CfgManager
64+
// 3. CfgManager.Load()/CfgManager.Save() to load/save from configure storage.
65+
ConfigList = []Item{
66+
...
67+
{Name: common.MySQLDatabase, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_DATABASE", DefaultValue: "registry", ItemType: &StringType{}, Editable: false},
68+
{Name: common.MySQLHOST, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_HOST", DefaultValue: "mysql", ItemType: &StringType{}, Editable: false},
69+
{Name: common.MySQLPassword, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_PASSWORD", DefaultValue: "root123", ItemType: &PasswordType{}, Editable: false},
70+
{Name: common.MySQLPort, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_PORT", DefaultValue: "3306", ItemType: &PortType{}, Editable: false},
71+
{Name: common.MySQLUsername, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_USERNAME", DefaultValue: "root", ItemType: &StringType{}, Editable: false},
72+
{Name: common.MySQLMaxIdleConns, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_MAX_IDLE_CONNS", DefaultValue: "2", ItemType: &IntType{}, Editable: false},
73+
{Name: common.MySQLMaxOpenConns, Scope: SystemScope, Group: DatabaseGroup, EnvKey: "MYSQL_MAX_OPEN_CONNS", DefaultValue: "0", ItemType: &IntType{}, Editable: false},
74+
...
75+
}
76+
```
77+
78+
Add DB type in configs: make/photon/prepare/utils/configs.py
79+
```
80+
if external_db_configs:
81+
config_dict['external_database'] = True
82+
# harbor db
83+
config_dict['harbor_db_type'] = external_db_configs['harbor']['type']
84+
config_dict['harbor_db_host'] = external_db_configs['harbor']['host']
85+
config_dict['harbor_db_port'] = external_db_configs['harbor']['port']
86+
config_dict['harbor_db_name'] = external_db_configs['harbor']['db_name']
87+
config_dict['harbor_db_username'] = external_db_configs['harbor']['username']
88+
...
89+
```
90+
91+
MySQL struct: src/common/models/database.go
92+
```go
93+
type MySQL struct {
94+
Host string `json:"host"`
95+
Port int `json:"port"`
96+
Username string `json:"username"`
97+
Password string `json:"password,omitempty"`
98+
Database string `json:"database"`
99+
MaxIdleConns int `json:"max_idle_conns"`
100+
MaxOpenConns int `json:"max_open_conns"`
101+
}
102+
```
103+
104+
Get database infos for different db types: src/common/dao/base.go
105+
```go
106+
func getDatabase(database *models.Database) (db Database, err error) {
107+
switch database.Type {
108+
case "", "postgresql":
109+
db = NewPGSQL(
110+
database.PostGreSQL.Host,
111+
strconv.Itoa(database.PostGreSQL.Port),
112+
database.PostGreSQL.Username,
113+
database.PostGreSQL.Password,
114+
database.PostGreSQL.Database,
115+
database.PostGreSQL.SSLMode,
116+
database.PostGreSQL.MaxIdleConns,
117+
database.PostGreSQL.MaxOpenConns,
118+
)
119+
case "mariadb", "mysql":
120+
db = NewMySQL(
121+
database.MySQL.Host,
122+
strconv.Itoa(database.MySQL.Port),
123+
database.MySQL.Username,
124+
database.MySQL.Password,
125+
database.MySQL.Database,
126+
database.MySQL.MaxIdleConns,
127+
database.MySQL.MaxOpenConns,
128+
)
129+
default:
130+
err = fmt.Errorf("invalid database: %s", database.Type)
131+
}
132+
return
133+
}
134+
```
135+
136+
MigrateDB with certain sql file: src/migration/migration.go
137+
```go
138+
// MigrateDB upgrades DB schema and do necessary transformation of the data in DB
139+
func MigrateDB(database *models.Database) error {
140+
var migrator *migrate.Migrate
141+
var err error
142+
143+
// check the database schema version
144+
switch database.Type {
145+
case "", "postgresql":
146+
migrator, err = dao.NewMigrator(database.PostGreSQL) // migrate db with postgres sql file
147+
case "mariadb", "mysql":
148+
migrator, err = dao.NewMysqlMigrator(database.MySQL) // migrate db with mysql sql file
149+
}
150+
151+
...
152+
}
153+
```
154+
155+
Migration sql file architecture
156+
```
157+
harbor/
158+
├── make
159+
│ ├── migrations
160+
│ │ ├── mysql
161+
│ │ │ ├── 0001_initial_schema.up.sql // File content may different with postgresql
162+
│ │ │ ├── ...
163+
│ │ ├── postgresql
164+
│ │ │ ├── 0001_initial_schema.up.sql
165+
│ │ │ ├── ...
166+
```
167+
168+
DAO layer make different databases compatible. We can extend database support by implementing mysql implementation.
169+
170+
DAO interface in src/pkg/repository/dao/dao.go
171+
```go
172+
// DAO is the data access object interface for repository
173+
type DAO interface {
174+
// Count returns the total count of repositories according to the query
175+
Count(ctx context.Context, query *q.Query) (count int64, err error)
176+
// List repositories according to the query
177+
List(ctx context.Context, query *q.Query) (repositories []*model.RepoRecord, err error)
178+
// Get the repository specified by ID
179+
Get(ctx context.Context, id int64) (repository *model.RepoRecord, err error)
180+
// Create the repository
181+
Create(ctx context.Context, repository *model.RepoRecord) (id int64, err error)
182+
// Delete the repository specified by ID
183+
Delete(ctx context.Context, id int64) (err error)
184+
// Update updates the repository. Only the properties specified by "props" will be updated if it is set
185+
Update(ctx context.Context, repository *model.RepoRecord, props ...string) (err error)
186+
// AddPullCount increase pull count for the specified repository
187+
AddPullCount(ctx context.Context, id int64, count uint64) error
188+
// NonEmptyRepos returns the repositories without any artifact or all the artifacts are untagged.
189+
NonEmptyRepos(ctx context.Context) ([]*model.RepoRecord, error)
190+
}
191+
```
192+
193+
**Core**
194+
195+
make/photon/prepare/templates/core/env.jinja:
196+
```
197+
DATABASE_TYPE={{harbor_db_type}}
198+
{% if ( harbor_db_type == "mysql" or harbor_db_type == "mariadb" ) %}
199+
MYSQL_HOST={{harbor_db_host}}
200+
MYSQL_PORT={{harbor_db_port}}
201+
MYSQL_USERNAME={{harbor_db_username}}
202+
MYSQL_PASSWORD={{harbor_db_password}}
203+
MYSQL_DATABASE={{harbor_db_name}}
204+
MYSQL_MAX_IDLE_CONNS={{harbor_db_max_idle_conns}}
205+
MYSQL_MAX_OPEN_CONNS={{harbor_db_max_open_conns}}
206+
{% else %}
207+
POSTGRESQL_HOST={{harbor_db_host}}
208+
POSTGRESQL_PORT={{harbor_db_port}}
209+
POSTGRESQL_USERNAME={{harbor_db_username}}
210+
POSTGRESQL_PASSWORD={{harbor_db_password}}
211+
POSTGRESQL_DATABASE={{harbor_db_name}}
212+
POSTGRESQL_SSLMODE={{harbor_db_sslmode}}
213+
POSTGRESQL_MAX_IDLE_CONNS={{harbor_db_max_idle_conns}}
214+
POSTGRESQL_MAX_OPEN_CONNS={{harbor_db_max_open_conns}}
215+
{% endif %}
216+
```
217+
218+
**Exporter**
219+
220+
make/photon/prepare/templates/exporter/env.jinja
221+
```
222+
HARBOR_DATABASE_TYPE={{harbor_db_type}}
223+
HARBOR_DATABASE_HOST={{harbor_db_host}}
224+
HARBOR_DATABASE_PORT={{harbor_db_port}}
225+
HARBOR_DATABASE_USERNAME={{harbor_db_username}}
226+
HARBOR_DATABASE_PASSWORD={{harbor_db_password}}
227+
HARBOR_DATABASE_DBNAME={{harbor_db_name}}
228+
HARBOR_DATABASE_SSLMODE={{harbor_db_sslmode}}
229+
```
230+
231+
**JobService**
232+
233+
JobService get configs from Core, then initDatabase with db config obtained.
234+
src/pkg/config/manager.go
235+
```go
236+
// GetDatabaseCfg - Get database configurations
237+
func (c *CfgManager) GetDatabaseCfg() *models.Database {
238+
ctx := context.Background()
239+
database := &models.Database{}
240+
database.Type = c.Get(ctx, common.DatabaseType).GetString()
241+
242+
switch database.Type {
243+
case "", "postgresql":
244+
postgresql := &models.PostGreSQL{
245+
Host: c.Get(ctx, common.PostGreSQLHOST).GetString(),
246+
Port: c.Get(ctx, common.PostGreSQLPort).GetInt(),
247+
Username: c.Get(ctx, common.PostGreSQLUsername).GetString(),
248+
Password: c.Get(ctx, common.PostGreSQLPassword).GetString(),
249+
Database: c.Get(ctx, common.PostGreSQLDatabase).GetString(),
250+
SSLMode: c.Get(ctx, common.PostGreSQLSSLMode).GetString(),
251+
MaxIdleConns: c.Get(ctx, common.PostGreSQLMaxIdleConns).GetInt(),
252+
MaxOpenConns: c.Get(ctx, common.PostGreSQLMaxOpenConns).GetInt(),
253+
}
254+
database.PostGreSQL = postgresql
255+
case "mariadb", "mysql":
256+
mysql := &models.MySQL{
257+
Host: c.Get(ctx, common.MySQLHOST).GetString(),
258+
Port: c.Get(ctx, common.MySQLPort).GetInt(),
259+
Username: c.Get(ctx, common.MySQLUsername).GetString(),
260+
Password: c.Get(ctx, common.MySQLPassword).GetString(),
261+
Database: c.Get(ctx, common.MySQLDatabase).GetString(),
262+
MaxIdleConns: c.Get(ctx, common.MySQLMaxIdleConns).GetInt(),
263+
MaxOpenConns: c.Get(ctx, common.MySQLMaxOpenConns).GetInt(),
264+
}
265+
database.MySQL = mysql
266+
}
267+
268+
return database
269+
}
270+
```
271+
272+
### Migration
273+
274+
#### Write a document Using official migration tool
275+
276+
MySQL community provide a [tool](https://dev.mysql.com/doc/workbench/en/wb-migration-database-postgresql.html) to migrate data.(Need further test)
277+
278+
#### Write a script to migrate data.
279+
280+
1. Define data models for postgreSQL and MariaDB/MySQL.
281+
2. Read data from postgreSQL and map to data model.
282+
3. Transfer postgreSQL data model to MariaDB/MySQL data model.
283+
4. Write data to MariaDB/MySQL database
284+
285+
### Database Compatibility Testing
286+
287+
**MySQL 8.0**
288+
289+
We have done some test for SQL compatibility. Here we list some SQL Incompatible points.
290+
291+
- TRIGGER is not needed in MySQL. Just use default CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP.
292+
- SERIAL type in MySQL is bigint unsigned. So a column reference other column type SERIAL must define as bigint unsigned.
293+
- Alter column type in MySQL must use MODIFY.
294+
- Loop keyword is different between MySQL and PostgreSQL.
295+
- Create index not support IF NOT EXIST in MySQL.
296+
297+
sql file | Compatibility test | comment
298+
------------|------------|------------
299+
| 0001_initial_schema.up.sql | Pass | xxx
300+
| 0002_1.7.0_schema.up.sql | Pass | xxx
301+
| 0003_add_replication_op_uuid.up.sql | Pass | xxx
302+
| 0004_1.8.0_schema.up.sql | Pass | xxx
303+
| 0005_1.8.2_schema.up.sql | Pass | xxx
304+
| 0010_1.9.0_schema.up.sql | Pass | xxx
305+
| 0011_1.9.1_schema.up.sql | Pass | xxx
306+
| 0012_1.9.4_schema.up.sql | Pass | xxx
307+
| 0015_1.10.0_schema.up.sql | Pass | xxx
308+
| 0030_2.0.0_schema.up.sql | Pass | xxx
309+
| 0031_2.0.3_schema.up.sql | Pass | xxx
310+
| 0040_2.1.0_schema.up.sql | Pass | xxx
311+
| 0041_2.1.4_schema.up.sql | Pass | xxx
312+
| 0050_2.2.0_schema.up.sql | Pass | xxx
313+
| 0051_2.2.1_schema.up.sql | Pass | xxx
314+
| 0052_2.2.2_schema.up.sql | Pass | xxx
315+
| 0053_2.2.3_schema.up.sql | Pass | xxx
316+
| 0060_2.3.0_schema.up.sql | Pass | xxx
317+
| 0061_2.3.4_schema.up.sql | Pass | xxx
318+
| 0070_2.4.0_schema.up.sql | Pass | xxx
319+
| 0071_2.4.2_schema.up.sql | Pass | xxx
320+
| 0080_2.5.0_schema.up.sql | Pass | xxx
321+
322+
**MariaDB 10.5.9**
323+
324+
sql file | Compatibility test | comment
325+
------------|------------|------------
326+
| 0001_initial_schema.up.sql | Pass | xxx
327+
| 0002_1.7.0_schema.up.sql | Pass | xxx
328+
| 0003_add_replication_op_uuid.up.sql | Pass | xxx
329+
| 0004_1.8.0_schema.up.sql | Pass | xxx
330+
| 0005_1.8.2_schema.up.sql | Pass | xxx
331+
| 0010_1.9.0_schema.up.sql | Pass | xxx
332+
| 0011_1.9.1_schema.up.sql | Pass | xxx
333+
| 0012_1.9.4_schema.up.sql | Pass | xxx
334+
| 0015_1.10.0_schema.up.sql | Pass | xxx
335+
| 0030_2.0.0_schema.up.sql | Pass | xxx
336+
| 0031_2.0.3_schema.up.sql | Pass | xxx
337+
| 0040_2.1.0_schema.up.sql | | xxx
338+
| 0041_2.1.4_schema.up.sql | | xxx
339+
| 0050_2.2.0_schema.up.sql | | xxx
340+
| 0051_2.2.1_schema.up.sql | | xxx
341+
| 0052_2.2.2_schema.up.sql | | xxx
342+
| 0053_2.2.3_schema.up.sql | | xxx
343+
| 0060_2.3.0_schema.up.sql | | xxx
344+
| 0061_2.3.4_schema.up.sql | | xxx
345+
| 0070_2.4.0_schema.up.sql | | xxx
346+
| 0071_2.4.2_schema.up.sql | | xxx
347+
| 0080_2.5.0_schema.up.sql | | xxx
348+
349+
### How To Use
350+
351+
Users can configure database type to use Mariadb/MySQL in external_database mode, PostgreSQL is used by default.
352+
353+
Set harbor_db_type configuration under external_database and db type under notary db configuration:
354+
```
355+
external_database:
356+
harbor:
357+
# database type, default is postgresql, options include postgresql, mariadb and mysql
358+
type: mysql
359+
host: 9.134.131.145
360+
port: 3306
361+
db_name: harbor
362+
username: root
363+
password: root
364+
ssl_mode: disable
365+
max_idle_conns: 2
366+
max_open_conns: 0
367+
notary_signer:
368+
host: 9.134.131.145
369+
port: 3306
370+
db_name: notary_signer
371+
username: root
372+
password: root
373+
ssl_mode: disable
374+
notary_server:
375+
host: 9.134.131.145
376+
port: 3306
377+
db_name: notary_server
378+
username: root
379+
password: root
380+
ssl_mode: disable
381+
```
382+

‎proposals/images/multidb/dbdriver.png

57 KB
Loading

‎proposals/images/multidb/initdb.png

94.5 KB
Loading

‎proposals/images/multidb/logic.png

164 KB
Loading

‎proposals/images/multidb/overview.png

71.2 KB
Loading

0 commit comments

Comments
 (0)