Using the Aria Storage Engine with MariaDB Server

storage engine

myisam

aria

↧

Introduction to MaxScale Administration Using maxctrl for MariaDB Cluster

July 8, 2020, 10:55 am

≫ Next: What is MariaDB Enterprise Cluster?

≪ Previous: Using the Aria Storage Engine with MariaDB Server

MariaDB Cluster consists of MariaDB Server with Galera Cluster and MariaDB MaxScale. As a multi-master replication solution, any MariaDB Server with Galera Cluster can operate as a primary server. This means that changes made to any node in the cluster replicate to every other node in the cluster, using certification-based replication and global ordering of transactions for the InnoDB storage engine. MariaDB MaxScale is a database proxy, sitting on top of the MariaDB Server that extends the high availability, scalability, and security while at the same time simplifying application development by decoupling it from the underlying database infrastructure.

In this blog series, we are going to look at the MaxScale administration using maxctrl for our MariaDB Cluster. In this first installment of the blog series, we are going to cover the introduction and some basics of maxctrl command-line utility. Our setup consists of one MaxScale server and a 3-node MariaDB 10.4 with Galera 4, as illustrated in the following diagram:

Our MariaDB Cluster was deployed and managed by ClusterControl, while our MaxScale host is a new host in the cluster and was not deployed by ClusterControl for the purpose of this walkthrough.

MaxScale Installation

The MaxScale installation is pretty straightforward. Choose the right operating system from the MariaDB download page for MaxScale and download it. The following example shows how one would install MaxScale on a CentOS 8 host:

$ wget https://dlm.mariadb.com/1067156/MaxScale/2.4.10/centos/8/x86_64/maxscale-2.4.10-1.centos.8.x86_64.rpm
$ yum localinstall maxscale-2.4.10-1.centos.8.x86_64.rpm
$ systemctl enable maxscale
$ systemctl start maxscale

After the daemon is started, by default, MaxScale components will be running on the following ports:

0.0.0.0:4006 - Default read-write splitting listener.
0.0.0.0:4008 - Default round-robin listener.
127.0.0.1:8989 - MaxScale Rest API.

The above ports are changeable. It is common for a standalone MaxScale server in production to be running with the read/write split on port 3306 and round-robin on port 3307. This configuration is what we are going to deploy in this blog post.

Important Files and Directory Structure

Once the package is installed, you will get the following utilities/programs:

maxscale - The MaxScale itself.
maxctrl - The command-line administrative client for MaxScale which uses the MaxScale REST API for communication.
maxadmin - The deprecated MaxScale administrative and monitor client. Use maxctrl instead.
maxkeys - This utility writes into the file .secrets, in the specified directory, the AES encryption key and init vector that is used by the utility maxpasswd, when encrypting passwords used in the MariaDB MaxScale configuration file.
maxpasswd - This utility creates an encrypted password using a .secrets file that has earlier been created using maxkeys.

MaxScale will load all the configuration options from the following locations, in the particular order:

/etc/maxscale.cnf
/etc/maxscale.cnf.d/*.cnf
/var/lib/maxscale/maxscale.cnf.d/*.cnf

To understand further on MaxScale configuration, check out the MaxScale Configuration Guide.

Once MaxScale is initialized, the default files and directory structures are:

MaxScale data directory: /var/lib/maxscale
MaxScale PID file: /var/run/maxscale/maxscale.pid
MaxScale log file: /var/log/maxscale/maxscale.log
MaxScale documentation: /usr/share/maxscale

MaxCtrl - The CLI

Once started, we can use the MaxCtrl command-line client to administer the MaxScale by using the MaxScale REST API listens on port 8989 on the localhost. The default credentials for the REST API are "admin:mariadb". The users used by the REST API are the same that are used by the MaxAdmin network interface. This means that any users created for the MaxAdmin network interface should work with the MaxScale REST API and MaxCtrl.

We can use the maxctrl utility in interactive mode, similar to the mysql client. Just type "maxctrl" and you will get into the interactive mode (where the prompt changed from the shell prompt to maxctrl prompt), just like the following screenshot:

Alternatively, we can execute the very same command directly in the shell prompt, for example:

MaxCtrl command options are depending on the MaxScale version that comes with it. At the time of this writing, the MaxScale version is 2.4 and you should look into this documentation for a complete list of commands. MaxCtrl utilizes the MaxScale REST API interface, which explains in detail here.

Adding MariaDB Servers into MaxScale

When we first start our MaxScale, it will generate a configuration file at /etc/maxscale.cnf with some default parameters and examples. We are not going to use this configuration and we are going to create our own instead. Create a backup of this file because we want to empty it later on:

$ mv /etc/maxscale.cnf /etc/maxscale.cnf.bak
$ cat /dev/null > /etc/maxscale.cnf # empty the file

Restart the MaxScale to start everything fresh:

$ systemctl restart maxscale

The term "server" in MaxScale basically means the backend MariaDB server, as in this case, all 3 nodes of our MariaDB Cluster. To add all the 3 MariaDB Cluster servers into MaxScale runtime, use the following commands:

$ maxctrl create server mariadbgalera1 192.168.0.221 3306
$ maxctrl create server mariadbgalera2 192.168.0.222 3306
$ maxctrl create server mariadbgalera3 192.168.0.222 3306

To verify the added servers, use the list command:

$ maxctrl list servers

And you should see the following output:

Adding Monitoring into MaxScale

The next thing is to configure the monitoring service for MaxScale usage. MaxScale supports a number of monitoring modules depending on the database type, namely:

MariaDB Monitor
Galera Monitor
Clustrix Monitor
ColumnStore Monitor
Aurora Monitor

In this setup, we are going to use the Galera Monitor module called "galeramon". Firstly, we need to create a database user to be used by MaxScale on one of the servers in the MariaDB Cluster. In this example we picked mariadbgalera1, 192.168.0.221 to run the following statements:

MariaDB> CREATE USER maxscale_monitor@'192.168.0.220' IDENTIFIED BY 'MaXSc4LeP4ss';
MariaDB> GRANT SELECT ON mysql.* TO 'maxscale_monitor'@'192.168.0.220';
MariaDB> GRANT SHOW DATABASES ON *.* TO 'maxscale_monitor'@'192.168.0.220';

Where 192.168.0.220 is the IP address of our MaxScale server.

It's not safe to store the maxscale_monitor user password in plain text. It's highly recommended to store the password in an encrypted format instead. To achieve this, we need to generate a secret key specifically for this MaxScale instance. Use the "maxkeys" utility to generate the secret key that will be used by MaxScale for encryption and decryption purposes:

$ maxkeys
Generating .secrets file in /var/lib/maxscale.

Now we can use the maxpasswd utility to generate the encrypted value of our password:

$ maxpasswd MaXSc4LeP4ss
D91DB5813F7C815B351CCF7D7F1ED6DB

We will always use the above value instead when storing our monitoring user credentials inside MaxScale. Now we are ready to add the Galera monitoring service into MaxScale using maxctrl:

maxctrl> create monitor galera_monitor galeramon servers=mariadbgalera1,mariadbgalera2,mariadbgalera3 user=maxscale_monitor password=D91DB5813F7C815B351CCF7D7F1ED6DB

Verify with the following command:

Adding Services into MaxScale

Service is basically how MaxScale should route the queries to the backend servers. MaxScale 2.4 supports multiple services (or routers), namely:

Avrorouter
Binlogrouter
Cat
CLI
HintRouter
Readconnroute
Readwritesplit
SchemaRouter
SmartRouter

For our MariaDB Cluster, we only need two routing services - Read-write split and round-robin load balancing. For read-write splitting, write queries will be forwarded to only a single MariaDB server until the server is unreachable, where MaxScale will then forward the write queries to the next available node. For round-robin balancing, the queries will be forwarded to all of the backend nodes in a round-robin fashion.

Create a routing service for round-robin (or multi-master):

maxctrl> create service Round-Robin-Service readconnroute user=maxscale_monitor password=D91DB5813F7C815B351CCF7D7F1ED6DB --servers mariadbgalera1 mariadbgalera2 mariadbgalera3

Create another routing service for read-write splitting (or single-master):

maxctrl> create service Read-Write-Service readwritesplit user=maxscale_monitor password=D91DB5813F7C815B351CCF7D7F1ED6DB --servers mariadbgalera1 mariadbgalera2 mariadbgalera3

Verify with:

All the successfully created components by MaxCtrl will generate its own configuration file under /var/lib/maxscale/maxscale.cnf.d. At this point, the directory looks like this:

$ ls -l /var/lib/maxscale/maxscale.cnf.d
total 24
-rw-r--r--. 1 maxscale maxscale  532 Jul  5 13:18 galera_monitor.cnf
-rw-r--r--. 1 maxscale maxscale  250 Jul  5 12:55 mariadbgalera1.cnf
-rw-r--r--. 1 maxscale maxscale  250 Jul  5 12:55 mariadbgalera2.cnf
-rw-r--r--. 1 maxscale maxscale  250 Jul  5 12:56 mariadbgalera3.cnf
-rw-r--r--. 1 maxscale maxscale 1128 Jul  5 16:01 Read-Write-Service.cnf
-rw-r--r--. 1 maxscale maxscale  477 Jul  5 16:00 Round-Robin-Service.cnf

Adding Listeners into MaxScale

Listeners represent the ports the service will listen to incoming connections. It can be a port or UNIX socket file and the component type must be "listener". Commonly, listeners are tied to services. In our setup, we are going to create two listeners - Read-Write Listener on port 3306 and Round-Robin Listener on port 3307:

maxctrl> create listener Read-Write-Service Read-Write-Listener 3306 --interface=0.0.0.0 --authenticator=MariaDBAuth
maxctrl> create listener Round-Robin-Service Round-Robin-Listener 3307 --interface=0.0.0.0 --authenticator=MariaDBAuth

Verify with the following commands:

At this point, our MaxScale is now ready to load balance the queries to our MariaDB Cluster. From the applications, send the queries to the MaxScale host on port 3306, where the write queries will always hit the same database node while the read queries will be sent to the other two nodes. This is also known as a single-writer setup. If you would like to use a multi-writer setup, where writes will be forwarded to all backend MariaDB nodes based on round-robin balancing algorithms. You can further fine-tune the balancing by using priority and weight.

Again, when changing the configuration options via maxctrl, all successfully created components will have its own configuration file inside /var/lib/maxscale/maxscale.cnf.d, as shown in the following output:

$ ls -l /var/lib/maxscale/maxscale.cnf.d
-rw-r--r--. 1 maxscale maxscale  532 Jul  5 13:18 galera_monitor.cnf
-rw-r--r--. 1 maxscale maxscale  250 Jul  5 12:55 mariadbgalera1.cnf
-rw-r--r--. 1 maxscale maxscale  250 Jul  5 12:55 mariadbgalera2.cnf
-rw-r--r--. 1 maxscale maxscale  250 Jul  5 12:56 mariadbgalera3.cnf
-rw-r--r--. 1 maxscale maxscale  259 Jul  5 16:06 Read-Write-Listener.cnf
-rw-r--r--. 1 maxscale maxscale 1128 Jul  5 16:06 Read-Write-Service.cnf
-rw-r--r--. 1 maxscale maxscale  261 Jul  5 16:06 Round-Robin-Listener.cnf
-rw-r--r--. 1 maxscale maxscale  477 Jul  5 16:06 Round-Robin-Service.cnf

The above configuration options can be directly modified to further suit your needs, but it requires the MaxScale service to be restarted to load the new changes. If you would like to start fresh again, you could wipe everything under this directory and restart MaxScale.

In the next episode, we will look into MaxCtrl's management and monitoring commands for our MariaDB Cluster.

Tags:

MaxScale

maxctrl

↧

What is MariaDB Enterprise Cluster?

July 9, 2020, 10:53 am

≫ Next: An Overview of Caching for PostgreSQL

≪ Previous: Introduction to MaxScale Administration Using maxctrl for MariaDB Cluster

MariaDB Enterprise Cluster is a subscription service of a highly available database solution from MariaDB Corporation which is managed with an Enterprise Lifecycle. There are three aspects of the Enterprise Lifecycle that are provided by MariaDB: Enterprise Builds, Enterprise Releases, and Enterprise Support.

Enterprise Builds ensure you will get the highest level of quality of software, which consists of optimized default parameters and priority of bug fixes available for subscription customers.

Enterprise Release gives you predictable releases for patches and updates based on a certain schedule.

Enterprise Support provides the user with customer support, professional services, training, and documentation.

The MariaDB Enterprise Cluster consists of MariaDB Enterprise Server with Galera Cluster for redundancy, and MariaDB Maxscale for load balancing.

MariaDB Enterprise Server & Cluster

MariaDB Enterprise Cluster comes with an Enterprise grade database server called MariaDB Enterprise Server. It provides enterprise features such as:

MariaDB Enterprise Audit, comprehensive audit plugin that provides detailed information of connections and also the changes of database.
MariaDB Enterprise Backup, it is an enhanced feature from MariaDB Backup that allows the writes and schema changes while the backup is running. The DDL blocking is reduced through backup stages and DDL logging.

Beside the enterprise features, there are some standard features that you might be familiar with in MariaDB, for example : SQL based account locking, password expiration, bitemporal tables, account automatic lock after failed login attempts.

MariaDB Enterprise Cluster and Galera Cluster

MariaDB Enterprise Cluster uses Galera Cluster for MariaDB which is already enhanced for the enterprise. It synchronizes data to achieve redundancy and high availability. Galera Cluster is a database clustering solution that enables multi master replication between the nodes with synchronous replication state.

The synchronous replication in Galera Cluster uses certification based replication where group communication and transaction ordering are used. The transaction is executed in a node, at the point when the commit happens, it will run coordination of the certification process to enforce global consistency. The broadcast service establishes a global total order between transactions to achieve global coordination.

Certification Based Replication requires some features of the database in order to be working. The features are:

Transactional Database; the database must be transactional, it needs to be able to rollback uncommitted transactions.
Atomic Changes; the transaction changes must occur completely or not occur at all in the database.
Global Ordering; the replication must be ordered globally. Transaction must apply to all instances within the same order.

MariaDB Enterprise Cluster and MariaDB Maxscale

MariaDB Enterprise Cluster also comes with MariaDB Maxscale as a database proxy which can provide a high availability, scalability environment. Other popular proxies that are used by MySQL and MariaDB users include HAProxy and ProxySQL.

There are some great features for Maxscale that give you benefit for your environment scaling:

Automatic Failover

Maxscale can monitor database server availability and automatically trigger failover for service resiliency if a crash happens. In MariaDB Enterprise Cluster where any node can accept writes and reads, Maxscale is used to minimize the database failures. In addition, maxscale also can be used to split write traffic.

Traffic Control

There are some features related to traffic controls in maxscale. You can set the max threshold of your query per seconds using Query throttling, SQL firewall can be used to restrict data access and block queries that have similar patterns based on the rules we defined. Authentication support that supports PAM and Kerberos.

Load Balancing

It provides load balancing for your traffic distributed to your database. It can be used to scale out your database (split read/write traffic through the nodes).

There are also some improvements on the latest Maxscale (version 2.4) such as Change Data Capture (CDC) adapter, connection attempt throttling, smart query routing, and ClustrixDB support.

We hope this short blog post gives you an understanding of what it is included in MariaDB Enterprise Cluster.

Tags:

↧

An Overview of Caching for PostgreSQL

July 10, 2020, 11:41 am

≫ Next: Exploring Storage Engine Options for MariaDB

≪ Previous: What is MariaDB Enterprise Cluster?

Most OLTP workloads involve random disk I/O usage. Knowing that disks (including SSD) are slower in performance than using RAM, database systems use caching to increase performance. Caching is all about storing data in memory (RAM) for faster access at a later point of time.

PostgreSQL also utilizes caching of its data in a space called shared_buffers. In this blog we will explore this functionality to help you increase performance.

PostgreSQL Caching Basics

Before we delve deeper into the concept of caching, let’s have some brush-up of the basics.

In PostgreSQL, data is organized in the form of pages of size 8KB, and every such page can contain multiple tuples (depending on the size of tuple). A simplistic representation could be like below:

PostgreSQL caches the following for accelerating data access:

Data in tables
Indexes
Query execution plans

While the query execution plan caching focus is on saving CPU cycles; caching for Table data and Index data is focused to save costly disk I/O operation.

PostgreSQL lets users define how much memory they would like to reserve for keeping such cache for data. The relevant setting is shared_buffers in the postgresql.conf configuration file. The finite value of shared_buffers defines how many pages can be cached at any point of time.

As a query is executed, PostgreSQL searches for the page on the disk which contains the relevant tuple and pushes it in the shared_buffers cache for lateral access. Next time the same tuple (or any tuple in the same page) needs to be accessed, PostgreSQL can save disk IO by reading it in memory.

In the above figure, Page-1 and Page-2 of a certain table have been cached. In case a user query needs to access tuples between Tuple-1 to Tuple-200, PostgreSQL can fetch it from RAM itself.

However if the query needs to access Tuples 250 to 350, it will need to do disk I/O for Page 3 and Page 4. Any further access for Tuple 201 to 400 will be fetched from cache and disk I/O will not be needed – thereby making the query faster.

At a high level, PostgreSQL follows LRU (least recently used) algorithm to identify the pages which need to be evicted from the cache. In other words, a page which is accessed only once has higher chances of eviction (as compared to a page which is accessed multiple times), in case a new page needs to be fetched by PostgreSQL into cache.

PostgreSQL Caching in Action

Let’s execute an example and see the impact of cache on the performance.

Start PostgreSQL keeping shared_buffer set to default 128 MB

$ initdb -D ${HOME}/data

$ echo “shared_buffers=128MB” >> ${HOME}/data/postgresql.conf

$ pg_ctl -D ${HOME}/data start

Connect to the server and create a dummy table tblDummy and an index on c_id

CREATE Table tblDummy

(

id serial primary key,

p_id int,

c_id int,

entry_time timestamp,

entry_value int,

description varchar(50)  

);

CREATE INDEX ON tblDummy(c_id );

Populate dummy data with 200000 tuples, such that there are 10000 unique p_id and for every p_id there are 200 c_id

DO $$

DECLARE

random_value integer:= 1;

BEGIN

FOR p_id_ctr IN 1..10000 BY 1 LOOP               

FOR c_id_ctr IN 1..200 BY 1 LOOP                                 

random_value = (( random() * 75 ) + 25);

INSERT INTO tblDummy (p_id,c_id,entry_time, entry_value, description )

VALUES (p_id_ctr,c_id_ctr,'now', random_value, CONCAT('Description for :',p_id_ctr, c_id_ctr));

END LOOP ;

END LOOP ;                      

END $$;

Restart the server to clear the cache. Now execute a query and check for the time taken to execute the same

SELECT pg_stat_reset();

EXPLAIN ANAYZE SELECT count(*) from tbldummy where c_id=1;



                           QUERY PLAN

--------------------------------------------------------------

 Aggregate  (cost=17407.33..17407.34 rows=1 width=8) (actual time=160.269..160.269 rows=1 loops=1)

   ->  Bitmap Heap Scan on tbldummy  (cost=189.52..17382.46 rows=9948 width=0) (actual time=10.627..156.275 rows=10000 loops=1)

         Recheck Cond: (c_id = 1)

         Heap Blocks: exact=10000

         ->  Bitmap Index Scan on tbldummy_c_id_idx  (cost=0.00..187.04 rows=9948 width=0) (actual time=5.091..5.091 rows=10000 loops=1)

               Index Cond: (c_id = 1)

 Planning Time: 1.325 ms

 Execution Time: 160.505 ms

Then check the blocks read from the disk

SELECT heap_blks_read, heap_blks_hit from pg_statio_user_tables where relname='tbldummy';

heap_blks_read | heap_blks_hit

---------------+---------------

10000          |             0

In the above example, there were 1000 blocks read from the disk to find count tuples where c_id = 1. It took 160 ms since there was disk I/O involved to fetch those records from disk.

Execution is faster if same query is re-executed, as all the blocks are still in cache of PostgreSQL server at this stage

SELECT pg_stat_reset();

EXPLAIN ANAYZE SELECT count(*) from tbldummy where c_id=1;

                                                               

                                 QUERY PLAN

-------------------------------------------------------------------------------------

 Aggregate  (cost=17407.33..17407.34 rows=1 width=8) (actual time=33.760..33.761 rows=1 loops=1)

   ->  Bitmap Heap Scan on tbldummy  (cost=189.52..17382.46 rows=9948 width=0) (actual time=9.584..30.576 rows=10000 loops=1)

         Recheck Cond: (c_id = 1)

         Heap Blocks: exact=10000

         ->  Bitmap Index Scan on tbldummy_c_id_idx  (cost=0.00..187.04 rows=9948 width=0) (actual time=4.314..4.314 rows=10000 loops=1)

               Index Cond: (c_id = 1)

 Planning Time: 0.106 ms

 Execution Time: 33.990 ms

and blocks read from the disk vs from cache

SELECT heap_blks_read, heap_blks_hit from pg_statio_user_tables where relname='tbldummy';

heap_blks_read | heap_blks_hit

---------------+---------------

    0          |         10000

It is evident from above that since all blocks were read from the cache and no disk I/O was required. This hence also gave the results faster.

Setting the Size of the PostgreSQL Cache

The size of the cache needs to be tuned in a production environment in accordance to the amount of RAM available as well as the queries required to be executed.

As an example – shared_buffer of 128MB may not be sufficient to cache all data, if the query was to fetch more tuples:

SELECT pg_stat_reset();

SELECT count(*) from tbldummy where c_id < 150;

SELECT heap_blks_read, heap_blks_hit from pg_statio_user_tables where relname='tbldummy';

 heap_blks_read | heap_blks_hit

----------------+---------------

      20331     |      288

Change the shared_buffer to 1024MB to increase the heap_blks_hit.

In-fact, considering the queries (based on c_id), in case data is re-organized, a better cache hit ratio can be achieved with a smaller shared_buffer as well.

In Data_Organization-1, PostgreSQL will need 1000 block reads (and cache consumption) for finding c_id=1. On the other hand, for Data_Organisation-2, for the same query, PostgreSQL will need only 104 blocks.

Less blocks required for the same query eventually consume less cache and also keep query execution time optimized.

Conclusion

While the shared_buffer is maintained at PostgreSQL process level, the kernel level cache is also taken into consideration for identifying optimized query execution plans. I will take up this topic in a later series of blogs.

Tags:

postgres

PostgreSQL

caching

↧

Exploring Storage Engine Options for MariaDB

July 13, 2020, 9:03 am

≫ Next: Using the MariaDB Audit Plugin for Database Security

≪ Previous: An Overview of Caching for PostgreSQL

MariaDB Server was originally derived from MySQL and has therefore inherited its pluggable storage engine architecture. Different storage engines have different characteristics in terms of performance but also features and possibilities. This allows users to pick the right tool for the job instead of using the same storage engine no matter what is the purpose of the data, what are the requirements regarding data storage and how the data should be accessed. In this blog post we would like to look at the options available in MariaDB and discuss potential use cases for the different storage engines available.

What is a Storage Engine?

First, though, let’s take a look at what is the storage engine? MariaDB consists of multiple layers that operate together. SQL is parsed by one of them, then MariaDB reaches out for data, using a common API. Under the hood there is a storage engine that contains the data and it reacts to the requests for data, extracts the data and makes it available to MariaDB.

In short, MariaDB sends a request for a row and it is all up to the storage engine to retrieve it and send it back. MariaDB does not care how exactly the row is stored or how it is going to be retrieved, it is all up to the implementation within the storage engine. Storage engines may also implement different features. Transactions are being handled also entirely on the storage engine’s side. That’s why some of the support transactions and some do not. With this architecture it is possible to write different storage engines, dedicated to solving different problems.

Storage Engines in MariaDB Server

MariaDB comes with a set of storage engines. You can check which ones are available through a simple command:

MariaDB [(none)]> SHOW STORAGE ENGINES;

+--------------------+---------+-------------------------------------------------------------------------------------------------+--------------+------+------------+

| Engine             | Support | Comment                                                                                         | Transactions | XA   | Savepoints |

+--------------------+---------+-------------------------------------------------------------------------------------------------+--------------+------+------------+

| MRG_MyISAM         | YES     | Collection of identical MyISAM tables                                                           | NO           | NO   | NO         |

| CSV                | YES     | Stores tables as CSV files                                                                      | NO           | NO   | NO         |

| Aria               | YES     | Crash-safe tables with MyISAM heritage. Used for internal temporary tables and privilege tables | NO           | NO   | NO         |

| SEQUENCE           | YES     | Generated tables filled with sequential values                                                  | YES          | NO   | YES        |

| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables                                       | NO           | NO   | NO         |

| MyISAM             | YES     | Non-transactional engine with good performance and small data footprint                         | NO           | NO   | NO         |

| PERFORMANCE_SCHEMA | YES     | Performance Schema                                                                              | NO           | NO   | NO         |

| InnoDB             | DEFAULT | Supports transactions, row-level locking, foreign keys and encryption for tables                | YES          | YES  | YES        |

+--------------------+---------+-------------------------------------------------------------------------------------------------+--------------+------+------------+

8 rows in set (0.000 sec)

As you can see, there are many of them, we will cover the most important ones.

InnoDB

InnoDB, obviously, is THE storage engine. Transactional, built to deal with OLTP traffic, can provide really great performance. It is the default engine used in MariaDB and, unless you know what you are doing, you probably want to stick to it for your database.

MyISAM

MyISAM is one of the “original” storage engines available in MySQL and then MariaDB. It is not transactional, making it not ideal for the replication setups and, well, most of other environments as well. It is still very fast engine, especially regarding index access, making it suitable for read-only workloads that won’t be affected by locking of INSERTs and overall fragility of MyISAM.

Aria

Aria is an engine created for MariaDB as a replacement for MyISAM. It is not transactional but it is crash-safe making it way more reliable. Currently it is used for system and temporary tables but it can also be used instead of MyISAM for workloads requiring fast, read-only access to data.

Memory

This is an all-in-memory engine that is typically used for temporary in-memory tables. It is not persistent but might work for some read-only workloads.

CSV

This storage engine is designed to store data in a file as comma-separated values. It is not the most used storage engine, it’s very specialized but it still can be used to easily extract data from MariaDB into any other database software as well as Excel or similar software.

Storage Engines in MariaDB Enterprise Server

MariaDB Enterprise Server comes with a couple additional storage engines over what is available in the community edition. Let’s take a look at them as well.

ColumnStore

This is a dedicated storage engine for analytical workload. Thanks to the specific way of storing the data it makes it faster to retrieve large volumes of data, frequently needed for reporting. This might be the storage engine of your choosing for OLAP (OnLine Analytical Processing) workloads.

S3

S3 engine allows you to access data located in S3. It is a non-transactional engine intended to give users the option to archive data in the S3. Read only access is available after the table is created.

Spider

Spider engine lets you connect multiple MariaDB databases across the network, creating a sharded storage. It is transactional and it makes it easier for users to scale out by splitting the data across numerous MariaDB Enterprise Servers, distributing the traffic and workload among them.

MyRocks

MyRocks is a storage engine developed in Facebook, it is intended to reduce the write amplification and minimize the wear out of SSD drives. It is a transactional engine which should handle OLTP workload quite well, especially workloads typical for social media websites. MyRocks comes with pretty good compression, better than InnoDB, which can help to significantly reduce expenses on storage if the dataset becomes too large for InnoDB to handle properly.

Conclusion

As you can see, there are numerous options provided by both MariaDB Enterprise and Community Server regarding the way in which data can be stored. There are storage engines that excel in read-only workloads, OLAP or large datasets. It is up to the user to pick a good fit. Please keep in mind that, when in doubt, you can always stick to InnoDB, which provides quite good performance in general and should be more than enough for the majority of the cases. It is for those edge cases where you may need to look for something more suitable.

Tags:

↧

Using the MariaDB Audit Plugin for Database Security

July 14, 2020, 8:59 am

≫ Next: What's New in MariaDB MaxScale 2.4

≪ Previous: Exploring Storage Engine Options for MariaDB

There are different ways to keep your data safe. Practices such as controlling database access, securing configuration, upgrading your system, and more are part of database security. It is even possible that you have security issues and don’t realize it (until it is too late), that’s why monitoring is a key piece to ensure that if something unexpected happens, you will be able to catch it. This includes not only your system, but also your databases.

Auditing is a way to know what is happening in your database, and it is also required for many security regulations or standards (e.g. PCI - Payment Card Industry).

MariaDB Server, one of the most popular open-source database servers, has his own Audit Plugin (which also works on MySQL), in order to help with this auditing task. In this blog, you will see how to install and use this useful MariaDB Audit Plugin.

What is MariaDB Audit Plugin?

The Audit Plugin is developed by MariaDB to meet the requirements to record user access to be in compliance with auditing regulations.

For each client session, it records, in a log file (or syslog), who connected to the server, what queries executed, which tables were accessed, and server variables changed.

It works with MariaDB, MySQL, and Percona Server. MariaDB started including by default the Audit Plugin from versions 10.0.10 and 5.5.37, and it can be installed in any version from MariaDB 5.5.20.

MariaDB Audit Plugin Installation

The plugin file (server_audit.so) is installed by default during the MariaDB installation in the plugins directory /usr/lib/mysql/plugin/:

$ ls -lah /usr/lib/mysql/plugin/ |grep server_audit

-rw-r--r-- 1 root  root  63K May  9 19:33 server_audit.so

So, you just need to add it into the MariaDB instance:

MariaDB [(none)]> INSTALL SONAME 'server_audit';

Query OK, 0 rows affected (0.003 sec)

MariaDB [(none)]> SHOW PLUGINS;

+--------------+--------+-------+-----------------+---------+

| Name         | Status | Type  | Library         | License |

+--------------+--------+-------+-----------------+---------+

| SERVER_AUDIT | ACTIVE | AUDIT | server_audit.so | GPL     |

+--------------+--------+-------+-----------------+---------+

And enable it using the SET GLOBAL command:

MariaDB [(none)]> SET GLOBAL server_audit_logging=ON;

Query OK, 0 rows affected (0.000 sec)

Or make it persistent in the my.cnf configuration file to start to auditing:

[MYSQLD]

server_audit_logging=ON

Another way to add it into the MariaDB instance is by adding the plugin_load_add parameter in the my.cnf configuration file:

[mariadb]

plugin_load_add = server_audit

It is also recommended to add the FORCE_PLUS_PERMANENT to avoid uninstalling it:

[mariadb]

plugin_load_add = server_audit

server_audit=FORCE_PLUS_PERMANENT

Now you have the MariaDB Audit Plugin installed, let’s see how to configure it.

MariaDB Audit Plugin Configuration

To check the current configuration you can see the value of the “server_audit%” global variables by running the following command:

MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE "server_audit%";

+-------------------------------+-----------------------+

| Variable_name                 | Value                 |

+-------------------------------+-----------------------+

| server_audit_events           |                       |

| server_audit_excl_users       |                       |

| server_audit_file_path        | server_audit.log      |

| server_audit_file_rotate_now  | OFF                   |

| server_audit_file_rotate_size | 1000000               |

| server_audit_file_rotations   | 9                     |

| server_audit_incl_users       |                       |

| server_audit_logging          | OFF                   |

| server_audit_mode             | 0                     |

| server_audit_output_type      | file                  |

| server_audit_query_log_limit  | 1024                  |

| server_audit_syslog_facility  | LOG_USER              |

| server_audit_syslog_ident     | mysql-server_auditing |

| server_audit_syslog_info      |                       |

| server_audit_syslog_priority  | LOG_INFO              |

+-------------------------------+-----------------------+

15 rows in set (0.001 sec)

You can modify these variables using the SET GLOBAL command or make it persistent in the my.cnf configuration file under the [mysqld] section.

Let’s describe some of the most important variables:

server_audit_logging: Enables audit logging.
server_audit_events: Specifies the events that you want to record. By default, the value is empty, which means that all events are recorded. The options are CONNECTION, QUERY, and TABLE.
server_audit_excl_users, server_audit_incl_users: These variables specify which users’ activity should be excluded or included in the audit log file. By default, all users’ activity is recorded.
server_audit_output_type: By default auditing output is sent to a file. The other option is syslog, meaning all entries go to the syslog facility.
server_audit_syslog_facility, server_audit_syslog_priority: Specifies the syslog facility and the priority of the events that should go to syslog.

After configuring it, you can see the audit events in the specified log file (or syslog). Let’s see how it looks.

MariaDB Audit Plugin Log

To see the events registered by the Audit Log Plugin, you can check the specified log file (by default server_audit.log).

$ tail -f /var/lib/mysql/server_audit.log

20200703 19:07:04,MariaDB1,cmon,10.10.10.116,64,915239,QUERY,information_schema,'FLUSH /*!50500 SLOW */ LOGS',0

20200703 19:07:05,MariaDB1,cmon,10.10.10.116,61,915240,QUERY,information_schema,'SHOW GLOBAL STATUS',0

20200703 19:07:05,MariaDB1,cmon,10.10.10.116,64,915241,WRITE,mysql,slow_log,

20200703 19:07:05,MariaDB1,cmon,10.10.10.116,64,915241,QUERY,information_schema,'SET GLOBAL SLOW_QUERY_LOG=1',0

20200703 19:07:06,MariaDB1,cmon,10.10.10.116,61,915242,QUERY,information_schema,'SHOW GLOBAL STATUS',0

20200703 19:15:42,MariaDB1,root,localhost,124,0,CONNECT,,,0

20200703 19:15:42,MariaDB1,root,localhost,124,917042,QUERY,,'select @@version_comment limit 1',0

20200703 19:15:48,MariaDB1,root,localhost,124,0,DISCONNECT,,,0

20200703 19:57:41,MariaDB1,root,localhost,135,925831,QUERY,,'create database test1',0

20200703 19:58:05,MariaDB1,root,127.0.0.1,136,0,FAILED_CONNECT,,,1045

20200703 19:58:05,MariaDB1,root,127.0.0.1,136,0,DISCONNECT,,,0

20200703 19:58:49,MariaDB1,root,localhost,137,926073,QUERY,,'SELECT DATABASE()',0

20200703 19:58:49,MariaDB1,root,localhost,137,926075,QUERY,test1,'show databases',0

20200703 19:58:49,MariaDB1,root,localhost,137,926076,QUERY,test1,'show tables',0

20200703 19:59:20,MariaDB1,root,localhost,137,926182,CREATE,test1,t1,

20200703 19:59:20,MariaDB1,root,localhost,137,926182,QUERY,test1,'create table t1 (id int, message text)',0

20200703 19:59:48,MariaDB1,root,localhost,137,926287,QUERY,test1,'insert into t1 values (4,\'message 1\')',0

As you can see in the above log, you will have events about database connections and the queries running there, depending on the server_audit_events configuration.

Using the MariaDB Audit Plugin in ClusterControl

In order to avoid manual configuration, you can enable the Audit Plugin from the ClusterControl UI. For this, you only need to go to ClusterControl -> Select the MariaDB Cluster -> Security -> Audit Log:

And you will have the plugin enabled without any manual installation nor configuration.

Using ClusterControl you can also take advantage of different features, not only security but also monitoring, managing, and backing up, among other useful features.

Conclusion

Auditing is required for many security regulations and it is also useful if you want to know what happened in your database, and when and who was responsible for that.

MariaDB Audit Plugin is an excellent way to audit your databases without using any external tool, and it is also compatible with MySQL and Percona Server. If you want to avoid configuring it manually, you can use ClusterControl to enable this Audit Plugin in an easy way from the ClusterControl UI.

Tags:

↧

What's New in MariaDB MaxScale 2.4

July 15, 2020, 9:09 am

≫ Next: Deploying MariaDB Replication for High Availability

≪ Previous: Using the MariaDB Audit Plugin for Database Security

MaxScale 2.4 was released on December 21st, 2019, and ClusterControl 1.7.6 supports the monitoring and managing up to this version. However, for deployment, ClusterControl only supports up to version 2.3. One has to manually upgrade the instance manually, and fortunately, the upgrade steps are very straightforward. Just download the latest version from MariaDB MaxScale download page and perform the package installation command.

The following commands show how to upgrade from an existing MaxScale 2.3 to MaxScale 2.4 on a CentOS 7 box:

$ wget https://dlm.mariadb.com/1067184/MaxScale/2.4.10/centos/7/x86_64/maxscale-2.4.10-1.centos.7.x86_64.rpm

$ systemctl stop maxscale

$ yum localinstall -y maxscale-2.4.10-1.centos.7.x86_64.rpm

$ systemctl start maxscale

$ maxscale --version

MaxScale 2.4.10

In this blog post, we are going to highlight some of the notable improvements and new features of this version and how it looks like in action. For a full list of changes in MariaDB MaxScale 2.4, check out its changelog.

Interactive Mode Command History

This is basically a small improvement with a major impact on MaxScale administration and monitoring task efficiency. The interactive mode for MaxCtrl now has its command history. Command history easily allows you to repeat the executed command by pressing the up or down arrow key. However, Ctrl+R functionality (recall the last command matching the characters you provide) is still not there.

In the previous versions, one has to use the standard shell mode to make sure the commands are captured by .bash_history file.

GTID Monitoring for galeramon

This is a good enhancement for those who are running on Galera Cluster with geographical redundancy via asynchronous replication, also known as cluster-to-cluster replication, or MariaDB Galera Cluster replication over MariaDB Replication.

In MaxScale 2.3 and older, this is what it looks like if you have enabled master-slave replication between MariaDB Clusters:

For MaxScale 2.4, it is now looking like this (pay attention to Galera1's row):

It's now easier to see the replication state for all nodes from MaxScale, without the need to check on individual nodes repeatedly.

SmartRouter

This is one of the new major features in MaxScale 2.4, where MaxScale is now smart enough to learn which backend MariaDB backend server is the best to process the query. SmartRouter keeps track of the performance, or the execution time, of queries to the clusters. Measurements are stored with the canonical of a query as the key. The canonical of a query is the SQL with all user-defined constants replaced with question marks, for example:

UPDATE `money_in` SET `accountholdername` = ? , `modifiedon` = ? , `status` = ? , `modifiedby` = ? WHERE `id` = ?

This is a very useful feature if you are running MariaDB on a multi-site geographical replication or a mix of MariaDB storage engines in one replication chain, for example, a dedicated slave to handle transaction workloads (OLTP) with InnoDB storage engine and another dedicated slave to handle analytics workloads (OLAP) with Columnstore storage engine.

Supposed we are having two sites - Sydney and Singapore as illustrated in the following diagram:

The primary site is located in Singapore and has a MariaDB master and a slave, while another read-only slave is located in Sydney. The application connects to the MaxScale instance located in its respective country with the following port settings:

Read-write split: 3306
Round robin: 3307
Smart router: 3308

Our SmarRouter service and listener definitions are:

[SmartQuery]

type=service

router=smartrouter

servers=DB_1,DB_2,DB_5

master=DB_1

user=maxscale

password=******

[SmartQuery-Listener]

type = listener

service = SmartQuery

protocol = mariadbclient

port = 3308

Restart MaxScale and start sending a read-only query to both MaxScale nodes located in Singapore and Sydney. If the query is processed by the round-robin router (port 3307), we would see the query is being routed based on the round-robin algorithm:

(app)$ mysql -usbtest -p -h maxscale_sydney -P3307 -e 'SELECT COUNT(id),@@hostname FROM sbtest.sbtest1'

+-----------+--------------------+

| count(id) | @@hostname         |

+-----------+--------------------+

|   1000000 | mariadb_singapore2 |

+-----------+--------------------+

From the above, we can tell that Sydney's MaxScale forwarded the above query to our Singapore's slave, which is not the best routing option per se.

With SmartRouter listening on port 3308, we would see the query is being routed to the nearest slave in Sydney:

(app)$ mysql -usbtest -p -h maxscale_sydney -P3308 -e 'SELECT COUNT(id),@@hostname FROM sbtest.sbtest1'

+-----------+-----------------+

| count(id) | @@hostname      |

+-----------+-----------------+

|   1000000 | mariadb_sydney1 |

+-----------+-----------------+

And if the same query is executed in our Singapore site, it will be routed to the MariaDB slave located in Singapore:

(app)$ mysql -usbtest -p -h maxscale_singapore -P3308 -e 'SELECT COUNT(id),@@hostname FROM sbtest.sbtest1'

+-----------+--------------------+

| count(id) | @@hostname         |

+-----------+--------------------+

|   1000000 | mariadb_singapore2 |

+-----------+--------------------+

There is a catch though. When SmartRouter sees a read-query whose canonical has not been seen before, it will send the query to all clusters. The first response from a cluster will designate that cluster as the best one for that canonical. Also, when the first response is received, the other queries are canceled. The response is sent to the client once all clusters have responded to the query or the cancel.

This means, to keep track of the canonical query (normalized query) and measure its performance, you probably will see the very first query fails in its first execution, for example:

(app)$ mysql -usbtest -p -h maxscale_sydney -P3308 -e 'SELECT COUNT(id),@@hostname FROM sbtest.sbtest1'

ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query

From the general log in MariaDB Sydney, we can tell that the first query (ID 74) was executed successfully (connect, query and quit), despite the "Lost connection" error from MaxScale:

  74 Connect  sbtest@3.25.143.151 as anonymous on 

  74 Query    SELECT COUNT(id),@@hostname FROM sbtest.sbtest1

  74 Quit

While the identical subsequent query was correctly processed and returned with the correct response:

(app)$ mysql -usbtest -p -h maxscale_sydney -P3308 -e 'SELECT COUNT(id),@@hostname FROM sbtest.sbtest1'

+-----------+------------------------+

| count(id) | @@hostname             |

+-----------+------------------------+

|   1000000 | mariadb_sydney.cluster |

+-----------+------------------------+

Looking again at the general log in MariaDB Sydney (ID 75), the same processing events happened just like the first query:

  75 Connect  sbtest@3.25.143.151 as anonymous on 

  75 Query    SELECT COUNT(id),@@hostname FROM sbtest.sbtest1

  75 Quit

From this observation, we can conclude that occasionally, MaxScale has to fail the first query in order to measure performance and become smarter for the subsequent identical queries. Your application must be able to handle this "first error" properly before returning to the client or retry the transaction once more.

UNIX Socket for Server

There are multiple ways to connect to a running MySQL or MariaDB server. You could use the standard networking TCP/IP with host IP address and port (remote connection), named pipes/shared memory on Windows or Unix socket files on Unix-based systems. The UNIX socket file is a special kind of file that facilitates communications between different processes, which in this case is the MySQL client and the server. The socket file is a file-based communication, and you can't access the socket from another machine. It provides a faster connection than TCP/IP (no network overhead) and a more secure connection approach because it can be used only when connecting to a service or process on the same computer.

Supposed the MaxScale server is also installed on the MariaDB Server itself, we can use the socket UNIX socket file instead. Under the Server section, remove or comment the "address" line and add the socket parameter with the location of the socket file:

[DB_2]

type=server

protocol=mariadbbackend

#address=54.255.133.39

socket=/var/lib/mysql/mysql.sock

Before applying the above changes, we have to create a MaxScale axscale user from localhost. On the master server:

MariaDB> CREATE USER 'maxscale'@'localhost' IDENTIFIED BY 'maxscalep4ss';

MariaDB> GRANT SELECT ON mysql.user TO 'maxscale'@'localhost';

MariaDB> GRANT SELECT ON mysql.db TO 'maxscale'@'localhost';

MariaDB> GRANT SELECT ON mysql.tables_priv TO 'maxscale'@'localhost';

MariaDB> GRANT SHOW DATABASES ON *.* TO 'maxscale'@'localhost';

After a restart, MaxScale will show the UNIX socket path instead of the actual address, and the server listing will be shown like this:

As you can see, the state and GTID information are retrieved correctly through a socket connection. Note that this DB_2 is still listening to port 3306 for the remote connections. It just shows that MaxScale uses a socket to connect to this server for monitoring.

Using socket is always better due to the fact that it only allows local connections and it is more secure. You could also close down your MariaDB server from the network (e.g, --skip-networking) and let MaxScale handle the "external" connections and forward them to the MariaDB server via UNIX socket file.

Server Draining

In MaxScale 2.4, the backend servers can be drained, which means existing connections can continue to be used, but no new connections will be created to the server. With the drain feature, we can perform a graceful maintenance activity without affecting the user experience from the application side. Note that draining a server can take a longer time, depending on the running queries that need to be gracefully closed.

To drain a server, use the following command:

The after-effect could be one of the following states:

Draining - The server is being drained.
Drained - The server has been drained. The server was being drained and now the number of connections to the server has dropped to 0.
Maintenance - The server is under maintenance.

After a server has been drained, the state of the MariaDB server from MaxScale point of view is "Maintenance":

When a server is in maintenance mode, no connections will be created to it and existing connections will be closed.

Conclusion

MaxScale 2.4 brings a lot of improvements and changes over the previous version and it's the best database proxy to handle MariaDB servers and all of its components.

Tags:

↧

Deploying MariaDB Replication for High Availability

July 16, 2020, 1:05 pm

≫ Next: Determining the Best Architecture for a MongoDB Cluster Deployment

≪ Previous: What's New in MariaDB MaxScale 2.4

MariaDB Server offers asynchronous and synchronous replication. It can be set up to have a multi-source replication or with a multi-master setup.

For a read and write intensive application, a master-slave setup is common, but can differ based on the underlying stack needed to build a highly available database environment.

Having a master-slave replication setup might not satisfy your needs, especially in a production environment. A MariaDB Server alone (master-slave setup) is not enough to offer high availability as it still has a single point of failure (SPOF).

MariaDB introduced an enterprise product (MariaDB Platform) to address this high availability issue. It includes various components: an enterprise version of MariaDB, MariaDB ColumnStore, MaxScale, and lightweight MariaDB Connectors. Compared to other vendors with the same enterprise solution offering, it could be a cost effective option, however not everyone needs this level of complexity.

In this blog, we'll show you how to use MariaDB Server using replication on a highly available environment with the option to choose from using all free tools or our cost-efficient, management software to run and monitor your MariaDB Server infrastructure.

MariaDB High-Availability Topology Setup

A usual setup for a master-slave topology with MariaDB Server uses asynchronous or synchronous approach with just one master receiving writes, then replicates its changes to the slaves just like the diagram below:

But then again, this doesn't serve any high availability and has a single point of failure. If the master dies, then your application client no longer functions. Now, we need to add in the stack to have an auto-failover mechanism to avoid SPOF and also offers load balancing for splitting read-writes and in a round-robin fashion. So for now, we'll end up having the type of topology,

Now, this topology serves more safety in terms of SPOF. MaxScale will do the read and write splitting over the database nodes from your master against the slaves. MaxScale does a perfect approach when dealing with this type of setup. MaxScale also has auto-detection built-in. So whatever changes occur on the state of your database nodes, it will detect and act accordingly. MaxScale has the availability to proceed a failover or even a switchover. To know more about its failover mechanism, read our previous blog which tackles the mechanism of MariaDB MaxScale failover.

Take note that MaxScale failover mechanism with MariaDB Monitor also has its limitations. It's best applied only for a master-slave setup with no overcomplicated setup. This means that a master-master setup is not supported. However, MaxScale has more things to offer. It does not only do some load balancing as it performs read-write splits, it has built-in SmartRouter which sends the query to the most performant node. Although this doesn't add the feature of being highly available but it helps the nodes from getting stuck in traffic and avoid certain database nodes from under-performing that can cause timeouts or to a totally unavailable server caused by high resource intensive activity on-going.

One thing as a caveat of using MaxScale, they are using BSL (Business Source LIcense). You might have to review the FAQ before adopting this software.

Another option to choose from is using a more convenient approach. It can be cost-efficient for you to choose using ClusterControl and have proxies in the middle using HaProxy, MaxScale, or ProxySQL, for which the latter can be configured to from light-weight to a more production-level configuration that does query routing, query filtering, firewall, or security. See the illustration below:

Now, sitting on top of them is the ClusterControl. ClusterControl is set up with a high availability i.e. CMON HA. Alternatively, the proxy layer can be chosen from either HaProxy--a very lightweight option to choose from, MaxScale, as mentioned previously, or ProxySQL which has a more refined set of parameters if you want more flexibility and configuration ideal for a high-scaled production setup. ClusterControl will handle the auto-detection in terms of the health status of the nodes, especially the master which is the main node to determine if it requires a failover or not. Now, this can be more self-sufficient yet it adds more cost due to a number of nodes required to implement this setup and also using ClusterControl auto-failover which applies on our advance and enterprise license. But on the other hand, it provides you all the safety, security, and observability towards your database infrastructure. It is actually more of a low-cost enterprise implementation compared to the available solutions in the global market.

Deploying Your MariaDB Master-Slave Replication for High Availability

Let's assume that you have an existing master-slave setup of MariaDB. For this example, we’ll use ClusterControl using the free community edition which you can install and use free of charge. It just makes your work easy and quick to set up. To do this, you just have to import your existing MariaDB Replication cluster. Checkout our previous blog on how to manage MariaDB with ClusterControl. For this blog, I have the following setup initially as my MariaDB Replication cluster as seen below:

Now, let's use MaxScale here as an alternative solution from MariaDB Platform which also offers high availability. To do that, it's very easy to use with ClusterControl by just a few clicks, you are then able to set up your MaxScale that is running on-top of your existing MariaDB Replication cluster. To do that, just go to Manage → Load Balancer → MaxScale, and you'll be able to setup and provide the appropriate values as seen below,

Then just enable or click the checkbox option to select which servers have to be added as part of your MaxScale monitoring. See below,

Assuming that you have more than one MaxScale node to add, just repeat the same steps.

Lastly, we'll set up Keepalived to keep our MaxScale nodes always available whenever necessary. This is just very quick with just simple steps using ClusterControl. Again, you have to go to Manage → Load Balancer but instead, select Keepalived,

As you've noticed, I've placed my Keepalived along with MaxScale on the same node of my slave (192.168.10.30). Whereas, on the other hand, the second (2nd) Keepalived is running on 192.168.10.40 along with Maxscale on the same host.

The result of the topology is production ready which can provide you query routing, high availability, and with auto-failover equipped with extensive monitoring and observability using ClusterControl. See below,

Conclusion

Using MariaDB Server replication alone does not offer you high availability. Extending and using third-party tools will equip you to have your database stack highly available by not only relying on MariaDB products or even using MariaDB Platform.

There are ways to achieve this and manage it to be more cost-effective. Yet, there is a huge difference to availing to these solutions available in the market such as ClusterControl since it provides you speed, less hassle,, and of course the ultimate observability with real-time and up-to-date events not only the health but also the events occurring in your database cluster.

Tags:

↧

Determining the Best Architecture for a MongoDB Cluster Deployment

July 17, 2020, 11:50 am

≫ Next: How to Design a Geographically Distributed MariaDB Cluster

≪ Previous: Deploying MariaDB Replication for High Availability

Cluster deployments are of great significance in ensuring the high availability of data as well as protecting it. MongoDB enhances this through replication and sharding, whereby replication ensures vertical scaling through lifting redundancy whereas sharding inflates horizontal scaling.

In general, both approaches try to distribute the workload among the members and thus reduce the workload over which a single node could be subjected to. The database performance can then be seen as fast in serving users with throughput operations. However, without a prime cluster architecture, you may not see the same level of results, even if you try sharding and replication.

If the replica set members are even, then it will be hard for the members to vote and elect for a new primary if the existing one fails at some point. In this blog we are going to discuss the standard deployment architecture one can use but this may vary in accordance to application requirements.

MongoDB Deployment Strategies

The architecture of replica sets is very determinant of the capacity and capability of MongoDB.

A three node replica set is the standard cluster deployment for MongoDB in any production environment as it provides data redundancy and fault tolerance. Redundancy is important especially in database recovery after disaster strikes. A three node replica set may be the basic deployment architecture, but this may vary according to application specifications and requirements. However, do not make it too complex as it may lead you to some bigger configuration problems.

MongoDB Sharding Strategies

Sharding reduces the workload over which the database is to work on for a given query by reducing the number of documents that have to be acted upon. It therefore uplifts horizontal scaling allowing the database to grow beyond the hardware limits of a single server. Depending on the workload demand, nodes can be added or removed from the cluster and MongoDB will rebalance the data in an optimal way without operation intervention.

Some of the best deployment strategies for a sharded cluster include:

Ensuring Shard Keys are Distributed Uniformly

The reason behind sharding is to scale the database horizontally and reduce the number of throughput operations a single instance could be subjected to. If you don’t distribute the shard keys uniformly, you may end up with a small number of shards. With few shards, operations may be limited by the capacity of a single shard hence making read and write operations slow.

Chunks Should be Pre-Split and Distributed First

Shards have data chunks that are grouped according to some shard key criteria. When creating a new sharded collection, before loading it with data, you should create empty chunks and distribute them evenly on all shards. When you will be populating MongoDB with data, it will be easy to balance the load across the involved shards. The numInitialChunks option can be used to do these automatically if you are using hash-based sharding. The integer value however should be less than 8192 per shard.

Number of Shards

Two shards are often required as the minimum number for sharding significance to be achieved. A single shard is only useful if you want to lay out the foundation of enabling sharding in future and no need during the deployment time.

Prefer Range-Based Sharding Over Hash-Based Sharding

Range-based sharding is beneficial as it provides more shards, hence operations can be routed to the fewest shards necessary and more often a single shard. Practically this may be difficult not unless you have a good understanding of the data and query patterns involved. Hashed sharding improves the uniform distribution of throughput operation at the expense of providing inferior range-based operations.

Use Scatter-Gather Queries for Only Large Aggregation Queries

Queries that cannot be routed based on a shard key should be broadcast to all shards for evaluation and since they involve multiple shards for each request, they do not scale linearly as more shards are added hence incurring an overhead that degrades the performance of the database. This operation is known as scatter-gather and can only be avoided if you include the shard key in your query.

The approach is only useful when dealing with large aggregation queries that each query can be allowed to run in parallel on all shards.

MongoDB Replication Strategies

Replication enhances vertical scaling in MongoDB such that the workload is distributed among the involved members. In the production environment, these are some of the considerations one should make for an optimal cluster architecture.

Number of Nodes

The maximum number of nodes a replica set can have is 50 with 7 voting members. Any member after the 7th is considered to be non-voting. A good cluster should therefore have 7 voting members to make the election process convenient.

Deploy an odd number of voting members and if you only have less than 7 but even number of members, then you will need to deploy an arbiter as another voting member. Arbiters do not store a copy of the data hence will require fewer resources to manage. Besides, one can subject them to an environment you could not subject the other members.

Fault Tolerance Considerations

Sometimes some members may become unavailable as a result of factors such as power outages or network transients and disconnections. The number of members that remain in the set and capable of electing a primary create a situation known as Fault tolerance. It is therefore the difference between the total number of replica set members and the majority of voting members needed to elect a primary. Absence of a primary dictates that write operations cannot be executed.

The table below shows a sample relationship between the three.

Total replica set members	Majority required to Elect new primary	Fault Tolerance
3	2	1
4	3	1
5	3	2
6	4	2
7	5	2

The relationship is not that direct in that if you add more members to the set it is not given that the fault tolerance will increase as seen from the table. Additional members provide support for dedicated functions such as backups and reporting.

Capacity Planning and Load Balancing for Heavy Reads

You need to have a spare capacity for your deployment by adding new members before the current demand saturates the capacity of the existing set.

For very high read traffic, distribute the throughput reads to the secondary members and whenever the cluster grows, add or move members to alternate data centers in order to achieve redundancy and increase data availability.

You can also use target operations with tag sets to target read operations to specific members or modify write concern to request acknowledgement from specific members.

Nodes Should be Distributed Geographically

Data centers may also fail because of some catastrophe . One is therefore advised to keep at least one or two members in a separate data center for data protection purposes. If possible, use an odd number of data centers and select a distribution that maximizes the likelihood that even with loss of a data center, remaining replica set members can form a majority or at minimum provide a copy of the data.

Employ Journaling for Unexpected Failures

By default this is enabled in MongoDB. You should ensure this option is enabled so as to protect data loss in an event of service interruptions such as sudden reboots and power failures.

Deployment Patterns

There are mainly two deployment approaches that is:

Three Member replica sets which provide minimum recommended architecture for a replica set.
Replica set distributed across two or more data centers to protect against facility-specific failures like power outages.

The patterns however are dependent on application requirements but if possible, you can combine features of these two in your deployment architecture.

Hostnames and Replica Set Naming

Use logical DNS hostname rather than ip address when configuring replica set members or sharded cluster members. This is to avoid the pain involved with configuration changes you will need to make as a result of changed ip addresses.

In the case of replica set naming, use distinct names for the sets since some drivers group replica set connections by replica set name.

Conclusion

Architecture layout for a replica set determines the capacity and capability of your deployment. Data protection and system performance are the core considerations when setting up the architecture. One should consider vital factors such as Fault tolerance, number of replica set members, optimal sharding key and deployment patterns for high availability & data protection. Geographical distribution of the replica set nodes can address much of these factors by ensuring redundancy and providing fault tolerance if one of the data centers is absent.

Tags:

↧

How to Design a Geographically Distributed MariaDB Cluster

July 20, 2020, 10:13 am

≫ Next: MaxScale Basic Management Using MaxCtrl for MariaDB Cluster

≪ Previous: Determining the Best Architecture for a MongoDB Cluster Deployment

It is very common to see databases distributed across multiple geographical locations. One scenario for doing this type of setup is for disaster recovery, where your standby data center is located in a separate location than your main datacenter. It might as well be required so that the databases are located closer to the users.

The main challenge to achieving this setup is by designing the database in a way that reduces the chance of issues related to the network partitioning.

MariaDB Cluster can be a good choice to build such an environment for several reasons. We would like to discuss them here and also talk a bit about how such an environment may look like.

Why Use MariaDB Cluster for Geo-Distributed Environments?

First reason is that MariaDB Cluster can support multiple writers. This makes the write routing way easier to design - you just write to the local MariaDB nodes. Of course, given synchronous replication, latency impacts the write performance and you may see that your writes are getting slower if you spread your cluster too far geographically. After all, you can’t ignore the laws of physics and they say, as of now at least, that even the speed of light in fiber connections is limited. Any routers added on top of that will also increase latency even if only by a couple milliseconds.

Second, lag handling in MariaDB Cluster. Asynchronous replication is a subject for replication lag - slaves may not be up to date with the data if they struggle to apply all the changes in time. In MariaDB Cluster this is different - flow control is a mechanism that is intended to keep the cluster in sync. Well, almost - in some edge cases you can still observe lag. We are talking here about, typically, milliseconds, a couple seconds at most while in the asynchronous replication sky is the limit.

Why Use MariaDB Cluster for Geo-Distributed Environments?

Third, segments. By default MariaDB CLuster uses all to all communication and every writeset is sent by the node to all other nodes in the cluster. This behavior can be changed using segments. Segments allow users to split MariaDB Clusters in several parts. Each segment may contain multiple nodes and it elects one of them as a relay node. Such nodes receive writesets from other segments and redistribute them across MariaDB nodes local to the segment. As a result, as you can see on the diagram above, it is possible to reduce the replication traffic going over WAN three times - just two “replicas” of the replication stream are being sent over WAN: one per datacenter compared to one per slave in asynchronous replication.

Finally, where MariaDB Cluster really shines is the handling of the network partitioning. MariaDB Cluster constantly monitors the state of the nodes in the cluster. Every node attempts to connect with its peers and exchange the state of the cluster. If a subset of nodes is not reachable, MariaDB attempts to relay the communication so if there is a way to reach those nodes, they will be reached.

An example can be seen on the diagram above: DC 1 lost the connectivity with DC2 but DC2 and DC3 can connect. In this case one of the nodes in DC3 will be used to relay data from DC1 to DC2 ensuring that the intra-cluster communication can be maintained.

MariaDB is able to take actions based on the state of the cluster. It implements quorum - majority of the nodes have to be available in order for the cluster to be able to operate. If node gets disconnected from the cluster and cannot reach any other node, it will cease to operate.

As can be seen on the diagram above, there’s a partial loss of the network communication in DC1 and the affected node is removed from the cluster, ensuring that the application will not access outdated data.

This is also true on a larger scale. The DC1 got all of its communication cut off. As a result, the whole datacenter has been removed from the cluster and neither of its nodes will serve the traffic. The rest of the cluster maintained majority (6 out of 9 nodes are available) and it reconfigured itself to keep the connection between DC 2 and DC3. In the diagram above we assumed the writer hits the node in DC2 but please keep in mind that MariaDB is capable of running with multiple writers.

Designing Geographically Distributed MariaDB Cluster

We went through some of the features that make MariaDB Cluster a nice fit for geo-distributed environments, let’s focus now a bit on the design. At the beginning, let’s explain what the environment we are working with. We will use three remote data centers, connected via Wide Area Network (WAN). Each datacenter will receive writes from local application servers. Reads will also be only local. This is intended to avoid unnecessary traffic crossing the WAN.

To make this blog less complicated, we won’t be going into details of how the connectivity should look like. We assume some sort of a properly configured, secure connection across all datacenters. VPN or other tools can be used to implement that.

We will use MaxScale as a loadbalancer. MaxScale will be deployed locally in each datacenter. It will also route traffic only to the local nodes. Remote nodes can always be added manually and we will explain cases where this might be a good solution. Applications can be configured to connect to one of the local MaxScale nodes using a round-robin algorithm. We can as well use Keepalived and Virtual IP to route the traffic towards the single MaxScale node, as long as a single MaxScale node would be able to handle all of the traffic.

Another possible solution is to collocate MaxScale with application nodes and configure the application to connect to the proxy on the localhost. This approach works quite well under the assumption that it is unlikely that MaxScale will not be available yet the application would work ok on the same node. Typically what we see is either node failure or network failure, which would affect both MaxScale and application at the same time.

Designing Geographically Distributed MariaDB Cluster

The diagram above shows the version of the environment, where MaxScale forms proxy farms - all proxy nodes with the same configuration, load balanced using Keepalived, or just simply round robin from the application across all MaxScale nodes. MaxScale is configured to distribute the workload across all MariaDB nodes in the local datacenter. One of those nodes would be picked as a node to send the writes to while SELECTs would be distributed across all nodes. Having one dedicated writer node in a datacenter helps to reduce the number of possible certification conflicts, leading to, typically, better performance. To reduce this even further we would have to start sending the traffic over the WAN connection, which is not ideal as the bandwidth utilization would significantly increase. Right now, with segments in place, only two copies of the writeset are being sent across datacenters - one per DC.

Conclusion

As you can see, MariaDB Cluster can easily be used to create geo-distributed clusters that can work even across the world. The limiting factor will be network latency. If it is too high, you may have to consider using separate MariaDB clusters connected using asynchronous replication.

Tags:

multi-dc

geo-location

↧

MaxScale Basic Management Using MaxCtrl for MariaDB Cluster

July 21, 2020, 11:37 am

≫ Next: Tips for Monitoring MariaDB Cluster

≪ Previous: How to Design a Geographically Distributed MariaDB Cluster

In the previous blog post, we have covered some introductions to MaxScale installation, upgrade, and deployment using MaxCtrl command-line client. In this blog post, we are going to cover the MaxScale management aspects for our MariaDB Cluster.

There are a number of MaxScale components that we can manage with MaxCtrl, namely:

Server management
Service management
Monitor management
Listener management
Filter management
MaxScale management
Logging management

In this blog post, we are going to cover the first 4 components which are commonly used in MariaDB Cluster. All of the commands in this blog post are based on MaxScale 2.4.11.

Server Management

List/Show Servers

List a summary of all servers in MaxScale:

 maxctrl list servers

┌────────────────┬────────────────┬──────┬─────────────┬─────────────────────────┬─────────────┐

│ Server         │ Address        │ Port │ Connections │ State                   │ GTID        │

├────────────────┼────────────────┼──────┼─────────────┼─────────────────────────┼─────────────┤

│ mariadbgalera1 │ 192.168.10.201 │ 3306 │ 0           │ Slave, Synced, Running  │ 100-100-203 │

├────────────────┼────────────────┼──────┼─────────────┼─────────────────────────┼─────────────┤

│ mariadbgalera2 │ 192.168.10.202 │ 3306 │ 0           │ Slave, Synced, Running  │ 100-100-203 │

├────────────────┼────────────────┼──────┼─────────────┼─────────────────────────┼─────────────┤

│ mariadbgalera3 │ 192.168.10.203 │ 3306 │ 0           │ Master, Synced, Running │ 100-100-203 │

└────────────────┴────────────────┴──────┴─────────────┴─────────────────────────┴─────────────┘

For MariaDB Cluster, the server list summarizes the node and cluster state, with its MariaDB GTID, only if the cluster is set to replicate from another cluster via the standard MariaDB Replication. The state is used by MaxScale to control the behavior of the routing algorithm:

Master - For a Cluster, this is considered the Write-Master.
Slave - If all slaves are down, but the master is still available, then the router will use the master.
Synced - A Cluster node which is in a synced state with the cluster.
Running - A server that is up and running. All servers that MariaDB MaxScale can connect to are labeled as running.

Although MariaDB Cluster is capable of handling multi-master replication, MaxScale will always pick one node to hold the Master role which will receive all writes for readwritesplit routing. By default, the Galera Monitor will choose the node with the lowest wsrep_local_index value as the master. This will mean that two MaxScales running on different servers will choose the same server as the master.

Show all servers in more detail:

maxctrl: show servers

Create Servers

This is commonly the first thing you need to do when setting up MaxScale as a load balancer. It's common to add all of the MariaDB Cluster nodes into MaxScale and label it with an object name. In this example, we label the Galera nodes as in "mariadbgalera#" format:

maxctrl: create server mariadbgalera1 192.168.0.221 3306

maxctrl: create server mariadbgalera2 192.168.0.222 3306

maxctrl: create server mariadbgalera3 192.168.0.222 3306

The server state will only be reported correctly after we have activated the monitoring module, as shown under the Monitor Management section further down.

Delete a Server

To delete a server, one has to unlink the server from any services or monitors beforehand. As an example, in the following server list, we would want to delete mariadbgalera3 from MaxScale:

  maxctrl: list servers

┌────────────────┬────────────────┬──────┬─────────────┬─────────────────────────┬─────────────┐

│ Server         │ Address        │ Port │ Connections │ State                   │ GTID        │

├────────────────┼────────────────┼──────┼─────────────┼─────────────────────────┼─────────────┤

│ mariadbgalera1 │ 192.168.10.201 │ 3306 │ 0           │ Slave, Synced, Running  │ 100-100-203 │

├────────────────┼────────────────┼──────┼─────────────┼─────────────────────────┼─────────────┤

│ mariadbgalera2 │ 192.168.10.202 │ 3306 │ 0           │ Slave, Synced, Running  │ 100-100-203 │

├────────────────┼────────────────┼──────┼─────────────┼─────────────────────────┼─────────────┤

│ mariadbgalera3 │ 192.168.10.203 │ 3306 │ 0           │ Master, Synced, Running │ 100-100-203 │

└────────────────┴────────────────┴──────┴─────────────┴─────────────────────────┴─────────────┘

List out all monitors and see if the server is part of any monitor module:

 maxctrl: list monitors

 ┌─────────────────┬─────────┬────────────────────────────────────────────────┐

 │ Monitor         │ State   │ Servers                                        │

 ├─────────────────┼─────────┼────────────────────────────────────────────────┤

 │ MariaDB-Monitor │ Running │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

 └─────────────────┴─────────┴────────────────────────────────────────────────┘

Looks like mariadbgalera3 is part of MariaDB-Monitor, so we have to remove it first by using the "unlink monitor" command:

 maxctrl: unlink monitor MariaDB-Monitor mariadbgalera3

 OK

Next, list out all services to check if the corresponding server is part of any MaxScale services:

  maxctrl: list services

┌─────────────────────┬────────────────┬─────────────┬───────────────────┬────────────────────────────────────────────────┐

 │ Service             │ Router         │ Connections │ Total Connections │ Servers                                        │

 ├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

 │ Read-Write-Service  │ readwritesplit │ 1           │ 1                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

 ├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

 │ Round-Robin-Service │ readconnroute  │ 1           │ 1                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

 ├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

 │ Replication-Service │ binlogrouter   │ 1           │ 1                 │                                                │

 └─────────────────────┴────────────────┴─────────────┴───────────────────┴────────────────────────────────────────────────┘

As you can see, mariadbgalera3 is part of the Read-Write-Service and Round-Robin-Service. Remove the server from those services by using "unlink service" command:

 maxctrl: unlink service Read-Write-Service mariadbgalera3

OK

 maxctrl: unlink service Round-Robin-Service mariadbgalera3

OK

Finally, we can remove the server from MaxScale by using the "destroy server" command:

 maxctrl: destroy server mariadbgalera3

OK

Verify using the "list servers" that we have removed mariadbgalera3 from MaxScale.:

  maxctrl: list servers

┌────────────────┬────────────────┬──────┬─────────────┬─────────────────────────┬──────┐

 │ Server         │ Address        │ Port │ Connections │ State                   │ GTID │

 ├────────────────┼────────────────┼──────┼─────────────┼─────────────────────────┼──────┤

 │ mariadbgalera1 │ 192.168.10.201 │ 3306 │ 0           │ Master, Synced, Running │      │

 ├────────────────┼────────────────┼──────┼─────────────┼─────────────────────────┼──────┤

 │ mariadbgalera2 │ 192.168.10.202 │ 3306 │ 0           │ Slave, Synced, Running  │      │

 └────────────────┴────────────────┴──────┴─────────────┴─────────────────────────┴──────┘

Modify Server's Parameter

To modify a server's parameter, one can use the "alter server" command which only takes one key/value parameter at a time. For example:

  maxctrl: alter server mariadbgalera3 priority 10

 OK

Use the "show server" command and look into the Parameters section, for a list of parameters that can be changed for the "server" object:

maxctrl: show server mariadbgalera3

...

│ Parameters       │ {                                         │

│                  │     "address": "192.168.10.203",          │

│                  │     "protocol": "mariadbbackend",         │

│                  │     "port": 3306,                         │

│                  │     "extra_port": 0,                      │

│                  │     "authenticator": null,                │

│                  │     "monitoruser": null,                  │

│                  │     "monitorpw": null,                    │

│                  │     "persistpoolmax": 0,                  │

│                  │     "persistmaxtime": 0,                  │

│                  │     "proxy_protocol": false,              │

│                  │     "ssl": "false",                       │

│                  │     "ssl_cert": null,                     │

│                  │     "ssl_key": null,                      │

│                  │     "ssl_ca_cert": null,                  │

│                  │     "ssl_version": "MAX",                 │

│                  │     "ssl_cert_verify_depth": 9,           │

│                  │     "ssl_verify_peer_certificate": false, │

│                  │     "disk_space_threshold": null,         │

│                  │     "priority": "10"│

│                  │ }

Note that alter command effect is immediate and the parameter's value in the runtime will be modified as well as the value in its individual MaxScale configuration file inside /var/lib/maxscale/maxscale.cnf.d/ for persistence across restart.

Set Server State

MaxScale allows the backend Galera servers to be temporarily excluded from the load balancing set by activating the maintenance mode. We can achieve this by using the "set server" command:

 maxctrl: set server mariadbgalera3 maintenance

OK

When looking at the state of the server, we should see this:

 maxctrl: show server mariadbgalera3

...

│ State            │ Maintenance, Running

...

When a server is in maintenance mode, no connections will be created to it and existing connections will be closed. To clear the maintenance state from the host, use the "clear server" command:

 maxctrl: clear server mariadbgalera3 maintenance

OK

Verify with "show server":

 maxctrl: show server mariadbgalera3

...

│ State            │ Slave, Synced, Running                    │

...

Monitor Management

Create a Monitor

The MaxScale monitor module for MariaDB Cluster is called galeramon. Defining a correct monitoring module is necessary so MaxScale can determine the best routing for queries depending on the state of the nodes. For example, if a Galera node is serving as a donor for a joiner node, should it be part of the healthy nodes? In some cases like where the database size is so small, marking a donor node as healthy (by setting the parameter available_when_donor=true in MaxScale) is not a bad plan and sometimes improves the query routing performance.

To create a service (router), one must create a monitoring user on the backend of MariaDB servers. Commonly, one would use the same monitoring user that we have defined for the monitor module. For Galera Cluster, if the monitoring user does not exist, just create it on one of the nodes with the following privileges:

MariaDB> CREATE USER maxscale_monitor@'192.168.0.220' IDENTIFIED BY 'MaXSc4LeP4ss';

MariaDB> GRANT SELECT ON mysql.* TO 'maxscale_monitor'@'192.168.0.220';

MariaDB> GRANT SHOW DATABASES ON *.* TO 'maxscale_monitor'@'192.168.0.220';

Use the "create monitor" command and specify a name with galeramon as the monitor module:

  maxctrl: create monitor MariaDB-Monitor galeramon servers=mariadbgalera1,mariadbgalera2,mariadbgalera3 user=maxscale_monitor password=MaXSc4LeP4ss

OK

Note that we didn't configure MaxScale secret which means we store the user password in plain text format. To enable encryption, see the example in this blog post, Introduction to MaxScale Administration Using maxctrl for MariaDB Cluster under Adding Monitoring into MaxScale section.

List/Show Monitors

To list out all monitors:

 maxctrl: list monitors

┌─────────────────┬─────────┬────────────────────────────────────────────────┐

│ Monitor         │ State   │ Servers                                        │

├─────────────────┼─────────┼────────────────────────────────────────────────┤

│ MariaDB-Monitor │ Running │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

└─────────────────┴─────────┴────────────────────────────────────────────────┘

To get a more detailed look on the monitor, use the "show monitor" command:

 maxctrl: show monitor MariaDB-Monitor

┌─────────────────────┬───────────────────────────────────────────┐

│ Monitor             │ MariaDB-Monitor                           │

├─────────────────────┼───────────────────────────────────────────┤

│ State               │ Running                                   │

├─────────────────────┼───────────────────────────────────────────┤

│ Servers             │ mariadbgalera1                            │

│                     │ mariadbgalera2                            │

│                     │ mariadbgalera3                            │

├─────────────────────┼───────────────────────────────────────────┤

│ Parameters          │ {                                         │

│                     │     "user": "maxscale_monitor",           │

│                     │     "password": "*****",                  │

│                     │     "passwd": null,                       │

│                     │     "monitor_interval": 2000,             │

│                     │     "backend_connect_timeout": 3,         │

│                     │     "backend_read_timeout": 1,            │

│                     │     "backend_write_timeout": 2,           │

│                     │     "backend_connect_attempts": 1,        │

│                     │     "journal_max_age": 28800,             │

│                     │     "disk_space_threshold": null,         │

│                     │     "disk_space_check_interval": 0,       │

│                     │     "script": null,                       │

│                     │     "script_timeout": 90,                 │

│                     │     "events": "all",                      │

│                     │     "disable_master_failback": false,     │

│                     │     "available_when_donor": true,         │

│                     │     "disable_master_role_setting": false, │

│                     │     "root_node_as_master": false,         │

│                     │     "use_priority": false,                │

│                     │     "set_donor_nodes": false              │

│                     │ }                                         │

├─────────────────────┼───────────────────────────────────────────┤

│ Monitor Diagnostics │ {                                         │

│                     │     "disable_master_failback": false,     │

│                     │     "disable_master_role_setting": false, │

│                     │     "root_node_as_master": false,         │

│                     │     "use_priority": false,                │

│                     │     "set_donor_nodes": false              │

│                     │ }                                         │

└─────────────────────┴───────────────────────────────────────────┘

Stop/Start Monitor

Stopping a monitor will pause the monitoring of the servers. This commonly being used in conjunction with "set server" command to manually control server states. To stop the monitoring service use the "stop monitor" command:

 maxctrl: stop monitor MariaDB-Monitor

OK

Verify the state with "show monitor":

 maxctrl: show monitors MariaDB-Monitor

┌─────────────────────┬───────────────────────────────────────────┐

│ Monitor             │ MariaDB-Monitor                           │

├─────────────────────┼───────────────────────────────────────────┤

│ State               │ Stopped                                   │

...

To start it up again, use the "start monitor":

 maxctrl: start monitor MariaDB-Monitor

OK

Modify Monitor's Parameter

To change a parameter for this monitor, use the "alter monitor" command and specify the parameter key/value as below:

 maxctrl: alter monitor MariaDB-Monitor available_when_donor true

OK

Use the "show monitor" command and look into the Parameters section, for a list of parameters that can be changed for the galeramon module:

maxctrl: show server mariadbgalera3

...

│ Parameters          │ {                                         │

│                     │     "user": "maxscale_monitor",           │

│                     │     "password": "*****",                  │

│                     │     "monitor_interval": 2000,             │

│                     │     "backend_connect_timeout": 3,         │

│                     │     "backend_read_timeout": 1,            │

│                     │     "backend_write_timeout": 2,           │

│                     │     "backend_connect_attempts": 1,        │

│                     │     "journal_max_age": 28800,             │

│                     │     "disk_space_threshold": null,         │

│                     │     "disk_space_check_interval": 0,       │

│                     │     "script": null,                       │

│                     │     "script_timeout": 90,                 │

│                     │     "events": "all",                      │

│                     │     "disable_master_failback": false,     │

│                     │     "available_when_donor": true,         │

│                     │     "disable_master_role_setting": false, │

│                     │     "root_node_as_master": false,         │

│                     │     "use_priority": false,                │

│                     │     "set_donor_nodes": false              │

│                     │ }                                         │

Delete a Monitor

In order to delete a monitor, one has to remove all servers linked with the monitor first. For example, consider the following monitor in MaxScale:

 maxctrl: list monitors

┌─────────────────┬─────────┬────────────────────────────────────────────────┐

│ Monitor         │ State   │ Servers                                        │

├─────────────────┼─────────┼────────────────────────────────────────────────┤

│ MariaDB-Monitor │ Running │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

└─────────────────┴─────────┴────────────────────────────────────────────────┘

Remove all servers from that particular service:

 maxctrl: unlink monitor MariaDB-Monitor mariadbgalera1 mariadbgalera2 mariadbgalera3

OK

Our monitor is now looking like this:

 maxctrl: list monitors

┌─────────────────┬─────────┬─────────┐

│ Monitor         │ State   │ Servers │

├─────────────────┼─────────┼─────────┤

│ MariaDB-Monitor │ Running │         │

└─────────────────┴─────────┴─────────┘

Only then we can delete the monitor:

maxctrl: destroy monitor MariaDB-Monitor

OK

Add/Remove Servers into Monitor

After creating a monitor, we can use the "link monitor" command to add the Galera servers into the monitor. Use the server's name as created under Create Servers section:

 maxctrl: link monitor MariaDB-Monitor mariadbgalera1 mariadbgalera2 mariadbgalera3

OK

Similarly, to remove a server from the service, just use "unlink monitor" command:

 maxctrl: unlink monitor MariaDB-Monitor mariadbgalera3

OK

Verify with "list monitors" or "show monitors" command.

Service Management

Create a Service

MariaDB> CREATE USER maxscale_monitor@'192.168.0.220' IDENTIFIED BY 'MaXSc4LeP4ss';

MariaDB> GRANT SELECT ON mysql.* TO 'maxscale_monitor'@'192.168.0.220';

MariaDB> GRANT SHOW DATABASES ON *.* TO 'maxscale_monitor'@'192.168.0.220';

Where 192.168.0.220 is the IP address of the MaxScale host.

Then, specify the name of the service, the routing type together with a monitoring user for MaxScale to connect to the backend servers:

 maxctrl: create service Round-Robin-Service readconnroute user=maxscale_monitor password=******

OK

Also, you can specify additional parameters when creating the service. In this example, we would like the "master" node to be included in the round-robin balancing set for our MariaDB Galera Cluster:

 maxctrl: create service Round-Robin-Service readconnroute user=maxscale_monitor password=****** router_options=master,slave

OK

Use the "show service" command to see the supported parameters. For round-robin router, the list as follows:

  maxctrl: show service Round-Robin-Service

│ Parameters          │ {                                          │

│                     │     "router_options": null,                │

│                     │     "user": "maxscale_monitor",            │

│                     │     "password": "*****",                   │

│                     │     "passwd": null,                        │

│                     │     "enable_root_user": false,             │

│                     │     "max_retry_interval": 3600,            │

│                     │     "max_connections": 0,                  │

│                     │     "connection_timeout": 0,               │

│                     │     "auth_all_servers": false,             │

│                     │     "strip_db_esc": true,                  │

│                     │     "localhost_match_wildcard_host": true, │

│                     │     "version_string": null,                │

│                     │     "weightby": null,                      │

│                     │     "log_auth_warnings": true,             │

│                     │     "retry_on_failure": true,              │

│                     │     "session_track_trx_state": false,      │

│                     │     "retain_last_statements": -1,          │

│                     │     "session_trace": 0

For the read-write split router, the supported parameters are:

  maxctrl: show service Read-Write-Service

...

│ Parameters          │ {                                                           │

│                     │     "router_options": null,                                 │

│                     │     "user": "maxscale_monitor",                             │

│                     │     "password": "*****",                                    │

│                     │     "passwd": null,                                         │

│                     │     "enable_root_user": false,                              │

│                     │     "max_retry_interval": 3600,                             │

│                     │     "max_connections": 0,                                   │

│                     │     "connection_timeout": 0,                                │

│                     │     "auth_all_servers": false,                              │

│                     │     "strip_db_esc": true,                                   │

│                     │     "localhost_match_wildcard_host": true,                  │

│                     │     "version_string": null,                                 │

│                     │     "weightby": null,                                       │

│                     │     "log_auth_warnings": true,                              │

│                     │     "retry_on_failure": true,                               │

│                     │     "session_track_trx_state": false,                       │

│                     │     "retain_last_statements": -1,                           │

│                     │     "session_trace": 0,                                     │

│                     │     "use_sql_variables_in": "all",                          │

│                     │     "slave_selection_criteria": "LEAST_CURRENT_OPERATIONS", │

│                     │     "master_failure_mode": "fail_instantly",                │

│                     │     "max_slave_replication_lag": -1,                        │

│                     │     "max_slave_connections": "255",                         │

│                     │     "retry_failed_reads": true,                             │

│                     │     "prune_sescmd_history": false,                          │

│                     │     "disable_sescmd_history": false,                        │

│                     │     "max_sescmd_history": 50,                               │

│                     │     "strict_multi_stmt": false,                             │

│                     │     "strict_sp_calls": false,                               │

│                     │     "master_accept_reads": false,                           │

│                     │     "connection_keepalive": 300,                            │

│                     │     "causal_reads": false,                                  │

│                     │     "causal_reads_timeout": "10",                           │

│                     │     "master_reconnection": false,                           │

│                     │     "delayed_retry": false,                                 │

│                     │     "delayed_retry_timeout": 10,                            │

│                     │     "transaction_replay": false,                            │

│                     │     "transaction_replay_max_size": "1Mi",                   │

│                     │     "optimistic_trx": false                                 │

│                     │ }

List/Show Services

To list out all created services (routers), use the "list services" command:

 maxctrl: list services

┌─────────────────────┬────────────────┬─────────────┬───────────────────┬────────────────────────────────────────────────┐

│ Service             │ Router         │ Connections │ Total Connections │ Servers                                        │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Read-Write-Service  │ readwritesplit │ 1           │ 1                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Round-Robin-Service │ readconnroute  │ 1           │ 1                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Binlog-Repl-Service │ binlogrouter   │ 1           │ 1                 │                                                │

└─────────────────────┴────────────────┴─────────────┴───────────────────┴────────────────────────────────────────────────┘

In the above examples, we have created 3 services, with 3 different routers. However, the Binlog-Repl-Service for our binlog server is not linked with any servers yet.

To show all services in details:

 maxctrl: show services

Or if you want to show a particular service:

 maxctrl: show service Round-Robin-Service

Stop/Start Services

Stopping a service will prevent all the listeners for that service from accepting new connections. Existing connections will still be handled normally until they are closed. To stop and start all services, use the "stop services":

 maxctrl: stop services

 maxctrl: show services

 maxctrl: start services

 maxctrl: show services

Or we can use the "stop service" to stop only one particular service:

 maxctrl: stop services Round-Robin-Service

Delete a Service

In order to delete a service, one has to remove all servers and destroy the listeners associated with the service first. For example, consider the following services in MaxScale:

 maxctrl: list services

┌─────────────────────┬────────────────┬─────────────┬───────────────────┬────────────────────────────────────────────────┐

│ Service             │ Router         │ Connections │ Total Connections │ Servers                                        │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Read-Write-Service  │ readwritesplit │ 1           │ 1                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Round-Robin-Service │ readconnroute  │ 1           │ 1                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Replication-Service │ binlogrouter   │ 1           │ 1                 │                                                │

└─────────────────────┴────────────────┴─────────────┴───────────────────┴────────────────────────────────────────────────┘

Let's remove Round-Robin-Service from the setup. Remove all servers from this particular service:

 maxctrl: unlink service Round-Robin-Service mariadbgalera1 mariadbgalera2 mariadbgalera3

OK

Our services are now looking like this:

 maxctrl: list services

┌─────────────────────┬────────────────┬─────────────┬───────────────────┬────────────────────────────────────────────────┐

│ Service             │ Router         │ Connections │ Total Connections │ Servers                                        │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Read-Write-Service  │ readwritesplit │ 1           │ 1                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Round-Robin-Service │ readconnroute  │ 1           │ 1                 │                                                │

├─────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Replication-Service │ binlogrouter   │ 1           │ 1                 │                                                │

└─────────────────────┴────────────────┴─────────────┴───────────────────┴────────────────────────────────────────────────┘

If the service is tied with a listener, we have to remove it as well. Use "list listeners" and specify the service name to look for it:

 maxctrl: list listeners Round-Robin-Service

┌──────────────────────┬──────┬─────────┬─────────┐

│ Name                 │ Port │ Host    │ State   │

├──────────────────────┼──────┼─────────┼─────────┤

│ Round-Robin-Listener │ 3307 │ 0.0.0.0 │ Running │

└──────────────────────┴──────┴─────────┴─────────┘

And then remove the listener:

 maxctrl: destroy listener Round-Robin-Service Round-Robin-Listener

OK

Finally, we can remove the service:

 maxctrl: destroy service Round-Robin-Service

OK

Modify Service's Parameter

Similar to the other object, one can modify a service parameter by using the "alter service" command:

 maxctrl: alter service Read-Write-Service master_accept_reads true

OK

Some routers support runtime configuration changes to all parameters. Currently all readconnroute, readwritesplit and schemarouter parameters can be changed at runtime. In addition to module specific parameters, the following list of common service parameters can be altered at runtime:

user
passwd
enable_root_user
max_connections
connection_timeout
auth_all_servers
optimize_wildcard
strip_db_esc
localhost_match_wildcard_host
max_slave_connections
max_slave_replication_lag
retain_last_statements

Add/Remove Servers into Service

After creating a service, we can use the link command to add our servers into the service. Use the server's name as created under Create Servers section:

 maxctrl: link service Round-Robin-Service mariadbgalera1 mariadbgalera2 mariadbgalera3

OK

Similarly, to remove a server from the service, just use "unlink service" command:

 maxctrl: unlink service Round-Robin-Service mariadbgalera3

OK

We can only remove one server from a service at a time, so repeat it for other nodes to delete them. Verify with "list services" or "show services" command.

Listener Management

List Listeners

To list all listeners, we need to know the service name in advanced:

maxctrl: list services

┌──────────────────────┬────────────────┬─────────────┬───────────────────┬────────────────────────────────────────────────┐

│ Service              │ Router         │ Connections │ Total Connections │ Servers                                        │

├──────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Read-Write-Service   │ readwritesplit │ 0           │ 0                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

├──────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

│ Round-Robin-Service  │ readconnroute  │ 0           │ 0                 │ mariadbgalera1, mariadbgalera2, mariadbgalera3 │

├──────────────────────┼────────────────┼─────────────┼───────────────────┼────────────────────────────────────────────────┤

In the above example, we have two services, Read-Write-Service and Round-Robin-Service. Then, we can list out the listener for that particular service. For Read-Write-Service:

 maxctrl: list listeners Read-Write-Service

┌─────────────────────┬──────┬─────────┬─────────┐

│ Name                │ Port │ Host    │ State   │

├─────────────────────┼──────┼─────────┼─────────┤

│ Read-Write-Listener │ 3306 │ 0.0.0.0 │ Running │

└─────────────────────┴──────┴─────────┴─────────┘

And for Round-Robin-Service:

 maxctrl: list listeners Round-Robin-Service

┌──────────────────────┬──────┬─────────┬─────────┐

│ Name                 │ Port │ Host    │ State   │

├──────────────────────┼──────┼─────────┼─────────┤

│ Round-Robin-Listener │ 3307 │ 0.0.0.0 │ Running │

└──────────────────────┴──────┴─────────┴─────────┘

Unlike other objects in MaxScale, the listener does not have a "show" and "alter" commands since it is a fairly simple object.

Create a Listener

Make sure a service has been created. In this example, taken from the Create Service section above, we will create a listener so MaxScale will listen on port 3307 to process the MariaDB connections in a round-robin fashion:

 maxctrl: create listener Round-Robin-Service Round-Robin-Listener 3307

OK

Delete a Listener

To delete a listener, use the "destroy listener" command with the respective service name and listener name:

 maxctrl: destroy listener Round-Robin-Service Round-Robin-Listener

OK

This concludes this episode of basic MaxScale management tasks for MariaDB Cluster. In the next series, we are going to cover the MaxScale advanced management tasks like service filters, MaxScale user management and so on.

Tags:

↧

Tips for Monitoring MariaDB Cluster

July 22, 2020, 10:14 am

≫ Next: Comparing MariaDB Server to MariaDB Cluster

≪ Previous: MaxScale Basic Management Using MaxCtrl for MariaDB Cluster

In previous blog posts, we have covered topics for Monitoring Your Galera Cluster whether it's MySQL or MariaDB. Although the technology versions don't differ much, MariaDB Cluster has some major changes since version 10.4.2. In this version it supports Galera Cluster 4 and has some great new features that we will look at in this blog post.

For beginners that are not yet familiar with MariaDB Cluster, is a virtually synchronous multi-master cluster for MariaDB. It is available on Linux only, and only supports the XtraDB/InnoDB storage engines (although there is experimental support for MyISAM - see the wsrep_replicate_myisam system variable).

The software is a bundled technology which is powered by MariaDB Server, MySQL-wsrep patch for MySQL Server and MariaDB Server developed by Codership (supports Unix-like OS), and the Galera wsrep provider library.

You might compare this product with MySQL Group Replication or with the MySQL InnoDB Cluster, which aims to provide high availability. (Though they differ diversely on principles and approaches for providing HA.)

Now that we’ve covered the basics, in this blog we are going to provide tips we think beneficial when monitoring your MariaDB Cluster.

The Essentials of MariaDB Cluster

When you start using MariaDB Cluster you have to identify what exactly is your purpose and why you have chosen MariaDB Cluster in the first place. First you have to digest what are the features and their benefits when using MariaDB Cluster. The reason to identify these is because those are essentially what have to be monitored and checked in order for you to determine performance, normal health conditions, and if it's running in accordance to your plans.

Essentially, it is identified as no slave lag, no lost transactions, read scalability, and smaller client latencies. Then questions can arise like, how does it make no slave lag, or lost transactions? How does it make read being scalable or with smaller latencies in the client side? These areas are one of the key areas you need to look and monitor especially for heavy production usage.

Although the MariaDB Cluster itself can be customized accordingly. Applying changes to the default behavior such as pc.weight or pc.ignore_quorum, or even using multicast with UDP for a large number of nodes, can impact the way you monitor the nature of your MariaDB Cluster. But on the other hand, the most essential status variables are usually your silver lining here knowing the state and flow of your cluster is doing fine or its degrading showing a possible problem leading to a catastrophic failure beforehand.

Always Monitor Your Server Activity (Network, Disk, Load, Memory, & CPU)

Monitoring your server activity can also be a complex task if you have a very complicated stack that is intertwined in your database architecture. However, for a MariaDB Cluster, it's always best to have your nodes always set up as dedicated yet simple as possible. Although that doesn't limit you from using all the spare resources, below are the common key areas you have to look into.

Network

Galera Cluster 4 features streaming replication as one of the key features and changes from the previous version. Since streaming replication addresses the drawbacks it had in the previous releases but allows it to manage more than 2GB of write-sets since Galera Cluster 4. This allows big transactions to be fragmented and is highly recommended to enable this during session level alone. This means, monitoring your network activity is very important and crucial to the normal activity of your MariaDB Cluster. This will help you identify which node did have the most or highest network traffic based on the period of time.

So how will that help you improve where nodes with the highest network traffic have been identified? Well, this provides you room for improvement with your database topology or the architectural layer of your database cluster. Using load balancers or a database proxy allows you to configure proactively your database traffic especially when determining which specific writes shall go to a specific node. Let's say, out of the 3 nodes, one of them is more capable of handling large and big queries due to differences with the hardware specifications. This allows you to manage more of your capex and improve your capacity planning as demands on a specified period of time changes.

Disk

As network activity matters also with your disk performance especially during flushing time. It's also best to determine how committed time and retrieval performs when high peak load is reached. There are times that you stock up your database host with not only being dedicated to a Galera Cluster activity but also mash up with other tools like docker, SQL proxies such as ProxySQL or MaxScale. This gives you control with low load servers and allows you to use the spare resources available that can be utilized for other beneficial purposes especially to your database architecture stack. Once you are able to determine which node upon monitoring has the lowest load but still capable of managing its disk IO utilization, then you can select the specific node while watching over the time passes by. Again, this still gives you better management with your capacity planning.

CPU, Memory, and Load Activity

Let me briefly put these three areas to look upon when monitoring. In this section, it's always best you have better observability of the following areas at once. It's quicker and easier to understand, especially ruling out a performance bottleneck or identifying bugs that cause your nodes to either stall and that can also affect the other nodes and the possibility of going down the cluster.

So how does CPU, memory, and load activity upon monitoring help your MariaDB Cluster? Well, as what I have mentioned above, those are one of the few things yet a big factor for daily routine checks. Now, this also helps you identify if these are periodical or random occurrences. If periodical, it might be related to backups running in one of your Galera nodes, or it's a massive query that requires optimization. For example, bad queries with no proper indexes, or in-balance usage of data retrieval such as doing a string comparison for such a large string. That can be undeniably inapplicable for OLTP type databases such as MariaDB Cluster especially if it's really the nature and requirements of your application. Better use other analytical tools such as MariaDB Columnstore, or other third-party analytic processing tools (Apache Spark, Kafka, or MongoDB, etc.) for large string data retrieval and/or string matching.

So with all these key areas being monitored, the question is, how it shall be monitored? It has to be monitored at least per-minute. With refined monitoring, i.e. per-second of collective metrics can be resource intensive and much greedy in terms of your resources. Although half-a-minute of collectivity is acceptable especially if your data and RPO (recovery point objective) is very low, so you need more granular and real-time data metrics. It is very important that you are able to oversee the whole picture of your database cluster. Aside from this, it's also best and important that whenever what metrics you are monitoring, you have the right tool to tap your attention when things are in danger or even just warnings. Using the proper tool such as ClusterControl helps you manage these key areas to be monitored. I'm using here a free version or community edition of ClusterControl and helps me monitor my nodes without any hassle from installation up to the monitoring of nodes by just a few clicks. For example, see the screenshots below:

The view is a more refined and quick overview of what's happening currently. A more granular graph can be used as well,

or with a more powerful and rich data model which also supports query language can provide you analysis of how your MariaDB Cluster performs based on historical data comparing its performance in a timely manner. For example,

That just provides you more visible metrics. So you see how important it really is to have the right tool when monitoring your MariaDB Cluster.

Ensure Collective Monitoring of Your MariaDB Cluster Statistic Variables

From time to time, it cannot be inevitable that MariaDB Cluster versions will produce new stats to monitor or enhance the nature of monitoring the database by providing more status variables and refine values to look upon. As what I have mentioned above, I am using ClusterControl to monitor my nodes in this example blog. However, that doesn't mean it's the best tool out there. I mean PMM from Percona is very rich when it comes to collective monitoring for every statistic variable that whenever MariaDB Cluster has newer statistic variables to offer, you can leverage this and also change it as PMM is an open-source tool. It's a great advantage that you also have all the visibility of your MariaDB Cluster as every aspect counts especially in a production-based database that caters hundreds of thousands of requests per minute.

But let's get more specific into the problem here. What are these statistical variables to look into? There's many to count on for a MariaDB Cluster but focusing again on the features and benefits that we believe you use the MariaDB Cluster what it has to offer, then we'll focus into that.

Galera Cluster - Flow Control

The flow control of your MariaDB Cluster provides you the overview of how the replication health performs on all over the cluster. The replication process in Galera Cluster uses a feedback mechanism, which means it signals all over the nodes within that cluster and flags whether the node has to pause or resume replication according to its needs. This also prevents any node from lagging too far while the others are applying the incoming transactions. This is how the flow control serves as its function within Galera. Now, this has to be seen and not to be overlooked when monitoring your MariaDB Cluster. This, as mentioned in one of the benefits upon using MariaDB Cluster is that the avoidance of having slave lag. Although that's too naive to understand about the flow control and the slave lag, but with flow control, it will impact your Galera cluster's performance when there's a lot of queue and commits or flushing of pages to the disk goes very low for such disk issues or it's just the query running is a bad query. If you're a beginner of how Galera works, you might be interested reading this external post about what is flow control in Galera.

Bytes Sent/Received

The bytes sent or received correlates with the networking activity and even is one of the key areas to look side-by-side with flow control. This allows you to determine which node is the most impacted or attributing to the performance issues that are suffering within your Galera Cluster. It is very important as you can check if there can be any degradation in terms of hardware such as your network device or the underlying storage device for which syncing of dirty pages can take too much time to be done.

Cluster Load

Well, this is more of the database activity of how much changes or data retrieval have been queried or done so far since the server's uptime. It helps you rule out what kind of queries are mostly affecting your database cluster performance. This allows you to provide room for improvement especially on balancing the load of your database requests. Using ProxySQL helps you out here with a more refined and granular approach for query routing. Although MaxScale also offers this feature, ProxySQL has more granularity although it also serves some performance impact or cost as well. Impact comes when you only have one ProxySQL as the SQL proxy to work out the query routing and it can struggle when high traffic is on-going. Having cost, if you add more ProxySQL nodes to balance more of the traffic which an underlying KeepAlived. Although, this is a perfect combo but it can be run at a low-cost until needed. However, how will you be able to determine if needed, right? That's the question remains here, so a keen eye to monitor these key areas is very important, not only for observability, but also for improvement of the performance of your database cluster as time goes by.

As such, there are tons of variables to look upon in a MariaDB Cluster. The most important thing here you have to take into account is the tool you are using for monitoring your database cluster. As mentioned earlier, I prefer using the free version license of ClusterControl (Community Edition) here in this blog as it provides me more ways with flexibility to look upon in a Galera Cluster. See the example below,

I have marked or circled in red those tabs that allow me to visually oversee the health of my MariaDB Cluster. Let's say, if your application is greedy over using streaming replication from time to time and it sends a large number of fragments (large network transfer) for cluster inter-activity, it's best to determine how well your nodes can handle the stress. Especially during stress testing before pushing specific changes in your application, it's always best to try and test to determine the capacity management of your application product and determine if your current database nodes and design can handle the load of your application requirements.

Even on a community edition of the ClusterControl, I am able to gather granular and more refined results of the health of my MariaDB Cluster. See below,

This is how you shall approach the monitoring of your MariaDB Cluster. A perfect visualization is always easier and quicker to manage. When things go south, you cannot afford to lose your productivity and also the downtime can impact your business. Although having free does not provide you the luxury and comfort when managing high traffic databases, having alarms, notifications, and database management in one area is a-walk-in-the-park add-ons that ClusterControl can do.

Conclusion

MariaDB Cluster is not as simple to monitor as compared with the traditional asynchronous MySQL/MariaDB master-slave setups. It works differently and you must have the right tools to determine what's going on and what's going into your database cluster. Always prepare your capacity planning ahead before running your MariaDB Cluster without proper monitoring beforehand. It's always best that your database load and activity is known prior to a catastrophic event.

Tags:

database monitoring

monitoring

performance monitoring

↧

Comparing MariaDB Server to MariaDB Cluster

July 23, 2020, 11:35 am

≫ Next: Logical Replication Partitioning With PostgreSQL 13

≪ Previous: Tips for Monitoring MariaDB Cluster

MariaDB Server and MariaDB Cluster are open source products powered by the MariaDB Corporation. MariaDB Server is one of the most popular relational databases, it was originally forked from MySQL server.

MariaDB Cluster is a high availability solution built from MariaDB Server, using a Galera Cluster wsrep library to synchronize the data between nodes. The replication method of Galera is synchronous (or ‘virtually synchronous’), ensuring the data will be the same on all the nodes.

MariaDB server can also be made highly available via standard replication. Replication can be asynchronous or semi-synchronous.

So how does the MariaDB server with standard replication differ from MariaDB Cluster with Galera Cluster? In this blog, we will compare those two. We will be using ClusterControl to illustrate some of the differences.

MariaDB Server Architecture

The architecture of MariaDB Server can be a single/standalone instance or master/slave replication as shown in the diagram below.

The MariaDB Server single instance architecture stands for one node only. The drawback of having a single instance is a single point of failure for the database. If your database crashes and does not come back up, you do not have any failover mechanism, and you need to do a restore to recover your database from the last backup.

The master/slave architecture is a distributed setup, with the master acting as writer and the slave(s) as reader(s). Using a load balancer like Maxscale or ProxySQL, you can split the database traffic so that writes are sent to the master and reads to the slave(s). Having a replication setup will eliminate a single point of failure for the database, but you need to be able to failover automatically if the master fails. Or else, applications will not be able to write to the database and they will be affected. ClusterControl can be configured to provide automatic failover and recovery for MariaDB replication.

MariaDB Cluster Architecture

MariaDB Cluster is a high availability solution consisting of MariaDB Server and Galera Replication as shown in the architecture diagram below :

It is synchronous (“virtually synchronous”) replication, all of the nodes are writable. The synchronous replication guarantees if the changes happen in one of the galera nodes, it will be available on all the other nodes in the cluster before being committed.

The big difference is that all the nodes are equal from the application point of view, they can send write traffic to any of the database instances. Also, all nodes should have exactly the same data so there is no data loss in case of node failure.

MariaDB Deployment

Both MariaDB Replication and MariaDB Cluster can be deployed via ClusterControl. When you deploy MariaDB Server, you need to start by choosing MySQL Replication while for MariaDB Cluster, you need to choose MySQL Galera.

For MariaDB Server, you can either deploy a single node MariaDB instance or you can setup master/slave and bi-directional replication. The minimum number of nodes in a replication setup is two, you need one master and at least one slave.Just fill the IP Address for the master and add slaves (if you want to have master/slave architecture). You can use the Add Second Master field if you want to set up bi-directional replication. A master-master setup will be provisioned with bi-directional replication, but one of the nodes will be set as read-only. The reason is to minimize the risk for data drift and ‘errant transactions’.

For MariaDB Cluster, you need at least 3 hosts for target database nodes to be installed. This is because it has to be able to handle network partitioning or “split brain” syndrome. You just need to fill the ip address when Add Node when defining MySQL Servers configuration.

Do not forget to choose MariaDB as the vendor of database, database version that you want to install and fill the root password. You can also change the non default datadir to any other path.

After we configure all of the things, just deploy the cluster. It will trigger a new job for database deployment.

Note that it is also possible to have 2 Galera nodes and one Galera arbitrator aka garbd on a third host.

MariaDB Server & Cluster Monitoring

Database monitoring is a critical part of the database, you can know the current state of database health. The difference between MariaDB Server and MariaDB Cluster monitoring is Galera Metrics for synchronization.

On MariaDB Server, you can check your current database health through the MySQL Metrics; MySQL Server - General, MySQL Server - Caches, MySQL InnoDB Metrics which is also visible on the MariaDB Cluster as shown in below:

MySQL Server- General gives you information about the current state of InnoDB buffer pool hit ratio, database connection, queries, locking, and database memory utilization.

MySQL Server- Caches, there is a lot of information provided in Caches. Mostly related to the caching in database, eg: buffer pool size, buffer pool instance. There is also information about table cache usage, hit ratio, Cache Hits and Misses. You can also find thread cache usage and hit ratio information .

MySQL Server- InnoDB Metrics shows metrics related to InnoDB storage eg : Bufferpool activity, InnoDB Row operations, InnoDB Log file size, InnoDB Data Read/Write.

On MariaDB Server, if you setup master/slave replication, there is one subcategory of metrics under MySQL Replication - Master. There is information related to master binary log file, master binary log position, and binlog creation frequency.

MariaDB Server has a lot of information related to the database, these are also available for MariaDB Cluster. The difference is there are two dashboards for MariaDB Cluster - Galera Overview and Galera Server Charts.

Galera Overview gives information related to the current state of Galera replication. There is information like cluster size, flow control sent, flow control received, flow control paused.

Galera Server Charts has information about cluster name, cluster status, size, Global cache size.

Conclusion

MariaDB Server with standard replication and MariaDB Cluster are not really different products in terms of database service, but they have different characteristics depending on your requirements on availability and scalability. ClusterControl supports both MariaDB Server with standard replication and MariaDB Cluster deployments, so do give both setups a try and let us know your thoughts.

Tags:

↧

Logical Replication Partitioning With PostgreSQL 13

July 24, 2020, 9:34 am

≫ Next: Configuration Management Software Integration

≪ Previous: Comparing MariaDB Server to MariaDB Cluster

Every PostgreSQL release comes with few major feature enhancements, but what is equally interesting is that every release improves upon its past features as well.

As PostgreSQL 13 is scheduled to be released soon, it’s time to check what features and improvements the community is bringing us. One such no-noise improvement is the “Logical replication improvement for partitioning.”

Let’s understand this feature improvement with a running example.

Terminology

Two terms which are important to understand this feature are:

Partition tables
Logical Replication

Partition Tables

A way to split a large table into multiple physical pieces to achieved benefits like:

Improved query performance
Faster updates
Faster bulk loads and deletes
Organizing seldom used data on slow disk drives

Some of these advantages are achieved by way of partition pruning (i.e. query planner using partition definition to decide whether to scan a partition or not) and the fact that a partition is rather easier to fit in finite memory as compared to a huge table.

A table is partitioned on the basis of:

List
Hash
Range

Logical Replication

As the name implies, this is a replication method in which data is replicated incrementally based on their identity (e.g. key). It is not similar to WAL or physical replication methods where data is sent byte-by-byte.

Based on a Publisher-Subscriber pattern, the source of the data needs to define a publisher while the target needs to be registered as a subscriber. The interesting use cases for this are:

Selective replication (only some part of database)
Simultaneous writes to two instances of database where data is getting replicated
Replication between different operating systems (e.g. Linux and Windows)
Fine grained security on replication of data
Triggers execution as data arrive on the receiver side

Logical Replication for Partitions

With the benefits of both logical replication and partitioning, it is a practical use case to have a scenario where a partitioned table needs to be replicated across two PostgreSQL instances.

Following are the steps to establish and highlight the improvement being done in PostgreSQL 13 in this context.

Setup

Consider a two node setup for running two different instances containing partitioned table:

Steps on Instance_1 are as below post login on 192.168.56.101 as postgres user:

$ initdb -D ${HOME}/pgdata-1

$ echo "listen_addresses = '192.168.56.101'">> ${HOME}/pgdata-1/postgresql.conf

$ echo "wal_level = logical">> ${HOME}/pgdata-1/postgresql.conf

$ echo "host postgres all 192.168.56.102/32 md5">> ${HOME}/pgdata-1/pg_hba.conf

$ pg_ctl -D ${HOME}/pgdata-1 -l logfile start

Setting ‘wal_level’ is set specifically to ‘logical’ to indicate that logical replication will be used to replicate data from this instance. Configuration file ‘pg_hba.conf’ has also been modified to allow connections from 192.168.56.102.

# CREATE TABLE stock_sales

( sale_date date not null, unit_sold  int, unit_price int )

  PARTITION BY RANGE ( sale_date );

# CREATE TABLE stock_sales_y2017 PARTITION OF stock_sales

  FOR VALUES FROM ('2017-01-01') TO ('2018-01-01'); 

# CREATE TABLE stock_sales_y2018 PARTITION OF stock_sales

  FOR VALUES FROM ('2018-01-01') TO ('2019-01-01');

# CREATE TABLE stock_sales_default

  PARTITION OF stock_sales DEFAULT;

Although the postgres role is created by default on the Instance_1 database, a separate user should also be created who has limited access – which restricts the scope only for a given table.

# CREATE ROLE rep_usr WITH REPLICATION LOGIN PASSWORD 'rep_pwd';

# GRANT CONNECT ON DATABASE postgres TO rep_usr;

# GRANT USAGE ON SCHEMA public TO rep_usr;

# GRANT SELECT ON ALL TABLES IN SCHEMA public to rep_usr;

Almost similar setup is required on Instance_2

$ initdb -D ${HOME}/pgdata-2

$ echo "listen_addresses = '192.168.56.102'">> ${HOME}/pgdata-2/postgresql.conf

$ pg_ctl -D ${HOME}/pgdata-2 -l logfile start

It should be noted that since Instance_2 will not be a source of data for any other node, wal_level settings as well as the pg_hba.conf file doesn’t need any extra settings. Needless to say, pg_hba.conf may need updating as per production needs.

Logical Replication doesn’t support DDL, we need to create a table structure on Instance_2 as well. Create a partitioned table using the partition creation above to create the same table structure on Instance_2 as well.

Logical Replication Setup

Logical replication setup becomes much easier with PostgreSQL 13. Up till PostgreSQL 12 the structure was like below:

With PostgreSQL 13, publication of partitions becomes much easier. Refer to the diagram below and compare with previous diagram:

With setups raging with 100’s and 1000’s of partitioned tables – this small change simplifies things to a large extent.

In PostgreSQL 13, the statements to create such a publication will be:

CREATE PUBLICATION rep_part_pub FOR TABLE stock_sales 

WITH (publish_via_partition_root);

Configuration parameter publish_via_partition_root is new in PostgreSQL 13, which allows the recipient node to have a slightly different leaf hierarchy. Mere publication creation on partitioned tables in PostgreSQL 12, will return error statements like below:

ERROR:  "stock_sales" is a partitioned table

DETAIL:  Adding partitioned tables to publications is not supported.

HINT:  You can add the table partitions individually.

Ignoring the limitations of PostgreSQL 12, and proceeding with our hands on of this feature on PostgreSQL 13, we have to establish subscriber on Instance_2 with following statements:

CREATE SUBSCRIPTION rep_part_sub CONNECTION 'host=192.168.56.101 port=5432 user=rep_usr password=rep_pwd dbname=postgres' PUBLICATION rep_part_pub;

Checking if it Really Works

We are pretty much done with the whole setup, but let’s run a couple of tests to see if things are working.

On the Instance_1, insert multiple rows ensuring that they spawn into multiple partitions:

# INSERT INTO stock_sales (sale_date, unit_sold, unit_price) VALUES ('2017-09-20', 12, 151);# INSERT INTO stock_sales (sale_date, unit_sold, unit_price) VALUES ('2018-07-01', 22, 176);

# INSERT INTO stock_sales (sale_date, unit_sold, unit_price) VALUES ('2016-02-02', 10, 1721);

Check the data on Instance_2:

# SELECT * from stock_sales; 

sale_date  | unit_sold | unit_price

------------+-----------+------------

 2017-09-20 |    12 |    151

 2018-07-01 |    22 |    176

 2016-02-02 |    10 |   1721

Now let’s check if logical replication works even if the leaf nodes are not the same on the recipient side.

Add another partition on Instance_1 and insert record:

# CREATE TABLE stock_sales_y2019

      PARTITION OF stock_sales 

     FOR VALUES FROM ('2019-01-01') to ('2020-01-01');

# INSERT INTO stock_sales VALUES(‘2019-06-01’, 73, 174 );

Check the data on Instance_2:

# SELECT * from stock_sales;

 sale_date  | unit_sold | unit_price

------------+-----------+------------

 2017-09-20 |    12 |    151

 2018-07-01 |    22 |    176

 2016-02-02 |    10 |   1721

 2019-06-01 |    73 |   174

Conclusion

I will definitely check the aforementioned two upcoming features in my next set of blogs. Till then food for thought – with the combined power of partitioning and logical replication, is PostgreSQL sailing closer towards a master-master setup?

Tags:

↧

Configuration Management Software Integration

July 24, 2020, 11:16 am

≫ Next: Using ClusterControl with Multiple Cloud Providers (Multi-Cloud)

≪ Previous: Logical Replication Partitioning With PostgreSQL 13

Any configuration management software such as Puppet, Chef, SaltStack, Ansible or Terraform can be used with ClusterControl to simplify automatic deployments of managed databases across hundreds of servers or VMs.

Delegate the responsibility for database provisioning, management, monitoring and the database lifecycle to ClusterControl which has decades of battled-tested know-how of full-ops database management from 1000s of our users and customers.

ClusterControl Integration with Configuration Management Software

Operations Team and End Users

In larger organizations, there are usually one or more operations teams who are responsible to ensure that their internal users (customers) are happy by delivering services reliably and in a timely fashion.

The operations team manages and monitors deployed databases using ClusterControl’s web application or using the Command Line tool. Our application provides in-depth insight into the performance and health of the database clusters which is invaluable for administrators.

End-users however usually will not need to be exposed to ClusterControl’s management application at all and can do with a much simpler custom self-service web portal. This portal sits on top of a configuration management service to trigger automated provisioning of infrastructure and managed databases.

A custom portal is relatively straightforward and easily implemented with a simple frontend and a backend service (Go or Python) which for example runs Ansible playbooks with ClusterControl’s Command Line Tool or make calls directly on the RPC API.

Related Resources

↧

Using ClusterControl with Multiple Cloud Providers (Multi-Cloud)

July 24, 2020, 11:25 am

≫ Next: Hybrid Cloud Database Deployments (Hybrid Clouds)

≪ Previous: Configuration Management Software Integration

Multi-cloud is the use of multiple cloud providers whether it’s deploying into multiple cloud infrastructures or using differentiated services from more than one provider.

There are several reasons for the rapid adoption of multi-cloud:

Vendor lock-in
- One key driver by going multi-cloud is to avoid becoming locked in with a cloud provider’s services and pricing model. A multi-cloud strategy from the start ensures an easier migration path from one provider to another if the architecture has been designed with portability in mind.
- It also gives an opportunity to develop cloud applications to meet different technical and business requirements in order to optimize costs, take advantage of differentiated services or in response to differing service levels between the cloud providers.
Geography
- Different providers have different strengths and weaknesses in a particular region either in terms of costs, services offered, support or latency. This is leveraged to improve cost effectiveness and service performance by deploying closer to end users.
- A service performance is highly dependent on where data is retrieved or stored. Optimal latency and user experience is best achieved when data is served from a cloud provider closest to end users.
Governance
- As we have seen with GDPR, new government regulations can have a profound impact on the way that data privacy and locality needs to be managed. Many public sector organizations for example want their data stored locally in their region. In some countries this is a legal requirement and in others, it’s a preferred option.
Resilience
- Multi-cloud is also a strategy for disaster recovery. Using multiple providers can help organizations ensure operations and core business functions are not severely impacted by a local disaster or unplanned incidents for a provider which can take critical services offline.
- Another risk to operate out of one cloud provider is what happens if a provider goes bankrupt. A few years back the cloud storage company Nirvanix went out of business and its customers had just a short time to retrieve all their data and find another place to store it.

Multi-Cloud Open Source Database Deployments Architecture

Going multi-cloud is usually easier said than done. Applications are made to run in containers that present few difficulties moving or running in different clouds however the database is a critical and key component of stateful applications and for the most part, still runs on dedicated hardware or virtual machines. Moving a database to a new environment is not a trivial task in addition to having small maintenance windows where a limited amount of downtime is allowed or perhaps not at all.

Disaster recovery solutions with multi-cloud database deployments are usually homegrown and maintained with custom scripts and manual effort to recover in time to meet the organization’s stated recovery time objective (RTO).

ClusterControl provides a number of features which facilitates recovery scenarios with “full-ops” database management in diverse environments such as multi and hybrid clouds:

Disaster Recovery
- Active-Active clusters (Asynchronous Replication)
  - MySQL Galera
  - MySQL NDB Cluster
Active-Active clusters (Semi-synchronous Replication)
- MySQL Galera
Active-Standby (or Master-Slave) clusters
- MySQL Galera
- MySQL Replication
- PostgreSQL
Workload management
- Highly available database aware load balancing with read-write routing
- Cluster provisioning
  - Streaming backup from Active cluster
  - Previous backups
- Node provisioning
  - Streaming backup from master node
  - Previous backups
Enterprise-grade backup and restore management
- Full, incremental and differential backups
- Physical and logical backups
- Encryption and compression
- Point In Time Recovery
- Offsite shipping to cloud storage with S3 API compliant cloud storage providers

Related Resources

↧

Hybrid Cloud Database Deployments (Hybrid Clouds)

July 24, 2020, 11:31 am

≫ Next: Database Disaster Recovery Using Cluster-to-Cluster Replication

≪ Previous: Using ClusterControl with Multiple Cloud Providers (Multi-Cloud)

One of the biggest draws of the hybrid cloud model is the ability to act fast (“agility”) and adapt to business changes whether it’s a need to scale out capacity when it is not timely available in your local data centers or move non-critical applications to the cloud which is more cost-effective in the long term.

Hybrid Cloud Open Source Database Deployments

Another benefit is providing business continuity which is one of the integral elements of an organization’s operations. Business continuity is all about continuing to do business during a failure or even a disaster and a continuity plan needs to be carefully planned, by taking into consideration to resilience, recovery, and contingency. Business critical databases can be deployed in a hybrid cloud setup, so that there is no data loss, even in the case of a disaster or emergency at one data center; this minimizes downtime and the costs related with the recovery of essential data.

ClusterControl can be used to provision and manage highly available database clusters across WANs with configurations optimized for increased latency and split-brain incidents with “full-ops” operational support for rapid scaling, cloning or restoring clusters with automated fault detection and recovery.

Related Resources

↧

Database Disaster Recovery Using Cluster-to-Cluster Replication

July 24, 2020, 11:39 am

≫ Next: MariaDB Server Database Encryption Basics

≪ Previous: Hybrid Cloud Database Deployments (Hybrid Clouds)

Disaster recovery in general relies on two key factors which are

Backups (and recovery)
- Backing up and being able to recover an organization’s data / databases is the most important responsibility an (IT) operations team has.
Disaster Tolerance
- Business critical applications and databases are usually deployed on two or more different locations whether it is in on-premise data centers or in public clouds. A switchover is then done upon failures.

A third often overlooked factor is consistently verifying that backups are actually restorable. There are several public cases where backups have been turned out to be corrupt and un-restorable when disaster actually strikes.

Cluster to Cluster Database Replication for Disaster Recovery

ClusterControl provides a complete enterprise grade backup management including verification. Backups can be taken as full, incremental or partial with offsite shipping to S3 compliant cloud storage providers. Offsite backups can be used to restore to a secondary site from scratch however it will take longer time (RTO) to provision and get the application/database back online. This is the most cost effective solution for applications that are not business critical.

In cases where Recovery Time Objectives - RTO is critical, database resilience is achieved by replicating from the primary database cluster in one location to a secondary database cluster in another location.

ClusterControl supports the following resilience options out of the box:

Active - Active clusters (Asynchronous Replication)
- MySQL Galera
- MySQL NDB Cluster
Active - Active clusters (Semi-synchronous Replication)
- MySQL Galera
Active - Standby (or Master - Slave) clusters
- MySQL Galera
- MySQL Replication
- PostgreSQL

Cluster-to-Cluster replication can also be used in many other use cases such as:

Migrating to a new datacenter
Geo-location performance - keep data closer to a set of users for that region
Replicating production to a dev/test environment for troubleshooting
Replicate and test a newer version of the database before doing a switchover
Use a secondary cluster for read-only access for reports or analytics

Hands-on PostgreSQL Cluster-to-Cluster Replication Demonstration

Related Resources

↧

MariaDB Server Database Encryption Basics

July 27, 2020, 8:40 am

≫ Next: Tips for Monitoring MariaDB Replication with ClusterControl

≪ Previous: Database Disaster Recovery Using Cluster-to-Cluster Replication

Encryption is one of the most important security features to keep your data as secure as possible. Depending on the data that you are handling, not always it is a must but you should at least consider it as a security improvement in your organization anyway, and it is actually recommended to avoid data theft or unauthorized access.

In this blog, we will describe two basic types of encryption and how to configure it on a MariaDB Server.

What is Data Encryption?

There are two basic types of data encryption: at-rest and in-transit. Let’s see what they mean.

Data-at-Rest Encryption

Data stored in a system is known as data-at-rest. The encryption of this data consists of using an algorithm to convert text or code to unreadable. You must have an encryption key to decode the encrypted data.

Encrypting an entire database should be done with caution since it can result in a serious performance impact. It is therefore wise to encrypt only individual fields or tables.

Encrypting data-at-rest protects the data from physical theft of hard drives or unauthorized file storage access. This encryption also complies with data security regulations, especially if there is financial or health data stored on the filesystem.

Data-in-Transit Encryption

Data transferred or moving around between transactions is known as data-in-transit. The data moving between the server and client while browsing web pages is a good example of this kind of data.

Since it is always on the move, it needs to be protected with proper encryption to avoid any theft or alteration to the data before it reaches its destination.

The ideal situation to protect data-in-transit is to have the data encrypted before it moves and is only decrypted when it reaches the final destination.

MariaDB Data-at-Rest Encryption

The encryption of tables and tablespaces was added in MariaDB from 10.1 version, and it supports encryption for XtraDB, InnoDB, and Aria storage engines, and also for binary logs.

You can choose different ways to encrypt:

All tables
Individual tables
Everything, excluding individual tables

According to the documentation, using encryption has an overhead of roughly 3-5%, so it is important to have a test environment to stress it and see how it responds, to avoid issues in production.

How to Configure Data-at-Rest Encryption on MariaDB

Let’s check an existing “city” table in a MariaDB database:

$ strings city.ibd |head

infimum

supremum

infimum

supremum

3ABW

3KHM

infimum

supremum

Kabul                              AFGKabol

Qandahar                           AFGQandahar

As you can see, you can read data from there without any issue using the strings Linux command for example. Now, let’s see how to encrypt it.

Generate an encryption keys using the openssl rand command:

$ mkdir -p /etc/mysql/encryption

$ for i in {1..4}; do openssl rand -hex 32 >> /etc/mysql/encryption/keyfile;  done;

Edit the generated file /etc/mysql/encryption/keyfile and add the key IDs which will be referenced when creating encrypted tables. The format should be as follows:

<encryption_key_id1>;<hex-encoded_encryption_key1>

<encryption_key_id2>;<hex-encoded_encryption_key2>

You can edit it using the sed linux command in this way:

$ for i in {1..4}; do sed -i -e "$i s/^/$i;/" keyfile; done

So the file should be something like this:

$ cat /etc/mysql/encryption/keyfile

1;f237fe72e16206c0b0f6f43c3b3f4accc242564d77f5fe17bb621de388c193af

2;0c0819a10fb366a5ea657a71759ee6a950ae8f25a5ba7400a91f59b63683edc5

3;ac9ea3a839596dbf52492d9ab6b180bf11a35f44995b2ed752c370d920a10169

4;72afc936e16a8df05cf994c7902e588de0d11ca7301f9715d00930aa7d5ff8ab

Now, generate a random password using the similar openssl command that you saw earlier:

$ openssl rand -hex 128 > /etc/mysql/encryption/keyfile.key

Before proceeding to the next step, it is important to know the following details about encrypting the key file:

The only algorithm that MariaDB currently supports to encrypt the key file is Cipher Block Chaining (CBC) mode of Advanced Encryption Standard (AES).
The encryption key size can be 128-bits, 192-bits, or 256-bits.
The encryption key is created from the SHA-1 hash of the encryption password.
The encryption password has a max length of 256 characters.

Now, to encrypt the key file using the openssl enc command, run the following command:

$ openssl enc -aes-256-cbc -md sha1 -pass file:/etc/mysql/encryption/keyfile.key -in /etc/mysql/encryption/keyfile -out /etc/mysql/encryption/keyfile.enc

Finally, you need to add the following parameters in your my.cnf configuration file (located in /etc/ on RedHat-based OS or /etc/mysql/ on Debian-Based OS):

[mysqld]

…

#################### DATABASE ENCRYPTION ####################

plugin_load_add = file_key_management

file_key_management_filename = /etc/mysql/encryption/keyfile.enc

file_key_management_filekey = FILE:/etc/mysql/encryption/keyfile.key

file_key_management_encryption_algorithm = aes_cbc

encrypt_binlog = 1



innodb_encrypt_tables = ON

innodb_encrypt_log = ON

innodb_encryption_threads = 4

innodb_encryption_rotate_key_age = 0 

…

And restart the MariaDB service to take the changes:

$ systemctl restart mariadb

At this point, everything is ready to use the encryption feature. Let’s encrypt the same table that we showed earlier, “city”. For this, you need to use the ALTER TABLE statement setting the ENCRYPTED parameter in YES:

MariaDB [world]> ALTER TABLE city ENCRYPTED=YES;

Query OK, 0 rows affected (0.483 sec)

Records: 0  Duplicates: 0  Warnings: 0

Now, if you try to access the table directly from the file system, you will see something like this:

$ strings city.ibd |head

PU%O

!ybN)b

9,{9WB4

T3uG:

?oiN

,35sz

8g)Q

o(o

q_A1

k=-w

As you can see, the table is unreadable. You can also specify the Encryption Key ID by adding the ENCRYPTION_KEY_ID = <ID> parameter in the MySQL command, where <ID> is the ID number from the keyfile created previously.

New tables will be encrypted by default as we set the innodb_encrypt_tables parameter in ON in the my.cnf configuration file.

MariaDB Data-in-Transit Encryption

MariaDB allows you to encrypt data-in-transit between the server and clients using the Transport Layer Security protocol (TLS), formerly known as Secure Socket Layer or SSL.

First of all, you need to ensure that your MariaDB server was compiled with TLS support. You can verify this by running the following SHOW GLOBAL VARIABLES statement:

MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE 'version_ssl_library';

+---------------------+----------------------------+

| Variable_name       | Value                      |

+---------------------+----------------------------+

| version_ssl_library | OpenSSL 1.1.1  11 Sep 2018 |

+---------------------+----------------------------+

1 row in set (0.001 sec)

And check if it is not currently in use using the SHOW VARIABLES statement:

MariaDB [(none)]> SHOW VARIABLES LIKE '%ssl%';

+---------------------+----------------------------+

| Variable_name       | Value                      |

+---------------------+----------------------------+

| have_openssl        | YES                        |

| have_ssl            | DISABLED                   |

| ssl_ca              |                            |

| ssl_capath          |                            |

| ssl_cert            |                            |

| ssl_cipher          |                            |

| ssl_crl             |                            |

| ssl_crlpath         |                            |

| ssl_key             |                            |

| version_ssl_library | OpenSSL 1.1.1  11 Sep 2018 |

+---------------------+----------------------------+

10 rows in set (0.001 sec)

You can also verify the SSL status using the status MariaDB command:

MariaDB [(none)]> status

--------------

mysql  Ver 15.1 Distrib 10.4.13-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

Connection id: 22

Current database:

Current user: root@localhost

SSL: Not in use

Current pager: stdout

Using outfile: ''

Using delimiter: ;

Server: MariaDB

Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic-log mariadb.org binary distribution

Protocol version: 10

Connection: Localhost via UNIX socket

Server characterset: latin1

Db     characterset: latin1

Client characterset: utf8

Conn.  characterset: utf8

UNIX socket: /var/lib/mysql/mysql.sock

Uptime: 4 hours 28 min 25 sec

Threads: 11  Questions: 111668  Slow queries: 0  Opens: 92  Flush tables: 1  Open tables: 85  Queries per second avg: 6.933

--------------

How to Configure Data-in-Transit Encryption on MariaDB

Let’s create the certs directory to store all the certificates:

$ mkdir -p /etc/mysql/certs

Now, let’s generate the CA certificates that will be configured to encrypt the connection:

$ openssl genrsa 2048 > ca-key.pem

$ openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem

This last command will ask you to complete the following information:

Country Name (2 letter code) [AU]:

State or Province Name (full name) [Some-State]:

Locality Name (eg, city) []:

Organization Name (eg, company) [Internet Widgits Pty Ltd]:

Organizational Unit Name (eg, section) []:

Common Name (e.g. server FQDN or YOUR name) []:

Email Address []:

Now, you need to generate the server certificates:

$ openssl req -newkey rsa:2048 -nodes -keyout server-key.pem -out server-req.pem

This command will ask you to fill the same information that before plus an optional certificate password.

$ openssl rsa -in server-key.pem -out server-key.pem

$ openssl x509 -req -in server-req.pem -days 365000 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem

And finally, you need to generate the client certificates:

$ openssl req -newkey rsa:2048 -nodes -keyout client-key.pem -out client-req.pem

This will also ask you to complete the information and an optional certificate password.

$ openssl rsa -in client-key.pem -out client-key.pem

$ openssl x509 -req -in client-req.pem -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem

Make sure you’re using a different Common Name on each certificate, otherwise it won’t work and you will receive a message like:

ERROR 2026 (HY000): SSL connection error: self signed certificate

At this time, you will have something like this:

$ ls /etc/mysql/certs/

ca-cert.pem  ca-key.pem  client-cert.pem  client-key.pem  client-req.pem  server-cert.pem  server-key.pem  server-req.pem

And you can validate the certificates using the following command:

$ openssl verify -CAfile ca-cert.pem server-cert.pem client-cert.pem

server-cert.pem: OK

client-cert.pem: OK

So now let’s configure it in the my.cnf configuration file (located in /etc/ on RedHat-based OS or /etc/mysql/ on Debian-Based OS):

[mysqld]

ssl_ca=/etc/mysql/certs/ca-cert.pem

ssl_cert=/etc/mysql/certs/server-cert.pem

ssl_key=/etc/mysql/certs/server-key.pem



[client-mariadb]

ssl_ca =/etc/mysql/certs/ca-cert.pem

ssl_cert=/etc/mysql/certs/client-cert.pem

ssl_key=/etc/mysql/certs/client-key.pem

Make sure you are adding it under the corresponding section (mysqld and client-mariadb).

Change the certificate’s owner and restart the database service:

$ chown mysql.mysql /etc/mysql/certs/

$ systemctl restart mariadb

After this, if you take a look at the SHOW VARIABLES output, you should have this:

MariaDB [(none)]> SHOW VARIABLES LIKE '%ssl%';

+---------------------+----------------------------------+

| Variable_name       | Value                            |

+---------------------+----------------------------------+

| have_openssl        | YES                              |

| have_ssl            | YES                              |

| ssl_ca              | /etc/mysql/certs/ca-cert.pem     |

| ssl_capath          |                                  |

| ssl_cert            | /etc/mysql/certs/server-cert.pem |

| ssl_cipher          |                                  |

| ssl_crl             |                                  |

| ssl_crlpath         |                                  |

| ssl_key             | /etc/mysql/certs/server-key.pem  |

| version_ssl_library | OpenSSL 1.1.1  11 Sep 2018       |

+---------------------+----------------------------------+

10 rows in set (0.001 sec)

Now, let’s create an user with the REQUIRE SSL parameter to use it:

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 's9s'@'%' IDENTIFIED BY 'root123' REQUIRE SSL;

Query OK, 0 rows affected (0.005 sec)

If you use this user to access the database, and check the status command, you will see the SSL in use:

MariaDB [(none)]> status

--------------

mysql  Ver 15.1 Distrib 10.4.13-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

Connection id: 15

Current database:

Current user: s9s@127.0.0.1

SSL: Cipher in use is TLS_AES_256_GCM_SHA384

Current pager: stdout

Using outfile: ''

Using delimiter: ;

Server: MariaDB

Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic-log mariadb.org binary distribution

Protocol version: 10

Connection: 127.0.0.1 via TCP/IP

Server characterset: latin1

Db     characterset: latin1

Client characterset: utf8

Conn.  characterset: utf8

TCP port: 3306

Uptime: 16 sec

Threads: 11  Questions: 136  Slow queries: 0  Opens: 17  Flush tables: 1  Open tables: 11  Queries per second avg: 8.500

--------------

How to Enable SSL Encryption with ClusterControl

Another way, and even an easier way, to enable SSL on your MariaDB database is by using ClusterControl. We will assume you have ClusterControl installed and you are managing your MariaDB database using it, so go to ClusterControl -> Select your MariaDB Cluster -> Security -> SSL Encryption -> Enable.

MariaDB Server Database Encryption Basics

And that’s it, you will have your SSL encryption enabled in your MariaDB database without any manual task.

At-Rest Encryption Limitations in MariaDB

There are some limitations related to MariaDB at-rest encryption to take into account:

Metadata (for example .frm files) and data sent to the client are not encrypted.
Only the MariaDB server knows how to decrypt the data, in particular
- mysqlbinlog can read encrypted binary logs only when --read-from-remote-server is used.
- Percona XtraBackup can’t back up instances that use encrypted InnoDB. However, Mariabackup can backup encrypted instances.
The disk-based Galera gcache is not encrypted in the community version of MariaDB Server, however, this file is encrypted in MariaDB Enterprise Server 10.4.
The Audit plugin can’t create encrypted output. Send it to syslog and configure the protection there instead.
File-based general query log and slow query log can’t be encrypted.
The Aria log is not encrypted. This affects only non-temporary Aria tables.
The MariaDB error log is not encrypted. The error log can contain query text and data in some cases, including crashes, assertion failures, and cases where InnoDB/XtraDB write monitor output to the log to aid in debugging. It can be sent to syslog too if needed.

Conclusion

Protecting data-in-transit is as important as protecting data-at-rest, and even if it is not a must in your organization, you should consider applying it as it can help you to avoid data theft or unauthorized access.

MariaDB has a pretty easy way to implement it following the steps mentioned earlier, but it is even easier using ClusterControl for sure.

Tags:

encryption in-transit

encryption

database security

↧

Tips for Monitoring MariaDB Replication with ClusterControl

July 28, 2020, 11:01 am

≫ Next: An Overview of MariaDB Xpand (formerly ClustrixDB)

≪ Previous: MariaDB Server Database Encryption Basics

MariaDB replication is one of the most popular high availability solutions for MariaDB and widely used by top companies like Booking.com and Google. It is very easy to set up, with some trade-offs on the ongoing maintenance like software upgrades, schema changes, topology changes, failover and recovery which have always been tricky. Nevertheless, with the right toolset, you should be able to handle the topology with ease. In this blog post, we are going to look into some tips to monitor MariaDB replication efficiently using ClusterControl.

Using the Topology Viewer

A replication setup consists of a number of roles. A node in a replication setup could be a:

Master - The primary writer/reader.
Backup master - A read-only slave with semi-sync replication, solely for master redundancy.
Intermediate master - Replicate from a master, while other slaves replicate from this node.
Binlog server - Only collect/store binlogs without serving data.
Slave - Replicate from a master, and commonly set as read-only.
Multi-source slave - Replicate from multiple masters.

Every role has its own responsibility and limitation and one must understand the correct topology when dealing with the database nodes. This is also true for the application as well, where the application has to write only to the master node at any given time. Thus, it's important to have an overview on which node is holding which role, so we don't screw up our database.

In ClusterControl, the Topology Viewer can give you an overview of the replication topology and its state, as shown in the following screenshot:

ClusterControl understands MariaDB replication and is able to visualize the topology with the correct replication data flow, as represented by the arrows pointed to the slave nodes. We can easily distinguish which node is the master, slaves and load balancers (MaxScale) in our replication setup. The green box indicates all the important services are running as expected with the assigned role.

Consider the following screenshot where a number of our nodes are having problems:

MariaDB Replication Topology Viewer

ClusterControl will immediately tell you what is wrong with the current topology. One of the slaves (red box) is showing "Slave IO Running" as No, to indicate some connectivity issue to replicate from the master. While the yellow box shows our MaxScale service is not running. We can also tell the MaxScale versions are not identical for both nodes. You can also perform management tasks by clicking on the gear icon (top right on every box) directly which reduces the risks of picking up a wrong node.

Replication Lag

This is the most important thing if you rely on data replication consistency. Replication lag occurs when the slaves cannot keep up with the updates happening on the master. Unapplied changes accumulate in the slaves' relay logs and the version of the database on the slaves becomes increasingly different from the master.

In ClusterControl, you can find the replication lag histogram under Overview -> Replication Lag where ClusterControl constantly samples the Seconds_Behind_Master value from "SHOW SLAVE STATUS" output:

Replication lag happens when either the I/O Thread or SQL Thread cannot cope with the demands placed upon it. If the I/O Thread is suffering, this means that the network connection between the master and its slaves is slow or having problems. You might want to consider enabling the slave_compressed_protocol to compress network traffic or report to your network administrator.

If it's the SQL thread then the problem is probably due to poorly-optimized queries that are taking the slave too long to apply. There may be long-running transactions or too much I/O activity. Having no primary key on the slave tables when using the ROW or MIXED replication format is also a common cause of lag on this thread. Check that the master and slave versions of tables have a primary key.

Some more tips and tricks are covered in this blog post, How to Reduce Replication Lag in Multi-Cloud Deployments.

Binary/Relay Log Size

It's important to monitor the binary and relay logs disk size because it could consume a considerable amount of storage on every node in a replication cluster. Commonly, one would set the expire_logs_days system variable to expire binary log files automatically after a given number of days, for example, expire_logs_days=7. The size of binary logs is totally dependent on the number of binary events created (incoming writes) and little that we know how much disk space it would consume before the logs are going to be expired by MariaDB. Keep in mind if you enable log_slave_updates on the slaves, the size of logs will be almost doubled because of the existence of both binary and relay logs on the same server.

For ClusterControl, we can set a disk space utilization threshold under ClusterControl -> Settings -> Thresholds to get a warning and critical notifications as below:

ClusterControl monitors all disk space related to MariaDB services like the location of MariaDB data directory, the binary logs directory and also the root partition. If you have reached the threshold, consider purging the binary logs manually by using the PURGE BINARY LOGS command, as explained and discussed in this article.

Enable Monitoring Dashboards

ClusterControl provides two monitoring options to sample the database nodes - agentless or agent-based. The default is agentless where sampling happens via SSH in a pull-only mechanism. Agent-based monitoring requires a Prometheus server to be running, and all monitored nodes to be configured with at least three exporters:

Process exporter (port 9011)
Node/system metrics exporter (port 9100)
MySQL/MariaDB exporter (port 9104)

To enable the agent-based monitoring dashboard, one has to go to ClusterControl -> Dashboards -> Enable Agent Based Monitoring. Once enabled, you will see a set of dashboards configured for our MariaDB replication which gives us a much better insight on our replication setup. The following screenshot shows what you would see for the master node:

Apart from MariaDB standard monitoring dashboards like general, caches and InnoDB metrics, you will be presented with a replication dashboard. For the master node, we can get a lot of useful information regarding the state of the master, the write throughput and binlog creation frequency.

While for the slaves, all the important states are sampled and summarized as the following screenshot. if everything is green, you are in good hands:

Understanding the MariaDB Error Log

MariaDB logs its important events inside the error log, which is useful to understand what was going on with the server, especially before, during and after a topology change. ClusterControl provides a centralized view of error logs under ClusterControl -> Logs -> System Logs by pulling them from every database node. You click on "Refresh Logs" to trigger a job to pull the latest logs from the server.

Collected files are represented in a navigation tree structure and a text area with syntax highlighting for better readability:

From the above screenshot, we can understand the sequence of events and what happened to this node during a topology change event. From the last 12 lines of the error log above, the slave had an error once connecting to the master and the last binary log file and position were recorded in the log before it stopped. Then a newer CHANGE MASTER command was executed with GTID information, as shown in the line "Previous Using_Gtid=No. New Using_Gtid=Slave_Pos" and then the replication resumes as what we wanted.

MariaDB Alert and Notifications

Monitoring is incomplete without alerts and notifications. All events and alarms generated by ClusterControl can be sent to the email or any other supported third-party tools. For email notifications, one can configure whether the type of events will be delivered immediately, ignored or digested (a daily summarized report):

For all critical severity events, it's recommended to set everything to "Deliver" so you will get the notifications as soon as possible. Set "Digest" to warning events so you are well aware of the cluster health and state.

You can integrate your preferred communication and messaging tools with ClusterControl by using the Notifications Management feature under ClusterControl -> Integrations -> 3rd Party Notifications. ClusterControl can send alarms and events to PagerDuty, VictorOps, OpsGenie, Slack, Telegram, ServiceNow or any user registered webhooks.

The following screenshot shows all critical events will be pushed to the configured telegram channel for our MariaDB 10.3 Replication cluster:

ClusterControl also supports chatbot integration, where you can interact with the controller service via s9s client right from your messaging tool as shown in this blog post, Automate Your Database with CCBot: ClusterControl Hubot Integration.

Conclusion

ClusterControl offers a complete set of proactive monitoring tools for your database clusters. Do use ClusterControl to monitor your MariaDB replication setup because most of the monitoring features are available for free in the community edition. Don't miss those out!

Tags: