Redis Sentinel Reset

Sentinel role

目前 Sentinel 系统是 Redis 的 unstable 分支的一部分， 你必须到 Redis 项目的 Github 页面 克隆一份 unstable 分值， 然后通过编译来获得 Sentinel 系统。. Sentinel 程序可以在编译后的 src 文档中发现， 它是一个命名为 redis-sentinel 的程序。. 你也可以通过下一节介绍的方法， 让 redis-server 程序. Redis Sentinel is a distributed system, this means that usually you want to run multiple Sentinel processes across your infrastructure, and this processes will use gossip protocols in order to understand if a master is down and agreement protocols in order to get authorized to perform the failover and assign a new version to the new.

# Host and port we will listen for requests on bind 127.0.0.1 port 16380 # # 'redis-cluster' is the name of our cluster # # each sentinel process is paired with a redis-server process # sentinel monitor redis-cluster 127.0.0.1 6380 2 sentinel down-after-milliseconds redis-cluster 5000 sentinel parallel-syncs redis-cluster 1 sentinel failover.

Sentinel is a distributed system, which guarantees the high availability of programs. It is used to monitor any number of master servers and all slave servers under these master servers. In case of failure, it selects a new master through voting mechanism and connects all slaves to the new master.

monitor

Continuously check whether the master and slave are running normally, master survivability detection and master and slave operation detection.

notice

When there is a problem with the monitored server, send a notification to other (sentinel, client).

Automatic fail over

Disconnect the master from slave, select a slave as the master, connect other slave to the new master, and inform the client of the new server address.

be careful

Sentry is also a redis server, but it does not provide data service. Usually, the number of sentries is odd

Activate the sentry

configuration file

Sentinel default profilesentinel.conf

General withsentinel_port.confName the Sentinel’s profile

configuration information

start-up

Configure master-slave structure, taking 1master 2 slave as an example.

1 start master and slave first

Please refer to the master-slave blog for master-slave configurationMaster and slave

2. Activate the sentry

Sentinel command

PING：PONG
Sentinel masters: lists all monitored master servers and their current status.
Sentinel Slaves: lists all slave servers of a given master server and their current status.
Sentinel get master addr by name: returns the IP address and port number of the master server with the given name. If the primary server is performing a fail over operation, or the fail over operation for the primary server has been completed, this command returns the IP address and port number of the new primary server.
Sentinel reset: resets all primary servers whose names match the given pattern. The pattern parameter is a global style pattern. The reset operation makes clear all the current states of the master server, including the failure transition in progress, and removes all the discovered and associated slave servers and sentinel of the master server.
Sentinel fail over: when the primary server fails, it is forced to start an automatic fail over without asking other sentinels for advice (however, the sentinel that initiated the fail over will send a new configuration to other sentinels, and other sentinels will update it accordingly).

Initialize Sentinel

initialize server

As can be seen from the following startup code, the startup mode is determined by the functioncheckForSentinelModeTo decide whether to use sentinelMode of a start, add the command is also usedsentinelcmdsCommand list for

From the source code, we can see that there are two ways to start sentry

Either way, it startsredisIt will be implementedinitServerConfigThe difference isSentinelIt will also be implementedinitSentinelConfig、initSentinelTwo initialization functions. Let’s see what these two functions do.

Replace Sentinel’s private code

initSentinelConfig()This function will useSentinelThe configured properties override the server’s default properties.

initSentinel()A command table will be loaded. A main query commandINFOIt is also different from ordinary server, but uses a special version.

Initialize sentinel state

After the command table is loaded, it is immediately followed bysentinelStateandsentinelRedisInstance An initialization of the structure.

SentinelIn statemastersThe dictionary records all the monitored master server information. The key is the name of the server and the value is the corresponding value of the monitored master serversentinel.c/sentinelRedisInstanceStructure. eachsentinelRedisInstanceThe instance structure represents a monitorRedisServer instance, which can be a master server, a slave server, or another sentinel server.

aboutsentinelStateThe initialization of themastersThe initialization of the master dictionary is based on the inputsentinelConfiguration file（sentinel_26379.conf）To do it. It is mainly monitoredmasterOfipand port。

be carefulThese are maintained and used by sentinel.

sentinelState

sentinelRedisInstance

name
The name of the instance
The name of the master server is set by the user in the configuration file
The name of the slave server and sentinel is automatically set by sentinel
The format is IP: port, for example, “127.0.0.1:26379”

runid
Run ID of the instance

sentinelAddr
The address of the instance

Properties specific to the primary server instance

sentinels
Others also monitor all sentinels of this master server

slaves
If this instance represents a master server
Then this dictionary holds the slave server under the master server
The key of the dictionary is the name of the slave server, and the value of the dictionary is the sentinelredisinstance structure corresponding to the slave server

quorum
The number of support votes needed to judge this instance as objective down

parallel_syncs
SENTINEL parallel-syncs The value of the option
The number of slave servers that can synchronize the new master server at the same time when performing a fail over operation

auth_pass
The password required to connect the master and slave servers

From server instance specific properties

master_link_down_time
The time when the master-slave server was disconnected

slave_priority
Slave priority

slave_reconf_sent_time
When performing a fail over operation, slaveof is sent from the serverTime of command

Redis Sentinel Reset Command

master
Instance of master server (used when this instance is a slave server)

slave_master_host
The master server IP recorded in the reply to the info command

slave_master_port
The main server port number recorded in the reply to the info command

slave_master_link_status
The master-slave server connection status recorded in the reply to the info command

slave_repl_offset
Copy offset from server

In structuresentinelAddrIt holds the address and port of the object.

Make a connection

Sentinel will connect firstsentinelin mastersEvery one of themmaster, and in everymasterandsentinelCreate two betweenAsynchronous connectionOneCommand connectionOneSubscribed Links . At this time, sentinel will become the client of the master. It can send commands to the master server and get relevant information from the command reply.

Command connection

It is specially used to send commands to the master server and receive command replies. For example, sentinel sendsINFOOrders.

Subscription connection

Dedicated to subscriber master_sentinel_:helloChannel. such assentinelTo the Lord, to the slave, to the othersentinelsend outsentinelAnd the main database information.

In the publish and subscribe function of redis, the information sent will not be saved in the redis server. If the client that needs to receive the message is not online or disconnected when it arrives, the client will lose the information. In order not to lose_sentinel_:helloAny information about the channel,sentinelYou must use a dedicated subscription connection to receive information from that channel.

Get master server information

SentinelBy default, every10Second frequency to the main serverINFOCommand to get the current information of the master server by analyzing the command reply. Sentinel can obtain the following two aspects of information:

1 information about the master server itself, including the server run_ ID, the server role of the role.

2 corresponding to the main serverAll slave serversFrom the server IP and port.

Get information from server

WhenSentinelWhen a new slave server appears,SentinelIn addition, the corresponding instance structure will be created for the new slave server（sentinelRedisInstance）In addition, aCommand connectionandSubscription connection。

SentinelIt will still send one every 10 seconds, just like the main serverINFOCommand to get the current information from the server.

run_ id、role、ip、port 、master_ link_ Status, slave_ Priority (priority of the slave server).

Send information to master and slave servers

By default,SentinelIt will send commands in the following format to all monitored master servers and slave servers through command connection once every 2 seconds:

This command is sent to the_sentinel_:helloThe channel sends a message. The content of the message consists of several parameters

(1) s_The first parameter recordssentinelIt’s a message of its own.

(2) m_The first parameter records the information of the master server. If sentinel is monitoring the master server, then these parameters are the information of the master server. If sentinel is monitoring the slave server, then these parameter records are the information of the master server being copied by the slave server.

for example

Receive channel information from master and slave servers

WhenSentinelAfter establishing a subscription connection with a master or slave server,SentinelIt will passSubscription connectionSend to serversubscribe_sentinel_:hello 。

For each and everySentinelConnected server,SentinelConnect to the server by command_sentinel_:helloChannel sends information, and then through the subscription connection from the server’s_sentinel_:helloThe channel receives information.

So when there’s a new oneSentinelWhen you connect in, send thesubscribe_sentinel_:helloBy existingSentinelReceive (I will also receive this message from myself).

Be aSentinelfrom_sentinel_:helloWhen the channel receives a message,SentinelI will analyze this information and extract IP, port and run in the information_ ID and other 8 parameters, and check the following: if this message is sent by itself, it will be ignored directly. If it’s newSentinelIn this case, sentinel will update the structure of the corresponding primary server instance, that is, the newly addedSentinelAdd tosentinelsIn the dictionary.

eachSentinelEach has its own onesentinelsDictionaries,SentinelsDictionary information holds all information except itselfSentinelInformation.

Offline status

There are two different concepts about offline in sentinel of redis: (1) subjectively down (sdown for short) refers to the offline judgment made by a single sentinel instance to the server, and there will be no failover at this time. (2) Objective down (owdown) means that when multiple sentinel instances make sdown judgment on the same server, the target sentinel will fail over the primary server. This article introduces in detail.

Subjective offline state

defaultSentinelWill be sent once per second to all instances (including master, slave, other sentinel) with which it creates a command connectionPINGCommand, and judge whether the instance is online by instance reply.

Legal response

+pong 、 -loading、 -masterdown。

Invalid reply

In addition, all replies or no replies are considered invalid. No reply means no reply within a specified time.

The value of the down after milliseconds option set by the user is not only used by sentinel to determine the subjective offline status of the master server, but also used to determine the subjective offline status of all slave servers under the master server and other sentinel monitoring the master server.

theremasterIs the name of the primary server, and the port is the default6379 ，2representativesentinelThere are many problems in the cluster2individualsentinelthinkmaster Only when the status is offline, can we really think that the master is not available (that isObjective offline)。

It’s not just going to besentineljudgemasterEnter the subjective offline standard, but also to judge allSlave Library, otherssentinelThe standard of entering subjective offline.

When multiple sentinels are set, the subjective offline duration may be different

For multiplesentinelWhen monitoring the same primary server together, thesesentinelIn the configuration filesentinle.confSet indown-after-millisecondsValues can also be different, so when asentinelWhen the main server is judged as subjective offline, the othersentinelYou may still think that the primary server is online. Only all of themsentineOnly when they have entered the subjective offline state, can they be regarded as the main channelmasterIn the subjective offline state.

Objective offline status

WhenSentinelAfter a master server is judged to be offline subjectively, in order to confirm whether the master server is offline or not, it will monitor other servers of the master serverSentinelWhen there are more than half (depending on the specific configuration, generally more than half), for examplesentinel monitor mymaster 127.0.0.1 6379 2In the middle2When a judge is offline, it is considered that he is offline objectively.)

WhenmasterAfter being determined to be offlinesentinelWe’ll pick onedecision makerTo perform a fail over operation. Objective offline conditionsPrimary server only。

is-master-down-by-addrThe command is used to determine whether to go offline objectively

sentinelThe current era of configurationcurrent_epochFor electiondecision maker sentinel, run_idIt can be * or the running ID of sentinel.

Redis Sentinel And Cluster

Decision maker selection

Suppose there are foursentinelThese foursentinelBoth voters and candidates (these four must be healthy).

1 cannot have one of the following three Tags: Sri_ S_ DOWN SRI_ O_ DOWN SRI_ DISCONNECTED

The heart rate is normal

3 priority cannot be 0 (slave > slave)_ priority）

4. Info data cannot time out

5. The disconnection time of master-slave connection cannot be timeout

The process of voting is very simple, every one of themsentinelAll will be their ownip、 port、current_epoch、run_idfromis-master-downSend tohelloChannel.

sentinelWho was the first to get itis-master-downInformation, vote for the correspondingsentinel。

After onecurrent_epochThe largest, and more than half. Is selected asdecision makerOtherwise, another round, every additional roundcurrent_epoch + 1Until the election.

Fail over

Select candidate slave

1 Online

2 fast response

3 the shortest time to disconnect from the original master

4. Priority principle

Priority > offset > runid

Finally, a new one is selectedmasterAnd then to the newmaster send out

Then declare the newmaster

Redis Sentinel Tutorial

Finally, the original master is used as the slave. When you go back online,sentinelWill sendsalveofThe command makes it a slave.

summary

Redis Sentinel Configuration

• sentinelIt’s just a computer running in a special moderedisServer, which uses a different command table from the normal mode, as well as different commands from the normal mode.

• sentinelSend to primary serverINFOCommand to obtain the address information of all the slave servers under the master server, and create the corresponding instance structure for these slave servers, as well as the command connection and subscription connection to these slave servers.

• Normally,sentinelTo the monitored master and slave servers every 10 secondsINFOCommand, when the primary server is offline, orsentinelWhile the primary server is being failed over,sentinelSend to slave serverINFOThe frequency of the command will be changed to once a second.

• For multiple servers that monitor the same master and slave serversentinelIn other words, they will send messages to the monitored_sentinel_:helloChannel sends messages to other channelssentinelDeclare your own existence.

• eachsentinelI’ll get it from you, too_sentinel_:helloReceive others in medium channelsentinelSend information, and based on this information for other userssentinelCreate the corresponding instance structure and command connection.

• sentinelOnly command and subscription connections are created with master and slave servers,sentinelAndsentinelOnly command connections are created between.

• sentinelSend data to the instance (including master, slave and others) once per secondsentinel）SendPINGCommand, and judge whether the instance is online according to the reply of the instancesentinelWhen an invalid reply is sent,sentinelThis instance will be judged as subjective offline.

• WhensentinelWhen a master server is judged to be subjectively offline, it will monitor other users of the master server to the same serversentinelAsk them if they agree that the master server has entered the subjective offline state.

• WhensentinelAfter collecting enough subjective offline votes, it will judge the primary server as objective offline and initiate a failover operation against the primary server.