13.6 Tokens and Cache Coherency


13.6 Tokens and Cache Coherency

So the CFS Server says to a CFS client, "Here is a token of my appreciation!" But seriously, the CFS Server tends to throw in a token as a free gift when sending I/O to a CFS client. Tokens are a lightweight synchronization method used by the CFS server to guarantee that every cluster member has the same view of a file – this is known as cache coherency.

There are two token modes:

  • READ

– Shared. Can be held by one or more members at a time.

  • READ/WRITE

– Exclusive. Can be held by one member at a time.

There are three token types:

  • MHOLD

– Present when a file's vnode[2] is cached on a CFS client, but does not imply that the member is actually caching any data.

The CFS server uses the MHOLD token to track where files are cached in order to notify those CFS clients when certain events occur such as the file becoming unlinked.

The MHOLD token is never specifically requested or revoked.

  • ATTR

– Protects the file's cached attributes and data. Exists when a CFS client is caching file data or attributes (file size, time of last modification, etc.)

  • DIOMAP

– Protects the AdvFS extent map. Used for Direct I/O and Direct Access Cached Reads to cache the AdvFS extent map for the file. The extent map for a file indicates where the data resides on the volumes.

[2]A vnode (or virtual node) is the in-memory representation of the file's serial number. In UFS, the serial number is called an inode. In AdvFS, the serial number is known as a tag or bfNode (bit file node).

A token is requested by a CFS client anytime a read, write, get attribute, or set attribute operation needs to be performed. The token is held until the operation completes.

A token is granted by the CFS server as follows:

  • If a CFS client requests a token for a file and no other CFS client has a token for that file, the CFS server will grant the token.

  • If a CFS client requests a token mode of READ/WRITE for a file and another CFS client has a token in READ/WRITE or READ mode for that file, the CFS server will send a revoke request to the CFS client currently holding the token. Once the token is returned, the CFS server will grant the token to the requesting CFS client.

  • If a CFS client requests a token mode of READ for a file and another CFS client has a token in READ/WRITE mode for that file, the CFS server will send a revoke request to the CFS client currently holding the token. Once the client downgrades the token to a mode of READ or returns the token, the CFS server will grant the token to the requesting CFS client.

  • If a CFS client requests a token mode of READ for a file and another CFS client currently has a token in READ mode, the CFS server will grant the token to the requesting CFS client.

For example, members 1-3 in a four-node cluster have the file MyDataFile opened for read. They all hold a token with the READ mode set. The fourth member requests a token to write to the file opened by the other three members. The CFS server will revoke the tokens held by members 1-3, and once the tokens are returned, the CFS server will then grant the token for MyDataFile in READ/WRITE mode to member4. When the tokens are revoked, the information in the members' caches for MyDataFile is invalidated. Once the write operation is completed, member4 will release or downgrade the token to READ mode, and then the remaining members can once again be granted a token in READ mode.

A token is currently granted on a per file basis, although greater granularity is being investigated.

IMPORTANT

Do not confuse tokens with file locking. Tokens do not synchronize access to a file to guarantee ordered I/O as file locking does. If you want your applications to synchronize access to files, you have two choices:

  • Cluster Unaware (POSIX – X/OPEN) – fcntl(2), lockf(3), flock(2)[3]

  • Cluster Aware – Distributed Lock Manager (DLM)[4]Application Programming Interface (API)

Note that the term "cluster unaware" is used to indicate that this method will not make your application "aware" that it is in a cluster. In other words, it is not an API unique to the TruCluster Server. In fact, due to support for the POSIX and X/OPEN APIs in the CFS, an application written to use these standard APIs for file locking on a standalone system can move the application to the cluster and it will provide cluster-wide file locking. No special modifications are required, which is pretty cool indeed!

The job of the token is to guarantee that the members' caches do not contain different views of the file at any particular time.

Here is a simplified example (note, we do not indicate token type, only token mode). In this example, Member1 is node sheridan and Member2 is node molari.

Token Mode:

Member1

Member2

On member1, open a new file with your favorite
editor. Write the file to disk but do not exit the editor.

READ/WRITE

No Token.

 [sheridan] # vi MyDataFile i La Dee Da Yadda yadda yadda  :w 

Token Mode:

Member1

Member2

On member2, type out the file.

Token downgraded
to READ.
Cache flushed.

READ

 [molari] # cat MyDataFile La Dee Da Yadda yadda yadda 

Both members have an identical view of the file.

Token Mode:

Member1

Member2

On member2, append some information to the file.

Token revoked.
Cache flushed &
invalidated.

READ/WRITE

 [molari] # cat >> MyDataFile toodle loo ^D 

Token Mode:

Member1

Member2

On both members, display the file.

READ

Token downgraded
to READ.
Cache flushed.

 [molari] # cat MyDataFile La Dee Da Yadda yadda yadda toodle loo 
 [sheridan] # cat MyDataFile La Dee Da Yadda yadda yadda toodle loo 

Both members still have an identical view of the file.

Token Mode:

Member1

Member2

On member1, write the file to disk again.
Remember, we're still in the editor.

READ/WRITE

Token revoked.
Cache flushed &
invalidated.

 [sheridan] - still in vi :w 

Token Mode:

Member1

Member2

On member2, type out the file again.

Token downgraded
to READ.
Cache flushed.

READ

 [molari] # cat MyDataFile La Dee Da Yadda yadda yadda 

The behavior shown here using tokens should be familiar to you since this is exactly how a standalone system behaves without tokens. Try this example using two processes on a standalone system. The results will be the same.

To gather additional information on tokens, you can use the cfsstat(8) command. You can obtain general information and statistics for tokens by using the cfsstat command with "tokens" and "tokstats" options respectively. To retrieve specific token information for a particular file by client or server, use "clitok_log" and "svrtok_log" options respectively.

For more information, see the TruCluster Server Cluster Administration Guide and the cfsstat(8) reference page.

[3]flock is technically not part of the POSIX or X/OPEN standard but uses the fcntl function

[4]See chapter 18.




TruCluster Server Handbook
TruCluster Server Handbook (HP Technologies)
ISBN: 1555582591
EAN: 2147483647
Year: 2005
Pages: 273

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net