Section 5.5. Observing File System Top End Activity

5.5. Observing File System "Top End" Activity

Applications typically access their data from a file system through the POSIX I/O library and system calls. These accesses are passed into the kernel and into the underlying file system through the VOP layer (see Section 5.1).

Using DTrace function boundary probes, we can trace the VOP layer and monitor file system activity. Probes fired at the entry and exit of each VOP method can record event counts, latency, and physical I/O counts. We can obtain information about the methods by casting the arguments of the VOP methods to the appropriate structures; for example, we can harvest the file name, file system name, I/O size, and the like from these entry points.

The DTrace vopstat command instruments and reports on the VOP layer activity. By default, it summarizes each VOP in the system and reports a physical I/O count, a VOP method count, and the total latency incurred for each VOP during the sample period. This utility provides a useful first-pass method of understanding where and to what degree latency is occurring in the file system layer.

The following example shows vopstat output for a system running ZFS. In this example, the majority of the latency is being incurred in the VOP_FSYNC method (see Table 14.3 in Solaris^™ Internals).

# ./vopstat VOP Physical IO                                              Count fop_fsync                                                      236 VOP Count                                                    Count fop_create                                                       1 fop_fid                                                          1 fop_lookup                                                       2 fop_access                                                       3 fop_read                                                         3 fop_poll                                                        11 fop_fsync                                                       31 fop_putpage                                                     32 fop_ioctl                                                      115 fop_write                                                      517 fop_rwlock                                                     520 fop_rwunlock                                                   520 fop_inactive                                                   529 fop_getattr                                                   1057 VOP Wall Time                                             mSeconds fop_fid                                                          0 fop_access                                                       0 fop_read                                                         0 fop_poll                                                         0 fop_lookup                                                       0 fop_create                                                       0 fop_ioctl                                                        0 fop_putpage                                                      1 fop_rwunlock                                                     1 fop_rwlock                                                       1 fop_inactive                                                     1 fop_getattr                                                      2 fop_write                                                       22 fop_fsync                                                      504