LevelDB数据存储大数据

4. LevelDB源码剖析之基础部件-操作系统操作

2017-08-13  本文已影响48人  随安居士

LevelDB被设计成跨平台数据库,支持在windows、linux、Android等多个操作系统。除了port文件夹中的AtomicPointer、Mutex等编程基础部件,和操作系统相关的其他操作统统被塞进了Env类中。

4.1 Env类

Env类从功能上划分包括如下几个部分:

4.2 文件相关操作

文件相关操作接口定义如下:

class Env
{
  ......

  //创建顺序读文件 -- 文件同时只能被一个线程访问
  virtual Status NewSequentialFile(const std::string &fname,
                                   SequentialFile **result) = 0;

  //创建随机读文件 -- 文件同时只能被一个线程访问
  virtual Status NewRandomAccessFile(const std::string &fname,
                                     RandomAccessFile **result) = 0;

  //创建写文件 -- 文件同时只能被一个线程访问
  virtual Status NewWritableFile(const std::string &fname,
                                 WritableFile **result) = 0;

  //创建或打开写文件 -- 文件同时只能被一个线程访问
  virtual Status NewAppendableFile(const std::string &fname,
                                   WritableFile **result);

  // Returns true iff the named file exists.
  virtual bool FileExists(const std::string &fname) = 0;

  // Store in *result the names of the children of the specified directory.
  // The names are relative to "dir".
  // Original contents of *results are dropped.
  virtual Status GetChildren(const std::string &dir,
                             std::vector<std::string> *result) = 0;

  // Delete the named file.
  virtual Status DeleteFile(const std::string &fname) = 0;

  // Create the specified directory.
  virtual Status CreateDir(const std::string &dirname) = 0;

  // Delete the specified directory.
  virtual Status DeleteDir(const std::string &dirname) = 0;

  // Store the size of fname in *file_size.
  virtual Status GetFileSize(const std::string &fname, uint64_t *file_size) = 0;

  // Rename file src to target.
  virtual Status RenameFile(const std::string &src,
                            const std::string &target) = 0;

  // Lock the specified file.  Used to prevent concurrent access to
  // the same db by multiple processes.  On failure, stores NULL in
  // *lock and returns non-OK.
  //
  // On success, stores a pointer to the object that represents the
  // acquired lock in *lock and returns OK.  The caller should call
  // UnlockFile(*lock) to release the lock.  If the process exits,
  // the lock will be automatically released.
  //
  // If somebody else already holds the lock, finishes immediately
  // with a failure.  I.e., this call does not wait for existing locks
  // to go away.
  //
  // May create the named file if it does not already exist.
  virtual Status LockFile(const std::string &fname, FileLock **lock) = 0;

  // Release the lock acquired by a previous successful call to LockFile.
  // REQUIRES: lock was returned by a successful LockFile() call
  // REQUIRES: lock has not already been unlocked.
  virtual Status UnlockFile(FileLock *lock) = 0;
  ......
}

文件按操作方式分为顺序读文件(SequentialFile)、随机读文件(RandomAccessFile)及写文件(WritableFile)。
通用的文件操作包括删除文件、获取文件大小等;通用的目录操作包括创建、删除目录,获取目录的子目录或文件列表,逻辑简单此处不提。另外,有一组配对的文件锁操作LockFile、UnlockFile。

4.2.1 SequentialFile

顺序文件和随机文件实现都非常简单,二者最大的区别在于:顺序文件不提供基于offset的读操作接口。

顺序文件每次从当前位置读取指定大小的字节数,另外支持跳过中间部分内容,除此之外再无其他。

// A file abstraction for reading sequentially through a file
class SequentialFile
{
public:
  SequentialFile() {}
  virtual ~SequentialFile();

  // Read up to "n" bytes from the file.  "scratch[0..n-1]" may be
  // written by this routine.  Sets "*result" to the data that was
  // read (including if fewer than "n" bytes were successfully read).
  // May set "*result" to point at data in "scratch[0..n-1]", so
  // "scratch[0..n-1]" must be live when "*result" is used.
  // If an error was encountered, returns a non-OK status.
  //
  // REQUIRES: External synchronization
  virtual Status Read(size_t n, Slice *result, char *scratch) = 0;

  // Skip "n" bytes from the file. This is guaranteed to be no
  // slower that reading the same data, but may be faster.
  //
  // If end of file is reached, skipping will stop at the end of the
  // file, and Skip will return OK.
  //
  // REQUIRES: External synchronization
  virtual Status Skip(uint64_t n) = 0;

private:
  // No copying allowed
  SequentialFile(const SequentialFile &);
  void operator=(const SequentialFile &);
};

4.2.2 RandomAccessFile

随机读文件只有一个基于offset读取指定大小字节数的接口,定义如下:

// A file abstraction for randomly reading the contents of a file.
class RandomAccessFile
{
public:
  RandomAccessFile() {}
  virtual ~RandomAccessFile();

  // Read up to "n" bytes from the file starting at "offset".
  // "scratch[0..n-1]" may be written by this routine.  Sets "*result"
  // to the data that was read (including if fewer than "n" bytes were
  // successfully read).  May set "*result" to point at data in
  // "scratch[0..n-1]", so "scratch[0..n-1]" must be live when
  // "*result" is used.  If an error was encountered, returns a non-OK
  // status.
  //
  // Safe for concurrent use by multiple threads.
  virtual Status Read(uint64_t offset, size_t n, Slice *result,
                      char *scratch) const = 0;

private:
  // No copying allowed
  RandomAccessFile(const RandomAccessFile &);
  void operator=(const RandomAccessFile &);
};

4.2.3 WritableFile

写文件的接口定义相对“复杂”,除了追加操作外,还包括了关闭(close)、刷新(flush)、同步(sync)操作。接口定义如下:

// A file abstraction for sequential writing.  The implementation
// must provide buffering since callers may append small fragments
// at a time to the file.
class WritableFile
{
public:
  WritableFile() {}
  virtual ~WritableFile();

  virtual Status Append(const Slice &data) = 0;
  virtual Status Close() = 0;
  virtual Status Flush() = 0;
  virtual Status Sync() = 0;

private:
  // No copying allowed
  WritableFile(const WritableFile &);
  void operator=(const WritableFile &);
};

Flush和Sync的差异在于:Flush讲数据从用户缓冲区刷新到内核缓冲区,而Sync则会将内核缓冲区数据刷新到磁盘。

4.3 线程相关操作

接口定义如下:

  // Arrange to run "(*function)(arg)" once in a background thread.
  //
  // "function" may run in an unspecified thread.  Multiple functions
  // added to the same Env may run concurrently in different threads.
  // I.e., the caller may not assume that background work items are
  // serialized.
  virtual void Schedule(
      void (*function)(void *arg),
      void *arg) = 0;

  // Start a new thread, invoking "function(arg)" within the new thread.
  // When "function(arg)" returns, the thread will be destroyed.
  virtual void StartThread(void (*function)(void *arg), void *arg) = 0;

  // Sleep/delay the thread for the prescribed number of micro-seconds.
  virtual void SleepForMicroseconds(int micros) = 0;

Schedule提供系统线程池,添加task后交由线程池调度,而无需用户创建线程池或者进行线程管理;StartThread负责创建单独线程。

4.4 其他操作

  // *path is set to a temporary directory that can be used for testing. It may
  // or many not have just been created. The directory may or may not differ
  // between runs of the same process, but subsequent calls will return the
  // same directory.
  virtual Status GetTestDirectory(std::string *path) = 0;

  // Create and return a log file for storing informational messages.
  virtual Status NewLogger(const std::string &fname, Logger **result) = 0;

  // Returns the number of micro-seconds since some fixed point in time. Only
  // useful for computing deltas of time.
  virtual uint64_t NowMicros() = 0;

4.5 总结

本节描述的是LevelDB系统构建基础,本身没有任何技术复杂度。作者设计遵循了单一职责、精简等基本设计原则,将接口做到最简,具有一定的借鉴意义。


转载请注明:【随安居士】http://www.jianshu.com/p/3c449af6fdac

上一篇下一篇

猜你喜欢

热点阅读