Struct LocalHummockStorage

pub struct LocalHummockStorage {

    mem_table: MemTable,
    spill_offset: u16,
    epoch: Option<u64>,
    table_id: TableId,
    op_consistency_level: OpConsistencyLevel,
    table_option: TableOption,
    instance_guard: LocalInstanceGuard,
    read_version: HummockReadVersionRef,
    is_replicated: bool,
    event_sender: HummockEventSender,
    memory_limiter: Arc<MemoryLimiter>,
    hummock_version_reader: HummockVersionReader,
    stats: Arc<HummockStateStoreMetrics>,
    write_limiter: WriteLimiterRef,
    version_update_notifier_tx: Arc<Sender<PinnedVersion>>,
    mem_table_spill_threshold: usize,
}

LocalHummockStorage is a handle for a state table shard to access data from and write data to the hummock state backend. It is created via HummockStorage::new_local.

Fields

mem_table: MemTable

spill_offset: u16

epoch: Option<u64>

table_id: TableId

op_consistency_level: OpConsistencyLevel

table_option: TableOption

instance_guard: LocalInstanceGuard

read_version: HummockReadVersionRef

Read handle.

is_replicated: bool

This indicates that this LocalHummockStorage replicates another LocalHummockStorage. It’s used by executors in different CNs to synchronize states.

Within LocalHummockStorage we use this flag to avoid uploading local state to be persisted, so we won’t have duplicate data.

This also handles a corner case where an executor doing replication is scheduled to the same CN as its Upstream executor. In that case, we use this flag to avoid reading the same data twice, by ignoring the replicated ReadVersion.

event_sender: HummockEventSender

Event sender.

memory_limiter: Arc<MemoryLimiter>

hummock_version_reader: HummockVersionReader

stats: Arc<HummockStateStoreMetrics>

write_limiter: WriteLimiterRef

version_update_notifier_tx: Arc<Sender<PinnedVersion>>

mem_table_spill_threshold: usize

Implementations

impl LocalHummockStorage

pub fn update(&self, info: VersionUpdate)

See HummockReadVersion::update for more details.

pub async fn get_inner( &self, table_key: TableKey<Bytes>, epoch: u64, read_options: ReadOptions, ) -> StorageResult<Option<StateStoreKeyedRow>>

pub async fn wait_for_epoch(&self, wait_epoch: u64) -> StorageResult<()>

pub async fn iter_flushed( &self, table_key_range: TableKeyRange, epoch: u64, read_options: ReadOptions, ) -> StorageResult<HummockStorageIterator>

pub async fn rev_iter_flushed( &self, table_key_range: TableKeyRange, epoch: u64, read_options: ReadOptions, ) -> StorageResult<HummockStorageRevIterator>

fn mem_table_iter(&self) -> MemTableHummockIterator<'_>

fn mem_table_rev_iter(&self) -> MemTableHummockRevIterator<'_>

pub async fn iter_all( &self, table_key_range: TableKeyRange, epoch: u64, read_options: ReadOptions, ) -> StorageResult<LocalHummockStorageIterator<'_>>

pub async fn rev_iter_all( &self, table_key_range: TableKeyRange, epoch: u64, read_options: ReadOptions, ) -> StorageResult<LocalHummockStorageRevIterator<'_>>

impl LocalHummockStorage

async fn flush_inner( &mut self, sorted_items: Vec<(TableKey<Bytes>, SharedBufferValue<Bytes>)>, old_values: Option<Vec<Bytes>>, write_options: WriteOptions, ) -> StorageResult<usize>

impl LocalHummockStorage

pub fn new( instance_guard: LocalInstanceGuard, read_version: HummockReadVersionRef, hummock_version_reader: HummockVersionReader, event_sender: HummockEventSender, memory_limiter: Arc<MemoryLimiter>, write_limiter: WriteLimiterRef, option: NewLocalOptions, version_update_notifier_tx: Arc<Sender<PinnedVersion>>, mem_table_spill_threshold: usize, ) -> Self

pub fn read_version(&self) -> HummockReadVersionRef

See HummockReadVersion::update for more details.

pub fn table_id(&self) -> TableId

pub fn instance_id(&self) -> u64

fn is_flush_old_value(&self) -> bool

Trait Implementations

impl LocalStateStore for LocalHummockStorage

type Iter<'a> = HummockStorageIteratorInner<'a>

type RevIter<'a> = HummockStorageRevIteratorInner<'a>

async fn get( &self, key: TableKey<Bytes>, read_options: ReadOptions, ) -> StorageResult<Option<Bytes>>

Point gets a value from the state store. The result is based on the latest written snapshot.

async fn iter( &self, key_range: TableKeyRange, read_options: ReadOptions, ) -> StorageResult<Self::Iter<'_>>

Opens and returns an iterator for given prefix_hint and full_key_range Internally, prefix_hint will be used to for checking bloom_filter and full_key_range used for iter. (if the prefix_hint not None, it should be be included in key_range) The returned iterator will iterate data based on the latest written snapshot.

async fn rev_iter( &self, key_range: TableKeyRange, read_options: ReadOptions, ) -> StorageResult<Self::RevIter<'_>>

fn get_table_watermark(&self, vnode: VirtualNode) -> Option<Bytes>

Get last persisted watermark for a given vnode.

fn insert( &mut self, key: TableKey<Bytes>, new_val: Bytes, old_val: Option<Bytes>, ) -> StorageResult<()>

Inserts a key-value entry associated with a given epoch into the state store.

fn delete(&mut self, key: TableKey<Bytes>, old_val: Bytes) -> StorageResult<()>

Deletes a key-value entry from the state store. Only the key-value entry with epoch smaller than the given epoch will be deleted.

async fn flush(&mut self) -> StorageResult<usize>

async fn try_flush(&mut self) -> StorageResult<()>

fn epoch(&self) -> u64

fn is_dirty(&self) -> bool

async fn init(&mut self, options: InitOptions) -> StorageResult<()>

Initializes the state store with given epoch pair. Typically we will use epoch.curr as the initialized epoch, Since state table will begin as empty. In some cases like replicated state table, state table may not be empty initially, as such we need to wait for epoch.prev checkpoint to complete, hence this interface is made async.

fn seal_current_epoch(&mut self, next_epoch: u64, opts: SealCurrentEpochOptions)

Updates the monotonically increasing write epoch to new_epoch. All writes after this function is called will be tagged with new_epoch. In other words, the previous write epoch is sealed.

fn update_vnode_bitmap(&mut self, vnodes: Arc<Bitmap>) -> Arc<Bitmap>

impl StateStoreRead for LocalHummockStorage

type ChangeLogIter = ChangeLogIterator

type Iter = HummockStorageIteratorInner<'static>

type RevIter = HummockStorageRevIteratorInner<'static>

fn get_keyed_row( &self, key: TableKey<Bytes>, epoch: u64, read_options: ReadOptions, ) -> impl Future<Output = StorageResult<Option<StateStoreKeyedRow>>> + Send + '_

Point gets a value from the state store. The result is based on a snapshot corresponding to the given epoch. Both full key and the value are returned.

fn iter( &self, key_range: TableKeyRange, epoch: u64, read_options: ReadOptions, ) -> impl Future<Output = StorageResult<Self::Iter>> + '_

Opens and returns an iterator for given prefix_hint and full_key_range Internally, prefix_hint will be used to for checking bloom_filter and full_key_range used for iter. (if the prefix_hint not None, it should be be included in key_range) The returned iterator will iterate data based on a snapshot corresponding to the given epoch.

fn rev_iter( &self, key_range: TableKeyRange, epoch: u64, read_options: ReadOptions, ) -> impl Future<Output = StorageResult<Self::RevIter>> + '_

async fn iter_log( &self, epoch_range: (u64, u64), key_range: TableKeyRange, options: ReadLogOptions, ) -> StorageResult<Self::ChangeLogIter>

fn get( &self, key: TableKey<Bytes>, epoch: u64, read_options: ReadOptions, ) -> impl Future<Output = StorageResult<Option<Bytes>>> + Send + '_

Point gets a value from the state store. The result is based on a snapshot corresponding to the given epoch. Only the value is returned.