Struct S3StreamingUploader

pub struct S3StreamingUploader {
    client: Client,
    part_size: usize,
    bucket: String,
    key: String,
    upload_id: Option<String>,
    next_part_id: i32,
    join_handles: Vec<JoinHandle<ObjectResult<(i32, UploadPartOutput)>>>,
    buf: Vec<Bytes>,
    not_uploaded_len: usize,
    metrics: Arc<ObjectStoreMetrics>,
    config: Arc<ObjectStoreConfig>,
}

S3 multipart upload handle. The multipart upload is not initiated until the first part is available for upload.

Reference: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

Fields

client: Client

part_size: usize

bucket: String

key: String

The key of the object.

upload_id: Option<String>

The identifier of multipart upload task for S3.

next_part_id: i32

Next part ID.

join_handles: Vec<JoinHandle<ObjectResult<(i32, UploadPartOutput)>>>

Join handles for part uploads.

buf: Vec<Bytes>

Buffer for data. It will store at least part_size bytes of data before wrapping itself into a stream and upload to object store as a part.

not_uploaded_len: usize

Length of the data that have not been uploaded to S3.

metrics: Arc<ObjectStoreMetrics>

To record metrics for uploading part.

config: Arc<ObjectStoreConfig>

Implementations

impl S3StreamingUploader

const MEDIA_TYPE: &'static str = "s3"

pub fn new( client: Client, bucket: String, key: String, metrics: Arc<ObjectStoreMetrics>, config: Arc<ObjectStoreConfig>, ) -> S3StreamingUploader

async fn upload_next_part(&mut self) -> ObjectResult<()>

async fn flush_multipart_and_complete(&mut self) -> ObjectResult<()>

async fn abort_multipart_upload(&self) -> ObjectResult<()>

Trait Implementations

impl StreamingUploader for S3StreamingUploader

async fn finish(self) -> ObjectResult<()>

If the multipart upload has not been initiated, we can use PutObject instead to save the CreateMultipartUpload and CompleteMultipartUpload requests. Otherwise flush the remaining data of the buffer to S3 as a new part.

async fn write_bytes(&mut self, data: Bytes) -> ObjectResult<()>

fn get_memory_usage(&self) -> u64