Struct SpillOp

Source
pub struct SpillOp {
    pub op: Operator,
}
Expand description

SpillOp is used to manage the spill directory of the spilling executor and it will drop the directory with a RAII style.

Fields§

§op: Operator

Implementations§

Source§

impl SpillOp

Source

pub fn create(path: String, spill_backend: SpillBackend) -> Result<SpillOp>

Source

pub async fn clean_spill_directory() -> Result<()>

Source

pub async fn writer_with(&self, name: &str) -> Result<Writer>

Source

pub async fn reader_with(&self, name: &str) -> Result<Reader>

Source

pub fn read_stream( reader: Reader, spill_metrics: Arc<BatchSpillMetrics>, ) -> Pin<Box<dyn Stream<Item = Result<DataChunk, BatchError>> + Send>>

spill file content will look like the below.

[proto_len]
[proto_bytes]
...
[proto_len]
[proto_bytes]

Methods from Deref<Target = Operator>§

pub fn inner(&self) -> &Arc<dyn AccessDyn>

Fetch the internal accessor.

pub fn limit(&self) -> usize

👎Deprecated since 0.52.0: limit is no-op for now

Get current operator’s limit. Limit is usually the maximum size of data that operator will handle in one operation.

pub fn with_limit(&self, _: usize) -> Operator

👎Deprecated since 0.52.0: limit is no-op for now

Specify the batch limit.

Default: 1000

pub fn default_executor(&self) -> Option<Executor>

👎Deprecated since 0.53.0: use Operator::executor instead

Get the default executor.

pub fn with_default_executor(&self, _: Executor) -> Operator

👎Deprecated since 0.53.0: use Operator::update_executor instead

Specify the default executor.

pub fn info(&self) -> OperatorInfo

Get information of underlying accessor.

§Examples
use opendal::Operator;

let info = op.info();

pub fn executor(&self) -> Executor

Get the executor used by current operator.

pub fn update_executor(&self, f: impl FnOnce(Executor) -> Executor)

Update executor for the context.

All cloned Operator instances share the same internal state, such as HttpClient and Runtime. Some layers may modify the internal state of the Operator too like inject logging and metrics for HttpClient.

§Note

Tasks must be forwarded to the old executor after the update. Otherwise, features such as retry, timeout, and metrics may not function properly.

pub fn http_client(&self) -> HttpClient

Get the http client used by current operator.

pub fn update_http_client(&self, f: impl FnOnce(HttpClient) -> HttpClient)

Update http client for the context.

All cloned Operator instances share the same internal state, such as HttpClient and Runtime. Some layers may modify the internal state of the Operator too like inject logging and metrics for HttpClient.

§Note

Tasks must be forwarded to the old executor after the update. Otherwise, features such as retry, timeout, and metrics may not function properly.

pub fn blocking(&self) -> BlockingOperator

Create a new blocking operator.

This operation is nearly no cost.

pub async fn check(&self) -> Result<(), Error>

Check if this operator can work correctly.

We will send a list request to path and return any errors we met.

use opendal::Operator;

op.check().await?;

pub async fn stat(&self, path: &str) -> Result<Metadata, Error>

Get given path’s metadata.

§Notes
§Extra Options

[Operator::stat] is a wrapper of [Operator::stat_with] without any options. To use extra options like if_match and if_none_match, please use [Operator::stat_with] instead.

§Examples
§Check if file exists
use opendal::ErrorKind;
if let Err(e) = op.stat("test").await {
    if e.kind() == ErrorKind::NotFound {
        println!("file not exist")
    }
}

pub fn stat_with( &self, path: &str, ) -> OperatorFuture<OpStat, Metadata, impl Future<Output = Result<Metadata, Error>>>

Get given path’s metadata with extra options.

§Options
§if_match

Set if_match for this stat request.

This feature can be used to check if the file’s ETag matches the given ETag.

If file exists, and it’s etag doesn’t match, an error with kind [ErrorKind::ConditionNotMatch] will be returned.

use opendal::Operator;

let mut metadata = op.stat_with("path/to/file").if_match(etag).await?;
§if_none_match

Set if_none_match for this stat request.

This feature can be used to check if the file’s ETag doesn’t match the given ETag.

If file exists, and it’s etag match, an error with kind [ErrorKind::ConditionNotMatch] will be returned.

use opendal::Operator;

let mut metadata = op.stat_with("path/to/file").if_none_match(etag).await?;
§if_modified_since

set if_modified_since for this stat request.

This feature can be used to check if the file has been modified since the given time.

If file exists, and it’s not modified after the given time, an error with kind [ErrorKind::ConditionNotMatch] will be returned.

use opendal::Operator;
use chrono::Utc;

let mut metadata = op.stat_with("path/to/file").if_modified_since(Utc::now()).await?;
§if_unmodified_since

set if_unmodified_since for this stat request.

This feature can be used to check if the file has NOT been modified since the given time.

If file exists, and it’s modified after the given time, an error with kind [ErrorKind::ConditionNotMatch] will be returned.

use opendal::Operator;
use chrono::Utc;

let mut metadata = op.stat_with("path/to/file").if_unmodified_since(Utc::now()).await?;
§version

Set version for this stat request.

This feature can be used to retrieve the metadata of a specific version of the given path

If the version doesn’t exist, an error with kind [ErrorKind::NotFound] will be returned.


let mut metadata = op.stat_with("path/to/file").version(version).await?;
§Examples
§Get metadata while ETag matches

stat_with will

  • return Ok(metadata) if ETag matches
  • return Err(error) and error.kind() == ErrorKind::ConditionNotMatch if file exists but ETag mismatch
  • return Err(err) if other errors occur, for example, NotFound.
use opendal::ErrorKind;
if let Err(e) = op.stat_with("test").if_match("<etag>").await {
    if e.kind() == ErrorKind::ConditionNotMatch {
        println!("file exists, but etag mismatch")
    }
    if e.kind() == ErrorKind::NotFound {
        println!("file not exist")
    }
}

§Behavior
§Services that support create_dir

test and test/ may vary in some services such as S3. However, on a local file system, they’re identical. Therefore, the behavior of stat("test") and stat("test/") might differ in certain edge cases. Always use stat("test/") when you need to access a directory if possible.

Here are the behavior list:

CasePathResult
stat existing dirabc/Metadata with dir mode
stat existing fileabc/def_fileMetadata with file mode
stat dir without /abc/def_dirError NotFound or metadata with dir mode
stat file with /abc/def_file/Error NotFound
stat not existing pathxyzError NotFound

Refer to [RFC: List Prefix][crate::docs::rfcs::rfc_3243_list_prefix] for more details.

§Services that not support create_dir

For services that not support create_dir, stat("test/") will return NotFound even when test/abc exists since the service won’t have the concept of dir. There is nothing we can do about this.

pub async fn exists(&self, path: &str) -> Result<bool, Error>

Check if this path exists or not.

§Example
use anyhow::Result;
use futures::io;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let _ = op.exists("test").await?;

    Ok(())
}

pub async fn is_exist(&self, path: &str) -> Result<bool, Error>

👎Deprecated: rename to exists for consistence with std::fs::exists

Check if this path exists or not.

§Example
use anyhow::Result;
use futures::io;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let _ = op.is_exist("test").await?;

    Ok(())
}

pub async fn create_dir(&self, path: &str) -> Result<(), Error>

Create a dir at given path.

§Notes

To indicate that a path is a directory, it is compulsory to include a trailing / in the path. Failure to do so may result in NotADirectory error being returned by OpenDAL.

§Behavior
  • Create on existing dir will succeed.
  • Create dir is always recursive, works like mkdir -p
§Examples
op.create_dir("path/to/dir/").await?;

pub async fn read(&self, path: &str) -> Result<Buffer, Error>

Read the whole path into a bytes.

§Notes
§Extra Options

[Operator::read] is a wrapper of [Operator::read_with] without any options. To use extra options like range and if_match, please use [Operator::read_with] instead.

§Streaming Read

This function will allocate a new bytes internally. For more precise memory control or reading data lazily, please use [Operator::reader]

§Examples
let bs = op.read("path/to/file").await?;

pub fn read_with( &self, path: &str, ) -> OperatorFuture<(OpRead, OpReader), Buffer, impl Future<Output = Result<Buffer, Error>>>

Read the whole path into a bytes with extra options.

This function will allocate a new bytes internally. For more precise memory control or reading data lazily, please use [Operator::reader]

§Notes
§Streaming Read

This function will allocate a new bytes internally. For more precise memory control or reading data lazily, please use [Operator::reader]

§Options

Visit [FutureRead] for all available options.

§Examples

Read the whole path into a bytes.

let bs = op.read_with("path/to/file").await?;
let bs = op.read_with("path/to/file").range(0..10).await?;

pub async fn reader(&self, path: &str) -> Result<Reader, Error>

Create a new reader which can read the whole path.

§Notes
§Extra Options

[Operator::reader] is a wrapper of [Operator::reader_with] without any options. To use extra options like concurrent, please use [Operator::reader_with] instead.

§Examples
let r = op.reader("path/to/file").await?;

pub fn reader_with( &self, path: &str, ) -> OperatorFuture<(OpRead, OpReader), Reader, impl Future<Output = Result<Reader, Error>>>

Create a new reader with extra options

§Notes
§Extra Options

[Operator::reader] is a wrapper of [Operator::reader_with] without any options. To use extra options like version, please use [Operator::reader_with] instead.

§Options

Visit [FutureReader] for all available options.

§Examples
let r = op.reader_with("path/to/file").version("version_id").await?;

pub async fn write( &self, path: &str, bs: impl Into<Buffer>, ) -> Result<Metadata, Error>

Write bytes into path.

§Notes
§Extra Options

[Operator::write] is a simplified version of [Operator::write_with] without additional options. For advanced features like content_type and cache_control, use [Operator::write_with] instead.

§Streaming Write

This method performs a single bulk write operation. For finer-grained memory control or streaming data writes, use [Operator::writer] instead.

§Multipart Uploads

OpenDAL provides multipart upload functionality through the [Writer] abstraction, handling all upload details automatically. You can customize the upload behavior by configuring chunk size and concurrent operations via [Operator::writer_with].

§Examples
use bytes::Bytes;

op.write("path/to/file", vec![0; 4096]).await?;

pub async fn copy(&self, from: &str, to: &str) -> Result<(), Error>

Copy a file from from to to.

§Notes
  • from and to must be a file.
  • to will be overwritten if it exists.
  • If from and to are the same, an IsSameFile error will occur.
  • copy is idempotent. For same from and to input, the result will be the same.
§Examples

op.copy("path/to/file", "path/to/file2").await?;

pub async fn rename(&self, from: &str, to: &str) -> Result<(), Error>

Rename a file from from to to.

§Notes
  • from and to must be a file.
  • to will be overwritten if it exists.
  • If from and to are the same, an IsSameFile error will occur.
§Examples

op.rename("path/to/file", "path/to/file2").await?;

pub async fn writer(&self, path: &str) -> Result<Writer, Error>

Create a writer for streaming data to the given path.

§Notes
§Writer Features

The writer provides several powerful capabilities:

  • Streaming writes for continuous data transfer
  • Automatic multipart upload handling
  • Memory-efficient chunk-based writing
§Extra Options

[Operator::writer] is a simplified version of [Operator::writer_with] without additional options. For advanced features like content_type and cache_control, use [Operator::writer_with] instead.

§Chunk Size Handling

Storage services often have specific requirements for chunk sizes:

  • Services like s3 may return EntityTooSmall errors for undersized chunks
  • Using small chunks in cloud storage services can lead to increased costs

OpenDAL automatically determines optimal chunk sizes based on the service’s Capability. However, you can override this by explicitly setting the chunk parameter.

For improved performance, consider setting an appropriate chunk size using [Operator::writer_with].

§Examples
use bytes::Bytes;

let mut w = op.writer("path/to/file").await?;
w.write(vec![0; 4096]).await?;
w.write(vec![1; 4096]).await?;
w.close().await?;

pub fn writer_with( &self, path: &str, ) -> OperatorFuture<(OpWrite, OpWriter), Writer, impl Future<Output = Result<Writer, Error>>>

Create a writer for streaming data to the given path with more options.

§Options

Visit [FutureWriter] for all available options.

§Examples
use bytes::Bytes;

let mut w = op.writer_with("path/to/file")
    .chunk(4*1024*1024)
    .concurrent(8)
    .await?;
w.write(vec![0; 4096]).await?;
w.write(vec![1; 4096]).await?;
w.close().await?;

pub fn write_with( &self, path: &str, bs: impl Into<Buffer>, ) -> OperatorFuture<(OpWrite, OpWriter, Buffer), Metadata, impl Future<Output = Result<Metadata, Error>>>

Write data with extra options.

§Notes
§Streaming Write

This method performs a single bulk write operation for all bytes. For finer-grained memory control or lazy writing, consider using [Operator::writer_with] instead.

§Multipart Uploads

OpenDAL handles multipart uploads through the [Writer] abstraction, managing all the upload details automatically. You can customize the upload behavior by configuring chunk size and concurrent operations via [Operator::writer_with].

§Options

Visit [FutureWrite] for all available options.

§Examples
use bytes::Bytes;

let _ = op.write_with("path/to/file", vec![0; 4096])
    .if_not_exists(true)
    .await?;

pub async fn delete(&self, path: &str) -> Result<(), Error>

Delete the given path.

§Notes
  • Deleting a file that does not exist won’t return errors.
§Examples
op.delete("test").await?;

pub fn delete_with( &self, path: &str, ) -> OperatorFuture<OpDelete, (), impl Future<Output = Result<(), Error>>>

Delete the given path with extra options.

§Notes
  • Deleting a file that does not exist won’t return errors.
§Options
§version

Set version for this delete request.

remove a specific version of the given path.

If the version doesn’t exist, OpenDAL will not return errors.


op.delete_with("path/to/file").version(version).await?;
§Examples

op.delete_with("test").await?;

pub async fn delete_iter<I, D>(&self, iter: I) -> Result<(), Error>
where I: IntoIterator<Item = D>, D: IntoDeleteInput,

Delete an infallible iterator of paths.

Also see:

  • [Operator::delete_try_iter]: delete a fallible iterator of paths.
  • [Operator::delete_stream]: delete an infallible stream of paths.
  • [Operator::delete_try_stream]: delete a fallible stream of paths.

pub async fn delete_try_iter<I, D>(&self, try_iter: I) -> Result<(), Error>
where I: IntoIterator<Item = Result<D, Error>>, D: IntoDeleteInput,

Delete a fallible iterator of paths.

Also see:

  • [Operator::delete_iter]: delete an infallible iterator of paths.
  • [Operator::delete_stream]: delete an infallible stream of paths.
  • [Operator::delete_try_stream]: delete a fallible stream of paths.

pub async fn delete_stream<S, D>(&self, stream: S) -> Result<(), Error>
where S: Stream<Item = D>, D: IntoDeleteInput,

Delete an infallible stream of paths.

Also see:

  • [Operator::delete_iter]: delete an infallible iterator of paths.
  • [Operator::delete_try_iter]: delete a fallible iterator of paths.
  • [Operator::delete_try_stream]: delete a fallible stream of paths.

pub async fn delete_try_stream<S, D>(&self, try_stream: S) -> Result<(), Error>
where S: Stream<Item = Result<D, Error>>, D: IntoDeleteInput,

Delete a fallible stream of paths.

Also see:

  • [Operator::delete_iter]: delete an infallible iterator of paths.
  • [Operator::delete_try_iter]: delete a fallible iterator of paths.
  • [Operator::delete_stream]: delete an infallible stream of paths.

pub async fn deleter(&self) -> Result<Deleter, Error>

Create a [Deleter] to continuously remove content from storage.

It leverages batch deletion capabilities provided by storage services for efficient removal.

Users can have more control over the deletion process by using [Deleter] directly.

pub async fn remove(&self, paths: Vec<String>) -> Result<(), Error>

👎Deprecated since 0.52.0: use Operator::delete_iter instead
§Notes

If underlying services support delete in batch, we will use batch delete instead.

§Examples
op.remove(vec!["abc".to_string(), "def".to_string()])
    .await?;

pub async fn remove_via( &self, input: impl Stream<Item = String> + Unpin, ) -> Result<(), Error>

👎Deprecated since 0.52.0: use Operator::delete_stream instead

remove will remove files via the given paths.

remove_via will remove files via the given stream.

We will delete by chunks with given batch limit on the stream.

§Notes

If underlying services support delete in batch, we will use batch delete instead.

§Examples
use futures::stream;
let stream = stream::iter(vec!["abc".to_string(), "def".to_string()]);
op.remove_via(stream).await?;

pub async fn remove_all(&self, path: &str) -> Result<(), Error>

Remove the path and all nested dirs and files recursively.

§Notes

If underlying services support delete in batch, we will use batch delete instead.

§Examples
op.remove_all("path/to/dir").await?;

pub async fn list(&self, path: &str) -> Result<Vec<Entry>, Error>

List entries that starts with given path in parent dir.

§Notes
§Recursively List

This function only read the children of the given directory. To read all entries recursively, use Operator::list_with("path").recursive(true) instead.

§Streaming List

This function will read all entries in the given directory. It could take very long time and consume a lot of memory if the directory contains a lot of entries.

In order to avoid this, you can use [Operator::lister] to list entries in a streaming way.

§Examples
§List entries under a dir

This example will list all entries under the dir path/to/dir/.

use opendal::EntryMode;
use opendal::Operator;
let mut entries = op.list("path/to/dir/").await?;
for entry in entries {
    match entry.metadata().mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir {}", entry.path())
        }
        EntryMode::Unknown => continue,
    }
}
§List entries with prefix

This example will list all entries under the prefix path/to/prefix.

NOTE: it’s possible that the prefix itself is also a dir. In this case, you could get path/to/prefix/, path/to/prefix_1 and so on. If you do want to list a dir, please make sure the path is end with /.

use opendal::EntryMode;
use opendal::Operator;
let mut entries = op.list("path/to/prefix").await?;
for entry in entries {
    match entry.metadata().mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir {}", entry.path())
        }
        EntryMode::Unknown => continue,
    }
}

pub fn list_with( &self, path: &str, ) -> OperatorFuture<OpList, Vec<Entry>, impl Future<Output = Result<Vec<Entry>, Error>>>

List entries that starts with given path in parent dir with more options.

§Notes
§Streaming list

This function will read all entries in the given directory. It could take very long time and consume a lot of memory if the directory contains a lot of entries.

In order to avoid this, you can use [Operator::lister] to list entries in a streaming way.

§Options
§start_after

Specify the specified key to start listing from.

This feature can be used to resume a listing from a previous point.

The following example will resume the list operation from the breakpoint.

use opendal::Operator;
let mut entries = op
    .list_with("path/to/dir/")
    .start_after("breakpoint")
    .await?;
§recursive

Specify whether to list recursively or not.

If recursive is set to true, we will list all entries recursively. If not, we’ll only list the entries in the specified dir.

use opendal::Operator;
let mut entries = op.list_with("path/to/dir/").recursive(true).await?;
§version

Specify whether to list files along with all their versions

if version is enabled, all file versions will be returned; otherwise, only the current files will be returned.

let mut entries = op.list_with("path/to/dir/").version(true).await?;
§Examples
§List all entries recursively

This example will list all entries under the dir path/to/dir/

use opendal::EntryMode;
use opendal::Operator;
let mut entries = op.list_with("path/to/dir/").recursive(true).await?;
for entry in entries {
    match entry.metadata().mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        EntryMode::Unknown => continue,
    }
}
§List all entries start with prefix

This example will list all entries starts with prefix path/to/prefix

use opendal::EntryMode;
use opendal::Operator;
let mut entries = op.list_with("path/to/prefix").recursive(true).await?;
for entry in entries {
    match entry.metadata().mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        EntryMode::Unknown => continue,
    }
}

pub async fn lister(&self, path: &str) -> Result<Lister, Error>

List entries that starts with given path in parent dir.

This function will create a new [Lister] to list entries. Users can stop listing via dropping this [Lister].

§Notes
§Recursively list

This function only read the children of the given directory. To read all entries recursively, use [Operator::lister_with] and recursive(true) instead.

§Examples
use futures::TryStreamExt;
use opendal::EntryMode;
use opendal::Operator;
let mut ds = op.lister("path/to/dir/").await?;
while let Some(mut de) = ds.try_next().await? {
    match de.metadata().mode() {
        EntryMode::FILE => {
            println!("Handling file")
        }
        EntryMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        EntryMode::Unknown => continue,
    }
}

pub fn lister_with( &self, path: &str, ) -> OperatorFuture<OpList, Lister, impl Future<Output = Result<Lister, Error>>>

List entries that starts with given path in parent dir with options.

This function will create a new [Lister] to list entries. Users can stop listing via dropping this [Lister].

§Options
§start_after

Specify the specified key to start listing from.

This feature can be used to resume a listing from a previous point.

The following example will resume the list operation from the breakpoint.

use opendal::Operator;
let mut lister = op
    .lister_with("path/to/dir/")
    .start_after("breakpoint")
    .await?;
§recursive

Specify whether to list recursively or not.

If recursive is set to true, we will list all entries recursively. If not, we’ll only list the entries in the specified dir.

use opendal::Operator;
let mut lister = op.lister_with("path/to/dir/").recursive(true).await?;
§version

Specify whether to list files along with all their versions

if version is enabled, all file versions will be returned; otherwise, only the current files will be returned.

let mut entries = op.lister_with("path/to/dir/").version(true).await?;
§Examples
§List all files recursively
use futures::TryStreamExt;
use opendal::EntryMode;
use opendal::Operator;
let mut lister = op.lister_with("path/to/dir/").recursive(true).await?;
while let Some(mut entry) = lister.try_next().await? {
    match entry.metadata().mode() {
        EntryMode::FILE => {
            println!("Handling file {}", entry.path())
        }
        EntryMode::DIR => {
            println!("Handling dir {}", entry.path())
        }
        EntryMode::Unknown => continue,
    }
}

pub async fn presign_stat( &self, path: &str, expire: Duration, ) -> Result<PresignedRequest, Error>

Presign an operation for stat(head).

§Example
use anyhow::Result;
use futures::io;
use opendal::Operator;
use std::time::Duration;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op.presign_stat("test",Duration::from_secs(3600)).await?;
    let req = http::Request::builder()
        .method(signed_req.method())
        .uri(signed_req.uri())
        .body(())?;

pub fn presign_stat_with( &self, path: &str, expire: Duration, ) -> OperatorFuture<(OpStat, Duration), PresignedRequest, impl Future<Output = Result<PresignedRequest, Error>>>

Presign an operation for stat(head).

§Example
use anyhow::Result;
use futures::io;
use opendal::Operator;
use std::time::Duration;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op.presign_stat_with("test",Duration::from_secs(3600)).override_content_disposition("attachment; filename=\"othertext.txt\"").await?;

pub async fn presign_read( &self, path: &str, expire: Duration, ) -> Result<PresignedRequest, Error>

Presign an operation for read.

§Notes
§Extra Options

presign_read is a wrapper of [Self::presign_read_with] without any options. To use extra options like override_content_disposition, please use [Self::presign_read_with] instead.

§Example
use anyhow::Result;
use futures::io;
use opendal::Operator;
use std::time::Duration;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op.presign_read("test.txt", Duration::from_secs(3600)).await?;
  • signed_req.method(): GET
  • signed_req.uri(): https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
  • signed_req.headers(): { "host": "s3.amazonaws.com" }

We can download this file via curl or other tools without credentials:

curl "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>" -O /tmp/test.txt

pub fn presign_read_with( &self, path: &str, expire: Duration, ) -> OperatorFuture<(OpRead, Duration), PresignedRequest, impl Future<Output = Result<PresignedRequest, Error>>>

Presign an operation for read with extra options.

§Options
§override_content_disposition

Override the content-disposition header returned by storage services.

use std::time::Duration;

use anyhow::Result;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_read_with("test.txt", Duration::from_secs(3600))
        .override_content_disposition("attachment; filename=\"othertext.txt\"")
        .await?;
    Ok(())
}
§override_cache_control

Override the cache-control header returned by storage services.

use std::time::Duration;

use anyhow::Result;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_read_with("test.txt", Duration::from_secs(3600))
        .override_cache_control("no-store")
        .await?;
    Ok(())
}
§override_content_type

Override the content-type header returned by storage services.

use std::time::Duration;

use anyhow::Result;
use futures::io;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_read_with("test.txt", Duration::from_secs(3600))
        .override_content_type("text/plain")
        .await?;
    Ok(())
}

pub async fn presign_write( &self, path: &str, expire: Duration, ) -> Result<PresignedRequest, Error>

Presign an operation for write.

§Notes
§Extra Options

presign_write is a wrapper of [Self::presign_write_with] without any options. To use extra options like content_type, please use [Self::presign_write_with] instead.

§Example
use std::time::Duration;

use anyhow::Result;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_write("test.txt", Duration::from_secs(3600))
        .await?;
    Ok(())
}
  • signed_req.method(): PUT
  • signed_req.uri(): https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
  • signed_req.headers(): { "host": "s3.amazonaws.com" }

We can upload file as this file via curl or other tools without credential:

curl -X PUT "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>" -d "Hello, World!"

pub fn presign_write_with( &self, path: &str, expire: Duration, ) -> OperatorFuture<(OpWrite, Duration), PresignedRequest, impl Future<Output = Result<PresignedRequest, Error>>>

Presign an operation for write with extra options.

§Options
§content_type

Set the content-type header returned by storage services.

use std::time::Duration;

use anyhow::Result;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_write_with("test", Duration::from_secs(3600))
        .content_type("text/csv")
        .await?;
    let req = http::Request::builder()
        .method(signed_req.method())
        .uri(signed_req.uri())
        .body(())?;

    Ok(())
}
§content_disposition

Set the content-disposition header returned by storage services.

use std::time::Duration;

use anyhow::Result;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_write_with("test", Duration::from_secs(3600))
        .content_disposition("attachment; filename=\"cool.html\"")
        .await?;
    let req = http::Request::builder()
        .method(signed_req.method())
        .uri(signed_req.uri())
        .body(())?;

    Ok(())
}
§cache_control

Set the cache-control header returned by storage services.

use std::time::Duration;

use anyhow::Result;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_write_with("test", Duration::from_secs(3600))
        .cache_control("no-store")
        .await?;
    let req = http::Request::builder()
        .method(signed_req.method())
        .uri(signed_req.uri())
        .body(())?;

    Ok(())
}

pub async fn presign_delete( &self, path: &str, expire: Duration, ) -> Result<PresignedRequest, Error>

Presign an operation for delete.

§Notes
§Extra Options

presign_delete is a wrapper of [Self::presign_delete_with] without any options.

§Example
use std::time::Duration;

use anyhow::Result;
use opendal::Operator;

async fn test(op: Operator) -> Result<()> {
    let signed_req = op
        .presign_delete("test.txt", Duration::from_secs(3600))
        .await?;
    Ok(())
}
  • signed_req.method(): DELETE
  • signed_req.uri(): https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
  • signed_req.headers(): { "host": "s3.amazonaws.com" }

We can delete file as this file via curl or other tools without credential:

curl -X DELETE "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>"

pub fn presign_delete_with( &self, path: &str, expire: Duration, ) -> OperatorFuture<(OpDelete, Duration), PresignedRequest, impl Future<Output = Result<PresignedRequest, Error>>>

Presign an operation for delete without extra options.

Trait Implementations§

Source§

impl Deref for SpillOp

Source§

type Target = Operator

The resulting type after dereferencing.
Source§

fn deref(&self) -> &Self::Target

Dereferences the value.
Source§

impl DerefMut for SpillOp

Source§

fn deref_mut(&mut self) -> &mut Self::Target

Mutably dereferences the value.
Source§

impl Drop for SpillOp

Source§

fn drop(&mut self)

Executes the destructor for this type. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
§

impl<T> AsAny for T
where T: Any + Send + Sync,

§

fn any_ref(&self) -> &(dyn Any + Send + Sync + 'static)

Obtains a dyn Any reference to the object: Read more
§

fn as_any(self: Arc<T>) -> Arc<dyn Any + Send + Sync>

Obtains an Arc<dyn Any> reference to the object: Read more
§

fn into_any(self: Box<T>) -> Box<dyn Any + Send + Sync>

Converts the object to Box<dyn Any>: Read more
§

fn type_name(&self) -> &'static str

Convenient wrapper for std::any::type_name, since Any does not provide it and Any::type_id is useless as a debugging aid (its Debug is just a mess of hex digits).
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
§

impl<T> Conv for T

§

fn conv<T>(self) -> T
where Self: Into<T>,

Converts self into T using Into<T>. Read more
§

impl<Choices> CoproductSubsetter<CNil, HNil> for Choices

§

type Remainder = Choices

§

fn subset( self, ) -> Result<CNil, <Choices as CoproductSubsetter<CNil, HNil>>::Remainder>

Extract a subset of the possible types in a coproduct (or get the remaining possibilities) Read more
§

impl<T> Downcast for T
where T: Any,

§

fn into_any(self: Box<T>) -> Box<dyn Any>

Converts Box<dyn Trait> (where Trait: Downcast) to Box<dyn Any>, which can then be downcast into Box<dyn ConcreteType> where ConcreteType implements Trait.
§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

Converts Rc<Trait> (where Trait: Downcast) to Rc<Any>, which can then be further downcast into Rc<ConcreteType> where ConcreteType implements Trait.
§

fn as_any(&self) -> &(dyn Any + 'static)

Converts &Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot generate &Any’s vtable from &Trait’s.
§

fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)

Converts &mut Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot generate &mut Any’s vtable from &mut Trait’s.
§

impl<T> DowncastSend for T
where T: Any + Send,

§

fn into_any_send(self: Box<T>) -> Box<dyn Any + Send>

Converts Box<Trait> (where Trait: DowncastSend) to Box<dyn Any + Send>, which can then be downcast into Box<ConcreteType> where ConcreteType implements Trait.
§

impl<T> DowncastSync for T
where T: Any + Send + Sync,

§

fn into_any_sync(self: Box<T>) -> Box<dyn Any + Send + Sync>

Converts Box<Trait> (where Trait: DowncastSync) to Box<dyn Any + Send + Sync>, which can then be downcast into Box<ConcreteType> where ConcreteType implements Trait.
§

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Send + Sync>

Converts Arc<Trait> (where Trait: DowncastSync) to Arc<Any>, which can then be downcast into Arc<ConcreteType> where ConcreteType implements Trait.
§

impl<T> FmtForward for T

§

fn fmt_binary(self) -> FmtBinary<Self>
where Self: Binary,

Causes self to use its Binary implementation when Debug-formatted.
§

fn fmt_display(self) -> FmtDisplay<Self>
where Self: Display,

Causes self to use its Display implementation when Debug-formatted.
§

fn fmt_lower_exp(self) -> FmtLowerExp<Self>
where Self: LowerExp,

Causes self to use its LowerExp implementation when Debug-formatted.
§

fn fmt_lower_hex(self) -> FmtLowerHex<Self>
where Self: LowerHex,

Causes self to use its LowerHex implementation when Debug-formatted.
§

fn fmt_octal(self) -> FmtOctal<Self>
where Self: Octal,

Causes self to use its Octal implementation when Debug-formatted.
§

fn fmt_pointer(self) -> FmtPointer<Self>
where Self: Pointer,

Causes self to use its Pointer implementation when Debug-formatted.
§

fn fmt_upper_exp(self) -> FmtUpperExp<Self>
where Self: UpperExp,

Causes self to use its UpperExp implementation when Debug-formatted.
§

fn fmt_upper_hex(self) -> FmtUpperHex<Self>
where Self: UpperHex,

Causes self to use its UpperHex implementation when Debug-formatted.
§

fn fmt_list(self) -> FmtList<Self>
where &'a Self: for<'a> IntoIterator,

Formats each item in a sequence. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> FutureExt for T

§

fn with_context(self, otel_cx: Context) -> WithContext<Self>

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
§

fn with_current_context(self) -> WithContext<Self>

Attaches the current Context to this type, returning a WithContext wrapper. Read more
§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self>

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> IntoRequest<T> for T

Source§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
§

impl<T> IntoResult<T> for T

§

type Err = Infallible

§

fn into_result(self) -> Result<T, <T as IntoResult<T>>::Err>

§

impl<Unshared, Shared> IntoShared<Shared> for Unshared
where Shared: FromUnshared<Unshared>,

§

fn into_shared(self) -> Shared

Creates a shared type from an unshared type.
§

impl<T, U, I> LiftInto<U, I> for T
where U: LiftFrom<T, I>,

§

fn lift_into(self) -> U

Performs the indexed conversion.
Source§

impl<M> MetricVecRelabelExt for M

Source§

fn relabel( self, metric_level: MetricLevel, relabel_threshold: MetricLevel, ) -> RelabeledMetricVec<M>

Source§

fn relabel_n( self, metric_level: MetricLevel, relabel_threshold: MetricLevel, relabel_num: usize, ) -> RelabeledMetricVec<M>

Source§

fn relabel_debug_1( self, relabel_threshold: MetricLevel, ) -> RelabeledMetricVec<M>

Equivalent to RelabeledMetricVec::with_metric_level_relabel_n with metric_level set to MetricLevel::Debug and relabel_num set to 1.
§

impl<T> Pipe for T
where T: ?Sized,

§

fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> R
where Self: Sized,

Pipes by value. This is generally the method you want to use. Read more
§

fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> R
where R: 'a,

Borrows self and passes that borrow into the pipe function. Read more
§

fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> R
where R: 'a,

Mutably borrows self and passes that borrow into the pipe function. Read more
§

fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R
where Self: Borrow<B>, B: 'a + ?Sized, R: 'a,

Borrows self, then passes self.borrow() into the pipe function. Read more
§

fn pipe_borrow_mut<'a, B, R>( &'a mut self, func: impl FnOnce(&'a mut B) -> R, ) -> R
where Self: BorrowMut<B>, B: 'a + ?Sized, R: 'a,

Mutably borrows self, then passes self.borrow_mut() into the pipe function. Read more
§

fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R
where Self: AsRef<U>, U: 'a + ?Sized, R: 'a,

Borrows self, then passes self.as_ref() into the pipe function.
§

fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R
where Self: AsMut<U>, U: 'a + ?Sized, R: 'a,

Mutably borrows self, then passes self.as_mut() into the pipe function.
§

fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R
where Self: Deref<Target = T>, T: 'a + ?Sized, R: 'a,

Borrows self, then passes self.deref() into the pipe function.
§

fn pipe_deref_mut<'a, T, R>( &'a mut self, func: impl FnOnce(&'a mut T) -> R, ) -> R
where Self: DerefMut<Target = T> + Deref, T: 'a + ?Sized, R: 'a,

Mutably borrows self, then passes self.deref_mut() into the pipe function.
§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
Source§

impl<P, T> Receiver for P
where P: Deref<Target = T> + ?Sized, T: ?Sized,

Source§

type Target = T

🔬This is a nightly-only experimental API. (arbitrary_self_types)
The target type on which the method may be called.
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
§

impl<T> Scope for T

§

fn with<F, R>(self, f: F) -> R
where Self: Sized, F: FnOnce(Self) -> R,

Scoped with ownership.
§

fn with_ref<F, R>(&self, f: F) -> R
where F: FnOnce(&Self) -> R,

Scoped with reference.
§

fn with_mut<F, R>(&mut self, f: F) -> R
where F: FnOnce(&mut Self) -> R,

Scoped with mutable reference.
§

impl<Source> Sculptor<HNil, HNil> for Source

§

type Remainder = Source

§

fn sculpt(self) -> (HNil, <Source as Sculptor<HNil, HNil>>::Remainder)

Consumes the current HList and returns an HList with the requested shape. Read more
Source§

impl<T> SerTo<T> for T

§

impl<T> Tap for T

§

fn tap(self, func: impl FnOnce(&Self)) -> Self

Immutable access to a value. Read more
§

fn tap_mut(self, func: impl FnOnce(&mut Self)) -> Self

Mutable access to a value. Read more
§

fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self
where Self: Borrow<B>, B: ?Sized,

Immutable access to the Borrow<B> of a value. Read more
§

fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self
where Self: BorrowMut<B>, B: ?Sized,

Mutable access to the BorrowMut<B> of a value. Read more
§

fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self
where Self: AsRef<R>, R: ?Sized,

Immutable access to the AsRef<R> view of a value. Read more
§

fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self
where Self: AsMut<R>, R: ?Sized,

Mutable access to the AsMut<R> view of a value. Read more
§

fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self
where Self: Deref<Target = T>, T: ?Sized,

Immutable access to the Deref::Target of a value. Read more
§

fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self
where Self: DerefMut<Target = T> + Deref, T: ?Sized,

Mutable access to the Deref::Target of a value. Read more
§

fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self

Calls .tap() only in debug builds, and is erased in release builds.
§

fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self

Calls .tap_mut() only in debug builds, and is erased in release builds.
§

fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self
where Self: Borrow<B>, B: ?Sized,

Calls .tap_borrow() only in debug builds, and is erased in release builds.
§

fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self
where Self: BorrowMut<B>, B: ?Sized,

Calls .tap_borrow_mut() only in debug builds, and is erased in release builds.
§

fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self
where Self: AsRef<R>, R: ?Sized,

Calls .tap_ref() only in debug builds, and is erased in release builds.
§

fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self
where Self: AsMut<R>, R: ?Sized,

Calls .tap_ref_mut() only in debug builds, and is erased in release builds.
§

fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self
where Self: Deref<Target = T>, T: ?Sized,

Calls .tap_deref() only in debug builds, and is erased in release builds.
§

fn tap_deref_mut_dbg<T>(self, func: impl FnOnce(&mut T)) -> Self
where Self: DerefMut<Target = T> + Deref, T: ?Sized,

Calls .tap_deref_mut() only in debug builds, and is erased in release builds.
§

impl<T> TryConv for T

§

fn try_conv<T>(self) -> Result<T, Self::Error>
where Self: TryInto<T>,

Attempts to convert self into T using TryInto<T>. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more
Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

impl<T> LruValue for T
where T: Send + Sync,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> Value for T
where T: Send + Sync + 'static,