[source-s3] Enhancement: Allow for latest file flag #40543
michael-greene-gravie
started this conversation in
Connector Ideas and Features
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Requesting support for latest file flag in [source-s3]
Description
I would like to request a new feature for the [source-s3] connector in Airbyte. Currently, the connector allows us to ingest data from an S3 bucket by specifying a prefix or a list of files. However, it would be very useful if we could also specify a flag to ingest only the latest file in the bucket.
Feature Request
Add a new flag, let's call it
latestFile
, to the [source-s3] connector configuration. When this flag is set totrue
, the connector should only ingest the latest matching file.Example Usage
A full refresh file is being processed.
I am willing to work on this feature request.
Beta Was this translation helpful? Give feedback.
All reactions