[API] Idempotent Resource Creation #33092
justenwalker
started this conversation in
Connector Ideas and Features
Replies: 2 comments 1 reply
-
@rileybrook any thoughts? |
Beta Was this translation helpful? Give feedback.
1 reply
-
@marcosmarxm @rileybrook -- I've attempted to implement Option 2 Here: airbytehq/airbyte-platform#298 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Feature Request
I would like an idempotent way to create resources in order to prevent duplicates from being created.
By resource, I mean:
Reason
I often run into this scenario; especially if the Airbyte Server is under load (when it starts returning 504/502 errors):
The more retries, the more duplicates.
Since no resource has any unique constraint besides its UUID (which is generated on creation) -- the server will gladly accept multiple create requests for an identical resource.
Possible Implementations
User-supplied ID
One way to achieve this with minimal backwards incompatibility might be to allow the requests to include the ID for the resource that is being created, just like the Update requests. This field would be optional on create. If provided, it will use it instead of generating a UUID. The ID conflict will cause all but one of the requests to fail. Failed requests can then use a GET to retrieve the resource by the ID it would have created.
Separate Idempotency Key
This would require some API and DB Changes, but we could define a separate Idempotency Key (
ikey
) in each table which will be a unique index in the DB that can block subsequent duplicate INSERTs from succeeding.This key can be optionally provided along with the create request. Upon conflict with the
ikey
, the API would return the object it found in the DB matching theikey
, instead of returning some "Already Exists" error; effectively simulating a "success" response.I suggest this because, in order to make this useful, there needs to be some way to get the object (or at least the primary ID) that ultimately was persisted to the DB and, without returning that object on create, there would need to be a separate API for querying by this new
ikey
which seems like it could expand the API in undesirable ways.Beta Was this translation helpful? Give feedback.
All reactions