Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compaction with no PKI #86

Merged
merged 15 commits into from
Mar 11, 2023
Merged

Compaction with no PKI #86

merged 15 commits into from
Mar 11, 2023

Conversation

valiantljk
Copy link
Collaborator

@valiantljk valiantljk commented Feb 17, 2023 •
edited
Loading

#81 #82 #83 #77

  • Removed pki related code
  • Added is_src_delta in delta_file_envelope
  • Modified delta rebatch to group manifest entries based on partition locator
  • Modified hash bucket to pass around the is_src_delta to dd and mat
  • Modified compaction_session to discover deltas from two sources: uncompacted and compacted
  • Added support for retrieving high watermark of source based on compacted partition locator Add DeltaCAT Storage API to Automatically Return an Input Table's Compaction High Watermark #77

Test 1: Rebase compaction (Spark compacted table with/without new deltas)--->Ok
Test 2: Incremental compaction (based on Ray compacted table and one days' new deltas)--->Ok
Test 3: Backfill compaction (Compaction from scratch) --->Ok

@valiantljk valiantljk marked this pull request as draft February 17, 2023 01:41
@valiantljk valiantljk mentioned this pull request Feb 17, 2023
Closed
@pdames pdames self-requested a review February 17, 2023 07:25
@pdames pdames linked an issue Feb 21, 2023 that may be closed by this pull request
Remove Primary Key Index Building in Main and Create a Separate Branch for PKI Support #81
Closed
@valiantljk valiantljk marked this pull request as ready for review February 21, 2023 20:06
raghumdani
raghumdani reviewed Feb 28, 2023
View reviewed changes
Copy link
Collaborator

@raghumdani raghumdani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making this critical change. Some comments are short-term and can easily be incorporated in this PR but some are long term and may need to be taken up as enhancements.

deltacat/compute/compactor/model/delta_annotated.py Show resolved Hide resolved
deltacat/compute/compactor/model/delta_annotated.py Show resolved Hide resolved
deltacat/compute/compactor/model/delta_file_envelope.py Outdated Show resolved Hide resolved
deltacat/compute/compactor/model/round_completion_info.py Show resolved Hide resolved
deltacat/compute/compactor/model/round_completion_info.py Outdated Show resolved Hide resolved
deltacat/compute/compactor/compaction_session.py Outdated Show resolved Hide resolved
deltacat/compute/compactor/compaction_session.py Outdated Show resolved Hide resolved
deltacat/compute/compactor/utils/io.py Outdated Show resolved Hide resolved
deltacat/compute/compactor/utils/round_completion_file.py Outdated Show resolved Hide resolved
deltacat/compute/compactor/utils/system_columns.py Show resolved Hide resolved
pdames
pdames requested changes Mar 2, 2023
View reviewed changes
deltacat/compute/compactor/utils/io.py Outdated Show resolved Hide resolved
deltacat/compute/compactor/utils/io.py Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
This was referenced Mar 2, 2023
Open
Open
@valiantljk valiantljk requested a review from pdames March 2, 2023 01:42
pdames
pdames approved these changes Mar 10, 2023
View reviewed changes
Copy link
Member

@pdames pdames left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

raghumdani
raghumdani approved these changes Mar 10, 2023
View reviewed changes
Copy link
Collaborator

@raghumdani raghumdani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Looking forward to picking up TODOs in next few weeks.

@valiantljk valiantljk merged commit 63e5e6c into main Mar 11, 2023
@valiantljk valiantljk mentioned this pull request Mar 11, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pdames pdames pdames approved these changes

@raghumdani raghumdani raghumdani approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Remove Primary Key Index Building in Main and Create a Separate Branch for PKI Support
3 participants
@valiantljk @pdames @raghumdani