Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Block-builder-scheduler initial structure #9650

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

Block-builder-scheduler initial structure #9650

wants to merge 24 commits into from
+207 −19

Conversation

seizethedave
Copy link
Contributor

@seizethedave seizethedave commented Oct 16, 2024 •
edited
Loading

What this PR does

Adds initial structure for block-builder-scheduler living at pkg/blockbuilder/scheduler. Adds a new block-builder-scheduler target to the binary. This target currently just connects to Kafka and records per-partition start and end offset gauge metrics. In the future it will compute jobs and assign them to block-builder workers.

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

@seizethedave
Initial folder.
170ceec
@seizethedave
Merge remote-tracking branch 'origin/main' into davidgrant/block-buil…
5cf7a81 
…der-scheduler
@seizethedave
Kafka particulars.
f8a1440
@seizethedave
Basic observing of partition start/end offsets.
fb6637b
@seizethedave
comment typo
701e51b
@seizethedave
block-builder-scheduler target, flag, config plumbing.
b5c0995
@seizethedave
Fix refs to BlockBuilder.
50b5530
@seizethedave
Add IngesterPartitionRing as dep of BlockBuilderScheduler.
d628855
@seizethedave
Learn active partitions from ring watcher.
30514f0
@seizethedave
use PartitionRingReader.
0d28890
@seizethedave
nix lint warning from unused logger.
1205869
@seizethedave
Appease linter.
a82fc3c
@seizethedave
AGPL headers.
1da551a
@seizethedave
Remove PartitionRingReader stuff.
bea9109
@seizethedave
Don't try to validate block-builder/scheduler configs, as the Kafka b…
1fab80f 
…its are injected later.
@seizethedave
Specify scheduler's consumer group.
4554355
@seizethedave
Move to pkg/blockbuilder/scheduler
2defdf6
@seizethedave
Compute backlog time.
7a17548
@seizethedave
Don't need the lag stuff yet.
b2631bd
@seizethedave
Fix monitor duration timing,.
ef3c19d
@seizethedave
Revert "Fix monitor duration timing,."
0019293 
This reverts commit ef3c19d.
@seizethedave
Revert "Don't need the lag stuff yet."
5d619d3 
This reverts commit b2631bd.
@seizethedave
Revert "Compute backlog time."
271992a 
This reverts commit 7a17548.
@seizethedave
Merge remote-tracking branch 'origin/main' into davidgrant/block-buil…
15ce592 
…der-scheduler
@seizethedave seizethedave marked this pull request as ready for review October 18, 2024 16:49
@seizethedave seizethedave requested a review from a team as a code owner October 18, 2024 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pracucci pracucci Awaiting requested review from pracucci

@codesome codesome Awaiting requested review from codesome

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@seizethedave