Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DSIP-66][k8s] YAML Pod for User Customization #16482

Closed
2 tasks done
Tracked by #14102
Mighten opened this issue Aug 19, 2024 · 1 comment
Closed
2 tasks done
Tracked by #14102

[DSIP-66][k8s] YAML Pod for User Customization #16482

Mighten opened this issue Aug 19, 2024 · 1 comment
Assignees
Labels
DSIP feature new feature

Comments

@Mighten
Copy link
Contributor

Mighten commented Aug 19, 2024

Search before asking

  • I had searched in the DSIP and found no similar DSIP.

Motivation

Supporting user-customized K8s YAML Pod has the following benefits:

  • Flexibility: Unlike the existing K8s low-code job with limited functionality, YAML Pod provide users with the flexibility to define sophisticated task instances in DolphinScheduler, similar to how custom JSON does in DataX.

  • Workflow Customization: Users can integrate operational and maintenance processes into DolphinScheduler using YAML Pod for complex workflows.

In short, by enabling user-customized YAML Pod, DolphinScheduler can better support a wide range of Kubernetes-based workflows and operational requirements.

Design Detail

Frontend Design

The frontend adds support for user-customized k8s YAML Pod while remaining compatible with the original k8s low-code jobs.

  1. The Web UI layouts

    2-2-1-frontend-design

    When the user switches on the Custom Template, the Low-code k8s Job fields should hide and YAML editor should appear (or vice versa), similar to the JSON Custom Template in the DataX plugin.

    This feature is implemented using the Vue component span, which is controlled by reactive variables (such as yamlEditorSpan) in the file dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts.

  2. The Request body

    When the user switches to Custom Template mode, the request body should include only YAML-related fields (customConfig and yamlContent), and all previously hidden fields should not be sent.

    This feature is implemented using the taskParams in the file dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts

  3. i18n/locales

    Apache DolphinScheduler is an international software and should support multiple languages.

    The text on the Web UI are retrieved from variables defined in the file dolphinscheduler-ui/src/locales/{en_US, zh_CN}/project.ts. And for user-customized k8s YAML tasks, there are three key variables to consider:

    • k8s_custom_template: the label for the switch to enable user-customized k8s YAML tasks.
    • k8s_yaml_template: the label for the text editor used to input user YAML.
    • k8s_yaml_empty_tips: the warning message displayed when a user tries to submit empty YAML

    This feature is implemented by invoking t('project.node.${variable_name}') (such as t('project.node.k8s_yaml_template')) in the file dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts.

Backend Design

The backend design describes the process of how the worker executes user-customized k8s YAML Pod tasks.

2-3-1-backend-design-overview

After the worker checks the parameters, K8sYamlTaskExecutor is loaded for the current user-customized k8s YAML Pod task. Once the YAML is parsed into HasMetadata, its kind field is used to assign abstractK8sOperation as K8sPodOperation for executing the YAML Pod task.

  1. K8s Task Executors

    2-3-2-backend-design-task-executors

    Three k8s task executor are involved:

    • AbstractK8sTaskExecutor is an abstract class that represents a k8s task executor.
    • K8sTaskExecutor is a concrete class that extends AbstractK8sTaskExecutor to represent a low-code executor
    • K8sYamlTaskExecutor is a concrete class that extends AbstractK8sTaskExecutor to represent a user-customized k8s YAML task executor.
  2. K8s Operation handlers

    2-3-3-backend-design-operation-handlers

    Two operation handlers are involved:

    • AbstractK8sOperation is an interface representing all k8s resource operations.
    • K8sPodOperation is a concrete class that implements AbstractK8sOperation to handle Pod operations

Usecase Design

A typical use case for a k8s YAML Pod task includes uploading YAML, online workflows, and starting workflows, similar to k8s low-code jobs, unless users switch to the Custom Template option to fill in Pod YAML.

2-4-1-usecase-design

  1. The user edits a k8s Pod YAML node in a workflow
  2. If the Custom Template is activated and YAML content is not blank, the user may online this whole workflow
  3. If the workflow is online, the user may start the workflow and review the logs generated during the execution of the workflow.

Compatibility, Deprecation, and Migration Plan

The user-customized k8s YAML feature requires only customConfig to be activated(1). By default, the value is 0, which applies to the existing k8s low-code jobs.

This DSIP does not propose any deprecation or migration plan, but community feedback is valued.

Test Plan

The User-customized k8s YAML Pod feature allows users to submit YAML Pod to k8s. And this test plan aims to ensure that the feature functions as expected and meets user requirements.

Test Case # Name Action Expectation
1 UI Display Edit YAML, save and reopen The YAML content stays up-to-date.
2 UI Validation try to submit empty YAML The UI modal dialog intercepts empty YAML.
3 Online Workflow Save workflow, and online The User successfully brings the workflow online.
4 Dryrun Workflow Run workflow as dryrun mode The Master successfully dry runs this task.
5 Test Workflow Run workflow as test mode The Worker successfully tests this task.
6 Run Workflow Run workflow The Worker successfully runs this task.

Code of Conduct

@Mighten Mighten added DSIP Waiting for reply Waiting for reply labels Aug 19, 2024
@SbloodyS SbloodyS removed the Waiting for reply Waiting for reply label Aug 19, 2024
@ruanwenjun ruanwenjun changed the title [DSIP-63][k8s] YAML Pod for User Customization [DSIP-66][k8s] YAML Pod for User Customization Aug 19, 2024
@fuchanghai fuchanghai added the feature new feature label Aug 21, 2024
@SbloodyS
Copy link
Member

Closing for no plans to do this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DSIP feature new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants