Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support retry on broken pipe #481

Open
rad-pat opened this issue Sep 25, 2024 · 2 comments
Open

feat: support retry on broken pipe #481

rad-pat opened this issue Sep 25, 2024 · 2 comments

Comments

@rad-pat
Copy link

rad-pat commented Sep 25, 2024

We are experiencing many issues calling Databend through the python driver because we have Databend in Kubernetes on spot nodes. The spot nodes can be reaped at any point and when that happens, we get errors such as:

APIError: ResponseError with 1067: transport error, source: Some(tonic::transport::Error(Transport, hyper::Error(Io, Custom { kind: BrokenPipe, error: "stream closed because of a broken pipe" })))

Can we have an option to retry on such errors at the driver level? Possibly even sent in as a config param?
databend://u:p@host/db?retry_on_broken_pipe=3

@everpcpc
Copy link
Member

This is not a problem within server and client, but an error between servers in the cluster. It's not safe to simply retry this error with client, since we have no idea about the current query.

Maybe we could retry at server level? cc @zhang2014

@zhang2014
Copy link
Member

When only network failures occur and the server is still available, it is safe to retry between nodes. However, if the instance has already been killed, it is not possible to retry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants