Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opensearch centric strategy #1120

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ locals {
# Provision OpenSearch domain
resource "aws_opensearch_domain" "opensearch" {
domain_name = var.eks_cluster_id
engine_version = "OpenSearch_2.9"
engine_version = "OpenSearch_2.13"

# Specify a single instance cluster
cluster_config {
Expand Down
16 changes: 16 additions & 0 deletions website/docs/observability/opensearch/control-plane-logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,22 @@ Depending on the level of the EKS cluster activity, some of the control plane lo

The scheduler logs are shown at the end of the page. Notice that the scheduler log messages indicates `Unable to schedule pod; no fit; waiting` for `scenario-c`. This schedule log message from the control plane logs is similar to the Kubernetes event we saw for `scenario-c` on the previous page.

:::tip
If for some reason you don't see any logs for the scheduler, it is because there are no new logs. To generate new logs just re-start the scenarios as follow:

```bash
$ kubectl rollout restart deployment/scenario-a -n test
deployment.apps/scenario-a restarted

$ kubectl rollout restart deployment/scenario-b -n test
deployment.apps/scenario-b restarted

$ kubectl rollout restart deployment/scenario-c -n test
deployment.apps/scenario-c restarted
```

:::

![Control plane logs detail](./assets/eks-control-plane-logs-scheduler.webp)

Expanding the row allows us to drill down and view details as a table or in JSON format.
Expand Down
12 changes: 11 additions & 1 deletion website/docs/observability/opensearch/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,16 @@ You can view the Terraform that applies these changes [here](https://github.com/

:::

In this lab, we will explore the use of [OpenSearch](https://opensearch.org/about.html) for observability. OpenSearch is a community-driven, open-source search and analytics suite used to ingest, search, visualize and analyze data. OpenSearch consists of a data store and search engine (OpenSearch), a visualization and user interface (OpenSearch Dashboards), and a server-side data collector (Data Prepper). We will be using [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/), which is a managed service that makes it easy for you to perform interactive log analytics, real-time application monitoring, search, and more.
The proposed observability strategy for this module lies the utilization of [OpenSearch](https://opensearch.org/about.html), a powerful community-driven, and scalable open-source search and analytics engine, and the Observability Pyramid (Logs, Metrics, Traces) framework. This strategy provides a starting point for participants to think about how to approach observability in their organizations.

The strategy begins with a focus on centralized logging, where all logs from the various EKS components (including worker nodes, control plane and containers), and application metrics and traceability are ingested into an OpenSearch cluster **(metrics and traces will be added in a future release)**. This allows for comprehensive log aggregation and tracing, enabling participants to leverage OpenSearch's robust search and analytics capabilities to gain visibility into system-wide events and identify potential issues.

Underpinning the technical aspects of this strategy is a focus on cultivating an observability-driven culture. By involving cross-functional teams, including developers, site reliability engineers (SREs), and DevOps professionals, in the adoption and maintenance of the OpenSearch-based observability solution, the participants can foster a collaborative environment where data-driven problem-solving and continuous improvement become the norm.

Now let's talk a little bit about [OpenSearch](https://opensearch.org/about.html). OpenSearch consists of a data store and search engine (OpenSearch), a visualization and user interface (OpenSearch Dashboards), and a server-side data collector (Data Prepper). We will be using [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/), which is a managed service that makes it easy for you to perform interactive log analytics, real-time application monitoring, search, and more.

Kubernetes events, control plane logs and pod logs are exported from Amazon EKS to Amazon OpenSearch Service to demonstrate how the two Amazon services can be used together to improve observability.

:::info
You can find more information related to the Observability Pyramid (Logs, Metrics, Traces) framework in the book ["Observability Engineering" by Charity Majors et al](https://www.amazon.com/Observability-Engineering-Achieving-Production-Excellence/dp/1492076449)
:::
Loading