From 69acc8f9728f9b73a864c0dcca8d1adeb70bec30 Mon Sep 17 00:00:00 2001 From: Luis Arcega Date: Fri, 23 Aug 2024 16:27:58 -0500 Subject: [PATCH 1/3] Updating Opensearch verion to 2.13 --- .../opensearch/.workshop/terraform/preprovision/main.tf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manifests/modules/observability/opensearch/.workshop/terraform/preprovision/main.tf b/manifests/modules/observability/opensearch/.workshop/terraform/preprovision/main.tf index 3e63b59cf..ad42922d1 100644 --- a/manifests/modules/observability/opensearch/.workshop/terraform/preprovision/main.tf +++ b/manifests/modules/observability/opensearch/.workshop/terraform/preprovision/main.tf @@ -21,7 +21,7 @@ locals { # Provision OpenSearch domain resource "aws_opensearch_domain" "opensearch" { domain_name = var.eks_cluster_id - engine_version = "OpenSearch_2.9" + engine_version = "OpenSearch_2.13" # Specify a single instance cluster cluster_config { From ffa36487b05778c715a14ffb4e03cbbb454e9430 Mon Sep 17 00:00:00 2001 From: Luis Arcega Date: Fri, 23 Aug 2024 16:29:08 -0500 Subject: [PATCH 2/3] Adding steps to generate scheduler logs --- .../opensearch/control-plane-logs.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/website/docs/observability/opensearch/control-plane-logs.md b/website/docs/observability/opensearch/control-plane-logs.md index 351129c50..36846f871 100644 --- a/website/docs/observability/opensearch/control-plane-logs.md +++ b/website/docs/observability/opensearch/control-plane-logs.md @@ -162,6 +162,22 @@ Depending on the level of the EKS cluster activity, some of the control plane lo The scheduler logs are shown at the end of the page. Notice that the scheduler log messages indicates `Unable to schedule pod; no fit; waiting` for `scenario-c`. This schedule log message from the control plane logs is similar to the Kubernetes event we saw for `scenario-c` on the previous page. +:::tip +If for some reason you don't see any logs for the scheduler, it is because there are no new logs. To generate new logs just re-start the scenarios as follow: + +```bash +$ kubectl rollout restart deployment/scenario-a -n test +deployment.apps/scenario-a restarted + +$ kubectl rollout restart deployment/scenario-b -n test +deployment.apps/scenario-b restarted + +$ kubectl rollout restart deployment/scenario-c -n test +deployment.apps/scenario-c restarted +``` + +::: + ![Control plane logs detail](./assets/eks-control-plane-logs-scheduler.webp) Expanding the row allows us to drill down and view details as a table or in JSON format. From d92b6d34a184f777bef81bc8a31c5c7063e5731c Mon Sep 17 00:00:00 2001 From: Luis Arcega Date: Sat, 28 Sep 2024 12:48:28 -0500 Subject: [PATCH 3/3] Adding how it is an observability strategy with opensearch --- website/docs/observability/opensearch/index.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/website/docs/observability/opensearch/index.md b/website/docs/observability/opensearch/index.md index 1ff7674e3..fccb2e14e 100644 --- a/website/docs/observability/opensearch/index.md +++ b/website/docs/observability/opensearch/index.md @@ -26,6 +26,16 @@ You can view the Terraform that applies these changes [here](https://github.com/ ::: -In this lab, we will explore the use of [OpenSearch](https://opensearch.org/about.html) for observability. OpenSearch is a community-driven, open-source search and analytics suite used to ingest, search, visualize and analyze data. OpenSearch consists of a data store and search engine (OpenSearch), a visualization and user interface (OpenSearch Dashboards), and a server-side data collector (Data Prepper). We will be using [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/), which is a managed service that makes it easy for you to perform interactive log analytics, real-time application monitoring, search, and more. +The proposed observability strategy for this module lies the utilization of [OpenSearch](https://opensearch.org/about.html), a powerful community-driven, and scalable open-source search and analytics engine, and the Observability Pyramid (Logs, Metrics, Traces) framework. This strategy provides a starting point for participants to think about how to approach observability in their organizations. + +The strategy begins with a focus on centralized logging, where all logs from the various EKS components (including worker nodes, control plane and containers), and application metrics and traceability are ingested into an OpenSearch cluster **(metrics and traces will be added in a future release)**. This allows for comprehensive log aggregation and tracing, enabling participants to leverage OpenSearch's robust search and analytics capabilities to gain visibility into system-wide events and identify potential issues. + +Underpinning the technical aspects of this strategy is a focus on cultivating an observability-driven culture. By involving cross-functional teams, including developers, site reliability engineers (SREs), and DevOps professionals, in the adoption and maintenance of the OpenSearch-based observability solution, the participants can foster a collaborative environment where data-driven problem-solving and continuous improvement become the norm. + +Now let's talk a little bit about [OpenSearch](https://opensearch.org/about.html). OpenSearch consists of a data store and search engine (OpenSearch), a visualization and user interface (OpenSearch Dashboards), and a server-side data collector (Data Prepper). We will be using [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/), which is a managed service that makes it easy for you to perform interactive log analytics, real-time application monitoring, search, and more. Kubernetes events, control plane logs and pod logs are exported from Amazon EKS to Amazon OpenSearch Service to demonstrate how the two Amazon services can be used together to improve observability. + +:::info +You can find more information related to the Observability Pyramid (Logs, Metrics, Traces) framework in the book ["Observability Engineering" by Charity Majors et al](https://www.amazon.com/Observability-Engineering-Achieving-Production-Excellence/dp/1492076449) +:::