diff --git a/docs/custom-json-configuration/alert_notification_handlers_configuration.md b/docs/custom-json-configuration/alert_notification_handlers_configuration.md new file mode 100644 index 0000000..28b963b --- /dev/null +++ b/docs/custom-json-configuration/alert_notification_handlers_configuration.md @@ -0,0 +1,63 @@ +## Alert Notification Handlers [Early Access] + +The `alertNotificationHandlers` section configures integrations for sending alerts when monitored resources meet specified conditions. This enables automated notifications to external platforms such as Slack. + +### Slack Handler + +The Slack handler allows alerts to be sent to a designated Slack channel. Configuration options include: + +| Property | Description | +|----------------------------|-----------------------------------------------------------------------------| +| `accessToken` | The Slack API token used for authentication. | +| `acknowledgeAlertEnabled` | Boolean flag to enable/disable alert acknowledgment in Slack. This is by dafault false since implementation is not done. | +| `channel` | The Slack channel ID where alerts will be posted. | +| `enabled` | Boolean flag to enable/disable Slack notifications. | +| `messageIntervalSeconds` | Interval (in seconds) between alert messages sent to Slack. It should be more than or equal to min slack interval in seconds i.e. 180 | + +--- + +### Configure Slack in Custom JSON Configuration File + +#### Prerequisites + +Before configuring Slack notifications: + +- Create a Slack App in your Slack workspace. +- Generate an OAuth token with the required permissions to post messages to channels. + +#### Configuration + +To configure Slack notification in the custom JSON configuration file, locate the `alertNotificationHandlers` section and update the configuration as below. + +- Provide OAuth Token in `accessToken`. +- Set `channel` to the Slack channel ID where alerts will be sent. +- Set `enabled` to `true` to enable Slack notifications. +- Set `messageIntervalSeconds` to define the interval at which messages are sent to Slack. By default, it is set to 180 seconds from the code base. If, we set it to less than 180 seconds, it will be overridden to 180 seconds. + +```json +"alertNotificationHandlers": { + "slack": { + "accessToken": "slack-access-token", + "acknowledgeAlertEnabled": false, + "channel": "slack-channel-id", + "enabled": true, + "messageIntervalSeconds": 60 + } + } +``` + +### Verification in Kibana + +- Navigate to Kibana Discover. +- Select `logs-*` Data View. +- Search for "The Environment Watch shared configuration object is not empty" which indicates that the EW Windows Service fetching values from the custom JSON configuration file successfully. + +![](/resources/custom-json-images/environment-watch-shared-settings-not-empty-generic.png) + +### Slack Notification Example + +![Slack Example](../../resources/slackalerts-images/SlackNotification.png) + +## Troubleshooting +Refer to the [Troubleshooting Guide](../troubleshooting/custom-json-troubleshooting.md) to resolve any custom JSON slack configuration issues. + diff --git a/docs/custom-json-configuration/certificates_configuration.md b/docs/custom-json-configuration/certificates_configuration.md new file mode 100644 index 0000000..30dac26 --- /dev/null +++ b/docs/custom-json-configuration/certificates_configuration.md @@ -0,0 +1,109 @@ +# Certificates Configuration + +This section describes how to configure certificate monitoring using the `environmentWatchConfiguration` JSON configuration file. + +--- + +## Overview + +Monitors the presence and validity of specified certificates in Windows certificate stores. By default, the Relativity Secret Store certificate is monitored without requiring additional configuration. Other certificates can be added based on the installed product or specific requirements. + +**Default Certificates** +| Certificate Name | Description | +|-----------------------------------|--------------------------------------------------| +| Relativity Secret Store | Certificate for Relativity Secret Store. | + +**Properties In Custom JSON Configuration File Related to Certificates** + +| Property | Type | Description | +|----------------|----------|------------------------------------------------------------------| +| `enabled` | boolean | Enables or disables monitoring for certificates. | +| `include` | array | List of certificate objects to monitor. | +| `storeName` | string | Name of the certificate store (e.g., `"My"`). | +| `storeLocation`| string | Location of the store (e.g., `"LocalMachine"`). | +| `thumbprint` | string | Certificate thumbprint to identify the certificate. | + +#### StoreLocation Enum Values + +The `storeLocation` field specifies the location of the X.509 certificate store to use. + +**Possible Values** + +| Value | Description | +|---------------|----------------------------------------------------------------| +| CurrentUser | The X.509 certificate store is located in the current user's profile. | +| LocalMachine | The X.509 certificate store is located in the local computer's profile. | + +#### StoreName Enum Values + +The `storeName` field specifies the name of the Windows certificate store where the X.509 certificate is located. + +**Possible Values** + +| Value | Description | +|----------------------|-----------------------------------------------| +| AddressBook | Other people | +| AuthRoot | Third party trusted roots | +| CertificateAuthority | Intermediate CAs | +| Disallowed | Revoked certificates | +| My | Personal certificates | +| Root | Trusted root CAs | +| TrustedPeople | Trusted people (used in EFS) | +| TrustedPublisher | Trusted publishers (used in Authenticode) | + +**Get Certificate Thumbprint** + +Depending on the Store Location and Store Name, run the following command on the host. For `LocalMachine` and `My`, use: + +```powershell +Get-ChildItem Cert:\LocalMachine\My +``` + +The command returns a list of certificates including their `thumbprint` and `subject`. Copy the `thumbprint` value for the certificate to be monitored and use it in the custom JSON configuration file. Adjust the command as needed based on the selected `storeName` and `storeLocation`. + +## Configure Certificates + +Certificates can be monitored at the "**hosts**", "**instance**", or "**installedProducts**" level. +For certificates to monitor, locate "**certificates**" under the desired section and update the configuration as below. + +- `enabled` : Set to `true` to enable certificate monitoring. +- When configuring the `include` section, specify the `storeName`, `storeLocation`, and `thumbprint` for each certificate to be monitored. + +**Example 1**: Monitoring two certificates from the LocalMachine\My store. The certificate is identified by its Thumbprint, which can be retrieved using the following PowerShell command: `Get-ChildItem Cert:\LocalMachine\My` + +```json +{ + "certificates": { + "enabled": true, + "include": [ + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "005501F9BA68A2ED7D9BD515B256F6298AEF7E5A" + }, + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "E62D7D4DD8D054072A7A58A577D500753A586C75" + } + ] + } +} +``` + +### Verification in Kibana + +- Navigate to Kibana Discover. +- Select `logs-*` Data View. +- Search for "The Environment Watch shared configuration object is not empty" which indicates that the EW Windows Service fetching values from the custom JSON configuration file successfully. + +![](/resources/custom-json-images/environment-watch-shared-settings-not-empty-generic.png) +- Navigate to the Kibana certificates dashboard. +- Ensure that the certificates defined in the custom JSON configuration file appear on the Kibana certificates dashboard. The example below demonstrates how a certificate specified in the custom JSON configuration file is successfully monitored and displayed on the certificates dashboard. + +![](/resources/custom-json-images/certificate-json-example.png) + +![](/resources/custom-json-images/certificate-dashboard-example.png) + +## Troubleshooting +Refer to the [Troubleshooting Guide](../troubleshooting/custom-json-troubleshooting.md) to resolve any custom JSON certificate configuration issues. \ No newline at end of file diff --git a/docs/custom-json-configuration/environment_watch_configuration.md b/docs/custom-json-configuration/environment_watch_configuration.md new file mode 100644 index 0000000..6a6c47e --- /dev/null +++ b/docs/custom-json-configuration/environment_watch_configuration.md @@ -0,0 +1,313 @@ +# Custom JSON Configuration + +This document provides an overview of the custom JSON configuration file used by Environment Watch. The configuration allows users to centrally define and customize monitoring for Windows services and certificates, as well as configure Slack notifications for alerting. + +The shared configuration file enables users to control what is monitored—such as specific Windows services or certificate conditions—and how alerts are delivered. Currently, Slack is the only supported notification platform. The notification configuration is designed to be extensible, allowing additional platforms to be supported in future releases. Because the configuration is external to the application, custom monitoring settings are preserved during Environment Watch upgrades, making the solution both extensible and upgrade-safe. + +--- + +## Configuration Structure + +The configuration is organized in a hierarchical JSON format, with top-level sections and nested objects for each monitored entity. It will be saved in the BCPPath and the inside folder EnvironmentWatch. The name of the file will be environment-watch-configuration.json. Below is an example structure and explanation: + +--- + +## Reference Structure + +- **Top-level object**: `environmentWatchConfiguration` +- **Sections**: + - `monitoring`: Contains configuration for instance, installed products, and hosts. + - `instance`: Defines sources monitored at the instance level. + - `installedProducts`: A list of installed products, where each product defines its own monitoring sources. + - `hosts`: A list of hosts, where each host defines its own monitoring sources. + - `alertNotificationHandlers`: Defines notification handlers (e.g., Slack). + +--- + +## Monitoring Sections + +### Monitoring by Instance +The `instance` section defines sources that are monitored at the environment or system-wide level, regardless of specific products or hosts. +- **Purpose:** Monitors general resources (like certificates or Windows services) that are relevant to the entire instance. +- **Use Case:** Useful for checks that apply everywhere, such as core system services. + +### Monitoring by Installed Product +The `installedProducts` section contains a list of installed products, where each product defines its own monitoring sources. +- **Purpose:** Monitors resources specific to each installed product (e.g., web server, agent). +- **Use Case:** Allows to tailor monitoring to the needs of each product, such as product-specific services or certificates. +- The following installed product values can be used in the `installedProducts` section of the custom JSON configuration file. + +| Property | Description | +|----------------------------|-----------------------------------------------------------------------------| +| `Generic` | No Relativity Product is Installed | +| `QM` | Queue Manager Server | +| `Worker` | Worker Server | +| `Agent` | Agent Server | +| `Web` | Web Server Server | +| `SecretStore` | Secret Store Server | +| `ServiceBus` | Service Bus Server | +| `ServiceHost` | Service Host Server | +| `SQLDistributed` | SQL Distributed Server | +| `SQLPrimary` | SQL Primary Server | +| `Caat` | Analytics Server | + + +### Monitoring by Host +The `hosts` section contains multiple host objects, each specifying its own monitoring sources. +- **Purpose:** Monitors resources on a per-host basis, such as Services or certificates unique to a particular server. +- **Use Case:** Enables granular monitoring for individual machines, supporting host-specific checks (e.g., SQL Services on a database server). + +--- + +### Monitoring Section Breakdown + +| Property | Description | +|----------------------------|-----------------------------------------------------------------------------| +| `sources` | Specifies what is monitored (certificates, windowsServices, sqlServers, etc.)| +| `enabled` | Boolean flag to enable/disable monitoring for the source. | +| `include` | List of specific items to monitor (service names, certificate details, etc.) | +| `otelCollectorYamlFiles` | List of OpenTelemetry Collector YAML files (empty in this example). | + +--- + +## Configuration File Location + +Base path: `BCPPath` + +Folder: `EnvironmentWatch` + +File name: `environment-watch-configuration.json` + +Environment Watch automatically reads this file and applies the defined monitoring rules to the relevant instances, products, and hosts. + +To identify the BCP path for the environment, execute the following SQL query against the '**EDDS**' database: + +```sql +SELECT +[TemporaryDirectory] +FROM [EDDS].[eddsdbo].[ResourceServer] AS rs WITH(NOLOCK) +INNER JOIN [EDDS].[eddsdbo].[ExtendedArtifact] AS ea WITH(NOLOCK) + ON ea.[ArtifactID] = rs.[ArtifactID] +INNER JOIN [EDDS].[eddsdbo].[Code] AS c WITH(NOLOCK) + ON c.[ArtifactID] = rs.[Type] +INNER JOIN [EDDS].[eddsdbo].[ResourceGroupSQLServers] AS rgss WITH(NOLOCK) + ON rgss.[SQLServerArtifactID] = rs.[ArtifactID] +INNER JOIN [EDDS].[eddsdbo].[ResourceGroup] AS rg WITH(NOLOCK) + ON rg.[ArtifactID] = rgss.[ResourceGroupArtifactID] +WHERE c.[Name] = 'SQL - Primary' +``` + +![](/resources/custom-json-images/sql-bcp-path-query.png) + + An example of the BCPPath and folder structure is shown below: + +![](/resources/custom-json-images/bcp-path-custom-json-file-name.png) + +--- + +## Monitoring Source Types + +This section describes the main types of sources that can be monitored using the Environment Watch configuration: Windows services, certificates, Kibana Alerts through Slack notifications, and SQL cluster instances. Each source type has its own configuration structure and properties. Following are the details for each source type: + +--- +### Windows Services + +For detailed instructions, see [Windows Service Configuration](windows_services_configuration.md). + +### Certificates + +For detailed instructions, see [Certificates Configuration](certificates_configuration.md). + +### SQL Cluster Instances + +For detailed instructions, see [SQL Cluster Configuration](../sql-cluster-configuration/sql-cluster-configuration.md). + +### Kibana Alert Slack Notification Handler + +For detailed instructions, see [Alert Notification Handlers](alert_notification_handlers_configuration.md). + +## Example Configuration + +```json +{ + "environmentWatchConfiguration": { + "monitoring": { + "instance": { + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "windowsServices": { + "enabled": true, + "include": [ + "WindowsAzureGuestAgent", + "mpssvc" + ] + } + }, + "otelCollectorYamlFiles": [] + }, + "installedProducts": [ + { + "productName": "Web", + "sources": { + "certificates": { + "enabled": true, + "include": [ + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "A54225760344699530649239D175BAA73C70DC1B" + } + ] + }, + "windowsServices": { + "enabled": true, + "include": [ + "IISADMIN", + "WinDefend" + ] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "productName": "Agent", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "windowsServices": { + "enabled": true, + "include": [ + "RpcSs" + ] + } + }, + "otelCollectorYamlFiles": [] + } + ], + "hosts": [ + { + "hostName": "SQL01", + "sources": { + "certificates": { + "enabled": true, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLClusterName", + "instanceName": "InstanceName" + } + ] + }, + "windowsServices": { + "enabled": true, + "include": [ + "MSSQLSERVER" + ] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "hostName": "DG01", + "sources": { + "certificates": { + "enabled": true, + "include": [] + }, + "windowsServices": { + "enabled": true, + "include": [ + "Dhcp" + ] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "hostName": "DG02", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "windowsServices": { + "enabled": true, + "include": [ + "Schedule" + ] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "hostName": "CORE01", + "sources": { + "certificates": { + "enabled": true, + "include": [ + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "F8809D2677E010477847C92C5A1A673784537CBC" + }, + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "984812C68F059EB19A346D538ECFB072968C11C3" + }, + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "984812C68F059EB19A346D538ECFB072968C11C3" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "hostName": "CORE02", + "sources": { + "certificates": { + "enabled": true, + "include": [ + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "F8809D2677E010477847C92C5A1A673784537CBC" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [] + } + }, + "otelCollectorYamlFiles": [] + } + ] + }, + "alertNotificationHandlers": { + "slack": { + "accessToken": "slack-access-token", + "acknowledgeAlertEnabled": false, + "channel": "ABC12M3PQR4", + "enabled": true, + "messageIntervalSeconds": 180 + } + } + } +} +``` diff --git a/docs/custom-json-configuration/windows_services_configuration.md b/docs/custom-json-configuration/windows_services_configuration.md new file mode 100644 index 0000000..dbe92b1 --- /dev/null +++ b/docs/custom-json-configuration/windows_services_configuration.md @@ -0,0 +1,82 @@ +# Windows Service Configuration + +This section describes how to configure Windows service monitoring using the `environmentWatchConfiguration` JSON object. + +--- + +## Overview + +This configuration monitors the status of specified Windows services to ensure they are running as expected. Some Windows services are monitored by default based on the installed products, even if they are not explicitly configured in the custom JSON configuration file. + +The table below lists the service names that are monitored by default based on the installed products. + +### Default Services + +| Service Name | Description | +|---------------------------------------|--------------------------------------------------| +| apm-server | Elastic Stack APM Server | +| elasticsearch | Elastic Stack Elasticsearch Service | +| W3SVC | Internet Information Services (IIS) | +| QueueManager | Invariant Queue Manager | +| RabbitMQ | RabbitMQ Message Broker | +| kCura EDDS Agent Manager | Relativity Agent Manager | +| Relativity Analytics Engine | Relativity Analytics Engine (CAAT) | +| Relativity Secret Store | Relativity Secret Store | +| kCura Service Host Manager | Relativity Service Host Manager | +| kCura EDDS Web Processing Manager | Relativity Web Processing Manager | + +### Properties Table + +The following table lists the properties used to configure Windows services monitoring in the custom JSON configuration file. + +| Property | Type | Description | +|-------------|----------|------------------------------------------------------------------| +| `enabled` | boolean | Enables or disables monitoring for Windows services. | +| `include` | array | List of Windows service names (not display names) to monitor (for example, "WinDefend").| + +To identify the correct service name: + +- Navigate to the **Services** application on the host. +- Right-click the desired service and select **Properties**. +- Copy the **Service name** value (not the display name). + +## Configure Windows Services + +> [!NOTE] +> Windows service names are case-sensitive and must match exactly as they appear in the Services application. + +Windows services can be monitored at the **hosts**", "**instance**", or "**installedProducts**" level. +For services to monitor, locate "**windowsServices**" under the desired section and update the configuration as below. + +- `enabled` : Set to `true` to enable Windows services monitoring. +- `include` : List the service names to monitor. + +**Example** +```json +{ + "windowsServices": { + "enabled": true, + "include": [ + "Spooler" + ] + } +} +``` + +### Verification in Kibana + +- Navigate to Kibana Discover. +- Select `logs-*` Data View. +- Search for "The Environment Watch shared configuration object is not empty" which indicates that the EW Windows service fetching values from the custom JSON configuration file successfully. + +![](/resources/custom-json-images/environment-watch-shared-settings-not-empty-generic.png) + +- Ensure that the Windows services defined in the custom JSON configuration file appear on the Kibana Windows services dashboard. The example below demonstrates how a Windows service specified in the custom JSON configuration file is successfully monitored and displayed on the Windows services dashboard. +![](/resources/custom-json-images/windows-service-json-example.png) + +![](/resources/custom-json-images/windows-service-dashboard.png) + +--- + +## Troubleshooting +Refer to the [Troubleshooting Guide](../troubleshooting/custom-json-troubleshooting.md) to resolve any custom JSON windows services configuration issues. \ No newline at end of file diff --git a/docs/environment_watch_install_other_integrations.md b/docs/environment_watch_install_other_integrations.md index dad18e0..688f4b2 100644 --- a/docs/environment_watch_install_other_integrations.md +++ b/docs/environment_watch_install_other_integrations.md @@ -21,6 +21,11 @@ The RabbitMQ integration enables monitoring of RabbitMQ queues, exchanges, and n - [Setting Up RabbitMQ Integration](rabbitmq/rabbitmq_integration.md) +### Custom-JSON Integration + +The Custom-JSON integration enables monitoring of Windows Services, Certificates, Kibana Alerts with Slack Notifications, and SQL Cluster Instances. + +- [Setting Up Custom-JSON Integration](custom-json-configuration/environment_watch_configuration.md) ### Manual Elasticsearch Configuration The Manual Elasticsearch Configuration helps to update and optimize essential Elasticsearch settings. diff --git a/docs/environment_watch_troubleshooting.md b/docs/environment_watch_troubleshooting.md index 0434434..6639846 100644 --- a/docs/environment_watch_troubleshooting.md +++ b/docs/environment_watch_troubleshooting.md @@ -10,4 +10,5 @@ This document provides quick reference links to detailed troubleshooting guides - [APM Server Troubleshooting](troubleshooting/apm-server.md) - [Environment Watch Monitoring Agent and Open Telemetry Collector Troubleshooting](troubleshooting/monitoring-agent-and-otel-collector.md) - [Relativity Server CLI Troubleshooting](troubleshooting/relativity-server-cli.md) -- [Relativity Alerts Troubleshooting](troubleshooting/relativity_alerts_troubleshooting.md) \ No newline at end of file +- [Relativity Alerts Troubleshooting](troubleshooting/relativity_alerts_troubleshooting.md) +- [Custom JSON Configuration Troubleshooting](troubleshooting/custom-json-troubleshooting.md) \ No newline at end of file diff --git a/docs/sql-cluster-configuration/sql-cluster-configuration.md b/docs/sql-cluster-configuration/sql-cluster-configuration.md new file mode 100644 index 0000000..7383a56 --- /dev/null +++ b/docs/sql-cluster-configuration/sql-cluster-configuration.md @@ -0,0 +1,120 @@ +# SQL Cluster Configuration +> [!NOTE] +> Use the steps below only if your environment uses SQL cluster servers for Environment Watch monitoring. + +An environment may include SQL cluster instances consisting of two or more nodes. Monitoring these instances using the Environment Watch Windows service requires defining specific configurations in a custom JSON configuration file. + +--- + +## Configure SQL Cluster Instances + +Specify SQL cluster configuration in the custom JSON configuration file within the "**hosts**" section. + +Locate the "**hosts**" section in the JSON file and add an entry for each SQL cluster instance to be monitored. Include each **hostName** with the following details: + +**Example:** +For an environment containing a SQL cluster with two nodes (`SQLNode1` and `SQLNode2`): +Update the "hosts" section for each node by: +- Specify the correct host name +- Set the `enabled` flag to `true` +- Include the appropriate SQL cluster name (`clusterVirtualName`) +- Provide the corresponding instance name (`instanceName`) +- Below configuration sets both `SQLNode1` and `SQLNode2` cluster nodes to monitor the SQL cluster instance `SQL_INSTANCE` with the virtual cluster name `SQLCLUSTER`. + +> [!NOTE] +> SQL cluster configuration in the custom JSON configuration file should always be specified within the "**hosts**" section. + +```json +"hosts": [ + { + "hostName": "SQLNode1", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "MSSQLSERVER" ] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "hostName": "SQLNode2", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "Dhcp" ] + } + }, + "otelCollectorYamlFiles": [] + } + ] +``` + +> [!NOTE] +> If the environment does not use SQL cluster instances, then update `enabled` flag to false for all SQL cluster instances in the "**hosts**" section to disable SQL monitoring. + +--- + +### Restart the Environment Watch Windows Service + +After updating the `environment-watch-configuration.json` file with the SQL cluster configuration, save the changes, restart the Environment Watch Windows service to apply the changes. This ensures that the service reads the updated configuration and begins monitoring the specified SQL cluster instances. + +Once the windows service has been restarted, verify that the SQL cluster instances are being monitored correctly by checking the Environment Watch discover, dashboards for relevant metrics, alerts. + +### Verification in Kibana + +- Navigate to Kibana Discover. +- Select `logs-*` Data View. +- Search for "The Environment Watch shared configuration object is not empty" which indicates that the EW windows service fetching values from the custom JSON configuration file successfully. + +![](/resources/sql-cluster-images/environment-watch-shared-settings-not-empty.png) + +- Search for "Processed SQL instance details" and "labels.IsProvidedByCustomConfiguration attribute" should be true indicating that the SQL instance details were processed through custom JSON configuration file successfully as shown below: + +![](/resources/sql-cluster-images/processed-sql-details-true.png) + +- Select `APM-*` Data View. +- Search for "SQL Metrics" to verify that SQL cluster instance metrics are being ingested. KQL query example: +- Example 1: +
 host.name : "SqlClusterNode" and sqlserver.user.connection.count: * and labels.sqlserver_computer_name : "SqlClusterName"
+ +![](/resources/sql-cluster-images/sql-cluster-distributed-metrics.png) + +- Example 2: +
 relsvr.agent.exists: * and host.name: "SqlClusterNode"
+ +![](/resources/sql-cluster-images/sql-cluster-primary-metrics.png) +- Navigate to the Kibana SQL Dashboards. +- Verify data is being populated for all Kibana Dashboards. +- Navigate to Kibana Overview/Alerts. +- Verify that there are no alerts triggered. + +--- + +## Troubleshooting +Refer to the [Troubleshooting Guide](../troubleshooting/custom-json-troubleshooting.md) to resolve any custom JSON SQL cluster configuration issues. \ No newline at end of file diff --git a/docs/troubleshooting/custom-json-troubleshooting.md b/docs/troubleshooting/custom-json-troubleshooting.md new file mode 100644 index 0000000..64c12c7 --- /dev/null +++ b/docs/troubleshooting/custom-json-troubleshooting.md @@ -0,0 +1,271 @@ +# Custom JSON Configuration Troubleshooting + +This document provides guidance for troubleshooting issues related to custom JSON configuration file in Relativity Server environments. + +## Common Issues + +If the log message “The Environment Watch shared configuration object is not empty” does not appear in Kibana, review the following potential causes: + +![](/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png) + + **1.1 The custom JSON configuration file is not placed in the correct BCP path.** + +Verify that the custom JSON configuration file is located in the correct BCP path. + +Base path: `BCPPath` + +Folder: `EnvironmentWatch` + +File name: `environment-watch-configuration.json` + +Environment Watch automatically reads this file and applies the defined monitoring rules to the relevant instances, products, and hosts. + +To identify the BCP path for the environment, execute the following SQL query against the '**EDDS**' database: + +```sql +SELECT +[TemporaryDirectory] +FROM [EDDS].[eddsdbo].[ResourceServer] AS rs WITH(NOLOCK) +INNER JOIN [EDDS].[eddsdbo].[ExtendedArtifact] AS ea WITH(NOLOCK) + ON ea.[ArtifactID] = rs.[ArtifactID] +INNER JOIN [EDDS].[eddsdbo].[Code] AS c WITH(NOLOCK) + ON c.[ArtifactID] = rs.[Type] +INNER JOIN [EDDS].[eddsdbo].[ResourceGroupSQLServers] AS rgss WITH(NOLOCK) + ON rgss.[SQLServerArtifactID] = rs.[ArtifactID] +INNER JOIN [EDDS].[eddsdbo].[ResourceGroup] AS rg WITH(NOLOCK) + ON rg.[ArtifactID] = rgss.[ResourceGroupArtifactID] +WHERE c.[Name] = 'SQL - Primary' +``` + +**1.2 Activate the BCP path if it is not active.** + +- Log in to the Relativity application. +- Navigate to **Servers**. +- Filter by `SQL - Primary` in the 'Type' column. + +![](/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png) + +- Ensure the BCP path is active and 'Visible in Dropdown' is set to 'Yes'. + +![](/resources/custom-json-troubleshooting-images/bcp-path-active.png) + +- Click on 'Edit' and toggle the **'Status'** to 'Inactive' and save it. +- Again, click on 'Edit' and toggle the **'Status'** to 'Active' and save it. +- Restart the Environment Watch Windows Service to apply the changes. + +**1.3 Check the custom JSON configuration file for syntax errors.** + +- Download the sample JSON file from the link below and compare it with your custom JSON configuration file to identify any syntax errors. + +[Sample Custom JSON Configuration File](/resources/custom-json-troubleshooting-images/environment-watch-configuration.json) + +- Restart the Environment Watch Windows Service to apply the changes. + +**1.4 Update the Environment Watch Windows Service to the latest version.** + +- Verify the installed version of the Environment Watch Windows Service. +- Ensure the service is updated to version 100.0.21 or later, as custom JSON configuration file support was introduced in this version. +- If the service is running an earlier version, upgrade it to the latest available version. + +> [!NOTE] +> Ensure the `enabled` flag is set to `true` in the custom JSON configuration file for the relevant monitoring section (hosts, instance, or installedProducts). +## Certificates + +Possible causes for the following alert in the Relativity application include: + +![](/resources/custom-json-troubleshooting-images/certificate-alert.png) + + **2.1 Configure the certificate thumbprint properly.** + +Run the following PowerShell command based on the certificate store location and name. For `LocalMachine` and `My`, use: + +> ```powershell +> Get-ChildItem Cert:\LocalMachine\My + + **2.2 Ensure the host name is correct for monitoring by Host or Installed Product.** + +- Run the following PowerShell command. + + > ```powershell +> hostname + +Ensure the `hostName` property in your configuration matches the output. Example: + +```json +"hosts": [ + { + "hostName": "SQL01", + "sources": { + "certificates": { + "enabled": true, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + }] + }, + "windowsServices": { + "enabled": true, + "include": [ "MSSQLSERVER" ] + } + }, + "otelCollectorYamlFiles": [] + }, + ] +``` + + **2.3 Avoid configuring certificate thumbprints in Monitoring by instance.** + + Example configuration for instance monitoring: + +```json +"instance": { + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "windowsServices": { + "enabled": true, + "include": [ + "WindowsAzureGuestAgent", + "mpssvc" + ] + } + } + } +``` + +## Windows Services + +If Windows services do not appear in the Kibana dashboard after configuring the custom JSON configuration file, review the following potential causes: + +![](/resources/custom-json-troubleshooting-images/windows-service-dashboard-json.png) + +**3.1 Include the Windows services configuration in the custom JSON configuration file.** +- Ensure the custom JSON configuration file contains the correct configuration for the Windows services to monitor. +- Place the configuration under the `windowsServices` section for the relevant hosts, instance, or installedProducts. + +Example: + +```json +"hosts": [ + { + "hostName": "SQL01", + "sources": { + "windowsServices": { + "enabled": true, + "include": [ "MSSQLSERVER", "AnotherService" ] + } + } + }, + ] +``` + +**3.2 Verify that the Windows services are running and exist on the host.** + +- Verify that the Windows services to be monitored are running and exist on the host machine. +- Use the Services management console (`services.msc`) or the following PowerShell command: + +> ```powershell +> Get-Service -Name MSSQLSERVER, AnotherService + +**3.3 Always include service name in the custom JSON configuration file** + +- Ensure that the correct service names (not display names) are used in the include section of the custom JSON configuration file. + +## SQL Cluster Instances + +**4.1 Ensure all instances/nodes in the SQL cluster are monitored.** + +- Include each node of the SQL cluster in the `hosts` section of the custom JSON configuration file with the correct `hostName`, `clusterVirtualName`, and `instanceName`. +- Set `sqlServers` -> `enabled` to `true` for each host entry. + +```json +"hosts": [ + { + "hostName": "SQLNode1", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "MSSQLSERVER" ] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "hostName": "SQLNode2", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "Dhcp" ] + } + }, + "otelCollectorYamlFiles": [] + } + ] +``` + +> [!NOTE] +> Always specify SQL cluster configuration within the "**hosts**" section of the custom JSON configuration file. + +**4.2 The log "Processed SQL instance details" is missing, and the "labels.IsProvidedByCustomConfiguration" attribute is not set to true in Kibana->Discover** + +- Refer to [Common Issues](#common-issues) section 1.1 to 1.4 to troubleshoot why the Environment Watch Windows service is not picking up the custom JSON configuration file changes. + +## Slack Notifications + +**5.1 Adjust Slack message frequency.** + +- Set the `messageIntervalSeconds` property in the Slack handler configuration to a value greater than or equal to 180 seconds to avoid rate limiting by Slack. +- Restart the Environment Watch Windows Service to apply the changes. + +**5.2 When Kibana Alert is triggered, unable to see: "Message successfully sent alert {AlertId} to Slack channel {SlackChannel}" log in Kibana Discover** + +- Pass the Kibana Alert ID (`{AlertId}`) and Slack Channel ID (`{SlackChannel}`). +- Address issues mentioned in [Common Issues](#common-issues) section 1.1 to 1.4. + +**5.3 Slack notifications are not being sent even though the alert is triggered in Kibana.** + +- Verify that the Slack handler is correctly configured in the `alertNotificationHandlers` section of the custom JSON configuration file. +- Ensure that the `accessToken`, `channel`, `enabled`, and `messageIntervalSeconds` properties are set correctly. +- See [Common Issues](#common-issues) section 1.1 to 1.4 to troubleshoot why the Environment Watch Windows service is not picking up the custom JSON configuration file changes. +- It can also occur if the Slack API token is invalid or does not have the necessary permissions to post messages to the specified channel. +- Verify the token and permissions. + +![](/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png) + +![](/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png) + +![](/resources/custom-json-troubleshooting-images/slack-notification.png) + diff --git a/resources/custom-json-images/bcp-path-custom-json-file-name.png b/resources/custom-json-images/bcp-path-custom-json-file-name.png new file mode 100644 index 0000000..fc2b9a2 Binary files /dev/null and b/resources/custom-json-images/bcp-path-custom-json-file-name.png differ diff --git a/resources/custom-json-images/certificate-dashboard-example.png b/resources/custom-json-images/certificate-dashboard-example.png new file mode 100644 index 0000000..9af9064 Binary files /dev/null and b/resources/custom-json-images/certificate-dashboard-example.png differ diff --git a/resources/custom-json-images/certificate-json-example.png b/resources/custom-json-images/certificate-json-example.png new file mode 100644 index 0000000..cc98b15 Binary files /dev/null and b/resources/custom-json-images/certificate-json-example.png differ diff --git a/resources/custom-json-images/environment-watch-shared-settings-not-empty-generic.png b/resources/custom-json-images/environment-watch-shared-settings-not-empty-generic.png new file mode 100644 index 0000000..12fff80 Binary files /dev/null and b/resources/custom-json-images/environment-watch-shared-settings-not-empty-generic.png differ diff --git a/resources/custom-json-images/sql-bcp-path-query.png b/resources/custom-json-images/sql-bcp-path-query.png new file mode 100644 index 0000000..0f19f21 Binary files /dev/null and b/resources/custom-json-images/sql-bcp-path-query.png differ diff --git a/resources/custom-json-images/windows-service-dashboard.png b/resources/custom-json-images/windows-service-dashboard.png new file mode 100644 index 0000000..c66dfcb Binary files /dev/null and b/resources/custom-json-images/windows-service-dashboard.png differ diff --git a/resources/custom-json-images/windows-service-json-example.png b/resources/custom-json-images/windows-service-json-example.png new file mode 100644 index 0000000..18f7aad Binary files /dev/null and b/resources/custom-json-images/windows-service-json-example.png differ diff --git a/resources/custom-json-troubleshooting-images/bcp-path-active.png b/resources/custom-json-troubleshooting-images/bcp-path-active.png new file mode 100644 index 0000000..020baac Binary files /dev/null and b/resources/custom-json-troubleshooting-images/bcp-path-active.png differ diff --git a/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png b/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png new file mode 100644 index 0000000..0120cfa Binary files /dev/null and b/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png differ diff --git a/resources/custom-json-troubleshooting-images/certificate-alert.png b/resources/custom-json-troubleshooting-images/certificate-alert.png new file mode 100644 index 0000000..c23b689 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/certificate-alert.png differ diff --git a/resources/custom-json-troubleshooting-images/custom-json-troubleshooting.md b/resources/custom-json-troubleshooting-images/custom-json-troubleshooting.md new file mode 100644 index 0000000..55c4163 --- /dev/null +++ b/resources/custom-json-troubleshooting-images/custom-json-troubleshooting.md @@ -0,0 +1,277 @@ +# Custom JSON Configuration Troubleshooting + +This document provides guidance for troubleshooting issues related to custom JSON configurations in Relativity Server environments. + +## Common Issues + +If the log message “The Environment Watch shared configuration object is not empty” does not appear in Kibana, review the following potential causes: + +![](/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png) + + **1.1 The custom JSON file is not placed in the correct BCP path.** + +Verify that the custom JSON file is located in the correct BCP path. + +Base path: `BCPPath` + +Folder: `EnvironmentWatch` + +File name: `environment-watch-configuration.json` + +Environment Watch automatically reads this file and applies the defined monitoring rules to the relevant instances, products, and hosts. + +To identify the BCP path for the environment, execute the following SQL query against the '**EDDS**' database: + +```sql +SELECT +[TemporaryDirectory] +FROM [EDDS].[eddsdbo].[ResourceServer] AS rs WITH(NOLOCK) +INNER JOIN [EDDS].[eddsdbo].[ExtendedArtifact] AS ea WITH(NOLOCK) + ON ea.[ArtifactID] = rs.[ArtifactID] +INNER JOIN [EDDS].[eddsdbo].[Code] AS c WITH(NOLOCK) + ON c.[ArtifactID] = rs.[Type] +INNER JOIN [EDDS].[eddsdbo].[ResourceGroupSQLServers] AS rgss WITH(NOLOCK) + ON rgss.[SQLServerArtifactID] = rs.[ArtifactID] +INNER JOIN [EDDS].[eddsdbo].[ResourceGroup] AS rg WITH(NOLOCK) + ON rg.[ArtifactID] = rgss.[ResourceGroupArtifactID] +WHERE c.[Name] = 'SQL - Primary' +``` + +![](/resources/custom-json-images/sql-bcp-path-query.png) + +**1.2 Activate the BCP path if it is not active.** + +- Log in to the Relativity application. +- Navigate to **Servers**. +- Filter by `SQL - Primary` in the 'Type' column. + +![](/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png) + +- Ensure the BCP path is active and 'Visible in Dropdown' is set to 'Yes'. + +![](/resources/custom-json-troubleshooting-images/bcp-path-active.png) + +- Click on 'Edit' and toggle the **'Status'** to 'Inactive' and save it. +- Again, click on 'Edit' and toggle the **'Status'** to 'Active' and save it. +- Restart the Environment Watch Windows Service to apply the changes. + +**1.3 Check the custom JSON file for syntax errors.** + +- Download the sample JSON file from the link below and compare it with your custom JSON file to identify any syntax errors. + +[Sample Custom JSON File](/resources/custom-json-troubleshooting-images/environment-watch-configuration.json) + +- Restart the Environment Watch Windows Service to apply the changes. + +**1.4 Update the Environment Watch Windows Service to the latest version.** + +- Verify the version of the Environment Watch Windows Service. +- Ensure it is updated to version **100.0.21** or later, as custom JSON support was introduced in this version. +- If it is not updated, upgrade the Environment Watch Windows Service to the latest version. + +> [!NOTE] +> Ensure the `enabled` flag is set to `true` in the custom JSON file for the relevant monitoring section (hosts, instance, or installedProducts). + +## Certificates + +Possible causes for the following alert in the Relativity application include: + +![](/resources/custom-json-troubleshooting-images/certificate-alert.png) + + **2.1 Configure the certificate thumbprint properly.** + +Run the following PowerShell command based on the certificate store location and name. For `LocalMachine` and `My`, use: + +> ```powershell +> Get-ChildItem Cert:\LocalMachine\My + + **2.2 Ensure the host name is correct for monitoring by Host or Installed Product.** + +- Run the following PowerShell command. + + > ```powershell +> hostname + +Ensure the `hostName` property in your configuration matches the output. Example: + +```json +"hosts": [ + { + "hostName": "SQL01", + "sources": { + "certificates": { + "enabled": true, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + }] + }, + "windowsServices": { + "enabled": true, + "include": [ "MSSQLSERVER" ] + } + }, + "otelCollectorYamlFiles": [] + }, + ] +``` + + **2.3 Avoid configuring certificate thumbprints in Monitoring by instance.** + + Example configuration for instance monitoring: + +```json +"instance": { + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "windowsServices": { + "enabled": true, + "include": [ + "WindowsAzureGuestAgent", + "mpssvc" + ] + } + } + } +``` + +## Windows Services + +If Windows services do not appear in the Kibana dashboard after configuring the custom JSON file, review the following potential causes: + +![](/resources/custom-json-troubleshooting-images/windows-service-dashboard-json.png) + +**3.1 Include the Windows services configuration in the custom JSON file.** + +- Ensure the custom JSON file contains the correct configuration for the Windows services to monitor. +- Place the configuration under the `windowsServices` section for the relevant hosts, instance, or installedProducts. + +Example: + +```json +"hosts": [ + { + "hostName": "SQL01", + "sources": { + "windowsServices": { + "enabled": true, + "include": [ "MSSQLSERVER", "AnotherService" ] + } + } + }, + ] +``` + +**3.2 Verify that the Windows services are running and exist on the host.** + +- Check that the Windows services you want to monitor are running and exist on the host machine. +- Use the Services management console (`services.msc`) or the following PowerShell command: + +> ```powershell +> Get-Service -Name MSSQLSERVER, AnotherService + +**3.3 Always include service name in the custom JSON** + +- Ensure you are using the correct service names (not display names) in the `include` section of the custom JSON configuration. + +## SQL Cluster Instances + +**4.1 Ensure all instances/nodes in the SQL cluster are monitored.** + +- Include each node of the SQL cluster in the `hosts` section of the custom JSON configuration with the correct `hostName`, `clusterVirtualName`, and `instanceName`. +- Set `sqlServers` -> `enabled` to `true` for each host entry. + +```json +"hosts": [ + { + "hostName": "SQLNode1", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "MSSQLSERVER" ] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "hostName": "SQLNode2", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "Dhcp" ] + } + }, + "otelCollectorYamlFiles": [] + } + ] +``` + +> [!NOTE] +> Always specify SQL cluster configuration within the "**hosts**" section of the custom JSON file. + +**4.2 The log "Processed SQL instance details" is missing, and the "labels.IsProvidedByCustomConfiguration" attribute is not set to true in Kibana->Discover** + +- Refer to [Common Issues](#common-issues) section 1.1 to 1.4 to troubleshoot why the Environment Watch Windows service is not picking up the custom JSON changes. + +## Slack Notifications + +**5.1 Adjust Slack message frequency.** + +- Set the `messageIntervalSeconds` property in the Slack handler configuration to a value greater than or equal to 180 seconds to avoid rate limiting by Slack. +- Restart the Environment Watch Windows Service to apply the changes. + +**5.2 Kibana Alert is triggered, still cannot see: "Message successfully sent alert {AlertId} to Slack channel {SlackChannel}" in Kibana Discover** + +- Pass the Kibana Alert ID (`{AlertId}`) and Slack Channel ID (`{SlackChannel}`). +- Address issues mentioned in [Common Issues](#common-issues) section 1.1 to 1.4. + +**5.3 Slack notifications are not being sent even though the alert is triggered in Kibana.** + +- Verify that the Slack handler is correctly configured in the `alertNotificationHandlers` section of the custom JSON file. +- Ensure that the `accessToken`, `channel`, `enabled`, and `messageIntervalSeconds` properties are set correctly. +- See [Common Issues](#common-issues) section 1.1 to 1.4 to troubleshoot why the Environment Watch Windows service is not picking up the custom JSON changes. +- It can also occur if the Slack API token is invalid or does not have the necessary permissions to post messages to the specified channel. +- Verify the token and permissions. + +![](/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png) + +![](/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png) + +![](/resources/custom-json-troubleshooting-images/slack-notification.png) + + + diff --git a/resources/custom-json-troubleshooting-images/environment-watch-configuration.json b/resources/custom-json-troubleshooting-images/environment-watch-configuration.json new file mode 100644 index 0000000..bb323fc --- /dev/null +++ b/resources/custom-json-troubleshooting-images/environment-watch-configuration.json @@ -0,0 +1,91 @@ +{ + "environmentWatchConfiguration": { + "monitoring": { + "instance": { + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "windowsServices": { + "enabled": false, + "include": [] + } + }, + "otelCollectorYamlFiles": [] + }, + "installedProducts": [ + { + "productName": "Agent", + "sources": { + "certificates": { + "enabled": true, + "include": [ + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "" + }, + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "" + } + ] + }, + "windowsServices": { + "enabled": true, + "include": ["Spooler"] + } + }, + "otelCollectorYamlFiles": [] + } + ], + "hosts": [ + { + "hostName": "emttest", + "sources": { + "certificates": { + "enabled": false, + "include": [ + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "" + }, + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "" + } + ] + }, + "sqlServers": { + "enabled": false, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER02", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "W3SVC" ] + } + }, + "otelCollectorYamlFiles": [] + } + ] + }, + "alertNotificationHandlers": { + "slack": { + "accessToken": "", + "acknowledgeAlertEnabled": false, + "channel": "", + "enabled": true, + "messageIntervalSeconds": 60 + } + } + } +} \ No newline at end of file diff --git a/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png b/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png new file mode 100644 index 0000000..ab18bef Binary files /dev/null and b/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png differ diff --git a/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png b/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png new file mode 100644 index 0000000..786ab04 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png differ diff --git a/resources/custom-json-troubleshooting-images/slack-notification.png b/resources/custom-json-troubleshooting-images/slack-notification.png new file mode 100644 index 0000000..9460000 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/slack-notification.png differ diff --git a/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png b/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png new file mode 100644 index 0000000..725bf13 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png differ diff --git a/resources/custom-json-troubleshooting-images/windows-service-dashboard-json.png b/resources/custom-json-troubleshooting-images/windows-service-dashboard-json.png new file mode 100644 index 0000000..7465963 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/windows-service-dashboard-json.png differ diff --git a/resources/slackalerts-images/ChannelId.png b/resources/slackalerts-images/ChannelId.png new file mode 100644 index 0000000..9dae6ed Binary files /dev/null and b/resources/slackalerts-images/ChannelId.png differ diff --git a/resources/slackalerts-images/CreateChannelVisibility.png b/resources/slackalerts-images/CreateChannelVisibility.png new file mode 100644 index 0000000..b8fb977 Binary files /dev/null and b/resources/slackalerts-images/CreateChannelVisibility.png differ diff --git a/resources/slackalerts-images/CreateSlackApp.png b/resources/slackalerts-images/CreateSlackApp.png new file mode 100644 index 0000000..380f030 Binary files /dev/null and b/resources/slackalerts-images/CreateSlackApp.png differ diff --git a/resources/slackalerts-images/NewChannel.png b/resources/slackalerts-images/NewChannel.png new file mode 100644 index 0000000..6efccae Binary files /dev/null and b/resources/slackalerts-images/NewChannel.png differ diff --git a/resources/slackalerts-images/OAuthToken.png b/resources/slackalerts-images/OAuthToken.png new file mode 100644 index 0000000..1356e38 Binary files /dev/null and b/resources/slackalerts-images/OAuthToken.png differ diff --git a/resources/slackalerts-images/PickAworkspace.png b/resources/slackalerts-images/PickAworkspace.png new file mode 100644 index 0000000..fc2680c Binary files /dev/null and b/resources/slackalerts-images/PickAworkspace.png differ diff --git a/resources/slackalerts-images/SlackNotification.png b/resources/slackalerts-images/SlackNotification.png new file mode 100644 index 0000000..77062bd Binary files /dev/null and b/resources/slackalerts-images/SlackNotification.png differ diff --git a/resources/slackalerts-images/createachannel.png b/resources/slackalerts-images/createachannel.png new file mode 100644 index 0000000..6a94dda Binary files /dev/null and b/resources/slackalerts-images/createachannel.png differ diff --git a/resources/sql-cluster-images/environment-watch-shared-settings-not-empty.png b/resources/sql-cluster-images/environment-watch-shared-settings-not-empty.png new file mode 100644 index 0000000..637c964 Binary files /dev/null and b/resources/sql-cluster-images/environment-watch-shared-settings-not-empty.png differ diff --git a/resources/sql-cluster-images/processed-sql-details-true.png b/resources/sql-cluster-images/processed-sql-details-true.png new file mode 100644 index 0000000..94e7d20 Binary files /dev/null and b/resources/sql-cluster-images/processed-sql-details-true.png differ diff --git a/resources/sql-cluster-images/sql-cluster-distributed-metrics.png b/resources/sql-cluster-images/sql-cluster-distributed-metrics.png new file mode 100644 index 0000000..5bf4b6b Binary files /dev/null and b/resources/sql-cluster-images/sql-cluster-distributed-metrics.png differ diff --git a/resources/sql-cluster-images/sql-cluster-primary-metrics.png b/resources/sql-cluster-images/sql-cluster-primary-metrics.png new file mode 100644 index 0000000..ba9f678 Binary files /dev/null and b/resources/sql-cluster-images/sql-cluster-primary-metrics.png differ