diff --git a/docs/troubleshooting/custom-json-troubleshooting.md b/docs/troubleshooting/custom-json-troubleshooting.md new file mode 100644 index 00000000..be93b8de --- /dev/null +++ b/docs/troubleshooting/custom-json-troubleshooting.md @@ -0,0 +1,283 @@ +# Custom-JSON Troubleshooting + +This document provides guidance for troubleshooting issues related to custom JSON configurations in Relativity Server environments. + +## Common Issues + +If the log "The Environment Watch shared configuration object is not empty" does not appear in Kibana after configuring the custom JSON file, the potential causes are: + +![](/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png) + + **1.1 The custom JSON file is not placed in the correct BCP path.** + +Check the custom JSON file path: + +Base path: `BCPPath` + +Folder: `EnvironmentWatch` + +File name: `environment-watch-configuration.json` + +Environment Watch automatically reads this file and applies the defined monitoring rules to the relevant instances, products, and hosts. + +To identify the BCP path for the environment, execute the following SQL query against the '**EDDS**' database: + +```sql +SELECT +[TemporaryDirectory] +FROM [EDDS].[eddsdbo].[ResourceServer] AS rs WITH(NOLOCK) +INNER JOIN [EDDS].[eddsdbo].[ExtendedArtifact] AS ea WITH(NOLOCK) + ON ea.[ArtifactID] = rs.[ArtifactID] +INNER JOIN [EDDS].[eddsdbo].[Code] AS c WITH(NOLOCK) + ON c.[ArtifactID] = rs.[Type] +INNER JOIN [EDDS].[eddsdbo].[ResourceGroupSQLServers] AS rgss WITH(NOLOCK) + ON rgss.[SQLServerArtifactID] = rs.[ArtifactID] +INNER JOIN [EDDS].[eddsdbo].[ResourceGroup] AS rg WITH(NOLOCK) + ON rg.[ArtifactID] = rgss.[ResourceGroupArtifactID] +WHERE c.[Name] = 'SQL - Primary' +``` + +![](/resources/custom-json-images/sql-bcp-path-query.png) + +**1.2 The custom JSON file is in the correct BCP path, but the BCP path is not active.** + +Check if the BCP path is active: + +- Login to the Relativity application. +- Navigate to **Servers**. +- Filter by `SQL - Primary` in the 'Type' column. + +![](/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png) + +- Ensure that the BCP path is active and 'Visible in Dropdown' is set to 'Yes'. + +![](/resources/custom-json-troubleshooting-images/bcp-path-active.png) + +- Click on 'Edit' and toggle the **'Status'** to 'Inactive' and save it. +- Again, click on 'Edit' and toggle the **'Status'** to 'Active' and save it. +- Restart the Environment Watch Windows Service to apply the changes. + +**1.3 The custom JSON file has syntax errors.** + +Verify the custom JSON file for syntax errors: + +- Please download the sample JSON file from the link below and compare it with your custom JSON file to identify any syntax errors. + +[Sample Custom JSON File](/resources/custom-json-troubleshooting-images/environment-watch-configuration.json) + +- Restart the Environment Watch Windows Service to apply the changes. + +**1.4 The Environment Watch Windows Service is not running the latest version.** + +- Verify the version of the Environment Watch Windows Service. +- Ensure it is updated to version **100.0.21** or later, as custom JSON support was introduced in this version. +- If it is not updated, upgrade the Environment Watch Windows Service to the latest version. + +> [!NOTE] +> Ensure the `enabled` flag is set to `true` in the custom JSON file for the relevant monitoring section (hosts, instance, or installedProducts). + +## Certificates + +Possible causes for the following alert in the Relativity application include: + +![](/resources/custom-json-troubleshooting-images/certificate-alert.png) + + **1.1 Thumbprint for the certificate is not configured properly.** + +Depending on the Store Location and Store Name, run the following command on the host. For `LocalMachine` and `My`, use: + +> ```powershell +> Get-ChildItem Cert:\LocalMachine\My + + **1.2 The host name is not proper for the monitoring by Host or Installed Product.** + +- Run below powershell command to get host name. + + > ```powershell +> hostname + +Ensure the `hostName` property in your configuration matches the output. Example: + +```json +"hosts": [ + { + "hostName": "SQL01", + "sources": { + "certificates": { + "enabled": true, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + }] + }, + "windowsServices": { + "enabled": true, + "include": [ "MSSQLSERVER" ] + } + }, + "otelCollectorYamlFiles": [] + }, + ] +``` + + **1.3 Certificate thumbprint is unique and should not be configured in Monitoring by instance.** + + Example configuration for instance monitoring: + + +```json +"instance": { + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "windowsServices": { + "enabled": true, + "include": [ + "WindowsAzureGuestAgent", + "mpssvc" + ] + } + } + } +``` + +## Windows Services + +If Windows services do not appear in the Kibana dashboard after configuring the custom JSON file, the potential causes are: + +![](/resources/custom-json-troubleshooting-images/widows-service-dashboard-json.png) + +**1.1 The custom JSON file does not include the Windows services configuration.** + +- Ensure that the custom JSON file contains the correct configuration for the Windows services to monitor. +- The configuration should be under the `windowsServices` section for the relevant hosts, instance, or installedProducts. + +Example: + +```json +"hosts": [ + { + "hostName": "SQL01", + "sources": { + "windowsServices": { + "enabled": true, + "include": [ "MSSQLSERVER", "AnotherService" ] + } + } + }, + ] +``` + +**1.2 The Windows services are not running on the host.** + +- Verify that the Windows services you are trying to monitor are actually running on the host machine. +- You can check this by logging into the host machine and using the Services management console (`services.msc`) or by using the following PowerShell command: + +> ```powershell +> Get-Service -Name MSSQLSERVER, AnotherService + +**1.3 Always include service name in the custom JSON** + +- Ensure that you are using the correct service names (not display names) in the `include` section of the custom JSON configuration. + + +## SQL Cluster Instances + +**1.1 All instances/nodes in the SQL cluster are not monitored.** + +- Verify that each node of the SQL cluster is included in the `hosts` section of the custom JSON configuration with the correct `hostName`, `clusterVirtualName`, and `instanceName`. +- Ensure that `sqlServers` -> `enabled` is set to `true` for each host entry. + +```json +"hosts": [ + { + "hostName": "SQLNode1", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "MSSQLSERVER" ] + } + }, + "otelCollectorYamlFiles": [] + }, + { + "hostName": "SQLNode2", + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "sqlServers": { + "enabled": true, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "Dhcp" ] + } + }, + "otelCollectorYamlFiles": [] + } + ] +``` + +> [!NOTE] +> SQL cluster configuration in the custom JSON file should always be specified within the "**hosts**" section. + +**1.2 The log "Processed SQL instance details" is missing, and the "labels.IsProvidedByCustomConfiguration" attribute is not set to true in Kibana->Discover** + +- See [Common Issues](#common-issues) section 1.1 to 1.4 to troubleshoot why the Environment Watch Windows service is not picking up the custom JSON changes. + +## Slack Notifications + +**1.1 Slack messages are being sent too frequently and require to extend the interval.** + +- Ensure that the `messageIntervalSeconds` property in the Slack handler configuration is set to a value greater than or equal to 180 seconds to avoid rate limiting by Slack. +- Restart the Environment Watch Windows Service to apply the changes. + +**1.2 Kibana Alert is triggered, still cannot see: "Message successfully sent alert {AlertId} to Slack channel {SlackChannel}" in Kibana->Discover** + +- {AlertId}: Pass Kibana Alert ID. +- {SlackChannel}: Pass Slack Channel ID. +- Above issue can occur due to issues mentioned in [Common Issues](#common-issues) section 1.1 to 1.4. + +**1.3 Slack notifications are not being sent even though the alert is triggered in Kibana.** + +- Verify that the Slack handler is correctly configured in the `alertNotificationHandlers` section of the custom JSON file. +- Ensure that the `accessToken`, `channel`, `enabled`, and `messageIntervalSeconds` properties are set correctly. +- See [Common Issues](#common-issues) section 1.1 to 1.4 to troubleshoot why the Environment Watch Windows service is not picking up the custom JSON changes. +- It can also occur if the Slack API token is invalid or does not have the necessary permissions to post messages to the specified channel. Verify the token and permissions. + +![](/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png) + +![](/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png) + +![](/resources/custom-json-troubleshooting-images/slack-notification.png) + + + diff --git a/resources/custom-json-troubleshooting-images/bcp-path-active.png b/resources/custom-json-troubleshooting-images/bcp-path-active.png new file mode 100644 index 00000000..020baacd Binary files /dev/null and b/resources/custom-json-troubleshooting-images/bcp-path-active.png differ diff --git a/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png b/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png new file mode 100644 index 00000000..0120cfa7 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/bcp-path-relativity-ui.png differ diff --git a/resources/custom-json-troubleshooting-images/certificate-alert.png b/resources/custom-json-troubleshooting-images/certificate-alert.png new file mode 100644 index 00000000..c23b689c Binary files /dev/null and b/resources/custom-json-troubleshooting-images/certificate-alert.png differ diff --git a/resources/custom-json-troubleshooting-images/environment-watch-configuration.json b/resources/custom-json-troubleshooting-images/environment-watch-configuration.json new file mode 100644 index 00000000..bb323fc7 --- /dev/null +++ b/resources/custom-json-troubleshooting-images/environment-watch-configuration.json @@ -0,0 +1,91 @@ +{ + "environmentWatchConfiguration": { + "monitoring": { + "instance": { + "sources": { + "certificates": { + "enabled": false, + "include": [] + }, + "windowsServices": { + "enabled": false, + "include": [] + } + }, + "otelCollectorYamlFiles": [] + }, + "installedProducts": [ + { + "productName": "Agent", + "sources": { + "certificates": { + "enabled": true, + "include": [ + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "" + }, + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "" + } + ] + }, + "windowsServices": { + "enabled": true, + "include": ["Spooler"] + } + }, + "otelCollectorYamlFiles": [] + } + ], + "hosts": [ + { + "hostName": "emttest", + "sources": { + "certificates": { + "enabled": false, + "include": [ + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "" + }, + { + "storeName": "My", + "storeLocation": "LocalMachine", + "thumbprint": "" + } + ] + }, + "sqlServers": { + "enabled": false, + "include": [ + { + "clusterVirtualName": "SQLCLUSTER02", + "instanceName": "SQL_INSTANCE" + } + ] + }, + "windowsServices": { + "enabled": false, + "include": [ "W3SVC" ] + } + }, + "otelCollectorYamlFiles": [] + } + ] + }, + "alertNotificationHandlers": { + "slack": { + "accessToken": "", + "acknowledgeAlertEnabled": false, + "channel": "", + "enabled": true, + "messageIntervalSeconds": 60 + } + } + } +} \ No newline at end of file diff --git a/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png b/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png new file mode 100644 index 00000000..ab18bef7 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/environment-watch-shared-settings-not-empty.png differ diff --git a/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png b/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png new file mode 100644 index 00000000..786ab044 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/slack-message-in-kibana-discover.png differ diff --git a/resources/custom-json-troubleshooting-images/slack-notification.png b/resources/custom-json-troubleshooting-images/slack-notification.png new file mode 100644 index 00000000..9460000a Binary files /dev/null and b/resources/custom-json-troubleshooting-images/slack-notification.png differ diff --git a/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png b/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png new file mode 100644 index 00000000..725bf133 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/triggerred-alert-in-kibana.png differ diff --git a/resources/custom-json-troubleshooting-images/widows-service-dashboard-json.png b/resources/custom-json-troubleshooting-images/widows-service-dashboard-json.png new file mode 100644 index 00000000..74659639 Binary files /dev/null and b/resources/custom-json-troubleshooting-images/widows-service-dashboard-json.png differ