Troubleshooting Notifications

The trigger configured for the message-sending resource might not have been satisfied in the time range you were looking for it. (The resource that sends the message could be an alarm, announcement subscription, event rule, connector, or contextual notification (alarm or event rule).)

For example, consider an alarm configured for a 90% threshold at a one-hour interval. Perhaps the most recent evaluation occurred before the compute instance exceeded the threshold.

How to diagnose

Review history of the message-sending resource and compare findings to the topic's published and delivered messages.

1. Note the time when the trigger condition occurred.

   View default metric charts for the resource to determine the time.

   For example, you might view metric charts for a Compute instance and find that it exceeded the threshold defined in the alarm at 10:01.

2. Find the related timestamp as recorded by the associated resource (alarm, event rule, or connector).

   • For an alarm: Look for relevant alarm state transitions close to the time of the trigger condition.

     Tip
     
     Assess alarms and messages using their unique identifiers. See Prevent Processing of Duplicate Items. To view the format used by alarm messages, see Message Format and Examples.

     View alarm history. A transition found around that time indicates that the alarm might have sent the message you're missing. An absence of transitions indicates that the alarm didn't send any messages. If you expected the alarm to transition, review its configuration.

   • For an event rule: Look for matching events that are close to the time of the trigger condition.

     View default metric charts for the event rule. See the Events Matched chart. A matching event around that time indicates that the event rule might have sent the message you're missing. An absence of matching events indicates that the event rule didn't send any messages. If you expected the event rule to detect a matching event, review its configuration.

   • For a connector: Look for written messages that are close to the time of the trigger condition.

     View default metric charts for the connector. See the Messages written to target chart. A written message around that time indicates that the connector might have sent the message you're missing. An absence of written messages indicates that the connector didn't send any messages. If you expected the connector to write a message, review its configuration.

3. In the subscription's parent topic, look for message publish and delivery times that are close to the related timestamp from the previous step.

   View the topic's default metric charts. Specifically, view the Published Messages Total Count and Delivered Messages Count metric charts. A published message that wasn't delivered could indicate an issue with the subscription's endpoint.

How to fix

You can remedy this situation for future trigger conditions. Update the trigger configuration of the message-sending resource so that the trigger is satisfied when you expect it to be.

For example, update an alarm to use a shorter interval.

Following are instructions for updating message-sending resources:

• Announcements: Updating an Announcement Subscription
• Alarms: Updating an Alarm
• Event rules: Editing an Events Rule
• Connectors: Updating a Connector

The message-sending resource might not have sent the message to Notifications. (The resource that sends the message could be an alarm, announcement subscription, event rule, connector, or contextual notification (alarm or event rule).)

For example, you might expect an email for an event, while the event rule was accidentally configured for a different event.

How to diagnose

Review history of the message-sending resource and compare findings to the topic's published and delivered messages.

1. Note the time when the trigger condition occurred.

   View default metric charts for the resource to determine the time.

   For example, you might view metric charts for a Compute instance and find that it exceeded the threshold defined in the alarm at 10:01.

2. Find the related timestamp as recorded by the associated resource (alarm, event rule, or connector).

   • For an alarm: Look for relevant alarm state transitions close to the time of the trigger condition.

     Tip
     
     Assess alarms and messages using their unique identifiers. See Prevent Processing of Duplicate Items. To view the format used by alarm messages, see Message Format and Examples.

     View alarm history. A transition found around that time indicates that the alarm might have sent the message you're missing. An absence of transitions indicates that the alarm didn't send any messages. If you expected the alarm to transition, review its configuration.

   • For an event rule: Look for matching events that are close to the time of the trigger condition.

     View default metric charts for the event rule. See the Events Matched chart. A matching event around that time indicates that the event rule might have sent the message you're missing. An absence of matching events indicates that the event rule didn't send any messages. If you expected the event rule to detect a matching event, review its configuration.

   • For a connector: Look for written messages that are close to the time of the trigger condition.

     View default metric charts for the connector. See the Messages written to target chart. A written message around that time indicates that the connector might have sent the message you're missing. An absence of written messages indicates that the connector didn't send any messages. If you expected the connector to write a message, review its configuration.

How to fix

You can remedy this situation for future trigger conditions. For example, update an event rule to match on the intended event.

Following are instructions for updating message-sending resources:

• Announcements: Updating an Announcement Subscription
• Alarms: Updating an Alarm
• Event rules: Editing an Events Rule
• Connectors: Updating a Connector

The subscription might be incorrectly configured.

For example, an email subscription's endpoint might not match the expected email address, or a Slack subscription's endpoint might not include the correct webhook.

One indicator of an incorrectly configured subscription is a published message that doesn't get delivered.

How to diagnose

Get the subscription's details and review configuration. For example, compare the endpoint of an email subscription to the expected email address.

Review history of the message-sending resource and compare findings to the topic's published and delivered messages.

1. Note the time when the trigger condition occurred.

   View default metric charts for the resource to determine the time.

   For example, you might view metric charts for a Compute instance and find that it exceeded the threshold defined in the alarm at 10:01.

2. Find the related timestamp as recorded by the associated resource (alarm, event rule, or connector).

   • For an alarm: Look for relevant alarm state transitions close to the time of the trigger condition.

     Tip
     
     Assess alarms and messages using their unique identifiers. See Prevent Processing of Duplicate Items. To view the format used by alarm messages, see Message Format and Examples.

     View alarm history. A transition found around that time indicates that the alarm might have sent the message you're missing. An absence of transitions indicates that the alarm didn't send any messages. If you expected the alarm to transition, review its configuration.

   • For an event rule: Look for matching events that are close to the time of the trigger condition.

     View default metric charts for the event rule. See the Events Matched chart. A matching event around that time indicates that the event rule might have sent the message you're missing. An absence of matching events indicates that the event rule didn't send any messages. If you expected the event rule to detect a matching event, review its configuration.

   • For a connector: Look for written messages that are close to the time of the trigger condition.

     View default metric charts for the connector. See the Messages written to target chart. A written message around that time indicates that the connector might have sent the message you're missing. An absence of written messages indicates that the connector didn't send any messages. If you expected the connector to write a message, review its configuration.

3. In the subscription's parent topic, look for message publish and delivery times that are close to the related timestamp from the previous step.

   View the topic's default metric charts. Specifically, view the Published Messages Total Count and Delivered Messages Count metric charts. A published message that wasn't delivered could indicate an issue with the subscription's endpoint.

How to fix

You can remedy this situation for future trigger conditions.

• If the subscription is pending, confirm it.
• If the subscription is incorrectly configured, recreate it with correct configuration. We recommend deleting the incorrectly configured subscription.

Notifications dropped the message received from an alarm, announcement subscription, event rule, connector, or contextual notification (alarm or event rule) that was intended for a subscription. This issue can occur when the subscription is pending or incorrectly configured.

How to diagnose

In the subscription's parent topic, look for dropped function messages. View the topic's default metric charts. Specifically, view the Failed Messages Count metric chart, and note the value of the metric dimension endpointType ("ORACLE_FUNCTIONS" for a dropped function message). When a dropped function message exists, the counter of this metric chart increments, showing "ORACLE_FUNCTIONS" for endpointType.

How to fix

You can remedy this situation for future trigger conditions.

• If the subscription is pending, confirm it.
• If the subscription is incorrectly configured, recreate it with correct configuration. We recommend deleting the incorrectly configured subscription.

For example, a Slack subscription is in pending status because of a lack of confirmation.

How to diagnose

Get the subscription's details to confirm active status. If you can't find the subscription, it might have been deleted.

How to fix

You can remedy this situation for future trigger conditions. Either confirm the subscription to activate it, or if you can't find it, recreate the subscription.

SMS messages might not be enabled for the message-sending resource. SMS subscriptions are enabled only for messages sent by the following Oracle Cloud Infrastructure services: Announcements, Monitoring, and Connector Hub. See Before You Begin (on the "Creating an SMS Subscription" page).

For example, consider an event rule configured to send messages to a topic. The topic contains an email subscription and an SMS subscription. However, SMS messages aren't enabled for the Events service. In this case, the SMS message is dropped.

How to diagnose

In the subscription's parent topic, look for dropped SMS messages. View the topic's default metric charts. Specifically, view the Failed Messages Count metric chart, and note the value of the metric dimension endpointType ("SMS" for a dropped SMS message). For example, if an unsupported resource sends an SMS message to a topic containing an SMS subscription, then the SMS message is dropped. The counter of this metric chart increments, showing "SMS" for endpointType.

How to fix

You can remedy this situation for future trigger conditions. Create a message-sending resource that is enabled for sending SMS messages:

• Alarms: Creating an Alarm
• Announcements: Creating an Announcement Subscription
• Connectors: Creating a Connector

The SMS message might be sent to or from an unsupported locale. International SMS capabilities are required if SMS messages come from a phone number in another country.

How to diagnose

Confirm that you have the ability to send and receive SMS messages to and from other countries. We continuously add support for more countries so that more users can receive SMS messages from local phone numbers. See SMS.

How to fix

You can remedy this situation for future trigger conditions. Get international SMS capabilities.

An email message might not be delivered if the email address is on a suppression list.

Reasons for suppression include bounce codes and user complaints. For more information, see Managing the Suppression List.

How to diagnose

In the subscription's parent topic, look for dropped email messages. View the topic's default metric charts. Specifically, view the Failed Messages Count metric chart, and note the value of the metric dimension endpointType ("EMAIL" for a dropped email message). For example, if the email address is on a suppression list, then the email message is dropped. The counter of this metric chart increments, showing "EMAIL" for endpointType.

How to fix

You can remedy this situation for future trigger conditions. See Deleting an Email Address from Suppression List. For help with avoiding suppression lists in the future, see Maintain a Positive Email Sender Reputation.

If your topic is in an Oracle Platform Services managed compartments (named "ManagedCompartmentForPaas"), then the Monitoring service might not have permissions to use it, and alarm messages sent to that topic might not be received.

How to diagnose

Get the topic's details to determine if its compartment is an Oracle Platform Services managed compartments (named "ManagedCompartmentForPaas").

How to fix

You can remedy this situation for future trigger conditions. For more details, including steps for resolution, see Alarm messages are not received in Oracle Platform Services managed compartments.

The message-sending resource might not have sent the message to Notifications. (A message-sending resource could be an alarm, announcement subscription, event rule, connector, or contextual notification (alarm or event rule).)

For example, you might expect an event rule to send a message to the configured topic because an event occurred. However, the event rule might be accidentally configured for a different event that didn't happen.

How to diagnose

Review history of the message-sending resource and compare findings to the topic's published and delivered messages.

1. Note the time when the trigger condition occurred.

   View default metric charts for the resource to determine the time.

   For example, you might view metric charts for a Compute instance and find that it exceeded the threshold defined in the alarm at 10:01.

2. Find the related timestamp as recorded by the associated resource (alarm, event rule, or connector).

   • For an alarm: Look for relevant alarm state transitions close to the time of the trigger condition.

     Tip
     
     Assess alarms and messages using their unique identifiers. See Prevent Processing of Duplicate Items. To view the format used by alarm messages, see Message Format and Examples.

     View alarm history. A transition found around that time indicates that the alarm might have sent the message you're missing. An absence of transitions indicates that the alarm didn't send any messages. If you expected the alarm to transition, review its configuration.

   • For an event rule: Look for matching events that are close to the time of the trigger condition.

     View default metric charts for the event rule. See the Events Matched chart. A matching event around that time indicates that the event rule might have sent the message you're missing. An absence of matching events indicates that the event rule didn't send any messages. If you expected the event rule to detect a matching event, review its configuration.

   • For a connector: Look for written messages that are close to the time of the trigger condition.

     View default metric charts for the connector. See the Messages written to target chart. A written message around that time indicates that the connector might have sent the message you're missing. An absence of written messages indicates that the connector didn't send any messages. If you expected the connector to write a message, review its configuration.

How to fix

You can remedy this situation for future trigger conditions. For example, update an event rule to match on the intended event.

Following are instructions for updating message-sending resources:

• Announcements: Updating an Announcement Subscription
• Alarms: Updating an Alarm
• Event rules: Editing an Events Rule
• Connectors: Updating a Connector

Notifications dropped the message received from an alarm, announcement subscription, event rule, connector, or contextual notification (alarm or event rule) that was intended for a function subscription. This issue can occur when the subscription is pending or incorrectly configured.

How to diagnose

In the subscription's parent topic, look for dropped function messages. View the topic's default metric charts. Specifically, view the Failed Messages Count metric chart, and note the value of the metric dimension endpointType ("ORACLE_FUNCTIONS" for a dropped function message). When a dropped function message exists, the counter of this metric chart increments, showing "ORACLE_FUNCTIONS" for endpointType.

How to fix

You can remedy this situation for future trigger conditions.

• If the subscription is pending, confirm it.
• If the subscription is incorrectly configured, recreate it with correct configuration. We recommend deleting the incorrectly configured subscription.

The function wasn't invoked even though Notifications delivered the message received from an alarm, announcement subscription, event rule, connector, or contextual notification (alarm or event rule).

How to diagnose

Note

The Notifications service has no information about a function after it's invoked.

If this is the first invocation, response might be delayed.

1. Confirm Notifications delivery: In the subscription's parent topic, confirm that Notifications delivered the message to the function. View the topic's default metric charts. Specifically, view the Delivered Messages Count metric chart, and note the value of the metric dimension endpointType ("ORACLE_FUNCTIONS" for a delivered function message). When a function message is delivered, the counter of this metric chart increments, showing "ORACLE_FUNCTIONS" for endpointType.
2. In the function, look for invocation and run times that are close to the time when the trigger condition occurred.
   • View the function's default metric charts. Specifically, view both the Invocations and Duration metric charts. An absence of data points close that timestamp indicates that the function wasn't invoked or run.
   • View the function's service logs.

How to fix

If the function was never invoked, then contact Support.

The function wasn't run even though it was invoked after Notifications delivered the message received from an alarm, announcement subscription, event rule, connector, or contextual notification (alarm or event rule).

How to diagnose

Note

The Notifications service has no information about a function after it's invoked.

If this is the first invocation, response might be delayed.

1. Confirm Notifications delivery: In the subscription's parent topic, confirm that Notifications delivered the message to the function. View the topic's default metric charts. Specifically, view the Delivered Messages Count metric chart, and note the value of the metric dimension endpointType ("ORACLE_FUNCTIONS" for a delivered function message). When a function message is delivered, the counter of this metric chart increments, showing "ORACLE_FUNCTIONS" for endpointType.
2. In the function, look for invocation and run times that are close to the time when the trigger condition occurred.
   • View the function's default metric charts. Specifically, view both the Invocations and Duration metric charts. A data point in Invocations that doesn't have a matching occurrence in Duration indicates that the function was invoked but not run.
   • View the function's service logs.

How to fix

See Troubleshooting OCI Functions.