The following steps will guide you through identifying and resolving common issues. Follow the steps in sequence to troubleshoot effectively. If the issue persists even after completing these steps, consider reaching out to Support for further assistance.

Step 1: Verify whether SysLog events coming to OpsRamp gateway from SysLog server by enabling the flags

  1. Login to the gateway.
  2. Execute gcli command.
  3. Execute below command to enable flags for print additional logs:
    flag add syslog.log on 40
    loglevel set com.vistara.gateway.syslog.SyslogEventProcessor DEBUG 60
    loglevel set com.vistara.gateway.syslog.SyslogMessageHandler DEBUG 60
  4. Execute exit command to exit from gcli.
  5. Replace the SysLog server IP address and execute the below command:
    tail -100f /var/log/app/vprobe.log | grep “UDP : Ip : <Ipaddress>”
  6. Send the SysLog event to the gateway and verify whether the log message appears in the gateway after the above command.
  7. If you do not find any log messages, refer Step 2 and capture the packets.

Step 2: Verify whether SysLog events are coming to the OpsRamp gateway from the SysLog server by capturing the packets

  1. Launch another gateway console and execute the below tcpdump:
    tcpdump -i any "udp port 514" -s 3000 -w /tmp/syslog.pcap
  2. Send the SysLog event to the gateway and verify whether the packet count increased.
  3. If the packet count increases, then press Ctrl+C to stop capturing packets.
  4. Download the packets to your machine and open them in Wireshark to verify the details.
  5. If no packets are captured, or if the required packets are not captured, it is necessary to check the configurations on the SysLog server.
  6. If required packets are captured, then go to Step 3 and verify SysLog configuration on the gateway side.

Step 3: Verify SysLog Configuration Pushed to the OpsRamp Gateway

  1. Execute the gcli command.
  2. Execute the syslogcache print command.
    Sample Output: Syslog Cache Collection is written to file: /var/log/app/tmp/syslogcache-1720088523726.json
  3. Execute the exit command to exit from gcli.
  4. Display the output file using the following command:
    cat /var/log/app/tmp/syslogcache-1720088523726.json
  5. Verify that the configured rules and configurations are present in the file.Specifically, check IP address, Severity, Facility of the event and RegEx pattern in the rules. If incorrect, create or update the configuration profile/rules accordingly.

Step 4: Verify whether any rule is dropping the event

  1. Login to the gateway.
  2. Execute gcli command.
  3. Execute the below command to enable flags for print additional logs:
    flag add syslog.log on 40
    loglevel set com.vistara.gateway.syslog.SyslogEventProcessor DEBUG 60
    loglevel set com.vistara.gateway.syslog.SyslogMessageHandler DEBUG 60
  4. Execute exit command to exit from gcli.
  5. Execute the below command:
    tail -100f /var/log/app/vprobe.log
  6. If you find the following messages in the log, it indicates events are reaching the gateway but are being skipped. If alerts are needed for these events, create or update the configuration profile accordingly.
    SyslogEventProcessor#96: Skipped:Severities. IP : 10.212.0.7, severity: 5, Profile: ABC Test syslog profile

Step 5: Check Gateway Alert Throttling

  1. Understand that OpsRamp Gateways apply alert throttling to reduce alert noise and optimize processing.
  2. By default, the Gateway limits 4 alerts per 10 minutes for each unique combination of:
    • Device IP
    • Metric name
    • Component name
      (Collectively referred to as deviceip_metric_component key)
  3. Once the threshold is reached:
    • Additional alerts for that key are not sent to the OpsRamp cloud until the 10-minute interval resets.
    • After the interval resets, the next alert will be:
      • Appended with “Attenuated Alert:" in the alert subject
      • Marked as CRITICAL
  4. In addition to gateway-side throttling, the OpsRamp cloud processes only one alert per key per minute. If multiple alerts are received within the same minute, only one is retained.
  5. If your SysLog events are not appearing as alerts in the platform:
    • Review /var/log/app/vprobe.log to determine if throttling messages are logged.
    • Confirm whether multiple alerts were generated for the same key within a short duration.
  6. If needed, you can customize the SysLog monitor rule to define unique metric and component values, which helps distribute alerts across different keys and avoid hitting the throttling limit.
  7. There are no specific prerequisites to customizing these fields, and such changes do not affect the throttling behavior unless explicitly reconfigured with Support assistance.
  8. If behavior persists or seems abnormal, capture Gateway logs and contact OpsRamp Support.