Production Ingest Pipeline Degradation

Incident Report for Halcyon

Resolved

All systems are operational and working as expected.
Posted Oct 24, 2025 - 12:28 PDT

Update

All services are operational and we are monitoring for any further issues.
Posted Oct 24, 2025 - 11:40 PDT

Update

We have re-enabled all services, webhook sending, and the display of alerts/events should be processing again.
We are monitoring for any further issues.
Posted Oct 24, 2025 - 10:59 PDT

Update

We have tracked the cause of the problem and are investigating to prevent it from happening again in the near future. Sending webhooks have been temporarily disabled as we investigate, but will still be sent once we are fully operational.
Posted Oct 24, 2025 - 10:25 PDT

Update

Data is still being processed successfully, but there may be a delay in receiving alert/event webhooks or assets/alerts/events showing in the console or API.
Posted Oct 24, 2025 - 07:36 PDT

Monitoring

We are currently recovering. Some systems may still be slow or partially inoperable, such as updating webhooks, receiving webhooks, and alerts/events showing in the console.
Posted Oct 24, 2025 - 06:29 PDT

Update

Team is continuing to take actions to restore operations. Until then, event ingestion, webhooks/alerting, new device registration, and some API operations in our US region will be severally degraded.

Cloud detections for our ransomware protections, Atlas API, are still online.
Posted Oct 24, 2025 - 05:48 PDT

Update

We are continuing to work on a fix for this issue.
Posted Oct 24, 2025 - 01:51 PDT

Identified

Root cause has been identified and the team is working to address the problem.
Posted Oct 24, 2025 - 01:17 PDT

Investigating

Halcyon has identified performance degradation in the ingest processing pipeline and is investigating
Posted Oct 23, 2025 - 23:44 PDT
This incident affected: REST API, Management Console, and Webhooks.