Cloudflare has revealed that a nation-state actor hacked into the companyâs self-hosted Atlassian server in November 2023, but the attack was stopped by the internal team within a few days of access.
The hack, which used stolen tokens and credentials, was able to access âsome documentation and a limited amount of source codeâ before being thwarted.
âOn Thanksgiving Day, November 23, 2023, Cloudflare detected the threat actor on our self-hosted Atlassian server,â Cloudflare said in a blog post. âOur security team immediately began an investigation and cut off the threat actorâs accessâ.
After investigating internally and blocking the malicious access, Cloudflare commissioned CrowdStrike for an independent investigation, which concluded that the hack had no impact on any Cloudflare systems or data.
Attacker gained access via the Okta compromise
The attack started with the actor gaining access to a few Cloudflare credentials through a recent Okta breach, the companyâs identity and access management (IAM) vendor.
âWe were the victim of a compromise of Oktaâs systems which resulted in a threat actor gaining access to a set of credentials. These credentials were meant to all be rotated,â Cloudflare said. âUnfortunately, we failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise.â
Among the stolen credentials was a Moveworks service token that granted remote access to Atlassian systems. Other compromises included a Smartsheet account with administrative access to the Atlassian Jira instance, a Bitbucket service account with access to the Cloudflare source code management system, and an AWS environment with âno access to the global network and no customer or sensitive data.â
âFrom November 14 to 17, the threat actor did reconnaissance and then accessed our internal wiki (which uses Atlassian Confluence) and our bug database (Atlassian Jira),â Cloudflare added. âThey then returned on November 22 and established persistent access to our Atlassian server using ScriptRunner for Jira, gained access to our source code management system (which uses Atlassian Bitbucket), and tried, unsuccessfully, to access a console server that had access to the data center that Cloudflare had not yet put into production in São Paulo, Brazil.â
The company added that the incident was in no way an error on the part of Atlassian, AWS, Moveworks, or Smartsheet, and happened because it failed to rotate the stolen credentials assuming they were unused.
Remediation helped by zero-trust efforts
Cloudflare said it was able to completely contain and remove the infection owing to its adoption of a zero-trust architecture.
âBecause of our access controls, firewall rules, and use of hard security keys enforced using our own Zero Trust tools, the threat actorâs ability to move laterally was limited,â the company said. âNo services were implicated, and no changes were made to our global network systems or configuration.â
Acknowledging the attackâs intention for establishing persistence and fearing overlooked persistence, Cloudflare resorted to a comprehensive remediation approach with additional proactive steps for future attacks.
âEven though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian products (Jira, Confluence, and Bitbucket),â the Cloudflare blog said.
Checking for outdated software packages, user accounts that might have been created, unused active employee accounts, and secrets left in Jira tickets or source code, were a few other remediation efforts Cloudflare made.