【譯】保護 Consul 在特定設置中免受 RCE 風險的影響

2018年11月27日 Consul 團隊

介紹

我們最近注意到了一組惡意軟體，它們主要針對具有允許遠程執行代碼這一特定配置的 Consul nodes 。我們的社區成員也 (負責任地) 報告了此惡意軟體造成的事件, 並與我們合作, 在最近版本的 Consul 中添加了一個補丁, 以保護我們免受這種威脅。

這篇文章詳細介紹了此惡意軟體可能會如何影響用戶, 具體取決於他們的配置, 並概述了我們已經採取的步驟, 來提供支持1.2.4、1.1.1、1.0.8 和0.9.4 版本的補丁, 使舊版本的 Consul 在沒有重大版本升級的情況下也能很容易受到保護。

總結

如果您已將-enable-script-checks設置為 true", 或者正在運行 Consul 0.9.0 或更早版本, 並且Consul api 可通過網路訪問的介面使用, 則應執行以下操作。

補救步驟:

升級到下面鏈接的版本之一。
如果需要腳本檢查, 請將 Consul 代理上的 "-enable-script-checks" 更改為 "-enable-local-script-checks。
禁用 Consul 伺服器上的腳本檢查。
確保 Consul HTTP API 綁定到 loopback 介面, 而不是一個可公開訪問的介面。
如果尚未啟用, 則啟用 ACL 。

有關詳細信息, 請繼續閱讀下面的內容。

如何利用腳本檢查

腳本檢查是 Consul 可以執行的健康檢查的一種類型，用於確定目標服務的健康狀態。支持的檢查類型包括 HTTP、TCP、gRPC、Docker、Alias、TTL 和腳本檢查。檢查可以通過 Consul API 註冊, 也可以由 Consul 代理從檢查定義目錄中載入。

腳本檢查將在配置的時間間隔內執行 Consul 進程的任何命令或內聯腳本。Docker 檢查是相同的, 但通過 Docker daemon 的 API 在正在運行的 Docker 容器中執行腳本。默認情況下, 作為 Consul 0.9.0 的 Consul 安裝中禁用腳本 (和 Docker) 檢查, 儘管它們在許多情況下非常有用, 但存在潛在的安全風險。

Consul API 是用於與 Consul 代理交互的 HTTP 介面, 包括註冊上面所述的運行狀況檢查定義。在大多數情況下, 我們建議將此 API 綁定到 loopback 介面, 以防止意外暴露。我們提供了一個廣泛的 ACLl 系統, 可用於保護 API 交互, 包括註冊檢查。

使代理容易受到這種攻擊的必要條件是:

API 可通過網路訪問的介面使用。
腳本檢查已啟用。
ACL 被禁用或 ACL Token 被破壞。

在上述條件下, 攻擊者可以對遠程代理註冊具有惡意負載檢查。根據設計, 腳本檢查允許任意執行代碼, 因此允許通過公開的 API 啟用檢查的服務註冊會帶來 RCE (遠程代碼執行) 威脅。

Consul的早期設計假定HTTP API僅限於受信任的本地網路訪問，直到後來才清楚許多用戶由於各種原因不知道（或仍然選擇）公開它, 這促使在 0.9.0 版本中更改為在默認情況下禁用腳本檢查。

建議將啟用 ACL 作為最佳做法, 但即使在限制性 ACL 到位的情況下, 如果攻擊者用至少一個可以註冊服務的 ACL 損害主機，則相同的ACL Token 將允許攻擊者在任何公開的遠程節點上註冊腳本檢查，因為根據設計，ACL的範圍不限於單個代理。我們尚未發現惡意軟體有利用本地可用的 ACL Token 針對受 ACL 保護的群集, 但主動指出此風險希望使用者們注意。

已知的影響

上個月我們有幾份關於這個問題的報告, 並證實它正在變得更加普遍。我們要確保我們的 Consul 用戶社區意識到這一威脅, 並知道如何保護自己免受這種攻擊。

在每種已知情況下, 攻擊者都通過 Consul 之外的一些不相關漏洞獲得了對數據中心中一台主機的訪問許可權。然後, 攻擊者在網路上檢測到暴露的 Consul API (可能通過自動掃描), 並使用腳本檢查將惡意軟體傳播到其他伺服器。

修復

許多用戶對其基礎結構中的服務使用腳本檢查功能, 因此完全禁用腳本檢查將禁用他們所依賴的功能。

在本月初發布的領Consul 1.3.0 中, Consul 社區的一名成員提供了一個補丁, 添加了一個新的配置選項-- -enable-local-script-checks, 它允許註冊腳本檢查僅通過本地配置文件, 從而防止使用 HTTP API 註冊惡意檢查。我們建議所有需要腳本檢查的用戶切換到此選項, 並找到避免的方法, 具體取決於對代理的遠程 API 訪問。

除此之外，我們建議檢查 ACL 系統和 Consul API 正在偵聽的介面的使用。默認情況下腳本檢查仍然完全禁用。

考慮到通過多個主要版本更改升級到 1.3.0 (在那裡發布了此功能) 的挑戰, 我們支持-enable-local-script-checks " 選項, 並發布了多個修補版本的 Consul, 以確保我們的用戶可以以最小的干擾和努力來緩解這一威脅:

1.2.4

更改日誌鏈接
下載鏈接

1.1.1

更改日誌鏈接
下載鏈接

1.0.8

更改日誌鏈接
下載鏈接

0.9.4

更改日誌鏈接
下載鏈接

這些是該系列中的用戶操作版本的替代。如果您當前使用的是 0.9.0 之前的某個版本, 我們建議升級到 0.9.4。與往常一樣, 我們建議您在所有升級中遵循我們的升級指南。版本1.3.0 及以後已包含此功能。同一版本的企業版也已可用。

通常, 我們更喜歡看到用戶升級, 很少出現這樣的 backport 修復。由於此威脅的性質, 我們決定支持此修復程序, 以便用戶可以快速保護自己, 而不需要可能帶來重大更改的主要版本升級。

依賴通過 HTTP API 註冊腳本檢查的用戶可以繼續使用-enable-script-checks, 但應格外小心地啟用 ACL, 並確保在網路上無法訪問 API 偵聽器。我們建議儘快找到不需要遠程 HTTP API 訪問的替代部署模式。

在 Consul 伺服器上允許腳本檢查尤其危險, 特別是為了全局工具或UI訪問，在伺服器上公開API是很常見的。伺服器上的遠程可註冊腳本檢查將面臨從網路上的任何節點向攻擊者暴露包括 ACL Token 在內的所有 Consul 狀態的風險。我們正在考慮在未來版本的 Consul 中將在配置伺服器上的腳本檢查變成一個很難犯的錯誤。通常，Consul伺服器無論如何都不會運行其他工作負載，因此禁用它們上的腳本檢查不太可能會導致問題，但是請檢查您的配置已確認沒有啟用它們，因為可能有其他客戶端需要。

我們還在探索其他一些選項，以幫助通知用戶這種潛在的不安全配置，並考慮對未來版本的腳本檢查提供額外的保護。我們為腳本檢查和代理配置在文檔中添加了更加突出的警告，, 並在安全模型中添加了一個新的部分, 突出顯示了配置 Consul 的已知不安全的方式。為了使運行中的 Consul 更易於安全訪問, 我們在 Consul 1.4.0 中重新設計了ACL系統的介面,並且建議當前沒有使用 ACL 的 Consul 用戶使用這些介面。

-----------------------------------------------------------------------------------------

【原文】Protecting Consul from RCE Risk in Specific Configurations

NOV 27 2018 THE CONSUL TEAM

Introduction

We』ve recently become aware of a set of malware targeting Consul nodes with a specific configuration which allows remote code execution. Members of our community also (responsibly) reported incidents caused by this malware, and worked with us to include a patch in a recent version of Consul that protects from this threat in the wild.

This post details how this malware may affect users, depending on their configuration, as well as outlines the steps weve taken to backport a patch for versions 1.2.4, 1.1.1, 1.0.8, and 0.9.4 to make it easy for older versions of Consul to be secured without a major version upgrade.

Summary

You should take action if you have -enable-script-checks set to true, or are running Consul 0.9.0 or earlier, and the Consul API is available on an interface that can be accessed over the network.

Steps to remediate:

Upgrade to one of the versions linked below.
Change -enable-script-checks to -enable-local-script-checks on Consul agents if script checks are required.
Disable script checks on Consul servers.
Ensure the Consul HTTP API is bound to a loopback interface instead of one publicly accessible.
Enable ACLs if not already enabled.

For full details, continue reading below.

How Script Checks Can Be Exploited

Script checks are one type of health check Consul can execute to determine the health status of a target service. Supported check types are HTTP, TCP, gRPC, Docker, Alias, TTL, and script checks. Checks can be registered via the Consul API or loaded from a directory of check definitions by the Consul agent.

Script checks will execute any command or inline script by the Consul process at the configured interval. Docker checks are the same, but execute the script within a running Docker container via the Docker daemons API. Script (and Docker) checks are disabled by default in Consul installations as of Consul 0.9.0, and though they are useful for many cases present a potential security risk.

The Consul API is the HTTP interface for interacting with the Consul agent, including registering health check definitions as noted above. We recommend binding this API to the loopback interface in the majority of cases to prevent accidental exposure. We provide an extensive ACL system that can be utilized to protect API interactions, including registering checks.

The necessary conditions that make an agent vulnerable to this attack are:

The API is available on an interface that can be accessed over the network.
Script checks are enabled.
ACLs are disabled or an ACL token is compromised.

Given the above conditions, an attacker can register a check on a remote agent with a malicious payload. By design, script checks allow arbitrary code execution, so allowing service registration with checks enabled via an exposed API presents an RCE (remote code execution) threat.

Consuls early design assumed that HTTP API was restricted to trusted local network access, and only later did it become clear that many users were not aware (or still chose) to expose it for a variety of reasons, prompting the change to disable script checks by default in 0.9.0.

Enabling ACLs is recommended as a best practice, but even with restrictive ACLs in place, if an attacker compromises a host with at least one ACL present that can register a service, that same ACL token will allow the attacker to register with a script check on any remote node that is exposed, since ACLs by design are not scoped to a single agent. We are not aware of malware exploiting locally-available ACL tokens against an ACL-protected cluster yet, but wish to proactively point out this risk.

Known Impact

Weve had several reports of this in the last month and have confirmed that it is becoming more widespread. We want to ensure our community of Consul users is aware of this threat and knows how to protect themselves against this type of attack.

In each known case, the attacker gained access to one host in the datacenter via some unrelated vulnerability outside of Consul. The attacker then detected exposed Consul APIs on the network (likely though automated scanning) and used script checks to propagate the malware to other servers.

Remediation

Many users use the script check feature for services across their infrastructure, so disabling script checks altogether would disable functionality they rely on.

In Consul 1.3.0, released earlier this month, a member of the Consul community contributed a patch that adds a new configuration option, -enable-local-script-checks, which allows script checks to be registered only via local configuration files, thus preventing use of the HTTP API to register malicious checks. We recommend all users who need script checks switch to this option and find ways to avoid depending on remote API access to agents.

In addition to this, we recommend reviewing usage of the ACL system and interfaces that the Consul API is listening on. Script checks remain disabled entirely by default.

Given the challenge of upgrading through multiple major version changes to 1.3.0, where this feature was released, we have backported the -enable-local-script-checks option and released multiple patched versions of Consul to ensure our users can mitigate this threat with minimum disruption and effort:

1.2.4

Changelog link
Download link

1.1.1

Changelog link
Download link

1.0.8

Changelog link
Download link

0.9.4

Changelog link
Download link

These are drop-in replacements for users operating versions in that series. If you are currently using a version before 0.9.0, we recommend upgrading to 0.9.4 - there may be some breaking changes from earlier versions, but these are minimal compared with the much larger changes made in 1.0.0. As always, we recommend following our upgrade guidefor all upgrades. Version 1.3.0 and onwards already contain this feature. Enterprise binaries for the same versions are also available.

In general, we prefer to see users upgrade, and rarely backport fixes like this. Because of the nature of this threat, we decided to backport this fix so that users could protect themselves quickly, without needing a major version upgrade that might potentially introduce breaking changes.

Users that rely on registering script checks via the HTTP API can continue to use -enable-script-checks, but should take extra care to enable ACLs and to ensure the API listener is not accessible on the network. We recommend finding alternative deployment patterns where remote HTTP API access is not required as soon as possible.

It is especially dangerous to allow script checks on Consul servers, especially since its common to expose the API on servers for global tooling or UI access. Remotely registerable script checks on servers risk exposing all Consul state including ACL tokens to the attacker from any node on the network. We are considering making it a hard error to configure script checks on servers in future versions of Consul. Generally Consul servers tend not to run other workloads anyway so disabling script checks on them isnt likely to cause a problem, but please check your configuration didnt enable them just because other clients need to.

We are also exploring several other options to help notify users of this potentially insecure configuration, as well as considering additional protections for script checks for future releases. We added more prominent warnings to our documentation for script checks and agent configuration as well as adding a new section to our security modelhighlighting known-insecure ways to configure Consul. As part of making running Consul securely more accessible, we redesigned the interfaces to our ACL system in Consul 1.4.0 and recommend usage of that for users not currently using ACLs with Consul.