Sentry 監控 - Snuba 資料中臺本地開發環境配置實戰

語言: CN / TW / HK

克隆倉庫

分別克隆 getsentry/sentry 與  getsentry/snuba

git clone https://github.com/getsentry/sentry.git
git clone https://github.com/getsentry/snuba.git

安裝系統依賴(以 Mac 為例)

Xcode CLI tools

xcode-select --install

Brewfile

進入 sentry 資料夾,你會看到一個  Brewfile 檔案:

cd sentry

Brewfile

# required to run devservices
cask 'docker'

brew 'pyenv'

# required for pyenv's python-build
brew 'openssl'
brew 'readline'

# required for yarn test -u
brew 'watchman'

# required to build some of sentry's dependencies
brew 'pkgconfig'
brew 'libxslt'
brew 'libxmlsec1'
brew 'geoip'

# Currently needed because on Big Sur there's no wheel for it
brew 'librdkafka'

# direnv isn't defined here, because we have it configured to check for a bootstrapped environment.
# If it's installed in the early steps of the setup process, it just leads to confusion.
# brew 'direnv'

tap 'homebrew/cask'

# required for acceptance testing
cask 'chromedriver'

如果你本地已經安裝了 Docker Desktop 並且已經啟動,可以把  cask 'docker' 註釋掉。

接下來,執行:

brew bundle --verbose

如果你之前本地沒有 Docker Desktop ,則還需要手動啟動一下它:

open -g -a Docker.app

構建工具鏈

Sentry 依賴於  Python Wheels (包含二進位制擴充套件模組的包),官方為以下平臺分發:

  • Linux 相容  PEP-513 ( manylinux1 )

  • macOS 10.15 或更高版本

如果您的開發機器沒有執行上述系統之一,則需要安裝 Rust 工具鏈。按照  https://www.rust-lang.org/tools/install 上的說明安裝編譯器和相關工具。安裝後, Sentry 安裝程式將自動使用  Rust 構建所有二進位制模組,無需額外配置。

官方通常會跟蹤最新的穩定 Rust 版本,該版本每六週更新一次。因此,請確保通過偶爾執行來使您的  Rust 工具鏈保持最新:

rustup update stable

Python

Sentry 使用 pyenv 來安裝和管理 Python 版本。它是在您執行  brew bundle 時安裝的。

要安裝所需版本的 Python ,您需要執行以下命令。這將需要一段時間,因為您的計算機實際上正在編譯  Python

make setup-pyenv

這裡假設你是 Zsh 使用者。

如果您鍵入 which python ,您應該看到類似  $HOME/.pyenv/shims/python 而不是  /usr/bin/python 的內容。這是因為以下內容已新增到您的啟動指令碼中:

cat ~/.zprofile ,你會看到如下內容:

# MacPorts Installer addition on 2021-10-20_at_11:48:22: adding an appropriate PATH variable for use with MacPorts.
export PATH="/opt/local/bin:/opt/local/sbin:$PATH"
# Finished adapting your PATH environment variable for use with MacPorts.

# It is assumed that pyenv is installed via Brew, so this is all we need to do.
eval "$(pyenv init --path)"

虛擬環境

您現在已準備好建立 Python 虛擬環境。執行:

python -m venv .venv

並激活虛擬環境:

source .venv/bin/activate

如果一切正常,執行 which python 現在應該會導致類似  /Users/you/sentry/.venv/bin/python 的結果。

Snuba 配置實戰

啟動 Snuba 相關依賴項容器

cd ../sentry
git checkout master
git pull

source .venv/bin/activate
sentry devservices up --exclude=snuba

# 11:17:59 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
# 11:18:01 [INFO] sentry.plugins.github: apps-not-configured
# > Pulling image 'postgres:9.6-alpine'
# > Pulling image 'yandex/clickhouse-server:20.3.9.70'
# > Not starting container 'sentry_relay' because it should be started on-demand with devserver.
# > Creating 'sentry_redis' volume
# > Creating 'sentry_zookeeper_6' volume
# > Creating 'sentry_kafka_6' volume
# > Creating container 'sentry_redis'
# > Creating container 'sentry_zookeeper'
# > Creating container 'sentry_kafka'
# > Starting container 'sentry_redis' (listening: ('127.0.0.1', 6379))
# > Starting container 'sentry_kafka' (listening: ('127.0.0.1', 9092))
# > Starting container 'sentry_zookeeper'
# > Creating 'sentry_clickhouse' volume
# > Creating container 'sentry_clickhouse'
# > Creating 'sentry_postgres' volume
# > Creating 'sentry_wal2json' volume
# > Starting container 'sentry_clickhouse' (listening: ('127.0.0.1', 9000), ('127.0.0.1', 9009), ('127.0.0.1', 8123))
# > Creating container 'sentry_postgres'
# > Starting container 'sentry_postgres' (listening: ('127.0.0.1', 5432))

這將在 master 上獲取最新版本的  Sentry ,並調出所有  snuba 的依賴項。

Snuba 主要依賴  clickhousezookeeperkafkaredis 相關容器。

docker ps 檢視一下:

1149a6f6ff23   postgres:9.6-alpine                  "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   127.0.0.1:5432->5432/tcp                                                       sentry_postgres
a7f3af7d52bb   yandex/clickhouse-server:20.3.9.70   "/entrypoint.sh"         3 minutes ago   Up 3 minutes   127.0.0.1:8123->8123/tcp, 127.0.0.1:9000->9000/tcp, 127.0.0.1:9009->9009/tcp   sentry_clickhouse
68913ee15c43   confluentinc/cp-zookeeper:6.2.0      "/etc/confluent/dock…"   3 minutes ago   Up 3 minutes   2181/tcp, 2888/tcp, 3888/tcp                                                   sentry_zookeeper
5a248eb26ed3   confluentinc/cp-kafka:6.2.0          "/etc/confluent/dock…"   3 minutes ago   Up 3 minutes   127.0.0.1:9092->9092/tcp                                                       sentry_kafka
0573aff7b5af   redis:5.0-alpine                     "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   127.0.0.1:6379->6379/tcp                                                       sentry_redis

設定 Snuba 虛擬環境

cd snuba
make pyenv-setup
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip==21.1.3
make develop

檢視遷移列表

snuba migrations list
#  system
#  [ ]  0001_migrations
#
#  events
#  [ ]  0001_events_initial
#  [ ]  0002_events_onpremise_compatibility
#  [ ]  0003_errors
#  [ ]  0004_errors_onpremise_compatibility
#  [ ]  0005_events_tags_hash_map (blocking)
#  [ ]  0006_errors_tags_hash_map (blocking)
#  [ ]  0007_groupedmessages
#  [ ]  0008_groupassignees
#  [ ]  0009_errors_add_http_fields
#  [ ]  0010_groupedmessages_onpremise_compatibility (blocking)
#  [ ]  0011_rebuild_errors
#  [ ]  0012_errors_make_level_nullable
#  [ ]  0013_errors_add_hierarchical_hashes
#  [ ]  0014_backfill_errors (blocking)
#  [ ]  0015_truncate_events
#
#  transactions
#  [ ]  0001_transactions
#  [ ]  0002_transactions_onpremise_fix_orderby_and_partitionby (blocking)
#  [ ]  0003_transactions_onpremise_fix_columns (blocking)
#  [ ]  0004_transactions_add_tags_hash_map (blocking)
#  [ ]  0005_transactions_add_measurements
#  [ ]  0006_transactions_add_http_fields
#  [ ]  0007_transactions_add_discover_cols
#  [ ]  0008_transactions_add_timestamp_index
#  [ ]  0009_transactions_fix_title_and_message
#  [ ]  0010_transactions_nullable_trace_id
#  [ ]  0011_transactions_add_span_op_breakdowns
#  [ ]  0012_transactions_add_spans
#
#  discover
#  [ ]  0001_discover_merge_table
#  [ ]  0002_discover_add_deleted_tags_hash_map
#  [ ]  0003_discover_fix_user_column
#  [ ]  0004_discover_fix_title_and_message
#  [ ]  0005_discover_fix_transaction_name
#  [ ]  0006_discover_add_trace_id
#  [ ]  0007_discover_add_span_id
#
#  outcomes
#  [ ]  0001_outcomes
#  [ ]  0002_outcomes_remove_size_and_bytes
#  [ ]  0003_outcomes_add_category_and_quantity
#  [ ]  0004_outcomes_matview_additions (blocking)
#
#  metrics
#  [ ]  0001_metrics_buckets
#  [ ]  0002_metrics_sets
#  [ ]  0003_counters_to_buckets
#  [ ]  0004_metrics_counters
#  [ ]  0005_metrics_distributions_buckets
#  [ ]  0006_metrics_distributions
#  [ ]  0007_metrics_sets_granularity_10
#  [ ]  0008_metrics_counters_granularity_10
#  [ ]  0009_metrics_distributions_granularity_10
#  [ ]  0010_metrics_sets_granularity_1h
#  [ ]  0011_metrics_counters_granularity_1h
#  [ ]  0012_metrics_distributions_granularity_1h
#  [ ]  0013_metrics_sets_granularity_1d
#  [ ]  0014_metrics_counters_granularity_1d
#  [ ]  0015_metrics_distributions_granularity_1d
#
#  sessions
#  [ ]  0001_sessions
#  [ ]  0002_sessions_aggregates
#  [ ]  0003_sessions_matview

執行遷移

snuba migrations migrate --force
#  ......
#  2021-12-01 19:45:57,557 Running migration: 0014_metrics_counters_granularity_1d
#  2021-12-01 19:45:57,575 Finished: 0014_metrics_counters_granularity_1d
#  2021-12-01 19:45:57,589 Running migration: 0015_metrics_distributions_granularity_1d
#  2021-12-01 19:45:57,610 Finished: 0015_metrics_distributions_granularity_1d
#  2021-12-01 19:45:57,623 Running migration: 0001_sessions
#  2021-12-01 19:45:57,656 Finished: 0001_sessions
#  2021-12-01 19:45:57,669 Running migration: 0002_sessions_aggregates
#  2021-12-01 19:45:57,770 Finished: 0002_sessions_aggregates
#  2021-12-01 19:45:57,792 Running migration: 0003_sessions_matview
#  2021-12-01 19:45:57,849 Finished: 0003_sessions_matview
#  Finished running migrations

檢查遷移

進入 Clickhouse 容器:

docker exec -it sentry_clickhouse clickhouse-client
# 執行如下 `sql` 語句:
select count() from sentry_local

# ClickHouse client version 20.3.9.70 (official build).
# Connecting to localhost:9000 as user default.
# Connected to ClickHouse server version 20.3.9 revision 54433.

# a7f3af7d52bb :) select count() from sentry_local

# SELECT count()
# FROM sentry_local

# ┌─count()─┐
# │       0 │
# └─────────┘

# 1 rows in set. Elapsed: 0.008 sec. 

# a7f3af7d52bb :)

檢視相關實體資料集

snuba entities list

# Declared Entities:
# discover
# events
# groups
# groupassignee
# groupedmessage
# metrics_sets
# metrics_counters
# metrics_distributions
# outcomes
# outcomes_raw
# sessions
# org_sessions
# spans
# transactions
# discover_transactions
# discover_events

啟動開發伺服器

此命令將啟動 api 和所有  Snuba 消費者以從  Kafka 攝取資料:

snuba devserver

轉到 http://localhost:1218/events/snql ,你將會看到一個簡易的查詢  UI