Skip to content
Kronotop Kronotop

OpenTelemetry

Kronotop pushes the numeric INFO fields to an OpenTelemetry Collector over OTLP/HTTP as JSON. Only metrics are exported, no traces and no logs. The export is disabled by default. A Docker Compose stack and Grafana dashboards are in Ready-to-run stack.

Configuration

Section titled “Configuration”
ParameterTypeDefaultDescription
observability.otlp.enabledbooleanfalseEnables the OTLP metrics export task.
observability.otlp.endpointstring"http://127.0.0.1:4318"Base URL of the collector’s OTLP/HTTP receiver. Kronotop appends /v1/metrics.
observability.otlp.export_intervalint (seconds)10How often the metrics are pushed. Failed pushes are logged and retried on the next interval.
observability {
  otlp {
    enabled = true
    endpoint = "http://otel-collector:4318"
    export_interval = 10
  }
}

The endpoint is the base URL of the collector. Do not add /v1/metrics yourself. A failed push writes one WARN line to the log and the task tries again on the next interval. Kronotop keeps no queue of failed pushes.

Collector

Section titled “Collector”

The collector needs an OTLP/HTTP receiver and a Prometheus exporter. resource_to_telemetry_conversion must be enabled. Without it the resource attributes are not turned into Prometheus labels, and you cannot filter by cluster or member.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318


processors:
  batch:


exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    resource_to_telemetry_conversion:
      enabled: true


service:
  pipelines:
    metrics:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ prometheus ]

Point Prometheus at port 8889 of the collector.

Ready-to-run stack

Section titled “Ready-to-run stack”

docker/observability(opens in a new tab) starts a collector, Prometheus and Grafana with Docker Compose.

extras/observability(opens in a new tab) holds the Grafana dashboards as JSON. Import them into your own Grafana. Ready for production use.

Labels

Section titled “Labels”

Every export carries four resource attributes. With resource_to_telemetry_conversion the collector adds them to every metric as labels.

OTLP attributePrometheus labelValue
service.namejobAlways kronotop.
service.instance.idinstanceMember id.
service.versionservice_versionKronotop version.
kronotop.cluster_namekronotop_cluster_nameCluster name.

Metric naming

Section titled “Metric naming”

Kronotop sends the INFO field name with the prefix kronotop_. Counters are sent as cumulative, monotonic sums. Gauges are sent as plain gauges. All values are integers. Byte-sized fields carry the unit By.

The Prometheus exporter in the collector builds the final name from this:

  • Counters: the word total is removed from the field name, then _total is added at the end.
  • Byte values get _bytes at the end, before _total. If the field name already contains bytes, nothing is added.
  • Fields with the value -1, text fields and _human fields are not sent.

For example, total_allocated_memory becomes kronotop_allocated_memory_bytes_total and transaction_bytes_read becomes kronotop_transaction_bytes_read_total.

Querying

Section titled “Querying”
  • Counters start at zero on restart. Use rate() or increase(), not the raw value.
  • Gauges: use the raw value.
  • Filter by kronotop_cluster_name and instance.
# Client commands per second on one cluster
rate(kronotop_external_commands_processed_total{kronotop_cluster_name="development"}[1m])


# Heap allocation rate of one member
rate(kronotop_allocated_memory_bytes_total{instance="dbeb680500cc5259d25865277d304b45dac8e694"}[1m])


# Heap usage ratio
kronotop_heap_used_memory_bytes / kronotop_heap_max_memory_bytes


# Restarts in the last hour: uptime dropped
resets(kronotop_uptime_in_seconds{instance="dbeb680500cc5259d25865277d304b45dac8e694"}[1h])

Exported fields

Section titled “Exported fields”

See INFO for what each field means.

Server

Section titled “Server”
INFO fieldPrometheus nameType
uptime_in_secondskronotop_uptime_in_secondsgauge
process_cpu_time_mseckronotop_process_cpu_time_msec_totalcounter

Kronotop

Section titled “Kronotop”

stash_shards is sent only when stash.enabled is true.

INFO fieldPrometheus nameType
known_memberskronotop_known_membersgauge
alive_memberskronotop_alive_membersgauge
bucket_shardskronotop_bucket_shardsgauge
stash_shardskronotop_stash_shardsgauge
primary_shardskronotop_primary_shardsgauge
standby_shardskronotop_standby_shardsgauge

Clients

Section titled “Clients”

clients_in_multi is sent only when stash.enabled is true.

INFO fieldPrometheus nameType
connected_clientskronotop_connected_clientsgauge
clients_in_transactionkronotop_clients_in_transactiongauge
clients_in_multikronotop_clients_in_multigauge
snapshot_read_clientskronotop_snapshot_read_clientsgauge
resp2_clientskronotop_resp2_clientsgauge
resp3_clientskronotop_resp3_clientsgauge

Memory

Section titled “Memory”

non_heap_max_memory is usually -1, so it is usually missing.

INFO fieldPrometheus nameType
heap_used_memorykronotop_heap_used_memory_bytesgauge
heap_committed_memorykronotop_heap_committed_memory_bytesgauge
heap_max_memorykronotop_heap_max_memory_bytesgauge
non_heap_used_memorykronotop_non_heap_used_memory_bytesgauge
non_heap_committed_memorykronotop_non_heap_committed_memory_bytesgauge
non_heap_max_memorykronotop_non_heap_max_memory_bytesgauge
direct_buffer_countkronotop_direct_buffer_countgauge
direct_buffer_used_memorykronotop_direct_buffer_used_memory_bytesgauge
direct_buffer_total_capacitykronotop_direct_buffer_total_capacity_bytesgauge
mapped_buffer_countkronotop_mapped_buffer_countgauge
mapped_buffer_total_capacitykronotop_mapped_buffer_total_capacity_bytesgauge
netty_used_direct_memorykronotop_netty_used_direct_memory_bytesgauge
netty_used_heap_memorykronotop_netty_used_heap_memory_bytesgauge
total_allocated_memorykronotop_allocated_memory_bytes_totalcounter
gc_countkronotop_gc_count_totalcounter
gc_time_mseckronotop_gc_time_msec_totalcounter
gc_freed_memorykronotop_gc_freed_memory_bytes_totalcounter

Threads

Section titled “Threads”

Only platform threads are counted. Virtual threads are not included.

INFO fieldPrometheus nameType
thread_countkronotop_thread_countgauge
daemon_thread_countkronotop_daemon_thread_countgauge
peak_thread_countkronotop_peak_thread_countgauge
total_started_threadskronotop_started_threads_totalcounter

Traffic

Section titled “Traffic”

external_commands_succeeded and internal_commands_succeeded are not sent.

INFO fieldPrometheus nameType
external_total_commands_processedkronotop_external_commands_processed_totalcounter
external_commands_failedkronotop_external_commands_failed_totalcounter
external_read_byteskronotop_external_read_bytes_totalcounter
external_written_byteskronotop_external_written_bytes_totalcounter
internal_total_commands_processedkronotop_internal_commands_processed_totalcounter
internal_commands_failedkronotop_internal_commands_failed_totalcounter
internal_read_byteskronotop_internal_read_bytes_totalcounter
internal_written_byteskronotop_internal_written_bytes_totalcounter

Transactions

Section titled “Transactions”

transactions_closed_without_commit, avg_commit_latency_usec and avg_transaction_duration_usec are not sent.

INFO fieldPrometheus nameType
transactions_createdkronotop_transactions_created_totalcounter
transactions_activekronotop_transactions_activegauge
transactions_watchingkronotop_transactions_watchinggauge
oldest_active_transaction_useckronotop_oldest_active_transaction_usecgauge
transactions_committedkronotop_transactions_committed_totalcounter
transactions_conflictedkronotop_transactions_conflicted_totalcounter
transactions_commit_failedkronotop_transactions_commit_failed_totalcounter
transactions_rolled_backkronotop_transactions_rolled_back_totalcounter
transaction_readskronotop_transaction_reads_totalcounter
transaction_range_readskronotop_transaction_range_reads_totalcounter
transaction_writeskronotop_transaction_writes_totalcounter
transaction_deleteskronotop_transaction_deletes_totalcounter
transaction_range_deleteskronotop_transaction_range_deletes_totalcounter
transaction_mutationskronotop_transaction_mutations_totalcounter
transaction_bytes_readkronotop_transaction_bytes_read_totalcounter
transaction_bytes_writtenkronotop_transaction_bytes_written_totalcounter
max_commit_latency_useckronotop_max_commit_latency_usecgauge

Tasks

Section titled “Tasks”
INFO fieldPrometheus nameType
task_countkronotop_task_countgauge
running_taskskronotop_running_tasksgauge

Volume

Section titled “Volume”

The volumeN lines are not sent.

INFO fieldPrometheus nameType
volume_countkronotop_volume_countgauge

Bucket

Section titled “Bucket”

index_maintenance_last_run is not sent.

INFO fieldPrometheus nameType
plan_cache_sizekronotop_plan_cache_sizegauge
plan_cache_hitskronotop_plan_cache_hits_totalcounter
plan_cache_misseskronotop_plan_cache_misses_totalcounter
open_cursorskronotop_open_cursorsgauge
bucket_insertskronotop_bucket_inserts_totalcounter
bucket_querieskronotop_bucket_queries_totalcounter
bucket_updateskronotop_bucket_updates_totalcounter
bucket_deleteskronotop_bucket_deletes_totalcounter
bucket_advanceskronotop_bucket_advances_totalcounter
bucket_vector_searcheskronotop_bucket_vector_searches_totalcounter
bucket_purgeskronotop_bucket_purges_totalcounter
bucket_removeskronotop_bucket_removes_totalcounter
index_maintenance_workerskronotop_index_maintenance_workersgauge
index_maintenance_processed_entrieskronotop_index_maintenance_processed_entries_totalcounter
index_maintenance_retried_conflictskronotop_index_maintenance_retried_conflicts_totalcounter

Vector

Section titled “Vector”
INFO fieldPrometheus nameType
vector_indexeskronotop_vector_indexesgauge
vector_bytes_usedkronotop_vector_bytes_usedgauge
vector_on_disk_indexeskronotop_vector_on_disk_indexesgauge
vector_flush_queue_sizekronotop_vector_flush_queue_sizegauge
vector_flush_failureskronotop_vector_flush_failures_totalcounter
vector_sync_queue_sizekronotop_vector_sync_queue_sizegauge
vector_sync_failureskronotop_vector_sync_failures_totalcounter
vector_lost_graphskronotop_vector_lost_graphs_totalcounter
vector_restore_failureskronotop_vector_restore_failures_totalcounter

ZMap

Section titled “ZMap”
INFO fieldPrometheus nameType
zmap_setskronotop_zmap_sets_totalcounter
zmap_getskronotop_zmap_gets_totalcounter
zmap_range_readskronotop_zmap_range_reads_totalcounter
zmap_deleteskronotop_zmap_deletes_totalcounter
zmap_mutationskronotop_zmap_mutations_totalcounter
zmap_watcheskronotop_zmap_watchesgauge
zmap_watched_keyskronotop_zmap_watched_keysgauge
zmap_watch_waiterskronotop_zmap_watch_waitersgauge
zmap_watch_fireskronotop_zmap_watch_fires_totalcounter