Observability — Metrics Collection Design Spec¶
Historical 1.x proposal: This specification is not a public 2.0 feature contract. See the current architecture, release gates, and roadmap.
Date: 2026-07-26
Target Release: v1.2.0
Status: Draft
Priority: P1 (HIGH value, LOW effort)
Author: Architecture profile (Phase 1 research), mcp-server-jvm-build-tools
1. Overview / Motivation¶
The server currently has domain-specific instrumentation in several services but no unified metrics collection:
BuildPerformanceService— build profiling (per-build timing, phase breakdown, history).BuildCacheService— cache health analysis (configuration scoring).ToolAuthorizationService— audit logging for tool invocations.
What's missing: - System-wide metrics — tool call counts, error rates, request latencies, active connections — the standard SRE/DevOps signals that operators need to monitor health. - Prometheus exposition — an endpoint that Prometheus can scrape for alerting and dashboards. - Cross-cutting instrumentation — metrics that span all services without coupling them to Micrometer directly.
Micrometer is the de-facto standard for JVM observability. Because the server already uses Spring Boot 4.1.0 (which bundles Micrometer and Actuator), adding Prometheus metrics requires minimal effort — primarily configuration and annotation.
Key Design Goals¶
- Add Micrometer metrics — counters, timers, and gauges for tool calls, build operations, cache hits, errors, and latency.
- Expose via Spring Boot Actuator —
/actuator/prometheusendpoint for Prometheus scraping,/actuator/metricsfor ad-hoc inspection. - Zero-coupling instrumentation — use Micrometer annotations (
@Timed,@Counted) andMeterBinderbeans to avoid hard dependencies in service code. - Label strategy — rich enough for dashboards, safe enough to avoid PII/cardinality explosion.
- Integrate with existing services — wire into
BuildPerformanceService,BuildCacheService,BuildToolsService,ToolAuthorizationServicewithout invasive refactoring.
2. Design Decisions¶
Decision 1: Micrometer + Actuator + Prometheus¶
Chosen: Micrometer core + Micrometer registry Prometheus via Actuator.
Spring Boot 4.1.0 already includes: - micrometer-core (meter registry, @Timed, @Counted AOP). - spring-boot-starter-actuator (actuator endpoints).
We add: - micrometer-registry-prometheus (Prometheus format serialization).
| Component | Purpose |
|---|---|
MeterRegistry (auto-configured) | Central meter registry |
@Timed on service methods | Record duration + count per tool |
MeterBinder beans | JVM, system, and custom metrics not tied to a single service |
PrometheusMeterRegistry | Format metrics for Prometheus scrape |
/actuator/prometheus | Prometheus-compatible exposition endpoint |
This is the lowest-effort, highest-compatibility approach. No new dependencies beyond the Prometheus registry jar. No changes to the application's request flow.
Decision 2: What metrics to collect¶
| Metric | Type | Tags | Source |
|---|---|---|---|
buildtools.tool.calls | Counter | tool_name, tool_service, status (success/error) | Per-tool method via @Timed |
buildtools.tool.duration | Timer | tool_name, tool_service | Per-tool method via @Timed |
buildtools.build.duration | Timer | tool (maven/gradle/sbt), command, success | BuildToolsService.executeBuildCommand |
buildtools.build.total | Counter | tool, command, result (success/failure) | BuildToolsService |
buildtools.build.errors | Counter | tool, error_type | Output parser |
buildtools.cache.score | Gauge | tool, category=overall | Latest cache health score from BuildCacheService.analyzeCacheHealth (0-100); sbt can incorporate locally parsed hit/miss counts |
buildtools.auth.requests | Counter | auth_type (api_key/jwt/none), result (allowed/denied) | OAuthResourceServerFilter + ToolAuthorizationService |
buildtools.async.tasks | Gauge | status (running/queued/completed) | AsyncBuildService (task queue) |
buildtools.tools.registered | Gauge | — | Startup registration count |
jvm.* | Various | — | Micrometer's built-in JVM metrics |
process.* | Various | — | Micrometer's built-in process metrics |
Decision 3: Instrumentation approach — annotations + MeterBinder¶
Chosen: Hybrid approach.
@Timedannotations on existing@Toolmethods for tool-call metrics.- Requires
TimedAspectbean registration. - Each
@Toolmethod gets@Timed(value = "buildtools.tool.calls", extraTags = {"tool_service", "BuildToolsService"}). - MeterBinder beans for custom metrics that span multiple services or need programmatic construction (gauges for cache health scores and async task counts).
- Manual MeterRegistry injection only where
@Timeddoesn't suffice, such as async task queue metrics. Cache scores are registered byCacheMetricsCollectorthroughMeterBinder.
No Micrometer imports in service code except where MeterBinder is used. The @Timed annotation is a Micrometer annotation, but it's purely declarative — the aspect bean does the actual work.
Decision 4: Endpoint exposure — Actuator Prometheus¶
Chosen: Spring Boot Actuator's /actuator/prometheus endpoint.
management.endpoints.web.exposure.include=health,info,metrics,prometheus
management.endpoint.prometheus.enabled=true
management.endpoint.metrics.enabled=true
This is consistent with the existing health endpoints (/health, /health/ready, /health/live) already exposed by ServerCardController. No new controller is needed.
The metrics endpoint (/actuator/metrics) provides ad-hoc inspection without Prometheus, while prometheus (/actuator/prometheus) provides the full exposition format.
Decision 5: Label strategy — avoid PII and high cardinality¶
Allowed labels (safe): - tool_name — MCP tool name (e.g., execute_build_command). Bounded set (~30 values). - tool_service — service class name (e.g., BuildToolsService). Bounded set (~15 values). - tool — build tool name (maven, gradle, sbt). 3 values. - command — build command (clean, test, package). Bounded set (~20 values). - result — success/failure. 2 values. - status — completed/failed/cancelled. 3 values. - error_type — error category (compilation, test, timeout, auth, unknown). ~10 values. - auth_type — api_key, jwt, none. 3 values. - cache_category — build_cache, config_cache, incremental_compilation. ~5 values.
Forbidden labels (PII or high-cardinality risk): - project_dir — user-provided file paths vary infinitely. - user_id — no user model; not collected. - request_ip — IP addresses are high-cardinality PII. - command_raw — raw build commands contain project-specific paths. - error_message — full error messages contain file paths and source code.
Decision 6: Actuator security¶
The /actuator/prometheus and /actuator/metrics endpoints follow the same security model as existing health endpoints:
- In stdio mode (default): no HTTP surface, no issue.
- In HTTP mode (
--http): the Prometheus endpoint is NOT behind the OAuth resource-server filter (it lives outside/mcp/**). For production deployments, restrict access via the reverse proxy or set: and expose Prometheus through a separate management port:
3. Metric Definitions¶
3.1 Tool Call Metrics¶
// Applied to each @Tool method in every service
@Timed(
value = "buildtools.tool.calls",
extraTags = {"service", "BuildToolsService"},
histogram = true,
percentiles = {0.5, 0.95, 0.99},
Description = "Total tool invocations and duration"
)
Exposed Prometheus metrics:
# HELP buildtools_tool_calls_seconds Tool invocation duration
# TYPE buildtools_tool_calls_seconds histogram
buildtools_tool_calls_seconds_count{tool_name="execute_build_command",service="BuildToolsService",exception="none",} 142.0
buildtools_tool_calls_seconds_sum{tool_name="execute_build_command",service="BuildToolsService",exception="none",} 1234.5
# HELP buildtools_tool_calls_total Total tool invocations
# TYPE buildtools_tool_calls_total counter
buildtools_tool_calls_total{tool_name="check_dependency_version",service="DependencyService",status="success",} 87.0
buildtools_tool_calls_total{tool_name="check_dependency_version",service="DependencyService",status="error",} 3.0
3.2 Build Execution Metrics¶
// Inside BuildToolsService.executeBuildCommand()
MeterRegistry registry; // injected
Timer.Sample sample = Timer.start(registry);
try {
String result = tool.executeCommand(home, dir, command);
sample.stop(Timer.builder("buildtools.build.duration")
.tag("tool", tool.getName())
.tag("command", command.split(" ")[0])
.tag("success", "true")
.register(registry));
Counter.builder("buildtools.build.total")
.tag("tool", tool.getName())
.tag("command", command.split(" ")[0])
.tag("result", "success")
.register(registry).increment();
return result;
} catch (Exception e) {
sample.stop(Timer.builder("buildtools.build.duration")
.tag("tool", tool.getName())
.tag("command", command.split(" ")[0])
.tag("success", "false")
.register(registry));
Counter.builder("buildtools.build.total")
.tag("tool", tool.getName())
.tag("command", command.split(" ")[0])
.tag("result", "failure")
.register(registry).increment();
throw e;
}
Exposed Prometheus metrics:
# HELP buildtools_build_duration_seconds Build execution duration
# TYPE buildtools_build_duration_seconds timer
buildtools_build_duration_seconds_count{tool="maven",command="clean",success="true"} 12.0
buildtools_build_duration_seconds_sum{tool="maven",command="clean",success="true"} 45.6
# HELP buildtools_build_total Total build executions
# TYPE buildtools_build_total counter
buildtools_build_total{tool="gradle",command="build",result="success"} 23.0
buildtools_build_total{tool="gradle",command="build",result="failure"} 1.0
3.3 Build Error Metrics¶
// Inside each BuildOutputParser.parse()
Counter.builder("buildtools.build.errors")
.tag("tool", toolName)
.tag("error_type", classifyError(error.message))
.register(registry)
.increment();
private String classifyError(String message) {
if (message.contains("cannot find symbol")) return "compilation";
if (message.contains("expected")) return "compilation";
if (message.contains("test") && (message.contains("FAILED") || message.contains("fail"))) return "test";
if (message.contains("timeout") || message.contains("Timed out")) return "timeout";
if (message.contains("permission") || message.contains("denied")) return "auth";
return "unknown";
}
Exposed Prometheus metrics:
# HELP buildtools_build_errors_total Build errors by type
# TYPE buildtools_build_errors_total counter
buildtools_build_errors_total{tool="maven",error_type="compilation"} 5.0
buildtools_build_errors_total{tool="gradle",error_type="test"} 12.0
buildtools_build_errors_total{tool="sbt",error_type="unknown"} 1.0
3.4 Cache Health Metrics — MeterBinder¶
@Component
public class CacheMetricsCollector implements MeterBinder {
private final BuildCacheService cacheService;
@Override
public void bindTo(MeterRegistry registry) {
// Cache health score, updated by analyzeCacheHealth
for (String tool : new String[] {"maven", "gradle", "sbt"}) {
Gauge.builder("buildtools.cache.score", cacheService,
svc -> svc.getLastScore(tool))
.tag("tool", tool)
.tag("category", "overall")
.description("Cache health score (0-100)")
.register(registry);
}
}
}
Exposed Prometheus metrics:
# HELP buildtools_cache_score Cache health score (0-100)
# TYPE buildtools_cache_score gauge
buildtools_cache_score{tool="maven",category="overall"} 55.0
buildtools.cache.hit.rate has been removed because no comparable per-tool hit rate was populated; the old gauge always reported 0.0. sbt cache scoring can incorporate hit/miss counts parsed from a local execution log, while Maven and Gradle scoring inspects configuration. Remove buildtools_cache_hit_rate queries, alerts, and dashboard panels. Use buildtools_cache_score{tool="...",category="overall"} to monitor the last cache health score, but do not interpret it as a hit rate. Dashboards that need actual hit rates must obtain measured hits and misses from a separate build execution source.
3.5 Auth Metrics¶
// Inside OAuthResourceServerFilter.doFilter()
Counter.builder("buildtools.auth.requests")
.tag("auth_type", token != null ? "jwt" : "none")
.tag("result", isValid ? "allowed" : "denied")
.register(registry)
.increment();
# HELP buildtools_auth_requests_total Authorization requests
# TYPE buildtools_auth_requests_total counter
buildtools_auth_requests_total{auth_type="jwt",result="allowed"} 45.0
buildtools_auth_requests_total{auth_type="jwt",result="denied"} 2.0
buildtools_auth_requests_total{auth_type="none",result="denied"} 10.0
3.6 Tool Registration¶
// During application startup, after MethodToolCallbackProvider registers tools
Gauge.builder("buildtools.tools.registered", () -> toolCount)
.description("Number of currently registered MCP tools")
.register(registry);
# HELP buildtools_tools_registered Number of currently registered MCP tools
# TYPE buildtools_tools_registered gauge
buildtools_tools_registered 28.0
4. Instrumentation Approach¶
4.1 @Timed on Tool Methods¶
Add @Timed annotation to each @Tool method across all 15+ service classes. The buildtools.tool.calls metric captures both count and duration for every tool.
Example (BuildToolsService):
@Timed(
value = "buildtools.tool.calls",
extraTags = {"service", "BuildToolsService"},
percentiles = {0.5, 0.95, 0.99},
description = "Tool invocation count and duration"
)
@Tool(name = "execute_build_command", description = "...")
public String executeBuildCommand(...) { ... }
How many annotations to add: ~30 (one per active @Tool method). This is mechanical and can be done in a single pass.
4.2 Manual Instrumentation (selectively)¶
Where @Timed doesn't expose enough labels (build success/failure, cache scores): inject MeterRegistry directly and use Timer.Sample and Counter.builder().
Manual instrumentation is needed in: - BuildToolsService.executeBuildCommand() — build-specific duration + status tags. - BuildToolsService.analyzeBuildOutput() — error-type classification. - BuildCacheService.analyzeCacheHealth() — cache score gauge (via MeterBinder). - OAuthResourceServerFilter.doFilter() — auth request counters. - AsyncBuildService — task queue depth gauges.
4.3 TimedAspect Registration¶
// New file: src/main/java/com/pragmatik/buildtools/observability/MetricsConfig.java
@Configuration
public class MetricsConfig {
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(registry);
}
}
Without this bean, @Timed annotations have no effect.
4.4 MeterBinder Beans¶
// New file: src/main/java/com/pragmatik/buildtools/observability/BuildMetricsBinder.java
@Component
public class BuildMetricsBinder implements MeterBinder {
// Bind cache, async task, and tool registration gauges
}
// New file: src/main/java/com/pragmatik/buildtools/observability/JvmMetricsBinder.java
// (Optional — Micrometer auto-configures JVM metrics with:
// management.metrics.jvm.enabled=true)
5. Prometheus Endpoint Configuration¶
5.1 application.properties¶
# ─── Actuator ──────────────────────────────────────────────
management.endpoints.web.base-path=/actuator
management.endpoints.web.exposure.include=health,info,metrics,prometheus
management.endpoint.health.show-details=when-authorized
management.endpoint.health.probes.enabled=true
management.endpoint.prometheus.enabled=true
management.endpoint.metrics.enabled=true
# ─── Micrometer ────────────────────────────────────────────
management.metrics.export.prometheus.enabled=true
management.metrics.tags.application=${spring.application.name:mcp-server-jvm-build-tools}
# JVM metrics enabled by default via spring-boot-starter-actuator
management.metrics.jvm.enabled=true
# HTTP request metrics (optional)
management.metrics.web.server.auto-time-requests=false
5.2 Security¶
The /actuator/prometheus endpoint is accessible without authentication by default. For production deployments:
# Option A: Exclude from public exposure, expose on management port
management.endpoints.web.exposure.exclude=prometheus
management.server.port=8081
# Option B: Keep on main port but protect via IP allowlist or token
# (Configure at the reverse proxy level)
5.3 Sample Prometheus Scrape Config¶
scrape_configs:
- job_name: 'mcp-server-jvm-build-tools'
scrape_interval: 15s
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ['localhost:8080']
# In production with management port:
# targets: ['localhost:8081']
6. Integration with Existing Services¶
6.1 BuildPerformanceService¶
The existing BuildPerformanceService already collects per-build timing data and persists it in .buildtools/history/. The new metrics are complementary, not redundant:
| Metrics Type | BuildPerformanceService | Micrometer/Prometheus |
|---|---|---|
| Per-build duration | ✅ Full detail, persisted | ✅ Histogram, aggregated |
| Phase breakdown | ✅ Yes, persisted | ❌ Not collected (too detailed) |
| Trend analysis | ✅ Cross-build trends | ❌ Prometheus doesn't store long-term |
| Real-time dashboards | ❌ | ✅ Prometheus + Grafana |
| Alert triggers | ❌ | ✅ Prometheus alert rules |
| Historical data > 30d | ✅ Keeps last 20 builds | ❌ Prometheus retention is storage-based |
Integration: BuildPerformanceService.profileBuild() increments the buildtools.build.total counter. No other changes needed.
6.2 BuildCacheService¶
The existing BuildCacheService calculates a cache health score (0-100) per tool. The BuildMetricsBinder registers a Gauge that reads the last score from BuildCacheService. The cache service needs a trivial getter addition:
// In BuildCacheService
private final Map<String, Double> lastScores = new ConcurrentHashMap<>();
public double getLastScore(String tool) {
return lastScores.getOrDefault(tool, 0.0);
}
// Update in analyzeCacheHealth():
lastScores.put(tool.getName(), (double) score);
6.3 BuildToolsService¶
The main build execution service is the primary target for instrumentation: - @Timed on all 6 @Tool methods. - Manual Timer.Sample in executeBuildCommand() for the buildtools.build.duration timer with build-specific tags. - Manual Counter in error handling for buildtools.build.errors.
6.4 ToolAuthorizationService / OAuthResourceServerFilter¶
Auth metrics are collected at the filter level (OAuthResourceServerFilter) rather than in the service, because the service only sees already-authenticated requests. The filter sees all requests, including unauthenticated ones.
7. Test Scenarios¶
Scenario 1: Prometheus endpoint returns valid data¶
GET /actuator/prometheus
Expect: 200 OK
Content-Type: text/plain; version=0.0.4
Body contains: buildtools_tool_calls_seconds_count{...,} 0.0
Body contains: jvm_memory_used_bytes{area="heap",...}
Body contains: JVM, system, and Micrometer metrics in Prometheus text format
Scenario 2: Tool call counter increments¶
Step 1: GET /actuator/prometheus → note buildtools_tool_calls_total value
Step 2: Call get_build_tool_version(buildToolName="maven")
Step 3: GET /actuator/prometheus → buildtools_tool_calls_total incremented by 1
Scenario 3: Build duration timer records¶
Step 1: Call execute_build_command(projectDir="/tmp/maven-project", command="clean compile")
Step 2: GET /actuator/prometheus
Expect: buildtools_build_duration_seconds_count{tool="maven",command="clean",success="true"} ≥ 1
buildtools_build_duration_seconds_sum{tool="maven",command="clean",success="true"} > 0
Scenario 4: Error counter on build failure¶
Step 1: Call execute_build_command with a project that has compilation errors
Step 2: GET /actuator/prometheus
Expect: buildtools_build_errors_total{tool="maven",error_type="compilation"} incremented
buildtools_build_total{tool="maven",command="compile",result="failure"} incremented
Scenario 5: Cache health gauge¶
Step 1: Call analyze_cache_health(projectDir="/tmp/gradle-project")
Step 2: GET /actuator/prometheus
Expect: buildtools_cache_score{tool="gradle",category="overall"} presents the cache score
Scenario 6: Auth metrics recorded¶
Step 1: POST /mcp/tools/call (with valid bearer token)
Step 2: POST /mcp/tools/call (with invalid bearer token)
Step 3: GET /actuator/prometheus
Expect: buildtools_auth_requests_total{auth_type="jwt",result="allowed"} incremented once
buildtools_auth_requests_total{auth_type="jwt",result="denied"} incremented once
Scenario 7: Tool registration gauge¶
Step 1: Start server
Step 2: GET /actuator/prometheus
Expect: buildtools_tools_registered equals the number of @Tool methods (~28)
Scenario 8: Metrics endpoint (ad-hoc inspection)¶
GET /actuator/metrics/buildtools.tool.calls
Expect: 200 OK
Content-Type: application/json
Body: {"name": "buildtools.tool.calls", "measurements": [...], "availableTags": [...]}
Scenario 9: Management port isolation¶
Step 1: Configure management.server.port=8081
Step 2: Start server
Step 3: GET /actuator/prometheus on port 8080 → 404 (or not routed)
Step 4: GET /actuator/prometheus on port 8081 → 200 with metrics
Scenario 10: No PII in labels¶
Step 1: Execute builds with various project paths
Step 2: Examine /actuator/prometheus output
Expect: No project_dir, user_id, request_ip, command_raw, or error_message labels present
8. Implementation Plan¶
Files to Create¶
| File | Purpose |
|---|---|
src/main/java/com/pragmatik/buildtools/observability/MetricsConfig.java | @Configuration — TimedAspect bean + MeterRegistryCustomizer |
src/main/java/com/pragmatik/buildtools/observability/BuildMetricsBinder.java | MeterBinder — cache, async, tool-registration gauges |
src/test/java/com/pragmatik/buildtools/observability/MetricsIntegrationTest.java | Prometheus endpoint integration test |
Files to Modify¶
| File | Change |
|---|---|
pom.xml | Add micrometer-registry-prometheus dependency |
src/main/resources/application.properties | Add actuator and Micrometer configuration |
BuildToolsService.java | Add @Timed to 6 @Tool methods; manual Timer/Counter in executeBuildCommand() |
DependencyService.java | Add @Timed to check_dependency_version |
PromptService.java | Add @Timed to 3 @Tool methods |
BuildResourceService.java | Add @Timed to 2 @Tool methods |
DependencyResourceService.java | Add @Timed to 2 @Tool methods |
ResourceTemplateService.java | Add @Timed to 2 @Tool methods |
SbtProjectService.java | Add @Timed to 3 @Tool methods |
BuildAuthService.java | Add @Timed to check_credential_status |
DependencyConflictService.java | Add @Timed to detect_dependency_conflicts |
JavaVersionService.java | Add @Timed to check_java_compatibility |
BuildPerformanceService.java | Add @Timed to 2 @Tool methods |
ToolAuthorizationService.java | Add @Timed to 4 @Tool methods |
BuildCacheService.java | Add @Timed to 2 @Tool methods; add getLastScore() |
OAuthResourceServerFilter.java | Add auth request counter |
docs/reference/tools.md | Document metrics endpoint and available metrics |
Dependencies to Add¶
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
No version needed — managed by Spring Boot 4.1.0 BOM.
9. Suggested Grafana Dashboard Metrics¶
| Panel | Metric | Type |
|---|---|---|
| Tool call rate (per tool) | rate(buildtools_tool_calls_total[5m]) | Bar chart |
| Tool call duration (p95) | histogram_quantile(0.95, rate(buildtools_tool_calls_seconds_bucket[5m])) | Time series |
| Build duration (p95, per tool) | histogram_quantile(0.95, rate(buildtools_build_duration_seconds_bucket[5m])) | Time series |
| Build success rate | rate(buildtools_build_total{result="success"}[5m]) / rate(buildtools_build_total[5m]) * 100 | Gauge |
| Error rate by type | rate(buildtools_build_errors_total[5m]) | Stacked area |
| Cache health score | buildtools_cache_score | Gauge |
| Auth request rate | rate(buildtools_auth_requests_total[5m]) | Time series |
| Active async tasks | buildtools_async_tasks | Gauge |
| JVM heap usage | jvm_memory_used_bytes{area="heap"} | Area |
| JVM GC pauses | rate(jvm_gc_pause_seconds_sum[5m]) | Time series |
10. Acceptance Criteria¶
- [ ]
/actuator/prometheusreturns valid Prometheus exposition format. - [ ]
/actuator/metrics/buildtools.tool.callsreturns the tool-call metric with tags. - [ ] All ~30
@Toolmethods are annotated with@Timed. - [ ] Build duration timer tracks
tool,command, andsuccesslabels. - [ ] Build errors counter classifies errors into
compilation,test,timeout,auth. - [ ] Cache health gauge is registered and updates on
analyze_cache_healthcalls. - [ ] Auth request counter works for both API key and JWT authentication.
- [ ] No PII or high-cardinality labels in any metric.
- [ ]
TimedAspectbean registered without errors. - [ ]
micrometer-registry-prometheusdependency added with no version conflicts. - [ ] Prometheus endpoint can be isolated on a management port.
- [ ] All 10 test scenarios pass.
- [ ] No breaking changes to existing service contracts.