Skip to content

Task 1-2: DLQ Alerting

Phase: 1 - Observability Priority: High Module: batch Depends on: task-1-1 Reference: docs/BountyHunter-Backend/details/feature-batch-async-processing/SPEC.md

Background

JMSHandleDlqMessage hiện chỉ log. Không có proactive alerting khi DLQ accumulate messages. Errors có thể tích lũy âm thầm.

Tasks

File: batch/jms/dlq/JMSHandleDlqMessage.java (hoặc thêm vào BatchQueueMonitorService)

1. Enhanced DLQ logging

DI Note: alertingService must be injected into JMSHandleDlqMessage (or whichever class hosts this method). Add it as a constructor-injected field using @RequiredArgsConstructor: 

private final AlertingService alertingService;
AlertingService should be a Spring @Service / @Component bean (e.g., wrapping a Slack webhook client or PagerDuty SDK). Required imports: - org.springframework.jms.annotation.JmsListener - org.springframework.messaging.handler.annotation.Header - org.springframework.jms.support.JmsHeaders

@JmsListener(
    destination = "${queue.dlq:DLQ}",
    containerFactory = "jmsListenerContainerFactory"
)
public void handleDlqMessage(
    String messageJson,
    @Header(JmsHeaders.DESTINATION) String originalDestination,
    @Header(value = JmsHeaders.REDELIVERED, required = false) Boolean redelivered
) {
    log.error("[DLQ] Dead letter received: originalDestination={}, redelivered={}, message={}",
        originalDestination, redelivered, messageJson);

    // Alert to monitoring (e.g., Slack webhook, PagerDuty)
    alertingService.sendAlert(
        "[DLQ ALERT] Dead letter in queue: " + originalDestination,
        messageJson
    );
}

2. Periodic DLQ depth check

File: batch/service/BatchQueueMonitorService.java

DI Note: This method is added to BatchQueueMonitorService (created in task-1-1). meterRegistry and getQueueDepth() are already available in that class. No additional injection is needed beyond what task-1-1 established.

@Scheduled(fixedDelay = 60000)
public void checkDlq() {
    Long dlqDepth = getQueueDepth("DLQ");
    if (dlqDepth != null && dlqDepth > 0) {
        log.error("[DLQ_ALERT] Dead letter queue has {} messages - action required!", dlqDepth);
        meterRegistry.gauge("jms.dlq.depth", dlqDepth);
    }
}

Verification / Acceptance Criteria

  • [ ] JMSHandleDlqMessage listener consumes a manually published message on DLQ and logs [DLQ] Dead letter received with correct originalDestination
  • [ ] alertingService.sendAlert(...) is called once per DLQ message (verify via mock/spy in unit test)
  • [ ] checkDlq() scheduler fires every 60 s and logs [DLQ_ALERT] only when DLQ depth > 0
  • [ ] Micrometer gauge jms.dlq.depth is registered and readable via /actuator/metrics/jms.dlq.depth
  • [ ] No NPE or uncaught exception when DLQ is empty (depth = 0)

Files to Modify

  • batch/src/main/java/com/figpop/batch/jms/dlq/JMSHandleDlqMessage.java
  • batch/src/main/java/com/figpop/batch/service/BatchQueueMonitorService.java