AOP實現系統告警

語言: CN / TW / HK

工作羣裏的消息怕過於安靜,又怕過於頻繁

一、業務背景

在開發的過程中會遇到各種各樣的開發問題,服務器宕機、網絡抖動、代碼本身的bug等等。針對代碼的bug,我們可以提前預支,通過發送告警信息來警示我們去幹預,儘早處理。

二、告警的方式

1、釘釘告警

通過在企業釘釘羣,添加羣機器人的方式,通過機器人向羣內發送報警信息。至於釘釘機器人怎麼創建,發送消息的api等等,請參考官方文檔

2、企業微信告警

同樣的套路,企業微信也是,在企業微信羣中,添加羣機器人。通過機器人發送告警信息。具體請看官方文檔

3、郵件告警

與上述不同的是,郵件是發送給個人的,當然也可以是批量發送,只實現了發送文本格式的方式,至於markdown格式,有待考察。郵件發送相對比較簡單,這裏就不展開贅述。

三、源碼解析

1、Alarm自定義註解

@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
public @interface Alarm {

/**
* 報警標題
*
* @return String
*/

String title() default "";

/**
* 發送報警格式:目前支持text,markdown
* @return
*/

MessageTye messageType() default MessageTye.TEXT;

/**
* 告警模板id
* @return
*/

String templateId() default "";

/**
* 成功是否通知:true-通知,false-不通知
* @return
*/

boolean successNotice() default false;
}

1.1、註解使用

@Alarm 標記在方法上使用,被標記的方法發生異常,會根據配置,讀取配置信息,發送異常堆棧信息。使用方法如下所示:

@Alarm(title = "某某業務告警", messageType = MessageTye.MARKDOWN, templateId = "errorTemp")

1.2、註解字段解析

  1. title

告警消息標題:可以定義為業務信息,如導師身份計算

  1. messageType

告警消息展示類型:目前支持text文本類型,markdown類型

  1. templateId

消息模板id:與配置文件中配置的模板id一致

  1. successNotice

正常情況是否也需要發送告警信息,默認值是fasle,表示不需要發送。當然,有些業務場景正常情況也需要發送,比如:支付出單通知等。

2、配置文件分析

2.1、釘釘配置文件

spring:
alarm:
dingtalk:
# 開啟釘釘發送告警
enabled: true
# 釘釘羣機器人唯一的token
token: xxxxxx
# 安全設置:加簽的密鑰
secret: xxxxxxx

2.2、企業微信配置文件

spring:
alarm:
wechat:
# 開啟企業微信告警
enabled: true
# 企業微信羣機器人唯一key
key: xxxxxdsf
# 被@人的手機號
to-user: 1314243

2.3、郵件配置文件

spring:
alarm:
mail:
enabled: true
smtpHost: [email protected]
smtpPort: 22
to: [email protected]
from: 132@qq.com
username: wsrf
password: xxx

2.4、自定義模板配置

spring:
alarm:
template:
# 開啟通過模板配置
enabled: true
# 配置模板來源為文件
source: FILE
# 配置模板數據
templates:
errorTemp:
templateId: errorTemp
templateName: 服務異常模板
templateContent: 這裏是配置模板的內容
  • spring:alarm:template:enabled ,Boolean類型,表示開啟告警消息使用模板發送。
  • spring:alarm:template:source ,模板來源,枚舉類:JDBC(數據庫)、FILE(配置文件)、MEMORY(內存),目前只支持FILE,其他兩種可自行擴展。
  • spring:alarm:template:templates
    errorTemp
    @Alarm
    

3、核心AOP分析

3.1、原理分析

3.2、自定義切面

@Aspect
@Slf4j
@RequiredArgsConstructor
public class AlarmAspect {
private final AlarmTemplateProvider alarmTemplateProvider;

private final static String ERROR_TEMPLATE = "\n\n<font color=\"#F37335\">異常信息:</font>\n" +
"```java\n" +
"#{[exception]}\n" +
"```\n";

private final static String TEXT_ERROR_TEMPLATE = "\n異常信息:\n" +
"#{[exception]}";

private final static String MARKDOWN_TITLE_TEMPLATE = "# 【#{[title]}】\n" +
"\n請求狀態:<font color=\"#{[stateColor]}\">#{[state]}</font>\n\n";

private final static String TEXT_TITLE_TEMPLATE = "【#{[title]}】\n" +
"請求狀態:#{[state]}\n";

@Pointcut("@annotation(alarm)")
public void alarmPointcut(Alarm alarm) {

}

@Around(value = "alarmPointcut(alarm)", argNames = "joinPoint,alarm")
public Object around(ProceedingJoinPoint joinPoint, Alarm alarm) throws Throwable {
Object result = joinPoint.proceed();
if (alarm.successNotice()) {
String templateId = alarm.templateId();
String fileTemplateContent = "";
if (Objects.nonNull(alarmTemplateProvider)) {
AlarmTemplate alarmTemplate = alarmTemplateProvider.loadingAlarmTemplate(templateId);
fileTemplateContent = alarmTemplate.getTemplateContent();
}
String templateContent = "";
MessageTye messageTye = alarm.messageType();
if (messageTye.equals(MessageTye.TEXT)) {
templateContent = TEXT_TITLE_TEMPLATE.concat(fileTemplateContent);
} else if (messageTye.equals(MessageTye.MARKDOWN)) {
templateContent = MARKDOWN_TITLE_TEMPLATE.concat(fileTemplateContent);
}
Map<String, Object> alarmParamMap = new HashMap<>();
alarmParamMap.put("title", alarm.title());
alarmParamMap.put("stateColor", "#45B649");
alarmParamMap.put("state", "成功");
sendAlarm(alarm, templateContent, alarmParamMap);
}
return result;
}


@AfterThrowing(pointcut = "alarmPointcut(alarm)", argNames = "joinPoint,alarm,e", throwing = "e")
public void doAfterThrow(JoinPoint joinPoint, Alarm alarm, Exception e) {
log.info("請求接口發生異常 : [{}]", e.getMessage());
String templateId = alarm.templateId();
// 加載模板中配置的內容,若有
String templateContent = "";
String fileTemplateContent = "";
if (Objects.nonNull(alarmTemplateProvider)) {
AlarmTemplate alarmTemplate = alarmTemplateProvider.loadingAlarmTemplate(templateId);
fileTemplateContent = alarmTemplate.getTemplateContent();
}
MessageTye messageTye = alarm.messageType();
if (messageTye.equals(MessageTye.TEXT)) {
templateContent = TEXT_TITLE_TEMPLATE.concat(fileTemplateContent).concat(TEXT_ERROR_TEMPLATE);
} else if (messageTye.equals(MessageTye.MARKDOWN)) {
templateContent = MARKDOWN_TITLE_TEMPLATE.concat(fileTemplateContent).concat(ERROR_TEMPLATE);
}
Map<String, Object> alarmParamMap = new HashMap<>();
alarmParamMap.put("title", alarm.title());
alarmParamMap.put("stateColor", "#FF4B2B");
alarmParamMap.put("state", "失敗");
alarmParamMap.put("exception", ExceptionUtil.stacktraceToString(e));
sendAlarm(alarm, templateContent, alarmParamMap);
}

private void sendAlarm(Alarm alarm, String templateContent, Map<String, Object> alarmParamMap) {
ExpressionParser parser = new SpelExpressionParser();
TemplateParserContext parserContext = new TemplateParserContext();
String message = parser.parseExpression(templateContent, parserContext).getValue(alarmParamMap, String.class);
MessageTye messageTye = alarm.messageType();
NotifyMessage notifyMessage = new NotifyMessage();
notifyMessage.setTitle(alarm.title());
notifyMessage.setMessageTye(messageTye);
notifyMessage.setMessage(message);
AlarmFactoryExecute.execute(notifyMessage);
}
}

4、模板提供器

4.1、AlarmTemplateProvider

定義一個抽象接口 AlarmTemplateProvider ,用於被具體的子類實現

public interface AlarmTemplateProvider {


/**
* 加載告警模板
*
* @param templateId 模板id
* @return AlarmTemplate
*/

AlarmTemplate loadingAlarmTemplate(String templateId);
}

4.2、BaseAlarmTemplateProvider

抽象類 BaseAlarmTemplateProvider 實現該抽象接口

public abstract class BaseAlarmTemplateProvider implements AlarmTemplateProvider {

@Override
public AlarmTemplate loadingAlarmTemplate(String templateId) {
if (StringUtils.isEmpty(templateId)) {
throw new AlarmException(400, "告警模板配置id不能為空");
}
return getAlarmTemplate(templateId);
}

/**
* 查詢告警模板
*
* @param templateId 模板id
* @return AlarmTemplate
*/

abstract AlarmTemplate getAlarmTemplate(String templateId);
}

4.3、YamlAlarmTemplateProvider

具體實現類 YamlAlarmTemplateProvider ,實現從配置文件中讀取模板,該類在項目啟動時,會被加載進spring的bean容器

@RequiredArgsConstructor
public class YamlAlarmTemplateProvider extends BaseAlarmTemplateProvider {

private final TemplateConfig templateConfig;

@Override
AlarmTemplate getAlarmTemplate(String templateId) {
Map<String, AlarmTemplate> configTemplates = templateConfig.getTemplates();
AlarmTemplate alarmTemplate = configTemplates.get(templateId);
if (ObjectUtils.isEmpty(alarmTemplate)) {
throw new AlarmException(400, "未發現告警配置模板");
}
return alarmTemplate;
}
}

4.4、MemoryAlarmTemplateProvider和JdbcAlarmTemplateProvider

抽象類 BaseAlarmTemplateProvider 還有其他兩個子類,分別是 MemoryAlarmTemplateProviderJdbcAlarmTemplateProvider 。但是這兩個子類暫時還未實現邏輯,後續可以自行擴展。

@RequiredArgsConstructor
public class MemoryAlarmTemplateProvider extends BaseAlarmTemplateProvider {

private final Function<String, AlarmTemplate> function;
@Override
AlarmTemplate getAlarmTemplate(String templateId) {
AlarmTemplate alarmTemplate = function.apply(templateId);
if (ObjectUtils.isEmpty(alarmTemplate)) {
throw new AlarmException(400, "未發現告警配置模板");
}
return alarmTemplate;
}
}
@RequiredArgsConstructor
public class JdbcAlarmTemplateProvider extends BaseAlarmTemplateProvider {

private final Function<String, AlarmTemplate> function;

@Override
AlarmTemplate getAlarmTemplate(String templateId) {
AlarmTemplate alarmTemplate = function.apply(templateId);
if (ObjectUtils.isEmpty(alarmTemplate)) {
throw new AlarmException(400, "未發現告警配置模板");
}
return alarmTemplate;
}
}

兩個類中都有Function<String, AlarmTemplate>接口,為函數式接口,可以供外部自行去實現邏輯。

5、告警發送

5.1、AlarmFactoryExecute

該類內部保存了一個容器,主要用於緩存真正的發送類

public class AlarmFactoryExecute {

private static List<AlarmWarnService> serviceList = new ArrayList<>();

public AlarmFactoryExecute(List<AlarmWarnService> alarmLogWarnServices) {
serviceList = alarmLogWarnServices;
}

public static void addAlarmLogWarnService(AlarmWarnService alarmLogWarnService) {
serviceList.add(alarmLogWarnService);
}

public static List<AlarmWarnService> getServiceList() {
return serviceList;
}

public static void execute(NotifyMessage notifyMessage) {
for (AlarmWarnService alarmWarnService : getServiceList()) {
alarmWarnService.send(notifyMessage);
}
}
}

5.2、AlarmWarnService

抽象接口,只提供一個發送的方法

public interface AlarmWarnService {

/**
* 發送信息
*
* @param notifyMessage message
*/

void send(NotifyMessage notifyMessage);

}

5.3、BaseWarnService

與抽象的模板提供器 AlarmTemplateProvider 一樣的套路,該接口有一個抽象的實現類 BaseWarnService ,該類對外暴露send方法,用於發送消息,內部用doSendMarkdown,doSendText方法實現具體的發送邏輯,當然具體發送邏輯還是得由其子類去實現。

@Slf4j
public abstract class BaseWarnService implements AlarmWarnService {

@Override
public void send(NotifyMessage notifyMessage) {
if (notifyMessage.getMessageTye().equals(MessageTye.TEXT)) {
CompletableFuture.runAsync(() -> {
try {
doSendText(notifyMessage.getMessage());
} catch (Exception e) {
log.error("send text warn message error", e);
}
});
} else if (notifyMessage.getMessageTye().equals(MessageTye.MARKDOWN)) {
CompletableFuture.runAsync(() -> {
try {
doSendMarkdown(notifyMessage.getTitle(), notifyMessage.getMessage());
} catch (Exception e) {
log.error("send markdown warn message error", e);
}
});
}
}

/**
* 發送Markdown消息
*
* @param title Markdown標題
* @param message Markdown消息
* @throws Exception 異常
*/

protected abstract void doSendMarkdown(String title, String message) throws Exception;

/**
* 發送文本消息
*
* @param message 文本消息
* @throws Exception 異常
*/

protected abstract void doSendText(String message) throws Exception;
}

5.4、DingTalkWarnService

主要實現了釘釘發送告警信息的邏輯

@Slf4j
public class DingTalkWarnService extends BaseWarnService {

private static final String ROBOT_SEND_URL = "https://oapi.dingtalk.com/robot/send?access_token=";
private final String token;

private final String secret;

public DingTalkWarnService(String token, String secret) {
this.token = token;
this.secret = secret;
}

public void sendRobotMessage(DingTalkSendRequest dingTalkSendRequest) throws Exception {
String json = JSONUtil.toJsonStr(dingTalkSendRequest);
String sign = getSign();
String body = HttpRequest.post(sign).contentType(ContentType.JSON.getValue()).body(json).execute().body();
log.info("釘釘機器人通知結果:{}", body);
}

/**
* 獲取簽名
*
* @return 返回簽名
*/

private String getSign() throws Exception {
long timestamp = System.currentTimeMillis();
String stringToSign = timestamp + "\n" + secret;
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
byte[] signData = mac.doFinal(stringToSign.getBytes(StandardCharsets.UTF_8));
return ROBOT_SEND_URL + token + "&timestamp=" + timestamp + "&sign=" + URLEncoder.encode(new String(Base64.getEncoder().encode(signData)), StandardCharsets.UTF_8.toString());
}

@Override
protected void doSendText(String message) throws Exception {
DingTalkSendRequest param = new DingTalkSendRequest();
param.setMsgtype(DingTalkSendMsgTypeEnum.TEXT.getType());
param.setText(new DingTalkSendRequest.Text(message));
sendRobotMessage(param);
}

@Override
protected void doSendMarkdown(String title, String message) throws Exception {
DingTalkSendRequest param = new DingTalkSendRequest();
param.setMsgtype(DingTalkSendMsgTypeEnum.MARKDOWN.getType());
DingTalkSendRequest.Markdown markdown = new DingTalkSendRequest.Markdown(title, message);
param.setMarkdown(markdown);
sendRobotMessage(param);
}
}

5.5、WorkWeXinWarnService

主要實現了發送企業微信告警信息的邏輯

@Slf4j
public class WorkWeXinWarnService extends BaseWarnService {
private static final String SEND_MESSAGE_URL = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=%s";
private final String key;

private final String toUser;

public WorkWeXinWarnService(String key, String toUser) {
this.key = key;
this.toUser = toUser;
}

private String createPostData(WorkWeXinSendMsgTypeEnum messageTye, String contentValue) {
WorkWeXinSendRequest wcd = new WorkWeXinSendRequest();
wcd.setMsgtype(messageTye.getType());
List<String> toUsers = Arrays.asList("@all");
if (StringUtils.isNotEmpty(toUser)) {
String[] split = toUser.split("\\|");
toUsers = Arrays.asList(split);
}
if (messageTye.equals(WorkWeXinSendMsgTypeEnum.TEXT)) {
WorkWeXinSendRequest.Text text = new WorkWeXinSendRequest.Text(contentValue, toUsers);
wcd.setText(text);
} else if (messageTye.equals(WorkWeXinSendMsgTypeEnum.MARKDOWN)) {
WorkWeXinSendRequest.Markdown markdown = new WorkWeXinSendRequest.Markdown(contentValue, toUsers);
wcd.setMarkdown(markdown);
}
return JSONUtil.toJsonStr(wcd);
}

@Override
protected void doSendText(String message) {
String data = createPostData(WorkWeXinSendMsgTypeEnum.TEXT, message);
String url = String.format(SEND_MESSAGE_URL, key);
String resp = HttpRequest.post(url).body(data).execute().body();
log.info("send work weixin message call [{}], param:{}, resp:{}", url, data, resp);
}

@Override
protected void doSendMarkdown(String title, String message) {
String data = createPostData(WorkWeXinSendMsgTypeEnum.MARKDOWN, message);
String url = String.format(SEND_MESSAGE_URL, key);
String resp = HttpRequest.post(url).body(data).execute().body();
log.info("send work weixin message call [{}], param:{}, resp:{}", url, data, resp);
}
}

5.6、MailWarnService

主要實現郵件告警邏輯

@Slf4j
public class MailWarnService extends BaseWarnService {

private final String smtpHost;

private final String smtpPort;

private final String to;

private final String from;

private final String username;

private final String password;

private Boolean ssl = true;

private Boolean debug = false;

public MailWarnService(String smtpHost, String smtpPort, String to, String from, String username, String password) {
this.smtpHost = smtpHost;
this.smtpPort = smtpPort;
this.to = to;
this.from = from;
this.username = username;
this.password = password;
}

public void setSsl(Boolean ssl) {
this.ssl = ssl;
}

public void setDebug(Boolean debug) {
this.debug = debug;
}

@Override
protected void doSendText(String message) throws Exception {
Properties props = new Properties();
props.setProperty("mail.smtp.auth", "true");
props.setProperty("mail.transport.protocol", "smtp");
props.setProperty("mail.smtp.host", smtpHost);
props.setProperty("mail.smtp.port", smtpPort);
props.put("mail.smtp.ssl.enable", true);
Session session = Session.getInstance(props);
session.setDebug(false);
MimeMessage msg = new MimeMessage(session);
msg.setFrom(new InternetAddress(from));
for (String toUser : to.split(",")) {
msg.setRecipient(MimeMessage.RecipientType.TO, new InternetAddress(toUser));
}
Map<String, String> map = JSONUtil.toBean(message, Map.class);
msg.setSubject(map.get("subject"), "UTF-8");
msg.setContent(map.get("content"), "text/html;charset=UTF-8");
msg.setSentDate(new Date());
Transport transport = session.getTransport();
transport.connect(username, password);
transport.sendMessage(msg, msg.getAllRecipients());
transport.close();
}

@Override
protected void doSendMarkdown(String title, String message) throws Exception {
log.warn("暫不支持發送Markdown郵件");
}
}

6、AlarmAutoConfiguration自動裝配類

運用了springboot自定義的starter,再 META-INF 包下的配置文件 spring.factories 下,配置上該類

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.seven.buttemsg.autoconfigure.AlarmAutoConfiguration

自動裝配類,用於裝載自定義的bean

@Slf4j
@Configuration
public class AlarmAutoConfiguration {

// 郵件相關配置裝載
@Configuration
@ConditionalOnProperty(prefix = MailConfig.PREFIX, name = "enabled", havingValue = "true")
@EnableConfigurationProperties(MailConfig.class)
static class MailWarnServiceMethod
{

@Bean
@ConditionalOnMissingBean(MailWarnService.class)
public MailWarnService mailWarnService(final MailConfig mailConfig)
{
MailWarnService mailWarnService = new MailWarnService(mailConfig.getSmtpHost(), mailConfig.getSmtpPort(), mailConfig.getTo(), mailConfig.getFrom(), mailConfig.getUsername(), mailConfig.getPassword());
mailWarnService.setSsl(mailConfig.getSsl());
mailWarnService.setDebug(mailConfig.getDebug());
AlarmFactoryExecute.addAlarmLogWarnService(mailWarnService);
return mailWarnService;
}
}

// 企業微信相關配置裝載
@Configuration
@ConditionalOnProperty(prefix = WorkWeXinConfig.PREFIX, name = "enabled", havingValue = "true")
@EnableConfigurationProperties(WorkWeXinConfig.class)
static class WorkWechatWarnServiceMethod
{

@Bean
@ConditionalOnMissingBean(MailWarnService.class)
public WorkWeXinWarnService workWechatWarnService(final WorkWeXinConfig workWeXinConfig)
{
return new WorkWeXinWarnService(workWeXinConfig.getKey(), workWeXinConfig.getToUser());
}

@Autowired
void setDataChangedListener(WorkWeXinWarnService workWeXinWarnService) {
AlarmFactoryExecute.addAlarmLogWarnService(workWeXinWarnService);
}
}

// 釘釘相關配置裝載
@Configuration
@ConditionalOnProperty(prefix = DingTalkConfig.PREFIX, name = "enabled", havingValue = "true")
@EnableConfigurationProperties(DingTalkConfig.class)
static class DingTalkWarnServiceMethod
{

@Bean
@ConditionalOnMissingBean(DingTalkWarnService.class)
public DingTalkWarnService dingTalkWarnService(final DingTalkConfig dingtalkConfig)
{
DingTalkWarnService dingTalkWarnService = new DingTalkWarnService(dingtalkConfig.getToken(), dingtalkConfig.getSecret());
AlarmFactoryExecute.addAlarmLogWarnService(dingTalkWarnService);
return dingTalkWarnService;
}
}

// 消息模板配置裝載
@Configuration
@ConditionalOnProperty(prefix = TemplateConfig.PREFIX, name = "enabled", havingValue = "true")
@EnableConfigurationProperties(TemplateConfig.class)
static class TemplateConfigServiceMethod
{

@Bean
@ConditionalOnMissingBean
public AlarmTemplateProvider alarmTemplateProvider(TemplateConfig templateConfig) {
if (TemplateSource.FILE == templateConfig.getSource()) {
return new YamlAlarmTemplateProvider(templateConfig);
} else if (TemplateSource.JDBC == templateConfig.getSource()) {
// 數據庫(如mysql)讀取文件,未實現,可自行擴展
return new JdbcAlarmTemplateProvider(templateId -> null);
} else if (TemplateSource.MEMORY == templateConfig.getSource()) {
// 內存(如redis,本地內存)讀取文件,未實現,可自行擴展
return new MemoryAlarmTemplateProvider(templateId -> null);
}
return new YamlAlarmTemplateProvider(templateConfig);
}


}
@Bean
public AlarmAspect alarmAspect(@Autowired(required = false) AlarmTemplateProvider alarmTemplateProvider) {
return new AlarmAspect(alarmTemplateProvider);
}
}

四、總結

主要藉助spring的切面技術,以及springboot的自動裝配原理,實現了發送告警邏輯。對業務代碼無侵入,只需要在業務代碼上標記註解,就可實現可插拔的功能,比較輕量。

五、參考源碼

編程文檔:
https://gitee.com/cicadasmile/butte-java-note

應用倉庫:
https://gitee.com/cicadasmile/butte-flyer-parent