彩经网

博睿宏远-企业应用性能管理服务商,赋能IT运营智能化

彩经网

大数据

Bonree Ants流式大数据处理引擎

让企业大数据处理更简单

立即体验

Ants(蚁群)名称源于其寓意:高效协作、小而有力、合纵连横,我们认为其可以诠释这套超级引擎框架的初衷:轻便灵活、场景通用、稳定高效。

大数据处理引擎框架是一个针对时序指标数据处理的通用流式计算引擎框架。此引擎框架负责范围包括原始数据预处理、准实时计算(支持默认算子和自定义算子)、多种时间粒度批量计算、数据落地及各种容错处理,同时支持自定义高级功能自定义扩展机制,默认支持如时序指标动态基线计算、报警条件判断计算等扩展等。

背景

彩经网YUELAIYUEDUODEQIYERENSHIDAODASHUJUDUIYUZISHENWEILAIFAZHANDEZHONGYAOXING,FENFENKAISHISHIYONGBINGZHUJIANYILAIDASHUJUCHULIDEXIANGGUANJISHU。DANSUIZHEXUYAOCHULIDESHUJUYUELAIYUEDUO,YEWUCHANGJINGYUEFAFUZA,ZAISHIJIZHIXINGGUOCHENGZHONGYUDAOHENDUOWENTI,RUDASHUJURENCAIJINQUE,XIANGGUANJISHUQUEFACHENDIAN,NANYITUPOPINGJING,TONGSHIBUTONGYEWUBUMENDEYEWUXUQIUFENFANDUOYANG,DAOZHIBUTONGXIANGMUDAIMAZHONGFUKAIFA、ZHONGFUZAOLUNZI、XIANGMUJISHUJIAGOUWUHUABAMENDEQINGKUANGYEHENCHANGJIAN,JIHOUQIWEIHUHEDIEDAIDAILAIHENDADETIAOZHAN。

DUIYUYISHANGDEXIANSHIWENTI,BORUIHONGYUANYI10NIANDEDASHUJUXIANGMUSHIJIANJINGYAN,CHOUXIANGHESHEJICHUYITAOLINGHUOQINGBIAN、CHANGJINGTONGYONG、WENDINGGAOXIAODECHAOJIDASHUJUCHULIYINQINGKUANGJIA—Bonree Ants,YIMANZUQIYEFUZADUOYANGDESHUJUCHULIXUQIU。

核心价值

KEYIBANGZHUQIYEBIANJIEKUAISUDESHIXIANHAILIANGSHIXUZHIBIAOSHUJUDEZAIXIANLIUSHICHULI。WOMENDEMUBIAOSHI,DUIYUCHANGJIANDESHIXUZHIBIAOLIUSHICHULIDEYEWUCHANGJINGKEWANQUANBUXUYAOYANFARENYUANCANYU,ZHIXUFEIYANFAYEWURENYUANDUISHUJUYINGYONGJINXINGJIANDANPEIZHIHEYEWUJIAOBENMIAOSHU,JIKESHIXIANQIMUBIAO;ERDUIYUFUZAYEWUCHANGJINGWOMENZEXIWANGYANFARENYUANTONGGUOYINQINGDECHAJIANJIZHI,JINXINGSHAOLIANGBIANMALAISHIXIANXIANGGUANYUYEWUQIANGGUANLIANDELUOJI,ERJIANGDASHUJUCHULIZHONGDICENGFUZADEZIYUANDIAODU、RENWUBIANPAI、RONGCUOCHULIJIAOJIYINQINGFUZE,KUAISUSHIXIANXIANGGUANDASHUJUCHULIYEWUKAIFA,JIDAJIANGDIQIYEXIANGGUANKAIFAHEWEIHUCHENGBEN。

TONGGUOZAIBORUIHONGYUANZHONGDUOFUWUXIANGMUDESHIJIAN,YINGYONGBonree AntsYINQINGKUANGJIAZHIHOU,DASHUJUCHULIKAIFAGONGZUOLIANGZHENGTIJIANGDILE80%,ZHENGTIXIANGMUZHOUQISUODUAN40%YISHANG。

整体架构

Bonree Ants整体架构图

Bonree AntsYINQINGKUANGJIAYILAIKAIYUANZUJIANYUNXING:ZooKeeper、Kafka、Storm、Redis,YILAISHAO,BUSHUJIANDAN,JIAGOUQINGBIAN,ZHEIYESHIQIHEXINYOUSHIZHIYI。

BonreeAntsDICENGJISUANKUANGJIAJIYUStorm,ZHISUOYIXUANZEStormZUOWEIDICENGJISUANKUANGJIA,ZHUYAOKAOLVDAOStormZISHENDESHISHIXINGGAO、ZIYUANKAIXIAOXIAO,WAIBUYILAISHAO、CHUNNEICUNJISUAN、RONGCUOXINGHAODENGTEXING。

Bonree AntsJIANGSHIXUZHIBIAODASHUJUCHULIGUOCHENGCHOUXIANGWEIYIXIAJIGEZHUYAOLIUCHENG,JI:YUCHULI、ZHUNSHISHIJISUAN、XIAOPILIANGJISUAN、DAPILIANGJISUAN、LUODIRUKUDENG。YISHANGLIUCHENGJUNYOUYUNXINGYUStormZHISHANGDEPreprocessing Topology、Calculation Topology、Storage TopologySANLEITUOPUFUZEWANCHENG。

Bonree AntsZHENGGESHUJUCHULIGUOCHENGZHONGSHUJUBULUODI,JUNZAINEICUNZHONGWANCHENG。YOUYUXUZHICHIDASHIJIANLIDUPILIANGJISUANYEWUCHANGJING,Bonree AntsYINRURedisZUOWEIStormJIQUNFUZHUNEICUNCUNCHUJIQUN,YIJIANGDIDAPILIANGJISUANSHIDUIStormJIQUNNEICUNDEKAIXIAO,YOUYUCAIYONGNEICUNJISUANDEFANGSHI,Bonree AntsSHISHIXINGGAO,DUICIPANI/OJIHUWUYINGXIANG。

彩经网Bonree AntsPILIANGJISUANSHIJIYUSHIJIANLIDUDEJUHEJISUAN,MORENZHICHI1MIAO、1FENZHONG、10FENZHONG、1XIAOSHI、1TIANDENGWUGEBUTONGSHIJIANLIDUDEJUHE。YOUYUGELIDUJISUANZHIJIANCUNZAIDIJINYILAIGUANXI,YINCIWEIJIANSHAOJISUANZIYUANKAIXIAO,JIASUJISUANGUOCHENG,Bonree AntsZAIRedisJIQUNHUANCUNLEGELIDUZHONGJIANJISUANJIEGUO,YIBEIXIAYISHIJIANLIDUJISUANZHIJIESHIYONG,CONGERJIANSHAOSHUJUCHULILIANGJI。

Bonree AntsDUIYUZUIZHONGJISUANJIEGUOLUODIZHITIGONGLEJIBENKUANGJIAZHICHI。YOUYUBUFUZEZUIZHONGSHUJUCUNCHU,YINCIYEBUHUIDUIZUIZHONGSHUJULUODICUNCHUZUJIANYOURENHEXIANZHI。Bonree AntsMORENNEIZHIZHICHIMysqlCUNCHUFANGAN,RUCAIYONGMysqlZUOWEIZUIZHONGLUODISHUJUKU,ZEJIANBIAOHEJIEGUOSHUJURUKUGUOCHENGJUNKEYOUBonree AntsZIDONGWANCHENG。RUXUCAIYONGQITALUODIFANGAN,RUHBaseDENG,ZEYOUYONGHUKAIFAStorage-PluginCHAJIANJINXINGZIDINGYIZHICHI。

ZHENGTIJIAGOUSHEJIDALIANGCAIYONGCHAJIAN(plugins)HEKUOZHAN(extensions)JIZHI,QIJIANGYUYEWUQIANGGUANLIANDEGEXINGHUACHULI,RUSHUJUYUCHULICELVECHOUXIANGWEIEtl-pluginJIANGSHUJUHEWEIDUZHIBIAOCHULIDESUANZI(NEIZHIZHICHIsum、max、minDENGJIBENSUANZI)KAIFANGWEIOperator-plugin,ERJIANGCHULIJIEGUOLUOPANCELVECHOUXIANGWEIStorage-plugin。

彩经网TONGSHI,ZAIZHICHICHAJIANJISHUWAI,WEILEFENGFUBonree AntsYINQINGKUANGJIAGONGNENG,HAIZHICHIextensionsKUOZHANJIZHI。YONGHUKEYIZAIXIANYOUYINQINGKUANGJIAJICHUSHANGKAIFAZIJIXUYAOextension,BINGYIDULIJISUANTUOPUDEFANGSHITIJIAOJIAnts,YOUYUAntsJIAZAIYUNXING,CONGERSHIXIANAntsGONGNENGDEYANSHEN。MUQIAN,Bonree AntsMORENNEIZHIDONGTAIJIXIANKUO(base-lineextension)HEBAOJINGTIAOJIANPANDUANKUOZHAN(alter-judge extension)。

CIWAI,Bonree AntsHAIZHICHICHAJIANDONGTAIGENGXINHESchema.xmlDONGTAIGENGXINGONGNENG,ZHEIYANGNENGGOUBANGZHUYONGHUSHIXIANBUZHONGQIJIKEGENGXINYEWUCHULILUOJIDEZHICHI,DUIYONGHUFUZADUOBIANDEYEWUCHANGJINGDAILAIJIDADELINGHUOXING。

运行机制

Bonree Ants运行机制

数据预处理拓扑(PreProcessing Topology)

彩经网CITUOPUFUZECONGkafkaDINGYUEYUANSHISHUJUYUANBINGDIAOYONGetl-pluginCHAJIANDUISHUJUJINXINGYUCHULI(etl-pluginCHAJIANYOUKEHUZIJILAISHIXIAN),BINGDUIJIANGetlHOUDEJIEGUOJINXINGZHUNSHISHIJUHECHULI(JISUANGUIZEYOUYONGHUZAIschema.xmlZHONGMIAOSHU)。

数据计算拓扑(Calculation Topology)

CITUOPUFUZEDUIetlZHIHOUDESHUJUJINXINGANBUTONGSHIJIANLIDUDEPILIANGJUHE(GUIZEYOUSchema.xmlMIAOSHU)。CIGUOCHENGNEIBUYOULIANGGEZIJISUANTUOPU:XIAOPILIANGJISUAN(FENZHONGJI),DAPILIANGJISUAN(XIAOSHIJIHETIANJI)。JISUANGUOCHENGZHONGXIAOLIDUDEZHONGJIANJISUANJIEGUODOUHUIHUANCUNDAOredisJIQUNZHONG,GONGXIAYIDALIDUJISUANSHIYONG。TONGSHI,GELIDUJISUANJIEGUOHUILUODIDAOkafkaXIANGYINGDEtopic,YOU“SHUJUCUNCHUTUOPU”DINGYUEJINXINGHOUXUCUNCHUCAOZUO,CONGERSHIXIANSHUJUJISUANHELUODIZHIJIANDEJIEOU。

数据存储拓扑(Storage Topology)

CITUOPUFUZESHUJULUODIRUKU,LUODISHUJUBAOKUOLIANGZHONG,JISHIXUZHIBIAOSHUJU(JIEGOUHUA)HEKUAIZHAOWENJIANSHUJU(FEIJIEGOUHUA,RUYEWUZHONGCUNZAICILEISHUJUDEHUA)LIANGZHONG,YONGHUXUGENJUZISHENYEWUXUYAOKAIFAStorage-pluginLAISHIXIANJUTIDELUOPANCELVE。

CHUCIZHIWAI,Bonree AntsMORENSHIXIANLELIANGGEZHONGYAODEextensionKUOZHAN,JIJIXIANJISUANextensionHEBAOJINGTIAOJIANPANDUANextension。

创新企业大数据处理方式

彩经网 YINQINGKUANGJIAZISHENYUYEWUWUOUHE,SHUJUCHULILIUCHENGGAODUCHOUXIANG,TONGYONGXINGQIANG,JIANJIEKAIFANGDEJIAGOU,JIAOSHAODEZUJIANYILAI,KAIFABUSHUJIWEIHUCHENGBENDI;MIAOJISHIYAN,SHISHIXINGHAO,NEIZHIPILIANGJISUANZHICHI;ZHICHIextensionJIZHI,YONGHUKEZIXINGFENGFUYEWUCHANGJINGGONGNENGZHICHI;NEIZHIDUOZHONGRONGCUOCELVE,BAOZHENGWENDINGYUSHUJUANQUAN;TONGSHI,Bonree AntsTIGONGQIANGDADEZAIXIANWeb ConsoleZHICHI,KEZHICHIKESHIHUAGUANLIHEJIANKONG,JUTIGONGNENGBAOKUORUXIA:

APP管理

APPZAIXIANCHUANGJIAN、SchemaZAIXIANPEIZHI、APPFABUSHANGXIAN、XIAXIAN

基础环境监控

zookeeper、storm、kafka、redisDENGJICHUJIQUNYUNXINGZHUANGTAIJIANKONG

日志监控

彩经网APPYUNXINGGUANJIANLIUCHENGJIEDIANZHUANGTAIRIZHISHOUJIHEJIANKONG。

Bonree AntsYIXIAOERMEIDELINIAN,YILINGHUOGAOXIAODEJISUANFANGSHIKUAISUCHULIQIYESHUZIHUAYEWUDAILAIDEPANGDASHUJULIANG,SHIQIYETIGAODASHUJUZAIXIANCHULINENGLIDEZHINENGDANAO。

博睿宏远-企业应用性能管理服务商,赋能IT运营智能化

彩经网