在海量数据处理环境下,传统的基于中心数据库的架构已经无法满足大规模的数据处理应用中高并发高数据读写的需求,而串行的工作模式也使得数据分析的时效性得不到有效的保证,已经严重地影响了用户体验。该文从应用架构的角度出发,提出了一种基于内存的高效在线数据处理服务框架,通过多索引的高效数据存取方法和基于发布/订阅模式的数据访问控制机制,在有效减少用户对中心数据库的读写请求的同时提高了数据处理的时效性。实验结果表明该文提出的基于内存的高效在线数据处理服务框架能够有效提高数据库的响应速度,缩短数据处理延时。
Abstract
Under the environment of big data, traditional database-centered data processing architecture cannot meet the requirement of high concurrency of read/write requests. At the same time, serial usages of data limit the effectiveness of data processing. This paper describes an effective on-line data process and service framework based on memory. This framework uses multi-index data access method and pub/sub data control mechanism to improve the effectiveness of data processing while reducing the interaction with the database. Experimental results show that the memory based on-line data process and service framework can significantly improve the response speed of database and shorten the latency of data processing.
关键词
海量数据 /
数据处理 /
在线缓存 /
发布/订阅
{{custom_keyword}} /
Key words
big data /
data process /
on-line cache /
publisher/subscriber
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] 李盛韬,余智华,程学旗,白硕. Web信息采集研究进展[J]. 计算机科学,2003(02): 151-171.
[2] 郭岩,刘春阳,余智华,等. 网络舆情信息源影响力的评估研究[J]. 中文信息学报, 2011,25(3):64-71.
[3] 曹鹏,李静远,满彤,等.Twitter中近似重复消息的判定方法研究[J].中文信息学报,2011,25(1):20-27.
[4] 郭浩,陆余良.基于信息传播的微博用户影响力度量[C]//CCIR2011.
[5] Fitzpatrick B. Distributed caching with memcached[J]. Journal Linux, 2004,124:7559.
[6] Krakowiak, Sacha. “Whats middleware?”. ObjectWeb.org. Retrieved 2005-05-06.
[7] 徐晶,许炜. 消息中间件综述[J]. 计算机工程, 2005, 33(16):73-76.
[8] 李文逍,杨小虎. 基于分布式缓存的消息中间件存储模型[J]. 计算机工程,2010,36(13):93-95.
[9] Birman K, Joseph T. Exploiting virtual synchrony in distributed systems[C]//Proceeding of SOSP 87 the eleventh ACM Symposium on Operating systems principles, 1987: 123-138.
[10] Hasan, Souleiman. Approximate Semantic Matching of Heterogeneous Events [C]//Proceeding of the 6th ACM International Conference on Distributed Event-Based Systems 2012, 252-263.
[11] Eugster P T, Felber P A, Guerraoui R. The Many Faces of Publish/Subscribe Proceeding of ACM Computing Surveys, 2003,35(2): 114.
[12] The Apache Software Foundation ActiveMQ[DB/OL] http://activemq.apache.org/, 2012.
[13] Snyder, Bruce, Bosanac, Dejan, et al. ActiveMQ in Action (1st ed.)[M], Manning Publications, 2010: 375.
[14] Iyengar A, Jessani V, Chilanti M. WebSphere Business Integration Primer: Process Server, BPEL, SCA, and SOA 1st[M], IBM Press , 2007.
[15] Kloppmann M. IBM Deutschland Entwicklung GmbH Business process choreography in WebSphere: Combining the power of BPEL and J2EE[J], IBM Systems Journal, 2004, 43(2): 270-296.
[16] B.Mann. Worldwide Product Manager Providing a Backbone for Connectivity with SOA Messaging[M], 2009.
[17] Kreps J, Narkhede N, Rao J. Kafka: a Distributed Messaging System for Log Processing[M], 2011.
[18] Videla A, Williams J J W. RabbitMQ in Action:Distributed messaging for everyone[M]. 2012.
[19] home page http://www.rabbitmq.com/[DB/OL], 2012.
[20] Seeger M. Key-Value Stores: a practical overview Computer Science and Media[A], Ultra-Large-Sites September 2009.
[21] Automatic Workload Repository (AWR) in Oracle Database[DB/OL], http://www.oracle-base.com/articles/10g/automatic-workload-repository-10g.php.
[22] Oracle Database Performance Method page[M/CD].
[23] http://docs.oracle.com/cd/B19306_01/server.102/b28051/tdppt_method.htm[DB/OL].
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金(60933005, 61173064,61100083),242专项项目(2011F45,2011F65)
{{custom_fund}}
PDF(4556 KB)
