Show simple item record

dc.contributor.advisor Ng, T. S. Eugene
dc.creatorLiu, Zhaolei
dc.date.accessioned 2016-01-25T16:47:38Z
dc.date.available 2016-01-25T16:47:38Z
dc.date.created 2015-12
dc.date.issued 2015-07-13
dc.date.submitted December 2015
dc.identifier.citation Liu, Zhaolei. "Leaky Buffer: A Novel Abstraction for Relieving Memory Pressure from Cluster Data Processing Frameworks." (2015) Master’s Thesis, Rice University. https://hdl.handle.net/1911/88106.
dc.identifier.urihttps://hdl.handle.net/1911/88106
dc.description.abstract The shift to the in-memory data processing paradigm has had a major influence on the development of cluster data processing frameworks. Numerous frameworks from the industry, open source community and academia are adopting the in-memory paradigm to achieve functionalities and performance breakthroughs. However, despite the advantages of these in-memory frameworks, in practice they are susceptible to memory-pressure related performance collapse and failures. The contributions of this thesis are two-fold. Firstly, we conduct a detailed diagnosis of the memory pressure problem and identify three preconditions for the performance collapse. These preconditions not only explain the problem but also shed light on the possible solution strategies. Secondly, we propose a novel programming abstraction called the leaky buffer that eliminates one of the preconditions, thereby addressing the underlying problem. We have implemented the leaky buffer abstraction in Spark for two distinct use cases. Experiments on a range of memory intensive aggregation operations show that the leaky buffer abstraction can drastically reduce the occurrence of memory-related failures, improve performance by up to 507% and reduce memory usage by up to 87.5%.
dc.format.mimetype application/pdf
dc.language.iso eng
dc.subjectbig data
JVM
memory
Spark
dc.title Leaky Buffer: A Novel Abstraction for Relieving Memory Pressure from Cluster Data Processing Frameworks
dc.contributor.committeeMember Cox, Alan L
dc.contributor.committeeMember Jermaine, Christopher M
dc.date.updated 2016-01-25T16:47:38Z
dc.type.genre Thesis
dc.type.material Text
thesis.degree.department Computer Science
thesis.degree.discipline Engineering
thesis.degree.grantor Rice University
thesis.degree.level Masters
thesis.degree.name Master of Science


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record