This blog is written and maintained by students in the Professional Master’s Program in the School of Computing Science at Simon Fraser University as part of their course credit. To learn more about this unique program, please visit {sfu.ca/computing/pmp}.

Ref: https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/files/2017/06/01_06.png

Spark Memory Management

To write programs in spark efficiently and with high performance, you will have to go over the memory management in spark. Most of the people either increase the cores, increase the memory of the executor and driver or play around with file size and .repartition() …

Akash Sindhu

Distributed Systems and Deep Learning Enthusiast

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store