In this tutorial we will be developing Spark listener for capturing runtime metrics of Spark's job, stages and tasks.
It's an analytics engine for processing large scale data stored in variety of file system and database.
Spark listeners allows you to hook custom code on different events emitted while the spark application is running. These events help us to capture metrics, that could be quite helpful in debugging and optimizing the code.
- A spark cluster running on YARN or standalone. To quickly spin up a local cluster, checkout this tutorial.
- Basic knowledge of Scala programming language and Spark APIs.
- Eclipse or IntelliJ IDEA which supports Scala language.
- Create a Scala project and add the following dependencies in the build.sbt file as shown below.
- Next, create a Listener class, which extends
SparkListenerabstract class, which internally implements
SparkListenerInterface. This class will consist of the overriden methods, that will receive the metrics once a Job, stage, task or the entire application ends.
- We need some variable to be initialized, that will be storing all the metrics.
- The listener interface allows us to hook custom code on certain events. For this basic example, we want to capture metrics once a job, stage, task or application completes. Following are the events -
- Implement the methods as shown below to capture the metrics.
- Now its time to use the above created Spark listener in any spark application. For simplicity, create a basic Spark application, that reads a text file and counts the number of lines present in the file. The highlighted line shows one way of adding your custom Spark listener to your application. You can also pass a list of custom spark listeners as mentioned in the official Spark configuration documentation.
- Let's run the above application and check the logs to get the metrics.