Here we will explain how to run the example from in minikube.

We assume the you have installed:


Now spark support Java 11, so we are going to use it. If you followed the tutorial How to install multiple java versions you can set it up by

jenv global 11

Create dockers (spark and spark-py)

First got to downloaded spark and use spark’s script to create docker for k8s:

./bin/ -r barteks -t v${SPARK_VERSION}-java11 \
    -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile \
    -b java_image_tag=11-slim build

Here barteks is my Docker Hub account

Then push dockers to Docker Hub:

docker push barteks/spark:v${SPARK_VERSION}-java11
docker push barteks/spark-py:v${SPARK_VERSION}-java11

Run minikube

minikube --memory 4096 --cpus 2 start

It’s important to reserve some memory and cpu’s.

Create service account

kubectl create serviceaccount spark-job
kubectl create clusterrolebinding spark-role \
    --clusterrole=edit --serviceaccount=default:spark-job \

This corresponds to:

apiVersion: v1
kind: ServiceAccount
  name:  spark-job
  namespace: default
kind: ClusterRoleBinding
  name: spark-role
  kind: ClusterRole
  name: edit
  - kind: ServiceAccount
    name: spark-job
    namespace: default

Submit example job


 kubectl cluster-info

get Kubernetes master. Should look like and modify in the command below:

./bin/spark-submit \
  --master k8s:// \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.container.image=barteks/spark-py:v${SPARK_VERSION}-java11 \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-job \

(check spark’s example directory for the proper link).

Check results

kubectl get pods

The check logs of the corresponding pod:

kubectl logs spark-pi-1751c5764c6efff5-driver

Updated: 2020-12-28