Keyerror: Spark_home During Sparkconf Initialization

I am a spark newbie and I want to run a Python script from the command line. I have tested pyspark interactively and it works. I get this error when trying to create the sc: File '

Solution 1:

It seems like there are two problems here.

The first one is a path you use. SPARK_HOME should point to the root directory of the Spark installation so in your case it should probably be /home/dirk/spark-1.4.1-bin-hadoop2.6 not /home/dirk/spark-1.4.1-bin-hadoop2.6/bin.

The second problem is a way how you use setSparkHome. If you check a docstring its goal is to

set path where Spark is installed on worker nodes

SparkConf constructor assumes that SPARK_HOME on master is already set. It callspyspark.context.SparkContext._ensure_initializedwhich callspyspark.java_gateway.launch_gateway, which tries to acccessSPARK_HOME and fails.

To deal with this you should set SPARK_HOME before you create SparkConf.

import osos.environ["SPARK_HOME"] = "/home/dirk/spark-1.4.1-bin-hadoop2.6"
conf = (SparkConf().setMaster('local').setAppName('a'))