kholohan / docker-hadoop-spark-workbench Public

forked from big-data-europe/docker-hadoop-spark-workbench

Notifications You must be signed in to change notification settings
Fork 0
Star 0

[EXPERIMENTAL] This repo includes deployment instructions for running HDFS/Spark inside docker containers. Also includes spark-notebook and HDFS FileBrowser.

0 stars 381 forks Branches Tags Activity

Star

Notifications

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
ant		ant
datanode		datanode
hadoop		hadoop
historyserver		historyserver
hue		hue
huebuildenv		huebuildenv
namenode		namenode
nodemanager		nodemanager
resourcemanager		resourcemanager
spark-master		spark-master
spark-notebook		spark-notebook
spark-worker		spark-worker
spark		spark
.gitignore		.gitignore
README.md		README.md
README.md.orig		README.md.orig
buildall.sh		buildall.sh
docker-compose.yml		docker-compose.yml
docker-compose.yml.orig		docker-compose.yml.orig
hadoop.env		hadoop.env
spawn-spark-worker.sh		spawn-spark-worker.sh

Repository files navigation

How to use docker-compose

To start a cluster:

    ./buildall.sh
    docker-compose up -d

To scale up spark-workers:

    docker-compose scale spark-worker=3

Docs

Motivation behind the repo and an example usage @BDE2020 Blog

Notes

Hadoop/Spark dockerfiles were forked from https://bitbucket.org/uhopper/hadoop-docker

About

[EXPERIMENTAL] This repo includes deployment instructions for running HDFS/Spark inside docker containers. Also includes spark-notebook and HDFS FileBrowser.

Readme