Jetty’s Server in Spark Stack

Spark has a web accessible monitoring interface, as well as, several other mechanisms like remoting, and file-serving that use http protocols. Spark uses the Jetty web server, client and java servlet container for these functionalities.

The web UI is started from within the SparkContext class by instantiated a SparkUI class, which is sub-class of the WebUI class. WebUI uses Spark utility and Jetty utility functions to spin up the web server and attach handlers to server web content.

Java on Mac OS X

Current version of Java

$> java -version

What java executable is command line using

$> ls -al /usr/bin/java

What versions of java exist on system

$> /usr/libexec/java_home -v version.xx

… i.e. if you execute..

$> /usr/libexec/java_home -v 1.7

… it’ll output the runtime lib directory. If you you use a version number that’s not installed it’ll just tell you that it can’t find a directory. In which case you need to install it.
You can also list the java frameworks library directory, but I’ve found this to be inconsistent with new java versions I install…

$> ls /System/Library/Frameworks/JavaVM.framework/Versions/

Installing JDK
To update the command line java we need to download the jdk, not just the jre that the default “google java download” prompts you to. We then need to make sure our env vars (ie JAVA_HOME) are set in .bash_profile.
JDK 8 Installation for OS X (oracle docs)
OS X Platform Install FAQ
Java SE Development Kit 8 Downloads

Sometimes it’s necessary to have multiple jdk versions on your system depending on what you’re running/compiling. Search for “java archive” (it’s at the bottom) to list previous java versions on downloads page…
Java SE Downloads

If we want to switch between java versions we can use the JAVA_HOME env var..

$> export JAVA_HOME=`/usr/libexec/java_home -v version.xx`

Java and Scala File and Directory Handling

Scala – How to open and read files in Scala

How to write to file in Java – BufferedWriter

Java File: Reading and Writing Files in Java Example (Api Docs)

4 Ways to Copy File in Java

File and directory Copying

Java File IO – How to test whether a file or directory exists

How to create directories in Java

How do I get all files with certain extensions in a directory including subdirectories?

How to set File Permissions in Java Easily using Java 7 PosixFilePermission