Posts By: Dan Osipov

Debugging Apache Spark Jobs

Posted by & filed under Big Data.

Would you like to step through your Spark job in a debugger? These steps show you how to configure IntelliJ IDEA to allow just that. Unlike a traditional Java or Scala application, Spark jobs expect to be run within a larger Spark application, that gives access to SparkContext. Your application interacts with the environment through… Read more »

Launching Robolectric tests from Android Studio 1.0

Posted by & filed under Android.

I recently upgraded my Android toolchain to the official Android Studio 1.0. I’ve set up a new project, and as with any Android app, included Robolectric to allow for easy unit testing. I was able to configure Gradle build to execute the tests fairly easily using existing documentation on Robolectric project. To my surprise however,… Read more »

Apache Spark on EC2

Posted by & filed under Big Data.

Its easy to get started with Apache Spark. You can get a template for a Scala job using the Typesafe Activator and have it running on a local cluster with a small dataset. You can also use a handy script spark_ec2 to launch an EC2 cluster as detailed in Running Spark on EC2 document. You could… Read more »