Schema migrations in the relational world are now common practice. The best practices for evolving a database schema are well known, where a migration gets applied before the code that needs to use it is rolled out. If there are any problems, migration can be rolled back. The same practices are not as well established… Read more »
Posts Categorized: Big Data
Type All The Things!
Scala is statically typed. Yet we rarely use it to our advantage. For example let’s say you have a function with the signature: def convert(a: Double, b: Double): String Of course, the author hasn’t written a comment, and clearly didn’t expressively label the variables. So lets fix it: def reverseGeocode(latitude: Double, longitude: Double): String The… Read more »
Spark on AWS EMR – The Missing Manual
Apache Spark recently received top level support on Amazon Elastic MapReduce (EMR) cloud offering, joining applications such as Hadoop, Hive, Pig, HBase, Presto, and Impala. This is exciting for me, because most of my workloads run on EMR, and utilizing Spark required either standing up manual EC2 clusters, or using EMR bootstrap, which was very… Read more »
Handling Avro records in Scalding
In this post I’ll try to cover how to write and read Avro records in Scalding pipelines. To begin, a reminder that Avro is a serialization format, and Scalding is a scala API on top of Hadoop. If you’re not using Scalding, this post is probably not too interesting for you. Let’s begin by defining… Read more »
Debugging Apache Spark Jobs
Would you like to step through your Spark job in a debugger? These steps show you how to configure IntelliJ IDEA to allow just that. Unlike a traditional Java or Scala application, Spark jobs expect to be run within a larger Spark application, that gives access to SparkContext. Your application interacts with the environment through… Read more »