Spark Platform Engineer
New York, NY
Posted Mar 29, 2016 - Requisition No. 49254
The Spark Platform team is building low-latency, distributed analytics infrastructure for Bloomberg based on Apache Spark.
Instead of building dozens of isolated Spark applications for individual problem domains, we are building a platform that makes it easy for teams to plug in their business logic without duplicating common functionality. This includes connecting to various datastores or real-time streams, figuring out a way to serverize Spark transforms or having to re-implement transforms such as currency conversion that are very common in financial analytics. But we can't just use Apache Spark as is. We need to enhance open source Spark to fit our low-latency, high throughput and high availability contexts. That's where you come in.
As part of our team, you'll collaborate with a number of the core Spark contributors to co-design and co-develop enhancements to Spark in areas ranging from Performance to HA.
We'll trust you to:
- Contribute to the development of the Bloomberg Spark Platform
- Interact with application teams to understand their specific requirements and drive the next set of features for the Spark Platform
- Meet with key players in the open source community to motivate necessary development efforts
- Contribute code to the Apache Spark community in areas ranging from Machine Learning to Spark HA
You need to have:
- Strong programming proficiency in Java and Scala (a minimum of two years)
- An expert-level understanding of Apache Spark - your work will involve modifying its internals
We'd love to see:
- Experience with MLlib
- Open source contributions of any sort
- Professional experience developing analytics infrastructure
If this sounds like you, submit an application and watch our talk at Spark Summit East to learn more about Bloomberg's Spark Platform (https://spark-summit.org/east-2016/events/spark-at-bloomberg/).