Senior Software Engineer - BQL Spark Platform
New York, NY
Posted Sep 20, 2022 - Requisition No. 105507
Our Team:
Bloomberg runs on data. It's our business and our product. From the biggest banks to the most elite hedge funds, financial institutions need timely, accurate data to capture opportunities and evaluate risk in fast-moving markets. Enabling clients to access Bloomberg’s data and perform custom analytics, the Bloomberg Query Language (BQL) and Analytics platform is at the forefront of innovation for our business. BQL is a domain-specific language that allows users to express complex analytics and quantitative models for execution on our platform. BQL allows our clients to re-imagine workflows of their financial data on their own terms, and has a rapidly growing user base.
Within the BQL organization, our team is building a low-latency, distributed interactive infrastructure based on Apache Spark. One major use case for this platform is the execution of BQL queries at scale, to support critical heavy workloads from Bloomberg clients. We are currently focused on BQL on Spark, as well as on Kubernetes orchestration of Spark and on extending our platform to provide ready-to-use solutions for making various Bloomberg content sets available via BQL. We collaborate with various other engineering teams to get things done.
As BQL is becoming the standard in accessing and querying Bloomberg data, the Spark Platform team is poised for growth both in terms of infrastructure and providing a rich feature set for managing interactive Spark. The interactive Spark Platform on Kubernetes project focuses on running Spark as cloud native service on Kubernetes and different cloud providers enabling us to provide various features like increased resiliency, dynamic allocation, autoscaling, and blue-green/canary deployments.
The Role:
As a member of the multi-disciplinary Spark Platform team, you’ll have the opportunity to make key technical decisions to keep this platform moving forward. Our team makes extensive use of open source (e.g. Spark, Kubernetes, Ansible, Terraform etc.) We collaborate widely with the industry, contribute back to the open source projects, and present at conferences. While working on the platform, the backbone for many of Bloomberg's upcoming products, you will have the opportunity to collaborate with engineers across the company and mutually learn about the technology that delivers products from the news to financial instruments. If you are a software engineer who is passionate about building resilient, highly available, and elastic infrastructure and seamless, usable full stack solutions, please apply!
We’ll trust you to:
- Design and develop Kubernetes-based products running as a SaaS offering and on managed clusters
- Interact with data engineers and Spark experts across the company to understand their workflows and requirements to inform the next set of features for the platform
- Design distributed systems and develop solutions for problems such as elastic load distribution, effective resource management, and fair scheduling
- Automate operations and improve telemetry in our Kubernetes infrastructure stack
- Regularly present and explain your work to peers, senior stakeholders (including our CTO), and clients
You'll need to have:
- Proficiency in two or more languages (Go, Java, Scala, Python) and willingness to learn more as needed
- Strong sense of curiosity to solve new problems and keep learning new technologies
- A passion for providing reliable and scalable infrastructure
- Experience with distributed systems eg. Kubernetes, Spark, Kafka
- Experience building and scaling mission-critical systems using any of Kubernetes, Swarm or Mesos
- Bachelor’s degree in computer science or a related field, and/or related professional experience.
We’d love to see:
- Experience with Kubebuilder and Kubernetes operator-based frameworks
- Experience with Spark
- Ability to identify and perform OS and hardware-level optimizations
- Experience with configuration management systems (Chef, Puppet, Ansible, or Salt)
- Experience with continuous integration tools and technologies (Jenkins, Git, Chat-ops)
Bloomberg is an equal opportunity employer, and we value diversity at our company. We do not discriminate on the basis of age, ancestry, color, gender identity or expression, genetic predisposition or carrier status, marital status, national or ethnic origin, race, religion or belief, sex, sexual orientation, sexual and other reproductive health decisions, parental or caring status, physical or mental disability, pregnancy or maternity/parental leave, protected veteran status, status as a victim of domestic violence, or any other classification protected by applicable law.
Bloomberg is a disability inclusive employer. Please let us know if you require any reasonable adjustments to be made for the recruitment process. If you would prefer to discuss this confidentially, please email amer_recruit@bloomberg.net
Salary Range: 160,000 - 240,000 USD Annually + Benefits + Bonus