Hadoop PySpark Developer Job at Purple Drive, Pittsburgh, PA

TmlDcHphbng4aEo0SFhGNXNNOWo3Y2la
  • Purple Drive
  • Pittsburgh, PA

Job Description

Job Title: Hadoop PySpark Developer
Location: Pittsburgh, PA (Onsite)
Experience Level: 8+ years preferred

We're hiring a Hadoop PySpark Developer to work onsite in Pittsburgh. You'll be using tools like Spark, Python, Hive, and Kafka to build and manage data pipelines on the Cloudera Data Platform. The role involves working with large datasets, building ETL solutions, and making sure everything runs smoothly and efficiently. You'll also work with other teams to understand data needs and help deliver reliable solutions.

Roles & Responsibilities

  • Build ETL pipelines using Spark, Python, Hive, and Kafka
  • Understand business requirements and turn them into technical tasks
  • Work with data analysts and business users to gather data needs
  • Create design documents and source-to-target mappings
  • Make sure data pipelines run on time and handle large volumes
  • Improve performance of Spark jobs and existing pipelines
  • Set up data governance and ingestion frameworks
  • Help with production deployments and fix issues when needed
  • Support data migration and rebuilding history data
  • Review code and test cases, and help solve technical problems
  • Work independently and coordinate with global teams
  • Bring in ideas and best practices to improve solutions

Key Skills

  • Apache Spark (PySpark)
  • Python
  • Hive
  • Kafka
  • Cloudera Data Platform
  • Linux/Unix
  • Impala
  • Big Data tools
  • Cloud technologies
  • Performance tuning
  • Data governance
  • Good communication and teamwork

Job Tags

Full time,

Similar Jobs

Jobright.ai

Junior ReactJS (Angular JavaScript Developer) Job at Jobright.ai

 ...verified openings from employers you can trust. Job Summary: The Dignify Solutions, LLC is seeking a ReactJS (Angular JavaScript Developer) to join their team. The role requires strong proficiency in JavaScript and a thorough understanding of React.js and Angular... 

MultiTech Systems

Data Entry Clerk - Remote Work From Home Job at MultiTech Systems

 ...Job Description: We are looking for a detail-oriented Remote Data Entry Clerk to join our team! This is a work-from-home opportunity with daily or weekly pay options. The ideal candidate will have strong typing skills, accuracy, and the ability to work independently... 

Monster

Audit Manager / Sr. Manager - Quality Control (cpa firm) - Remote Job at Monster

CPA firm currently has an opening for a Manager / or Sr. Manager in our Accounting and Auditing Quality Control Department. This position can be remote. Responsibilities Review client attest engagements, which include planning executing, directing, and reviewing financial... 

Ardurra Group, Inc.

Civil CAD Designer Job at Ardurra Group, Inc.

 ...Job Description Job Description Ardurra is seeking a CAD Designer to join our staff in Oakdale or Sacramento, CA! Primary Duties Prepare plans, specifications and estimates for a wide variety of civil engineering projects that include the following elements... 

Digital NEST

General Intern, bizzNEST Video Production Job at Digital NEST

 ...Organization size: 50+ employees Reports to: Video Production Lead Location(s) : Interns are on-site...  ...by working with clients, build professional level portfolios and resumes, and become competitively qualified for entry-level, career-focused employment....