Engineer - Design and Development

Siemens

Bengaluru, Karnataka

Data Engineer - Big Data Developer

Spark: Spark 3.x, RDD/DataFrames/SQL, Batch/Structured Streaming
Knowing Spark internals: Catalyst/Tungsten/Photon
Databricks: Workflows, SQL Warehouses/Endpoints, DLT, Pipelines, Unity, Autoloader
IDE: IntelliJ/Pycharm, Git, Azure Devops, Github Copilot
Test: pytest, Great Expectations
CI/CD Yaml Azure Pipelines, Continuous Delivery, Acceptance Testing
Big Data Design: Lakehouse/Medallion Architecture, Parquet/Delta, Partitioning, Distribution, Data Skew, Compaction
Languages: Python/Functional Programming (FP)
SQL: TSQL/Spark SQL/HiveQL
Storage: Data Lake and Big Data Storage Design
additionally it is helpful to know basics of:

Data Pipelines: ADF/Synapse Pipelines/Oozie/Airflow
Languages: Scala, Java
NoSQL: Cosmos, Mongo, Cassandra
Cubes: SSAS (ROLAP, HOLAP, MOLAP), AAS, Tabular Model
SQL Server: TSQL, Stored Procedures
Hadoop: HDInsight/MapReduce/HDFS/YARN/Oozie/Hive/HBase/Ambari/Ranger/Atlas/Kafka
Data Catalog: Azure Purview, Apache Atlas... More Detail

Apply Now

Similar Jobs

Companies Hiring in Bengaluru, Karnataka

DataCore Software

Roku

GE

Anand Healthcare

Toast

Cloud Software Group

Trimble

EY

RTX

Capital One

Google

GE

WARNERMEDIA

Oxbridge International School

TARGET

Roku

Strata

Morgan Stanley

GE HEALTHCARE

Micron Technology