Azure Databricks Engineer – SQL Migration

We are seeking a skilled and experienced SQL to Databricks Migration Specialist to join our team. The primary responsibility of this role is to migrate SQL-based Crystal Reports to Databricks, ensuring seamless integration and optimal performance. The ideal candidate should have a strong background in SQL, Databricks, and Crystal Reports, along with excellent problem-solving skills and attention to detail.

Responsibilities: As a SQL to Databricks Migration Specialist for Crystal Reports, your key responsibilities will include:

1. Analyzing Existing Crystal Reports:
- Reviewing existing Crystal Reports that utilize SQL queries.
- Identifying the underlying data sources and dependencies.
- Assessing the complexity of each report and its compatibility with Databricks.

2. Designing Data Models on Databricks:
- Collaborating with stakeholders (business analysts, developers) to understand reporting requirements.
- Translating existing SQL queries into optimized SparkSQL or Pyspark code for execution on Databricks.
- Designing efficient data models using Delta Lake or other appropriate storage formats.

3. Migrating Crystal Reports:
- Developing migration strategies based on the analysis of existing reports.
- Rewriting or modifying existing reports using Apache Spark APIs within the Databricks environment.
- Ensuring data integrity during the migration process.

4. Performance Optimization:
- Profiling and optimizing migrated reports for improved performance on Databricks clusters.
- Implementing best practices for query optimization, caching mechanisms, partitioning techniques, etc.

5. Documentation and Training:
- Documenting the migration process including steps taken, challenges faced, and solutions implemented.
- Providing training sessions or documentation for end-users on how to use migrated reports in Databricks.

Qualifications:
- Bachelor's degree in Computer Science or related field (or equivalent work experience).
- Strong proficiency in SQL programming language.
- In-depth knowledge of Apache Spark ecosystem including SparkSQL , Pyspark and Python programming languages.
- Familiarity with cloud-based platforms such as Azure Databricks would be advantageous
- Familiarity with Crystal Reports or any reporting tools would be advantageous
- Excellent problem-solving skills with an ability to analyze complex systems efficiently
- Strong communication skills with an ability to collaborate effectively across teams

Looking for someone senior who can work independently strong skill on Azure databricks, Datalake, and other Azure services and worked on migration