Senior Data Engineer

منذ 7 ساعات

doha, قطر Management Solutions International دوام كامل
Job description Senior Data Engineer

The Senior Data Engineer will own the data side of our client engagement in Doha. Joining the founding team in Doha and working day-to-day with the Senior Solutions Architect, the successful candidate will bring the Client’s own data and new sector-specific sources into the platform, extend the data model and ontology to cover them, and ensure the data is reliable enough for AI workflows to reason upon.

The Senior Data Engineer will build and operate pipelines using Databricks and Azure Data Factory on an Azure-hosted platform while upholding agile engineering practices, including unit testing, automation, and continuous integration and deployment through CI/CD.

The successful candidate will be someone who takes initiative, stays curious, and has an unrelenting drive to push the boundaries of AI and data science. They will be adaptive and introspective, willing to learn, guide, lead, and follow, with a low-ego mindset where outcomes matter more than individual recognition.

This is an opportunity to play a key role in advancing AI capabilities within the financial technology sector, working with highly experienced professionals as one of the first engineers in the Doha office and helping shape how the Client delivers solutions across the region.

As part of a fast-paced, high-growth environment, the successful candidate will have a direct impact on business growth and strategic direction, with significant opportunities for personal and professional advancement while contributing to AI-driven financial solutions.

Key Responsibilities ETL Development
  • Design, develop, and implement robust, efficient, and scalable data pipelines into the Client's data platform.
  • Ingest structured data feeds, APIs, and unstructured documents.
  • Build batch pipelines for pre-populated datasets.
  • Build on-demand enrichment pipelines triggered by user requests and business requirements.
  • Build and operate pipelines using Databricks and Azure Data Factory.
  • Ensure all pipelines include unit testing, automation, and CI/CD.
  • Develop production-grade pipelines that are tested, automated, repeatable, and scalable from the outset.
Client Data Integration
  • Onboard the Client's proprietary and licensed datasets into the platform.
  • Establish data contracts, schema mapping, secure connectors, and permission-based access controls.
  • Work within agreed data residency, security, and governance requirements.
  • Ensure client data is securely integrated and managed according to agreed access and residency rules.
Data Modelling & Ontology
  • Extend the data ontology to accommodate each new data source.
  • Map new sources across entities such as organizations, people, transactions, funds, and their relationships.
  • Work with the Head of Data to determine whether new concepts should be represented as entities or attributes.
  • Register new entities, relationships, and attributes within the field catalogue to ensure exposure through the platform API.
  • Build the data layer for sector-specific ontology packs, including new source types such as patents, scientific literature, regulatory records, and project registries.
  • Develop new entity types based on the Client's priority use cases.
  • Ensure new data is consistently modelled and accessible through the platform API.
Entity Resolution
  • Extend record-linkage and entity-resolution capabilities to new data sources.
  • Ensure client and third-party records are accurately matched to the correct organizations and entities.
  • Apply techniques such as fuzzy matching and blocking where appropriate.
  • Measure matching precision and recall to validate data quality and resolution accuracy.
Data Quality & Observability
  • Define and enforce data-quality checks across every pipeline.
  • Monitor completeness, freshness, duplication, and schema conformance.
  • Implement alerting and dashboards to identify data issues proactively.
  • Ensure issues are identified and addressed before reaching client-facing workflows.
Cost & Scale Trade-offs
  • Evaluate the economic impact of every new data source.
  • Estimate acquisition, processing, storage, and refresh costs.
  • Recommend whether sources should be pre-populated through batch processing or retrieved on demand.
  • Ensure each data source has a clear cost and maintenance strategy.
Documentation & Handover
  • Develop comprehensive documentation for pipelines, data contracts, and operational runbooks.
  • Ensure solutions can be operated and maintained by the wider data team and future hires in Doha.
  • Build sustainable systems that do not depend on a sin