۳ ساعت پیش

استخدام Data Engineer برای مجموعه بیت پین در تهران

بیت پین

تهران

۳ ساعت پیش اطلاعات تماس

حضوری

مقطع تحصیلی اعلام نشده

سابقه دارد (۴ سال)

حقوق توافقی

آقا و خانم

تمام وقت (Saturday To Wednesday 9 to 17)

مشاهده اطلاعات تماس

اطلاعات بیشتر

امروز

مجموعه بیت پین در تهران جهت تکمیل کادر خود به افراد واجد شرایط ذیل نیازمند است:

Description

job title

Job Description:
As a Data Engineer on Bitpin's AI & Data team, you will help architect and operate the data platform behind our cryptocurrency exchange. You will own the data's journey from source to value building pipelines from our operational PostgreSQL databases into our ClickHouse analytical warehouse, serving governed data marts to BI consumers, and managing the data ecosystem for our Qdrant-powered AI applications.
Key Responsibilities:
ETL/ELT Pipeline Development: Design, build, and maintain robust, scalable data pipelines using Apache Airflow and PySpark to transform data from PostgreSQL sources into our ClickHouse analytical warehouse.
Real-Time & Streaming Ingestion: Build and operate streaming ingestion paths that move data from operational systems into the warehouse with low latency, ensuring correct ordering, deduplication, and delivery semantics.
ClickHouse Modeling & Optimization: Design performant ClickHouse data models and write highly optimized SQL for large-scale analytical workloads.
Performance Tuning: Continuously profile and optimize query performance, storage layout, and resource utilization across the analytical warehouse as data volumes grow.
Data Quality & Reliability: Implement validation, reconciliation, and completeness checks across pipelines to guarantee that downstream consumers can trust every number — no silent data loss, no unnoticed drift.
Monitoring & Observability: Instrument pipelines and data services with metrics, alerting, and freshness monitoring so that failures and anomalies are detected before stakeholders notice them.
Application-Level Service Administration: Take full ownership of the configuration and administration of our data services. This includes creating Kafka topics and managing ACLs, designing database schemas, managing users and permissions, and setting up S3-compatible object storage buckets.
Workflow Orchestration: Develop, schedule, and monitor complex data workflows (DAGs) in Apache Airflow, ensuring they are efficient, reliable, and well-documented.
Serving Analytics & Governance: Build and maintain governed data marts consumed by our BI layer (Metabase) and data catalog (OpenMetadata), including supporting access-control enforcement jobs.
AI Data Provisioning: Develop processes to prepare, embed, and load data into our Qdrant vector database, often orchestrating these jobs within Airflow.
Collaboration & Ownership: Work closely with software developers, analysts, and AI specialists to define their data needs and build the solutions to meet them.
What We're Looking For:
Proven experience as a Data Engineer.
Hands-on experience with ClickHouse and its feature set.
Experience with CDC pipelines (Debezium, Kafka Connect) and streaming data processing paradigms.
Expert-level proficiency in Python for data processing, with experience in PySpark for transformation workloads.
Deep knowledge of SQL and experience with relational databases (PostgreSQL is a must).
Hands-on experience developing, scheduling, and monitoring complex data pipelines using Apache Airflow.
Experience with ETL/ELT architecture, data modeling, and data warehousing concepts.
Familiarity with containerizing applications using Docker.
Bonus Points:
Familiarity with BI tools (Metabase) and data catalogs (OpenMetadata).
Experience with vector databases (Qdrant, Weaviate, etc.).
Working days: Saturday To Wednesday 9 to 17

Data Engineer

متقاضیان واجد شرایط می توانند با کلیک روی لینک تکمیل فرم استخدام، رزومه خود را ارسال نمایند.

اطلاعات تماس

گزارش مشکل آگهی