Who are we?
We are a globally expanding software technology company that helps brands communicate more effectively with their audiences. We are looking forward to expand our people capabilities and success in developing high-end solutions beyond existing boundaries and establish our brand as a Global Powerhouse.
We are free to work from wherever we want and go to the office whenever we like!!!
What is the role?
We are looking for a highly skilled and motivated Senior Data Engineer to join our dynamic team. The ideal candidate will have extensive experience in building and managing data pipelines, noSQL databases, and cloud-based data platforms. You will work closely with data scientists and other engineers to design and implement scalable data solutions.
Key Responsibilities:
- Design, build, and maintain scalable data pipelines and architectures.
- Implement data lake solutions on cloud platforms.
- Develop and manage noSQL databases (e.g., MongoDB, Cassandra).
- Work with graph databases (e.g., Neo4j) and big data technologies (e.g., Hadoop, Spark).
- Utilize cloud services (e.g., S3, Redshift, Lambda, Kinesis, EMR, SQS, SNS).
- Ensure data quality, integrity, and security.
- Collaborate with data scientists to support machine learning and AI initiatives.
- Optimize and tune data processing workflows for performance and scalability.
- Stay up-to-date with the latest data engineering trends and technologies.
Detailed Responsibilities and Skills:
- Business Objectives and Requirements:
- Engage with business IT and data science teams to understand their needs and expectations from the data lake.
- Define real-time analytics use cases and expected outcomes.
- Establish data governance policies for data access, usage, and quality maintenance.
- Technology Stack:
- Real-time data ingestion using Apache Kafka or Amazon Kinesis.
- Scalable storage solutions such as Amazon S3, Google Cloud Storage, or Hadoop Distributed File System (HDFS).
- Real-time data processing using Apache Spark or Apache Flink.
- NoSQL databases like Cassandra or MongoDB, and specialized time-series databases like InfluxDB.
- Data Ingestion and Integration:
- Set up data producers for real-time data streams.
- Integrate batch data processes to merge with real-time data for comprehensive analytics.
- Implement data quality checks during ingestion.
- Data Processing and Management:
- Utilize Spark Streaming or Flink for real-time data processing.
- Enrich clickstream data by integrating with other data sources.
- Organize data into partitions based on time or user attributes.
- Data Lake Storage and Architecture:
- Implement a multi-layered storage approach (raw, processed, and aggregated layers).
- Use metadata repositories to manage data schemas and track data lineage.
- Security and Compliance:
- Implement fine-grained access controls.
- Encrypt data in transit and at rest.
- Maintain logs of data access and changes for compliance.
- Monitoring and Maintenance:
- Continuously monitor the performance of data pipelines.
- Implement robust error handling and recovery mechanisms.
- Monitor and optimize costs associated with storage and processing.
- Continuous Improvement and Scalability:
- Establish feedback mechanisms to improve data applications.
- Design the architecture to scale horizontally.
Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
- 5+ years of experience in data engineering or related roles.
- Proficiency in noSQL databases (e.g., MongoDB, Cassandra) and graph databases (e.g., Neo4j).
- Strong experience with cloud platforms (e.g., AWS, GCP, Azure).
- Hands-on experience with big data technologies (e.g., Hadoop, Spark).
- Proficiency in Python and data processing frameworks.
- Experience with Kafka, ClickHouse, Redshift.
- Knowledge of ETL processes and data integration.
- Familiarity with AI, ML algorithms, and neural networks.
- Strong problem-solving skills and attention to detail.
- Excellent communication and teamwork skills.
- Entrepreneurial spirit and a passion for continuous learning.
Join our team!
Top Skills
What We Do
We provide a seamlessly integrated all in one omni-channel marketing solution that enables organizations to engage in relationship marketing with their target audience, customers across complex marketing mix in an efficient and cost effective way. Our clients can communicate with their target audience via e-mail, SMS, MMS, Social Media, Push Notifications, Whatsapp, Text to Speech (IVR) or any other channels they have...We provide a boutique solution through its easy integration feature and it makes campaign management much more efficient as a "centralized" solution. When integrated into the organizations’ CRM systems, SmartMessage tracks actions in many different fields such as customer interaction history, salesforce automation, lead management, marketing & campaign management, service requests and complaints management, contact center automation, social media integration, customer loyalty and constantly provides feedback in these exact fields. In organizations using SmartMessage a "Customer Oriented" structure emerges. Depending on our clients requirements, we can provide this service as an on-premise and/or on-cloud.
