Lead Data Engineer- Java/Big Data/SQL/Architect-14+yrs

Posted 2 Days Ago
Be an Early Applicant
Bangalore, Bengaluru, Karnataka
Expert/Leader
Fintech • Information Technology • Payments
Join a world leader in payments and technology!
The Role
The Lead Data Engineer will manage and develop the Enterprise and Visa Data Catalog platforms, implementing metadata processes, ensuring performance optimization, and enhancing data discovery features. This role includes heavy coding responsibilities across various languages and collaboration with business stakeholders to enrich metadata and maintain data governance.
Summary Generated by Built In

Company Description

Visa is a world leader in payments and technology, with over 259 billion payments transactions flowing safely between consumers, merchants, financial institutions, and government entities in more than 200 countries and territories each year. Our mission is to connect the world through the most innovative, convenient, reliable, and secure payments network, enabling individuals, businesses, and economies to thrive while driven by a common purpose – to uplift everyone, everywhere by being the best way to pay and be paid.

Make an impact with a purpose-driven industry leader. Join us today and experience Life at Visa.

Job Description

The Universal Data Catalog (UDC) / Visa Data Catalog (VDC) is a comprehensive metadata management platform designed to provide a centralized repository for all data-related information across the organization. The platform enables efficient data discovery, governance, and utilization by offering detailed metadata definitions for tables, columns, and other data assets. It supports various business units by improving data accessibility, enhancing data quality, and facilitating compliance with data governance policies. Our implementation leverages the open-source Datahub project, ensuring a robust, flexible, and scalable solution.

Key Responsibilities

  • Develop and maintain the Enterprise Data Catalog / Visa Data Catalog platform, ensuring it meets the organization’s evolving needs.

  • Implement and manage metadata ingestion processes to ensure the catalog is up-to-date with the latest data definitions and business context.

  • Collaborate with data stewards, data owners, and other stakeholders to enrich metadata with business definitions, data lineage, and usage context.

  • Enhance the catalog’s search and discovery capabilities to provide users with intuitive and efficient access to data assets.

  • Integrate the data catalog with other enterprise systems and tools to support data governance, data quality, and analytics initiatives.

  • Monitor and optimize the performance of the data catalog to ensure scalability and reliability.

  • Provide training and support to users on how to effectively leverage the data catalog for their data-related needs.

  • Actively collaborate with the open-source community to contribute to and leverage the latest advancements in the Datahub project.

  • Analyze industry best practices and keep the catalog functionality up-to-date with feature sets provided by the market, while focusing on Visa’s scalability requirements.

  • Champion the adoption of open infrastructure solutions that are fit for purpose while keeping technology relevant.

  • Spend 70% of the time writing code in different languages, frameworks, and technology stacks.

This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.


Qualifications

Basic Qualification
•12 to 15 yrs+ of experience in architecture design and development of large-
scale data management platforms and data application with simple solutions
•Must have extensive hands-on coding and designing skills on
- Java/Python for backend
- MVC (model-view-controller) for end-to-end development
- SQL/NoSQL technology. Familiar with Databases like Oracle, DB2, SQL Server, etc.
- Web Services (REST/ SOAP/gRPC)
- React/Angular for front-end (UI front-end nice to have)
•Expertise in design and management of complex data structures and data
processes
•Expertise in efficiently leveraging the power of distributed big data systems,
including but not limited to Hadoop Hive, Spark, Kafka streaming, etc.
•Deep knowledge and hands on experience on big data and cloud computing
technologies.
•Strong service architecture and development experience with high
performance and scalability
•Technical background in data with deep understanding of issues in multiple
areas such as data acquisition, ingestion and processing, data management,
distributed processing, and high availability is required.
•Strong on driving for results and self-motivated, strong learning mindset, with
good understanding of related advanced/new technology. Keep up with the
technology development in the related areas in the industry, which could be
leveraged to enhance current architectures and build durable new ones.
•Bachelor's degree in Computer Science or related technical discipline required.
Advanced degree is a plus.
•Strong leadership and team player.
•Strong skills on mentoring/growing junior people
•Payment industry experience is a plus.
Preferred Qualification
•Experience with ETL / ELT tools / applications
•Experience with Apache NiFi and Apache Spark for processing large data sets
•Experience on Elastic Search
•Knowledge on Data Catalog tools
•Experience in building Data Pipeline development tools
•Experience with Data Governance and Data Quality tools

Additional Information

Visa is an EEO Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status. Visa will also consider for employment qualified applicants with criminal histories in a manner consistent with EEOC guidelines and applicable local law.

Top Skills

Java
Python
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: San Francisco, CA
26,500 Employees
On-site Workplace
Year Founded: 1958

What We Do

At Visa, we are driven by a common purpose – to uplift everyone, everywhere by being the best way to pay and be paid. As our products and technology have evolved with the world, Visa remains ubiquitous, reaching new customers in new and often invisible ways. We are at the center of this digital revolution with a network that connects people with over 80 million businesses all over the world. And Visa’s network is expanding, accelerating our growth. Our resilient business model, with its strong track record of success, will provide you with amazing opportunities to grow in your career, as well.

We are looking for people like YOU. Come join a people-centric company where you can invest in your career.

For more information, visit visa.com/about, visacorporate.tumblr.com and @VisaNews on Twitter.

Why Work With Us

Our employees are our company. Creating an inclusive and diverse workplace has been our key priority. With our purpose to “uplift everyone, everywhere” as our guide, we’re building an environment where diverse backgrounds and perspectives are celebrated and drive success inside our company and out in our communities.

Gallery

Gallery

Similar Jobs

Bangalore, Bengaluru, Karnataka, IND
26500 Employees

Streamhub Logo Streamhub

Senior Data Applications Engineer

Artificial Intelligence • Information Technology • Analytics
Bengaluru, Karnataka, IND
24 Employees

CHEP Logo CHEP

Software Developer

Logistics • Other
Bangalore, Bengaluru, Karnataka, IND
6172 Employees

Target Logo Target

Principal Engineer - Data Engineering

eCommerce • Other • Retail
2 Locations
172344 Employees

Similar Companies Hiring

Jobba Trade Technologies, Inc. Thumbnail
Software • Professional Services • Productivity • Information Technology • Enterprise Web • Consulting • Cloud
Chicago, IL
45 Employees
InCommodities Thumbnail
Renewable Energy • Machine Learning • Information Technology • Energy • Automation • Analytics
Austin, TX
234 Employees
HERE Technologies Thumbnail
Software • Logistics • Internet of Things • Information Technology • Computer Vision • Automotive • Artificial Intelligence
Amsterdam, NL
6000 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account