The AI/ML Engineer - Senior Analyst is a critical role with the objective of exploring, developing, testing, deploying, enhancing, and maintaining Artificial Intelligence (AI) and Machine Learning (ML) capabilities integrated with Internal Audit’s (IA) management systems.
A broad understanding of AI/ML concepts and techniques, implementations of digital services via SDLC and Agile methodologies, Cloud Computing, experience engaging in end to end projects, and related aspects is essential for the role. Additionally, a key aspect of this role involves the design and refinement of prompts to effectively communicate with generative AI models and to guide them, optimizing their output for a variety of technical applications and producing high-quality and contextually relevant content.
The role entails collaborating with the Digital Solutions and Innovation (DSI) and the Strategy teams within Internal Audit, business audit Teams, Model Risk Management, external third parties and technology teams on AI/ML use case ideation, prioritization, developing plans and timelines for implementation, thorough testing, deployment in a systemic way, maintenance to keep the capabilities up to date and relevant, and support in the onboarding of new technology for DSI.
Another success factor for the role will be documentation of model performance and other metrics and communication/presentation of results to pertinent audiences, including Senior Leadership Teams. Along with a strong technical foundation, a creative, imaginative, forward-looking attitude and mindset is needed to continue to exert a positive impact on DSI and Internal Audit. Furthermore, a total quality assurance orientation is indispensable in all activities and outputs.
Key Responsibilities:
- Design, test, deploy, and maintain AI/ML use cases, and integrate with the relevant audit applications
- Prompt engineering to elicit the desired outputs from generative AI models, ensuring clarity, accuracy, effectiveness, and contextual relevance
- Perform analytics on AI/ML capability usage to understand trends, patterns, and pain points, and implement a sustainable feedback loop to ensure positive and continually improving user experiences
- Explain results in a methodical, appealing, and easy to interpret manner.
- Collaborate with teams on updates, enhancements, sprint releases, and other procedural requirements
- Prepare, manage, and maintain Model Risk Management (MRM) documentation on model performance and accuracy, and engage with the MRM team on adhering to and fulfilling model requirements
- Apply strong project management skills to develop project plans, timelines, milestones, and demonstrate progress toward objectives
- Utilize excellent communication skills to ideate, educate, and present internally, often at senior levels (across functions); expected to communicate with external parties
Qualifications:
- 4+ years’ experience in AI/ML model development and deployment projects
- Demonstrable experience with Google Cloud Platform (GCP), IBM Watson Services, or any well-known cloud solution
- Extensive experience in machine learning use case design, testing and deployment
- Strong statistical foundation, with in-depth knowledge of supervised, unsupervised, and semi-supervised machine learning models, with a particular focus on NLP/NLU
- Demonstrable experience in working with Generative AI models, with a deep understanding of their capabilities, prompting strategies, model performance, and inherent limitations
- Extensive experience testing and using APIs
- Possess an innovative, forward-looking, and imaginative mindset to continually reach for new horizons in Audit Innovation
- Demonstrable experience working with stakeholders and exceeding their expectations
- Take responsibility and accountability for execution and delivery
- Positive attitude and a change-driven and change-oriented outlook
- Adept at developing new ideas and improving current processes
- Ability to thrive in a fast-paced, results-oriented environment
- Excellent communication and interpersonal skills, with ability to develop strong working relationships and foster team collaboration
- Possess excellent writing skills to deliver clear and concise messages, able to articulate complex concepts
- Strong presentation skills – comfortable with public speaking across various forums and be able to effectively and logically communicate when ideas are being challenged in an open forum
- Experience in analytics, reporting, and visualization for the analysis and presentation of results; Cognos analytics experience is a plus
- Familiarity with Financial Services industry and/or Auditing is a plus
Knowledge and Skills:
- Analytics, Artificial Intelligence (AI), Machine Learning (ML), Natural Language Processing (NLP), Natural Language Understanding (NLU), Generative AI (Gen AI), Large Language Models (LLMs)
- Prompt engineering and tuning; experience with Retrieval-Augmented Generation (RAG) and similar techniques is desirable.
- Design of AI/ML model performance metrics and optimization
- SaaS environments, Cloud Computing. Information Security experience is a plus
- Agile Project Management; JIRA is a plus
- Intermediate to Advanced level Python coding
- Visualization in MicroStrategy, Power BI, Tableau, or other Business Intelligence software
- Intents, entities, dialogs, actions, and related chat bot concepts are a plus
- Knowledge on IBM products (Watson Suite, OpenPages, Cognos) is a plus
- Understanding of UI/UX concepts is a plus
Education:
- Bachelor’s degree in AI, Data Science, Computer Science, Engineering or related disciplines; Master’s degree in similar areas is a plus
- CISA, PMP, Agile/Scrum, Cloud Computing, or related certifications are a plus
------------------------------------------------------
Job Family Group:
Decision Management
------------------------------------------------------
Job Family:
Specialized Analytics (Data Science/Computational Statistics)
------------------------------------------------------
Time Type:
Full time
------------------------------------------------------
Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.
If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.
View Citi’s EEO Policy Statement and the Know Your Rights poster.
Top Skills
What We Do
Citi's mission is to serve as a trusted partner to our clients by responsibly providing financial services that enable growth and economic progress. Our core activities are safeguarding assets, lending money, making payments and accessing the capital markets on behalf of our clients. We have 200 years of experience helping our clients meet the world's toughest challenges and embrace its greatest opportunities. We are Citi, the global bank – an institution connecting millions of people across hundreds of countries and cities.
