Monitoring and Observability Administrator

The Aerospace Corporation is the trusted partner to the nation's space programs, solving the hardest problems and providing unmatched technical expertise. As the operator of a federally funded research and development center (FFRDC), we are broadly engaged across all aspects of space- delivering innovative solutions that span satellite, launch, ground, and cyber systems for defense, civil and commercial customers. When you join our team, you'll be part of a special collection of problem solvers, thought leaders, and innovators. Join us and take your place in space.
At Aerospace, we are committed to providing an inclusive and diverse workplace for all employees to share in our common passion and aspiration - to carry out a mission much bigger than ourselves.
Job Summary
We are looking for an experienced Monitoring and Observability Administrator (Applications Administrator Staff III/IV) to join our Computational Services department. In this role, you will be the subject matter expert responsible for configuring and managing our enterprise observability tool stack which includes Splunk, Dynatrace and Nagios. This role will report into the Director of Computational Services and collaborate with cross functional teams that include Information Security, Enterprise Engineering, Networking, Data Center and Application Development. The ideal candidate should have hands-on experience implementing monitoring and observability solutions based on industry best practice. We are looking for a team player who has strong troubleshooting and problem-solving skills, is able to work independently and collaborate effectively in a team setting.
Work Model
This position will be a hybrid work model and offer partially remote, with the expectation of being on site in El Segundo, CA 60% or more (as enterprise needs arise) of the work week.
What You'll Be Doing

• Design, configure, implement, maintain, and document our enterprise monitoring and observability solutions which include Splunk, Dynatrace, and Nagios.
• Work with system and data owners to design, implement, and maintain integrations between observability technologies and the systems and data sources being observed.
• Work closely with infrastructure and application teams to design and build observability dashboards, alarms, and reporting, providing end-to-end insight into the status and health of monitored systems and platforms.
• Implement a monitoring program for applications and infrastructure, leveraging appropriate internal resources and third-party managed services.
• Within the context of the monitoring program, tune alerting and escalations to reduce false positives and non-actionable alerting and to escalate high-impact issues.
• Develop and implement strategies for cost control including tuning Splunk data volumes, optimizing the scope of monitoring, and leveraging open-source technology where appropriate.
• Lead ongoing migration efforts to migrate Splunk to SaaS.
• Maintain and upgrade the monitoring and observability systems and underlying infrastructure as needed to maintain cybersecurity and functionality.

What You Need to be Successful
Minimum Requirements for the Applications Administrator Staff III:

• 6+ years of expertise and hands on experience with installation, setup, configuration, maintenance (major and minor version upgrades) and tuning of monitoring and logging tools such as Splunk, Dynatrace, Nagios in hybrid environments (i.e. on premises and cloud instances)
• Solid understanding of IT infrastructure monitoring and observability best practices
• Splunk specific qualifications:
  • Experience with onboarding data, troubleshooting, and ensuring data availability with Splunk Universal and Heavy forwarders.
  • Generate and maintain search queries.
  • Experience developing log ingestion and aggregation strategies per Splunk best practices.
  • Understanding of Role-Based Access Controls (RBAC) within Splunk
• Dynatrace specific qualifications:
  • Experience developing Dynatrace dashboards for business processes.
  • Experience with configuration and customization of Dynatrace solution that includes integration with other tools.
  • Experience integrating Dynatrace API with other tools such as Dockers, Kubernetes, DevOps tools like Ansible, Jenkins
  • Experience with Synthetic monitoring and integration of third-party tools with Dynatrace.
  • Admin level tasks - Alert configuration, maintenance window, anomaly detection and changing threshold, custom event creation and agent upgrade/install.

• This position requires ability to obtain and maintain a security clearance, which is issued by the U.S. government. U.S. citizenship is required to obtain a security clearance.

In addition to the above, the minimum requirements for the Applications Administrator Staff IV:

• Experience in working with DevOps and agile methodologies.
• Proficient in developing and maintaining technical documentation, runbooks, and procedures.
• Knowledge of ITIL concepts and principles.

How You Can Stand Out
It would be impressive if you have one or more of these:

• Current TS/SCI clearance
• Experience working with an internal or service provider provided Network Operations Control/Security Operations Control (NOC/SOC)
• Experience migrating from Splunk on-premises to SaaS.
• Experience with gathering and organizing large amounts of data to use for instrumentation into an Enterprise monitoring solution Dynatrace or Splunk
• Dynatrace related certifications
• Splunk Core Power User, Enterprise Admin or Cloud Admin certification
• Programming skills in languages such Perl, Power Shell, Python, Bash or JavaScript
• Experience with Linux and Windows systems, including a solid understanding of system administration.
• Experience with automation tools such as Ansible, Puppet or Terraform
• Experience with container orchestration tools like Kubernetes.
• Experience with cloud platforms such as AWS, GCP, or Azure
• Experience in coordinating with vendors to resolve issues.
• Knowledge of security frameworks and standards (e.g., COBIT, NIST 800-53, ISO27001, SSAE16, SOC1, SOC2, etc.
• Knowledge of Syslog and network protocols

We offer a competitive compensation package where you'll be rewarded based on your performance and recognized for the value you bring to our business. The grade-based pay range for this job is listed below. Individual salaries within that range are determined through a wide variety of factors including but not limited to education, experience, knowledge and skills.
(Min - Max)
$100,805 - $170,000
Pay Basis: Annual
Leadership Competencies
Our leadership philosophy is simple: every employee, regardless of level and role, can demonstrate leadership. At Aerospace, our commitment is our people. To cultivate our talent and ensure that we have a strong pipeline of future leaders, we want individuals who:

• Operate Strategically
• Lead Change
• Engage with Impact
• Foster Innovation
• Deliver Results

Ways We Reward Our Employees
During your interview process, our team will provide details of our industry-leading benefits.
Benefits vary and are applicable based on Job Type. A few highlights include:

• Comprehensive health care and wellness plans
• Paid holidays, sick time, and vacation
• Standard and alternate work schedules, including telework options
• 401(k) Plan - Employees receive a total company-paid benefit of 8%, 10%, or 12% of eligible compensation based on years of service and matching contributions; employees are immediately eligible and vested in the plan upon hire
• Flexible spending accounts
• Variable pay program for exceptional contributions
• Relocation assistance
• Professional growth and development programs to help advance your career
• Education assistance programs
• An inclusive work environment built on teamwork, flexibility, and respect

We are all unique, from diverse backgrounds and all walks of life, yet one thing bonds all of us to each other-the belief that we can make a difference. This core belief empowers us to do our best work at The Aerospace Corporation.
Equal Opportunity Commitment
The Aerospace Corporation is an Equal Opportunity/Affirmative Action employer. We believe that a diverse workforce creates an environment in which unique ideas are developed and differing perspectives are valued, producing superior customer solutions. All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, age, sex (including pregnancy, childbirth, and related medical conditions), sexual orientation, gender, gender identity or expression, color, religion, genetic information, marital status, ancestry, national origin, protected veteran status, physical disability, medical condition, mental disability, or disability status and any other characteristic protected by state or federal law. If you're an individual with a disability or a disabled veteran who needs assistance using our online job search and application tools or need reasonable accommodation to complete the job application process, please contact us by phone at 310.336.5432 or by email at [email protected]. You can also review Know Your Rights: Workplace Discrimination is Illegal, as well as the Pay Transparency Policy Statement.