HPC Operations Engineer

We have an opening for an HPC Operations Engineer to develop, deploy, and maintain the tools, automation, and processes that keep Livermore Computing high-performance computing systems available and performing at high levels of reliability and performance.

You will work in a challenging, team-oriented environment, applying technical knowledge of HPC systems and software development to engineer monitoring, alerting, and diagnostic solutions for Livermore Computing’s (LC) high-performance computing clusters, storage systems, networks, and data center facilities.

You will contribute to technical projects as a member of a technical team, and collaborate with system administrators, software developers, and facilities engineers to improve the reliability and efficiency of LC operations. This position is in the Livermore Computing Division within the Computing Directorate.

This position will be filled at either level based on knowledge and related experience as assessed by the hiring team. Additional job responsibilities (outlined below) will be assigned if hired at the higher level.

This position requires full-time on-site presence due to the nature of the work.

You will

• Develop, deploy, configure, and maintain monitoring, alerting, and observability tools for LC HPC clusters, storage systems, high-speed networks, and data center facility infrastructure.
• Create and maintain software tools and processes that streamline alarm response, diagnostics, and routine operational tasks using established methods and procedures.
• Configure and integrate data from building management and facility systems, such as power and cooling, with HPC system monitoring to provide a unified operational view under established technical direction.
• Share alarm response and monitoring duties with operations staff, and analyze, diagnose, and resolve hardware, software, network, and facility problems of limited complexity, determine root cause, and implement corrective actions in collaboration with system administrators, vendors, and facilities staff.
• Participate in the deployment, testing, and validation of new HPC systems and components, and the decommissioning of retired systems.
• Develop and maintain technical documentation, procedures, and runbooks for operational tools and processes.
• Respond to customer problems and questions, working with Hotline, system administration, and other LC groups to improve the quality of service for end users.
• Perform other duties as assigned.

Additional job responsibilities, at the SES.2 level

• Design and develop moderately complex software tools and automations that improve the operation, reliability, and efficiency of LC computing and facility systems.
• Analyze system, alarm, and incident data to identify trends, and implement moderately complex strategies to improve monitoring coverage, alarm accuracy, and operational processes.
• Lead discrete technical projects or work packages, such as the deployment, validation, and decommissioning of HPC systems, coordinating tasks with system administration, facilities, and other LC groups.
• Diagnose and resolve moderately complex to complex system and facility problems and serve as a technical escalation resource.
• Develop and deliver training and provide guidance and informal mentoring to less experienced staff.
• Independently prioritize assigned work across multiple concurrent operational and project tasks.
• Ability to secure and maintain a U.S. DOE Q-level security clearance, which requires U.S. citizenship.
• Bachelor’s degree in computer science, computer engineering, or related field or the equivalent combination of education and related experience.
• Fundamental knowledge of and experience with Linux/UNIX systems and enterprise data storage, including installation, configuration, networking, RAID concepts, system logs, and troubleshooting.
• Understanding of programming and scripting languages, such as C/C++, Rust, Go, Python, or bash/csh/ksh.
• Experience with version control and configuration management systems, such as Git, Ansible, or CFEngine.
• Fundamental knowledge of system monitoring, alerting, and incident management practices and tools.
• Sufficient verbal and written communication and interpersonal skills necessary to collaborate effectively with system administrators, software developers, facilities staff, LC staff, and end users, and to present and explain technical information.
• Ability to work all shifts, including Owl shift (12am – 8am), weekends, and holidays.

Additional qualifications at the SES.2 level

• Comprehensive knowledge of HPC environments and technologies, such as Linux clusters, InfiniBand, Slurm, Lustre, and archival storage systems.
• Proficient experience developing software tools and automation in C/C++, Python, or similar languages, and broad application of technical principles to resolve moderately complex system, network, and facility issues.
• Experience leading technical work packages or discrete projects and mentoring less-experienced staff, and ability to work with limited direction in a dynamic environment with competing priorities.
• Relevant previous experience and demonstrated ability to perform independent technical work.

Qualifications

We Desire

• Experience with monitoring and observability tools, such as Grafana, Prometheus, Splunk, or Nagios.
• Experience with data center utility infrastructure and building management systems, including power distribution, water distribution, and cooling distribution units supporting HPC environments.
• Experience with virtual machines, container technologies, and orchestration, such as Docker, Podman, or Kubernetes.
• Experience working in a Closed Area or other secure computing environment.

Pay Range

$121,830 - $185,544 annually

$121,830 - $154,500 annually for SES.1 level

$146,340 - $185,544 annually for SES.2 level

This is the lowest to highest salary we in good faith believe we would pay for this role at the time of this posting; pay will not be below any applicable local minimum wage.

An employee’s position within the salary range will be based on several factors including, but not limited to, specific competencies, relevant education, qualifications, certifications, experience, skills, seniority, geographic location, performance, and business or organizational needs.

#LI-Onsite

Position Information

This is a Career Indefinite position, open to Lab employees and external candidates.

Why Lawrence Livermore National Laboratory?

• Included in 2026 Best Places to Work by Glassdoor!
• Flexible Benefits Package
• 401(k)
• Relocation Assistance
• Education Reimbursement Program
• Flexible schedules (*depending on project needs)
• Our values - visit https://www.llnl.gov/inclusion/our-values

Security Clearance

This position requires a Department of Energy (DOE) Q-level clearance. If you are selected, we will initiate a Federal background investigation to determine if you meet eligibility requirements for access to classified information or matter. Also, all L or Q cleared employees are subject to random drug testing.

Q-level clearance requires U.S. citizenship.

Pre-Employment Drug Test

External applicant(s) selected for this position must pass a post-offer, pre-employment drug test. This includes testing for use of marijuana as Federal Law applies to us as a Federal Contractor.

Wireless and Medical Devices

Per the Department of Energy (DOE), Lawrence Livermore National Laboratory must meet certain restrictions with the use and/or possession of mobile devices in Limited Areas. Depending on your job duties, you may be required to work in a Limited Area where you are not permitted to have a personal and/or laboratory mobile device in your possession.

This includes, but not limited to cell phones, tablets, fitness devices, wireless headphones, and other Bluetooth/wireless enabled devices.

If you use a medical device, which pairs with a mobile device, you must still foll