Software Engineer, ML Systems

Job Description: Software Engineer, ML Systems Location: San Jose, California, United States (On-site, 5 days per week; relocation support available) Employment Type: Full-time Experience: 1–15 years Position Overview We are seeking an ML Systems Engineer to optimize the performance and efficiency of large language model (LLM) inference powering an agentic AI platform.

This is a hands-on, low-level systems optimization role focused on improving LLM throughput, latency, and cost efficiency in production environments. You will design and optimize inference systems across multi-node clusters, build benchmarking and evaluation frameworks, and work closely with research and engineering teams to bring advanced optimizations into production.

Your work will directly impact the responsiveness and cost-efficiency of AI systems used by leading semiconductor companies.

Key Responsibilities

• Design, deploy, and optimize LLM inference systems across multi-node clusters to maximize throughput and minimize latency for production workloads.
• Implement, benchmark, and validate concrete inference optimizations.
• Profile and analyze inference bottlenecks at the systems level, including GPU kernel execution and memory bandwidth constraints.
• Build robust evaluation harnesses and benchmarking frameworks to measure accuracy, throughput, latency, and resource utilization across parallelism strategies.
• Collaborate with research scientists to integrate new model architectures and optimizations into production inference infrastructure.
• Evaluate and apply relevant techniques from research papers and open-source projects to continuously improve inference performance.
• Work directly with customers to clarify technical requirements and represent the organization professionally.

Required Qualifications

• 1–15 years of experience as a Software Engineer or ML Systems Engineer.
• Experience optimizing performance for high-performance computing (HPC) and machine learning workloads.
• Strong proficiency in Python and C++/CUDA.
• Hands-on experience with SGLang, vLLM, PyTorch, or similar inference frameworks.
• Experience building compiled native software deployed customer-side using C, Rust, or Go.
• Bachelor’s, Master’s, or PhD in Electrical Engineering, Computer Science, or a related field.
• Software engineering experience at hardware companies, such as EDA, electrical engineering, or consumer electronics hardware organizations.
• Ability to work on-site in San Jose, California, full time; relocation support is available.
• Strong communication skills and ability to work directly with customers.

Preferred Qualifications

• Experience engineering core engines, algorithms, or performance-sensitive software at EDA companies such as Synopsys or Cadence.
• Experience developing on-device ML models or intensive numerical computing for devices such as Google Pixel or iPhone.
• Experience working with C++ performance models and simulation libraries.
• Experience in numerical computing or scientific computing.