Senior ML Engineer at CloudLinux

About CloudLinux

CloudLinux is a global remote-first company driven by its principles: do the right thing, employees first, we are remote first, and we deliver high-volume, low-cost Linux infrastructure and security products that help companies to increase the efficiency of their operations. Every person on our team supports each other and does what we can to ensure we all are successful. Learn more at: https://cloudlinux.com/

Imunify360 is CloudLinux's security suite for Linux web servers. It provides real-time and background malware scanning, automated cleanup, a Web Application Firewall (WAF), runtime PHP protection (Proactive Defense), a database malware scanner, and IP reputation services. This system runs on hosting providers' shared, VPS, and dedicated servers, allowing new web-malware campaigns to be identified early within our fleet. We maintain one of the largest labeled corpora of live web-server malware in the industry, collecting files, verdicts, cleanup outcomes, and analyst decisions daily. This data drives our signatures, heuristics, and Machine Learning models.

We are searching for a Senior Machine Learning Engineer to transform this rich data into effective detection systems, including classifiers, LLM-assisted triage, and automatically generated signatures. The success of this role will be measured by the product's false-negative and false-positive rates, rather than notebook accuracy. Our dataset specifically focuses on web malware, including PHP, JavaScript, HTML phishing kits, infected WordPress and Joomla plugins and themes, and database injections, providing context around each file.

What You Will Do

The primary objective is to achieve 10x greater malware detection while maintaining the same or lower false-positive rate. This requires moving beyond hand-written rules and faster pipelines, suggesting that models trained on our proprietary data are the path forward.

• Determine which files on a server require deeper inspection, while staying within strict CPU and memory budgets.
• Differentiate malicious code from benign code at fleet scale and at a sensible cost per file.
• Automatically generate detection signatures from newly identified campaigns.
• Detect malware that appears malicious only within context, such as phishing pages and script injections.
• Direct each file to the correct detection engine.
• Ensure cleanup is precise so that affected websites remain clean.
• Prove that every model produces genuine improvement in detection before it is deployed.

How We Work

This role offers an experimentation track, access to GPUs, and a direct line to leadership. In return, we expect data-driven outcomes, not opinions.

• Remote and Asynchronous: We operate remote-first, spanning time zones from UTC-5 to UTC+8. Decisions are documented using ADR-Lite and are public by default.
• Hardware Driven: Training initiatives begin with dedicated GPUs on our own bare metal. Leadership is committed to funding further resources once the case is proven with data.
• Agentic Engineering: Tools such as Claude and Codex are the standard. They are integrated with GitLab, Jenkins, Redash, Jira, and Slite via MCP servers. We analyze how engineers utilize agents, focusing on context, specificity, and review quality.
• Close to Ground Truth: You will collaborate daily with malware analysts, detection engineers, infrastructure engineers, and the broader ML team.
• Ship Gates: No model or build will proceed if it increases false positives or false negatives when tested against our past error sets. Blameless post-mortems are completed within 24 hours.

Deliverables

You will own the entire lifecycle from raw data samples to a production model and quantify its impact.

• First Month: Deliver a first model trained on production data and validated against current engines, focusing on either suspicious-file triage or a metadata model. The evaluation harness must be in place, utilizing splits by time and campaign to prevent data leakage, along with a regression set from past false positives and false negatives.
• By Month 3: Have several models deployed, including some in shadow mode for evidence gathering and others actively producing real detections, all governed by the past-error regression set. A training setup must be operational to keep GPUs utilized. A written proposal (ADR) for the detection cascade, detailing cost per file and hardware requirements, must be completed.
• By Month 12: Models become a primary source of new detections, measured by their share of new detections and the trend in false negative rates. This includes retraining on fresh analyst and false negative labels at a fixed cadence and the generation of signatures within the analyst workflow. You will also lead two or three parallel training directions that other engineers can leverage with your established tooling.

Requirements

What You Bring

• Five or more years of experience building Machine Learning systems that function in production, including at least two years within an adversarial domain such as security, abuse, fraud, or spam.
• Proven ability to train and fine-tune transformer models yourself, covering data preparation, code tokenization, LoRA/QLoRA or full fine-tunes, distillation, and quantization, rather than just calling hosted APIs.
• Depth in classic Machine Learning, including gradient boosting, linear models on sparse features, calibration, handling class imbalance, and making sound judgments about when classic models outperform LLMs.
• Strong evaluation discipline: ensuring leakage-free splits by time and family, precision at a fixed false-positive budget, and maintaining an honest account of how production will differ from offline results.
• Expertise in Inference cost engineering, including measuring throughput and latency to optimize models for CPU or memory constraints (e.g., using ONNX, llama.cpp, vLLM, int8/int4).
• Strong proficiency in Python, PyTorch, and the Hugging Face stack, along with experience using SQL on large analytical tables like ClickHouse or similar databases.
• Familiarity with PHP or JavaScript, or a strong willingness to learn them while collaborating with our malware analysts.
• Demonstrated daily use of AI coding agents and the ability to articulate how you do this.
• English at an upper-Intermediate (B2) level or higher, with particularly strong written communication skills essential for asynchronous work.

Strong Plus Qualifications

• Experience with web malware, webshells, phishing kits, or WordPress security.
• Focus on models over code, such as AST or token features, code LLMs, or graph models applied to program structures.
• Experience with malware clustering and similarity analysis.
• Proficiency in multi-GPU training on personal bare metal setups, and familiarity with MLflow or Weights & Biases for versioning.
• Experience with Go or Rust for implementing on-server inference solutions.
• Published work or conference talks related to Machine Learning for security.

Benefits

• A dedicated focus on professional development.
• Engagement in interesting and challenging projects.
• Fully remote work with flexible hours, allowing you to schedule your day and work from any location worldwide.
• Paid 24 days of vacation per year, plus 10 days of national holidays and unlimited sick leaves.
• Compensation for private medical insurance.
• Reimbursements for co-working spaces and gym/sports memberships.
• A budget allocated for education.
• An opportunity to receive a reward for proposing the most innovative idea that the company can patent.

By applying for this position, you consent to the processing of your personal data as detailed in our Privacy Policy: https://cloudlinux.com/candidate-privacy-notice. Apply for this job