Remote Backend Pipeline Engineer at CloudLinux at CloudLinux

About CloudLinux and Imunify360

CloudLinux is a remote-first, global company dedicated to providing high-volume, low-cost Linux infrastructure and security products. Our mission is to enhance operational efficiency for companies worldwide, guided by principles of doing the right thing, prioritizing employees, and embracing a remote-first culture. We foster a supportive environment where every team member contributes to collective success. Learn more about us at cloudlinux.com.

Imunify360 Security Suite, a flagship product of CloudLinux Inc., stands as a leading security solution for hosting providers, renowned for stability. Designed for shared and VPS/dedicated servers, Imunify360 offers an automated, user-friendly, six-layer approach to deliver comprehensive and complete attack prevention.

About the Role: Pipeline Engineer

We are seeking a highly experienced Backend Pipeline Engineer to take ownership of our automated pipelines. These critical systems transform real-time threat intelligence into protective measures for tens of millions of websites globally. When a new vulnerability emerges, our automated chain must detect it, acquire the source, generate Web Application Firewall (WAF) rules, create tests, validate rule effectiveness without impacting legitimate traffic, and deploy it across our vast network. The system also actively monitors production, with the capability to automatically roll back if issues arise.

This existing, 24/7 pipeline is essential; any unreliability could compromise customer protection. We need a strong Engineer to rapidly expand this infrastructure, ensuring it remains reliable and transparent. This role involves solving complex engineering challenges, working with data at scale, and navigating unexpected scenarios. This is a builder's role, focused on creating resilient systems with sophisticated underlying logic that are externally boringly reliable.

This position is fully remote with flexible hours, offering the freedom to plan your day and work from anywhere in the world.

What You'll Own

• Automated Protection Pipeline: End to end ownership of the multi-stage, largely autonomous chain from threat intelligence to validated and deployed rules, required to complete within daily fixed time windows.
• Progressive Release Automation: Implementing controlled, staged rollouts of protection across our fleet, with automated guardrails that can halt or roll back a stage without human intervention.
• Quality Gates: Establishing systems to determine from live production signals if a deployed feature is causing harm, and acting proactively before it progresses further. Balancing the risks of missing a problem versus over-correcting.
• CI at Scale: Developing validation processes that provision real, disposable environments across a wide matrix of software versions and configurations, delivering trustworthy verdicts quickly within release windows.
• LLM Orchestration and Cost Control: Managing AI agents within purpose-built harnesses for various subsystems, including robust evaluation, budgeting, and spend accounting as core components.
• Observability and Alerting: Building systems that self-report their condition, confirm health, and escalate issues autonomously. This operates on a petabyte-scale threat intelligence store and live telemetry from over 60 million websites, with strict end to end latency budgets measured in hours.

Key Responsibilities

• Design, build, and operate the automated pipelines described above, from conception to deployment.
• Transform fragile, multi-stage batch jobs into resilient, resumable, idempotent, and observable systems with clear state machines and recovery paths.
• Define and enforce latency budgets and Service Level Objectives (SLOs) for each stage, ensuring violations are visible and actionable.
• Develop the observability layer, including metrics, dashboards, alerting, and health gates, allowing the pipeline to report its own status.
• Design and implement robust guardrails: automatic hold and rollback, blast radius limits, kill switches, and safe by default behavior for upstream dependency failures.
• Engineer low maintenance systems by eliminating manual steps, reducing the need for human supervision, and simplifying the operational surface.
• Write and maintain unit and integration tests for complex logic involving concurrency, partial failures, external API flakiness, and multi-stage state.
• Investigate and resolve intricate issues across technologies like ClickHouse, GitLab CI, S3/object storage, Prometheus/Grafana, and third party APIs.
• Collaborate with security analysts and the Server team on architecture, challenging designs that may not withstand production realities.

Requirements

• 5+ years of professional experience in backend, platform, or infrastructure engineering.
• Demonstrable experience building and operating multi-stage data or automation pipelines, such as CI/CD systems, ETL/ELT, build and release automation, job orchestration, or ML/data platforms. This is the most crucial requirement, and we will ask for a detailed walkthrough.
• Deep expertise in at least one of Python, Go, or Rust. We use all three, and proficiency in any one will demonstrate the required engineering depth.
• Strong systems design judgment, prioritizing thoughtful architecture over raw coding volume. The core challenge is deciding what to build, anticipating failure points, and ensuring self-correcting systems.
• Practical experience with workflow orchestration and job scheduling tools that you have deployed in production (e.g., Airflow, Temporal, Prefect, Dagster, Argo, or custom schedulers).
• An intuitive understanding of reliability engineering principles: idempotency, retries with backoff, exactly once vs at least once processing, checkpointing, resumability, graceful degradation, backpressure, and safe handling of partial failures.
• Hands on observability experience with Prometheus/Grafana, LGTM stack, or similar, including designing metrics rather than solely consuming dashboards.
• Extensive CI/CD experience, ideally with GitLab CI (including dynamic/child pipelines and self-hosted runners) and comfort with Docker and container based test environments.
• Experience with object storage (S3/Ceph or equivalent) and large scale analytical stores (ClickHouse or another columnar database).
• Comfort designing state machines and long running processes that tolerate restarts, and reasoning about concurrency across multiple in flight rollouts.
• Excellent debugging skills across system, network, and data layers.
• Strong communication skills and comfort collaborating in a distributed team.
• Proficiency in spoken and written English.

Nice to Have Skills

• Experience with progressive delivery: canary and percentage based rollouts, feature flags, automated rollback, and blast radius control.
• Experience running AI/LLM systems in production, particularly related to cost control, token accounting, evaluation harnesses, and managing non deterministic components within pipelines.
• Experience with fleet scale telemetry and building quality gates using noisy production signals.
• Familiarity with WordPress, PHP, or WAF/ModSecurity concepts.
• Experience with configuration management (Ansible, Puppet, Salt) and Linux service operations.

A cybersecurity background is not required. The primary challenges in this role involve orchestration, reliability, correctness under concurrency, and observability. Domain knowledge is learnable, and we have specialists to provide it; pipeline engineering judgment is what we cannot substitute.

What We Value in Engineers

• Curious and Fearless Problem Solvers: Eager to investigate existing systems, identify root causes, and propose improvements.
• Skeptical by Default: Questioning assumptions, seeking falsification, and trusting measurements over plausible reasoning.
• Pragmatic and Detail Oriented: Focused on building reliable, maintainable systems, and adverse to solutions requiring manual intervention.
• Owners: Comfortable being accountable for the correct operation of a pipeline.
• Effective Communicators: Able to articulate ideas clearly, provide constructive feedback, and foster team collaboration.
• Engaging and Proactive: Contributing energy, initiative, and a positive presence to strengthen team culture.

Benefits

• Focus on professional development and challenging projects.
• Fully remote work with flexible hours, allowing you to work from any location worldwide.
• 24 days of paid vacation per year, 10 national holidays, and unlimited sick leaves.
• Compensation for private medical insurance.
• Reimbursement for co-working spaces and gym/sports memberships.
• Education budget for continuous learning.
• Opportunity to receive a reward for patentable innovative ideas.

By applying, you consent to the processing of your personal data as outlined in our Privacy Policy.