Senior Infrastructure Engineer

Company: tastytrade Role: Senior Infrastructure Engineer Location: Chicago, IL – Hybrid (3 days/week in-office)   As a Senior Infrastructure Engineer, you’ll be part of a small team of engineers who are ultimately responsible for the production infrastructure which powers our retail brokerage business. We care about the speed and stability of our back-end systems as much as we care about the customer-facing applications we build. Our architecture is consistently evolving as our growing business faces new technical challenges, with a focus on practicality and performance. You'll help to maintain our infrastructure by working on existing configuration management code and troubleshooting unexpected issues. You'll work on building its future state by participating in the architectural discussions, building proof of concept systems, writing new configuration management code, and standing up new production systems.  Our infrastructure is a hybrid of self-hosted hardware at multiple colocation facilities and cloud-managed services. We leverage technologies such as SaltStack, Terraform, Docker, Nomad, and Vault.  The markets are open 24 hours a day, 7 days a week, and our customers are trading throughout. We engineer for durability, but occasional outages do occur. The infrastructure and development teams work together to troubleshoot and remediate these issues. There would also be very occasional off-hours or weekend work required for higher risk deployments or changes.  What You'll Do:  Use configuration management tools (primarily Salt and Terraform) to build and maintain a hybrid infrastructure hosted both at colocation facilities and in the public cloud.  Work directly with the development team to build supporting infrastructure for specific new application functionality.  Run proof of concept projects on early-stage infrastructure improvements to validate the feasibility of an approach, evaluate performance, and spike an implementation.  Manage a curated set of open source tools (PostgreSQL, RabbitMQ, Redis, etc.) via both configuration management tools and direct troubleshooting when required.  Manage telemetry (Graphite, Grafana, Prometheus), log aggregation (Elastic Stack), and monitoring tools (Checkmk, Nagios, Icinga).  Understand version control, create, and review pull requests as part of our mandatory peer review process.  Help maintain existing build/deploy pipelines for both configuration management and application development (GitHub Actions).  Implement security initiatives, identify risk, and mitigate or fix when required.  Who You Are:   You should have 6 to 10 years of experience as a System Administrator or Engineer.  You must be an expert Linux user with hands-on experience in shell scripting, process management, and troubleshooting in a production environment.  You must have experience with declarative configuration management tools like Chef, Salt, or Puppet as well as IaC tooling like Terraform.  You must be able to document processes and procedures clearly.  You should enjoy or at least be comfortable working with critical production systems where real revenue is at stake.  You must be able to work as part of a team, both giving feedback to more junior team members and taking direction from more senior team members.  You must be able to make good critical decisions balancing technology, frameworks, strategy, and practical constraints.  You must have knowledge of general networking principles and troubleshooting.  You must be able to learn quickly in areas where you don't have deep expertise.  Company Perks + Benefits:     Performance Bonuses     Stock Purchase Options     Medical/Vision/Dental Benefits   401k Plan    Annual educational / conference stipend Sustainable workday with minimal on-call responsibilities  20 Paid Vacation Days (plus an additional paid vacation day the month of your birthday!)     10 Paid Sick Days     Gym Membership Reimbursement     Commuter Benefits     Pet Insurance     W...