Once again, Yalla DevOps turned out to be a great opportunity to meet and hear about what’s new in the DevOps world. This community conference, organized by JFrog, was held on Thursday, July 8 at the lovely Tel-Aviv University campus. Here’s a nice video to feel the vibes:
This blog post will provide a summary of highlights and insights from that busy day. It is of course subjective, focusing on what caught my attention. I apologize in advance if I’ve missed any important insight or misrepresented the information provided at the conference in any way. In this case, instead of shooting the messenger (that’s me :)), you can shoot me an email: [email protected] and I’ll be sure to look into it and correct as needed.
In addition to the great lectures I’m summarizing here, it should be mentioned that the event also hosted great companies that showcased cool products and solutions in the DevOps domain, such as Granulate,
env0, Replicated, DoubleVerify, Spectral, Coralogix, Simloud, and minute media. You can find the complete sponsors list here.
Image source: Yalla DevOps – env0’s swag – summer treats
Some Trends to Get Us Started
Shlomi Ben Haim, JFrog CEO, got us started with a trend that involves the talents surrounding DevOps and their areas of expertise. It seems that these days, expectations from a DevOps engineer are higher, companies are looking to hire full-Stack DevOps talent rather than a “Kubernetes specialist” or a “Jenkins expert”. Such talent will be able to oversee the entire ecosystem and vision the Ops as a whole.
Another trend that was discussed (don’t be too surprised) is the cloud. Shlomi presented a JFrog survey showing that the world is rapidly marching towards the cloud. If two years ago, the number of companies using the cloud in some way in their production was 30%, this number doubled, rising to 60% in 2021.
However, some ecosystem players are not seeing the same rise. For example, Jenkins saw a decrease of 14%, coming to a (still very high) market share of 62%, as the market is looking for DevOps end-to-end solutions and is less willing to settle on solutions that are only CI or only CD.
As for the trend, or buzzword, of DevSecOps: Shlomi views it as a term which is still a fluff – if there are no automatic tools that are considered reliable by both development and security, closing significant gaps without the need for manual mitigations, this would still be a fluffy domain, but one that would attract new tools and products.
Shlomi invited Shay Banon, co-founder and CEO of Elastic, to join the talk and discuss open source and DevOps. As an inspiring leader, having been frustrated by commercial products out there (back in 2004) and overseeing a top open-source product from its first stages (the first lines of code were written by Shay himself), Shay was a good fit for this discussion.
After sharing his enthusiasm with the world of open-source (a world where if you have a problem and you open a defect report in Bugzilla, or if you ask a question in the user group, the code contributor would answer you, usually within a matter of hours, rather than the support rep who is not really into the code).
Shlomi asked Shay who in his opinion was more suitable to manage a software company: a technical person (like Shay) or someone from a business background (like Shlomi himself). The natural reference to their companies’ market share was, of course, not lost in this discussion (Elastic at 13.5B vs. JFrog at “only” 4.33B). As for the answer to that question, the jury is still out.
Quality and Velocity: A Journey to High Availability
This talk was given by Avital Tzubeli from Vonage.
Avital spoke about the journey made by Vonage, moving from up to 3 or 4 releases a year, to being able to release on a daily basis. Just like cooking, where the recipe might seem simple, but if you’re not careful you end up with a messy kitchen and a lame result – this move required a lot of work on testing of all kinds, from functional to load, stress, etc., new processes, and attention to details. You just can’t wing it.
Microservices Are ‘Easy’, Dependencies Are Hard: The Right Way to Build a Cloud Native CI/CD
Itiel Shwartz, CTO & Co-Founder at Komodor, focused on the question of how to break a monolith, not only at the code level (yes, we know how to do Microservices) but also at the CI/CD dependencies, build, testing and deployment levels.
Release Fast or Crash Your Satellite
Image source: Yalla DevOps panel
The following session was a panel hosting: Kfir Damari, co-founder and leader of the SpaceIL project, Omri Green from Grove Ventures, Fred Simon, co-founder and chief data scientist at JFrog, Dr. Adi Ninio Greenberg, a senior advisor and expert in the space industry, and Eliran Rubin from TLV Partners as the moderator.
This discussion around new markets and development aspects of the space industry was highly inspiring, despite not being directly connected to the DevOps world. Why was it inspiring? Because the engineering challenges solved in the space industry later affected other industries: from agriculture to materials engineering, 3D printing, batteries, food tech, hardware, software and more.
How do ventures realize a business model in the space industry where any project starts with a high investment before you see any return? And how does it all relate to DevOps?
Well… when you invest millions of dollars, you try to use the most up-to-date practices. You must have a good software process. And though many of the space projects initially started in the defense industry, which is more old school, new projects nowadays use real cutting-edge processes and technologies. Software updates during the mission, in production, are something that happens in the space domain, and they must work, as there is usually no second chance.
Adi discussed the challenges of having about 40,000 satellites on air within a decade, floating around the globe, owned by companies and governments, and having to control navigation and routing rules that would help avoid collisions.
An emphasis on the technical inspiration and education of the young generation should be made a mission that was part of the vision and actions of SpaceIL, Kfir concluded.
Deep-Dive Into Open Policy Agent (OPA) + Conftest + GateKeeper: Kubernetes Policy in Action
Shimon Tolts and Noaa Barki from datree.io took us on a journey through the challenges and solutions of managing policies in Kubernetes, including a glimpse at how they do it at datree in the policy enforcement domain, with the goal of preventing false or bad configuration from reaching production.
As we become more and more reliant on automated DevOps processes, the cost of errors might be anomalous. It’s no longer just a glitch here and there; we need strict monitoring and control.
Being Good Neighbors – Rate Limiting In a Serverless World
Boaz Ziniman from AWS provided tips that were more focused on AWS architecture, but are relevant to any distributed architecture.
Being good neighbors means using our resources wisely (even as a process you have to be aware of others – be a mensch, don’t over utilize, leave enough space for others).
Boaz started with the case study of Nielsen, processing 55T of data per day (presented as part of the great AWS series “This is my Architecture”).
Focusing on serverless and auto scaling, you have to correctly handle the following things: reserved concurrency, throttling, asynchronous flows and “fan out”. Waiting for a response from another service is both being paid as cloud$ and as waiting time. Use queues, such as Amazon sqs, or pub/sub, such as Amazon sns, or use an event bridge with proper throttling configurations.
Boaz discussed the problem of RDBMS connections, especially as lambda cannot do connection pooling! There are solutions for that: you can use a queue between the lambda and the DB and persist the data to the DB from the queue. Or you can use the new capabilities of RDS Proxy by AWS. You can also use caching solutions, such as Amazon ElastiCache. In addition, it is worth noticing that a hot lambda may use persisted storage of up to 500MB in its local /tmp, which could also be used for caching.
The next important tip was to always use timeouts and backoff retries!
Timeouts are important to avoid long waits and close processing if not relevant anymore.
Default timeouts (connection, db) are usually bad: either too long or infinite. Backoff retries means adding some time between retries, the recommendation is to add some randomness into the retry cycle, to avoid unintended correlated retry cycles, which tend to cause circular failures. And remember that configuring timeouts should be done as things that relate to each other.
Zombie Busters!
Shir Chaimi gave a very nice talk on how to handle a real problem and solve it by implementing your own solution (yes, it’s OK for DevOps to implement code!). She talked about how to automatically restore “Zombie” machines in a private cloud environment, by analyzing IPMI (actually, IPMI screenshots!). This solution, used at outbrain, currently supports 36 different issues. Mitigation options include: sending a signal to create keyboard input, releasing the machine from waiting for a certain user input, running CLIs, and automatically opening tickets for HW failures, such as failed memory or faulty disk.
Knative – Deploy and Manage Modern Container-Based Serverless Workloads
Recovering from the Zombies, Elad Hirsch, CTO at Agmatix took the stage with an excellent talk on Serverless (yes, it is a server, just being nicely hidden from you) and the Knative project for using K8S for serverless management and deployments.
When Prometheus Can’t Take the Load Anymore
Liron Cohen from Riskified followed presenting a remarkable methodical work of selecting the proper tool for scaling the usage of Prometheus for monitoring, examining three tools: M3, Cortex and Thanos.
Riskified chose Thanos, with the pros being: simple, well documented, supports both push and pull modes.
This choice is, of course, and as said by the presenter, related to the needs and considerations of their work environment. But the pros and cons for each tool were clearly explained.
Rethinking Observability for off-the-Shelf Tools
Next came Dexter Horthy, VP Customer Engineering at Replicated, who spoke about the challenge of bringing “on the shelf software” that is architectured for a cloud environment to an on-prem or private cloud environment, where many of the decisions are not yours.
How as a SW provider shall I provide installation for customers’ environment (on-prem, private cloud, etc.)? Do I need to provide my product with a bunch of bash scripts? Should I certify it with all sorts of deployment combinations? Dexter did the math – there are about 12K possible combinations today (platform, logging, monitoring, etc.) and growing…
The proposed approach is to let K8S work for you, to allow a hassle-free setup, which doesn’t sound like something new, but having it ‘cooked’ right is surely a challenge.
Lightning Talks
The event had eight lightning talks. All were short and good! You may see the abstracts in the agenda.
Some points taken from the lightning talks (from any of the 8… Sorry for not attributing points to specific speakers, it went so fast!, once you grasp an idea you are not sure from whose talk you got it):
- Services and code can be hacked. Codecov was hacked, or more specifically, its cloud storage, and there are still projects using the hacked version. Your pipeline is part of your code, so you must secure it and follow security alerts and updates.
- A lightning talk by AWS that is relevant for various scenarios: Multi accounts: using different cloud accounts to manage different environments or users, but then deploying to the different accounts using the same pipeline.
- Joining a new company as a DevOps manager: Where to invest first? Ask yourself: what scares my developers/ops the most? Start there. Things that are done manually often break. Analyze why.
- We talk about events and metrics in DevOps but we don’t use them enough, or correctly. DevOps code can be done in many ways, and different approaches can have enormous effects on the maintainability of your pipeline.
Closing Session – We Are the SRE Champions!
Ant(on) Weiss – SRE Advisor and Mentor, talked about the process of mentoring SRE champions, mentioning the details of the difference between DevOps and SRE (mainly, 15% raise in salary) and why (without being sexist) men tend to take additional job “titles” without extra pay while women usually decline.
And then there is the SRE bottleneck, solved only when you have a 50-50 split between Ops and Engineering tasks. To achieve that you want to have SRE champions in the team, mentored by an “embedded” DevOps engineer.
You should prepare a TRIP: Training, Responsibility, Involvement and Preparation.
And if it is a good TRIP you end up with an SRE champion in the team!
We ended the session singing Queen’s “We’re the Champions”.
Really, no kidding.
Anton prepared the YouTube beat as a closing, but as things always go, it didn’t really work in production, so we actually sang like nobody’s watching, which was a suitable ending to the session, and to this blog post if you ask me.
