EP12: Exploring eBPF Cloud Native Security

Extended Berkeley Packet Filter (eBPF) allows us to tap into the kernel to implement monitoring, observability, networking, and security.  In this episode, we invited Chris Kranz and Liz Rice to discuss the usage and adoption of eBPF within Cloud Native solutions.

References

• http://www.brendangregg.com/
• https://nathanleclaire.com/
• https://github.com/iovisor/bpftrace
• https://ebpf.io/what-is-ebpf
• https://github.com/lizrice/ebpf-beginners
• eBPF for Windows: https://www.youtube.com/watch?v=LrrV-eo6fug
• Community: http://slack.cilium.io/

eBPF Summit 2021
https://ebpf.io/summit-2021/

Please visit our YouTube Channel to see Chris present in our June 2021 Gathering (monthly meet-up).

Guest Speakers

Chris Kranz

Chris supports the Sales Engineering team in EMEA at Sysdig, helping make cloud native easier and more secure for Sysdig customers. Before joining Sysdig, he spent time building microservices and cloud applications with various end users, and before that lived a life of cloud, virtualisation and storage!

https://www.linkedin.com/in/ckranz/
@ckranz

Liz Rice
Liz is focused on containers, cloud native technologies, security and distributed systems, and  heavily involved in open source as the chair the Technical Oversight Committee of the Cloud Native Computing Foundation (CNCF), and an ambassador for OpenUK.

https://www.linkedin.com/in/lizrice/
@lizrice

Your Hosts
Michael Man: https://www.linkedin.com/in/mman/
Glenn Wilson: https://www.linkedin.com/in/glennwilson/

DevSecOps - London Gathering
Keep in touch with our events associated with this podcast.

• https://www.meetup.com/DevSecOps-London-Gathering/
• https://twitter.com/DevSecOps_LG
• https://www.youtube.com/c/DevSecOpsLondonGathering

Transcript

[00:00:00] MM: [00:00:00] So welcome we're here on another episode of DSO Overflow. We have two wonderful guests here. Chris and Liz. So Chris, would you like to introduce yourself? 

Chris: [00:00:09] Thanks, Michael. My name is Chris Kranz. I work at Sysdig. I look after the sales engineering team here across EMEA. 

MM: [00:00:15] Super. 

Liz: [00:00:17] Hi , I'm Liz Rice. I am the chief open source officer at Isovalent  and I'm also the chair of the technical oversight committee at the CNCF. 

MM: [00:00:27] We had the pleasure of Chris presenting at the monthly DSO London Gathering this month. And he covered one of the CNCF projects Falco as well as the commercial spin to that provided by his company, Sysdig.

So that solutions based on a newish technology called EBPF so. Maybe let's dig into that a bit more. Chris, would you like to kind of explain at a very high level as our audience understands what this new tech. 

[00:01:00] Chris: [00:01:00] Yeah. So I think EBPF is, is an extension of something that's actually quite old.

BPF has been around for a long time, or was originally developed as a way of kind of making the, the networking more programmable, the E part of the EBPF as the extended. So it's giving it more functions and it's no longer just about the network. So if I kind of rewind on that one and moved to where it is now, it's kind of a, a way of, of taking a, a safe application or we can probably dig into more what safe means and actually embedding it directly in the kernel.

Now it's much more lightweight than the kernel module. It's much more dynamic than the kernel module, because you don't need to go through the whole kernel contribution, although all those sorts of rigmarole of that. And it's got that safe aspect, which I touched on. So it means that people can.

In a relatively structured way, create ways of tapping into this information feed that EBPF provides, which is not just networking these days. It's we can extend it into things like system calls other low level information. And it just gives you a nice way of doing that. Now [00:02:00] the other side of EBPF is that you don't need to write EBPFs program.

So I think there's a lot of great projects out there. You know, the, the isovalent  team, I've got a number of them have been contributing to a number of them as well. Which makes it easier for them to people to tap into those, to those data streams in user space. So you've got this rich, lightweight, high performance data feed for something that people can tap into relatively safely.

MM: [00:02:25] Liz, our understanding your 2021 prediction, you, you called out EBPF what what's attractive about this technology for you? 

Liz: [00:02:35] I mean, as Chris has just been describing EBPF is making the kernel programmable. There's a really nice analogy that Brandon Greg used, which is. Bringing to it is to the current or what JavaScript is to HTML.

So allowing you to run custom code inside the kernel and the can, or is there is one kernel per virtual machine and in the world [00:03:00] of cloud native in particular, we tend to think about containers. And we think about, you know, these little micro services and, and things changing dynamically. But all of those containers are running on these virtual machines.

There is one kernel per virtual machine, and if you can be running EBPF code. You know, in each channel, you've got visibility into everything that's happening across those machines. And that gives it huge power. And, you know, you can hook into these really low level events. Like there's a sort of, part of that.

EPF called XDP the express data path. So you can get hold of a network packet before it's even really reached the computer. It's still on the network interface. So the speed is astonishing. You can hook into. Pretty much any event that's happening across the entire machine. You can tie that into knowledge of what's happening across the machinery, or perhaps [00:04:00] you're combining information from the kernel with information about what's happening, your Kubernetes.

At the end of the day, you've got this amazing platform for building new networking tools, security tools, observability tools. With huge performance benefits and huge ability to watch what's happening across the entire system. There's really, you know, we're seeing a really exciting range of new capabilities being developed on top of EBP.

MM: [00:04:31] So let me just play back my, my simple version of that. So is it effectively a capability to allow to hook into the kernel so that we can avoid the apps, the various technology abstractions on from a compute point of view so that we can work out what we want to see and then. Detect or remediate or, or prevent.

Liz: [00:04:58] So I think there's an [00:05:00] interesting distinction between people writing EBPF programs, which are C code they're dealing with kernel level data structures that, you know, pretty complex beasts to deal with. And then the abstractions and the tools that we can use built on that technology. I don't think everyone, you know, there's going to be a smaller set of people who are doing.

E BPF code itself and a much larger people who are going to use the tools built on top of it. 

Chris: [00:05:28] I completely agree with what there's a say. I think it there's, there's a danger of scaring people off. If you say you're writing something for the kernel, if you have to write, you know, when you could have done 90% of what you can do with EBPF by writing a kernel module, how many people know how to write kernel module?

How many people know how to tap into that information stream? Whereas EBPF. Yes. You still need to know level, low level. See if you're going to write the EBPF module. But the real advantage there is the, is the people that have already written those modules that provide this data stream. So then all you're doing really it's like tapping into an API [00:06:00] and we all know how easy it is.

Querying an API. So saying, give me the information about what the network's doing. Give me the information about what. Either the entire system call or just this one system call. I want to know when anyone's reading this one file. That's all I want to know. So you've got that, that obstruction layer, but in a, in a safe, clean high-performance way.

So it's obstruction without the overhead of obstruction as you often get. 

Liz: [00:06:23] And in some cases you can even use EBPF to kind of update your kernel. There's a really great example of a security issue. Whether it was like a, a packet of death a packet arriving that was going to be parse where the pals and code had a bug and was unable to parse it correctly if the length was zero.

So if you sent this packet, Kind of will crash, you're doomed, but with EPF, you can dynamically load a program that intercepts that packet before it gets to this bug checks for the [00:07:00] zero length packet, drops it in that case. And you've basically patched that security issue dynamically in a very safe way.

And without having to do a whole kernel upgrade, which is you know, really powerful. 

MM: [00:07:14] Oh, right. So that's, that's interesting. Cause I limited knowledge with EBPF  I thought it was just a monitoring capability rather than a prevent having the preventative function. 

Liz: [00:07:29] So that ability to do prevention is very dependent on what type of EBPF program you're running.

So depending on where in the kernel you're hooking into, or even where in user space you're hooking into you may or may not be able to actually change behavior. In some cases, you can only really observe it and read data. In the case of network packets, you do have this ability to drop them. For example. You know, can read 

MM: [00:07:59] legitimate [00:08:00] route rootkit, 

Liz: [00:08:01] I suppose. So, I mean, another really interesting example of how we're using it in cilium is so cilium is a networking layer for Kubernetes, and that means it's assigning the IP addresses for pods as they appear. Now, normally in container networking, you've got a networking stack inside each container, and you've also got a networking stack in on the host.

And there's a virtual ethernet connection between your container and your host. So when you send a packet out of a container, it's got to go down one stack and then back through another stack to be sent, you know, externally from that machine. But. With cilium , because we know those IP addresses and we can intercept the packets at the, kind of at the, either at the socket or at the network interface level, we can say, oh, well, we, we can already see that this IP [00:09:00] address it's come from a container.

We know it's hosted elsewhere. It's on a different machine. So we can just send it straight to the external network interface. And we don't need to go through. The whole stack on the host. So it's really high performance. And yeah, that's a good example of something where you can manipulate behavior. I think the history of EBPF being about packet filtering means that there's a lot of things you can do at the networking layer.

Really interesting where the BPF 

MM: [00:09:30] yes. That's a very good use case on whatever you is, are there or use cases are there To manipulate this technology.

Chris: [00:09:39] Manipulate is an interesting word, but so I guess so you know, I can talk about, on what we're doing on the Falco side is much more taking that on these strings. We want to use this for observability, and this is one of the things, again, Brendan Gregg talks around is as observability for not just application performance, but also security.

It turns out that what applications are doing from a performance [00:10:00] perspective is very similar to what applications are doing for a security perspective, using that same data stream. We can time how long something happens, and we can also see if it's doing what it's meant. So that's where we use it more of a, of a data feed.

So we're just looking at that information. And then from Falco, we're reading all those system calls to diagnose security information. So are you, are you executing system calls that you're meant to be doing you're running processes, you run into certain areas of file system, even oversight on the networking.

I think on the net, when we talk about security, you've got the detection and the prevention that kind of two sides to it. Yes, you, of course, you want to be preventing, you want to be blocking or to be rooting accordingly, but you also want to be having that detection phase. So you can have a look to see what's going on to make sure it's doing what it's meant to be doing.

It's behaving as it's meant to be doing. So I think they're the two that, that the kind of the quite popular use cases, I think observability is a really interesting one because you can, you can start injecting. Well think about debugging and troubleshooting. If you're going to debug your production application, and it's a compiled application, you need to take it [00:11:00] offline, inject your debugging code, recompile it, publish it up to production.

Argue with change control. You can put debugging in production and then deal with the performance overhead. While you're taking that feed with the BPF. What you can do is dynamically inject. Some debugging code to say, when these certain system calls, when this sort of activity, when these certain conditions are met, create a timer, give me the events, give me a data, feed, whatever it is of the activity, that's point in time.

And then once you finish that unloaded. So you've got live kind of injections of this debugging code,  these observability, the troubleshooting stuff to do this without disrupting anything. And you don't really affect the application. You're not taking anything down. You're not affecting anything.

I think that's there. That for me, that's one of the really exciting parts of EBPF is it's completely. So Liz's point earlier, you know, that that live patching, it's not just upgrading the kernel , it's doing that in a live way. So I'm not changing anything. I'm just literally putting in a new EBPF program and suddenly I've patched the kernel [00:12:00] without restarting anything without rebooting anything.

I'm just dealing with those, that information in a different way now.

Liz: [00:12:06] Another great reason why I think EBPF  is good. Really revolutionize observability is you can, you don't need to instrument your application because your application is going to be making system calls. It's going to be asking the candle to do things on its behalf.

And if you can add the hooks, the observability hooks that you need in the kernel using your BPF, you just don't need to change anything about the application at all. And in fact, there's a Rooney nice like a prototype bumper sticker that Nathan LeClaire kind of drew and it's Basically my, my other sidecar is the kernel.

And it's really getting at this idea that you're no longer going to need to add sidecar containers into your Kubernetes pods, because you don't need to, you can [00:13:00] just add the instrumentation directly into the kernel dynamically as, and when you need it, turn it on and off as you need it to look at different applications.

Without having to affect anything about the application or even how the application is configured. It's I think that's a real game changer. 

Chris: [00:13:18] And I think extending on that as well is, is even if you have instruments at your codes, you can look for that instrumentation. So you can use, I don't think we're doing this yet, but what you can use is using BPF to look for those triggers, where you've instrumented certain signals and use that as the timing.

And that's actually going to improve where you've got that instrumentation with using open telemetry or something else you're already any sort of APM tool. But again, as Liz been talking, offload that to the kernel, which is much more high-performance than pushing it into users. So I think that's exciting.

It's something that we we've been doing it again with, with Falco and some Sysdig open source tools. For example, looking for certain certain traditional monitoring signals, we're looking for them being sent to local host. So [00:14:00] they don't even need to hit the network, but we can see there's a system called that, sending that signal outs and we can pick it up.

Even if it goes nowhere, we can still pick it up because you've got that. 

Liz: [00:14:09] Hmm. There's all sorts of really interesting events that you can hook into say for example, things like fileless execution. Yeah. Do you really expect to be running something that was in memory and that never hit disk? Maybe that's normal behavior for your application, but a lot of the time that's kind of, you know, a sign of something.

Yeah. Unexpected. And it's the kind of thing that you can pick up the signature of really BPF, pretty straightforwardly 

MM: [00:14:37] forensics. 

Liz: [00:14:39] Oh. And forensics is a whole other, you know, cause you can collect all this data, send it off to a SIEM. You know, if there is something you need to look back at later, you can get this really rich data.

Something that we've been experimenting with within, within Cilium is. Yeah, we've got all this [00:15:00] information about the network connections that are being made and you through EBPF have information about the process that's initiating that network connection. So you can provide all this forensic information about not just which pod was involved, but which container, which executable exactly which process ID everything about that.

You know, if you saw a suspicious looking network connection to some kind of command and control or crypto currency mining, Paul, you'd be able to find out what had initiated that connection CPU. 

Chris: [00:15:40] Yeah, it's something that we've been doing you know, assisting with the open source tools there as well is, you know, our founders heritage with Wireshark.

If you think about being able to dynamically take a Wireshark packet capture just based on detecting certain signals. And then extend that to say, actually, I want to take a system capture what I record the system, [00:16:00] calls the file activity, the network activity, based on the certain signals. And there's a saying you suddenly you're extending what you can do with forensics.

I think forensics tools in the past have been very SIEM base. And then leveraging the log data to, to go back and forensically build something. And you, sometimes you, you're only, you've only got as much information as you've got locks. And so it leads everyone to store all the logs forever. But here, instead, if you're looking for those certain or signal patterns, you can say, I want a store, lots and lots of information, but only when I see these certain signals only when I see this potentially undesirable behavior.

And then when you do find that you can dig really, really deep into exactly what's happening. Which I mean, I, I find that I kind of geek out on some of those captures and some of that information you can get into. Cause it's amazing. Are you actually, you know, I find myself learning literally just how the Linux works itself, you know?

Cause you see the calls that Linux is making in terms of opening files, your read, modify rights and how it's event based and all those sort of things. I learned a lot from Linux itself by analyzing these. 

[00:17:00] MM: [00:17:01] But presumably when, when you're starting to explore this this technology, it's a bit overwhelming because in my mind, it's almost like I'm going to switch it on and use it.

And then I've got this tsunami of information coming at me. So, and any tips or advice as to how to get started? 

Liz: [00:17:22] Th there are some good tutorials out there. There's, there's tons of information. So, for example, using tools like BPF trace, which gives you a higher level abstraction over these EBPF, you know, detailed C programs.

So with BPF trace, you've got this language that allows you to express things about what kind of events you want to hook into and, you know, are you counting things or what are you trying to do? But it's a good way of exploring the kind of capabilities. EBPF is going to open up a and there are tons of really good [00:18:00] examples.

So I would say that's a good place to start just to get a feel for the power of it. There's also the ebpf.io site, which has a ton of useful resources. It goes, it goes pretty deep. So there's, there's a lot of expert material there as well, but it, it will point you in the right direction. 

Chris: [00:18:19] I find the, one of the things that can be a little bit off putting, as, as you say, you, you, you actually look at EPF itself and you look at the sheer amount of data and complexity of writing SICO or pseudo C and, and, and having to do that, I would highly recommend anyone to stay away from EBDF itself.

And instead look at the helper apps that attached EBF, as Liz was saying, what MEPF trace, keep cuddled, trace. Things like Falco things like psyllium, they actually leveraged that because then you get a, you get a good idea of this type of data, even though you're not, maybe not indirectly interacting with EDPs, you get a good understanding of the sort of data that's available to it.

And you can start, you can start kind of working out different use [00:19:00] cases that maybe those projects don't. Don't cover today, or maybe you want to do some custom stuff, but you can still, all those projects will be using an existing EDPL library. That's already written already baked already.

Battle-hardened then you can use that as your data source. So I think that that can be quite powerful as well. So have a look at the, the, the open source projects that are already using EBP F as their base. I think that's an easier way to get started 

Liz: [00:19:24] if you do want to sort of, you know, if you're the kind of person who really wants to get into the nuts and bolts and you want to understand, like, how do these C code C programs work and how do they get injected into the carnal?

I have got a get hub repo called E BPF beginners, which has got a couple of examples and some like really hello world level. You know, that can get you through that conceptual. You know, barrier of like, okay, how do I get my, how is the VPF even getting the C code? Load into the current or how is that all working?

But I think for most people that is not [00:20:00] what they, they don't need to be aspiring to writing any BPF code, unless they're really wanting to get into the details. I don't want to put people off from doing that if they're really interested in it, but for most of us, it's not necessary. 

Chris: [00:20:13] I think there's a few kinds of obstruction libraries as well.

So if you're go programmer there's there's attraction library, so you can interface with the BPF I go or Python or other languages. So there's a lot of good work. I mean, like I say, BPF isn't particularly new. It's it's been with us for awhile. A lot of people have done the hard work of creating sort of extraction layers to make it.

But yeah, I think as Lisa, depending where you want to enter, there's multiple levels, you can get straight in that pseudo, see, you can go in that middleware way. I'm kind of writing into EDF EDPR providers, or I'm going into some of the open source or some of the community projects that leverage UPPF in there in that, in that core.

MM: [00:20:51] I saw an announcement last month, maybe a couple of months ago where Microsoft they've started a, an equivalent project. That sounds quite exciting. [00:21:00] So the, the usual windows versus Linux type discussion comes, comes to my mind. So it's great that they've started that. 

Liz: [00:21:09] Yeah, it's more of a coming together actually.

So I'm actually speaking with Dave Thaler  from Microsoft tomorrow on a video live stream that we do for EBP F and psyllium office hours. So he's gonna, he's the main driving force behind the EPF windows project. So 24 hours after we're recording this, I will know a lot more than I do now.

But but what I do know is the. The roots of this were from last year's EBPF summit people at the summit, you know, users saying, well, why can we only do this EPF thing on Linux? Why can't we also do it in windows? I think Microsoft heard that and thought, well, yeah, actually, could we bring some of these concepts into windows?

Some things are going to be different. So, you know, the kernel data structures are definitely [00:22:00] very different, but there's a lot in common. They're reusing lots of community open source projects like the, the jet compiler and the verifier existing projects that they've been contributing back into, or they've been enhancing and adding those contributions back upstream and.

There's definite sense that I'm getting from that project that they want to make it as much in common between the windows implementations and the Linux implementations as, as possible. So maybe we will start to see some higher level obstructions that can run on both. And that would be amazing. 

Chris: [00:22:37] I think it will be really interesting to see.

I think from the first pass that I've seen of it is it's actually, they, as we've talked all about EDPs being in the kernel and being the core of it, it looks like the first Microsoft first iteration of that is all in user space. So it's taking an abstraction and then presenting it up. So interesting. I mean, they, they're going to have to evolve that and develop it more for it to give that the full functionality.

[00:23:00] But as they said, the first iteration of that, maybe I'm wrong. But the first iteration of that, I think it's exciting because it does give that abstraction layer, which is accessible to different 

Liz: [00:23:07] people. I think the difference is they're, they're running the verification and the compilation in user space, but they do actually hook the event in the kernel that the end result should be reasonably equivalent.

Yeah. 

MM: [00:23:21] PR presentation presentation, they say in your talk yesterday, Chris we briefly touched on the UI aspects of, of seeing the outputs from EBP F tools. Do, do you want to talk a bit more about that? 

Chris: [00:23:38] So I guess, yeah, that was, that was where I was, I guess the, the, the the topic there was around Falco sidekick.

So taking the, the output. So what we do within Foucault is when you get the events being triggered, we generate them in, in, you know, in Jason, which is programmatically. You can read them and you can manipulate them. And that's great from a machine perspective, not particularly useful, if you want to visually.

And I think that's, that's where you get [00:24:00] into focus sidekick, which takes that event stream of activity that's happening. And just give you a quick and easy way of visualizing it again. Liz was talking about that in terms of taking activity and events and putting it into something, putting into a SIM.

And that's the, I think that's the important part of it. You need, the information needs to be programmatically, right? But then the visual aspect where you make sense of it needs to be something that's a pretty UI. And I think that's the challenge of every great engineering project is it's, it's fantastic technology, but how do I make it easy to understand?

We are dealing with millions of data points you know, potentially per second, not often that high, but you know, you're, you are talking, talking millions, certainly millions per minute, minutes per hour. And then just making context of that is really important. Again, something that Liz touched on in terms of.

Understanding. Okay. This particular activity happens, but it happened to this pod that was in this, even at these namespace, in this Kubernetes cluster on these hosts, whatever it is, that's the context you need, rather than just saying, I received a network packet from this IP to this source. [00:25:00] That's no longer useful in the Kubernetes world.

Cause I have no idea what my source IP is or my destination IP is so can probably 

Liz: [00:25:07] changed since, you know, 

Chris: [00:25:10] it actually happened. Yeah, yeah, yeah, absolutely. It should have changed. So that's where you kind of, you double up the, the forensic side and the observability side. Cause I want to know. A point in time, what happened?

We're coming from a world where, as Liz just said that the IP is we're the same for, for years because I stood up my server. I gave it a name that I liked and I fixed like the address and it walked away and come back five years later, I know that's called Homer Simpson and that's the IP and Kubernetes, I shouldn't know the name or the IP address.

I should be able to access a programmatic, but I still need to be able to make sense. And that's where the obstruction layer, but combining it with the metadata is really important because I need to visualize it in a way that's easy to understand. 

MM: [00:25:53] Do you see this as a day two adoption for enterprises?

[00:26:00] Liz: [00:26:00] So I'm going to say that in the part of psyllium, you need a connectivity layer for your Kubernetes cluster anyway, and you have a choice. A number of different networking solutions. And you can change it. You know, it's not like once you've certainly on, on day two, you have the option to change. Maybe on day 10, you've kind of locked yourself into some particular capabilities that are specific to your particular networking solution.

But I think something that we've, that we've seen with with psyllium is people experiment. You know, if you're running mini cube, you, you probably. Yeah, you might be using weave net for example, but you're going to get to a point where you need, you know, a really scalable and capable networking solution.

There is no reason why you shouldn't run something like sodium that's CPF based from day one. And at day one, you can, [00:27:00] it's just as easy to tread waters. So if you want to take advantage of APPF that, you know, that's quite a nice thing to do on day one, but you don't have to. I think from a sort of security side of things, all the things like, you know, network policies and you know, Falco rules, you're going to want to start looking at those relatively early on in your enterprise production life cycle, but you don't need to, it doesn't need to be the apps.

First thing on your list. 

Chris: [00:27:30] Yeah. I mean, I guess philosophically I'd like to see people do it from day one. Because, because I think just, just look at the, the, the number of Kubernetes clusters exposed to the internet, the number of people running outdated people at these classes, all those sort of things.

And I think people, the danger of saying it's a day, two thing is day two quickly becomes day 10 quickly becomes day 100. So I think we, we want to be embedded in it and there's no, you don't have to use it. But if it's there from day one, you've got the capabilities to switch it on. I don't know. You know, we just [00:28:00] talked about the fact that EPF is pluggable, you know, where things like the, the, the networking layer, you need to make that choice up front because you need the networking layer.

You have to make a choice, but in terms of taking that, you, you can absolutely plug. Something like Foucault. So observability tools, you can plug them in dynamically from day two, but I think it does make sense doing that from the start, even if you're not fully writing all those rules and fully customizing it, having that information source.

So I can go back and see what actually, I think this application is doing something different today, going back and seeing, oh, actually, you know, it's always done that. I think that's really important. So I wouldn't want to overlook that technically. No, you don't need to do it from day one. And I certainly see a lot of people not doing things like this.

Excuse me, things like the security and there was variability from day one, but I do think it's important. I think it's something that people need to mature more into the same way that I wouldn't stand up a mainframe without security and observability. I wouldn't stand up a VMware infrastructure or cloud infrastructure.

We all know how much hours in cost. If I don't mind. So everyone knows that when I stand up Amazon, I [00:29:00] secure it because I don't want a Bitcoin farm. So when you stand up Amazon in infrastructures, we know that you want to make sure you've got bidding thresholds and you might want to make sure you've got security on there because I don't want to accidentally spend thousands of dollars one weekend.

And I certainly don't want to be running a Minero farm in my class. So I think people are getting into that habit of saying, I do need these things from day one, because I need to know what's going on. And I think that. Of that is just a maturity thing. Which I appreciate, isn't always there for day one.

So it's kind of a, a bit chicken and the egg from my point of view, I think, 

Liz: [00:29:28] yeah. I think some of these things also are they're complex from a cloud native perspective. So thinking about things like network policy, if you have no network policy, then you're not going to have any problems with Netflix.

Preventing traffic that you wanted to flow, you know, it's quite easy to make mistakes with network policy. And, you know, I, I think that's one of the reasons why people might put, put off doing it. So yeah, one of the things that we've done to try and help with that, there's a, a network policy editor.

We were [00:30:00] talking about visualization and just being able to plug a policy into this sort of visual format and see, oh, okay. So that would block traffic from. You know, externally, but it would allow traffic from certain pods and it would allow traffic to certain namespaces being able to see that visually can help you get over this obstacle, that the quite complex things that can shoot you in the foot, if you get them wrong.

Yeah. Classic example, network policy that blocks DNS. Then nothing can resolve anything and nothing can talk to anything. 

Chris: [00:30:42] But I don't want my users running DNS

necessarily a new challenge here because the amount of people that disabled se Linux or app armor, cause it gets in the way when you're trying to install something. Now we all know that you shouldn't and we all know that you shouldn't do curl , pseudo bash, [00:31:00] but we also all do it because you want to do something quickly.

You want to get up and running. I think that there's, there's a difference between trying stuff out. Proof of concept doing it quickly and then making sure you shake those bad habits when you move into day two, day three. 

Liz: [00:31:15] Yeah, for sure. I completely agree. 

MM: [00:31:17] One more question. Do you see the security operational center?

So the security monitoring team, typically most organizations, do you see them adopting this technology?

Chris: [00:31:31] It's a good question. I think. Not directly. No, I don't. It's the same way that I probably wouldn't see the security team kind of making too many technology choices. I think they would definitely be a consumer of it. Yeah. But I'm not sure I would see them kind of choosing to, to use it and deploy it. Certainly they would see the benefits and they would probably put the pressure on to say, actually, I've seen the light here.

I can see the information we could get, which we couldn't get before we should deploy it. But I probably wouldn't see them [00:32:00] necessarily being the people to make that particular choice, the exception might be around the forensics piece, but that's when you kind of bridge the gap between SOC and and so on.

And so yeah. 

Liz: [00:32:10] Yeah, for psyllium and for the networking piece, because you have to have a networking piece in place before. I mean, we talk about DevSecOps and we hope that security teams are involved in decisions quite early on. I think the reality is quite often a platform team has already made some of those fundamental decisions, but hopefully they're pleased with the kind of security capabilities that you can build on top that make it attractive to the security team.

MM: [00:32:40] In data. I see lots of benefits as, as Chris pointed out for that, the likes of the SOC and other security functions to benefit from the information. Okay. Any final words, Liz, that you would like to share with our listeners? 

Liz: [00:32:55] Well, we just opened the call for papers for EBPFs summit 2021.

[00:33:00] So if there are any listeners who are using your and doing exciting things with it, we would love to hear those stories. So ebpf.io/summit-2021/. 

MM: [00:33:12] Chris, what about you? Anything, any last words to share with Allison? 

Chris: [00:33:16] I I'm, I'm always going to encourage people to get involved in the community. I think the, the EBPF community much like the cloud native community in the Kubernetes community is very, very welcoming.

You know, there's, there's no newbie questions. The more people involved, the better, it becomes, whatever that involvement is, even if it's as ever, even if it's just helping out write documentation, going through things. If you read the online guides and they don't make sense to you. Perfect. Example of something.

So please do get involved to some great slack channels. And we talked about them yesterday with some of the Falco staff and the city of we've got some great community as well around that as well. So, yeah, please get involved. Or even if it's just chatting to us on Twitter and slack and stuff. 

MM: [00:33:55] Super great.

Well, thank you to you both for coming onto the show. 

[00:34:00] Chris: [00:34:00] Can I say thank you for having us.