So get a check. Our speaker tonight is Matt Hargrave from Red Hat, and he'll be talking to us about Boot C. That'll be our topic for the evening. Before that, just a few announcements here.
So let's see next slide.
Okay. If there's any details there. Anybody sees anything that tracks your attention on the.
On the list that bye-bye to Windows 10. That was, of course, back in October. So they're still friends and businesses that no doubt are bugging you about, hey, Windows 10 doesn't work, what do we do? Well, we know what to do. You put Linux on the box.
So we've got an archive of our presentations. You can find that on our homepage. And it's listed by dates and pretty easy to find stuff on there. If you've got any questions, go to the discuss mailing list.
And just to ask for some help, and we'll see if we can't find them for you. Calendar. We maintain a calendar of techy events. Originally, it was just our own events, and we made it events in St. Louis, and when the age of COVID hit, we started.
Listing the out-of-town meetings that look like could be of interest to our group. So it's kind of divided up into 3 3 bits. Uh, you don't have to officially join anything for the calendar. Just go to slug.org slash calendar, and.
You can see all the different calendar listings. As I said, mailing list, we do run a discuss mailing list. You can ask questions, anything helps a heck of a lot. It has nothing to do with Linux, Unix, or some sort of open standards.
At least computer-type stuff. There's also an announce mailing list we run. That's a moderated list, so you can't post to it unless it gets past the moderator, and that way it's a limited number of mailings that come out each month. It's primarily announcements.
So, uh, that's why we call it the announce list.
Let's see. our own sponsored meetings. I don't know what to say about that. Oh, this is also election month. Every year here in February, we hold elections. Slug is actually a corporation, 501c corporation, and it's a.
board of directors of 5 people. 2 of those are up for election this month. Lee Lambert sitting next to me here in the Batcave and Omni tech is one of the people up for election, and I believe Grant Taylor.
I think is the other person who's in a seat that's up for election. So you can find our ballot out online. You can vote online, or if you prefer to keep it secret, you can put it in the envelope and send it by us mail.
the instruction to how to do that are out there on the web page. So we appreciate everybody who keeps the organization running. It's all volunteer effort, and we appreciate the membership for occasionally just.
making a ballot, making suggestions. So thank you all.
Let's see. Uh, any, uh, any particular calls for help? We'll also have time after the main speaker for a longer call to help. But if anybody needs to throw something out on the table right away.
Just be time for a little quick help if you need it.
Okay. Like I say, there'll be more time afterwards.
And with that, um… Let's see. I guess, yeah, we're ready for our main speaker. Matt Hargrave from Red Hat is going to be our speaker this evening. We've been kind of looking forward to this. Matt's going to tell us about Bootsy.
Bootable containers, the wonderful world of Atomic Linux. So, Matt, I think I'm going to turn it over to you.
Awesome. Let me, uh…
Make sure that works. Okay, at this point, can you guys see my slide?
Yes.
Awesome. All right. Hey, so thank you for letting me come and present to you all. I'm super excited about that.
I am also super excited about bootable containers. Um, I've been using Linux for about 25 years now, which I do appreciate the audience on this call is probably a little different than what I normally have presented to on this topic, so that's probably not really that impressive.
Um, but, um… I've seen a lot of changes of Linux over the years, and this one just gets me really excited.
But before I dive into that, does anybody know who this is?
Ah, all right, quiet. So this is Jfk, right? And.
Yep.
Yeah, can anyone, um, tell me what what the speech is about here? Anybody has a guess?
Is that down at Rice and Rice University in Houston? And he's doing the 1st version of the.
Mission to the moon.
Yeah, yeah, that's exactly, that's exactly it. Yeah. So September 12th, 1962, President John F. Kennedy. He gave his famous speech right that would kick off one of man's greatest engineering feats.
And so… You know, he says, uh, we choose to go to the moon.
Yeah, in this decade. and do other things, not because they are easy, but because they are hard. And so if you think about like, what do we think those other things were? Maybe like migrating to the cloud.
Probably not really what he's thinking. But, you know, um… doing the hard things. And so I want to tell a little story about the Apollo mission, but really kind of about.
the Apollo computer. Now, I'm curious, if anyone knows what this is.
discrete logic.
Ah, this is what's referred to as rope core memory.
The bubble memory?
Wow.
Uh, yeah. So, um… NASA needed 74 kilobytes for the Apollo computer.
And back in the 60s, memory was very large, very heavy, the opposite of what you need for space travel.
So, they solved this with the inventive rope core memory.
And essentially what you did is you created a read-only memory. And you did that by threading copper wire in and around magnetic cores to form your ones and your zeros.
Each one of these cores had about half a mile of copper wire.
And the way they were built. was they hired women from textile factories, and typically these were older, older women. And so they referred to it as little old lady memory. But these women would be given the instruction sets, and they'd weave these computers.
essentially. So. So I think that's a really cool piece of history around the Apollo computer. But what essentially they did here is they built an immutable computer.
So if anything were to go wrong, they could simply reboot the system, and it would come up back to its default state. So you didn't have to worry about any kind of data corruption.
It is hard-coded. And so, obviously, we made a lot of advancements nowadays, but bootable containers are kind of in that same space. You're kind of creating this immutable image of a system.
And if anything were to go wrong, you can simply reboot and go back to a known state.
So I'm going to talk a little bit about the history of how.
the atomic systems kind of got started. So you had a group called Project Atomic, and they introduced a tool called Rpmos tree in 2020, uh, sorry, 2014.
And so this was a really cool project that created an immutable operating system. So basically.
Everything was mounted read-only, except for the Etsy and the var file system.
It created this really easy way to do upgrades and then rollbacks.
And you have this image-based system. So Rpm OS tree is is the tool that really made all this magic happen.
And so it allowed you to compose RPM packages into this immutable image.
It was kind of like Git for operating system binaries.
Every commit is going to have. a checksum that then hard links files.
to that commit. And of course, you know, it's like Git where you can have branches, you can have a stable branch or a testing branch.
And this is handled with the content addressable storage.
And so… what that kind of looks like is, as you make your commits, um… You hardlink the binary.
To a hash. And so, as you… Create a new version of that file, and add that to the commit. The hardlink just points to the new file.
And so if you have. two commits with the same file, it's just going to use that same file, so the file system doesn't really grow super large.
Um, so it's a little bit efficient in that way, but obviously you can't have multiple versions of things as you have different commits.
downloaded between, um, essentially a side A and a side B, so you kind of have two partitions.
and those partitions are hard linking dependent on the commit hash that you have.
So it was a really cool tool. And and we built a few things at Red Hat. Actually, we have what was called RHEL for Edge.
that use this core Os is another piece of software that you may be aware of is a mutable Linux and designed for running containers. And that's actually the the base OS of what OpenShift runs on top of.
But some of the challenges around this is that, one, it was a really steep learning curve, learning all these new tools, and they were kind of complicated. They weren't well documented.
And you had to do a reboot to apply any changes to the system.
So you had to start thinking about how you use this system very differently, because you couldn't just make changes to files directly. You couldn't install something easily. So you use a tool called Bubble Wrap, which is the underlying tool of what we use today as Flatpak, but it was… it's a sandboxing app that kind of.
containerizes your applications. Um, but this is all new, so it was… it was all very hard to figure out.
How you did the repo hosting was complex. Oh, sorry, was there a question?
Uh-huh.
Yeah. Okay. Oh.
Yeah, there there was in the chat, and I meant to ask you this, Matt, if you wanted us to interrupt you during the talk.
Yeah, you can. Uh, let me… sorry, I didn't have the, um…
The the question in chat was, I'm guessing you vent Crc instead of checksum.
Uh, yeah, I think that's more accurate, yes. Uh, I mean, ultimately, I think it is a checksum, but for the files.
But anyways, and um… So… so when you made changes, or if you wanted to make real-time changes for when you, um… create a system to customize the system itself, you use something called Ignition File, and essentially you're injecting this file through pixie booting. It was a huge pain. It wasn't a great tool set, but it was a really good idea.
So Red Hat kind of. went back to the drawing table and figured out, hey, how do we make this better?
And so we did this through the inventive bootc. And bootc is a set of tooling that.
takes the way we do containers and allows you to manage these atomic.
OF images. I'm gonna dive into that a little bit later. But right now.
The question on a lot of people's mind is probably, what the heck is a bootable container?
I'm familiar with a normal container. You've got an image that's essentially a tar file.
That patches up just the base level of the userland OS, and then adds in the application files.
And so it's used as sandbox on a Linux kernel that's running.
Using C groups and SE Linux to sandbox that application.
And so, bootable containers are simply the next evolution of this, and we simply inject the Linux kernel into this.
Image. Oh.
And so what this is going to do is… give you a comprehensive container native workflow and user experience in the way that you.
host and install updates on your running operating system.
And it's important to remember that. Bootable containers are normal Linux.
It's just a different way that you manage and handle all the bits and the way you deliver those packages, or the changes to the operating system, versus your traditional package mode.
So it is essentially still, if you're running for running Cintas or rel.
It's the same. It just is managed in a different way.
So that looks just like a VM to me, but if it has its own kernel.
Sure, sure, yeah. Uh, the difference here is that it's actually built as a package.
Um, as an image, and then you will, and I'll show you what that looks like here. So essentially what you'll do to build these things is you write a container file. I'm going to go dive in a little bit more detail on this. This is just a quick highlight.
But it is a container image that you build.
Nothing goes to a container registry. There's a process in which we're going to convert that to a disk image, and then you simply deploy that.
to wherever makes sense, right? Is that on a bare metal server, laptop, public cloud?
Now, bear with me. You're probably going to see some references in my slides to image mode for rel. That is what Red Hat has productized boot C. We call that image mode for rel. So a lot of this presentation.
was, um, I built this around image mode for RHEL and gave it at a Red Hat conference. I tried to change it to be a little bit more community-driven, but just if you see that, that's the image mode for RHEL is the marketing thing around this.
I don't even see that on your slide. Oh, okay.
But Boot C is the underlying technology. You'll see it later, probably. So so when you make an update, you change the container file and rebuild your container image and upload it to the container registry.
Um, and then your running server would essentially pull down.
that change and apply that with a reboot. Uh, so I'll go into detail on each piece of this of the… deployment lifecycle.
So when you build a container file. I guess really, I guess I should ask.
Is anybody not familiar with containers?
Perfect. Alright, because that was gonna… we're gonna start, uh, back up a little bit, but alright, so everyone has a basic understanding of a container. So if you've ever built a container file.
Okay.
Um, this is about what it looks like. You start off with the from, and that defines the base image of where you're starting from. And so you see that looks just the same as anywhere else. The difference here is that we're actually pointing to a bootc image.
And then, you're going to do all of your runs to install the dependencies and the software you want, add configuration files, run scripts.
Identical. However, there's a few caveats here in that some commands are going to be ignored, simply because they don't make any sense in a, um… when you boot this as a full system, whether it's a VM or a bare metal, so you're not gonna have an entry point because you're not running a single command, you're running a full operating system. Same with the command, um, environmental variables, you're not setting those, because that's handled.
In the system, per user logged in, right? Um, expose, you're not exposing ports because you're running a system, and then the users. This is actually using the regular Linux user system, Etsy password, Etsy shadow system. So, um, so these commands would essentially be ignored.
That you would potentially see in other container files.
And so, I mentioned how this is immutable, and a lot of this is read-only. So the way this works is everything is writable at build time, so when you write your container file and you need to make changes, you can do that.
When you run this, the image content is read-only. The only file systems that are actually writable are going to be your var and your Etsy file system.
VAR is completely persistent, and it never gets updated after the initial build. So you don't want to put any content in your VAR system that you'd want to manage.
do the image, uh, so things like, um, application data that's gonna change, like, a database, for example. You never want that touched.
Uh, this is really where you're holding the state of the operating system.
And then you have Etsy. Um, it's also writable, however, it uses a three-way merge.
Um, to… to do that, so those things will change. If you make changes on the image layer, that'll get merged into… The system when you upgrade.
So I know this is about RHEL, but really this applies to both the Fedora and the CentOS image as well.
Right now, I don't know, there's no other dis… uh, Alma, Alma has a Bootsy.
image, but other than that, I don't think any other non-REL-based distro is doing Boot C right now. This is something that's really cool that's happening at Red Hat, is that typically Red Hat is what's referred to as a follow-fast company.
And we will find really cool, really, um, what we think is the best in breed in the open source space, because we're an open source company, and we go find those projects and we say, hey, this is a really cool project. We think it's got a lot of potential, and we think this is the best one, because a lot of times in.
Open source, you will have multiple flavors solving the same problem.
In this case, uh, this is completely Red Hat Hat has created this from the ground up, so no one has really adopted this yet that I've seen. I know that Ubuntu does have some atomic stuff using the OS tree technology, but it's not using the boot seed.
Um, I imagine. I wouldn't be surprised if it does, because I think it's… this is really cool. But, um… Sorry, I… Got a little off on a tangent there, but, um, ultimately here is…
These images are quite large. Yes.
Hey, Matt. You'll have to answer it now, but just to note, you mentioned atomic stuff. You may need to define atomic stuff.
Yeah, yeah, sorry, okay, so Atomic… originally you had the project Atomic, um, and that was a group of folks, and they created the Core OS. Red Hat actually purchased that company. Um, but Atomic is usually referring to as these immutable systems that are using this OS tree technology. And under the hood, BootC is still using RPMOS tree.
And that's, you know, composing the file system using the… the Git-like binary file system.
Um, and… and… including these things in an immutable fashion that are read-only. So that's essentially, when I say atomic, that's typically what I'm referring to.
Okay. Alright.
is an image-based… System that's immutable. I think that would be a pretty good definition. Hopefully that clears that up. Yeah. So these images are not like your application OCI container images where they're typically super small. It's a full operating system.
So it's quite large. What you won't see on there is things like Cloudinary agents. That's because the way you're managing these are through the container files. So most of that configuration is happening at the.
image layer, or the configuration layer. So, you could install those if you wanted to, and… in the post-install.
Boot C is obviously going to be in there to manage the system. Oh, yep. Other question.
Oh, no, maybe just hearing things. All right. Um, alright, so… so once you've built your container image, and the way you do that, right, you write your container file, and you use Podman build, um, and I'm gonna show… I actually have a demo that I can show you guys, um, we're gonna do it live, see how that works.
But once you have your container image, now you need to convert that to a actual.
bare metal image. And so we do that with the disk builder. Podman desktop. I don't know if you've ever used that before. It's a super cool tool. It's open source. It is sponsored by Red Hat, but it's not a Red Hat product. So, completely free to use.
Um, it's really gone a long way, and I think it's far surpassed what Docker Desktop was.
There's a command line tool as well. It just… this gives you a nice GUI to manage Podman and the disk builder, so you can do all this through the command line, which actually it shows… there's actually a container that is the bootcimageBuilder container, and you can.
Um, give it all the arguments you need, and convert your… container image into the disk image.
Uh, and it can handle just about any kind of disk image that exists. So you've got Qcat2, you can go AMI if you want to deploy this to the cloud, the MDKs, RAWs, and ISO, so you can do bare metal installs, uh, just like your typical.
Click through Installer. Oh, yep, yep.
which was bare metal. Which one is… oh, there it is at the bottom. Okay, I hadn't gotten out that far yet. And how do they come up with Q, cow 2?
Yeah.
Uh, Qcow2 is… I think it's QEMU, it's a copy on right, too. That's the disk image that, uh, QEMU.
Users?
Oh, okay. Well, that's a… that's a very moving explanation. Thank you.
Yeah, yeah, that's what I always think, too. Um, yeah, yeah, so that's, uh, if you ever run, uh, KVM.
liver. That's that's the the disk image that's going to use.
Oh. Okay, okay.
Yeah. Any other questions before I go on?
And of course, it supports building multiple architectures, so if you want to build this for ARM or PowerPC, or I don't know what folks are building nowadays.
When I was growing up, I ran to a friend that he was doing some weird stuff with PowerPC back in the day.
Oh.
I think those days are gone. Um, so when you build your image, sometimes there's customizations that you want to inject, uh, but you don't want it necessarily handled by the container image itself. Now.
There's some debate and philosophy around this, right? When is it appropriate to do it in the container image itself? And when do you want to inject it at the disk build?
there's various… ways of doing this.
So, if you want to customize and add users to it, like injecting SSH keys and passwords and managing a user, you can do that. Um, so it might make sense here in that you want an image that's your base image, but when you deploy it, you have.
Different users per deployment, that might make sense to use this configuration file.
You can also set up file systems here, so if you want to set the size of the root file system, that's where you do that, or create custom mount points on VAR.
Now, that's pretty much all you can do. You can't create a file system in Opt or Etsy, or Home, it's gotta be either just the root file system, or somewhere in VAR, and that is because var is the only place that is actually writable. Everything else is going to be read-only.
And everything is basically built off the root file system, so there are no other file systems. Um, that's… that's read-only, except for Etsy.
However, you might think, hey, I've got this application, but it needs to be in a very specific location, and it needs to be writable. The way you'd handle that is with bind amounts.
And so you can configure bind mounts to point to somewhere in VAR, say, var data.
And and that's how you'd kind of handle that.
That's what the sample is on the right hand side.
Yes, correct. So, so I got the VAR data. If I, if I wanted that to be an option data, I would just create in the container file I'd probably upload in the FS tab.
Something that, uh, configured the bind mount to that, or in system D.
You can install via Kickstart as well. So now you just simply have the OS tree container command in your Kickstart file, and you would point directly to the container registry.
URL. So, that's what's really cool about these boots C images, is that we're using all the tool sets that containers have given us.
And you kind of have this parallel thing where… The OS tree folks were building this immutable operating system. We've got these layers that you're dealing with.
But it was complex, and the developer side of the house was doing all this really cool stuff with containers, very similar style of work, where you're defining everything in a predefined way early on in a config file, i.e. The Docker container file.
And so it's it's really handy to build to latch onto that and just reuse all that technology is already built.
Craig.
Um… The pre and the post still work here. What wouldn't work here is if you were defining packages, because that's defined in the image itself, so it just would ignore those.
So Boot C is actually a tool itself as well.
Amen.
That's gonna… you're gonna find that on the container image that… or the system image. And so you've got a few different commands that you would use to manage the system. So bootc upgrade, that is going to.
Look at the image itself that it's already attached to, and go see if there's any changes to that.
If there are, it will pull those down and attach that to… the other partition, and then when you reboot, it would apply.
Uh, those changes. If something were to go wrong, you can simply roll back with the rollback command, and that will go to the exact state that you were previously in. So everything's staged.
In an A or B side, and so once you roll back, it would discard all the staged updates, but it would go back to exactly how the system was.
So, this is super helpful, right? Because sometimes you upgrade stuff and it breaks. Rpm does have… Yum does have a rollback ability.
But, I don't know, every time I've used it, it's been hit or miss, because you get in these weird dependency issues.
You can use bootc switch to completely change your reference image.
Um, because you're just laying down the file system directly to the disk, it… you can complete… you could completely change major versions, or distros, potentially. Uh, that really would have little impact, because you'd lay down everything on the disk, and then you… when you reboot it, point to all those new files.
So, you completely change the way a running system is, and then roll back if you wanted to.
Um, bootc install is something that you would use probably only really if you were running this in a container, so you can run bootc as a container, and then have a running normal system.
And then lay down directly to the file system, and… convert the running system into a atomic system. So I tested this out, actually, I was running a RHEL system in AWS. Your traditional package.
Mode RHEL, and then I ran this, uh, and pointed it to the file system. It needs root privileges, obviously, but it lays it directly to the disk, you reboot it, and now you're running in a BootCase, or what we refer to as image mode, Pharrell is. So, really cool way to convert systems if you want.
Um, so I talked a little bit about the A side and the B side, right? So now you've got these transactional upgrades. So what happens is it downloads that container.
image and lays it down using Compose FS. And uses that OS tree to convert the container image into the root file system and makes those writes. So I was saying is, as you upgrade, right, you can go major or minor versions really easy.
by just pointing it to that new container image. And if something goes wrong, the rollback's super easy, so you don't have to worry about, like, snapshots or restoring from backups. It's literally a reboot to go back to the way you were.
And if you've ever dealt with containers, you know, we love our tags in the container world.
Your tags work identical to the way you'd use them in containers or traditional containers.
So you have your dev, your test, and your prop branch.
And as you start to tag your versions. If you have a system that's… Um, set to be in the dead side as you push out your changes, those updates can roll through, and then, obviously, moving to your production side.
As those versions get pushed through. they're gonna download it as you as you get it up, and… tag it with the… the higher environment tag.
So you can kind of branch it that way.
And this is great for… really good for servers. Um, desktops, probably less so.
If you do want to experience this in a desktop style environment, run something on your laptop, Fedora has a version of their distro called Silver Blue.
And SilverBlue uses the RPMOS tree, now BootC, tooling.
Uh, to create this immutable OS, and you can run it as a desktop. So all of your applications are going to be installed as flat packs.
And those are kind of like our containers that are, you take an app and you essentially containerize it.
I've ran it before. Uh, Red Hat just changed our policy on our work laptops, where we now have to use the, uh, corporate build, so I had to move away from it, unfortunately. But, um, but I was running it, and it runs fine as a…
Oh.
desktop OS actually better, in my opinion, because… I never had to worry about installing patches breaking something, because if something broke, I just simply used the rollback. It was super easy.
Um, so I've got a demo I'm going to run through and show you what this looks like in the real world.
Uh, but if there's any questions while I kind of switch, uh, do a little context switching here as I get my laptop, uh, set up for that.
No questions. Man.
Good evening. My name is Tim. I'm here in Afton, Missouri, and I joined half an hour late, I forgot about the meeting. But I'm a relatively new Fedora user looking to get my RHCSA, Linux plus and all that, and I'm curious to know, are you… well, first off, are you a Red Hat employee?
Yes, yeah, I'm the solution architect. With Red Hat?
Okay, cool, cool, cool. Um, so my question for you is, I've got a little Framework Laptop 12, um, and I installed CentOS on it, and I'm just working through… like, I guess, would you have any advice for someone who, um.
who's looking to get their RHCSA certification. Um, I'm just doing it by reading books and playing around with an OS that, you know, it's just on a computer that has no other purpose in my life, and I'm just, you know, breaking things and.
Sure. Yeah, yeah, no, great, great question.
Um, yeah, I would say yes, and… and I'll get… I guess I'll give you some other resources. So, um, if you want to run actual RHEL, so CentOS Stream.
is essentially tied to the major versions, but it doesn't have minor versions.
And so it's probably about 6 months or so ahead of where REL is. It is the upfront of REL.
So for the most part, nine times out of 10, this should be absolutely identical to REL. It's just applied patches that are going to.
go to RHEL as the lifecycle happens. If you want free RHEL subscriptions.
Um, you can sign up on developers.redhat.com. and you get access to, uh, 16 free RHEL.
That's what I was going to recommend.
systems through the developer program. Yeah. And… Developers.redhat.com is super awesome in that it has labs and all kinds of really cool blogs on it.
and I like it because it's not markety. Um, a lot… if you go to, like, red hat.com and look at a lot of our stuff.
Um, it is more focused on the business marketing side of things, where developers.redhat.com is.
Very much. geared towards the technical folks who just want to figure stuff out.
Uh, there are an amazing amount of free resources for learning these things. And yeah, so I recommend that you absolutely can.
go through, uh, there's several good books out there.
And to get your art CSA, because if you're doing that as an individual, it can be quite expensive and can price folks out on an individual basis. I do appreciate that, so… Um, we do offer our learning subscription.
Um, but that's typically, um, honestly, like, it just is out of reach for most individuals.
One of the advantages that the developers
Price-wise.
method has.
is it is the same RHEL software that you would be certified… testing to be certified on. It's not something that's close,
It is the same.
And it's not just student licensing, it's…
um, kind of, if you're an individual,
not doing it as a business, you can get a full-blown
red-head Enterprise Linux license, and I think you said it's 16 of them.
So, definitely check out
the developer solution.
Yeah, and you get access to, like, web console.
Yeah, yeah, just, I guess, to reiterate, it is like the the console, when you log in, that's what our enterprise customers log into. It's identical. It's… the only difference is the SKU that you get access to is called the developer SKU, but… Otherwise, yeah, it is identical.
If you have any problems, feel free to reach out. I'll happily help you out with that. Okay, so I have my Linux terminal here starts. I'm working from 2 systems.
So hopefully don't get too confused. Okay, so this is what you would typically see as a… I'm doing this live in front of people while it's recording, so we know it's gonna go real smooth.
Yeah, yeah, yeah. So so this is a container file here.
that I've built. We are pointing to the Cintas boot C.
image. So it's up in Quay.io. I'm copying over some Ssh keys to root just to make this really easy. Probably wouldn't really do it that way. But we're making this quick and dirty.
Um, installing Apache. Removing the no index file, and then I'm adding an an a simple HTML file. I do have the exposed commands on there. Mostly because I do want to show some things. Alright.
Um, so… We're going to build.
So I know very much what I want to do. So this makes it a little easier.
I'm going to call this version one, or tag it with version one.
Okay? And so if you've ever built container. You're going to notice this looks identical.
It's running through the steps. It's installing Apache.
and makes the commit. All right. So we have our.
Amish. So I'm going to run this as a…
Normal container. I am going to… Um… make a expose the port so I can Ssh into it just to show off some things.
Okay, so I am running. That, I mean log just to show you what this looks like.
So that's really the consoles. The log's kind of wonky. But if I Ssh into this.
Okay, so… What you'll notice here is the kernel is 618. That would not normally be the kernel that you would run on Cintas stream 10.
That's because it's actually using the underlying system, because right now it is running as a container. So it's using the host kernel, which is my fedora laptop. However, if I look at the Rpms.
I see a kernel is indeed installed here. running system D, so I can do a status on htpd, and I see.
It is running. So this is a container that I just built using that container file.
It's running HTTPD, it's running SH, it's a full system, though, ultimately. I mean, this is… I guess it's hard to show, like, what you would, uh… show that it's a full system. But hosting and control. It is the the icon name is a container as the chassis. But it is OS stream 10 as the operating system.
So… I'm going to stop that, and then we will convert that to a disk image.
So you can do this command line. But it's a lot of parameters that I don't have memorized. So we're going to use the fancy.
Podman Desktop. So, this is an extension that you would install.
An extension to Podman Desktop?
Yes. So if you go to extensions here, and you go to catalog, and you search route C.
Ah, okay.
There it is. So super easy, probably my desktop is so cool. I can't stop being impressed. And it's not a product. We don't sell it, like, we just… develop it. So that's great. Okay. So now.
I'm going to select this and tell to… Sorry. That's not what I wanted. Okay, wrong wrong tab. Okay, try this again.
So I'm going to go and tell this to the output folder is going to be here to my or my demo is.
I'm gonna. I selected Qcow 2. I've done this already. So some of these are already selected. Normally this would be all empty, but.
QCAL 2, because I'm going to run it on this laptop. I am going to inject the config file, which I'll show you what that looks like, actually.
So if I cap my config file, all this does is create my local user, so I give it the password of password 1, pretty super secure as my SSH key, adds my self-to-wheel. It also creates a var data.
file system at 2 gigs.
We're gonna build it. It's pretty quick, but there's a little little paint drawing here, so I didn't queue up any music, unfortunately.
It does need elevated privileges, because there are some… mounts that it takes advantage of work is being done to try to be… able to build it without root privileges. But as of today, to be able to convert a.
container image into a disk image. It does require elevation.
I don't know if there's anything interesting in the logs that I can point to.
Can we take one step back and go through that sentence you just said again, the… you're converting a.
Oh. Yes.
Disk image into a.
When you're converting in a container image into a disk image.
It requires elevated. privileges. So it requires Podman requires root access.
And that is because of the way it does the file system mounting.
So development work is happening to try to make it so that this can all happen as a user.
Okay.
But as it stands today, it you do have to have root access to DV.
disk image build.
Okay.
Okay, so this extension… uh, doesn't meet all of the rootless container things that.
is one of the benefits of Podman, but it's in work.
Yeah, yeah. So, yeah, so while you can run things in as as rootless in in Podman sometimes, depending on what you're doing. Yeah, you do need elevated permission. And this is one of those instances do that. All right.
So we've built our disk image. And let's see here… So the output to our output is here. So if I get a few cow 2 directory. There's my disk dot.
QCAW2. Oh, I don't want to stop sharing my screen.
Um, I don't know if you all are familiar with Copilot.
But it is a nice web interface that… You can manage your system through.
So I'm just going to use that to build my VM.
So I'll call this the St. Louis log. I do hate the way it has you browse.
files, though, it's not.
Pretty. So just have enough.
File system come up. Okay, so pretty straightforward. I'm pointing to the disk.
qao2, which was the disk image that we created after converting the container image.
And this is a pretty. basic Vm build to cockpit.
And, uh… So it's going to boot. There it goes.
And while that's booting, I'm going to need to push this. I've actually already pushed this image to the Quay. So we're going to do some cooking show magic. But here's my Quay account.
And so, if I go into the slug demo and look at my tags, I've got version 1, but then I also… Have latest.
That added that tag. And then we'll do an upgrade. I'll.
Oh, there's my tags here? Uh, or Matt. Okay, so it's completely booted, so I'm gonna SSH into this, so…
Alright, so now you'll notice that the kernel version's a little different, because I am actually running.
the synth Os kernel. But now you see that the chassis is a VM.
Excuse me. And we are running… Hbd.
So let's. Go to the website.
I'm gonna remember that. 6 and a half.
Cool. Uh… So oh, hello, Kansas City. So that's where I'm from.
So there's a joke here that I want to throw out. Do you know where the best barbecue is in St. Louis?
Well, you hop on I-70, you had about 4 h west.
Where?
er.
Yeah, uh… Okay, so had to throw that one out.
can…
Okay.
I kind of heard that one coming. There used to be a KC BBQ at in Chesterfield Mall. Neither one of which exists. So yeah.
Oh, yeah?
Yeah, yeah, you gotta go to Joe's.
Yeah, but there there was a Casey barbecue restaurant.
in Chesterfield Mall, but neither one of them made it to 2026.
Alright, so we're going to make our fix, right? Hello, St. Louis. Replace it with Kansas City.
And now I'm going to do a podman. Build… 3.0…
Okay, so once again, if you're familiar, you'll see how it went really, really fast this time, and that's because everything is cached.
Except for the only thing that really had to happen was it needed to upload the new version of the index file, so it did copy that. Everything else pretty much stays the same.
So now I have version 2. So I'm going to.
push this to… Quay?
So when you say cash, I… I'm assuming you cash the same way that the different container layers are?
Yes, yeah, because this is containers, right? This… everything that's happening here is identical to any other kind of container you deal with. Yes, the layers.
Yes. So all you needed to download was the new parts.
Yes, yes. So the layers were already cached. Okay, so I go to my container image here.
Oh, that's not the refresh button. Alright, so I've got version 2 here, and I'm going to now tag, actually.
The list of people.
Let me, I realize I made a mistake here, but we can.
Okay.
Okay. So I look at boot C, we'll see here that we're booted with the image.
pointing to the… Slug demo version 1, and we've got the SHA hash.
So I'm going to switch it to… Latest.
which is still going to be the same version.
Yep, I knew that was gonna happen.
Oh, no. Oh, I know what my problem is. I think this is a private repo. I need to make it public.
Well, it is a live demo, so… Something had to go wrong.
Yeah, yeah. Absolutely. Make it public.
Ah, couldn't download it.
Now, yeah, you could set up authentication, just haven't done it, so… But if that's… if that's all that goes wrong in my live demo, I'm pretty excited.
Okay. So the fact that it's delaying now. means it's probably downloading it. Yeah, boom.
Yes. Oh, I felt. Oh.
Or maybe not.
Well…
That is odd. Let me see if I can switch it to boop.
Oh, there it goes. Okay. So this would look the same if you did an upgrade. But so I'm switching to version 2 now.
Okay, so what? Yeah.
So it's downloading those those layers.
Well, dialing the image.
Normally, this would be faster because of the way I did it. I… There's technically a local image when I built the disk image, not pointing to the quay image.
and because I'm well, actually, no, it's because I'm using switch because I'm using switch. It's downloading the entire new image and replacing that. If I had done a.
upgrade. It would only update the changes. And that's… Because of the way I did the tagging, I made a mistake. I should have built this as the latest instead of, um… Do I misspelled at me?
Oh, I should still… I don't know why that didn't work. That's odd.
Yeah, that's why this is taking and having it downloading all the layers, because I did a switch instead of an upgrade. So it is taking longer. Normally, this would be very fast, because you're only really downloading the one index file.
Luckily, I have pretty fast internet, so we're… we're getting there.
The paint is dry.
Unrelated, but while we're waiting, is that, um, KDE that you're running on your Fedora instance?
Got that. Yes, yeah, ultimately, just Katie Lippert.
Oh, wait, I'm sorry. I was thinking Kvm. Yes, it is Kde. The window manager. Yes.
Uh, used to be a big, uh, Gnome fan for a long time, but uh… I don't like the… I don't like the new new version. All right. So we've we've downloaded right? But if we still look at our our website.
Nothing's changed because we haven't rebooted. Now we're going to reboot.
And we can watch it reboot over here. Hmm.
There you go. I saw something red go by.
Right? So it's booted.
Oh, I think it's cached. There it goes. Hello, St. Louis. So now our change has been applied by applying the the new update and rebooting.
So I can also show that. Oh, boy.
Somebody does later. So, propose FS here en route, but it's, uh… read only Etsy here.
is writable. And then, if I can find var.
There's my guard data. It's writable. There's my var. It's writable.
So if I were to go into VAR.
as it was running, I thought I saw a line go by in red.
Oh, yeah. There probably was an error of some sort.
in there. But uh… Or, yeah, I'm not sure. Some Systemd thing that wanted to highlight in red, but it is booted just fine.
Okay.
Uh… Let's see here. I think if I remember, yeah.
Oh yeah, I think I do change where the, uh…
That's right, I do change. So, I changed the, um… the index file to be an user share. And I did that because I left it in var. If I made any changes.
to my image, they wouldn't be applied in var, because VAR only happens when you do the initial build. It'll never push changes to bar, so I had to change my, uh, my location for the HTML.
But, uh, but I can write here if I wanted to.
Or if I go to bar data.
Polarable. I think that's all I can think of. Does anyone want to see anything else?
But that's so so that's that's bootable containers. I think what's really cool here. And so I appreciate that this audience a lot of times is probably focused on desktop things. I know a lot of Linux user groups are more focused on that. But from my perspective, from an enterprise perspective, when you're building systems.
What's really cool is that from a, uh… container aspect. In this instance, if I'm only testing out that I made changes to the website, I could make my change, spin up the container to validate my.
tests against that system, say, hey, my changes worked, everything's fine. Now I can push that update out. I don't need to boot up a full system to test a.
a website change. So you can kind of create these CI/CD pipelines just like you do with your application containers, but for your whole OS. So now when you're managing at scale, you can define everything as code in your container file and start to do GitOps.
And there's a really cool blog. for, um, that we… oh.
Any other advantage or use case for these bootable containers?
Hey, Matt, what?
Oh, I'm sorry, can you say that again?
Any other advantage or use case of this bootable containers? I understood what you're trying to say, but we have different deployment strategies also, right? I mean, but any actual use case which solves the problem.
Uh, it's actually needed over containers, the bootable containers.
Versus, like, traditional Linux systems. So in this case I know exactly what my system looks like. So everything is defined. You you shift completely left in that you're defining your entire Os configuration.
Yeah, yeah, I'm gonna get that.
Got it, correct.
in your container file. I know this is exactly how my container is. Nobody logs in and makes changes to the system. That whole SSH in unless it's like a break glass scenario you need to troubleshoot, which you really still want to get away from.
And so anytime I make changes, that's going to go to my version control system, realistically.
So this container file, I'd make the commit, I'd push it to git, and that would run it through a pipeline that eventually.
updates that image into the repository, and then. I'm gonna update my system when it's convenient, and I've got my change window, but I know exactly what that system looks like. And if.
something is to break, I can roll back. I don't need a VM snapshot, so I can start rolling back bare metal servers, edge devices, without needing any.
backups or VM snapshots. It's… and it's crazy fast. The speed in which, you know, restoring a.
a large VM snapshot can take a while. This is literally the speed of a reboot I can roll back entire systems.
Hey, Matt, quick question for you. The bootable container.
Yeah.
Uh, is that, you know, in this case, it took you 3 or 4 minutes to build it.
and you know it took you at some time, you know, to when you had to swap it out. Would it make sense to keep your the OS in a in a bootable container, and then.
tag other, you know, add other containers to that when it's running, so you don't have to update everything type of thing.
Absolutely. And that's exactly how it works. If you were to run this, uh… And so that's… that's how OpenShift works, essentially, or Fedora Silver Blue.
Okay.
So, um, so Core OS on OpenShift, everything that runs OpenShift is a container. So SSH is running the container, the Kubernetes kubelets are running as containers. So yeah, all of your applications. So you would manage your applications independently of the system.
And that's a big thing that Red Hat is pushing for, I think, or the… really the Linux server world is pushing for is.
a strong separation between application and system, because when they get super intertwined, it's really hard to update the system and not break the application. And so you really want independent workflows on.
Right.
your dependencies and and building and running your application versus your system. And so those updates don't have to one coincide.
I have a flag on that play.
Up. Okay.
If your application has dependencies on things that are on the system needing to be a specific version,
Those are copied into a container,
And it's separate from the system, and you can upgrade the system, but you still have that old code
running in the container.
Yes, but you can upgrade the system without breaking the container.
I would argue not everything on the system is updated, but okay.
Uh, well, the container code isn't. Right? But, like, I can apply an OS patch, so if I'm moving versions.
Yeah.
I can upgrade the version. Yeah, so from… If it's a different team that manages that application, I can say, hey, I'm operating the system, I don't have to worry about breaking their… their application, and it's up to them to upgrade their code. Yeah, I understand what you're saying is the application, that code's not updated, but if your responsibility is a system.
And there's some times that you just have legacy applications that you can't… I work with a lot of customers that have really, really old applications that the company doesn't exist anymore.
And that application is mission critical for the business, which, you know.
describing the day job.
Yeah, I'm sure. Yeah. So any Linux admin has probably seen this. Uh, it's scary. That's the reality of the world. But now, as the the Linux team can say, hey, I'm operating the system, at least the operating system's upgraded.
That might not be. But, um, but yeah, and then back to whoever originally said that, I think that was Lee, right?
No, it was me that was talking about the…
old code.
Oh,
Uh, no, but back to, like, you run all of your applications as containers, so that you can make changes and install new applications without having to reboot, and that's exactly how you do it. And so in a desktop world, you use what's called flat packs.
which are a… it's a sandbox very similar to containers in the way you you do the apps of it.
But, um, slightly different technology used for servers. You're going to use your traditional Podman container situation.
Hey, hey, Matt, Craig's had his hand up for a bit. Greg.
So, does this let…
Yeah, yeah, sorry, go ahead.
basically use the container build system to build…
like a VM image? Is that what we're…
trying to accomplish here?
Okay, okay.
Exactly. Pretty much, yes, in a roundup. Yeah, with some nuance. But yes.
Is there… yeah, yeah, okay. I'd like to hear about the nuance, too, a little bit. Um, like…
Dig into that a little bit, but um…
Sure.
So… is it possible to have an image that will run as a container or as a VM, and…
How do you even decide which to use in that case?
Yeah, absolutely. So the nuance here. So not just a VM, but you can run it as a bare metal system as well.
So there's no distinction between virtual machine or bare metal, but you build a disk image. So you're using the tool sets of containers, which.
Right, right.
Yeah.
is a really good toolset system. The workflow for building containers is matured really well, and it's and we, I think most folks have agreed that it it's pretty nice.
So we just use that same tooling. to manage the underlying technology was the OS tree part.
Right.
Um, to build this… this image, and then convert it to a disk image, which you then install. Now, between… before you convert it to a disk image, it is a container image. It's a your traditional OCI container. Yeah, so I can run that image. Let me go back to sharing.
Okay. So…
So my VM's running right? This is still running.
Oh, we're not seeing your screen yet.
Oh, sorry. Uh… Okay.
There we go.
Okay, so my my. Well, how do I get rid of that?
So this this VM is running. Um, and that's based on the image. But if I go to my Podman images.
So right now I'm not running any anything. But if I do.
Hotman run…
Make sure it's right.
So this is the exact identical image that my DN is running as.
So that's now running, and if I do a curl on… that that is that same image.
So I'm running it right now, side by side. One is a container, one is a VM.
identical images. The difference is one's a full kernel booting on a virtual machine. The other ignores the kernel part. It doesn't have a bootloader. The kernel, the files exist for the kernel, but there's no bootloader.
Right, right, okay.
And it's just running as a container, but it's a full like I can Ssh into it. I didn't forward that port, but I could SSH into it, see systemd running.
Do anything you would up to, you know, there's limitations what I can do in a traditional container like.
Right, right.
But but outside those limitations, I can do that in that system in that container. Yeah. So yeah, so I can do it side by side.
So, how do you decide which one to go with for, uh…
Um, so it depends on your use case. Yeah, so most of the time, if you're using this technology and using a Bootsy image, you're going to want to boot that up.
Any specific purpose.
And at some point, the endpoint, the end goal is that you're going to want to run that machine. And the reason why is, if you're running it as a container, you want an actual container to be as lean as absolute possible. You want to remove attack vectors, you want to make it.
just not be big, so you want it small, so it's completely sandbox. It's only the application.
Right, right.
This is a full operating system. So the images are quite large.
So, what's nice is that as a container, I can spin this up and do validation testing against it without having to convert it and boot up a whole machine.
So I can do like Cicd. for the operating system.
Right.
And the only time I need a full boot up is I'm applying that update to the system itself.
Or I make changes to the kernel. I have to test that out because that that's a limitation that I would have with the container.
But a lot of times. you're making changes to even system level changes that don't need the kernel. I can validate that by just booting up. If I'm making Ssh changes, I can make those changes, boot up the container image, and validate that it's still good, and never touch my running system.
until I validated everything works, and then push that change.
And then reboot into that change. I don't know if anybody's broken.
mass amounts of Ssh servers. at one time, but I have. It's not a fun time.
I pushed out a bad automation. So luckily, I was using configuration management tooling. But I I pushed out a bad config and broke.
Like, 200 servers. SSH config, so their SSH was just broken, and we couldn't get into them to fix it. I was lucky.
Yeah, yeah. Luckily, because I had configuration management, I could push that change, it just took a while for it to apply.
That was the day the internet went down.
But now there's probably better ways I could have done that, and… There was a lot of things that went wrong there in that whole chain of events that could have prevented it. But one thing is if I had, you know, made those changes.
if I was doing it this way, you know, you're not using things like puppet or Ansible or Chef to manage these systems, because you're doing it all at the the container level. So the need for doing it that way goes away.
And I think that's really cool.
Yeah.
I have one question. What do you think, like, it's very latest thing, right? Uh, what do you think in future, where it will replace the traditional CICD?
pipeline. Or do you think it will, or it will stay, like, sandbox?
I don't know about… the traditional CIC. I do think this is the future of Linux. I think package mode, or the way you… I think that's going to go away, and this is definitely atomic systems. It's it's very similar akin to the way your phone works.
Um, you know, every app is independently managed, and you download those, and then when you apply a firmware update on your.
your Android phone, your iPhone, right? You just download it and you reboot into the new version.
Um, I think that's going to be… what the future of Linux is, I really strongly believe that. I've seen this over several years kind of evolve into what it is now. Now it's super easy to use by anybody.
Um, the traditional RPMOS tree was a little. Complex. I struggled with it.
Okay.
And, but, um, but yeah, as far as, you know, using to, yeah, this, I don't think it's going to evolve the CICD pipeline aspect of it, but I think this is going to allow system admins.
Because a lot of system admin work is very Wild West, or it has been traditionally.
Um, some companies are better than others. But this allows you to manage your systems in a very same way that you manage your application software. I think that's in in a good way.
By the way, Hiro, I apologize. You were asking questions earlier, and then we kind of.
started forking off from, and I didn't know if your original question got answered or not.
Is it answered adequately before?
Uh, yeah, yeah, I'm good. Actually, I joined late. This was totally new to me, but I'm good now.
I don't know how to do it, but I know what is this.
Okay, good.
Good. Uh-oh.
I have a quick question. So Linux Noop here, and I've got all kinds of information coming from all different kinds of perspectives. Can you compare and contrast this against LXCs? And the only reason I ask is because I have a Linux nerd friend in KC who is all about LXCs, and um.
Oh, man. I have not messed with LXC in a minute.
So LXCs actually are isolating via. So it works very similar to Oci containers.
Um, so it uses namespaces to isolate processes. I don't believe, uh… I'm gonna be honest. I haven't messed with Lxc, and I don't know if I can give an accurate answer, because I forget a lot of the details on how Alexei works.
I will say that it's an older technology that just never really. I say older. It's not that old. It's like 17 years old. I only know that because I googled it real quick. But, um… It hasn't as caught on. It didn't in the container world… OCI containers kind of won the landscape.
Um, I think Lxc is cool. But other thing… other… the OCI technology just has… has really caught on, that's what everyone really uses. Now, when you say containers, people don't think, like.
C containers. But yeah, I don't know if I give a fair contract. It has been so long since I've messed with them that I… I don't remember enough to… To give you a fair shake on that one. Happy to… happy to do some research and get back to you, though.
Yeah. Oh, no, that's okay. I just… like I said, you know, the thing about being a beginner is that information is presented to you, and you don't know how to weight that information, right? It's like a polynomial equation where you assign, you know, coefficients or whatever, and it's like, some stuff's more important than others.
Well, yeah, and I will say… In this space, specifically the Linux and the open source space, there are folks with very strong opinions about things.
that are very loud, and that's not necessarily means they're more right. Um, I don't want to ruffle any feathers. Um, I've been in my fair share of highly, uh.
um, heated debates about things, right? I mean, just… Emacs and VIM, like, you can Google that, right? That's the old-school one. But, um… So yeah, I appreciate that it's hard to know, um… You're gonna be probably here and read a lot of things, and you don't know how accurate it is, and it's tricky. And I don't know if I have a good answer for you outside of you can look at.
the industry and what folks are using, what becomes popular, and think, okay.
you know, and I'm gonna go… so I might, uh, see, I might upset somebody here. So if I go to the systemd route, I think it's safe now. I think it's been long enough now I can talk about system view without upsetting people. Um, but, uh… you know, system D and system at Sys 5. You know, that upset a lot of people, and there's a… I think a lot of that was because there's so much change, and I can appreciate that. And actually, that system D is super helpful with boot C.
Let me go down this rabbit hole real quick. Uh, so there's a lot of things in Systemd that in traditional Linux, you're like, why does it have this? Like, I can do this the old way, I don't know why I need systemd to manage this. Um, because it handles it in more of a dynamic way.
um, mounts, for example. Systemd can handle mounts, right? In a traditional Linux user, why do any mount have an SS tab? I don't want system to be managing my mounts. However.
In boot C, it all of a sudden, Systemd makes a lot of sense to start managing your system.
Uh, through that mechanism. Uh, stuff like, uh, Network Manager as well.
It really clicks together nicely. But I just want to say that little caveat. So all right back to back to my previous thing. So you had a lot of folks that were really upset about it, and and you know you could debate a lot of ways. I personally.
As a system admin, as in the, you know, Fortune 500 level, managing a whole lot of Linux servers.
I was very happy with SMD that I never had to write another init script in my life. That makes me happy. It made managing the systems that I was working with.
easier to deal with, and. That's my personal experience. I really like system G. It had some hiccups, and and there's some very valid criticism. But if you look at it today, you know, most, I think just about every major distribution has moved to System D. I know there's a few outliers, though, but those are outliers that are very specific, like, hey, we're different because we don't run SystemD.
Um, and so you look at the landscape, and you kind of see what becomes popular, and you're like, alright, you know, there might be someone who's very loud and really hates on System D, but if it was really so bad, would all these distributions run it?
Microsoft's still selling a product.
You know, there's caveats to that, right? And you're not going to be 100% right all the time. I think, but that's the best way. I don't know. Otherwise, you know, you just have to research it and, you know, use the tool that makes sense for you. And that's what's cool about Linux and open source is that there's typically alternatives around it and you have access to the source code to make changes.
to things that, if they don't work to your needs.
Question for you.
And I got here late, but isn't…
Oh, yeah.
the goal of BootSea?
to boot what…
starts as a container, or is a container during the build phase.
but turn it into an actual standalone virtual machine.
Okay, so would it be fair to say the container aspect
Yes, yeah, absolutely. And so, yeah.
is a byproduct
on the way to the goal of the virtual machine.
Um, yeah, kind of. I think so, because originally, when you look at, like, RPMOS tree, you know, there was no container side of it. Um, we simply latched on to the container toolset because a lot of it made sense and overlapped with getting there to your immutable operating system atomic style.
operating system. So yes, the bootable container or the container aspect of it has some advantages, like, you know, like I was saying, you can validate in testing by just spinning up a container and not having to boot full OS.
But yes, the end goal is ultimately that you're getting to that.
Great, thank you.
full system disk image.
So in my former career. I had a.
started working with containers to you know, modernize our.
different build systems, that kind of thing. Um, and it was seriously looking into using Kubernetes in particular, your Rosa product, which is Red Hat OpenShift on Amazon, I believe it is.
Um, just the platform. Um, there was one snag we had, though, that we needed because we had some dinosaurs to maintain, we still needed some virtual machines.
Uh, perhaps two. And at that time.
Um… the Red Hat model was you had.
They could support virtual machines. that would attach to Kubernetes kind of like a container and went to some of your demo training things in St. Louis.
Where they kind of demo… you had a chance to dem… you know, kind of do a test run where you actually.
Use a Kubernetes-type interface and. build up and run a Windows Vm.
Which is pretty cool. Um… But from our application at that time, that VM or whatever VMs you were running had to be.
On a bare metal machine, and not something that's virtualized, at least that's the way the sales model was.
Now, from watching this, I think… what used to have been a.
full virtual machine, which you just showed us, it can be a, I'll say a plumper container.
Um, so much… So you wouldn't need libvert or.
I wonder if that that problem that I ran into where you had to have a, you know, Windows VMs, if that problem is gone now with this.
Okay.
Um… Oh, I think it answered your question, but real quick, I think, Steve, you've gone to some of the rugs out there in St. Louis, right? I think we've met.
Okay, yeah. Probably so, because I work with Valerie Hill.
Alright, yeah, I know who you are. Uh-huh. Via right now by chance?
Um, yeah. What's that?
Did you go… did you work at Ameren as well, by chance?
Okay, I still think we've… I think I've run into you. I think… Yeah, if you've come to some of our workshops.
Uh… yep. So yeah, yeah. So okay.
But… yeah. Oh, there you are. Yeah, I… I think we were on some webinars together, at least, or like some… I think we did some Zoom intro kind of things.
Absolutely.
Um… Yes.
Yeah, yeah. Well, hey, good seeing you. Um, so to answer your question, okay, so we're kind of conflating two different technologies. So running virtual machines on OpenShift underlying technology is what's referred to as Cube Vert.
And essentially what you're doing is containerizing KVM and running that and using Kubernetes as an orchestrator to manage the pod itself.
So this does not change any of that. So you can still run. So that that technology still exists, right? Is running.
Right.
a VM on Kubernetes, or in… so, OpenShift, for those of you who don't know, OpenShift is simply Red Hat's distribution of Kubernetes. It's got more pieces on it, but it's… that's the best way to describe it. Um, the reason it runs as bare metal is because it doesn't support nested virtualization.
Okay. Yeah, and they always said that it was like a performance issue.
Um, not to say it doesn't work. I've seen people do nested virtualization on OpenShift. It does… it runs horrible. Uh, and just like, if you try to do it with any other hypervisor, it's gonna run horrible, because you have a… That's… yeah, you're gonna lose a lot.
which which agrees with what you just said.
Yeah, yeah. So so as of today, and I think even running QVERT.
Can you do it? Yes. Is it going to be a good experience? No, it's going to be horrible. Um, but, um, but yeah, so that's… but that's a completely different technology you still really want to be on bare metal when you're virtualizing systems.
Um… Yeah, so… But that's kind of a very different technology than what this is. This is really just the OS itself, which you could then run on, like, Kubernetes.
Does that, uh, answer your question?
Yeah, because I was, um… Well, I mean, the ones that need to be virtualized were, of course, Windows applications or Windows compilers, which.
Sure.
Yeah, I don't know if that. Yeah, because Microsoft came out with their.
Yeah, I mean, so…
net tools that would run on. Linux. And right, that was one thing that I did want to convert over, but they they kind of hesitate like, well, we don't have time to.
test that, so I didn't invest any time in doing it, but uh… But I think for anything… Certainly anything that could be run on a Linux-type operating system than I was, maybe thought I was stuck on a Windows machine, which would have been a VM.
I think this would have helped that along quite a bit.
Yeah, I mean, at the end of the day, if the VM requires or the application requires Windows, you're going to be stuck on that. There are Windows containers. Nobody really does those. They're kind of horrible.
Right.
Some folks have tried and it works. Like you can do it. It's just not a great experience. Um… From what I've seen, I don't… I have yet to see anybody do it and say, yes, this is the way forward. Most of the time, they say, you know what, we're just going to stick it in a Windows VM and.
Because sometimes you just… sometimes you need windows, and that's just the way it is. Um… And and so… As of today, I don't think… they're not doing anything that I am aware of, Microsoft is… that's like this, where you can do immutable Windows, um, installs. It'll be interesting to see. I wouldn't… well, because you have stuff, you know, like iPhone and Android, which…
you know, those do it. And so I don't know if windows is going to go to that path or not.
Yeah. Yeah, I. had done all that research with you guys, and um… We got to talking to our IS group because they had to kind of… they weren't going to end up managing it.
But then they say, well…
I don't think they had any experience with even containers or… Kubernetes, at least to speak of, and they said, well, what we're going to do, our solution is we're going to support you better with virtual machines.
I thought, well. It's a shame he didn't do that a decade ago.
And, uh, so that's what I kind of, uh… I thought, well, it's time for me to retire, and so…
Fair enough.
It's kind of sad, because I really wanted to get that working today. It was really close. I actually had, um… a Red Hat VM running probably Red Hat 9 at the time, just a minimal install, but I had Podman on that, and I was running…
build containers that were driven by… We were using Team City for our… Uh, continuous integration. And so that container was supporting container build. So I was that, uh… Red Hat VM was running containers in containers.
which was just the coolest thing I'd ever done in my life. And so I wanted to see it through. But once they say, well, no, we're not going to do that.
We're gonna do VMs. I thought, wow… Okay.
kind of wanted to go anyway, so that was my exit cue.
But yeah, tell Valerie I said hello. and the rest of the gang as well.
Absolutely. Yeah, yeah.
It's kind of…
maybe… is a variation on what Steve was just asking. But just in the last month.
we've had kind of a discussion about one of our members who.
It has a machine. Because of.
It originally being purchased by work, I think maybe I shouldn't presume that. But anyway, the the machine has to.
be running Microsoft Windows. And we got into a discussion of, well, wouldn't it be nice if you could dual boot it and also have… have Linux on there as well. And that then kind of morphed into, well, let's not dual boot it, but rather, let's just do it with.
containers. So let's let it be running Microsoft Windows, but.
So would… would… With this kind of feed right into that? Would this be an appropriate tool to to consider for this?
So no, um, so to run containers on Windows.
The ways that I know to do it is either with Docker or with Podman Desktop.
And both of those… actually run a VM under the hood, a Linux VM, and that's what actually runs your Linux containers.
Um, so… Now, you could run one of these as a container on there, but at that point, like, you're still gonna run a virtual machine. If you run a Linux on Windows.
And so this is just a different style of or deployment method, or… type of Linux.
That's not the right word I want to use, I don't think. But it's so your traditional package mode Linux is how I refer to it. It's this is the same as that. It just but you build it as a container image. But to run that container, you still have to have something, a Linux kernel at the end of the day.
And so, for on Windows, like I said, you're still going to be running a VM, and at that point.
you might as well… it depends on what your use case is, but you probably might as well just run the Linux VM.
I don't… Now, if you want to dual boot, like, yeah, this would be an easy way of doing it, but no different than if running your traditional. This just is a different way of managing the system itself.
Hmm. Okay. Okay.
Um… And it almost sounds like a variation in the question, but it's.
it, instead of talking about different operating systems, talking about different hardware, much earlier in the conversation here, you were saying about, yeah, back in the day and people were talking about getting stuff to run on PowerPC.
Now, of course, what people want to know is, can I get my Linux to run on an Apple M chip any experience with that? I mean, can can you get.
the what you were talking about here this evening.
Will this run on an M chip?
So, my understanding is that the M chips are ARM?
So, I believe so. So so it does. the disk converter does support multi-architectures.
And ARM is one of those. I'm trying to look at it right now, actually, to see if it'll tell me.
all the architectures, it will…
handle, but, uh, trying to find it. Um, yeah, so in theory, yes, I believe there have been folks who do get Linux running on.
the M chips. And so in this case, you would simply.
Build it for the right architecture, which would be arm, I believe. Oh, is it? I'm trying to look at that.
Um, I've been following a developer in the SI
I'm not sure how you pronounce it, Linux Distro.
And it has specifically been the process of
Corning, and I use that term loosely,
to the Apple…
M-series architecture.
Okay, very cool. Yeah, I'm gonna be honest, I haven't followed that very much, but I… I do my interest… in… My understanding is that it is ARM, but I think there's probably… I know Apple does a lot more customization than this.
It's ARM as in its…
arm adjacent, it is not arm…
proper intellectual property from Armed Corporation.
Ah, okay. Gotcha. So the only thing that would limit you here is… is really that the architecture of the kernel for the architecture. So.
in Cintas or Fedora or REL's situation where these Boot C images exist. I don't believe we do compile it for the whatever this M chip is. And so that's probably not going to work. Look, assuming that we've got one distribution that's probably doing and has figured it out.
Um…
Um, it's probably pretty unique, so… But if if the hardware, whatever that is.
Does have an R… if there's a kernel with that architecture, you can build it. So, maybe not this M chip, but if you start thinking about, like, Raspberry Pis or other small devices like that.
The bootable containers is super good for those, because a lot of those, you're running something as, like, a service.
And so you don't want to just go apply patches, so it's really easy to manage that.
from an image-based system.
Okay. Um… different direction here completely for a moment.
You use the term. atomic earlier, and in one of the little discussion.
mailing list things we were having earlier this week talking about this. You know, we saw the term atomic Linux. Would you care to define atomic Linux for us, and how that fits in?
Yeah, so Atomic Linux is going to be an immutable container-managed Linux.
So immutable being that. the majority of the file system is read-only.
Okay.
and image-based being that you're pointing to an entire image, and that's how you're pulling down your updates.
rather than managing the files directly independently.
Okay. input on it.
Uh, Matt, if I may, I just have a simple question here. What is the real purpose of this? Is it just to… protect the operating system from any malware that you might be dealing with in your operations.
Uh, is it to make it easy to install 2,000 workstations with the same distro, the same setup or what exactly is the reason for this?
Sure, yeah, I think it's all the above, right? So, first, you're defining your system.
Yeah.
From the get-go, from the start. So, you have your container file where you define exactly how this system is set up, all the packages that are installed, the configuration. So that's all shifted left to where you're probably going to manage that in version control.
Or you're backing that up. So, even if I'm looking at, like, a desktop system, I could customize my desktop system and the container file, and then if I have to redeploy because I broke something, because, hey, we like to tinker here, right? We're in the Linux user group. We have a bunch of hackers who want to tinker, and you're gonna… and we probably break our systems all the time from doing that.
Well, I've already written out, I've codified how my system is defined, and so now I can completely redeploy that really easily and get it back to the way that I had it.
with minimal… disruption. So so there's that. And and so that scales really, really easy back to the what you mentioned, you know, 2,000 workstations, absolutely, because it scales really easily. I only have to think about one container file to manage the image for 2000 systems.
The security aspect is another side effect that's super handy here, right? Because you have a read-only file system, so you can't just go and install.
malware that's going to get directly attached to the kernel. Um, that's going to be a lot harder to change, because it has to be written to the actual file system, and those hard links… so you have to do the composeFS piece of it. Um… And, you know, eventually malware is going to figure that out. Somebody's going to write it. But as of today, you know, it's really hard. Your attack vector is going to be much, much smaller. Sure, you can run rogue containers, but that's a lot easier to deal with than malware that's getting, um…
added to your kernel.
Okay, in terms of operation, how does this differ then from running a live distro from, say, a thumb drive?
So, you're, uh, I mean, so your day-to-day operation, you're probably not gonna notice, right? Um, so actually Chromebooks were using this style. They were using the… The underlying technology of how… OS tree works where you're you're hard linking your files. That image based style operating. That's how Chromebooks were right. So you when you log in and you use your browser.
Right.
you won't notice. It's all the same to you.
Mm-hmm.
It's when you apply an update instead of doing like, um… So I'm a Red Hat guy, right? So your DNF update, or if you're a Debian person, your apt get update, uh… You know, instead of doing that, you're just saying, hey, update based on my image. So you're saying, upgrade to the next image version, and you're pulling down the layers and just laying out layers, and nothing changes until you reboot. So your experience becomes very similar to your cell phone.
in that you're not thinking about it as individual files, but holistic image versions.
And if something were to break, you can roll back really easily, so you don't have to be scared of patching.
uh, you know, I think these days, patching is… boy, I don't remember the last time I've updated a Linux system and a patch broke things, like, really bad.
maybe the application gets a little funky, but the system itself, I haven't… I'm knocking on wood. I haven't had some bad patches in a good while, but, you know, traditionally, it's… It used to be kind of a scary thing if you had to apply a bunch of patches and reboot and you hope for the best. Um, but now, you don't have to worry about it, because you can roll back super easy. So I think there's a lot of benefits. Some of them are kind of side effects. I think the security aspect is kind of a side effect.
Um, if you think about it from a server aspect, right? You don't have people logging into servers. That's a huge security.
footprint that just. or service, a tax service that kind of disappears. So, um.
Yeah, I think there's… so those are some of the advantages. I think… it kind of depends on the use case. But but yeah, your your day to day operation, the way you use this system, that doesn't change. It's still Linux. You're still using your applications the same way. Your system differently.
Mm-hmm. Yeah, because there are distros that you can go in and apply updates, do whatever, and and then burn a new ISO for your thumb drive and and boot from that. And and it is essentially an immutable.
Yeah, yeah.
operating system. Everything you do on it doesn't matter, including a, you know, RM-F, you know, big things like I've actually done it just for fun. And, uh, and it's completely immutable. And you do your work, uh, writing to a temporary file system or to an external drive.
Uh, and the operating system is safe. So. I'm not sure, of course, you know, that's 2,000 thumb drives. I don't know how you would apply what you're doing to the 2000 workstations, I guess. Would you download the ISO?
or the container onto each workstation from a central server, or how would you do that?
Yeah.
Sure, yeah. So yeah, so the experience would be identical to that, yeah, just like an ISO. Boy, it's been a long time since I've done that. So USB drives, but thinking back to the days of Nopix, right? You'd burn your ISO to your… Your CD, and that was one of my first experiences with, like, a live Linux distribution. So yeah, it's going to be the same, right? Because you can't edit anything, but exactly that experience. In this case, it's the tool set that really makes the big difference, right? We have a toolset that is identical to the way we manage containers, and so it's just, it's the ease of.
Making your your configuration changes. So you define your OS, and we have a toolset that's just really good at managing.
images. And so for 2,000 workstations, you're going to have your initial deployment, and that can be through pixie booting use like kickstart. Or if you have someone going around an intern that throws a bunch of ISOs on there on the, uh.
CDs or USBs, right? But once you get the initial system down.
Got it, yeah. Right.
Sure.
After that, you're pointing to a repository, and you're only downloading the differential. And so your downloads are going to be smaller, and then you can stage those upgrades as well. So, once it's all staged, you can wait to reboot at 2 o'clock in the morning so you don't disrupt your users.
Got it. Thank you.
Um, am I right to say that, like, minimizing computing resources is also, like, a component of that, or, like, a motivation for wanting to virtualize or containerize? Like, if we were to… I'm gonna make a bleak analogy here, but, like, if we were to imagine a dystopia where you own nothing and everything is a monolith and compute resources are controlled by the government, and the government is trying to squeeze as much.
computing power, you know, per application or per resource as they can, then…
Um, actually, no, this, it's actually almost maybe the opposite when it comes to resource utilization. Okay.
The resource utilization thinking of processor memory doesn't change versus traditional package.
Linux or image mode. container bootable container image Linux c sorry I'm mixing up my words here. A bootc image in a traditional package mode Linux.
It's gonna run identical as far as processor memory is concerned for the most part. I'm going to say that in theory. There's always weird stuff that happens in computers. But for the most part, those should run identical.
I could actually see how
Bootsy image-based might actually be…
a measurable…
percentage better than a traditional
fat VM, because you have some inconsistencies between FATVMs.
Whereas the booty image-based VM,
is going to be cookie-cutter stamped out,
And you can leverage hypervisor deduplication, and the each cookie cutter
will be identical.
not close, but a little bit off, it will be a rubber stamp.
as long as they're the same image.
pulled from the repository.
So you might get more savings…
That way, but it is still…
AVM that is running.
Yeah, at scale, yeah, I'd say that would apply. Now, disk space, on the other hand, in this case, you'll probably see slightly larger disk space utilization for for boot C.
Simply because of the way that you're downloading, um… the different… your… well, I don't know, because you are holding on to different versions of the binaries.
that turns into a repository management, but that's still a shared resource.
Um… Yeah, it'll get trimmed up.
And you you'll probably run into the the layering problem that you have with containers to where if you.
Some parts that you changed… Um, you might have a new version of entity X, but the older may stay still be there if it's in a, you know, higher level layer.
just used.
So, it doesn't… it doesn't use over… overlay FS like containers do, so it does purge… it'll only keep… The version of files that a… one of the… so you only have two partitions, basically. You have a Part A, Part B. Um, it's only going to keep the files that are in one of those two sides, or it will purge it, so it's not going to get astronomically bigger.
And when you have RPMs when you do an update, you do replace like so if you update SSH, it's going to delete the old SSH and install the new version. So you could potentially have two versions of a lot of binaries.
So, similar to the way containers happen, but it won't be quite as drastic, or containers can get quite large as you add a whole lot of layers.
Yeah, there were times when I thought it's going to be best just to.
I'd clear out all the images I had locally so that it, you know, was forced to rebuild from all new. I'd have to download all new stuff if I'd made a bunch of changes.
That's how I clean house.
Yeah, and there's tricks that you can do to… Yeah, there's tricks you can do to help optimize that and fix that issue, but it won't be quite as bad, but it will potentially take up more disk space, um, than traditional, like, simply because you have two versions of binaries at times. Um, and then if you're running a bunch of flatpaks are because they.
They are containerized applications, not OCI, they use a bubble.
Um, little bit different technology, but under the hood, they all point to the same thing, really. Um… But because you're duplicating libraries.
on that side of things, Flatpaks become significantly more disk space utilization than… Traditional package.
versions, um, a little easier to manage, right? Because now you're not having conflicts between.
Your application version libraries and your system version libraries, which was always a challenge, right, with Linux, and so that goes away, but you do lose out on the disk-based savings of dynamically.
linked binaries.
Would it be fair to say that
this… not necessarily repeating myself, but this starts off as a container
to amalgamate all the pieces.
And then, as part of the build process, it essentially does a recursive copy
of that amalgamated file system to a monolithic file system,
that it is just…
a single copy of files, there may be the A-B duplication.
But instead of overlays, it's…
Dare I say, analogous to, like, a recursive tar.
of the files from the container.
copied to an image, and I think you said it's a QCO2,
disk image, and then
That is what you're booting.
henceforth is the VM
of… that's running that QCal image.
I think most of what you said is accurate, yes. So it uses what's called ComposeFS to build the file system, the root file system. And so it's kind of like object storage. It's referred to as content addressable storage, if you're familiar with that.
But essentially, you've got an object layer that has the hashes of each binary.
Um, and then you simply do a hard link to the binary, um, based on the commit.
Right, but once you have…
that…
source file set, you're
copying those…
into a disk image.
And then subsequently, the disk image is what's booted and run inside the VM.
This image is running the ComposeFS file system, the OS tree that does, it still uses this.
addressable file system.
Can you… so if you boot back up one of your BootSea…
Can you see the different versions?
And that content addressable from inside that.
VM.
I think there's a way to do it. Um, but I'm not sure I know how.
But let me see if I can… Because, oh, I think you have to use OS tree to do it.
So I'm testing out some commands to see. I think there is a way to do it.
Check some more file. I yes, there is a… I believe there's a way to do it. I don't know how to do it.
Okay.
But yes, I think there's a way to actually show the.
all the links. Where the file is hardlinked to.
Can you…
log in to Boot C.
And share your screen and run a couple of test commands.
Yeah. Give me one second. Let me swap over.
What are you laughing at, Craig?
Lane, you're on… you're unmuted.
I think that's just a, uh…
Cool thing to ask, like, yeah.
I know, you're probing some things.
Well, I'm curious, like, I wanna…
long listing of…
User Ben Bash, or whatever.
and look at, uh, the mount points.
And, um…
I wonder how many hard links there are.
Right, right.
On it, etc.
By the way, that's the… that's the joy emoji, not the laughing emoji.
Oh, okay.
So here is, uh… These are Ben.
Yeah. Most of those look… uh, do a mount, please?
And then you probably want to see… So, as you can see, so it's ComposeFS.
Okay?
Alright, uh, do a DF…
Mounted read-only.
spaced dot.
Okay, so it's coming from ComposeFS.
Um…
I think if I might be able to go to…
do… the other thing I'd be interested in is, can you show your kernel command line?
demessage, and then I think it's…
grip casein-sensitive command.
minus i.
Thank you. Command.
Yeah, I believe.
Command line boot image! Okay… bootOS tree default, what? Yeah, I see that, and there's your kernel, and…
Yeah, so I think this is it. I think this.
Hang on, I'm still reading a line, stop.
Boy. Oh, sorry.
Come on, please, that's okay.
It's few UIDs all the way down.
Yeah. Um…
Okay, nice reboot, zero.
Okay, thank you.
Yeah, because I think that I think this is where it's this might be where it's stored, because that's the commit.
Right? These are yeah. So these are the layers.
Um, yeah, so this is the actual layers right here.
CD dot dot.
Yeah, thank you. Yeah. So one and 2 right? So I have my part one and my part two.
Okay, um…
So I do a group C status. So currently, I'm booting the version 2 version one is probably the the.
The one here.
Lee will recall something from an email we were discussing. I almost feel like you're approaching what is
called boot environments in ZFS-based systems.
Interesting. I don't know enough about.
It's basically under ZFS, you can have many, many, many versions pointing to different things through mere
methods. And you're just picking a set of them that is this
Ah, okay.
call it commit, for lack of a better word.
to use as your running file set, and if you need to switch over to the other, you reboot, you pick a different boot environment, and it comes up.
That is that that's exactly what's happening here. Yep.
And I know… I'm pretty confident there is an OS tree command that will show you the hashes and the hard links.
Yeah, um…
PWD, please?
I just don't know that.
go to OSTreeBoot0.
default?
Oh, Justice Wooden. Yeah.
uh, go up one more.
And one more?
LSLA, please.
Okay, so you've got your current…
All right. So, buried in there that…
various directory tree structure are the different files.
stored by… yeah, you called it Content Addressable Storage.
Um, okay.
Thank you. Dat.
Yeah, no problem.
Katrunes. Box does that where it has a… ZFS… And then every time I run an update, it downloads a new.
Image. So if there's anything that goes sideways with the system update.
It will allow me to swap back to the previous image.
It seems like we're having convergent, uh…
Yeah.
Um, options here, like, all these things are converging in the way they do things. We've got now Docker, and we've got…
It's almost like the Alta…
this, and then we got… and then we got ZFS, like, like, they all kind of are starting to… to look the same, which is…
We do…
They do similar things, but how they do them
Almost like they all use these.
is different. Like, you've got amplitude modulation, frequency modulation, and phase modulation.
They can all carry music and news, but how they work is different.
But yeah, I think the… as far as, like, the way Linux does it under the hood, it's probably, yeah, pretty much the same.
It's almost like they all share the same internet and people talk. It's like, this is how I solved it.
It's almost as if talking and sharing ideas and learning and growing from each other is a good thing.
you're not sharing at the moment.
Okay, I found the command. Let me, uh… Yeah, I'll share it.
That's Wanda. Um, so essentially this is a command I'm pointing to the.
the specific commit, which I think is the commit that I'm… But so you can see the hashes that point to the all the files.
on the file system.
But there's a lot here. So if I were to do something like.
R. Oh, there's a lot more bashes than I…
So you can see the the hash of the.
Yeah.
the bash binary there.
Okay, thank you.
Yeah, no problem. That's why I wonder. I want to spin up a live system so we can do that. And I figured people have questions and want to.
Show me what happens when you type this.
What does it take to switch over to the…
other commit.
So in this case, because I I did a switch, I can in the old commits already on the disk. I can just do a rollback command and then reboot.
Okay.
That would be pretty much the speed of a reboot.
And how fast I can take. The rollback's pretty fast when you type command, but in the reboot is fast. If you update, uh… If I just do a… if I publish an update.
Then it was download the differential, the next layer and and compose that. And then I'd reboot into that.
which that speed's gonna dictate how much change there was.
Right.
But because nothing happens until I reboot, you can stage whenever you're, you know.
Yeah, because you're…
files in your normal…
the day before.
Unix directory tree structure,
are not actually changing. It's when you reboot
it's mounting a completely different set of…
file structures, and that's…
up with your new version.
Exactly. Yeah, yeah. So… In the enterprise case, where you're like, hey, I need to… I want to stage my upgrade, but I can't apply it.
you know, for in my experience as a Linux engineer, it'd be great if I can go ahead and stage all my.
My updates during the day. That's what takes all the time and then reboot, you know, in the middle when I, when I wake up 2 a.m. To do my actual change, a quick reboot and yep, all right, system come back. I can go back to sleep.
Yeah, so… I was under the impression when you
did an update and pushed it out.
And then systems subsequently pulled it down.
They were pulling down the entire new image.
Now it sounds like they're pulling down
I'll call it a delta that gets applied to a new set of files.
And then you subsequently reboot.
to that new set of files.
Correct. And that was because I made a mistake when I initially deployed the system. So I used the switch, the Bootsy switch command.
Which switches references entirely.
Okay, so you're not redeploying an entire new image.
each time you do updates. Your…
actually doing something into the running system to cause it to pull things down and integrate it.
into the running system in a non-active mode,
But it's…
not the same as a Yum or DNF.
or apt update, but it's something that runs…
inside the VM to pull in updates and prepare pointers for rebooting?
Okay.
Correct. Yeah. And that's what the C, the boot C tool does. So you publish. So the Yum update piece or the DNF update piece happens when you build the container image itself.
Yes.
And then once you publish that image, then yes, it downloads that differential. Bootsy will… look at that image and pull down that layer.
And then use the ComposeFS to do the pointing piece. Yeah, but yes, exactly what you described. I think is that's accurate.
Okay.
So, in an enterprise level, you have your…
build or development system, if you will, that is…
probably doing the YUM or DNF or whatever update.
or rebuilding the container, referencing the new versions.
And then your…
pushing the image from that.
to the repo, and then boot C subsequently in a thousand VMs goes,
oh, I have 200 megs of Delta to pull down and prepare.
And when I next reboot, I activate it.
Yep. Exactly.
Interesting.
Um…
Yeah, so…
Say that again, Grant, will you?
if… to compare this to a traditional software lifecycle,
you have programmers updating code or updating packages,
And then you…
take those and you, big air quotes, compile them.
or you build the new container on your development system,
And then once you have the container built,
you push that image to the repo,
And then you go to your thousand or so deployed systems,
And you use a bootc command, I think it was, of some flavor.
to tell that container to pull in its updates.
And it pulls in a small amount.
from the central repo,
adds it to the existing
file set space on the VM,
and updates pointers,
So that when you reboot the VM, it's now booted to the new version.
the fact it starts as a container is incidental.
And you can ignore that.
that's just a byproduct. The idea is you get the image…
put it in the repos, and then your systems are running based on that image,
And they can pull updates
from the repo into their own install, and it's not…
redeploying a new…
VM, it's pulling in small amounts of updates using BootC methodology.
I believe.
Matt's probably on mute and correcting me.
No, you're absolutely correct. I think nothing you said raises any flags on my side.
So, Lee, another way to say it is…
you could build…
use Boot C.
to create the…
image. I use air quotes around image.
that you want deployed on your client's systems,
You push it, go through the process, it happens to create a container, but ignore that fact.
And it ultimately ends up with a disk image,
you push that disk image…
to a repo, and then you go to all your clients' offices, remotely, log into them.
and tell them
I want you to download updates
using bootc commands,
And they pull from the central repo
and add pieces to their local storage repository,
And then update some pointers
So that when those systems reboot,
they start using the new version.
They have no real concept that there was ever a container involved.
Right?
Let me make one small correction, because you're pretty much right on point.
When you push an update, you don't have to convert it to a disk image.
Okay?
Okay.
converting to a disk image only happens when you need to do the initial deploy, and that like lays down the bootloader and all those things. Once you have a running booty system.
Boot C is able to pull from the container registry and pull down a normal Oci container, which is really just a tar file.
and do the necessary steps to. lay down that file system and make the new pointers.
Okay, so…
But otherwise, yes. Yep. So you just have to convert it into a disk image once you've already deployed.
you… you tweaked the…
central repo…
transport vessel, if you will.
But the existing system at Lee's offices would
download a copy of that transport vessel,
pick and pull the pieces they need to and add them to the persistent storage locally.
update some pointers.
and ready to reboot, but they're not actually using a container.
on the Raspberry Pis that Lee uses for many of his customers' things.
Is that accurate?
Yeah, yeah, and stuff like, if yeah, if you're managing a bunch of customers' Raspberry Pis, boy.
Does that make more sense, Lee?
I think this would make life way easier.
Yeah, yeah, I think so. I'm I'm just relating it back to the normal Docker container where everything is an overlay.
But I know Matt said that wasn't detected and involved here.
kind of helpful in the mental image of them.
The container image is an overlay, but the running system is not.
But…
Well, it's… it's got an overlay FS, though.
And and Bootsy convert does that conversion.
How does… how does the one overlay correspond to the other? Is it, like, converting?
That's pretty cool.
Yeah. It's not… like, the file system itself isn't using… it's not using a overlayFs. Like, if I look at the mount point.
Oh, I thought I did see… I thought I saw overlay in there.
It's it's it. No, it's Compose FS, not overlayFS.
Okay.
Okay. Okay, I was wondering about all that, too. So this is making it.
clearer for me. Thank you.
I almost feel like Overlay FS and what it's being used is in some ways very similar
to what Git does with the repo and the working directory.
Yeah, more convergent evolution.
Jerry, you scared everybody quiet.
I'm looking at this diagram, and it's helpful, except that
Yeah, yeah.
What does manage mean? And apparently, manage…
could even mean just to pull in a new image, right?
That's really, yeah. So that manages, think of running the Bootsy upgrade command.
a, uh, comparison I might make.
is deploy might be what my company calls the, um,
integration team.
they are in charge of bringing customers into the company, migrating them from
legacy systems to new things.
And then my steady-state team
does the month-to-month management
And figuring out what's going on, doing updates, troubleshooting, etc.
I think IBM emails.
I think Dowry. Yeah, I bet that. Yeah, is accurate.
If anybody has been in IBM land, Deploy would also be
RFS, or request for service or solution.
I, uh, I do my best to stay away from that land.
And that's all I'll say because we're on the recording.
Huh.
Fair enough. I'm a former IBMer, I could say what I want.
Um, what other alternatives are there? Like, are there…
other options I might want to choose in certain situations?
Um, I mean, we've talked about
What other options are we?
you know, containers versus VMs, but…
Are there… are there other tools that can…
give me this sort of…
I guess I'm not so worried about images, I'm worried more about the, uh, the overlay FS and how that's built, and, um…
I guess the immutable file system that you can basically switch a point or reboot in your…
Back on your previous version.
I'll ask the question another way. Is there anything other than Boot C
that is using Overlay FS.
like this.
or ComposeFS, right?
sorry, composed of a commit.
So the hood. Yeah. So originally this was all built on what was called RPM OS tree or OS tree and and boot C is really.
Bootsy used to do a lot of aliases to OS tree, and I think now they just share a lot of the code. I think Ubuntu has something called Ubuntu Atomic, which uses OS tree.
But they are not yet using a tool like Bootc that is easy. Os tree and Rpm. Os tree is really.
obnoxious in that in the complexity and and. difficulty it is to set up and manage.
Would…
The reason Bootsy was created was simply to make this.
easier to to build and manage.
Would it be fair to compare BootSea to…
tooling and automation around
OS tree, like you might be able to do most of what, if not everything, BootC does manually.
Potentially with OSTree.
But I don't have enough hair left in the world to do it.
Yeah, I think so. Because, yeah, we had so you'd have to like the way you do the repositories was different. It was an Http server.
Um, I don't… and so it was more like, um… It was more like the Git repository sort of thing, where you actually saw the hashes and things. You didn't have, like, the container toolsets and the.
a repository that's already easy to to manage and and deploy that everyone has already.
kind of like…
But yeah, it's the underlying technology is the same. Just like, you know, with containers, you know, containers were around for a long time. You've got C groups and, you know, managing your namespaces.
Uh, in doing Runci, and then that… you could do that for a long time. Dude, the namespace stuff, you know, but LXC was doing it.
But it was a pain to do it, and the tooling wasn't consistent. And then Docker came around and was like, hey.
Yeah, it's…
You can do all this neat sandboxing stuff with one command, and everyone's like, this is the best thing ever. It's… yeah, so it's like that. And eventually, those things evolve, and more feature sets get added to where it does grow beyond what it originally it was.
Right. Um…
Would you also be willing to accept
that Bootsy is to OSTree what Yum is to RPM.
Oh, that one might be a stretch.
evolution and enhancement of the latter.
Maybe. I'm gonna say maybe.
Okay. It's that last caveat you said other features are being added that make it
Morton.
Yeah, I like Grant's questions. One and the same thing. Yeah.
Yeah, yeah, yeah, it…
It may be… yeah, you know, it's not bad. You know, analogies fall apart sometimes.
3 or 4 beers at 1,000 yards.
Yeah, yeah, give me a couple beers. I'll say yes.
Okay, I'll be quiet for at least 30 seconds.
That's right. I know who I could ask, though, because I'm just a solution architect, but I'm pretty good friends who… with, uh, who used to be the project manager of this, so… I think I don't know, I can always… I can always relate questions to Ben Briar.
Good.
Stay tuned, readers. You might find a note and discuss from Matt this coming week saying.
Gary, did you receive the text message, lose my phone number?
Signed, Matt.
Well, we appreciate it if it shows up that, but certainly you've done a lot of work here tonight for for our group, and there's a lot of good information we've already picked up from you. So, uh…
Yeah, I'm glad it was… alright, I hope it was valuable.
if nothing else, you've made me think and go, hmm, and want to poke and try things.
It's super fun. I… so, I initially tried this with the the Rpmos tree toolset. And I spent a good amount of time tinkering with that, and I gave up in frustration.
Um, and so in Boot C came around. I tried out, and I was like, oh, this is what I wanted. And I I've been on the hype train since… since it's released. So it's… It's neat. And honestly, I mean, I've been doing Linux engineering for a long time. It's easy to get bored, so having something that excited me is… that's why I think it's special, is because I'm like, I got excited about it, and…
Boredom in an enterprise environment is actually a good thing, in my opinion.
Like, I'm pretty bored around Linux stuff these days, just because…
When you're working at NORAD, yeah, you don't want an exciting, exciting day.
you want your excitement to be, they've changed brands of coffee.
Right, right, uh-huh.
I'm gonna distract everyone for just a moment, and you can, in the back of your mind, be.
depositing any other questions before we wrap up for the evening, but I did mention at the beginning, this is election month. There's election for the two seats on the board of directors. It's a 5.
There are five people on the board of directors. They go on three year terms. So two of them are due this year. Lee and Grant are up for reelection. So anyone who has been at two of our meetings in the last calendar year, uh, basically the 2025 calendar year.
If you were at any two of our meetings, hey, you're a voting member. There are no dues. So just being here was enough to authorize you to vote. So you can either vote on the website, or if you want to keep it anonymous, there's instructions there how you can shove it in the U.S. Mail envelope and send it to us anonymously.
So with that, we encourage you to vote. Thank you very much.
Let's see, Lee, was there anything else I should be announcing?
Oh, I just started one of the things next Wednesday next month, the Wednesday meeting, the Wednesday general meeting in March, second Wednesday of March. The main talk will be by Scott Granaman.
It will be the next in his continuing series of The History of Computing, and it will cover the years of 2000 to 2010, which many of you probably are going to go, well, I was around then. It ain't going to be anything new to me.
This will kind of be that summary of, yeah, at the beginning of that decade, we thought we knew what we were doing and everything was coming, and then this happened. You know, when Y2K was a non-event. Yeah, when Y2K was a non-event.
So, uh, so yeah, all those little gotchas would somehow, by the end of that decade, changed our perspective and and what we were thinking we were doing. So yes, come to the general meeting.
on the second Wednesday in March, and hear all about it.
You ready for a bookend? Not yet. Let's see any any other questions directed to our speaker this evening and his topic.
Well, I have a question that may or may not be relevant, but, you know, I waited until the end.
Okay.
So I'm playing with Ollama. Uh, which, you know, is a… agentic AI thing. I just wanted to run it to see what it would take.
And unfortunately, it's, like, requires VRAM. which, you know, I don't really want to put into my laptop.
I feel like, um…
A lot of lots and lots of rebrand. Is there anything that you know of that is… doesn't require that much, like…
Sean Tweehouse has done some presentations.
on some LLMs, and I know some of what he presented on
did not require GPU, and it would use standard CPU, albeit at the expected performance hit.
But it would still function without super fancy GPUs.
Yeah, um…
Yeah, that's fine. I I I understood. I'm okay with that.
go to the SLUG website,
And staying around to help me, or Gary, or Lee, um, they've…
Okay?
Video Archive, I think the front page has some listings towards the bottom.
And I want to say it's within the last 12 to 18 months.
Um…
Yeah.
Well, I could do a search, too. I'll I'll look for that. Okay.
Sean Tweehouse would be the presenter, I believe.
Lee, do you have any follow-ups or corrections?
Okay, very good. Thanks.
I remember the presentation, but don't remember when we had it.
Okay.
We're scrolling right now. We might. We might come up with that.
Yeah, I looked up the Red Hat stuff. Apparently it requires quite a bit of VRAM, too.
Yeah, um, I'll say, so… so we acquired a company called Neuromagic, and they have some really cool stuff. Um, one thing that came out of them, I believe, was the VLLM.
I don't remember.
That was supposed to allow us to do inferencing. The way they talked about it is it does do, um… There's model compressions and VLLM can do better, more optimized inferencing.
Awesome.
Supposedly, originally, like, yeah, you'll be able to do it on CPUs. Um… I've talked to some of my AI guys. I work for these great AI guys that are, like, stupid smart, um… wants a PhD in physics, like, these guys know what they're talking about. So, I get over my skis pretty fast when it comes to AI stuff, it's just not my area of expertise. But I do work with these folks quite a bit.
Yeah, for sure.
And what I've been told is, they're like, yeah, there's some folks who, you know, they've got some things that they say they can… they can do stuff on CPU, but the reality is, it's gonna be horrible, it's not really gonna work.
Yeah, that's kind of what I… what I've been reading.
So there's some small models and stuff you can do. I think you could probably do some interesting granite, some pretty small stuff. Um, I would try and use granite on my… I've got a 4080, and I was still struggling to do, um…
See, the thing is, is I think that the… some of the applications that users, especially small businesses.
They don't really need very much at all. So I'm.
We're…
I'm really playing with the… the edges, of course, you know, I'm gonna… I'm trying to see what I can get done.
But, you know, whatever. I mean, if it doesn't work, it doesn't work, I… Might require a 16 gig.
you know, video card, I guess.
Grant, Grant, Grant your your suggestion on that PDF and the archive. March 2024.
Oh, okay, very good. Thank you.
Using LLM and RAG. By Sean.
Yeah.
I'll tell you what I'm seeing in the professional world is that, and you'll probably see this in the news, right, is that, you know, the memory stuff is getting super expensive because they've sold out, like.
Yeah, it's kind of ridiculous.
several years in advance, hard disks are becoming in the same… same way. And yeah, so right. Gpus have been this this way for a few years now. I don't see that slowing down at all. So anybody who can develop a technique.
to more efficiently run models, even if that's just inferencing.
Um, on CPU is going to make a lot of money.
I get… I gotta tell you that, you know, there's… I have some friends of mine who have lots of CPU available.
They're like, well… What about AI? And then I'm like, well, no, you can't really run it without GPUs.
So, you know, it's kind of funny, but so I'm trying to figure out a way around it.
Yeah, no, I can appreciate it, it just, but yeah, if that gets solved, whoever solves that's gonna be a rich person, because companies will go crazy over it simply because it will solve a problem that a lot of companies are having.
Well, I'm sure it's going to be very. specialized, but who cares?
almost one page.
If you need specialized, then it's specialized.
Well… Well, the problem is the video RAM, not the video card.
It sounds like something like the direct access. method that iGPUs use to use the system memory for.
VRAM or, um… Current.
Uh, certain graphics cards are able to load textures straight off of the.
The, um… Uh, NVMe SSDs.
Oh.
Instead of having to go through the CPU. I don't know anything about how they do it, but… That's the only way I can think of going through.
Working around it.
Yeah, it requires a little investigation on my part. It was just a question. Thanks. I appreciate the answers.
That's cool.
Sure.
Hey, Tony, I got 2 things for you. Number one, there's a bunch of Tesla P. 40s on eBay right now. They're Nvidia.
per 270 bucks, because I got one of them a little while ago.
Oh.
And that would be a card you could actually put in a full-size machine and have a decent amount of memory and Gpus to play with.
And the other suggestion is, if you don't want to go the hardware route, open claw.
O-P-E-N-C-L-A-W. Okay.
Yeah, yeah, I'm familiar with it.
Uh, will you describe what it is for others watching the recording that might not be familiar with it?
30-second spiel.
Open claw is one of the newer pretty much. I'm not sure if it's all open source, but I I think it's is. It's a GitHub project.
And it has a lot of capabilities, including building your own agents.
Thank you.
Hmm. So, you understand what happened with that on the security world.
Yeah, but from what I've heard, they've been doing doing some patching for it.
Right, there's a… there's a 3-hour interview with Lex Friedman with the Open Claw developer.
Steinberger or Spienberger, or something like that, I don't know, I'm butchering it probably.
He's from Austria. But I mean he is like a savant. He the way he created that whole thing. He.
He used 8… open cloud, I mean, or clawed.
Subscriptions, and he was running them simultaneously. you know, developing the code for it.
I mean, if you're interested in that, that's an interesting YouTube video, but whatever.
But yeah, I mean… I… I'm not sure if I want to create agents with it, but it's… this is, of course, a… uh, you know, I'm trying to help… I'm doing it for the same reasons Red Hat talked about.
It's for the privacy reasons. you know, small businesses don't. You're not supposed to put.
private information into the… Cloud LLMs.
Yeah.
Although a lot of people do. But you're not supposed to. So what do you do? How can you use AI technology?
That's why I'm looking into it.
Yeah, that's a that's a big thing on on my bucket list, too, is sovereign AI.
I feel like there's a need for…
what I'll call a shim.
that appears as…
a GPU with certain amount of memory,
that is really software,
that can take…
the commands they're given and hand them out across the network to multiple compute nodes,
for the CPU,
the distributed CPUs to do the computing,
and hand it back to the virtual GPU to hand it to the AI
LLM software.
Right, and and it could even reside in RAM, because if, you know, some of these people have tons of RAM.
Mm-hmm.
In terms of CPU at this point. you know, and anyway.
So… There is a company that creates, it's called software defined memory, and essentially they built what is like a SAN, but for memory.
It's called Code. Now, I can't speak to any of the validity.
Right. Okay.
of the product itself. I'm familiar with it. I've heard of it. I've heard their sales pitch, but I don't know if it works. They claim they can get.
memory performance that is the same as local memory. I feel like that is not.
Possible, because of physics. But. Yeah, KOVE. But they essentially build these software towers or the memory towers, and I feel like that's kind of what, you know, like Grant was saying, you kind of have this. You treat it like we've treated storage now. Um…
Will you type that name into the chat, please?
And when I first heard about this, I was like, this is amazing, I don't… see the value of this. But now…
that kind of clicked. I think that might make sense in this scenario.
I'll do you one better and put the whole URL.
Thank you.
Yeah, there there's one problem with that right now.
Memory is almost more expensive than GPUs.
But then you can get… More dense memory into a system, I mean… I got thread openers that can.
or Xenos, I can take what? Dozen sticks.
Whereas with the GPUs, you're stuck with whatever's soldered on the board.
Yeah, but the advantage of Gpu is they're pretty. They pretty much optimize token processing to run in that hardware.
Right, so you would have to do something to emulate the NVEC.
So that's going to give you.
Yeah, the NVAC, NVIDIA encoder or the. What is the… We are.
GPU and, uh… The reason why everything runs on the Nvidia GPUs is because it…
That or, you know, you create your own inferencing chips to that can use standard DDR memory.
Yeah.
Or the system memory. Which is the big bottleneck right now is…
Everybody who's running into is that you… you're stuck with the, well, the only other way around that is, um… How did Apple get the, uh… open claw to run across four machines.
It was over Thunderbolted and it was somehow the the four.
Apple IMAX were sharing memory over the Thunderbolt. It's like RDNA Jeff Keerling did it.
If you want to know he he had a mock up that got or a demo unit sent to him and he did a video on how to.
Basically… Cluster for… IMAX together so that they can share their unified memory.
Across both the… EM4.
System memory and… Uh, the M4 GPU.
And you… It was like, oh, I'm never going to have 40 grand to go buy.
Yeah.
Well, I think he returned them.
for IMAX. Well, he, he got those, yeah, he got those sent to him and he had to send them back, but…
Um, so looking at Cove, I see what it's doing.
It is using some software features
to abstract some things,
And it is using InfiniBand
As the network technology, which is…
usually, um…
multiple years.
ahead of Ethernet.
for higher speed and lower latency,
So they're using very specialized interconnections between systems.
And they're aggregating memory
through controlled access.
between nodes in the pool.
So you can run something on…
some systems, and it can think, oh my god, I have
terabytes and terabytes and terabytes of memory,
And it is accessed.
across InfiniBand.
And RDMA over InfiniBand, and…
has… it's been a standard. It just… it's…
That's it.
very, very, very niche.
Oh, anything that's needed to go that fast, it's like… Call for price.
Yeah.
So yeah, the whole… sovereign AI, the biggest problem.
I guess everybody's… has, is that the… Yeah, whenever you shove something into one of these… Online LLMs… They're seeing everything.
So it's… they can do with whatever they want, with whatever you put in there.
Yeah, we're still at the very beginning of this technology. So it's going to change quite a bit.
Yeah, right. Yeah, oh yeah.
Oh, I'm ready for the collapse. This all smells like .com I'm like… Okay, I'm going to sit over here, watch… A bonfire? Because I remember this thing called Yahoo Pipes.
Mm-hmm.
Okay.
I think we're already starting to see…
Yeah, it's gonna be interesting.
early signs of the collapse.
Vibe coding is…
really hastening the…
failure, and there's already people going,
we've tried to use AI LLM vibe coding for things, and it has caused problems galore. We need
people that actually know how to think to come in and fix things now.
Yeah.
Yeah, the the open club. developer, he's like… You have to try to understand, to think like the.
agent or the… the way the the the AI is thinking, which is.
No.
Not like a regular programmer. So that's really hard to conceptualize.
because basically the AI doesn't know anything. Do you have to build everything for it?
It's easy. It's an interesting guy that… But, you know, I mean, that's why I say we're at the beginning of it. Everything is still being built.
A lot of AIs are going to go by the wayside.
Right?
Oh yeah, it's it's .com 2.0. Right now, everything, anything related to AI, the people with money are throwing money at it.
Yeah.
And within, I would say, the next 18 months, there's going to be a great refining while you're not working, so we want our money back.
Not alone. I hate to put it this way, but Sam Altman has pissed off anybody there in the technical field.
The moron went out and decided, hey, what happens when I try to corner the market on memory and.
Yeah, they… they've overdone that that and they're probably not even going to build everything.
broke… Oh, I won't be surprised if they never finished that Stargate project.
Yeah. Well, look at… just… I don't know if you get the West News magazine here in Baldwin. I'm in Baldwin.
But they had an article about 2 or 3 data centers in this area.
that they're discussing building. And of course, you know, there's a few… People upset over that.
Missouri, right?
I mean, but. It's kind of crazy. Even around our area. There's.
They're building this all over the place in this country.
Yeah. Oh, everybody's like, oh, all I have to do is.
Buy a whole bunch of GPUs and hardware and I'll get this guaranteed income. I'm like.
No.
No. But there's just, but I could go to the bank and get a loan for this because it says AI on it. And I'm like.
Yeah.
Any bank throwing money at an AI startup. without some kind of… plan to monetize it?
is currently a complete moron. And deserve to go.
Well, I'm gonna… how many railroad companies were consolidated to what 2 or 3?
I think… Oh, there were hundreds of them, and we were down to, like, 4.
there was like hundreds of them.
Um, same thing with aerospace, same thing with airplane, with, um.
oh airlines, um…
The thing that I just really irks me is that.
Yeah, it's.
When I joined IT. It was kind of this unspoken democratization of power. If you were willing to put in the time to learn how the hardware worked.
You could go out and get yourself a used… XYZ system, and you can make things happen.
But now it's like, well, no, you're gonna have to be one of the… Mosh.
Oh. Uh, have the massive amount of collateral.
Or debt… To even… uh actually play this game anymore I'm like.
Why would anybody want to deal with you guys anymore?
Yeah, well. Alright, well, I got lots of information. I appreciate everybody's answers, Allah.
if I come up with something, I'll let you guys know. But in the meantime, I'll just… Work on trying to get something with very little.
video stuff.
It's one of the smaller, uh, smaller… Alarms running?
Yeah, I wanna… I wanna see what can be done with the smallest there is.
We'll see you.
Well, Jeff has a whole bunch of stuff talking about running that open law system and the models that can run on it.
Some of them are down to only a few gigabytes.
So I'll check that out on the on the March 24.
Let alone.
Thanks a lot. So I appreciate the it was an interesting talk. Thanks a lot, man.
All right.
And thank you, Tony, for the interesting discussion. Any other things that we need to touch on? I know we've run a little bit over time, but I think we were all.
well involved in that conversation, but, uh… Probably we could be wrapping up.
You guys see what… The, uh, shenanigans that Boeing's up to?
Any final thoughts?
For those who haven't heard, it was announced yesterday that Boeing's defense systems headquarters are going to be moved back to St. Louis in 2017. They were moved out of St. Louis and moved to Arlington, Virginia.
Many people said that was so the people in the C-suite could appropriately suck up to politicians in the Washington, D.C. Area. I didn't say that. Am I still on the recorder?
However, last year during the union strike, which went on for months and took forever to resolve, apparently the board of directors thought, you know.
Maybe it's more important that the bosses actually be in the factory occasionally and know what the heck's going on in the factory, and how the plane to be cooked.
Who worked dog.
What a thought. Yes. So.
So, that's along with their building some kind of… Intelligence.
hub with it for the gov or for the Department of.
fence in North County. So yay more jobs!
Yeah, and that did. Well, I don't have any details on that or knowledge of it. You elaborate on it a little bit?
Oh. Just what the News 4 put out, plus, you know.
Any idea what they're doing there?
Don't tell my boss, but I've been applying to those jobs.
Mm-hmm.
I found one the other night. That's literally my old man's job.
Oh, well.
Literally just put together a managed training hardware and events. So I'm like.
For 180,000, I could do that.
Yes, sir.
Gary. My grandson. My grandson left Raytheon in Tucson, Arizona.
And now… Working at Boeing.
here in St. Louis. Well, okay.
Here in St. Louis. And he is a math whiz.
And so… He was a top secret clearance at Raytheon and is now doing that at Boeing.
Ah.
So… Yeah, he, he can't tell me what he's doing.
on.
But you might know something about that.
Mm-hmm.
That would be… yeah, that, I mean… So yeah, if, uh… If you can get him to tell you any unclassified things, maybe you can bring it to the next meeting, and.
Let Tyler and others know what's being discussed or what's what's being prepared.
Yeah, you don't even.
Oh, I have three people. Uh, at my local church that worked there, and… fourth that retired and I'm seeing here going so is it what?
What should I do? Prepare for them. They're going, it's all secret.
You got to get used to not talking to it about it outside of work. I'm like.
Damn.
Are they willing to sponsor anybody to get clearance for jobs?
There you go.
Yes. So that's one thing I did know. Most of these jobs I've looked at are they're willing to.
The security clearances post-start requirement. I am a boring white guy with, you know, you guys as friends. I don't think I would have any problem getting a security clearance.
I resemble that.
I got to give you all my quick classified secret story. So.
uh, over at St. Louis U was working in the wind tunnel water tunnel lab. And we had a technician come in.
From one of the companies that built the hardware. And so he's working in there on the tunnel and he keeps looking up on the wall and on the wall.
We had a bunch of models that had been, you know, done as part of people's master's theses in aerospace engineering.
And so anyway, he keeps looking up there and staring and finally at some point he kind of shakes his head and grumbles and said.
I was told that that was classified and I was never supposed to.
tell anybody about that or mention it or describe it. And we finally realized that, yes, one of our… one of our graduate students, apparently his thesis got became a classified project up at Boeing.
So there's actually a wind tunnel up here.
Oh, yeah, yeah, there's wind tunnels and water tunnels at.
But, uh, we're… Holy.
at the engineering school that blew.
Oh, a slew. I thought you meant Boeing.
Yeah, no, no. Well, at Boeing, they they may have them also, but uh…
Anyway, um… This has been a really enjoyable evening. Thank you very, very much, Matt. All kinds of good stuff here.
And thank you all for coming this evening. And let's see, I guess I think we've taken care of all final comments we need to. Yes.
All right. Well, our bookend for the. for the whatchamacallit? The recording. This has been, uh… Thursday, February the 19th, 2026. This has been the monthly St. Louis Linux user group, St. Louis Log Meeting. Our speaker tonight was Matt Hargrave from the Kansas City Office of Red Hat.
City Office of Brentca.
And, uh, his topic was Boot C, the wonders of Atomic Linux and Containers.
So thank you all for coming and see you hopefully general meeting or one of our other meetings before that. General meeting, Scott Graneman Wednesday next month. Bye.
Good evenThank you.Have a good