Alrighty. Hang on, recording.
Good evening, everyone. Welcome to the St. Louis Linux Users Group, St. Louis Lug.
Uh, this is Thursday, the, uh. 90? 18.
18th.
18th of December, excuse me, 18th December 2025. Uh, tonight, our speaker is Lee Lambert.
Uh, he is doing a talk on the wonderful world of file systems.
It's kind of a follow-on talk to his talk last Wednesday at our general monthly meeting.
Where he told us how he rescued his file system.
Uh, when it had a power crash. And, uh, so this is kind of more of a background of.
What are file systems, how they all… What the different ones could do for you. So, I'm going to turn it over to Lee Lamardt.
Uh, we're not doing the intro stuff. Oh, uh, let's see, we… Let's see… Stan, do we need other stuff in the intro?
Alrighty.
Well, you know, I do have all these slides, you know, and… They do cover all these points, and you could run through them in less than 10 minutes if you were really, really fast.
Okay, we'll do that. About us, the St. Louis Lug.
is, uh, part of the St. Louis Linux Unix users group. We're a 501c3 corporation.
We've been around for, I don't know, it's been over 30 years?
And, uh, so the lug meetings are usually on the third Thursday of the month.
Sometimes on the fourth Thursday, if there's a little conflict.
Uh, and we also run a general meeting, uh.
Uh, you're picking up stuff that's not on the slide, Gary.
the Unix in general, open standards in general. Um… Okay, uh, so… Alright, uh, the agenda for tonight, um… Okay, the meeting electronically opens at 6, lets you check out your microphone, sound system, screen sharing.
6.30, we actually start with the introductions, announcements, and uh… Uh, description of, uh, what we have, like the calendar and the archives that are available on our homepage.
And then, uh, about 6.45, we'll actually go into the meeting.
Share of Lug is James Conroy, your emcee tonight is me, Gary Meyer.
Our host is Lee Lambert, who's the Chief Scientist at Omnitech.
He's also the treasurer of SLUG. Uh, our production head is Stan Reichert, who's the Executive VP of SLUG.
And boy, does he keep everything going, including keeping me on the darn, uh, agenda page here, so… chat monitor. Uh, we're willing to have anyone who wants to to volunteer. It's basically to monitor the Zoom chat area tonight.
And, uh, if somebody. Uh, type something in, and it should be addressed right away.
Uh, you bring it to We's attention that, hey, Lee, can you answer this right now?
So, anybody want to volunteer to monitor the chat?
How much are you offering?
Don't all do it at once. Okay. It was an offer? All right.
No, no, I'm asking how much you're offering, like, how many
mutations can I skip?
Oh!
He's offering, he's offering not to put you on the ballot.
Damn it, I thought I was bribing Stan for that.
There we go, yes, there we go!
Oh my god. There we go, there we go.
It's a team effort.
All right, I'll take it for the team and watch the chat.
Yep.
Thank you, sir. Alrighty. Topic regular. We are always looking for some people to help us, uh, find, uh, speakers for future meetings. You can either be.
finding interesting topics, and uh… and then use that as a guide to find a speaker who can do that topic.
Or you can find somebody who's a good speaker and just wants to talk about something, and then we can find a good topic for him, so… If anyone would like to be a topic wrangler for us, uh, talk to me, or Stan, or Lee.
Uh, or somebody else in the steering committee, and uh… we'll help get you organized.
So, if anybody has any ideas. Again, talk to those people, and uh… we'll try and follow through.
Speakers, as I just said, we're always looking for good speakers, good topics.
So, uh, yeah, if you'd like to volunteer. Say something to one of us, and we'll… we'll get you set up.
announcements five days from now, usually the fourth Tuesday of the month, is the new Linux meeting.
Uh, so, uh, the new Linux meeting does not have a specific topic.
speaker, it's just whatever they decide to talk about that night.
The next slide, the St. Louis Area Computer Club meeting. It's usually on the first Thursday of each month.
Speed.
Same thing, they don't have a set topic, a set speaker, they just talk about whatever they want to talk about that one.
Uh, slew general meetings the second Wednesday of each month. That does have a set topic and speaker, actually usually two of them.
And, uh, note there, uh, this last October, in other words, 2 months ago.
Uh, Microsoft Windows 10 had an end of life, and there's still people who are going, what do you mean by.
Microsoft Windows is out of date, so a lot of people all of a sudden say, we want to switch to Linux.
Come on, Gary, speed it up, speed it up.
So, um… Okay. Uh, Project Hail Mary, uh, the link to the official track.
Uh, next year, on March the 20th. So, that's coming up.
And then, uh, call for other announcements. Who's got any other announcements that need to be made?
Well, usually they're…
requests for questions.
Go and watch, going twice.
from people that can't wait, or is that coming up yet?
Okay.
It's coming up yet.
I do have a question for Stan, uh… The, uh… the Slack meeting?
Uh… anyway, uh… Something has come… one of our meetings is coming up on January the 1st.
Uh, we haven't talked about that. Are we going to have that meeting on January 1st?
No, it's gonna be moved to the following week.
Okay. So… Okay, so we'll get that on the calendar, get that straightened out, but for those of you who are thinking of coming to the January 1st meeting, it'll be a week later.
Okay, next slide.
Okay, bye-bye Windows 10. As we said, October the 14th, Windows 10 hits end of life.
Uh, there's still people who are looking to. how to deal with that. There's a paperback book out by Amazon, there's a Kindle edition.
There's a PDF that's freely available. Uh, it clues people in, but yeah, a lot of people want to just get off Microsoft and get to.
Linux, so help your friends out. Next slide.
Uh, St. Louis Unix user group, new Linux Beginner Workshop, uh… There's our website.
And our mailing addresses?
It's on the forest Tuesday. Normally, Forest Tuesday of every month.
From 6 to 9 central time. Hmm. Is it end of 10.org is a project out of Europe is to help find Linux support.
A good sight, um. Next.
Privacy and security. Um… We, uh, record these two meetings each month, where we have official speakers.
And we put those into our archive, so they're available for people to look at later.
However, do remember that not just us, but anyone who is in this meeting can be recording it. So, uh.
Uh, don't say anything that you don't want recorded.
Presentation archives, as we said, that is available. If you go to either the SLUG or the SGL Linux homepage.
section down there that's called Archives. Book it up by date, and hopefully it has the title.
And it'll hopefully have recordings, uh, slide decks. Chat, session, all right there.
on our calendar, SLU maintains SLUG.org slash calendar. Uh, it has both our meetings, which are highlighted in red.
And it also has… meetings done by people other than us.
or presentations by other than us that are of a technical interest, we think.
And we think people in this group may be interested.
Mailing lists, um… Uh, let's see… yeah, our members sometimes have questions, uh, want to ask them, so there… our mail is there.
When we're holding a meeting, the announce list is very limited traffic, it's an edited list, and usually.
Well, the only thing that goes in there is an announcement of our upcoming meetings. Every now and then, something's important enough.
It gets an announcement there. The discuss list is open to anyone who's signed up for it can post anything they want to question.
An observation. It's just like being here in the meeting. You can chit-chat about whatever you want.
steering committee, if you like seeing sausage being made.
come to our steering committee meetings. Uh, you can help run this group.
Uh, talk about your ideas, and talk about what we could do better.
sysadmin group. Obviously, we run our own mailing lists, uh… We've got our archives, our web pages.
And we keep Linux systems running to do all that, and the sysadmin list is basically a list of the volunteers who are being sysadmins on our group.
Um, so we hope that people on that group.
Uh, sysadmin list would be working. And helping to administrate the systems.
And, uh, so it's not exactly open. Uh, to anyone, uh, since we might slip and divulge.
You don't have to be on the sysadmin list to ask system administrator questions.
some security.
Right. Correct, correct.
You can do that on Discuss.
Okay, groups that are affiliated with SLUC, there's the St. Louis Linux Group.
the lug, uh, which is basically a subdivision of the group that I said it's specifically for Linux, rather than just open standards.
Uh, the new Linux group is, uh, for people who.
beginner workshop.
consider themselves relatively new. It's great, however, for. I'm gonna call this one. Yeah, it's a beginner workshop, but… Those who are experienced, please come, because you can learn a lot dealing with the new users.
And it's really helpful for you to give your insights to the new users. So, uh, so yeah.
Older users, please go. Slack, St. Louis Area Computer Club.
Uh, it's more hardware-oriented, and we'll even talk about Microsoft systems or other systems.
Um, so, again, both the new Linux and Slack group have no preset agenda, no preset topic, it's just whatever people want to talk about.
Next. Call for help. Before we go on! If anyone would like to, I mean, after Lee's talk, uh, we can go off into any general questions or general comments.
Uh, but, uh, if anybody feels they can't wait that long, or they have to leave early, and you'd like to get in a quick question.
Go on once. Going twice?
Now is the time. Uh…
hang on. Um, Jeff Jackson had a question.
in the chat, and it also sounds like he might be having
Problems not hearing.
Uh, Jeff, did you get that resolved, or do you want me to…
relay your question?
And you are muted.
So, Jeff, as you can see in the chat, had a question about printing from Linux, which…
Ouive. That's, uh, can be a loaded topic, and uh…
I'm sorry, say again, Lee?
Unlimited topic, I mean. I can say, I consider it an unlimited topic because there's so many rabbit holes you can end up going down.
Yes. Um, Jeff, if you want to type in
And I'll type this to him.
That's, uh, working in the right direction.
He's unmuted now.
But, uh, we are multimodal here.
Happy to, uh…
Converse via text as well.
Um, looks like his microphone might be working.
Zoom is a problem.
Stan will tell you that.
Oh, gosh. You wanna know how I really feel?
Let's keep this PG-13, not R or X.
All right, um…
Excuse me.
perhaps Jeff can work on his audio issue, and we can try to help him with the printers.
or Windows issue later on.
I'd like to know what…
issue he has with his Windows laptop.
Fair enough. He hasn't piped it yet, and he's having audio issues. Oh, he can hear us now! Okay.
Uh, do you want to…
hey, we can hear you!
Winning!
Yep, I can. Um, yeah, we're doing good now.
It was my browser, there was a little button for the speaker, and I clicked on it, and it gave me the audio.
Oh, okay. Um, so what's your question about printing?
Real quick.
Well, we've got, like, 2 Linux systems, laptops. And a, uh, Windows laptop.
They're having problems printing to an HP laser or office jet.
80-20, it's a Wi- wireless printer. And it has an IP address, and I tried pinging the IP of the printer, and I couldn't get any response back.
Do any of your systems have the ability to ping it?
I haven't tried more than one workstation. I've only tried one workstation with the.
with the pain, so…
Okay. Um, I would suggest testing to see if any of the workstations can ping it.
And, uh, let's start with the network aspects.
Mm-hmm. Okay.
And once we know we have a communications path,
then we can divulge into one of the rabbit holes that is printing.
Well, the weird thing is that it actually finds it automatically.
Dante.
the printer, I removed it, and it re-found it.
We've discovered it.
Dan has 2 questions. what Linux distro are you running, and do you have your firewall turned on?
Yeah, I had my firewall turned on. And it's Ubuntu is the Linux version.
Uh, you gotta…
In the firewall, you may have to go in there and tell it that it will allow the.
Printing. Cups. She, she UPS.
Which allows port 631. And then… okay.
Uh, you can configure it. either shoe… Um, graphical things or whatever, and sometimes you can go into.
The browser, and you can say localhost colon 631, and that'll take you into the COPS configuration setup.
If you don't… if your distro doesn't have. a specific.
Uh, application for setting up your printer. Most of them do now.
But the cups… via way of the browser.
Local host colon 631 sometimes is a good way to do your configuration.
Those are just some suggestions.
Okay.
I've done.
Um, do we want to continue with the presentation and give Jeff some time to…
poke at things.
Okay.
I think it'd be a good idea.
To point out that our presentations are at this meeting are usually focused toward Linux.
Either at the… higher level at an enterprise level, or at the personal level.
Uh, sometimes more advanced and. and so forth. Uh, the apprentices send notes in PDFs and scripts and so forth to the editor at SLUG for inclusion in our presentation archive page.
Um, do you want people to interact with questions and comments, or.
Or hold till later.
Question to you, Lee. Coming inverse. So, yep, go ahead and speak up if you've got a question for Lee.
I mean, during his talk.
Any other questions? Go ahead. Lee, introduce yourself quickly, and then continue.
Okay. And now, here's Lee Lambert from Omnitech.
That was…
Ah. Alright, guys, uh, uh-oh.
Aye, aye.
Is Zoom a problem?
No, my trackball's a problem. Uh… see if we can find a mouse over plugged in over here, I may have to plug in a separate mouse.
Uh-oh.
While we're solving this technical problem that just cropped up here.
Let's do a little background. Last week, we spent… The session walking through the problem I had.
And by the way, Lee Lamard, uh, Omnitech. We do all kinds of interesting tech consulting, specializing in Unix, Linux, and open source.
And I'm also a private pilot. I fly airplanes, I work on airplanes.
In fact, we've just got. Put the new engine in one and doing the engine on the other right now, so a lot of my time is taken up with that.
So, we're good. Last week, we did what was called ButterFS100.
See if I can get a share here now.
Oh, that sucks. It still is a grandkid.
Grant, can you look up the keystroke to share screen?
You're going to wisely.
Uh, looking.
Sharp keyboard shortcut, please.
on my Mac, it is Shift-Command-S, so I would try…
Like…
Thank you.
All tests may work.
All directs nothing.
See, Alt-Sierra.
Oh, wait a minute. Oh, yeah, here.
Mm-hmm. That's interesting. Where's my window?
Stan, you're a little too gleeful there.
Did Zoom hide your window?
I can see your email.
I didn't do anything, I didn't do anything.
Oh, gosh.
You want to know how I really feel about Zoom?
Uh, that's not the right window, sorry. So, Stan, tell us how you feel about, uh…
This is BG13.
Gosh… Hmm… I… I probably should.
It's this one. And a presentation next month will be What to Hate About Zoom.
I'm gonna… I'm gonna shut off my camera to conserve bandwidth and.
God, how the are you explain about it? Yeah.
I probably ought to mute myself, too.
I am not going to make a comment about that.
Sorry, guys.
At least I got him looking.
Nope, it's the wrong one. So…
My weight, Stan sharing his camera now.
Is your video shut off?
No, whenever I hit the share. It's not showing me the windows of share.
Oh, Zoom.
Uh, ChatGPT is what we see.
I'm viewing your screen, Lee. At least on anonymous coward who's seen it.
And… There we go, sorta.
Hey! Intro to Linux File Systems!
Penguin and a Disc.
This is a prequel to what we did last week.
Uh, no, I don't think it's a prequel.
Um…
Part zero?
I think it's a follow-on. Lee, we are not hearing you. We can see you.
Oh.
or your presentation,
Hey! 5x5!
Okay, how about now? Oh, good. Alright, uh, what a nightmare.
Okay.
The presentation last week, we called Butter FS100. And that was a run-through of the serious issues I had with his workstation.
Notably, last spring sometime. Excuse me, when I updated it.
The root partition became unbootable. So I had to roll back.
to the previous version. Which is a trivial task in ButterFS, but I… and I could keep on trucking.
But I could never… I could never update anything in the OS.
I could install new applications. I could update individual applications, but I could not do an update.
And we looked at that problem, we looked at the symptoms.
We walk through it. And I demonstrated what ButterFS looked like, and how I isolate the problem and ended up fixing it.
My idea tonight. is to go back.
To a more generalized view and talk about where all file systems.
architecture, all file system architectures are similar. How they fit together, how they compare to each other.
And then sometime in the near future, we have 2 more.
Butterfs sessions. With… a deeper dive into the basic concepts of ButterFS, and then the 500 is troubleshooting.
And… the route, uh, data that I needed.
to fix the problem from the last session.
The main reason for this is that. We need background, and I'm trying to make sure that everybody is.
is following along, and we don't lose everybody, and nobody gets disappointed because.
I want to edumificate. folks as much as I can.
Any questions as we get started?
All right, why start with file systems?
If you've been working with Linux, or any. IT environment.
from modern systems, Linux. Wind blows, Mac, Apple, iOS, Android, whatever.
You've all got file systems under the covers. If you grab an old one, 50 years old, deck 20, which we had when I was working at Emerson.
That had a file system. Although it was probably… you store it in megabytes as opposed to terabytes.
The point is here. The file system is really the foundation layer for everything that is stored.
on the system. Including your bootloader.
Including all OS services. applications that you have.
installed on the system, either. by the OS, or by you as a user application.
And most importantly, a big job that you have with the file system.
system is maintaining. Data integrity.
That is why we call it the foundation. Of the entire… of the entire environment.
Some of the modern systems in 2025 have improved a lot of the.
a lot of the aspects of the file system, for example.
Most file systems today are much more reliable than they were 50, 40, 30, 20 years ago.
Part of that is in the structure of the file system. Part of that is in the improvements in hardware reliability.
As I showed you last time. If something blows up.
And ButterFS, it is fairly straightforward to roll back.
to the previous snapshot. There are some other.
Very significant file systems out there, no, most notably ZFS.
That also has that sort of capability. But I would argue that it is not as easily manageable as it is with ButterFS.
But it can certainly save snapshots and allow you to restore a snapshot at a given point in time.
Data integrity is improved. Again, both from a hardware and a software side.
And we're going to see how… The snapshot actually.
integrates into the entire system. to make a lot of these advanced techniques.
Possible. So the goal here… is to give you a background you need for butterfish internals, which will be the next one.
And then we'll talk about real-world failures and recovery strategies.
To start off with, what is a file system?
If I'm looking at a command prompt on my system.
And I don't want any gray bears answering this.
What is one command. That will show me… Every storage file system.
That has been configured on this system.
Bye. Yeah, okay.
Try DF.
I'll buy that. DF gives me a lot of… stuff?
I don't know that it would be every file system, though.
As opposed… well, are you thinking about, uh, in-memory?
No, um…
I will say, without giving away another command, it will show file systems that are online and ready for use
There could be others that are not.
And you have to do something…
to access them, and there could also still be more.
that, uh, are not.
And also, one word.
Okay, uh, I'll agree with that. No.
Uh, 20C, which…
might qualify as a gray beard, as a command in chat.
M… if you have a microphone, how do you pronounce your name?
Thank you. Great.
in my day job.
listed another command.
Ah, okay. Uh, I will get those in a minute.
Okay.
Let me continue on with DF here. Now, first thing you'll notice.
A lot of you would probably. Be able to use… The block counts.
But personally. I prefer DF-H.
For human readable. And to me, it's a lot easier to grok.
185 and 16, 85%. As opposed to… The block counts we had up here.
For the same 85%. Excuse me.
Notice that… We talked about this last time.
a file system here. is… has a… A physical file system, I should say.
virtual file systems might not. But a physical file system has a… Device associated with it.
Notice that there are a number of different instances of that physical device. We'll look down here in a few minutes.
For example, here is root. monitored on slash.
Size the partition, use available, use percentage. Here are some virtual file systems, our… Non-physical file systems, which is the third type, I believe Brant was alluding to.
As opposed to unmounted file systems. Remember we talked about snapshots last time?
Here are two sub-volumes. Notice that the sub-volumes share the same physical characteristics as the physical volume.
The two boots, here is your grub boot, here's your EFI partition.
Up root temp, user local var, and booty FI.
Those are the remaining sub-volumes. And in this case, I have my home partition mounted as a separate partition.
As partition 3 here, P3. We didn't talk about that last time.
Somebody want to, uh, toss out an advantage of doing this?
very, very… important advantage.
kind of isolates problems to a certain area.
Okay?
It allows you also to separately backup. Your home data.
Okay. Those are good points. To that, I would add.
Or your system data separately.
If P2 munches itself for some reason. I am not affecting my home data.
No, if the OS were to entirely blow up his partition.
I've still got my home… my user data. As well as whatever the last backup was that I made of that root partition.
In this world, on this machine, I have a separate 2TB drive.
Which is for… uh… workspace.
is probably the best way to put it. I'm not too concerned with it. If it goes away, yeah, I lose some archived data, but there's very little active on there.
And that also makes a nice, fast place. to back up home and whatever I need to en route.
In fact, if I look… if I look at that root directory.
Uh, that Shepard SDC partition. Here's my backup.
I'm having backup root lately, but. My system back up here.
is all the… all the critical stuff on the system.
Including home. Okay. And notice I've got some old… Uh… rotation snapshots rotated.
I'm not sure why they haven't been deleted. Normally, our snapshot will take care of it.
Anything else would be good to mention here you can think of, Grant?
Um, what I was thinking of when you were asking…
is file systems being mounted or not. DF will show what is mounted, so you could have something that is not mounted. DF wouldn't show it.
Um, Etsy FS tab probably would.
If you had things configured.
Um, but I was also thinking you may have a drive that is not defined,
in FSTab that you manually mount,
Or maybe you've got, uh, something on a flash drive you use
rarely you plug it in.
And type the mount command, that wouldn't… probably wouldn't show up in FSTab, or…
various other places you could have file systems.
that it's like, how do you know where those are? And, um…
Tony mentioned LS Block.
tends to help for things, too.
Right.
But you can get into some really weird things.
Yeah, yeah. Ellis Block was mentioned in the chat.
You can see, excuse me, LS block, lists block devices on the system.
And this will show you your RAID configuration. SDA and B, which are the two physical drives on this… on this.
particular machine? Here's your booty FI.
Here's your root drive, partition 2. Is your home partition?
Partition 3. Here's SDB, the other one, it mirrors it exactly.
Here's SDC, which we talked about.
If anybody remembers… I've been using slash U for almost 50 years.
I'm probably 40, closer to 40. Because back in the early 80s, we were selling VSD era.
Yeah, um… Oh, I'm drawing a blank.
SDI or something?
Oh… No, no, uh, version 4, BSD4, was that… The, uh, parallel to AT&T Unix back then?
I don't know.
Anyway, we were selling business systems. I think… I thought it was BSDP… no, it was, uh… Merkley, Berkeley's be… yeah, Berkeley standard distribution, V4.
Yeah, BSDI.
Yeah. And that was what we were selling.
And in that world, everything… everything user. programs with Montereno slash U.
And slash home with a separate partition, so I've been using it ever since.
There also are a bunch of drives configured down here.
that are not mounted. If you… If you added a thumb drive in here, it would show up on SDD.
Is that, like, a media reader?
See, like…
SD card, etc.
Uh. Yes, because it… if I insert a thumb drive.
Yeah.
It'll detect as SDI.
So, between… DF, DF-H.
And LSBLK. You can glean a lot of information about any configure system.
Uh, I tend to reach for what Michael suggested, and that's
FDISC-L.
of course, I kind of hark back to before LS Block was commonly available command.
The thing… the thing I don't… I don't use that by myself a lot, because, like I say, I learned.
LS Block 40 years ago. This gives an amalgamation of presence.
Physical configuration. And… let's see… GPT, possible errors.
We'll talk about partition tables at, I believe in 200.
Might behooves to mention partition tables at this point.
In the modern world, GPT is what you have.
up until… was it… Windows 11, or was it Windows 10 that went GPT? Anybody know?
I thought it was before that.
Not… not too… not too long before. But anyway, Windows has finally made a transition to GPT.
And that's a whole other can of worms, which I will not.
get into.
Yeah, one of the big drivers is bios partitioning had problems with drives larger than 2TB, so as those became mainstream, we were compelled.
to GPT.
Okay, uh, the other one I was looking for… What I use a lot is H.
There was a question in the chat you may want to look at.
T as in tango, capital T. Which… which…
If Michael was asking or saying that, uh, slash U is…
new for him the first time he's seen that.
Um, and I was getting ready to reply, it wins at VimGolf.
So, you know…
typing it frequently, it's two characters.
Okay.
Um, I have seen a few other
gray beards using it, um,
gray as in almost white beards.
So, it's been around for a while.
At least 40 years. Because the old BSD version 4.
From the, uh… Original AT&T days.
Anyway, capital T. Somebody, somebody that doesn't have a gray beard, tell me what it is.
I feel the need to go color my beard real quick.
All right, all right. Type.
tells you what type of file system it is.
We've got my Butterfs, which we talked about, we've got TempFS, which are.
Memory file for systems. We've got VFAT for booty FI, which is the only way.
You can do secure boot. And then we'll talk about it here, but XFS for both home partition and slash U.
is one of the more widely used partitions. That does not need.
snapshotting capability. Yes.
Question. Uh… That slash run stuff, is that always in the temp?
FS file system structure. Is it where they're always pointed to?
contemporarily.
I have a spiel, but it's your show, Lee.
Yeah, Gary.
Say again.
Um, slash run?
That's okay. You were talking.
I consider to be a…
Linux-ism, um, particularly a SystemDism.
Traditionally, it was a run… it was a directory
under VAR, so it was VAR run, and that's where things would go.
And particularly on Linux and SystemD-based distros,
They just moved run…
to be on the route.
Um, it has been a…
frequently been a memory-based file system for many years.
It is usually intended for relatively small, transient files.
things that you don't need to keep.
across boots, and you want to be
fairly fast to access.
Did I hear somebody else chiming in there?
Okay. Um…
Thank you.
a question or comment on your capital T?
Sure. Go ahead.
I believe you can append a file system type to that,
Okay.
And it will constrain the output to that type of file system.
No space.
As in… Dig in?
Take the space out.
Okay.
Yeah.
Probably depends on the version.
Okay, um…
the lowercase t, not the capital. I'm sorry.
So you can't… Okay, so you have to use a lowercase t.
To filter.
Yeah, the capital T, it looks like, prints a column, and the lowercase t allows you to filter the type.
Wahoo.
Yep.
Good point.
Anybody else? Uh… that's true.
Here's Bach. Bach is a little bit simpler.
Because the rate… the rate is done outside. physically of the virtual machine.
We've got a root partition, EXT4. We've got to serve partitions, XFS.
One for media, and one for Gary's, uh… How much is it?
And Gary's 13 gig worth of email he hasn't looked at in 10 years.
No, it's only 8. It's got 13 available.
He doesn't have to check email for another decade.
Alright, so, file systems.
that each of those file systems. Whether they're temporary or ButterFS, or XFS, or EXT4.
or VFAT, or whatever. That file system has to define how the data is stored, how it is organized, and how it's protected.
has to manage the files and directories, the metadata.
Permissions and ownership, block allocation. I won't get into this too deep.
But suffice to say that. Every file.
Every file of a file system. is composed of blocks.
And a major… part of that file systems tasks.
in operation is to keep track of which blocks belong to which file.
Which blocks have been changed? We'll talk about here in a few minutes.
Which blocks are free. As well as a separate flag, so the system always knows which block.
has been used.
And that's critical, because without that file system, you have no structure, no safety, and… No recovery.
Does that, does that, does that involve the super block stuff?
Yes.
in my youth, I used to fiddle with that.
Yes, that does involve the super block stuff, but.
The way the blocks are managed. within a file system is unique to that file system.
Which is probably, uh… we'll touch a little bit of that in Butterfs 500, but it's… well beyond, you know, what we're acting about here.
I'll toss out there, there's some…
Thank you, thank you, thank you.
common patterns. I made a comparison earlier for some other things.
you know, a whole bunch of people can make
chilly, uh, but each chili can be a little bit different while still largely being the same type of dish.
File systems have a lot of patterns.
But each one has its own particular…
flavors and seasonings.
Most of them.
But they still have the same ingredients. Most of it. Most of the time, yeah.
I'm gonna throw in an odd observation at this point.
Uh, is to… without a file system. We were just talking about this the other day.
Back before Unix. There was Baltics, and before that, GCOS.
One of the things that GCOST and a couple of other.
non-IBM mainframes came up with. was this whole idea of file names and then directories for… put files in.
Because back in those days, on an IBM computer.
When you created a file, you burned the tape. You basically told it, okay.
This file you're opening on disk number 3. Uh, at track 151, uh, block number 17.
That's the way you kept track of files on an old IBM mainframe.
And so, yeah, this whole idea of a file system.
Was a radically new, amazing idea back in those days, and it's one of the things that led us to Unix eventually.
Unique. You mean politician with names?
I bet… I bet the, uh… I bet there's 360s running with those old-style file systems.
You mean by associating names with those files?
Correct. This would have been back in the… in the… early 70s, uh, if you were a big-time IBM mainframer.
Yeah, you had to know track number, block number, in order to write something onto a file.
And you had a pre-allocated before you could use it.
Mm-hmm.
Well, I'm just trying to figure when CPM came into the picture, because that was in the late 70s.
Control, Program and Monitor.
Yes.
What that acronym stand for, Lee, CPA?
Oh, okay.
But I believe CPM was more designed for. desktops, small systems.
Yeah.
Yeah, but it had a file system, and… I'm trying to remember CPM had a directory.
I was just wondering the same thing. I know DOS 1.0 did not.
I don't think. the old 360 system, I don't think that had… I mean, there were just blocks and cylinders and.
discs, but… There was no file system, per se, where you could.
You know, I don't think it had a concept of a directory.
Subdirectories did not show up until DOS version 2.
Correct, that's what I was saying.
I know that…
on VOS and lineage.
Yes.
for a long time, there has been what is called the catalog.
of what…
files are where…
But there's not really a directory structure, it is more…
convention and file naming…
So, there's some patterns, and it kind of…
simulates directory structure, but it's all just one big list.
Yeah, when I was at Mac, we used to use the 360s and.
I distinctly.
I'm pretty sure that that file, that… the file system, if you'll… allow that. Really, it was just… You know, that… people thought they had directories and things, but what they really had was.
you know.
file names with, uh… Ah, it's just in a big morphous… thing that your file name, you segmented the file names to give it a little bit of structure, but it wasn't.
Yeah.
It had nothing to do with the underlying file system, if you can call it a file system.
I distinctly remember DOS 1 was a problem because it had a limit of the number of files.
And if you went over that limit, the stuff disappeared.
I didn't know would disappear,
But I know, and even still today, you can only have so many…
Oh, it would, yeah.
things in the root of the file system,
And my experience is if you hit that…
You can't add more, and it…
Lee, doesn't the same apply?
too many file systems. I mean, the number of things before you hit it.
is vastly different, but, like, um…
With extent or XFS, or butter,
If you put too many things in one directory,
You run into some real weird edge cases, I believe.
Yeah, but I think those numbers have been, uh… Dramatically increasing over the last year.
Yes, light.
number 64 bits.
On some of them.
Yeah. Yeah. Okay.
All right, anybody else taking a trip down memory lane?
I'm searching my index, hang on.
Go ahead.
Alright. All right, so file system is responsible for maintaining the directory structure.
all… each… all the elements in the current directory, as well as.
directories down the tree. In the… in the Linux world, you've got a root file system here in slash.
Uh, we got a lot of crap in there.
The root directory on Bach. Everybody's seen been moved, Dev, Etsy, home.
on Debian, you've got Anitrd Links. In the root directory, which you don't have on other versions.
live, Lib, and Live64. Lost and Found, if a block on the disc in certain file systems.
I like EXG4. is found to not belong to a file.
What file name does it have in there?
It gets thrown into lost and found.
Number, yeah.
Yeah, I know name.
I think there's any in there… oh.
I used to have to. Uh, keep a list of all the major.
files on my systems, because when the system crashed, I had to go in and figure out what I know to rename to.
Oof.
I noticed that on Butterfs, there is no loss and found.
Um, I have been on many extent-based systems.
where somebody has removed the directory, and that has never caused a problem.
Um, FSCK will recreate it.
If it is needed.
Okay. Uh, so this is Butterfest, doesn't have a lost and found XFS, which is slash U, doesn't have a lost and found either.
In the XFS world, there's XFS repair. There's a lot of XFS tools that you have.
What is XFS estimate?
Uh, just a couple trivial no… trivial… notes… Uh, let's see, uh…
I'm not sure, but I know XFS Repair is worth its weight in gold.
The only caveat is you have to have the file system cannot be mounted to do a repair.
Uh, and that is one of the big reasons why you never… well, I guess it wouldn't make much difference. You can always boot from a.
thumb drive, if that was your root partition. But that's the biggest reason.
to use XFS for large, extensive file systems that just keep growing.
XFS check instead of FS check. If you want to make an XFS file system bigger.
You, uh, in fact, I went through this year 3 or 4 years ago, and we went from.
2TB drives to 4 terabyte drives, and our core backup server here in the shop.
Uh, basically… I went through the procedure, if anybody's interested in it, I can.
share the presentation with you, but… Basically, you swap.
A 2TB for 4… you… Take the 2TB drive out of the RAID 10 array.
Put the 4TB in. Copy the partition table over from a 2TB drive.
and re-added to the array. And once you've gone through and swapped every.
2TB drive for 4TB drives and rebuilt. the array, you will have a… basically a 2TB file system spread out over our physical 2TB file system spread out over X number of disks.
So basically, you just go in and delete. The last partition, re-add it with the full extent of the disk.
And then when you're done, you hit GrowFS, and all of a sudden, instead of a… 20-20 terabyte.
storage array, you've got a 40 terabyte storage array.
Nice. Yeah. In fact, we did it. I never took the machine down.
Any other questions about XFS? REXT4.
How do you shrink an XFAS file system?
What do you do when you need to shrink an XFS file system?
You don't…
You'd have to recreate the smaller one, and then copy the data across.
I'm not trying to be a…
smart, you know what, but uh…
within the last week, we actually had
something come up at work.
Oh!
Where it's like, well, crap, we can't shrink that, it's XFS.
And…
Uh, the customer's paying for disk space they didn't need.
What'd you do?
Because it's allocated to their VM and root file system.
And the amount of work to undo it.
is prohibitive, and they're like, I'll pay the extra $5 a month, just
go on.
Because to do it would require additional space to create a new file system
copy things across at an outage and booting off of live media and remote consoles. It's like, no, it's not worth it.
Whereas if it was an extent or other things,
We could shrink it, remove the extra disk that was added to the wrong, uh,
Basically, somebody was trying to grow
a logical volume, and they grew the wrong one.
And there was no way to ungrow it.
Ah, okay. Yep. Oh, wow.
Yeah.
Yeah, that is, uh… feature or flaw, or whatever you will, of XFS.
That's not a bug, that's a feature.
Oh, okay. In terms of… All right, anybody else?
Okay, so we've talked about the directories, uh, file naming.
in the current Linux world, it's… naming isn't a problem, but… I ran into an issue a month ago with one of my clients who's an attorney.
He's a character. But he thought it was simple to use OneDrive.
Uh-oh. There are so many flaws in OneDrive, it's like.
Walking through a swamp with alligators every 12 feet.
Waiting to chomp on you. So, you know, he had one machine that was complaining about.
Name about the file path, or file name, including path too long.
Well, that OneDrive said it's not a problem. But that machine did.
So, we… I went down there, I spent 4 or 5 hours with him.
We sucked everything off of OneDrive to iCloud, threw away OneDrive.
And so, so far, he's been cruising. But, you know, it's just really, really, really, really, really weird.
And obviously, that file system has to keep track of the paths for every file there.
permissions, ownership, access control. I just love the simple world.
LS-L. You got… 7 permissions here to deal with.
With a couple weird ones throwing in. That is just so much easier to wrap your mind around than the 45,000 permissions you have in Active Directory.
Yeah, but what do you do when there's a plus after that?
last character.
Um…
Uh, extended ACCL, I believe, right? Or faculty, rather.
post-6 ACLs.
Yeah, there… there's POSIX ACLs, there's extended attributes,
Which don't even show up, and then there's, um…
SELinux context.
Yeah, ACLs are Linux access control lists.
Yeah, and there's also extended attributes on something you can use on some file systems, which I have not gotten involved with.
Yeah.
We've never… We've never had a base presentation on.
But anyway, the… go ahead.
Uh… access control list.
Okay. Extended permissions.
On what? On access control, this ACL. Oh, okay.
it gets complicated.
fast.
Yeah.
Yeah. Mm-hmm.
And if I recall correctly,
That's why I never volunteered for it.
ACLs do not add permission.
they remove permissions. So, like, your standard user group other?
has to allow it,
And then ACLs say, no, no, no,
Um, different grouping have
permissions, not…
they're a subset of
what you have there.
Yeah, I don't even remember how to use it. You can do… you can do a set fackle.
Do you remember any examples, Grant?
No. I think in 20-plus years, I think I've run across it.
once myself and one other person that had run into it.
used all the time in the Windows world.
But… never in Unix.
Effectively, never.
Yeah.
Alright. That file system also has to handle reliability. What happens on a crash?
Well, the file system has to know what was open.
has to know where everything was. has to be able to recover.
An accurate representation of the file system? Before the crash?
And depending on the file system handle, or how it handles files.
If it crashes and loses track of blocks, they'll end up in lost and found.
and EXT4 system.
And again, we'll be talking about copy on right here in a little bit.
You'll see why copy on right is actually. a very good.
technology to have in case of a crash are.
In case of a disruption in. a file writing process.
That file system also has to give you reasonable performance.
We… we talked about Butterfs. a while, or yet, last week.
I remember that ButterFS system is based on beetrieve.
Which is a very fast and very efficient binary.
Uh… tree… That it uses to organize every… all the files and blocks in the system.
You have to allow our official rights and give you some sort of caching capability. And again, we'll talk more about that in a little bit.
Question.
Would you rather have performance or reliability?
Yes, question.
Good question. What do people think?
liability, because it's really bad performance when you gotta go back and recreate the friggin' wheel, because it… Never got out to the disc.
Yep.
Ideally, you can have both, but if I have to pick, I'll take reliability.
Yeah. Unless, unless… You have the luxury of using a scratch pad or temp storage.
Yeah. But those are specialized types of files.
You know, then performance might be important.
Right. Our specialized application. Yeah, specialized application. I mean, if you're talking about the controls for, uh.
a jet fighter or something like that. It's making decisions on, you know, where the rudder goes in the next.
half a second. Reliability, no, if you gotta worry about restoring the data, it's not useful anymore.
Alright, everybody else?
How does I.O. play into here, is that… just a measure of performance, or what?
Well, there's… there's two different levels of performance. I.o. Is a performance.
parameter of the underlying hardware. Uh, in general.
Okay.
You know, there may be hardware caching on there, or the controller might have some memory, but basically.
I know is how fast you can get the data into and out of that storage media.
Whereas the performance we're talking about here is the performance in file data retrieval.
FileData Writing. hashing, almost all just… almost all storage media nowadays have some sort of caching.
system built in, so that if you want one block.
The controller will automatically grab the next. say, 10 blocks and put it in the cache.
So, when you come back for second, third, fourth, and fifth block, you can get it out of memory, as opposed to physically taking it from disk.
I would argue that there is also some aspect of performance to the file system.
in the way it organizes structure and maintains things.
Both on disk and in memory, because searching those, the algorithms,
start to come into play.
And you can get some surprising results, like…
I have seen faster…
I.O. with compressed file systems,
then uncompressed.
Because there is less data,
to cut off the disk.
Right. Right?
There's some non-intuitive things.
You know, you can take that 2 gig. Directory structure, compress it into a couple hundred meg.
suck that off a disc, and the fraction of the time it would take to pull the whole 2G directory structure.
And then, uh, use it in memory. You don't even have to uncompress it, just use utilities that.
will uncompress the pieces you need on the fly.
Yep.
Like, like, uh… ZCAT and ZLS, and ZGREP type of thing.
Everybody follow that?
Good. Anyone else for the question.
I did.
All right, in the Linux world. The basic application.
is the Linux storage deck, Virtual File System. You have a… storage device at the bottom.
I'm gonna walk up the stack. That storage device could be a solid-state drive, non-volatile memory.
Physical hard drive. The storage device is… Where the data is kept.
You know, if, you know, basically, where the data resides if you turn the power off.
A storage device is designed to survive a normal shutdown.
On top of that physical storage. There are multiple ways to organize the blocks.
For example, we talked about LVM. A few months ago, Logical Volume Manager.
There is one nice thing about LVM. Which a lot of people like, which is why the Red Hat world uses it by default when you build a system.
With LVM, you can extend a partition. with other space somewhere else.
close on another physical drive, another media, or whatever.
Without taking the system down. LVM will operate on a running file system.
Um…
DM is Device Mapper.
Uh, I'm not sure what DM and MD are. Grant, do you have any…
And it is a… an abstraction layer. I think it came out of.
MD. MD is multiple device.
And that is ostensibly.
Yeah, um, and…
at some point,
Oh, the RAID system, okay.
the RAID system used…
device mapper to do part of what it does.
Um, and I would argue that list is missing traditional partitions.
Okay. I'll buy that.
Uh, example.
Where's the first place you go to look at the health of your rage system?
Depends on the RAID system.
Uh, TWC…
any rate system, any normal rate system.
three CLI sneeze on your keyboard.
Uh, yeah.
Nah, simpler than that. Md Stat. Uh, back up. I'm speaking soft grade.
Okay.
uh… ZPool stack.
Software managed by Linux.
If you have… If you're running… DFS.
Yeah, uh, ButterFS has something comparable, too, I believe.
Lewis?
LS? Anyway, we'll cover those in detail later.
Yeah.
I got a… I got a good question for you.
Gary, this one's for you.
Uh…
Okay?
I'm having empty chat on this system. And I know for a fact there's 12 4TB drives in it.
How about an LS block?
Ooh!
Uh, do you have a hardware rate?
or a firmware raid?
Yeah, yeah.
LS block, uh… tick up.
I'm sorry, say it again, Stan.
I will.
LS block space dash… F as in Foxtrot.
It's nothing. Oh, that just shows the, uh… UUID, I'm sorry.
Uh, FDISC space dash L space slash…
dev slash SDA.
Oh, it's a perk!
So, you can either get to that with a boot time menus in the firmware,
Or there's gonna be…
commands to talk to the rain card and ask it status.
And those drives are hidden behind
The RAID card, and it is presenting a logical drive.
to the OS, and yeah, MegaCLI is one of the commands that, oh god, the syntax, that is an abomination.
Yep.
No, but I wrote the shell scripts 20 years ago.
Yeah. You and every other admin that has to deal with it has done that.
to pull the status.
when you have 8,000 of the machines, and they're all by a whole bunch of different clients, and you can't copy files between them,
You have to use the esoteric syntax.
Right. Uh, Mega CLI.
I use the command line. I have a shell script that will.
uh, generate this status file every 15 minutes. Or every… I think every hour. And then every… every 15 minutes, I use Nagios to summarize it.
And it'll tell me if I've got… let's see where… Where's my 90 years?
So, when I said TWCLI, that is…
a 3-wear raid cards counterpart to Mega CLM.
Right, right, right. Uh, anyway, you know, here's, uh… FE archive right here.
Where did we lose you?
Yeah, go ahead.
You lost me 2 minutes ago.
After I said what I said.
Okay. Uh… Let's back up.
Yeah.
Uh, when… Grant was talking, you know, when I was quizzing people here, we were throwing different eyes… different ideas out on the table.
Alright.
It's not software RAID. It's not… or the Linux system only sees.
Two physical discs, which we showed up here. SGA1, SGA 2, and actually 3 partitions.
Right?
Physically, all it sees is SDA. So I built this thing 5, 6, 7 years ago. It's a Butterf system.
It has been updated ever since. And here's the 13 terabytes out here on slash U.
Hang on a second…
I would argue that, uh…
PowerEdge RAID controller, aka PERC,
is an actual hardware RAID card.
Yeah, this one's a little bigger out here. Uh, yes, it is a hardware RAID card.
And you can hang… drives in… I think they're mostly 12.
Drive chassis. You can string them out as long as you want.
Yeah.
With controllers and buses. But anyway, this is another… I think it's a perk system, but… Uh… make a steel and I work on.
fdisk-l slash dev slash SDA.
Chicken?
Yep, it's a PERC 710, and that's one of the reasons I tend to reach for FDISC.
Is it will give me some hints about the hardware.
PERC is a hardware…
PERC stands for Power Edge RAID Card, or Controller.
It is Dell's brand for their power edge servers.
hardware RAID card.
And normally, you can use perk cards.
in other brands. It's just…
a very… it's from Dell, and it's almost…
very frequently used in Dell hardware.
Unless you cross vendors?
Like I do.
Thank you.
Yeah, thank you, Grant. I never knew. what Perk meant, or the acronym. Thank you.
So what's a DRAC?
Yeah, really, and it's…
Uh, Dell Remote Access Controller. It's the…
hardware that you can remote control the computer with.
Yeah, but basically any Dell server has a hardware.
Uh, out-of-band controller, I would call it. that you can shell into, or connect into, you get a GUI, and you can manage the hardware on the system.
Yeah, and…
And that has evolved over time, years and years and years ago, they were
Okay. You know, without an OS running. Yeah.
separate add-in cards.
Yeah, and today they're… in fact, today, people call them IDREP.
For internet-based.
Well, also because they are integrated to the motherboard.
Yep. It's on board.
Oh, iDrag is integrated? Okay. Got it.
Anyway, if you ever start dealing with server stuff.
You know, you're talking about, you know, 2U, 4U.
Huge things, you know, in fact, I gave Grant to 1U.
Which actually had 6 or 8 2-inch. Drives in. Instead of 3.5 inch drives.
That was for services rendered.
Yes.
Yes, it was, and for hardware traded.
And this guy has the same Mega CLI.
Uh… And let's see, how do I get…
Um, if you have a path to the actual Mega CLI command, do a double-dash help on it, just to show
how bad it actually is.
People are laughing.
Okay.
Yeah.
Uh… Hang on a second, let me find the darn, uh… Mega CLI one that runs every hour.
Where the heck did I put it?
like, I live at the Unix command line.
Hi.
And I deal with a lot of really terse commands.
Mega CLI, PERC CLI, and other things in that family.
absolutely break my brain.
You got to bite me? I heard this.
Uh…
run that through number line.
It is a terse mother.
Well…
Yeah.
Is it sending it to standard error, not standard out?
No. Oh.
Yeah, and they are terse.
296.
Your built-in help.
296 lines in the health frame.
Um,
That's interesting. Warp Drive Info?
I want to say that's related to boss cards. Um…
Shields up.
It was… so, Dell has something they call a Boss Card. I don't know what it stands for.
But it was intended not as an SSD, but it's in the family…
It was intended for, like,
caching layer,
for various things…
Um, kind of like a hybrid between a RAM drive and an SSD.
And their commands to manage have kind of…
amalgamated together.
This is the command. To generate that template.
Megasi alive, and there's a full path for it.
There's the log file. Minus PD list, minus all.
And it was not worth my time to figure out how to extract the information I need.
So I just dropped it all in the log file.
And then I went to, uh…
I've worked with LSI hardware RAID cards, Mega Raid, which perk…
tend to be based on now. I've used Adaptec RAID cards.
compact smart RAID cards, some IBM RAID cards…
Three wear raid cards.
MegaCLI…
is, to me, two orders of magnitude
more terse.
as friendly as ED,
And as verbose as ED.
it just…
I… I can get it groan out of many Unix admins by mentioning MegaCLI.
Okay, this is the actual bash script. Huh?
I coughed.
Oh, okay. I thought you were having a reaction.
Fair.
Well, that's relatively clean and short, particularly with what you're working with.
That's not bash, that's Pearl.
I'm sorry, it is. Uh, yeah, Don actually did this about 15 years ago.
hey, written once that's running how many places without a problem.
Yeah. This is probably close to two dozen, although a lot of them have.
Mm-hmm.
dropped off since then.
All right, so if anybody ever runs into a perk or mega LSI.
I'll be happy to help. Because I've got all the tools.
All right, back to the presentation. So, we've talked about the devices, the blocks, the file systems.
And Linux looks at all of that. as a virtual file system.
That… the VFS layer…
That goes back to BSD, or…
AT&T Unix proper.
Mm-hmm.
You're… you're saying people won't use it in the outside Linux world?
I don't think it's as much of a first-class citizen.
as it is in Unix.
Yeah, okay.
virtual file system.
Anyway… VFS is… Right. In other words, the fact that it's treated as a file system.
Even though it is different kinds of devices. Different kinds of media.
It's basically abstracted into a virtual file system. metaphor so that the kernel.
has a standard interface to it. Makes sense.
Multiple kinds of virtual cloud systems? Question?
Are there multiple kinds of virtual file systems that you use?
Well,
Do you understand?
No, yes, no. Um…
VFS-layered virtual file system layer acts as an abstract
to present a common interface
to various things, applications above it,
While allowing
disparate file systems underneath of it.
Before VFS was a thing,
your file system
had to…
know about your disk controller.
had to know how the drive was physically laid out.
Once upon a time,
There were no partition tables.
It was compilation parameters or values in your kernel. Like, if you wanted to change partitions,
You were compiling a new kernel.
Um, and the VFS
came along around that same time,
or shortly thereafter, and provided an abstraction layer
to provide a lot of flexibility.
Yeah, the analogy would be today. At the low level, you've got a device.
And you've also got a file, and a lot of the OS components.
don't care what they're talking to, because the. At the abstraction layer allows them to simplify the interfaces.
Yep.
So I like your stack here, Lee.
Mm-hmm.
But what if my storage device…
is an image file that is on a file system.
that is on top of a VFS, on top of a file system, on top of a block, on top of a storage device.
You follow what I'm saying?
Yeah, but I'm not sure that… So…
you ever take an ISO file and mount it as a loopback?
it basically doubles your stack.
Yeah, no, no, I'm just… trying to… well… Yeah, right.
So, if you're mounting an ISO. Buying a boot image.
are… if you're mounting a… G-zipped, or TGZ.
you're basically exploding that object into the components within the object.
So, we could have… 1, 2, 3, 4, multiple.
Mm-hmm.
virtual file system layers. Between the user and whatever data you're looking for.
Yep.
Okay. Yeah, that makes sense? Yeah.
Yeah.
A little brain warp there, but yeah. Do we need to unwarp it?
Then you've got bastards like me that will put an image file inside of an image file, inside of an image file.
And I do things like that to test, to see if something is possible, because when you stress test,
It helps demonstrate weaknesses or…
how robust something is.
Okay, so here's a file system. Which is one layer down.
one virtual file system down to a storage device. Okay.
On that storage device is profile.tgz. Okay… Uh, okay, we're open here.
Which is basically a second virtual file system. in function, although it's… It's not as complicated as it would be if it was another physical device.
So, you have to open that virtual. Object, profile.tcz, and within that.
is a directory, is files. Could be another TGZ tar ball in here. Mm-hmm.
Okay, that's the point Grant was making. Okay.
Did it work? Yeah.
Okay, so each layer… File system, the blocks, and the storage device.
Has one job. And by… Stacking these where each of the layers performs that function.
storage block and file system. That makes… This virtual interface, if you will.
Flexible. It hasn't… it doesn't care what hardware's under it.
Because we've just shown that it works the same way.
All of the hardware… all of the hardware layers.
I don't know my size room. Okay. All the hardware… all the hardware is handled.
At the storage level. The… each block of the file is handled by the block layer, which.
manages them. And then the files themselves. are managed and organized into blocks.
That you can utilize. From the outside world.
Asterisk.
Asterisk?
ButterFS tends to do some things that the others don't.
True, and we'll… we're gonna see those here in a few minutes.
Anybody, uh… I have questions on the layers at this point.
Alright, let's talk about them in detail.
virtual file system. provides that unified interface.
For all file systems. Whatever time.
And it takes… the system calls.
Open file, read file, uh… re-block.
Rename file, whatever. into the actual operations.
on the files. Or the blocks, depending on what you're using, or depending on what the operation is.
And to make sure that everybody. to live at the same time.
Remember when we looked at our DF here?
You know, every one of these. is a file system.
Where there's physical. Virtual, temporary.
EFI file system. More physical, more temporary.
Yeah, the DF command and LS and etc.
they don't have to know how to talk to those different types of file systems.
They talk to the VFS.
Right. They talk to the ver- they talk to the VFS layer.
Uh, here's, uh… Abnormality.
We've got overlays and TempFS, these are virtual file systems.
That are part of… Docker containers.
Did you have a question, Gary?
Nope. Not yet. Nope.
I actually have two… Um, containers running right now.
One of them is… Microsoft SQL Server.
I did present… I did a presentation on this 6 or 7 years ago when I first.
start using it. That work's been… it's been run ever since.
I think this is 2000… I started with 17, I think this is 19.
If I remember correctly. So, that is a container running SQL Server.
And this is a separate container. Running the registry.
In the Docker world, the registry is the… storage for each.
container image. in history.
So, if I want to pull a… if I want to grab a container, I can grab.
The latest? Or… If there's 50 of them in the registry, I can grab any one of the previous 50.
her previous 49. This was created 6 years ago.
And this machine has been running for 4 hours since I got back to San Francis.
And then this is the actual name of the container.
OTS SQL 1, registry. Now, the reason I'm going down this path is I want everybody to understand that Docker.
is another file system. On top of your base file system.
Which is actually… Another good example of that interface, because.
Look at all the images of, you know, I've got on the system.
I play with Greenbone Vulnerability Scanner a lot. Not recently, though.
Some of these I can pro- I can get rid of.
But here's the SQL Server, and it is 2019.
Uh, here's the old 2016. In 2017, so 16 was when I re… originally started playing with it.
Which was right at 10 years ago. And notice how much space this little pickyune thing takes. 1.5 gig.
You realize you install it, it takes about 8GB on your system?
The reason being, in fact, all these 1.35 gigs are SQL Server versions.
1.5 is the latest one, it's a little bigger.
This is the express version you get for free.
And it is just the server. Not the rest of the crowd. Okay.
But it does… it does everything you need.
Not the rest of what, did you say?
Not the rest of the croft that you get when you install SQL Server on a workstation.
Okay.
Because you get everything from AD integration to… The, uh, SQL Server Management Studio tools to.
There's GUI extensions built into it, so you can actually drag and drop Cruft to the GUI tools, and it'll actually change the database for you.
Yeah, that… another way to describe that is it's the actual database engine.
It's none of the management stuff
or wizards, or…
Right, this is the actual admin.
what have you, around the database engine.
Good point. I like that.
And same thing with Greenbone. This is the vulnerability scanner, we've… talked about it more than once over less.
For 10, 15 years. I actually run it on client networks.
Every week, just to make sure there aren't devices out there that.
I want to know about. Or, I want to make sure that nobody has a.
halking vulnerability that is. Gonna bite me in the butt some day if some hacker finds it.
Okay, questions about the Docker? quote-unquote file system.
Because it is another virtual entity on top of.
That physical file system. And I will not get into… overlays and things like that.
Because that is way beyond what we're talking about here.
I will argue that Docker is…
Wow.
Amongst other things, a management
leveraging things that are built into the Linux kernel.
Uh, true? Uh… I'm not sure how much has to be built in, though.
the overlays and much of that is…
That's, uh, what is that, that G space, uh, C… Uh, C Group, right?
Uh, CGroup is…
more process control, um…
mount namespaces and various other things.
So, a whole other discussion.
Well, never mind. Yeah, I just… It's just a tool. It's a tool, and… It solves a lot of problems.
what is probably very powerful.
Yep. If I could only… if I could explain Kubernetes that fast, I'd be, uh…
All right, so we talked about VFS. what it is, how it works… It enables the kernel.
to talk to any file system. Using a standard virtual interface, if you will, in quotes.
And then neither the kernel or applications care what file system is actually used.
And, you know, to physically perform it. So the key takeaway is… VFS equals compatibility plus flexibility.
Alright. Most people have used the EXT4, it's been around for, what, 25 years?
I remember AEXT3, and then came the one where the guy got thrown in jail, and the project was abandoned.
Yeah, I feel like…
Extend foreign journaling was…
What was that?
late 2000s, Extend 3 was early 2000s, Extend 2 was…
90s and 80s.
What was the file system after EXT3 that, uh…
Riser.
You have a Reezer, or riser?
Oh, riser, that was it, R-E-I-S-E-R thing, yeah. Thanks, Sean.
I am a riser advocate. The thing that took me away from Riser was ZFS.
Um, the same workload
I could get consistently better performance out of Riser than Extend 3 or Extend 4.
on the same drives.
And XFS was not…
really a thing in Linux at the time.
And I have actually had riser outperform XFS as well, just not.
by as wide a margin.
Okay, if she 4. It's been around… About 15 years or something like that, it's very stable.
It… Journals, metadata?
Which means if the system crashes, it can easily roll back to the.
previous state. I have not played with that a lot, but I know it's possible.
Grant, would you want to comment on that?
I have never needed to do it. FSCK has…
always been able to bring an extent4 file system.
back into service.
faster, um…
Extend 3 was a little weird. You could add a journal to Extent 3, and it was slightly different than Extend 4.
But the journaling allowed file system checks.
to be much faster.
And if you have not had the…
shall I say, luxury.
of waiting on a file system check.
journaling with Extent 3 and Extent 4 could allow something to happen in a single-digit number of minutes.
that I have seen take double-digit hours.
Hmm.
on Extent 2.
Yeah, because it lacked that journaling capability, couldn't have a real.
Yeah.
handle on the previous state.
Journaling was a tremendous boom.
for file systems.
Oh.
Was somebody trying to say something?
Okay. Go ahead, Lee.
Alright. Est4 is fairly minimal in terms of. capabilities, it doesn't have any snapshots.
But again, the biggest advantage of EXT4 was the fact that it did implement.
journaling systems. our German capabilities.
Journaling was standard with extent for it existed in Extend 3.
Yeah.
Right, but you had to add it. XFS.
High throughput, excellent scaling, large files. XFS is… my… Go to File System for any kind of storage media.
It's very efficient, good performance. The only downside is no copy on write, which we're going to talk about here more.
But… the argument there is XFS is for storage.
Copy on right. is where you need to be extremely careful about.
Maintaining the previous state of the file. So that if the system crashes, you can recover it very quickly.
My argument is XFS is only designed for storage, it shouldn't be crashing unless.
Something major goes on with… outside that environment. So the overhead of copy on right, which significantly, significantly affect.
The throughput of that system. at the data volumes it's designed for.
Yeah, I also see copy on right.
Does that seem fair to you, Grant?
for some other things…
like, if you're dealing with…
image files for virtual machine hard drives or something.
You don't… if you have multiple
And it's weird, but if you have multiple Drive images pointing to the same backing file,
the… it doesn't copy the entire file.
for the subsequent instance.
Until…
a change is made, and depending on configurations,
It will only copy…
the part that has changed. It actually writes the change to a new location,
for the new file, but there's some…
linkage between them.
And you get some deduplication. That's also some space copy on right plays in.
And don't… don't worry about copyright right now, because we've got.
A deeper dive coming up here in a few minutes.
Yeah.
And then Butterfs.
So, the nice things about ButterFS, it has copy on write, it also manages snapshots, which we've seen.
It uses checksums throughout the file system. And it is self-healing from the standpoint that.
You can always… We revert to a previous.
version of the current previous snapshot of the current file system.
I would argue, or add to that list of…
ButterFS also has some
read capability and some other things.
But it's not used as often.
Right. Yeah, to be… to my mind.
I build… Usually RAID 10.
That is not the best performing RAID, but is the only RAID I've seen.
Where I have physically lost 3 drives. And recovered it. Hmm.
I have questions for over a drink some other point in time.
Uh… Brandy just put up a, uh, something in chat.
Uh… nice, uh, comparison of, uh. different file systems and… Kyle was showing which ones have which, uh… drinks. Oh, okay.
Cool.
I believe that's out of, uh, G-parted.
Okay. Yeah, we… we're not talking about tools tonight, but… Stan mentioned G-parted, or parted, G-parted the graphical version, parted the command line version.
That is the partition doctor tool that. If you ever have to touch a drive, you want to be familiar with.
Thank you, Randy. Yeah.
Okay, a little more detail on each of the file systems, EXT4… We talked about reliability, compatibility, simple behavior.
Uh… remember we talked… we already mentioned a couple things.
in a… ButterFS file system, you don't have a lost and found.
in a EXG4 file system, you have lost and found.
Because you don't always know the previous state of every block on the system.
Whereas in Butterfs, you do. So if a block gets lost, the disc is damaged, logically, or the logical structure of the disc is damaged.
And ButterFS can… take you back to a… known state for every file on the disk.
And then if blocks are untagged as free, and they are not used, it will automatically tag them.
What is that known state mean?
After you… known state is my way of describing the fact that.
the file system counterpart of you knowing who is on the call tonight, Stan?
Uh…
Fair enough.
You're assuming I'm still awake.
Uh, the analogy I'm trying to draw, or trying to show, is.
The fact that ButterFS and ZFS. are… and XFS to some ex… to some degree.
But ButterFS is the only file system out there.
That can automatically. Revert back to a previous version of a partition subvolume.
are, in some cases, a file, but that's normally included with a sub-volume.
GFS allows you to. physically restore.
that previous state. And XFS… Repair will fix a lot of problems in the file system.
by deciding if a block belongs to a file or not.
So, the bottom line there is. Those 3 file systems both have a way.
Some ways to different degrees. of recovering data if the logical structure of the disk is damaged.
Or, by extension, if a physical damage occurs. You don't want to be there, but there is a small chance that you could revert the file.
to a previous version and at least retrieve that intact version.
And again, there's a lot of variables there, which I don't want to get into.
But… butterfest right now is the best file system for.
Maintaining a history. of a file, our data on the disk.
ZFS has the same capability, but it's not as easily managed.
And XFS is very reliable. in terms of the way it manages data internally.
And it has some degree of capability in terms of maintaining data history.
Does that help, Stan? You shall wait?
Hmm.
My brain melted about 3 minutes ago.
Uh, that is a fair assessment when you're getting below the beaten path for most file system work.
Like, when you need to deal with a recovery utility,
you're having a bad day.
Agreed.
And hopefully they come few and far between.
Okay, so EHC4 is liable, it's widely compatible, it's simple.
Remember the best thing about EXC4, in terms of added capability was the journal.
I would like an extent
Which gave you that data history capability.
4 or 3 with a journal as a station wagon with snow tires and a whole bunch of
Sand in the back end, it's gonna get through. It'll be slow, it'll be ugly, but it will be reliable.
It's a Volvo.
I like it. I like it.
Just remember, bring oats for the horses.
horses, I'm eating the oats.
Yeah.
All right, uh, the limitations for EXT4 snapshots. They had not been implemented when EXT4 was matured.
No copy, alright. Same… same deal.
DHT4 is not as robust as Butterfus or ZFS in terms of data integrity.
EXT4, best for boot partition, simple servers. Legacy compatibility.
I would, uh… I don't… I haven't done any XT4 partition in 5 or 6 years myself.
primarily because SUSI defaults to Butterfest. Uh, although… Uh, and Grant, would you know, is the XT4 what, uh, Red Hat uses for LVM compatibility?
Um, oh, it's very compatible with LVM.
But Red Hat has defaulted to XFS.
Okay.
I think that might have even been, like, version 7.
Is there a licensing issue? Is there a licensing issue on these?
Yeah. Yeah, I don't do much red hand stuff, so…
Um, butterXFS,
They are native in kernel.
Um, ZFS…
There are some lawyers on both sides of the aisle that are making a lot of money.
Um, that being said,
Probably back in, like, 2016, 2017, Canonical…
said, we think we can ship
ZFS. Support.
directly. And they did. And they still are.
And Oracle's lawyers have not sued Canonical out of existence yet.
And boy, does Oracle have the lawyers to do it. If they can,
Which tells me canonical was right.
you can use it.
I digress.
Yeah. But, you know, ZFS historically has had copyright.
waiting conflicts, not copyright.
Conflicts? I know.
license conflicts.
Say again? license, okay. Well, I… Sorry, I was conflating license and copyright.
Pharaoh.
Uh, I do remember 20 years ago, when somebody in the OpenBSD world asked about ZFS.
And Leo said, I wouldn't touch that. license, if I… if you paid me.
Uh-huh.
Thank you.
Alright, so, next up. XFS.
We've talked about most of its. Benefits?
In summary, it's excellent on the I.O. Side. It can handle more than… You know, it can handle multiple.
Interfaces, huge files. Enterprise workloads.
As Grant and I have both mentioned, XFS is pretty much a workhorse for us.
Because anytime we have a large storage volume, XFS is it.
And remember, ButterFS is typically the root partition. Because it is… provides some serious advantage in terms of rollback to a previous version if something breaks.
or isolating data in sub-volumes. So that if you roll back, you don't have to roll back everything.
are… track everything in snapshots.
The limitations, though, XFS, no snapshots, no rollback. You have to… Uh, have external tools to manage it.
a battle.
I would say that also applies to Extent 4.
Whereas, like, butter… chicken?
Okay, good point.
XFS databases, media storage, data warehouses. Again, large volumes.
Where the data is… might not be static. But at least the risk is low of it being corrupted.
And XFS that we have in Linux,
by the OS.
is descendant from XFS from SGI that was used to store graphic files.
back in the, uh, Toy Story days.
when having huge
terabyte at the time, file systems was, oh my god, how do you do that?
SGI made XFS to do it.
Uh, Vincent has a question in the chat.
Um, are there some situations where it's preferable to use Extent 3 instead of Extent 4?
I feel certain the answer is yes.
Um, and I think part of that is Extent 3?
can be faster than extent 4.
And you can actually get journaling added on to Extend 3.
Um, but it's kind of weird corner cases,
And you might have better luck
with ex… unjournaled Extent 3
compatibility-wise with older things that don't know
about extent 4.
Um, and Vincent did that answer your question?
And while you're responding, I will say I would consider XFS to be the sports car hot rod.
file system.
And again, if you notice, if you recall. Oh, on this machine… But RFS for root.
XFS for you, XFS for home. So, but… ButterFS is a system drive.
XFS, or the other two partitions?
And it didn't look like Vincent has said anything else yet.
All right, the modern FS, Butter FS. By the way, ButterFS is… the way I pronounce it, some people have different variations.
I picked that up from SUSI about 10 years ago.
And I don't remember who was from SUSI or in a group.
Anyway, that's why I call it. Beatrieve FS.
Better.
Butter… butter. Better… better, better. Butter.
Yep.
Core design. Copy on right, we're going to talk about it here, and sub-volumes are the two main advantages that it provides.
In addition to… beetrieve, as we mentioned before.
is an old… Technology.
For lack of a better word. But it's also one of the most efficient and fastest.
Butterfest also implements checksumps. So, throughout… And if it… detects a problem.
I have not looked into it very much. I've never had a problem with it, but.
It does have some self-healing capabilities. Although I do not know much about him.
Anybody else play with butter? Have any comments there?
What does it mean by self-healing?
The way I understand it, if ButterFS. detects a problem with a bad block checksum or something, or… a file on the system.
Since it's doing copy on write, it can throw away the current block and revert to the previous block automatically.
I note… That, uh, with… I always use extent for.
That, uh… the time to worry is when it goes beyond 85% of.
Filling up a partition, is there any similar. Kind of saying that you've observed for butter FTS?
I have not for butter. But, uh, I have… seen that… 10-15% as a possible blank space on NVM.
Non-volatile memory? Uh, because there are some… Optimization routines and repair routines that NVM runs automatically.
And it will grab… It will need space to grab if it detects a problem.
Because I note that. That seems that the extent for.
does a lot of cell cleanup, and once it gets beyond that 85%, then it starts… could start having problems.
Uh, by cleanup, you mean it's using lost and found?
Yes, sir. Uh… I don't know what it's doing, it just seems like, uh… It… things are… life is a lot happier if you keep everything but 85% or… Lower.
I… I remember from years ago, Extend 3 Days, and I think Extent 4 had some similar problems, just not as severe.
Once you've filled the file system beyond a certain point,
It's internal operation
And striving to keep things defragmented.
really started to suffer.
between 80% and 90% utilization.
And by self-healing, you could kind of call that…
very, very, very loose analogy is, like,
memory parity. It can detect some…
simple, correctable errors, like if it's reading a block-off disk,
And it doesn't match checksum, but it's a small enough problem, it can fix that.
both into what is read…
and provided to the program asking for the disc read,
But it can also correct the copy that is on disk, so subsequent reads.
L. Correct.
Um, the thing I'm not sure of…
And do you know, Lee,
Does butter check some…
metadata and…
actual data, or is it just metadata?
Okay.
I believe it is actual data. But, uh, I don't have any proof of it right now.
Right.
Um…
Question.
reading the chat, Vincent says, one reason to use Extent 3 would be to reduce writing on an SSD.
journaling, uh, will greatly increase writing.
that is very valid.
Journals are inherently chatty.
Good point.
I wonder what… if EXG3 would compare to XFS.
How are you prepared XFS as far as chattiness.
I don't know.
Interesting. Well, I will, uh… comment that.
this system here… storm, which we've talked about before.
It's 85%, and it runs ButterFS on SSDs. Not NVM, but SSDs.
And I… Knock on wood, I have not had any physical problems with it.
All right, in terms of operational power. Butterfus gives you snapshots.
We mentioned this last time, we were talking about send and receive.
You can actually send and receive data between Butterfs file systems.
The commands are there. I've never played with it.
Uh, there also is a built-in RAID capability between two ButlerFS file systems.
And again, I have not played with it, but that capability is there.
And that's one of the reasons the series is focused on.
But RFS. Because you have, you now have capabilities.
That you didn't have before. And also, the B-Tree organization structure within the Butterfs file system.
gives you some performance advantage over things like EXT4, the way I understand it.
Yeah.
And I don't know how it compares to ZFS. Grant, have you heard anything like that?
Um…
It may actually be a little faster, especially
ZFS on Linux when I started messing with it.
15 years ago.
Um, maybe…
10 years ago, was kind of slow.
for some operations.
But I chose it because it had…
some features that I found very advantageous.
And it is…
multi-platform.
It is on Solaris, obviously, it is on FreeBSD.
Maybe NetBSD, but not open.
Um, I believe certain versions of Mac
support ZFS.
And they're, I believe, are some other platforms, so…
In some ways,
butter was a Linux
re-implementation
of many ZFS.
concepts.
But the way ZFS hooks into the kernel, and it's a completely separate subsystem,
There's a performance penalty.
So, butter being Linux native, and actually
Hmm.
in stock Linux kernel source,
has some advantages in the Linux
ecosystem. It is Linux only.
though.
At this point, although it is, um, it is open source, right?
ZFS, or butter?
So… For better… and I know ZFS is not a pure open source license.
Um…
The source code is available, you are allowed to use it. It's…
Uh, I think CDDL.
So, like, how you redistribute it.
is called into question.
Yeah, it's built into the Linux kernel.
Okay, but our fence is open, right?
It is inherently open.
Right.
Okay. All right, questions?
How did… how do these, uh… file systems relate.
When you're dealing with a server that needs to deal with 80,000 interactions in a second.
Um…
on Linux…
It…
is going to be dependent on your disk subsystem.
and your interface for it, you're gonna cry if you use SATA.
Um, SAS, or…
in VME will be a whole lot better.
It also depends
what you're doing.
with those I.O. operations.
Um, if their reads, and for this very similar data,
You want to give your system extra memory, and the Linux disk cache is your best friend.
And beyond that, Stan, if you did have. in the real world, 80,000 transactions per second.
You would not be using a single server. You would have that workload distributed over 10, 15, 20.
clusters, or pods, or nodes, whatever you want to call them.
Um…
on multiple… multiple servers.
No, um, extent for…
Could you do that with EXT4 and XFS, and…
you would play hell trying to scale to that.
Um, XFS…
Depending what they are, maybe?
Lee, I will toss out there, you can get…
a kernel to do that.
on a single system,
If the disk subsystem.
can keep up. The kernel is not the problem. The application is not the problem.
It's…
The I.O. Yeah.
Yep. And when you have multiple parallel servers, you have inherently parallel I.O.
But if you have…
one or two…
systems active standby against an enterprise all-flash-based.
Sand? Yeah.
go for it. The sand's gonna be bored.
you could actually have multiple systems doing that.
Yeah, and as I was saying, and depending on the workload.
sandwich could carry the workload, but do you want the sand to carry the workload?
Yeah, it depends what the workload is.
Right, you know, if you're dealing with. uh, Microsoft.
Office 365. I guarantee you they don't have any sort of single.
workload configured in that system. They have. the workload spread over 500 different nodes all over the country and all over the world.
Yeah.
Uh, and then… something that I'm very interested in myself, which I haven't gotten into yet, is Kubernetes.
Which is how you configure a workload like that, so it automatically scales.
You know, if it… if it… Once it reaches 1,000 workloads.
It launches another 10 nodes. for the next thousand workloads.
or the next 20 nodes, and those nodes are automatically managed.
Depending on the workload. If the workload shrinks, the nodes shrink.
Um, where I see this type of… not necessarily Kubernetes, but trying to get that
I.O. throughput on a single system,
is large, um…
AX, Solaris, or Linux servers that are running enterprise database-type things, like Oracle Rack.
Yep.
or J.D. Edwards, or SAP, something where you're paying 6 or 7 digits in software support
annually. So, throwing whatever
hardware that's just a cost of doing business.
And if your business is at large, you definitely have to have it.
Yeah.
and a whole team of people to make it work, too.
I'll be back. Alright.
Uh… What have my title?
10-minute warning.
Title slide's gone. That's it for me, guys.
Um, I'll toss out there that…
ButterFS, ZFS…
And I feel like there's some other players in this space.
tend to have layering violations, in that they…
do the raid… they do the volume management, they do file system, they do snapshots.
things that traditionally, those were disparate layers.
And it's kind of been realized that when you take those discrete layers,
and reintegrate them homogenously,
It is absolutely amazing.
what can be done with them. You end up with things like CFS or ButterFS.
Um, I feel like…
This may cause some people to shudder.
Veritas File System.
from the 90s and early 2000s.
still around, um…
It similarly crossed multiple layers.
And I want to say that Microsoft had…
something they were trying to…
bring out that was…
a complex thing, like…
their dynamic disks taken to the next level to do…
super fancy things. I have no idea if they ever came out with it, though.
And Ron BC says, yup, I'm not sure what he's saying that to, but…
Yep, uh, talking about the Unix philosophy of one thing and one thing well.
over from Ron.
Um, but not universal law.
So, the layering…
really helps clarify
different things and how they interact with each other.
Um, but when you have…
somebody or a team of somebodies
that very well and truly understand.
all those layers and how they interact.
If you turn a blind eye and say, I want you to create something that will do these things,
They can work magic with data, and you get
ZFS and BetterFS and the legs.
Did we cover the thing that, uh, Ron B.C. Put in the chat, where he talks about.
Asking, uh, if butterf is… FSH, uh, was recommended, uh… It's using some forms of braid.
To avoid data corruption, loss, etc. Did you?
I believe… I thought I replied in chat, um, to this day.
ButterFS core documentation advises against
using the RAID 5, and I think
RAID 6 capability.
Uh, that butter has. It is…
known to have problems, just…
Don't go there.
Unless you're a developer purposely working on them.
I… I presume you covered that while my brain was melting down.
I don't know that…
Lee went into that. He talked about, or mentioned that butter has RAID support.
Yeah, I've never used it or explored it, because.
I typically do, you know, in my… projects, I usually build RAID as… at the hardware level.
And we're not dealing with… rate of the software at the file system level.
I'm thinking… my… I'm thinking, Grant, correct me if I'm wrong.
There are a number of synchronization utilities that would enable you to sync.
One file system to another, like. DRBD.
Uh, but I'm not, and I can wrap my mind around that.
But I'm not sure how you would implement a file.
File system raid, any thoughts on that?
Well, um… butter and ZFS do it.
And it's the way they function
they actually internally know
There are multiple block devices underneath, and they kind of aggregate
Those together, analogous to how LVM does,
And there's… when we write to this file that is this
pseudo-file system,
it is backed by blocks on this device and blocks on this other device.
So it's applying the same RAID concepts
It is just integral to the file system
not, as you said, like, for extent XFS or Extent 4, an external
component. I think you put, uh, butter on top of MDADM
or multiple device software RAID.
butter purists would say the RAID 1 that you're doing,
Butter FFs could do
without needing MD ADM.
Seems to me that would add another layer of complexity when you have two.
sub-volume systems, multiple sub-volume systems to RAID, as opposed to the physical device underneath.
Um, if you're dealing with something like software RAID,
you can either… you're doing that multiple devices in kernel.
It is just…
Is it outside of the file system, but still in kernel?
Or is it inside a file system, inside of kernel?
You still have the duplication effort and RAID management, but is it…
external to the file system, or internal.
And… I can tell you for sure from dealing with ZFS,
having it internal,
allows more features.
like, I can…
do some interesting things…
to…
grow, and I think newer versions allow shrinking a pool
ZFS pool, as long as there is sufficient storage for the allocated
File. Space.
Whereas if they are two separate things, like the RAID is outside of,
you end up with that
barrier between the two, they have less flexibility.
Um, and it's by breaking down those barriers that we've been able to get some of the flexibility
We have.
Um…
So, ISCSI is really… ROMBC's asking if iSCSI or Ceph or Gluster…
for distributed. To me, iSCSI…
is a San…
communications method. It's a way to access block devices across
an IP network.
Seth, my understanding, I've never worked with it,
is it is more…
a, uh…
I almost want to say NAS.
like where you're accessing some things
more NAS protocol than block,
And it does replication…
at a higher level.
Um… I believe Gluster?
is more closely related to a clustered
file system, like you might use it on top of iSCSI or traditional fiber channel SAN,
or shared physical parallel SCSI.
Wherein you have multiple
machines mount the same file system at the
same time. It is a very special capability to be able to do that.
Um…
You could put ZFS on top of…
iSCSI, I would not put ZFS on top of Ceph or Gluster,
Because I really don't have a reason, normally, to put a file system on top of a file system, unless I'm storing, like, an ISO disk image,
And then I'm rarely mounting that image.
Um, and no, I don't think you're off-topic, Ron. You're talking about things squarely within the storage concept.
And how you access it, and what layer it operates on.
It is a complicated space that comes from multiple operating systems.
And multiple decades of hardware, software capability evolution.
Linux just tends to be a melting pot.
for a lot of them. And it's, which one is the most proper for this application?
And do you prefer…
COLA or uncola to drink?
Sometimes it comes down to that.
Did that speak to everything you were…
asking about Ron?
Cool.
Yep, there it is, great, thanks. Ron says, great, thanks.
Yep. Take it, run with it!
Cool.
Are we far enough along to end the presentation?
Do a bookend? Question?
I think so. Gary, you ready for bookend? Not yet.
Uh, for those of you who are still with us here, uh… if you are someone who either… this is your first time coming to a, uh… uh, SLUG meeting, or lug meeting, St. Louis LUG.
Uh, or if you're just somebody who hasn't been here for a couple few months, and you've come back.
Uh, be real curious to know. What brought you here tonight? Uh, did you see this topic and went, oh my gosh, I want to hear this topic, it's so interesting.
Or you were just bored because everybody, all your other friends went to a Hanukkah party tonight, or… Uh, so if you could, you can either open your microphone and just give us a clue what brought you, what your interest was tonight.
Or, uh, if you want to type it into chat, that's fine too. If you're really shy and you just want to send it to, uh.
Uh, send it to editor at sloop.org, uh, as an email. You can do that too, but uh… Uh, hopefully they just say it, or type it in the chat, be interested to know what brought you here this evening.
Looking at the current participants, everybody has been here at least once or twice before.
Yeah, they're familiar with us.
No, like I say, some people just have not been around for a while, and maybe just come back after.
A period. So, what brought you back?
Somebody needed to heckle Lee.
There you go. Oh, you do a pretty good job, Grant, and I appreciate it.
Well, and you return the favor, and I appreciate that.
Now, if Gary would just stop asking questions…
And if we could get Stan to ask a few more.
As long as you number the pages, I know where to go.
But he, he did. Oh, why?
Oh, I think Stan's been very good tonight. He's stayed awake long enough to, uh.
flew us in when we need to provide more data.
Uh… We love you, Stan.
Storage is a complex topic.
Yeah.
And it's kind of one of those ones that you tend to
deal with it when you're setting up a computer.
And a lot of people leave it the heck alone.
Other than times when they're forced to do it.
Lest they run the risk of losing data.
Ah, yeah, yeah.
Eric, I'm sorry.
And Merry Christmas to Norman.
Or, as I usually say, merry ho ho and happy ha-ha.
Alright. Um, well, I guess, uh… Okay, maybe that is it for the evening, then. Um… If there's no other, uh, comments that should be recorded. We'll put a bookend on it.
This was the St. Louis Linux user group, the St. Louis LUG. It was our monthly meeting on.
Thursday, the 18th of December, 2025. Our speaker this evening was Lee Lambert from Omnitech Corp.
And, uh, he's also the treasurer of this group for a number of years and a frequent speaker. We always appreciate.
That, and all of, uh, all of the, uh, machine time and effort, uh.
Dedicated to administrating the systems lead. And, of course, we also really appreciate a mind-bending topic tonight, so… Wait, wait on the next one.
It gets better or worse. And don't forget, we've got 2 more to go. That's right.
We're only halfway through. Oh, boy. Um, so with that, thank you all for coming this evening, and uh… we'll turn off the recording.