Uh, thanks for the reminder. Hang on, recording first.
Okay.
Okay, this is the bookend. This is the monthly general meeting of SLUG, St. Louis Linux Unix Users Group.
The date is Wednesday, 10th of December 2025. Uh, we have our usual Oreo cookie version of a meeting.
We have our basic tutorial on the front. Our main talk on the back, and GUI middle, which has the announcements and the introductions.
Uh, that, uh, main talk in the back will be.
the, uh, better file system, uh, it's by Lee Lambert.
And our, uh… Uh, base tutorial topic tonight is.
Need a router, firewall? Added by Grant Taylor. And with that, I turn it over to Grant.
Um…
Yep, and I am operating at a deficit here.
Screen here…
And I am relying on Lee.
So, um…
Some of this is…
Fairly simple, but I anticipated there…
would be questions to fill in, and uh…
Audience participation with everybody except…
Gary.
You got your number. I've been a bad boy again.
So, routers 101.
What are they? What do they do? Why do you want one?
And firewalls as well.
Clicky the button.
Okay.
Oh, where's the button? There it is.
So, what is a router?
What is a routing table? What is a route?
What's a firewall? What's NAT? Various other questions that are all in the scope.
of router firewall devices.
Um, a router. It's a device that does routing.
So what's routing?
determining where to send somebody or something, and doing it.
It's that simple.
Uh, real-world analogies.
Let's say, uh, you need to park at a hospital and get redirected to the proper building.
The parking attendant kind of functions as a router. Yeah, you want the next building down, park over here.
he or she is routing you.
When you walk in the building, you ask the receptionist.
Hey, where's this department?
Down the hall, take the elevator to the third floor, and it's on your right.
That's another form of routing.
When you get to the nurse's station where you're at, hey, where's my friend? Uh, they're in room…
506 down the hall.
These are all forms of routing that I expect many people have either experienced themselves,
or know somebody that has.
The exact same idea,
happens with IP packets on the little blue box,
It's plugged into your internet connection.
Um, comments, questions, before we move on?
Gary, you are allowed to ask questions.
Oh, that's a relief.
Okay, uh, routing table.
It is a list of directions, where things go, and how to get there.
An example might be the building directory.
or the parking attendant looking at which departments are in what building.
Are the receptionists looking up your friend's name on the computer?
just… the piece of information you're looking for, and where it's at, how to get there.
Um, okay? And summary, route table is like a map.
of networks, or map of people, or…
You ever looked up where you find the, uh…
salsa on a directory in the…
grocery store? That's a form of a routing table.
Next.
So what's a route?
Basically, it's a piece of information.
That says what you want,
where it is at, how to get there.
The routing table is a collection of those.
Um, okay.
comments, questions?
I like this analogy. Makes a lot of sense.
I tried to make it approachable.
So, what's a firewall? We hear about firewalls in the context of routers, or on our computer.
Um, but on a router, because most small office, home office routers,
Like, you plug in, or your ISP provides, or you rent from them.
They are routers that also act as a firewall.
So what's the firewall doing?
Well, the firewall,
is really something that just controls what passes through.
Um, if you've done any home repair, or been around inspections,
You might have heard about that drywall is fire-rated for a certain amount of time.
It keeps one thing, fire.
On one side for a certain length of time. It is a firewall.
We have borrowed that name and applied it to
computer and network security devices to keep fire or…
bad things outside the
virtual wall that is protecting our network.
Any comments, questions, or concerns?
Okay? Um…
Stan or Gary, I think you were asking this one in particular.
What is NAT?
Um, well, real snarky. It's network address translation. Don't you know what that is?
NAT is really a way to translate
nomenclature or identifiers.
the hospital analogy is, maybe your friends are in Suite 6.
But multiple departments have Suite 6, so you can't just say Suite 6 to the receptionist.
She won't know what that is.
But if you say maternity Suite 6, oh yeah, I know what that is, go here.
Or maybe maternity is on the fifth floor, and Suite 6 is room 506.
If you ask for room 506,
in the area of the hospital,
people will know what that is.
That is a different name or identifier.
4 Suite 6.
Does that make sense? Because this is a repeated pattern.
Gary?
Do you… make sense?
Yes. Yep.
Okay? So, we can also expound upon that.
And, uh, I feel like something in…
my original copy didn't make it through.
I made a reference to…
two different buildings on the right brothers, uh, hospital campus.
go back. Um…
So, you may have the Orville building that has room 506,
And you may have, um…
I think it's Wilbur Building that also has a room 506, so if you ask the parking attendant,
I need room 506.
Which building? So, we prefix it with Orville.
506. Again,
We have added a new name,
Sweet 6 is known as room 506 is known as
Orville506.
Clear as mud?
Those of you who aren't aware, that's another joke to Gary here, who, once upon a time, was parked in the West parking garage, and the attendant.
Had to explain to me that, no, my car was over in the east parking garage.
Oh, I didn't even know about that one. Talk about winning!
After you…
Um, but that is an example of how the same thing
can be known by different names,
or by different addresses. We're translating the address
through certain contexts.
Make sense, Stan?
Okay?
I like the Orville and, uh… Wilbur was it?
Yep, I'm pretty sure, and…
I think my internal draft had a few more notes, but…
I have to interject real quickly. Thank you, Lee. You took my notes and produced something while work exploded on me, so I am grateful for it.
Okay, um, so network address translation…
You're very welcome, my friend.
Yeah.
It's really converting between those different names or numbers.
or addresses, pardon me, it is translating between those addresses
to allow the same thing to communicate.
Um, and yeah, it helps conserve IPv4 addresses.
There are probably a whole bunch of 192.168-something-something addresses around the world.
Your little blue box,
whatever color it is that connects you to the internet is converting your Suite 6
version to Orville 506 to talk on the internet.
Does that stand? I know you were kind of…
really helped me understand this, uh, when we were talking about it. Did I answer all your questions, or…?
Let's make this interactive.
Yeah, yes, and I often ask really dumb questions to get.
good answers that other people are afraid to ask.
And other people can ask questions here besides those that have already spoken up.
Uh, there's gonna be a couple of questions from the chat room.
Okay. Yeah, I do not have that open. If you think now is a germane time.
Uh, and a little bit later.
Read them off, please.
How not yet, not yet, a little bit later.
Okay. All right.
Alright, clicky-clicky, Lee, please.
Okay, so…
A very heated topic is…
Is NAT a security mechanism?
In its purest sense, no, not really.
The way NAT is used, and some other aspects,
provide what seemed to be security, but it's really the other pieces not
NAT itself.
Um, combined with other things NAT can provide some security.
Uh, what happens when you translate…
So you've got somebody in Suite 6,
Maybe they ordered a pizza.
Um…
And let's, uh, maternity ward.
Sweet Six, or Room 506.
You're an uncle, or an aunt, or somebody in a family you're going to visit friends who just had a baby type thing.
You get there after work,
And they're hungry, you decide to spring for a pizza for them.
And need to tell the pizza delivery how to bring it in.
So there was some other things that kind of fell on the cutting room floor.
But… for the pizza delivery guy to be able…
or gal, to be able to bring the pizza to you,
The receptionist…
is probably gonna say, no, you're not family.
Um, they need to send somebody down to get the pizza from them, so you…
walk out of the ward, go down to the lobby,
and get the pizza, and pay the delivery person.
And walk back up…
And when you get back up to the maternity ward, you buzz and say, hey, can I get back in? I walked out to get pizza,
that, can I get back in?
I just walked out to get pizza,
is what we call session state.
the nurse working in the ward, he said, I'm gonna go down and get the pizza, I'll be back in a minute,
goes, oh yeah, you did say that, come on in, because
she knew you were in before, you're returning, it is okay for you to return.
Does that state make sense?
And then that state is very, very, very critical.
Yes, it does, quite a bit.
Because the way NAT
As a security mechanism,
works is it's really…
that session state…
under the hood.
If somebody else…
walks back up and says, hey, I just went out to get pizza, can I get let in?
She looks at you and says, you know this Joker?
You shake your head no, and she says,
Sorry, you can't come in.
So, the lack of state…
for that other person,
and her not opening the door…
is what provided the security.
it's having the session state. Is it known session or not?
And then the blocking unknown things, that's where the real security comes from.
Not NAT.
Okay.
I'm wondering about tailgating at this point.
Hypothetically, we're presuming tailgating is not an option.
Um, one of the things… I don't know if it hit the cutting room floor, was I made a comparison
to the turnstile.
at, uh, Menards or various other places where you have a physical barrier that essentially only one person can get through at a time.
And you have a badge reader or something to say, let me through it.
there's some state associative.
Does that make sense? Do you understand why NAT by itself, changing the names,
isn't the security, but the security is really
I saw you go out to get the pizza, yes, you can come back in.
That makes sense?
Yes, and that's… Nat alone, or is it dealing with the firewall?
It is more firewall session state.
At this point.
It is possible to do that session state without
Okay.
Nat.
And it's possible to do NAT,
without session state.
Okay?
Yep. It's… they're kind of linked at the hip,
Good food for thought. Thank you.
But they're two separate things, and the session state is where the security is.
Not Nant.
Okay? Lee?
Uh, let me… let me ask a question that I've heard before.
Okay.
Uh, since… Well, since the.
Mm-hmm.
activity across NAT that you would envision, or. Happening. It's always outbound.
If there was not a session. There… A lot of people would think that your inbound security is fairly tight.
You said something very critical.
If the session is not there.
Which you are completely correct.
And it's really that session, or the absence of it, that is providing the security.
And it's kind of like, um…
You could think of it as a parking attendant. I…
Show me the stub from when you parked your car for you to be able to get back into the building to go get your car.
that parking stub is a form of session state. It has today's date and time on it.
Without that, you can't get in.
Makes sense?
Yeah, and that's, uh… Differentiate… differentiation in that.
Normally, you think of Nat, you don't think about the session.
Right. And NAT, in its most common sense,
is what we call many internal LAN IP addresses.
to a singular external internet IP.
So if somebody is banging on the outside, let me in, let me in!
If there is no session state, go, yeah, it's associated with the person sitting on the couch, little man.
Have their pizza. Um…
If there's no state,
it… even if it… it cannot function, there's no…
way to come in. Um, one thing I don't think I really talked about is you'll hear about port forwarding. Used to be a common thing for games, or if you want to run a web server at home.
That is a special, permanent session.
that says, if somebody comes and says, can I get a glass of water? Yes, you can go to the kitchen and get a glass of water, but you can't do anything else.
All right.
Okay, sounds good.
Uh, yeah, the NAT can be many to one, which is what…
most people have at home.
It can also be many-to-many.
I can have an entire network
that I change the first
two sets of numbers, but the second two sets
stay the same.
you ever have nicknames that you call your friends?
that you have, uh…
name for each friend,
So you've translated their name
But you have separate names for all of them. That is a many-to-many
relationship.
Does that make sense?
Uh…
No. I lost… I lost it.
So, we could have…
CEO, CFO…
And, uh, PFY.
as people that are in positions in the SLUG organization.
I believe CEO…
is a name or title.
In this case, for Gary.
CFO is a title
For Lee, I think I'm getting these right.
And PFY…
is a title I hope to achieve someday.
So, I can say, hey, CEO, talk to the CFO,
to get the information you need.
That is a one-to-one translation of CFO.
equals Lee.
C-E-O equals Gary.
P-F-Y equals grant.
Do you see the one-to-one relationship there?
Does that make more sense?
So, um…
Yeah, somehow I lost it with the many-to-many.
And I kind of should have put those in the… done the one-to-one first.
The many-to-many…
Usually, not always.
is…
one-to-one, just multiple sets of it. Like, you've got
three one-to-one relationships that I just described.
Does that… okay. It can also be…
Right.
We have volunteers…
that can reference multiple people working on the server migration project. That would actually be a one-to-many.
Does that make sense?
A lot of smoke coming out of my ears, I'm not sure.
Fair enough, and…
The main thing to take away is the many-to-one. That is what
probably everybody on this meeting.
is using right now. I would almost…
Guaranteed. The other…
our specialty situations that usually it's networking professionals.
configuring a firewall for the corporate location.
Does that slow the smoke?
Okay.
A bit, yeah.
So the… are we on? Yeah. The first part of… the statement.
Uh, like, let me take the, uh… Uh, many to one, for instance. In that case.
The leading word, many. Is describing… addresses on the outside.
Inside.
Inside, excuse me, that's probably, yeah. And then one is describing.
addresses on… The outside of your network.
Yes, sir. So, Stan probably has many…
devices on his network at his house.
And he very likely has won
globally routed IP,
On the outside.
Does that help?
Sure.
Can I ask a clarifying question? Oh, you heard me!
That makes sense.
Yeah.
I got a new configuration, just didn't know if the audio was working. Okay, uh.
It works.
Um… so… If you… let's say you had a corporate network, okay.
And let's say inside the corporate network. And I'm just making this up, was all 192.168.something-something, right?
Sure.
And then… you had… Um, you know, so all of your… Your employees, when they exited.
The corporate network to go to visit various websites.
They would… they would have that, you know, many-to-one relationship, right?
But, let's say that the same… Network also had behind the firewall.
a web server? For the… for the company.
And that had a different IP address. Right? And that even that different IP address.
Was identified externally with a DNS number. Or, excuse me, name, and… I'm just asking Claire, I don't even know if this is right.
Well, no.
But is that, like, some example of a many-to-many relationship?
Yes. Uh, well, I wouldn't call that a mini-to-many.
I would call that a mini-to-one
and a one-to-one.
So you've got many people on their workstations
to one IP address for general web browsing.
You also have
a one-to-one relationship
Mm-hmm.
for the web server to its singular external IP.
Right, and that would kind of be port forwarded or something, maybe.
Maybe. Um…
it would… you kind of get into some minutiae, but do you see how that is two separate NAT things side by side? A many to one?
and a 1 to 1.
Okay.
Wouldn't that be better described as, uh, that web server being on the DMZ?
that, um…
Yes, you could.
you…
really start to get into what is the DMZ, how is it configured,
Um, it used to be…
DMC…
25, 30 years ago,
used to imply…
a network configuration where you had an external
firewalling router,
the DMZ, where your web server or web servers,
And then an inside router,
the firewalling router
separating the office LAN.
So, for the office LAN to get to the internet,
it would pass through the first router,
go across the DMZ network out the second router.
And DMZ, in the classic sense, is…
systems in the DMZ,
can talk to the internal systems,
and external systems.
And there was no guarantee that internal systems
could talk to the internet
without using a proxy or some other application on a system in the DMZ.
a more common…
in the last 15 to 20 years, use of DMZ.
is a specific IP…
is set up that if you come to…
And try to bang on the door, let me in!
It says, well, I don't know who you are.
I don't have session.
But I have this special rule at the bottom of my session table,
Anything I don't have stayed on, send it to this one internal computer.
That's what…
I was kind of… I was thinking, in effect, you know, when we talk about port forwarding.
Mm-hmm.
You know, that's, like, one port to one computer, but… when you enable DMZ, in effect, you're saying.
Anybody on the outside. Redirect mode to this web server.
Yes.
You know, all the ports, which, you know, that would… Keep the… all the other people on the inside could not have their own routers, or I mean… their own servers going on their machines.
Because all the… all the… the ports, or whatever, are directed to that one machine.
Largely, yes, I agree. You get into some real minutiae, there's…
a thousand and one ways to deal with this, but in the common contemporary sense of DMZ, you are correct.
Oh, and getting back earlier, when you were using the analogy.
Yeah.
You know, getting to the hospitals or whatever, and you had Wilver, and uh… Anyway, I was thinking that I would have used that as a good example of, uh.
Port forwarding, you know. People saying, well, you know.
I want to come in and see so-and-so, you know?
And the guy's in there, wait a minute, he says.
No, I got two… two big… buildings behind me, I got Wilbur and Arnold, you know? You gotta tell me which one.
And…
And that's where, you know, the port-forwarding rule, that would be the guy goes down to the.
to the guard down there and says, if somebody asked for, you know, this room, send them up to Wilber.
Uh… yeah, I would probably…
we get into some minutiae, but I would probably say, like, if somebody is asking,
The guard, I need in, I need in, and no details. It's like, the emergency department is over there in the Orville building,
they can help you. Go there.
that might be analogous to port a DMZ port forward.
Does that work for you?
Okay.
Yeah, I think that'll work, yeah. Thanks.
I was gonna say, it's not your last one.
Go ahead, Ed, you're fine. I wanted this to be interactive.
I have one final question, not final question, but can… if… Yeah, never missed. Anyway, um… Um, yeah, so… Uh, so getting back to real… the real… life scenario. Let's get back to, uh, people's home networks.
Okay.
with the 192.168. scenario. Um… And this isn't about port forwarding, this is about… The, uh, the other report… So, what I mean by that is… Okay, let's say, for example.
You've got your web… you've got two machines, and you have one, the web browser on the first machine.
Those out to the internet. Right? And because of NAT translation, it appears.
to the internet that's coming from. Whatever your ISP assigned you for that IP address.
Yeah.
And then, um… You connect up to something.
And it needs to have a reverse. Connection back to you.
Right, and then… It chooses some kind of, like.
Sort of random port. And then, any packets coming back.
are, like, going to this random port. And when they come in on that random port.
At the IP address that your ISB gave you.
And then they hit your NAT firewall. The net translation goes, oh, okay.
I remember that guy. That was. Uh, machine number one.
And it routes the incoming packets. To machine number one.
Um…
you're close.
Do I have that right?
Um, the quintessential example
Yeah.
of this is File Transfer Protocol.
which uses multiple ports.
Okay.
And you might have heard active versus passive modes.
So, what happens is my internal client…
Mm-hmm.
establishes an outgoing connection
to an FTP server, say I'm wanting to download the Linux kernel source code,
Presuming they still offered it off of FTP.
And I told the FTP server I want to get
the latest dot.
TGZ or whatever.
Historically,
The FTP server on the outside would try to open a new connection
Inbound…
to send the data to you.
And what's really happening under the hood
is your FTP client says, okay, send me this file on my internal IP and this alternate port.
And it starts listing on that.
port on your computer.
And then the firewall…
As it is passing out, the firewall and NAT component work in conjunction with each other,
The NAP piece translates the internal IP
to be the external IP,
But it also goes, oh…
The internal client is saying, use this port number.
So it establishes some additional state in the session table,
that says, when the outside server comes back in,
on this other port,
here's the session information, send it to me.
So it's really two…
can be more than two.
sessions working in
concert with each other,
and the NATning firewall,
has to analyze and intercept
The details, so it knows what happens when the external server
initiates the new connection to send the data.
That is the classic
active connection.
Um, the newer came about because of firewalls.
Passive mode
flips the data connection.
When you do passive mode,
instead of the client telling the server,
create a new connection to me.
the client and server talk,
And the client creates a new connection.
outbound.
Does that make sense? Does it…
Yeah, it made sense on the FTP side of it, right? I was thinking more of the HTTP side.
Um, short of newer…
But, okay.
I'm trying to think…
HTT…
Well, the reason I'm saying that, you know, if you had the.
Mm-hmm.
Uh, Machine 2. Right? And it… let's say it's going to the same website, okay? Right? But, you know, maybe it's a different page on that website, okay?
Right.
Um, you know, and… and the server on the other end, the web server.
has to send packets back to a single IP address.
Yeah.
Uh-huh.
Right? And… how does it know. Which… Which, which, uh, you know, when it gets to the IP address, like, how does it disambiguate.
The, uh, incoming packets. that were originally outbound, it was the GET request.
And then you have the response. Right? So you have two of those coming in.
Yep.
coming to the same IP address. Where… how do they get to machine one, and then how do the other ones get to machine two?
that is a very good question.
And I have a one-word answer, which I'll subsequently explain, and that word is
ports.
So, what happens is when you're
client, and I'm gonna pick on… let's say you've got your…
computer, and you've got your phone in your house, and you're surfing the web with both of them.
Your computer sends a connection from a given random port
on its LAN IP address,
out to the server,
IP address and server port.
As that connection goes out, that session state
includes…
your computer's IP and the port.
your computer is using, as well as the server IP and server report.
Similarly, when your phone goes out to the same server,
It has its LAN IP address and a random port.
So what happens is the NATing firewall router goes, oh,
Ed's computer port
1, 2, 3, 4…
is talking to server…
port 80. Ed's phone…
on port 3456,
is talking to…
Server and port as well.
So, what happens is, as it's being transmitted going out,
there are some different…
complexities in NAT stacks.
implementations, but many of them, if they don't have to change the port,
They won't. So…
what happens is the web server sees two connections.
from your router. It sees one connection on port,
Uh, 1234…
And a second connection from port
2345. So it's actually replying from the web server,
IP import back to the outside of your router,
port, sending it to the IP, your service provider gives you,
port 1234,
your router goes…
internal, or, um…
Outside IP import.
to outside server,
maps in the session data,
to your internal workstation,
port 1234,
And the reply to your phone,
Well, that's outside IP port 2345,
Hey, session data says that goes to Ed's phone.
port 2345.
Okay, so to clarify, the, um… Uh, the browser on each of these devices.
is the… is the guy who's making the decision about.
Which random port to use? Not the NAT firewall.
Um,
it is probably more the kernel on the device than the browser, but
Okay. What's…
the firewall…
Depending on its specific implementation,
may go, there's no conflict between
port 1234,
import 2345…
So, send them out as they are, change the address!
But don't change the ports.
Now, if Gary is over at your house for a slug meeting,
And he's trying to browse the web, maybe look at the calendar on his phone, or his Chromebook,
And his Chromebook says, eh, roll the dice, I'm using port 1234.
Firewall goes, wait a minute…
We got a conflict. We got two people.
using port 1234. We can't have that.
So the first person gets to use the port,
The second person, it will change the port
2, 3, 4, 5, 6.
to go to the web server. So the reply coming back to Gary's phone
or Chromebook comes to the outside,
port 3456…
firewall looks through its state, oh, okay, I've got some session state.
for, uh, external web server port and my outside IP, 3456,
That goes to Gary, let's send it over to him.
Does that make sense?
Yes, sir.
And it gets retranslated to Gary's 1234. Okay, that's what I was trying to clarify. Thank you.
Mm-hmm.
That also explains why I have this picture of Ed on my phone.
Every month.
We're not talking about the, uh, remote code execution Ed used against you.
Dang it! I mentioned Fight Club.
Yes.
Okay, anything else?
But, by the way, I meant to get in the question.
So you explained what CEO was. What, who, and what CFO was, what, who.
Uh, PFY.
P YO, was it?
PFY.
Does anybody know what BOFH is?
Nope. That's BOF.
Birds of a feather? No, that's not right. Bastard operator from.
Yeah, some operator from. Yeah!
bastard operator from hell.
What was the BOFH assistant called?
Okay.
Pimply-faced youth.
I don't know.
So, I…
Okay. Carry on!
you know, I'm just a peon in a cog in the system, so I called myself the PFY.
Any other questions?
How far along are we? How far along are we?
I think we're probably relatively close, uh…
SlidePresenter, how are we doing?
out of…
Uh, we're on number 9. I don't know.
Oh, Godwin.
72.
Uh, so we actually talked about the stateful packet inspection.
Uh, that should be…
That is the session
table of…
I look at the, uh… I inspect the incoming packet,
Do I have session state for it?
Yes or no.
And based on that decision,
do I allow it through?
An example might be, is the patient still checked into the hospital?
Are you in the right building?
Or have they been discharged and gone home? So no longer…
in the session table.
Does that make sense?
Okay.
Yes, it does.
Uh-huh. Now, sorry about… sorry about the cutting room floor, Grant. I didn't look at these too closely. I didn't realize that we're missing some.
splits where you take a long slide and make 2 or 3 short ones out of it.
Yeah. Um, you know what? It is better than the slides I had, which are null and void.
So, I am grateful for what you have done.
Another example is the nurse might say the baby is sleeping, you're not allowed to wake them right now.
Session denied.
So…
Any other questions about this, or are we ready for the next slide?
I think you're going to the next slide.
Uh… final summary and key takeaway… key points.
Ready.
Routers forward traffic.
literally take something in one side, send it out another.
Routing table is where things get sent.
or as a collection of where things get sent.
A route is a singular, where does this thing…
get sent. Firewall…
controls access.
NAT changes the names as things pass through.
You might think that NAT is not the real source of security.
And that it is the session state.
That is the real source of security.
But NAD is very frequently used
with session state.
the session state,
uh, stateful packet inspection is the real security mechanism.
And that looks like it is time for a CTRL-D, unless anybody has any questions.
I'm going to ask if, uh… I don't know if this will take too much time or whatever, but… People would be interested in knowing about, uh.
Uh-huh.
The default route? And… and the net mask.
Okay. Uh…
Stan, do I have 5 more minutes?
Certainly.
Okay. So, the default route, or the default gateway,
is kind of special.
It is literally…
If there is no…
route or gateway, and routes and gateways are the same thing.
historical nomenclature.
But if I don't have a specific location,
By default,
Send them that way.
That's really all the default gateway is.
Match everything that I don't have something better for.
send it over to Lee's house.
Um, and net masks.
I like to compare
net masks.
to the size of a room.
As in, if it's a real small room, kind of like an elevator,
It might be a net mask of, like, slash 29.
kind of small. Doesn't hold many people.
But maybe a more realistic net mask.
is a slash 24.
might be…
Lee's Batcave, where multiple people can
hang out and chat and have interesting comments with speakers and Chromebooks.
And then we might have a football stadium.
that is really big and holds a lot of people.
That might be a slash 8.
Does that make sense to you? And the people are analogous to the IP addresses.
in the network.
And I can add on to that,
I really like those analogies.
A gateway…
is a door…
to another room, or another network.
And sometimes you have a door to get into the hall,
to go down the hall to get into yet another room.
So, let's say the kitchen is down the hall, we have a route to the kitchen,
through that door.
And in the event of an emergency,
The fire exit is there. That is the default gateway. If ever in doubt,
go that way.
How do you like them apples, Stan?
Very good. And emergency exit is this way. Yeah.
Yep. Um, or any other time, it doesn't necessarily have to be an emergency,
But it's like, if ever in doubt,
go that way.
Uh, 3 items from the chat room, if we're ready for them.
I want to take a drink of water, but go for it.
First one is from, uh… Carrie, will you cover WAN.
failover, or is that too advanced for this presentation?
Uh, not to chicken out.
that gets quite more complex, and not tonight?
Um…
I would be…
happy to do something on it.
Um, in the most basic sense,
instead of having one
default. There's a…
Try to go out that door. If it's blocked,
go out this other door. Or vice versa.
Beyond that, it gets into a lot of technical minutiae.
And it's kind of situationally dependent, and…
lots of aha and gotcha and ouch that hurt.
things.
Okay.
I think that's good enough at this level. Next question from BJ, I believe that's Brad Jones.
It says, that parking stub. Analogy is probably the best.
analogy I've heard when explaining session state, mind if I borrow it?
I'd like to point out that these sessions are recorded and.
Uh, people are welcome to use the whole recording, and they're pointing things to people, besides using.
Uh, bits and pieces. Comment?
Um, my…
request, not requirement.
is that you give a nod towards me, or buy me a Coke Zero.
And then, by all means,
I would love for you to share the information and help other people learn.
And really, the Coke Zero is just…
I'm kind of selfish, and I want something.
Uh… A third item is from S, I believe that's Stuart. He asked, what is double net? I think that's what it's called.
Seems to cause issues.
The tackle is…
there is what has become…
what most people refer to as double NAT.
And then there is what I refer to as double NAT.
I will explain the former first.
And… double NAT to most of the internet today,
is you have NAT,
happening on two separate devices.
ostensibly back-to-back.
So if you have your little blue box router that is NATting
your many-to-one, your many LAN IP addresses
to the one address on the outside of your connection,
And then your ISP.
is natting the many
ISP customers' connections,
as it passes out to the internet, so you have NAT happening
at your router, and again,
at your ISP's router.
That is what…
Most people refer to as double NAT.
In both of those cases,
the address that's being translated
is the source address of an outgoing packet.
So, we might say Sweet 6…
gets translated to 506.
And then, as you leave the building,
Room 506 gets translated to Orville.
506.
Not quite, but it's close.
Um, does that make sense? You see how we're translating the source address
two different times, or double translating.
makes sense to me. I'm not sure… I don't know if it makes sense to… To Stewart.
Um, we'll give Stuart a few minutes to respond while I say what I consider DoubleNet to be.
I hark back from when Nat was a special program run on special computers.
30 years ago.
Um, at the time,
we used to do far more complex things where…
we would translate the source address,
like normally happens.
But we would also translate
The destination address.
So I'm doing two different translations.
on a single device.
Does that make sense?
Makes sense, but it sounds like a nightmare.
Oh, it can be, and it usually comes into play
When you're trying to interconnect two companies that are both
192.1680 dot.
Uh, slash 24, and how do we get the…
GIS person in this office,
to talk to the GIS server at the other office.
when it's the same subnets on either side.
So you usually end up
translating the source
and the destination, because what you would do in that scenario
is, let's say the GIS client is…
call it A, going to what it thinks is B.
it gets translated to something else,
C and D.
Where D is the real destination,
But we make the GIS person look
to the FAR network as C.
So we've translated source and destination
across that business-to-business connection.
Or, of course, the real reason there is the PFK had no clue how to design a network.
Um…
That is a real concern.
But acquisitions and mergers cause far more of that than you might realize.
Yeah, but I would, to that, respond. You shouldn't be using 192, 168 for an internal network.
In any case, because if you ever set up a VPN, I guarantee you're going to run across clients that are 192, 168.
Okay, fair. You, in…
to your point, I could, instead of 192.1680-something,
It could be 10-10-10 slash 24.
I believe a lot of that is addressed to Stuart's earlier response.
that, uh… Why does that then cause problems? So, you've explained some of these problems already, I think.
Um, there…
Believe.
One of the other real common problems is, like, when you're wanting to set up gaming for your Xbox,
or pick your console device, or your web server.
You can go to your router.
and say, I want to turn my server
into the contemporary DMZ, anything unknown, send to my server.
But…
That's your router.
What about the ISP's router?
that is also NATing. You have no control of it.
on a residential connection, it is almost guaranteed the ISP will refuse
any connect… any request
to help set something up for port forwarding to you.
They'll say, if you want that, you have to upgrade to a higher connection.
class, where you'll get a globally routed IP.
give us the money.
So that is another big problem.
Um, and you kind of run into some things like FTP or
SIP for VOIP,
where…
the protocol has IP information
inside the payload that has to be translated
In addition to the addresses and possibly ports passing through the net,
And there's no guarantee that your ISP's device
is capable of…
or configured to do that. So it's additional…
boundaries, one of which you have no control over.
That's one of the bigger problems.
with double NAT, as most people think of it.
Okay?
I think there was some, uh… ISPs or whatever that… We're kind of had their routers… You know, locked down.
And they didn't want you to use yours. And one of the ways around it, these guys came up with was.
They would go into that router, and they would configure it.
To do the, uh, DMZ. And they would hook their router up.
To the… to the ISP's router, and everything would go.
from the ISPs into their router as the DMZ.
And then in their router, they could divvy out to whatever other.
computers they got on their internal network. And that was kind of their way of being able to use their.
own router, as opposed to using the ISPs router.
Yeah, um…
And I think what you're referring to there is
let's say I have a…
cable modem from your favorite cable company, pick your own name.
And they have a modem router combo and thou shall use
ISP-provided device, no ifs, ands, or buts.
So, yeah, the thing in that case is…
that router is at your house, you have a modicum of control,
and say, uh, DMZ everything to my singular client,
Which happens to be
my router that I have full control of and can do whatever I want.
Mm-hmm.
And I think your example I was using was, uh… I think it was AT&T, and it might have been there early, uh… U-verse or DSL, one of the two.
They had it set up where they wanted, you know, they didn't want you to use your box, they wanted you to use their box.
Uh, you've responded on the list, all that.
Yeah. Um…
A lot of guys wanted to use their own, so they did that.
Yeah. I see that.
Um, for AT&T Fiber at home.
contemporarily, and I cannot justify the expense
for what is needed to…
remove…
the AT&T device from the mix.
Um…
So…
Well, AT&T ONTs do give you multiple. Connection nodes. They will even do a pass-through.
Yeah.
Which basically… which basically bypasses their router function in favor of whatever box you hooked on.
Um, maybe contemporary firmware, but when I set mine up here at my house in St. Louis,
two and a half years ago.
they… the pass-through…
did not work for me.
But I had the complication of I had an 8 block of globally routed IPs.
And that may have prevented me from using pass-through.
is highly situationally dependent.
Because… because… Yeah.
Okay.
So I suppose that IPv6 routing is outside the scope of the stock.
Alright.
Um, no.
Here's what's different in IPv4 and IPv6.
Address length.
Everything I just talked about doing with IPv4, I can do with IPv6.
the technical execution is…
Well, I guess my question was, like, if… Um, if you had a… IPv6 internal network… And you had your router.
And… the external stuff… was a combination of.
IPv4 and IPv6 in the. And the wider World Wide Web.
Uh… in your little box was IPv6.
Right? And it's going out, and it says, uh, okay.
You've hit the external network, and that's an IPv4.
Um, so…
Right? Is that Nat doing that, or is that… The router doing that…
are… it sounds like you're asking about what I would… what I would refer to as protocol conversion,
from V6 to V4,
or V4 to V6.
Which is a thing.
It is complicated.
Mm-hmm.
And I don't see it happen…
Very often.
what I see happen far more frequently…
is dual stack.
Which I have here at the house.
Um, or…
IPv6 only.
And getting from IPv6 only to IPv4…
requires…
something to do a conversion.
And other way around, to go from IPv4 only.
two IPv6 only.
also requires a conversion.
It is a thing…
It is usually referred to as NAT64 or NAT.
4-6.
It is a complex thing.
I…
I would run dual stack.
Okay. Yeah, just curious to know if, uh… If you think that that's a good idea.
Well,
For security reasons. Of course. I think it's more security through obscurity, which is never… You know, inherently safe.
Keep in mind, the security that you're ascribing to NAT is really…
session state.
Make sense?
I think we need to move along here, folks.
Okay.
Um, I'd like to point out that a normal process is described.
Our sequence of events, there's a, uh… I, uh… group of slides that's available on the.
front page, uh, this that I'm linking to right now, and I'm gonna run through it real quickly, because if I let Gary run through it, we'll be here another 20 minutes.
Because…
Actually, Gary wants… Gary wants to interject a quick question to the group before people… Start making up their minds on something.
Uh, uh, it can… Hopefully people can do a show of hands, they know where the little button is down there under Moore, where they can raise and lower their hand.
Uh, how many people are here. Uh… exclusively because they wanted to hear the tutorial tonight.
We're gonna count that up. When I'm driving at here is… I wanted to get a feeling for.
How many people want to hear and elaboration kind of networking.
Introduction part two. Uh, not tonight, but at some future meeting.
Uh…
I would vote for that.
Good.
Um, yeah, that's good. Go for that, too.
Anyway, I wanted to talk out, there's a number of slides in that thing there, and this is the list of them, and I want to run through them real quickly.
Uh, the one slide talks about us. We're a… a 501c designated not-for-profit professional association. Our meetings are free and open to the public.
Um, we have a web page, www.sloop.org. And a lot of guidance available on that webpage, so our agenda usually goes through… there's a… we start at 6 o'clock for early arrival for people to test their chat, microphone, sound, and screen sharing, and then we start the actual meeting, and then do the base session when they have.
It's a middle… as we go through, try to quickly do the announcements, archives, sponsored meetings, and talk about calendars, mailing lists, and so forth.
And then get into our main topic thereafter. We're running a little short on time here.
After the meeting, with further discussion, is possible on our mailing list, and we… just because the main topic is over, we don't have to end the session.
These sessions are recorded along with the chat, the slide work, and videos are available on our archive page within a few days.
Our introductions… Gary's the president, vice president is me.
I do the lot of production. Treasurer is Lee Lambert, he's our host.
Uh, we have a vacancy for secretary. Uh, any… I've done some… the chat monitoring for the earlier portion here. We need top… Wranglers to help us find other speakers, and we always need volunteers.
Um, that… we just covered. We just covered our base topic.
Frances are principal place.
Um, and we try to keep it simple and small, uh, that this is the gooey middle now.
And then we stop there, and we went back and talked about all that.
And… Next, we're gonna do a… Call for help, real quickly here.
Go ahead.
Okay, any… again, there'll be another call for help at the end.
Uh, but anybody who feels like they can't wait till the end and just needs a quick answer.
Uh, now's a chance to do a very quick call for help in the middle here.
Going once? Going twice… Third time, going right along.
Announcements, um… These meetings are on the second Wednesday of every month.
Uh, the next one's gonna be January 14th. Uh, 8 days from now, we're gonna have the… next week, a Thursday, is going to St. Louis Linux users.
group meeting, uh, 5 days after that is going to be the new Linux users meeting.
It's every… yeah, every fourth Tuesday. Uh, the next Slack meeting is usually on the first Thursday of each month.
Um… anybody that's been under a rock, Microsoft Windows 10, end of life just occurred in October.
Uh, any other announcements people would like to make?
Going once, going twice. 3 times. I like to point out there's a project that started in Europe.
Well, that's… that's the next slide, sorry. Bye by Windows 10 is a free PDF that talks about people going from.
the Microsoft Windows 10 to either Windows 11, or they can switch over and go, uh… to Linux. Half the book covers going to Linux, the other half goes… tells you how to get from Windows 10 to 11.
Yeah, it's a free book. It's available from Amazon, paperback or Kindle.
Or you can get a free PDF from those sites that are available.
There's the end of 10, uh… dot org project that came out of Europe, it points out.
Where you can go find help for moving from Windows 10 to wherever you want to go, primarily.
get assistance with getting to Linux. Uh, privacy and security, big brother's watching here, anybody can be recording this thing. We only record this… the slug and the St. Louis Linux meetings, because our archives.
Because they do presentations. The other meetings we have are free format. We may record it, but we don't post those meetings.
The archives are available, the SLUG. on every… the bottom of every… one of our web pages is… is listings.
We also have a calendar, and on the calendar, we… have all our sponsored events are highlighted in red. If you could put www.loob.org slash calendar, that'll bring it up for you.
We have mailing lists where people can, after the meeting, they can ask questions there.
Uh, we have the analysis, very limited traffic, and usually it's just the, uh, meeting instructions.
discuss, you can ask system administration questions there. You don't have to… belong to our sysadmin.
mailing list, because that's our volunteer support staff. It's actually the people who do the actual work.
Um, our sponsored meetings are… Listed here, the St. Louis Linux users group, they have our presentations.
that New Links groups is a beginner workshop, the Slack is a configuration workshop for, uh, hardware and software.
And… we didn't call for help. And it's time for our main presentation.
Which usually covers Unix, PSD. Mac, whatever, that kind of process.
cross-platform interoperable systems is interesting. They're often more advanced and detailed, important, more so than the basic sessions.
And are often focused on enterprise computing. Our presenter tonight is going to be Lee Lambert. Lee, uh… Do you want people to interrupt you, or as you go along and ask questions, or what?
Whatever you feel like.
Okay. Any other questions before we go right into our main presentation?
Go ahead, I'm gonna end my, uh… session here, if I can find the doggone button.
It's this one. Sorry, I gotta figure out which desktop it is.
Where'd it go?
Yeah, hmm…
Well, thank you, Stan. Uh, this has been an interesting project.
I'm gonna… we've got multiple… slide decks to go through. You can see them on the screen up here.
Make that a little bit bigger for you.
No… There we go…
The whole idea is I've got a main workstation here in the office, I built it a couple years ago.
You know, it's a… one of these fancy i9 things, I put it in the car.
put them in an old case I had, and so it looks like everything else.
But the point is, it's my main workstation. It is the heavy duty hauler here in the office.
And last winter. Again, almost a year ago.
I try to do an update on it. And for some reason, which I have not been able to figure out how.
After that update, the system would not boot. And so, for 8 or 9 months, I could install new packages, but I couldn't push updates because.
After the update, the machine would not boot, but.
One of the nicest things about ButterFS, which I'm going to show you in detail as we go through the material.
is the fact that even though. The disc was corrupted by some fashion.
I could quote-unquote, roll back. To the previous copy of the dish.
Of the disc, and again, I'm going to cover all this here in a few minutes.
And keep on truckin' with the previous state of the machine.
So tonight, we're going to start here on what's… what we're calling Butter FS100.
And I'm going to walk you through, step by step.
What happened, the steps I took to do the troubleshooting.
And how I solved the problem with some. Help from a certain nameless face called Artificial intelligence.
But then I've got a… we've got a couple other slide decks.
To go further into the Butterfs details. 200. Hopefully we may get into that tonight. If not, we can push it to a future meeting.
And next Thursday is an option, and if we have more material than that.
Uh, you know, we can do… even do another meeting if we need to.
And Butterfest 500 is some of the advanced behavior.
And I also put a slide deck together on Linux file systems.
Which may fit at the end, or may fit here towards the middle.
So, just to give you a perspective of what we've got going here.
Let's go ahead and get started. The storm incident. It's a real-world case study.
All the history has been recorded. And we'll use examples as we go through.
The slides here.
To start off with. Hang on a minute.
I gotta get this thing out of the way.
So I can see it. Sorry about that, my, uh… Zoom toolbar was covered in the slide.
So, as I briefly mentioned a few minutes ago.
The situation occurred last early last spring. When, after an update.
The machine wouldn't boot. There is an emergency mode.
Which I will show you. And… let's, uh… I'll have to find it here in a minute.
Normally, when you boot a machine. The… you'll see the BIOS.
Logo screen… After the BIOS logo screen, you'll see a boot menu that has been produced by the operating system.
that's installed. Uh, some OSes, most notably wind blows.
hide all that, or they try to hide it.
to make sure that you can only boot their operating system.
Just a quick comment. That drove me fruit on my new laptop a couple… 3 months ago.
Because I could not get into the BIOS. To boot a USB drive.
And that took me hours to figure out. Even though I had a blank hard drive in there. Okay. I put a new hard drive in there.
But I could not get it to boot a USB drive.
Okay. Because there was no way to get to a boot menu.
Alright. Because everything was… secure. Well, that's a side issue.
Oh, okay. Um, but anyway… The point is that various manufacturers.
Like to take liberties and play whatever games they want to play.
to customize the boot process for their own. Uh, devices… So, that boot screen may or may not be the first or third thing you.
screen activity that you see when you boot a machine.
And if that boot process fails. for some reason, there's emergency mode.
And again, uh, I'm… I'll… Gotta find that screen…
Okay, well… Sorry, guys. Um… I've got some examples here later on we can talk about.
So, in that emergency mode, you will see a screen.
That is not a fancy colorful boot menu from the OS, like OpenSusi does.
But it will be typically a black and white screen.
With a few entries on it, which you can pick, you can pick an entry and boot.
Or you've got two commands you can use. You can… you do an E?
For edit that boot entry. Or you can do a C as in Charlie.
For a command line, where you can build your own group.
Boot entry. But the basic problem was… The disc was missing. There were pieces of the disc missing.
Don't worry, don't worry about the terms here, we'll talk about sub-volumes and snapshots here as we get into the material.
But it seemed to be… That the sub-volumes were missing.
And, at the time. The snapshots were… were missing also.
But the bottom line is.
Rub could not physically boot the machine. Because the grub configuration.
Had gotten corrupted. Which was the final conclusion that I came to.
But basically, I had a big brick… brick of a machine.
Which ticked me off tonight. So, when I tried to boot the machine.
It gave me the recovery show. If I can't show it to you tonight, I'll show it to you next time.
It seemed to be missing binaries. Yeah, go ahead.
was that…
A recovery shell, or was that the grub panic shell?
It was the grub menu. And then… Uh, back up. The reco… I think the recovery shell we're talking about here.
is the one you can get from a thumb drive.
Okay, thank you.
Uh, in other words, I was… I'd use the open SUSY thumb drive.
years of recovery shell on that. Or sometimes I'd use this rescue because it booted about 4 times as fast.
Okay. And then… again, the config was broken.
It took a long time of mucking with it to figure out exactly what it was.
We sort of… we sort of… Alright, as the process continued, I've… sort of concluded that the environment was incomplete.
But the end result was… We could not get to a GUI.
At one point, there was a clue that we saw, bad shim.
Which we'll talk about here in a few minutes.
By no GUI, no multi-user boot. That was the end symptom.
of the corrupted boot process. Now, at any point, I could abandon that.
Roll back to the previous snapshot. And boot normally.
And end up with the system I had before we did the update.
So, it's not a linear process here. Probably 5 or 6 times over this year time period.
I would try to do an update. It would fail, I'd end up in the GUI.
Could not boot the system, so I rolled back.
And booted a previous snapshot. To get the system up and running.
And as we'll see here in a few minutes, to do that, that staff shop comes up read-only.
So, you have to either… Roll back to that snapshot.
Which activates that snapshot as the current root partition, then you reboot, read-write.
Or you can play with it directly. And look at the individual sub-volumes, again, which we'll talk about.
Within the Butterfest file system. The bottom line conclusion here.
is that for 8 months, I thought I had a whole system.
Because I could not do an update. Without breaking.
That root partition. And so, after each one of those failed updates, I would have to do a rollback.
Which created another snapshot, and I've still got some extra ones laying around, which I need to clean up eventually, but… I do have to… you know, this is the best machine I've got.
And I need it for… billable hours.
Does that kind of make sense to everybody?
Now, we talked about a grub prompt. Uh, I hope to be able to… I don't know if I have time to show it tonight, we'll see… see how far we get.
But basically, when you get into a Grub menu.
When you get into that black screen that shows.
boot options. That are not what you're used to seeing.
If you read the paragraph underneath it, you will see two options.
You will see an E to edit any one of those lines you have selected.
And it brings up a block of text. I'll show you one here in a few minutes.
Or, you can hit C. for command prompt.
If you hit that C for command prompt. Then you get a grub prompt.
And I've used Grub for 25, 30 years. But I never knew how to do the… use the grub command prompt.
But it's really kind of neat. If you do a C, drop to a shell.
Or drop to the Grub command prompt. The most interesting.
command you can have is a Grub LS. And what that does is it lists.
List all devices that Grobe has detected. Typically, they are disk drives.
In my case, they arrayed arrays. Because the three partitions on a system drive.
Our RAID 1 with 2, uh… you know, solid state drives.
Which is not another issue entirely, which I'm not even gonna get into.
If you wish to. boot from the Grub environment.
You may have to load some modules. Number one is ButterFS, and number 2 is the MD RAID.
Which tells Grub. Or which installs the components within Grub.
To accelerate array. And talk to the ButterFS file system.
rub, set. Allows you to specify what the root partition is to boot from.
This is just an example. Uh, in my case, it was HG0, I believe it was, uh… System MD127, P3.
Or P2, rather. So, this will tell… this will tell you what devices.
The grub kernel is found. And you can try to boot any one of those.
And the other command you have is… the Linux EFI.
EFI is a nightmare that has been promulgated by some.
Notorious vendors over the last 10 years. And it really ticked me off, because.
when I was trying to install on my laptop.
I could not figure out how to get the USB drive to boot. Number one, select it.
And number two, physically boot it. Because it… it, uh… I don't remember what the problem was, but I had to figure out how to boot Legacy the first time.
And it was impossible. So, EFI is the 3-letter cuss word.
in the… in the Linux world, because it always gets in the way.
In fact, on a laptop, I ended up turning it off. I ended up turning off Safe Boot because it wasn't worth.
Fair enough. Thank you.
Surely. Thank you.
there's a four-letter variant that has a U in front of it, too.
Yeah, Eufy. Uh, I think, if I remember correctly, EFI is the Linux module that talks.
EFI, and Eufy is the hard code in the BIOS, is that?
Um, EFI…
Sound reasonable?
Yeah. EFI is the…
E-I-E-I-O.
new counterpart
to BIOS. It's enhanced firmware. I don't know what it…
Yeah.
Okay.
They're extensible firmware interface. And now that you as unified extensible firmware interface.
Okay.
So… We talked about the disc being unusable, unreadable, till the mod… or, I'm sorry.
You can't talk to a Butterfest disc until you can.
Load it, or you can attach it. To attach it, you have to be able to.
build the RAID array, which… is done automatically.
And you have to have the file system driver, Butterfs.
As I kept working on it, I could find.
The Colonel was there on the system. on NIDRD?
was there on the system, all the data was there.
Here's a, uh… leading question for some of you guys.
How would you… boot… a partition like I had here.
Without a bootloader.
with EFI, you can get a command line at the firmware,
I don't need EFI.
Without a bootloader on disk and, um…
BIOS or legacy mode, you're gonna need a bootloader somewhere, either…
on disk or on Flash.
Well, I didn't say I didn't have a bootloader, I said without the bootloader on the disc.
you need it on something else.
Right. And… The key there is… If you boot from a USB drive.
And I tried both the OpenSUSI and the SysRescue disk at different times.
As long as that bootable disk. As… A RAID… RAID module?
And the ButterFS module. You can boot the thumb drive.
Mount the partition you want to boot. Do your bind mounts, bar, dev, sys, proc.
and run. route into the mount.
And you're in the running system.
But it's predicated on the kernel you're booting,
Nice, that makes sense? Yeah.
having the right version and drivers and everything that the running system you're getting to.
needs.
usually it works, except when it doesn't.
Oh, right.
And then you custom, you get a different boot device.
Right. Well, and, you know, like I say, the… I tried, uh, OpenSeries 15.6, or 15.5, it worked, but it took.
I don't know, a couple minutes to boot. And at one point, I was booting and testing, booting and testing.
fairly rapidly, so I started using the CIS rescue disk, which took about a fifth the time to boot.
Which meant I… which meant I could… Go ahead and make a change, come out and reboot, go ahead and make it… reboot, you know, boot with the thumb drive again.
Mount the volume, true. Uh, do the buying mounts to root into it.
Unfortunately. That is going to be hard to show tonight.
Because I have not been successful in building a.
virtual machine where I can enlarge the font on the console.
where you guys could see it. So, I will promise you some demos of that.
Next week. But the bottom line here was, I figured out that grub wasn't the failure.
Because all the components were there. It was the configuration that was… that was broken.
configuration of… wait a minute, what configuration of the file system, or of Grub?
And…
No, the bootable… the boot configuration. Between Grub, and there's multiple.
aspects of that configuration. Number one, you have a grub.config.
which is built with all the boot options. That you should have in the system, and it's basically built every time you make a change.
You have the modules that have to be in the kernel to properly talk to the hardware.
You have to get through the EFI boot process.
with your configuration. And then you have to boot the file system, and once you boot the file, once you start booting the file system, you have to find the Linux kernel on it.
So, so much of this is… is grub-specific, it's not necessarily.
Relating to the ButterFS file system, or it could be EXT4, for that matter, right?
I would consider grub.
to be hurdles that had to be overcome to be able to address his underlying
Butter FS problems.
Grub is kind of effectively its own teeny-tiny operating system that's sole purpose is to…
Access storage devices, read config,
for itself, and chain load other operating systems.
Yeah. Yeah, and I will say again.
Thank you both. Thank you both.
What I learned in this process here. was… was invaluable.
Because now I have some idea. as to how I can test, troubleshoot.
and possibly boot a system manually. If the grub configuration is broken.
Because at one point, I could actually boot the machine.
By running through these steps. And completely bypass the Grow configuration.
The grub configuration is really just running those steps automatically.
In some ways, it's analogous to, like, AutoExec on DOS.
Automatically, right.
Yep. Uh, in fact… Give me a second here. I just got…
Okay, this is Grub Config. On… on this machine.
Let me blow it up a little bit for you.
First off, as the note says, it's automatically generated by Grub2make config.
And I probably use that 100 times in the last month.
Every one of these entries. has something to do with the boot process.
I don't care about them. A lot of them are just templates.
Because a lot of these are placeholders. Each one of these is a file.
in Etsy Grubdee. System D artifact.
And… 80% of these files are blank.
Grub actually has a lot of…
Particularly grub 2.
a lot of…
fluff that isn't necessarily needed to boot any given machine.
But it is…
placeholders for features that a small percentage of people use.
And you end up with a whole lot of scaffolding that is… most people don't need.
Right. And this is a testing.
grow config… And this replaced, in fact, I don't even have the OpenSusi Grub config in here.
I'm using the manual one. This is a custom manual entry for a direct boot.
The menu entry says… On the grub menu, open sushi 515.6 manual.
At… at is a symbol you're going to see all the time in the ButterFS world.
At is a pointer to the current root partition.
In this case, we're… Go ahead. Yeah.
Wait, never mind. Okay. We're installing some modules here.
Here's a very important one, RAID. Here's another one, Butterfs.
Uh, it really… another thing that amazed me. I could start a cyst rescue drive.
assemble… From the hardware.
I'll show… I'll show it to you, but MDADM-assemble dash dash scan.
That tells MDA ADM, which is the. MD Administrator.
Which is there on every Linux system. to dash dash assemble, build whatever RAID arrays you can find on a system.
dash dash scan is scan the hardware for. Any raid flag partitions.
MD is short for Multiple Device or Multiple…
mobile device, okay, I knew it was something simple like that.
In this case, remember I said the boot partition?
on my system was a little bit different. MD stash, NMD slash, multi… multi-device.
System, which is what… What it detected system is the name I gave to that.
RAID 1 configuration. And GPT-2.
is boot the GPT. bootloader on partition 2.
Um, I take GPT-2 in that context,
to be a reference to the second GPT partition.
Uh, maybe I misunderstood, I thought you were saying…
That is what I said.
It's a reference to a hunt loader.
Well, good… Okay, well, GPT is the partition format for this device.
And number 2 is a second partition of that partition table.
Yeah.
Yeah, if you… if you still had classic, uh.
Yeah, I better understand.
MBR format, and it's a GPT, that would say MS DOS.
Uh, if it was classic, it wouldn't be able to interpret it.
Sure can. You just load, uh…
part underscore MS-DOS instead of part GPT.
Yeah, but it's part MS-DOS going to identify a GPT, flag a GPT.
So, he's saying if you were using DOS-based partitioning instead of GPT,
Do you participate.
That would be what would be different.
Okay, I haven't… I haven't done that in 10 years.
Okay. Anyway, here is the… Kernel configuration line.
Linux EFI… This is the actual file name.
On the boot partition. root device.
File system type. Sub-valve at.
No splash screen? No other splash screens?
And here's the initRD. For EFI.
Which is the other file in… in slash boot.
Mhm.
What's the, in it or. RDIF I.
So this is the…
Uh, I don't know if I can explain it right, but I think of this as the kernel.
Right.
This initial kernel, and this is the… System D initial.
Oh, RAM disk? Okay. Thank you, Phil. Nice.
an initial RAM disk.
Thank you. Yes. Okay, makes sense.
Now, what really freaked me out. is in the grub menu.
I could hit C for command line. And I could build this.
Well, Bill, this at least. in the grub bootloader.
On the command line. And physically bypass the entire growth configuration by typing it manually.
Yep.
Okay. Go ahead.
No, I'm agreeing. It's like, yeah, I've…
That's… I guess you've not had that, um…
Shall we say pleasure before.
Welcome to the cloning, I'm sorry.
I had… I had not, no.
I would think are good.
And to be up. Yeah, go ahead.
A good analogy. I think a good analogy would be.
Just like when you're at the command line on Linux or whatever, and you type out separate commands and do them.
And you get tired of doing it, so you make a little script.
So, in effect, you're doing the same thing on that.
The context is in… Uh, grub, as opposed to, you know, the Linux command line.
Yeah, except for one critical factor that's missing on growth.
Depends…
Tab completion.
I've actually had tab completion on newer versions of Grub, particularly looking at file systems.
That's true, yeah.
Yep. Maybe it's an optional feature and not all distros include it.
Really?
I was floored when I saw it.
Yeah.
Yeah, tell me about it. Okay, any other questions about A?
I was also extremely grateful when I saw
We're a boot configuration.
So, you're using RAID…
outside of ButterFS, and I thought one of Butter's claims was it had RAID capability built in.
is… are you… so you… were you using external?
Raid.
Uh, the RAID is built from the physical drive.
And the partition you build… filled with RAID is formatted with Butterfs.
Okay. I…
prior conversations, I thought you were leveraging
what I understood to be RAID capabilities
inside of, or part of,
ButterFS.
No, no. Uh…
Hey, I'm waiting for the other example to go on.
I'm not sure why… People would say that.
Okay.
Because… There are a number of features within ButterFS, which we'll talk about.
But I'm wondering if people are confusing snapshots. With rate, in terms of… You know, once you create a snapshot.
You have permanently flagged a particular section of that disc.
That can never be changed.
Does that seem like an analogy to you, Grant?
Um, there may be other people making that, and that is not…
what I was talking about.
I'll try to find some documentation for it. Go ahead.
Okay, okay. Sure. All right, anybody else at this point?
Now, so… Once I started booting and rebooting and trying to troubleshoot.
I started looking at. What snapshots were on the machine? Now, this is a good.
Good point in time to define a snapshot. I… mentally picture in my mind a snapshot as being.
X number of blocks on the discs. That are tagged with a special.
identifier. And are… can no longer be changed.
You know, you cannot change them unless you mechanically go in there and.
Uh, change this… change the structure, or whatever you want to do it, but the OS cannot write to that snapshot.
In fact, that snapshot is not within the OS's.
mounted data. Unless you physically add it.
And this gets back, this… gets into a point we're going to talk about here in a little bit.
The cow principle. Again, we'll define it, but here's a good time to introduce it. It's called Copy on Write.
This snapshot… could be used by the OS in a read-only state.
It might be… I… you'd have to look at exact configuration.
If that is true. If a change is posted to this snapshot, or the data in a snapshot.
It never gets in the snapshot. But it is saved in the active partition.
As being a change. 2… the previous snapshot.
Copy on right. Instead of writing to the original.
snapshot, you… Copy that data to a working read-write snapshot, and then link them together.
Nice. Yeah. Nice.
Okay, let's, uh, try a couple exercises here.
Butterfs? Volume… flash.
First, this is the… one of the basic commands, like, uh.
ls space slash. Sorry.
This is my working workstation. Or my workstation I'm using right now.
This is the ID. For this particular sub-violume.
Now, sub-volume is not a snapshot.
A sub-volume is a logical grouping. of blocks on the physical device.
Which has a unique identity. And they are identified by the sub-volume ID.
26, 27, 258, 9, 60. Yadda yadda. Uh, I think… yeah, I'm up to, like, close to 500.
460. 469 means this is the 469th snapshot that has been created.
on this Butterf's volume.
Here is the… Active root partition.
Identified by the ad.
Notice the 5. Subvolume 5.
is the… sub-volume that is pointing to the current.
Or a root partition. That is a physical location for that definition to be stored.
Notice that I got a bunch of garbage volumes down here.
You know, 9.02, that's a heck of a lot cli… uh… newer than 256 or 257, rather.
Now, if you look at this for a minute.
You'll actually notice a few things.
You know, number one. Var… UserLocal.
Temp, route, opt. And a lot of these here.
Including the original dot snapshot. Sub-volume.
Look at their IDs.
256 is the first sub-volume created. And that is… or sub-volume 5.
The ID is 256. It is sub-volume 5, index-wise.
These are still… Use sub-volumes in use.
from when this disk was built 2 years ago.
Does that make sense? I gotta think about that. Yeah, think about that.
These sub-volumes, each one of them identifies. A collection of blocks.
on the RAID 1. That physical collection of blocks has been in use since this machine was built.
And it is uniquely identified by the VAR tag.
Because that sub-volume. When you boot the system.
is mounted on slash bar. just like a network drive, but in this case, ButterFS is handling it.
Because any changes, uh… Okay, back up. A snapshot would not normally be used.
In an active file system.
I would liken your VAR and user local and temp.
to be very, very, very similar to if you use separate partitions,
or separate logical volumes for those file systems.
you're using a separate
Right.
container inside ButterFS for those file systems.
Correct. In fact.
So with that in mind, it makes sense that your
VAR file system,
Today is the same container,
from when you installed the system.
Unless you actively replace…
that file system.
Mm-hmm.
it… to me, I would expect it to be the same, just having contents changed,
as time progresses, but it's the same file system.
Right. Yeah. Here is another, uh… Hardware.
It's just the same file system with the same sizes and parameters for permissions.
But the data inside it can change.
Okay, a sub-volume by itself. Up here, 92.57, uh, I'm not sure what the generation is.
that looks like an inverted tree.
Top level, uh… anyway.
um, ID…
257 is…
mounted on VAR, but its parent is ID256.
Likewise, for a number…
Uh, that's right, that's right. This is the parent. Thanks, Grant.
Yep, but if you get down, like, look at your snapshot slash 418,
its parent is 265,
Right.
Which is .snapshots.
Right. Okay, so where I was… we'll go to answer your question, Stan.
This sub-volume… is a logical isolation of X number of blocks on the disks.
Mounted on slash R. And so, tell me what the, uh… What's… what's the big advantage of having that?
Anybody off the top of your head?
Well, uh, what's the relationship between any mounted. Disc, and the disc is mounted on.
with traditional partitions and file systems,
For mounted at.
you provide some isolation,
So, like, it's really hard to put more data on a file system than the size of the file system.
Um, so…
But true, but I was looking for the isolation.
Yeah. Uh…
You know, that gives you a level of security.
and isolation between any sub-volume. And the parent. Yeah, go ahead, Grant.
Security in what sense?
Well, a common thing that I see is root…
Oh.
will be mounted with options to allow device files to exist.
But home will be mounted with, like, no dev.
option, so people can't put a surreptitious device file
in their home directory,
and do something. Or if you have an archive file system that is just raw data,
you could mount it as no exec.
So you can't execute things off of it, but you can access files, read-write.
Um, it gets into various mount-level options.
You can apply them to different…
different options to different directories.
Okay, I understand all that, but that it comes in my head, it comes back to the question of why do I do it this way?
Um…
Security is one thing. Another option has to do with snapshots.
Like, I very common have…
application data live on its own?
file system or data set.
kind of, like, opt, hypothetically, for this example.
I can snapshot
opt independently.
once a minute. If I want to, but I may snapshot the rest of the system.
once an hour or once a day.
I can roll OPT,
back…
to whatever snapshot's there.
So I may want to do snapshots.
at different amounts of time. Anything that might motivate you to have, well, I want this directory
to be treated differently
than other directories,
that is probably a good opportunity for a separate
file system or data set.
Yeah, and the other… the other aspect of that is.
root as a separate sub-volume. is you can up… you can… the root directory.
is you can update the root partition all day long.
And never touch anything in… That subway.
Which are r slash home, but the slash slash home is not listed here.
Uh, which means that whatever data you keep in root.
We'll still be there. Even if you blow away.
the system.
Because in testing this. I replaced root.
50, 40, 50 times. But I always had the data in the sub-items.
Some of this gets into more…
lifecycle management of data storage.
at a block-level
or file system level, instead of, yeah, I have copies of my data 8 different places,
It's kind of…
I don't want to say enterprise-y type activity, but it is an activity that is probably atypical for your average home user.
Lee, you are not an average home user.
I have to…
Oh, I often think of snapshots and things like that as a recovery point.
Yes?
Well, that's what the snapshots are for.
Is that what all this stuff is for?
But…
The other?
Let's say I'm getting ready to…
I mean…
try to compile X11 or something else that's gonna explode all over the place.
And I want to contain it into something and sequester it,
So I could create a new file system
PDQ do all my work,
in there, and then when I'm done, I can just destroy the file system.
And I get all the space back, and everything is contained in there.
Does that make sense?
Kind of, but I… I'm still having trouble grasping why I'm doing this.
You have to care about using some of the options provided.
Just like some of the networking things I mentioned in the base,
there are things I have done most people
have never had a desire to do, and some that know about them say, I want to never do that.
But I do them. And…
Butter FS provides a lot of features
that many people
Just load the OS installer did it, and that's what I have.
Um, somebody like Lee and myself,
plays with a little bit more, does more things…
Um, I don't know if Butter can do it, but ZFS, I can…
transport…
a given file system through a snapshot or unmount it and transport the file system.
to a file or through a pipe to another system, and import copies.
Almost…
like database journaling,
create on one side, transfer the journals,
import on the other side, and I have a point in time
copy of the source.
but on the destination system.
How quickly can these… these snapshots be created?
It's instantaneously.
Yeah, like, order of seconds or decimals thereof.
Because all you're doing is selling ButterFS to take those blocks.
And freeze them.
And… and…
Um, yes, Carrie, you can do exactly what you are
Yeah.
suggesting you can take a snapshot
Mount that snapshot…
Which is read-only, back that snapshot up while your otherwise using the system, making changes.
to what the snapshot was from.
So, like, I could…
Hi. I guess this would be for systems that… are processing 87 million.
processes per second, or something like that.
Or maybe it's processing 50 things on a slow, first-generation Raspberry Pi.
I don't think you run better about our fish on a first-gen pie, it'd probably take Gen 3 at least.
time.
Fair enough. I…
bad example, but point still stands.
Hmm…
Capital. All right, anybody else?
Well, it should turn out better than I thought.
Okay, so we looked at a sub-volume list, we went through it in detail.
Anybody on the system can do a sub-volume list, just like, uh, mount command, or… oh, I was going to mention.
This is… the favor command I learned over the last 8 months.
I'm sorry, I'm sorry, wrong one. Right here.
Fine Mount. Instead of looking… pawing through a list.
The 50 mounts… Which are all kinds of sub-volumes and Docker can, you know, all kinds of.
Will you do a quick test for me?
doofus like that, pardon my French.
do a DF space slash bar.
See you again.
slash var.
Okay. So, it…
DF shows the device, but it doesn't know the ButterFS specifics.
and find out, we'll show those. Thank you!
Right.
So, as far as a mount point. MD127. By the way, Susie likes MD127.
Every time you create a system. RAID array, multi-device, multiple device array, it's MD127.
And notice that it is… At… What's the add again?
referenced current, isn't it?
Yeah, so ButterFS's root rotation, right?
It is a sim link. to the current Butterfs root partition.
And if you think about it, that's pretty important because.
With a mount like this. I can swap root partitions all day long.
and have the same varsal biome. Without changing this configuration.
What does the, uh, NCFS tab…
for VAR look like?
Uh…
Okay.
Thank you.
All right. Here is another Butterfs.
Save your bacon. More than… Multiple times in working on this, I would have to go in and.
Physi- look at the physical. configuration.
Of the drive, or of the disk.
And you do that by mounting. Sub-vol ID5, remember, that is the physical sub-volume of the.
root partition. on a device, or which is that device.
In other words, that's the sub-volume, this is the volume.
on a mount point. And then you can see… Physically, what sub-volumes are in there.
Let's grab that. So I start with that.
I'm okay for here. So what goes here?
Is that where you're at sign goes?
No, that's a, uh, NVMe zero.
And be your device containing the…
Oh, the MD127?
Right. And in this case, it's P2 the same.
On M&T. And of course… I need a pseudo in front of that.
These are the… root partitions.
That have been defined in ButterFS.
Can you say LS space dash L?
No, I can do LL.
You know, November 22nd. is, uh, is the last time I changed it.
And these were all ones that got broken. Or ones that were broken.
Now, I did find out something interesting. If… you know, uh, I tried at one point.
to rsync. One snapshot.
So, volume, to root. Sub-volume's updated.
Unfortunately, it didn't work. I found out the hard way.
that our sync will not copy. a sub-volume mount point.
Because it has special attributes. You know, very similar. In other words.
if I… if I, uh… How can I do that?
Basically. VAR is a sub-volume mount point.
to the user, it appears to be a normal directory.
But there are extended attributes in there. I don't… I don't know what they are.
But I know that if you try to r-sync.
amount point… It won't work.
Was stat show you anything? S-T-A-T?
Um…
Uh… not sure. I'm not sure if any of this… Grant, have you ever looked at any of this stuff?
Um…
I am taking that output
to indicate that stat…
is ignorant.
to ButterFS. Looks like stock.
standard POSIX
Okay.
Yeah, I mean, this is all base… base perms, uh… Yeah, and notice this is… I know 256.
Yeah.
Right.
Has one link to it. Yeah, and… But anyway, that is one of the lessons I learned, and I'm just.
Passing it along. You know, because at one point.
I tried to take… A… the broken… Sub-volume. After the update.
and R-Sync it to the old one. Which, which I was… had rolled back to.
And I got the data, but the sub-volumes wouldn't mount.
So, I think that is… Uh, you know, one of these broken ones down here.
I haven't purged these yet, because I want to be able to show them to you.
Alright, oh, we were gonna… I was gonna do this, yeah.
do the sub volume out.
And those are the… are the ones that are… that remain on.
This physical array.
And technical… or… From a functional standpoint.
I should have cleared all these and just deleted them.
But… I wanted to leave him here, so, you know, you guys could, uh… See the trail or the traces of the case may be.
Do you have an understanding of what is broken within this system?
Uh, yes or no. Uh, the route… Broken.
issue is that if this. Root sub volume is broken.
It won't boot. Now, you know, for example, this…
I take it that it's something that you renamed to read it as broken?
Okay, yes, I… instead of deleting this. Um, I renamed it, and then created a new one.
Alright, you renamed it.
Okay.
Or create a new… a new SIM link. And, you know, that was, like, a month ago.
And I don't remember the exact steps.
But these are all 3 phase… these are… these 3 are all failed.
a couple of… Would not boot?
Because the grub configuration was torched. And… and we'll see what exactly happened here later on.
Uh, at least one of the other ones. was one I had tried to fix it by our syncing from a Google.
partition, or a good volume. But that broke this one because.
It rsync to slash var. And so then, when I tried to boot it, slash bar wouldn't mount.
FC… FSCK wouldn't do anything to it?
No, there's nothing wrong with a subway. The mount point itself, the slash VAR mount point.
like, slash MNT. Did not have the special attributes that told ButterFS that it was.
a sub-volume mount point. It's like if you… and, you know, I mounted… I mounted this on MNT.
Which is a regular directory. But… slash var, slash opt.
The amount points… for each of these sub-volumes and AND slash boot.
for these to be legal mount points. per sub-volumes in ButterFS.
They have to have additional attributes.
So, you cannot mount a sub-volume. Butterfesh on a directory.
It has to be a mount point created as a sub-volume mount point, the way I understand it.
Does that make sense? Kind of.
But it surprises me.
Kinda.
Like, if you unmount MNT,
and try to mount…
Instead of subval,
5. If you try to mount SubVault 256,
Do you see the contents of var?
No. Because… Uh, 256 includes a reference to VAR.
257.
Not 56.
For good. Oh… Yeah.
So, ID257 is VAR.
Mm-hmm.
Right. That is the VAR sub volume. That is not the mount point.
The slash, uh… This root alias.
points to…
Uh, which one does it point to?
I don't… I don't remember how to show that.
Uh, you know, but basically, the root subvolume. includes a slash var directory.
Yeah.
But to mount the VAR sub-volume. on the slash bar en route.
requires that. That directory in the roots of VAL have.
attributes that tell ButterFS it is a valid mount point.
Interesting. I'm surprised they're stored
on the directory, rather than…
as part of the subfall. Uh, ZFS does the latter.
Okay.
Well, it's… I don't know that they're not. All I know is that if you.
Create a directory. en route, and then try to mount a sub-val on it.
Unless that mound point has been created for a sub-valve mount, it won't work.
No, no, and I know that's kind of weird, but… I'm just trying to pass along stuff I picked up over the last few months.
Fair enough.
Okay, so we looked at the sub-volumes. The point here is… When I was doing the troubleshooting.
Everything was here. And I tried about 15 different.
Ways to do it. And I had my trusty helper give me all kinds of rabbit holes to go down.
Which sometimes is not fun.
Okay, so… The… eventually.
I got an error. Bad shim signature.
What is a shim? A shim is a connector.
That physically loads the kernel. The reason I found out was that the kernel verification was booked.
Because it never loaded, and it already never ran.
And that was what actually broke the boot process.
Because of that, and NIDRD could not mount the file system.
And it launched the recovery shell instead.
Recovery Shell, in this case, had no tools, no drivers, no modules.
Because it could not… Pull them from the file system.
So, during the testing, first test. Boot from assist rescue are Open SUSI.
I could see the physical drives.
Once secure… once Secure Boot was disabled. I could see him in grub.
But RFS would mount, snapshots were shown, sub-volumes were correct.
The lesson there was the hardware was alive, file system was okay, so… It was definitely a configuration issue.
Another test would be to manually. Boot?
And I could. I could run like that.
By manually specifying the kernel. And basically.
The grub boot lines we were looking at before.
Putting those directly into the… Where am I?
Oh, that's right, they're on a slide a while back.
Um, remember I… remember I showed you how that you could actually.
hit C on the grub menu, get a command prompt.
You can enter… the four parameters you need at the Grub2 prompt.
And actually boot the system. And that proof that the system itself.
Would boot normally. Okay.
This is starting to feel like Secure Boot.
No, because, uh, on my desktop. I did not have to disable Secure Boot.
It was my laptop I had to disable Secure Boot because I couldn't get into the freakin' BIOS.
To boot a thumb drive.
That, uh, that sham error, I think, is related to Secure Boot.
Yeah, and by secure boot, I'm referring to…
cryptographic signature verification,
And I could see not necessarily what devices you're allowed to boot or not.
Um, and if a signature failed,
It's natural response would be to refuse to load.
the next binary. And when you are running the commands, grub commands, manually,
your only using the ones…
needed to bring it up, not the extra scaffolding.
to do signature verification.
Yeah, I think that might have been the case.
Yeah.
Yeah, and your secure boot shim.
your update changed the kernel, it updated the kernel,
But for some reason, the SUS update process did not update the signature, so subsequent boots
Looking to verify the signature of the kernel, well, it doesn't match. The kernel has changed,
Noble for you!
Yep.
That was it.
And when you say recovery mode is you were dropped at a grub prompt?
Correct. Yep.
That's grub having bailed out. I can't do what my config file says to do.
So… pout! My arms are crossed. I'm not moving.
In the interest of sanity, Gary. I'm going to suggest we push the next one.
to actually… I was gonna say, was this the slide deck for… Better FS500? No, this is 100. This is what? So 200 will be next week, and then we'll decide what to do with 500. Yeah.
So… I think that's probably a wise decision.
Very interestingly, thank you.
All right, anybody else?
Oh, definitely. At least they got a working system again.
Yeah, and I have been in the situation of, oh my god, my main workstation that has everything is not working.
Oh, thank God my dad is there. Now, why in the hell isn't it booting?
Yeah.
EIEIO.
Did you ever… did it ever make you want to go get a drink?
Been there, done that.
Oh, more than once.
Bullet holes right over the, uh…
No, you might miss and damage the data that you're trying to get back.
Uh… how do I stop sharing?
You stop sharing like you uninstall McAfee.
Shoot the thing.
You have the floor. All right. Thank you very, very much, and I, of course, was.
Only half jesting about with these slide deck for the inter… a better FS500, uh, because yeah, this definitely, uh… Uh, dug in.
I think more than, uh, most of us expected for the 100 level.
But it was excellent. Yeah, well, you know, realize the 100 level is what I did to isolate the problem and solve it. And then next week, we start going back to fill in.
The wife, okay.
Very nice. Unless somebody comes up with a better offer, we confer.
Um, quickly polling the crews. I realize there's some people who are.
Yeah, their Wednesday nights are free, other people their Thursday nights are free.
Um, are any of you upset if we, uh… Pick up with, uh, this and go to… Better FS200 next Thursday, that's 8 days from now.
Or… are any of you screaming in horror, going, no, I can't make a Thursday night meeting?
I don't care.
And then nobody else should care either. We record this stuff, like, it… You know, they don't show up then, they can watch it on the recording.
Well, it's, of course, nicer if you're actually here, and you can say, wait a second, that doesn't work that way.
I know that that does make a difference, yeah, that, you know, it's nice to show up so you can ask questions, but.
Um…
But anyway, I'm not hearing anybody object and say, no, no, please don't do this on next Thursday, so… I'm thinking we probably will.
Uh, move forward and go to better FS200, uh, at the Thursday.
It's not like we're gonna get a better offer, Gary.
Well, you never know. But, uh, like I said, here I was looking more just to see if anybody here tonight, uh.
I was afraid they were being left out if we moved to Thursday night next week, so…
If you don't mind me asking, how, uh, far back… how many recordings do you have of meetings at this point?
Uh, pretty much since the beginning of COVID, uh.
Managing is substantial.
We've, uh… I think… And Stan might be able to correct me. I think we have every… meeting since, uh, somewhere in the year 2000 recorded.
We got four brothers.
I think we only missed one. Out of that whole mess.
So you… so if you do one, uh, every week, cause… That, like, 250 or so.
Uh, well, we really only record 2 meetings a month, and uh…
Zoom now. Okay, so 24 times… For about almost about 100, or in the vicinity of 100 meetings.
Yeah, and we did do some spot recordings prior to that in previous years, but uh… Nowhere at this level of consistency, so…
Cool. Cool. Do you have any, uh… Any, uh, recordings on, let's say, drive imaging?
Define drive imaging.
That's something I'm working… or perennially working on. Well, clones… well, Clone Zill is one, and then other open source tools.
Don't forget Data Destroyer.
Yeah, DD, yeah.
Oh, I've been through all of them, I couldn't… I can't remember.
You know, go to our presentation archive page and do a search and see what you get on the titles.
Okay, alright, thanks.
And sometimes the title's really… are deceptive because the titles, uh, you know, mean one thing, or one thing, and then you go in and there's maybe much more depth than you expect.
Okay. Alright, I'll have to check it out. Okay, thank you.
Good questions.
Dead air.
Okay. Alright, well, um… Well, if there are any other questions, uh… germane and specific to tonight.
We'll move on past that. Uh, as we said, I think we're going to move forward next Thursday and go to.
Uh, better FS200 next Thursday night. And, uh, um… So I think we've covered both of those two topics.
That gets us to general Q&A, just to know, hey, what we're working on earlier this week that broke and no longer works, or how did you.
What marvelous thing did you stumble on that fixed something this week?
Uh, if anyone has anything they would like to, uh.
I'll put up on the table, and… Tell us about, or ask us about.
This is the chance.
I don't think anybody wants to know about the VPN hell that I've been dealing with at work.
I did, uh… run across screennotes, which is a… not screennotes, um… What's it called?
Um, give me a second here. It's called, uh, speech notes.
Which is a speech-to-text recorder that seems to be working really, really good.
I haven't been able to really verify that. Plus, it ran me into some really weird and nasty problems when I did the install.
But it seems to be something that. I… provides a… opportunity and features that.
I really want to get into later on, but.
I do not want to do any kind of presentation on it.
Period. Nope.
How about… how about a tutorial?
Mm.
Just letting you know that there's something out there that seems pretty good.
take a snapshot first.
Mm-hmm.
Do it your own risk.
Yeah, that's for sure. Yeah.
And Stan? Not with your brown… not with your brownie, please.
All right, unless anybody has a better idea, Gary, how about a book in?
Hmm.
Okay, last chance. I heard somebody start to say something?
Wait, wait. Yeah, it does!
Wait, wait. Wait, wait.
Does anybody… is there any other Rust developers in the meeting today?
Yeah, it's… it's… I'm getting back into it after maybe a year and a half, and I forgot everything. I mean, literally everything.
Except… except for the… how to write a function. I could do that.
Okay.
That's all the other stuff, right? Anyway, so I have, like, everything in one file, and I want to split it off into two files, like, separate files, right?
Um… and… That's… tricky, I guess.
I…
You know, for example, like, you… you have a… your main… your main.rs, which has your, like.
Main function in there, and then… You have, like, this structure… And… it's, you know, in the top of that file.
And you keep adding… Methods to that structure.
And then it keeps getting bigger and bigger and bigger and bigger, you end up, like, 300 lines down.
It, you know, it was the top of your main function.
Right? And you want to take all… it's all a single structure.
Mm-hmm.
So you want to take that whole thing and put it in a separate file. Now, in, like, C.
Right? You could just create a header file. And then, uh… a separate .c file, and then in your main file, you could just include that error file, right?
But in Rust, it's more complicated. You have to use modules, and… Uh, you have to worry about.
Hi.
isn't the crude…
Visibility and things like that. So anyway, I'm just asking.
really a preprocessor directive, not…
Mark?
a C language feature itself.
Yes, you're absolutely right. It is a pre-processed director.
I would want…
I should also mention that the preprocessor, right. Works at the tokenizer level, right? So, if you think about a compiler.
It has several phases. There's a tokenizer, a parser.
You know, a type checker, you know, an intermediate language.
And then a code optimizer, and then a code generator, right?
And… Uh, the first phase is that tokenizer part, right?
And if you look at GCC, and you look at its preprocessor.
It uses the exact, and I'm telling you, the ex… even though they're separate processes.
It uses the exact same code. for processing the, um… Uh, uh, the preprocessor directives.
Uh, like, defines and includes statements and. If in-depth and all that kind of stuff, right? Uses all, that's the same tokenizer that the compiler itself uses for the CE language.
That's how it's able to… to do funny things with, like, macros and stuff.
Um, but the… there's no such thing as a preprocessor under Rust.
This doesn't exist. So I was just asking if anybody had any experience.
I…
So, why do you… why do you want to split it up?
Sounds like your classic…
It's getting big. The file's getting really big.
I just want to move this chunk of code into two separate files, or multiple files instead of one file.
and aggregate it.
Um, I…
Exactly. Exactly, right, exactly, yeah.
Is there a reason you can't use a preprocessor, even if it's external to the Rust compiler? Like, if you…
used M4 or something to combine your files right to standard out.
And have the Rust compiler read standard in, or something like that?
Hmm. I suppose that's possible. I mean, it, uh… I wouldn't call that idiomatic rust. I mean, if, you know, the idea here would be that.
If you had some code, and let's say you wanted to put it up on CodeBerg, which is like a… Open source replacement for Git, GitHub.
Um, you know, you want somebody to be able to download that, and you don't have to… You have to say, well, in order to get this to work, you know, you gotta have M4 and write this kind of crappy thing where you go through the Rust…
Standard in and stuff, you know, so… I just want to do it the rest way.
Um, then I just… you know, I got… I got a version working in a separate.
Ah.
In a separate directory, just like a test project.
Um, but I… I don't like it.
Anyway, I just asked if there's any Rust developers, so…
Um… Ed, I just put something in, uh… Uh, the chat, uh, the… the St. Louis Rust.
Group, uh, has… had a rebirth, uh, they started meeting again.
I believe it was last year? Uh, there's a gentleman named Nathan Tippey.
Really?
Uh, and I believe he teaches, I think, over at SIU Edwardsville.
Uh, and he teaches the Rust course. Um, so anyway, he's… Got this going. Obviously, the fact that no one else.
In tonight's meeting here has spoken up. I presume none of us… none of these folks have.
Regular meeting. Uh, they meet once a month. Uh, they were nice enough at one point when it got reborn.
Uh, somehow they started meeting. On the third Thursday of the month, which is, of course, the normal meeting over the log here.
Um, so, uh, when I kind of go to them on that, Nathan was nice enough to.
Move the meeting to the first Thursday. They pretty much only do physical meetings. I don't think they ever… I think maybe once or twice they try to do a.
Uh, an online version, but… Um, but they do, uh… had that meetup. I put it there in the chat.
Um, so… Uh, I would think you could go out into meetup and go into the.
To discuss the… the Q&A part of the meetup.
And, uh, you know, post questions and just see if anybody from the St. Louis Rust Group might have an answer.
Okay. Yeah, I've got sort of, you know. So, you know, scoured the… Various resources, Google, Stack Overflow, and… One thing is, uh.
Uh, getting the AI answers right. Uh, actually type the code in and did not compile.
So, that's… I keep having… Failure after failure after failure after failure after failure.
With a, uh, very rarely does it work. So, anyway, that's just me. Okay, thanks anyway. I'll check that out. Thanks, Gary.
All right, sure.
All right, anybody else? All right. If there's no one else.
A big thank you to, uh, Grant, uh, that was a wonderful, uh.
Okay.
Uh, uh, tutorial tonight, and Lee… Uh, obviously, yeah, that was a… Oh, heck of a 100 presentation.
Uh, and, uh, thank you all for coming tonight. You all asked excellent questions, made grim.
Great observations, and uh… You all contributed a lot to the meeting. Our bookend, this was Wednesday, the 10th of December 2025. This was the.
Uh, general monthly meeting of SLU, the St. Louis Linux Unix Users Group.
And, uh, our present centers tonight, uh, Grant Taylor did the, uh.
what is a router?
Uh, the intro… Uh, tutorial, and uh, that was on… What is a router?
And, uh, Lee Lambert did the presentation on the Better FS file system.
Uh.