Oh, I guess you gotta… Welcome. This is the St. Louis Unix Users Group.
Virtual meeting… Remote meeting for… September 10th.
2025. We're gonna start out with a basic presentation by Ken Johnson.
Kenshin? Uh, excuse me. Ken, go ahead and introduce yourself and go right into your…
Presentation, if you would, please.
And start by sharing your screen. You are muted at this point.
You have to go down to the bottom for the kick panel, and… Click on… That's good, yes.
Thank you. This is better. Alright, let's see about sharing the screen.
That's on there.
We see it.
Okay, great, great. So, this is the tutorial for September 2025.
I'm going to… Give an overview of 4 different Android apps that I've used for network troubleshooting.
Along the one, uh…
Put that out of the way. I have one bonus app.
Oh, for goodness sakes. How do I get rid of that?
Well, at least that works. So your work as an employment consultant, performing system and small network administration.
I sometimes write technical documentation for people. My cell phone is a Google Pixel 3a, which was released in 2019.
I purchased an unlocked refurbished unit in 202. Will have different results.
Um… yeah. So, um, some things we're not going to do.
I'm not gonna give you any demonstrations, but I will show a few screenshots.
And there's not going to be a comprehensive evaluation of all of the apps in the particular categories.
Um, I'll show you things that have worked for me. That doesn't necessarily mean that I have found the best possible.
So… Uh, someone has the capabilities of that app.
We'll talk about what I use it for. And then I'll show you a screenshot.
But first. This is a picture of a TP-link USB-C to Ethernet adapter.
If I connect this to my phone. It gives me more phone than internet interface.
This turns out to be really hopeful for certain kinds of network troubleshooting.
I have lost your screen share. I don't know if it's my client or what.
It's still okay on our end, or on my end. Okay.
Alright, alright, so… I'm not claiming that there's anything special about this particular adapter. Again, this is a case of…
I want to go down here every little possible one. This one works for me.
So, these are the four apps that we're going to talk about.
Wi-fi analyzer, you'll note that Wi-Fi Analyzer is unknown. And we will get to that.
To those details in a little bit. Network tools, from working electric and PCAP droyed by…
Manuel Feranda. So…
Provides network information, uh, basically. What's the network you're connected to? What's your own IP address? What's the IP address?
Of… in order, what's the IP address of your DNS server?
One got assigned by GHCP. And that's the thing that I use this for the most.
Um, we are typically, and most frequently used in this particular app.
Is to either. More network ports.
Are they working? Are they connected to the correct network? Or, in a situation where somebody says to me.
More worried connection isn't working anymore. So I can walk over there with a phone and…
Um, that network adapter. Can I turn up my phone to the network and say, hey.
What's going on? Um…
Only at least one occasion, it turned out that. A pretty recently installed port.
Had failed me to be, you know, wired over again. I… I don't know what the electricians were up to that day.
Um… Every little case, it turned out that, uh.
The dock. For their laptop had failed.
At least… all the little parts of the dock were working, but the ethernet.
And failed. Some of those ones with landsca, so it will basically scan its…
You know, the network that it's connected to. For everything that'll answer a ping.
And we will show you a list of… Wi-Fi access points. The stars basically indicate.
One of the things that I use the most. Network information, followed by ping, followed by the land scan.
And here is…
An example of the network information. And on this particular case, I did not have it hooked up to an Ethernet address, but…
You are doing Ethernet, but you see the same kinds of information.
You know, shows you the connected type. The IP address, the mask.
All that stuff. And that's, of course, is enough for me to see.
Yes, it's correctly connected to the primary network. Oh, yes, it's correctly connected to the network for the voice over IP phones.
Yeah, certainly, questions are welcome.
Again, can I interrupt a second? Your audio seems to be flat-topping a bit. I don't know if it would help if you could move your microphone away a little bit from your mouth.
Okay. Okay.
Okay, go ahead.
Um… Yeah. So… right here…
So, there used to be… an app in the Play Store called Wi-Fi Analyzer.
And that was my favorite app for looking at. The status of Wi-Fi access points.
In my immediate neighborhood. Unfortunately, it's no longer there.
This does show…
Like, the breadth… of the, uh…
Of the channel usage, um… the channels aren't all independent.
You know… They have a fair amount of overlap from one to the next.
Or at least from one to the adjacent one. And this gives you an indication of that.
How much overlap there is.
So, failing that…
There's what looks like a similar tool. Which… I have run exactly once.
Called the Wi-Fi analyzer by VRAM Software Development. They claim that it's open source.
They do seem to have a GitHub. I… you know, whether that matches what they're actually distributing, of course, is another question.
Um, I tried off the channel graph, it's got a slightly different look.
But it does give you… Similar kinds of information.
And that looks like this. You'll notice they're using a, uh…
Fought-top trapezoidal. Um…
Uh… approach. You can also see maybe a little more clearly.
In this one… Oh, there are some access points that are set up to use…
Bunch of channels, and not just be centered on one. Um…
That one that says hidden, I… don't know anything about that.
Alright.
Network tools? By Hurricane Electric.
Um, it has a nice set of features for looking up.
Dns names. You know, shows you a bunch of the different name types, and you can select which name type you're trying to look up.
It offers ping and traceroute. And it also offers a basic, uh.
Ssl or TLS test. It will attempt to make a secure connection to the.
System and port that you… indicate, and then it will tell you.
What was negotiated? So it doesn't try all the different flavors.
I think it's probably starting at the top. You know, TLS 1.3, working its way down, and then the first one that hits.
That's what you've got. I have not used this as often.
As network analyzer. But it does have some slightly different.
Slightly different set of tools. And it is on my phone.
Um, this is their list. Oh… tools, um…
You can see the DNS, there's a simple IP calculator. Um, I have not tried their IP performance tests.
Well, the Mac browser, I'm not sure how well that would work now.
Um… ping, and then down at the bottom, there's… you can see the, uh…
The SSL, TOS analyzer. Um, that might be…
A bit of a fancy name for a fairly simple test.
But it… it will do that simple test. So that's useful.
And then there's PCAPDroid. Um… this is pretty cool.
What it does is to set up an internal VPN. To capture network traffic from a particular app.
And then it stores it in a PCAMP file. Which is the kind… same kind of…
Saved packet file that's used by Wireshark. So, then if you move that system to a…
Move that file to a system that has Wireshark, you can.
Analyze it, you can look at it, analyze it from… with using the tools of Wireshark.
Um… I just recently came across this.
Um… When I tried it on my own phone.
It worked just the way I wanted it to. It showed me the… DNS Exchange.
And then… and IMAP exchange. Between Mozilla Thunderbird.
And a mail server. Now, I really wasn't very interested in this for my phone.
Uh, the CEO of… The client was having trouble getting his phone to work with the mail server.
And, uh. My remote counterpart…
Didn't really seem to get interested in trying out the… list of settings that I said, hey, these settings work for me.
Until I emailed him the information about. This app, and said, you alright?
We can put this app on his phone. And capture the packets.
Um, so… It worked as a threat, anyway.
Um… And, uh, if we'd had to do it, it would have worked to.
Actually capture the packets. Getting it set up is fairly straightforward.
They have documentation. On… for the app on GitHub.
Um, it's a Markjohn file.
It can do a bunch of other things besides this. But from, you know, the network troubleshooting situ…
For the network troubleshooting case. Which is what I was primarily interested in.
This ability to capture traffic is the big one, from my point of view.
Um… you know, the alternative would have been that my counterpart would have needed to figure out.
What switches… Uh, the CEO's phone was using.
At the main office. And then… find one of those with a monitor port.
Uh, hook a system up to the monitor port. That's all doable?
But, uh… you know, I could have walked the CEO through getting this set up on his phone.
So, I was pretty pleased to find this. Here's, uh…
An example of the setup. The interface wasn't super obvious to me until I went to look at the documentation, but once I did that, it was easy to say, oh.
The app that I want is Thunderbird. Capture…
Save, you know. Save the packets to this file.
And then, you know, once it was done, just… hook that can hook my phone up to a computer, and…
Drag that file over. And then at that point, um…
You know, I could open it up with Wireshark and see exactly what was going on.
So… no. I was very pleased.
Because the next time I have a problem of that… of this kind, this is what I'm going to reach for.
And the bonus, since we seem to have time… This is a programmer's calculator, patterned after the HP16C.
When I came across this, I thought, oh, that'll be really handy.
All have, you know, a hex-octal decimal. Binary calculator on my phone.
And… I got it on my phone, and then I noticed that.
I don't actually use it that much. Um… when… well, the distinction that I make is that I'm not writing or debugging kernel mode software anymore.
Um… So… I'm not looking at system crashes, and…
I don't need a Xcalculator as often. All the same, I think it's pretty cool.
There you go. Your HP-style programmer's calculator.
X decimal, octal, binary. Um…
If you need to do that stuff. Having one on your phone can be handy.
All right. So that was quick, um…
These are the points, in summary, that we went over. Questions?
Comments.
This is Stan, uh… I have a question. I'll…
Yes.
Can you just tell me why… Uh, you would use these applications versus something on a laptop, or…
A workstation? Is it because they're… There's just more convenient, or there's this kind of stuff is not available on… that you could find someplace else?
Or is it just…
Okay, that is a very good question. Um…
My phone is a lot smaller to carry around. You know, even with that Ethernet adapter.
So, if I'm doing something simple, like. Either checking a trouble report.
On an Ethernet port, you know. One of this client that has basically two Ethernet ports.
In every work location. You know, that is every place where somebody sits to work.
One of them is for their main network. One of them is for the voice over IP phones.
Um, and if I get an eternal report about one of those.
I can carry a laptop over there, but… My phone is smaller, faster, more convenient, and it's in my pocket.
So that kind of covers the NetAnalyzer. Things, uh, tests.
Wi-fi analyzer, yes. I think there are probably tools like that available.
For a PC. Um… I came across this on my phone.
Before I actually came across one for a PC. And so I've just stuck with it.
That's… you know, that's a personal choice, I guess.
Um… But sometimes it isn't.
Do you find it's… do you find it's also that it's just a very easy-to-use secondary device?
It's… so you're doing… you can do something on the PC or laptop, and also on your phone, so you've got.
Multiple devices going at once, kind of thing?
Well, so I think there's some of that. And, of course, the other thing is that once in a while.
I will be someplace where I do not have a laptop with me, or my usual tools, but I do have the phone.
And I have this… You know, I have this tool available to me, the Wi-Fi analyzer. If for some reason I need to check out.
What's the situation here? Um…
The working electric tools. Um…
The… Let's… let's back up a little bit.
Oh, let's see here…
Sure, Grant.
Can I add my answer to that question? Um, I haven't had to do it in a long time, but in…
Prior jobs, I have needed to put. Tools on clients' devices, like if their particular device.
Is having a problem at some location that I cannot get to, but we're.
We can meet at a coffee shop or something. I can put the app on their device and train them how to use the tool.
So they know how to do the troubleshooting when they are at the problem location.
So I put the tools on and teach. For somebody to do it themselves.
The other thing I have done with cell phone-based. Is when I have an occasion to need to sniff.
The cellular internet. Connection. I like to see if a cell provider is filtering traffic or something.
That's real hard to get into without putting a capture application on the cell phone itself.
Yeah, so that's… That's not something I've needed to do, but I can certainly see what you're getting at.
Um… the… the PCAP droid would probably do that work.
And there very well may be other things. You know, as I mentioned up front.
These were things that I've found that have worked for me.
And at that point, I stopped looking. Um.
So… The DMS network tools here, um…
That's certainly reproducible. On, you know, the Linux command line, so is the ping, and so is the trace route.
And you can actually do better. Well…
You can take a somewhat different approach to the SSL TLS test in the sense that.
Um… You can use…
Openssl… S underscore client.
And make a connection, and specify a particular version of TLS.
Of course… that will only let you do tests with…
The levels or versions of TLS that your system allows. Um…
But sometimes I think there, you know, there can be a convenience thing.
Um, my… the particular client where…
I use the PCAPDroid stuff. I don't have…
I don't have a Linux laptop there. So…
You know, this is… this is another option.
Question about the Mac browser in what you just showed. A moment ago.
Is that… to look up OUIs for MAC addresses, or is that…
Some reference to, like, the ARP cache. What the phone sees.
Fair enough.
Honestly, I have no idea. Um, there's one of these apps, as I recall, that does have an OUI lookup.
Um… My guess is that that's probably good for…
Commonly seen OUIs… that are at least a few years old.
You know, there's only so much you can expect to get for free.
But this… This was good, because it… like I say, it, um…
If… if my counterpart had not.
You know, decided, yeah, okay, I'll go ahead and try those settings that Ken sent me.
Um… I could have put this on and seen exactly what was going on, and then at that point, it would have been…
Um, pretty difficult to… ignore my recommendations. And then he also understand that solving a non-technical problem with technical means, but…
Sometimes you have to do that kind of thing. Other questions?
Did you ever, uh, expand on the thing about Wi-Fi Analyzer by unknown?
Oh, yeah, so… My favorite is no longer there.
So I can't tell you who it was by, because… You know, I can't go to the Play Store and see that information.
Mm, okay.
And so then… what I can tell you instead is…
There seems to be this. Um… They seem to respond to…
I don't know… I don't know how big this team is.
But they do seem to respond to… uh… ratings and comments at the Play Store.
Um… It seems to more or less work, at least in the sense that.
It looks like… It looks pretty much like what the other one did, other than the…
Um… Square drop maces instead of parabolas.
Um, so… I…
I was completely surprised when I went to the Play Store to look up, alright, who's this by?
Because really, the only one I know of to identify something is the Play Store is… name and author.
I… there may be some other way, but it's less obvious, and I couldn't find it.
So I was going to the Play Store to get the author's.
On all of these, and… Discover Dough, that one's not there.
Which I think is kind of unfortunate, but there it is.
Anything else?
On your, uh, over pixel, are you still… with the original firmware, or did you go with third party?
All still the original firmware. Um, I don't have any secrets on my phone.
Okay.
That aren't protected by very strong encryption. So…
I'm hoping to get at least another 2 years out of it.
Um… So far, it's been working… really well for me.
And at the time that was really attractively priced. Um, you know, buying…
I've had some really good luck buying really nice hardware. That's a few years old.
You know, that also… that also has worked for me for buying laptops.
Um, and I'm… probably gonna buy… I'm probably going to buy a Pixel 6a for my wife.
In the next month. Um, the 3As are old enough now that they're not really available, but…
The 6A is about the same size, and… Um, it's…
Reasonably priced, and I expect it to be at least as good as the 3A.
Anything else? I'm happy to answer any other questions you might have.
How… different is the data you get out of. Tools like this versus, like, having.
Unified toolset.
Well, one difference is that, uh. Pcapd droid?
He wasn't setting… because it's setting up a VPN. There is… IP address translation going on.
So you're not seeing the real IP addresses. For the situation that I need to look at.
You know, that I needed to look at, which was to see.
You know, is this IMAP exchange trying to fetch mail from a server working?
And what's it trying to do? That doesn't matter, because I know…
I know that request is coming from this cell phone, and I know that it's going to… the mail server, um…
That I administer, okay? But I can imagine that there would be situations where that would be not so good.
I think you can also… I think that… application will also support what they call root capture.
So, if you have a rooted device. It can just capture directly from.
The network interface. Without, you know, without having to set up a VPN.
Um, otherwise, for the most part, I guess that I would say that.
These just… These present…
Reasonably pleasant, um… GUI interfaces.
That's different. I mean, that's… it can be quick when you're trying to.
Get something done if you're trying to, say.
Capture the data. Like, the NetAnalyzer is using, oh…
I woke up to this port, and here's the data, and here's the IP address I get, and here's the router that I get, and so on and so forth.
If I want to actually capture that data. Than a command line application is a lot better.
I guess I could take a screenshot. Let's kind of clumsy to work with.
Does that help?
Yeah.
All right. Anything else?
It looks like we're coming up on 7 o'clock, so…
I have a very important question. I have a very important question. That 16C calculator emulator.
Uh, can you use that to compute the winning, uh, Powerball lottery number?
Ensure you have accurate data, yes, you could.
Um… It wouldn't be my first choice, Gary.
Where do you get the data?
Um… I think there would be other HP calculators. That would be… more suited to that calculation.
Or even getting, you know. Closer estimates.
I mean, well, maybe you're not aware, but… Um…
It should be made a series of calculators that had. Roughly this look.
The one you see most frequently for sale at the cheapest price is the business calculator.
There's also a scientific calculator.
Am I taking you seriously enough?
Yeah, I know. Yes, oh yes, you definitely took me…
More than seriously enough. Although I will point out that there was a physics professor at.
Lou, uh, his name. With Jim McGee, uh, and way back, I am talking.
25, 30 years ago. When the state of Missouri first started running their own lottery.
And, uh, he actually did have a formula that. Apparently considerably increased his odds.
Of, uh, coming up with, uh, winning ticket numbers. And, uh, so yes, after the state of Missouri found out that.
Yeah, he was basically beating the curve. They did change how they ran the lottery, and it no longer worked. So it is possible.
With some lotteries to, uh. Come up with a, uh, maybe not always winning strategy, but something that gives you an edge.
Doing random numbers well. Is not trivial.
No. Um…
Okay, and that's what the lottery really needs. You know, to avoid predictability, and doing that well is non-trivial.
Um, probably… 30 years ago?
A high school friend of mine and I. Took, like, 90 drawings worth of.
Some… lottery in the state of Missouri, I don't remember which one, Powerball or what.
But did a statistical analysis. Of all the numbers.
And there was a slight variance, more likelihood for certain numbers than others.
But for the sake of conversation. It was, like, 1.995.
Versus 2.005. So it was a .01.
Spread between the most likely and the least likely number. It really wasn't enough to… make a difference.
You have to… have to buy a lot of tickets to… to catch that small of a difference, so… Anyway, I threw that in just to give people time to think of a truly technical question they may still want to ask Ken in reference to the presentation.
Uh, anybody got anything they need to ask at this point?
Okay, hearing none… I'd like to… from everybody, offer a big thank you to Ken Johnson for doing a wonderful presentation.
A lot of good tools there. Thank you very, very much. We've been hoping to get Ken back as a presenter for a number of months, but he's been tied down.
On a couple projects, not been able to schedule us, but we're grateful that he was able to do it tonight. Thank you very much, Ken.
You're welcome. Good night, gentlemen.
With that, um…
Thank you, Ken.
I still have a Wednesday gig.
Right. All right, um… Let's see. So, uh…
Stan? Is that our… okay. Uh… Okay. Uh, okay, uh, this is the middle of the meeting. Uh, we do our.
Announcements and all those kinds of things in between our two talks. We call this the Oreo Cookie Model.
So, Ken did our introductory tutorial there on the front, uh, did a wonderful job at it.
And, uh, now we're here in the, uh. Gooey middle of the meeting, uh, so I'm Gary Meyer, I'm our MC for this evening.
And I'm also the president of Slug. Uh, see, I think we have… See, Stan, is there a slide with, uh…
The other names on it? Okay, Stan's thinking about it. There we go, okay. Uh, Stan Reichert, he's our Vice President, he's also the man who takes care of sooo much of the production.
Uh, as far as the recordings, the, uh… Uh, changing the slides, uh, so thank you for all that hard work, Stan. And, uh, Lee Lambert, uh, is not here this evening. He's the treasurer of Oregon…
He also hosts a lot of our, uh. Online systems, web pages, our archives.
Uh, on his, uh, company's systems. Uh, secretary position is vacant. If anyone would like to act as a corporate secretary.
We'd love to have you. Please, somebody. Come to the meeting, volunteer. Obviously, the secretary thing does kind of require you to be at a meeting. The chat monitor is something where you're already at the only meeting where it matters, just, uh.
Uh, here in Zoom, there's a little thing at the bottom of the page that says chat. Uh, it's where people, instead of going to their microphone, can just type something in.
So, um, if anyone would like to help out tonight by just.
Eyeball in the chat, and if something comes up in chat that the speaker may need to know about, or may be able to quickly answer.
Uh, somebody would like to volunteer to be the chat monitor?
Please speak up soon. Okay, well, you're thinking about that.
Gary, you should…
Hold on.
Yeah.
Is that Tyler who said you'll do it? Thank you, Tyler, appreciate it. Okay, uh, Topic Wrangler. We always need somebody to help us find, uh, presenters, find speakers for the meetings.
Uh, so, uh, uh, if either you just have some idea of what would be good topics.
Or you have a good idea of who might be a good speaker, or you're just willing to do the arm wrestling and legwork to put the.
Two things together and get somebody. Uh, we'd love to have your help being a topic wrangler. And of course, we'd also love to have your help being a speaker, so anybody who'd care to volunteer, we always really appreciate it. Thank you.
Um, so with that, I guess, uh, we'll pop on to the next slide. By the way, I'd also like to thank Ken for another thing. Ken has been a great contributor.
Uh, he's been the one who's been, uh. Uh, keeping, uh, our, uh.
Meetup account available, uh, so for those of you who find us by Meetup.
Uh, we have Ken to thank for, uh. Organizing and subsidizing that. So thank you, Ken.
Okay, next slide.
Okay, the next SLUB meeting, uh, well, tonight, September the 10th, is, uh, 1. Uh, next one on the actual SLUB general meeting is always the second Wednesday of the month, the next one will be.
October the 8th. Uh, 8 days from now, uh, next week on Thursday.
Is the St. Louis Linux user group meeting. It's basically a subset, a subgroup of Slug.
And, uh, as I say, that's next Thursday. Don't have a topic yet, uh, so if anyone has one they'd like to offer.
I'm all ears, uh, uh. Me offline, or drop something in chat, and we'll get something worked out.
Five days after the St. Louis LUG meeting, uh, is the new Linux users meeting.
Uh, that's always, uh. There's no set presentation. Instead, it's just a thing of.
Whoever considers themselves the newest person to be the user of Linux, they get to ask the first question.
And, uh, um, so it sounds like, well, gee, they should all be simple questions, right?
The amazing thing is, is those of us who are more experienced users.
It really helps if you come to the meeting, because not only can you provide the answers.
I've always found it amazing if you just have a few.
Different experienced users, they usually all come up with a different.
Answer, and amaze each other. It works that way, you can do that? Um, so, uh, um… Yeah, you more experienced people, please come to the new Linux meeting.
It actually is kind of fun. You wind up just… hearing other people's unusual way of doing things, so…
Slack meeting is usually the first Thursday of each month. Slack is a St. Louis area computer club. It's a very old.
Group here in St. Louis, it's now part of SLUG. And, uh, basically it's more hardware-oriented, so you can come in and talk about, you know, you got this PC, your laptop, and it's not working, and this is the hardware, and this is…
Uh, we'll even talk about… we'll even talk about Microsoft there. Um, so, uh… Uh, so, yeah, that's the first Thursday of each month.
Uh, let's see, coming up, uh… October the 14th, that's just about 5 weeks away, people. Uh, it's the end of life for Windows 10.
And, uh, yes, many of you have already been getting, uh.
Panicked phone calls, uh, from your friends and family saying, uh, oh.
Windows 10, end of life? Okay, how do I… I moved to Linux. Uh, so, uh…
Uh, hopefully… We're all fielding those questions.
Um, any other announcements that anyone would like to make at this point?
Okay, while you're thinking, um… We keep an archive of all of our past meetings. Since the beginning of the pandemic, back in 2000 and…
Zero, I guess? 2020. Um… We've recorded all of our meetings, and we actually have all those recordings available, so the video, the sound, the chat.
The slide deck, uh, it's all out in our archives, so go to our archive.
Uh, there's, like I say, almost 60 years of, of, uh, good recordings there. There are… some things in the archive from previous meetings, occasionally it's a full recording, occasionally it's a slide deck, or other references for that meeting, so…
Uh, look in our archives, uh. Our mailing list, we have a discussed mailing list, discuss email list.
And, uh, it's very useful when you go out there, and it's kind of like.
Just at the meeting here, you can ask a question, and it's amazing sometimes how, uh.
Uh, how you can get answers. Um, maybe for just a minute or two here, we'll do a call for help session here. Uh, if anyone doesn't feel like they can stay till after the main speaker tonight.
Um, you know, you can put in a quick question, get a quick reaction here, either a call for help or a quick statement or something you've been working on.
And, uh, we'll see if we can get that answered in a couple minutes before our main speaker starts.
Uh, and… for that Microsoft end of life.
Uh, there's, uh… an interesting book that Stan has there on the slide.
Uh, it's, uh, something that's freely distributed, you can download it.
Gives you a clue of what things you can do, uh, to accommodate this end.
Of Windows 10. You can also go to the bookstore. And pick it up at the bookstore, either as a… Amazon Kindle.
Edition, or a paperback edition. Let's see.
And…
New Linux Beginner Workshop Online. That's what I already mentioned there, that's the new Linux user group.
End of 10 dot… endof 10.org is a project that started in Europe.
And people that have got. Events going on to do, uh, you know, Stall Fest kind of things. They can post it there.
Awful if they want to provide commercial. Paid support, they can list places in the U.S. You go to that website, and it's got a, uh.
Got a map, and you can click, you know, click on the map, and it'll show you the four places in California that have already posted.
That they've got a storefront, or whatever, somebody's home. For 10 bucks, or 20 bucks, or 50, I'll move all your data off of your Windows machine and put it on a.
Or I'll just install Linux over it and not lose your data, et cetera, et cetera.
That kind of stuff. Cool.
Um. Go ahead. Gary.
Keep going.
All right. Um, let's see, uh… Just a reminder, um, although we, as a group, are recording tonight's meeting, as we recorded most of them.
Uh, anybody can be potentially recording this meeting, and… Um, you know, so we might offer to be able to, uh.
Uh, screen out or edit something out of our recording, if there's just an accident and you release some.
Classified information from work. But, uh, if somebody else is recording it, may not be able to help you.
So, uh, beware. Um, let's see…
And, uh, yes, there's simple screen recorder there down at the bottom, which is how you can set that up, set up your Linux machine to do this recording.
Okay, as I said before, uh, we do keep an archive, and all of these recordings are in there. The only.
Uh, well, I should say, the only meetings that we keep a record of are the ones that we have set presentations, and that's the general meeting, Slug.
And the St. Louis log meeting. The new Linux and the Slack meeting are not, uh.
Not kept in the archives. Uh, there's a calendar, slug.org slash calendar. Uh, we try and put things of interest on there.
Uh, when we started it, it was basically our own meetings, and it became meetings in St. Louis.
We know of that would appeal to our… membership, and now, especially since the pandemic, we've really just expanded it out, and any good meeting.
Worldwide, we're just as likely to put out there, uh, on.
On our calendar, just so people can find it. The meetings that we run are highlighted in red, so…
That's how you can find ours most easily. And there is an example of it right there.
So…
Okay. Just a couple of, uh…
Anniversaries on there, for instance, uh… uh… today, actually, it's the 35th anniversary of Archie, the first search engine. We now just think of.
Google as being the way everybody searches, or DuckDuckGo, if you're a little more secure, but Archie was the first one 35 years ago.
Yesterday was the 80th anniversary of the first book. Found in a computer. It actually was a moth that was stuck in the relay of an old mechanical computer.
So, uh… Anyway, a few interesting historical tidbits on there besides the current.
The current meetings. All right, next.
I think yesterday was also, uh, the birthday of Dennis Ritchie.
Um, I'm trying.
Unless anybody has anything else to come. Oh, really?
Are Richie and Kernian still alive?
Dennis? Dennis Ritchie, of course.
That's a good question. I'm not sure. Do you happen to know?
Now, he passed away. Right about the same time when Steve Jobs did, so, you know, all the attention was on Steve Jobs.
And so a lot of people didn't realize it, but yeah, that's when he passed away.
Okay. Okay.
And for those of you scratching your head going, okay, who was that? Um, that's, uh, if you remember.
Uh, the book, uh, by Kernighan and Ritchie. Uh, kind of the definitive book on, uh, C programming, uh, some years ago.
Many years ago. Many decades ago. All right. Uh…
Yeah. Many years ago.
You know, that book is cool. Outstanding. If you look… if you look at it, it's still outstanding.
Uh-huh. Okay, uh, does anybody have any…
You look at it, you go, well, geez, that's not much, a lot of C books are, like, 2 or 3 inches thick.
And this one's a little bitty teeny book. And it's all in there, though, that's the thing.
It's all in there, it's all clear. It's a great way to read it.
Uh-huh.
Great way to learn that. I'm sorry?
Quality, not quantity. Quality, not quantity.
Mm-hmm.
Oh, yeah. Yeah, it's a well-written book.
Oh, yeah. Alright, uh, unless anybody has any, uh…
Questions they'd like to throw out real quick, and like I say, we're kind of running up on time here, so… But like I say, after the main speaker, there'll be time to just go off into general conversation and.
Ask general questions afterwards. So, uh… Last call for any emergency, get it in now.
Hearing none. Okay, our main speaker tonight. Uh, is, uh, Andrew Denner from the CIA log.
Yes, it's a secret organization, a secret lug. Well, actually, it's a central Iowa lug, but uh… Uh, so, uh, Andrew's been very good about, uh.
Uh, re-presenting some of the things that he's originally done, or at least originally thought of up there.
And, uh, so we're very, very happy to have him come down and do a reprised version.
Of, uh, his talk tonight. And, uh, uh, yeah, this was one of those things where… I meant to mention earlier, when we were looking at anniversaries on the calendar, it's, uh, the couple anniversaries of SUSE, uh, this, uh, this month, like the creation of the company.
And, uh, we're hoping to get a CSU speaker. Uh, and, uh, it just hasn't…
It hasn't gotten out yet, or hasn't… actually been properly arranged, so… With that, uh, Andrew is going to tell us all about…
Network time protocol, NTP. And, uh, how to best, uh, set up… control the time on your system.
So, uh, with that, I'm going to turn it over to Andrew, and I love your subtitle.
Uh, Dr. Stratum.
Okay, thanks for having me again. Uh, and yeah, the title, uh, this is actually a reprise of a talk.
Which, pretty much everything changed in 10 years, but about 10 years ago, I gave this, so… Uh, yeah, uh, NTP is still a thing, and it's been around for forever, just about.
So the title, yeah, I was playing with Dr. Strangelove. And, uh, thanks to our friends, AI, who, uh.
Created up the graphics for a few of these slides as well, as you can see.
And… there we go. Uh, so the… the sort of pitch that I sent on was.
Basically, hey, we're gonna cover everything from, uh, what happens when your computer wakes up and it's, uh, January 1st, 1970.
Uh, well, stuff breaks. And we're going to take a look at everything from how time works to…
Ntp, and all of the different software that you can use to keep track of stuff.
What is a stratum? What is… how do time zones work?
And, uh, why… uh, 3 watches are better than, uh, 1 watch.
But, uh, whether you're dealing with GPS, or just trying to keep, uh, figuring out when something happened.
We're gonna keep your system ticking like clockwork, all without. 1.21 gigawatts.
So, briefly little bit about me, uh, if you forgot from last time.
Uh, I'm a scientific software scientist by… or scientific computing scientist by day, and… Linux nerd from Knight.
I've been, uh, a Linux user since way back in 2003.
And, uh, yeah, somehow I still remain the president of CIA LUG, and… Uh, if you didn't get enough of this talk, uh, this… today, uh, I'll be giving it again.
Uh, next week, on Wednesday, up in, uh, Des Moines here.
So, uh, thanks for having me yet again. So really, the first question is, why does time matter?
Well, for your Linux computer, or just any computer in general.
If you don't have the right time and it disagrees with the rest of the world.
Things like security certificates, or even just being able to log in if you're using Kerberos, or one of those sort of systems.
It's not gonna work well. Logging? Who knows what time stuff actually happened? If, uh…
An event happens and it's in the wrong time, you're never gonna find it again.
It plays nice with distributed systems. When files were created even matters.
Compliance, well, if you want to know that somebody logged in at TimeX, well, that time better be agreeable and right.
And also, if you're one of those, uh, poor suckers who, uh.
I have something that's licensed and have a licensed server running and stuff like that. If you don't have your time set right.
It's probably gonna kick you out of your license, or, uh, worse.
Or let you use an old license, and I mean, ethically, that's not great either.
So, uh, that's why time really matters. Uh, the next thing would be just a few.
Level-setting things about NTP and time in, uh, general. Uh, so when you, uh… We talk about, uh…
Utc time. Everybody, uh, remembers… That is, like, Greenwich Mean Time.
Well, that is another time, a name for, uh, sort of a similar time.
But the thing about Greenwich, meantime, is about half the year it's actually not.
Utc time, because it's British summertime. But, uh, basically, the idea is that it's the solar ti… also, Greenwich Mean Time is based off of solar time versus atomic time.
So, what… Go ahead, Steve.
I was just gonna ask, what is UTC and TAI… what are those acronyms mean?
Yep, so, uh… UTC is, uh…
Universal, uh… I think it's French, but it's Universal Standard Time, basically.
Uh, I thought it was Universal Time Coordinal. Print translation.
Yes, that sounds right. Sorry, I drew a total blank to look that one up, but yes, you're correct.
Cordone universal time, or something like that.
And then TAI is actually… I did look up, because that one I didn't know off the top of my head.
Tai is International Atomic Time, and that is French. Junk, because, of course, it's international.
And the thing about that, and the difference between the two is.
Pai is, uh… Uh, based off of what the atomic clocks read, there are no such things as leap seconds.
And it's the average time across 450-plus. Public atomic time clocks that all coordinate and sync up their time, and.
They basically agree and average out. Where UTC is for civil timekeeping.
It's kept mostly in sync with the UT1, which is astronomical time.
Plus or minus about 9 tenths of a second. So, uh, how do they keep that in sync?
We have things called leap seconds. Those are either plus or minus 1 second, as needed.
And, uh, you find out about them roughly 6 months before they happen.
They'll, uh, let everyone know, and then basically just everybody agrees at a certain time that.
We're gonna pretend like there's an extra second, uh. At a certain time.
Now, the good news is that adds all sorts of complexity and heartburn.
Um, and in roughly 2035. There is a plan to completely abandon this because everybody just doesn't want it, and they'll just let time drift.
But, uh, till then. We still have to deal with it.
So… Uh, the next question is stratum.
What are stratums? Where do they come from? What… What are we talking about here? Well, basically, in, uh…
The, uh, world of network time protocol. There are certain layers of at least theoretical trustworthiness.
Stratum Zero is the ground truth. It is an atomic clock, GPS.
You can't get any more pure than that. Stratum 1 is something that's plugged into one of those.
And in theory. It's as good as you're gonna get on the network.
Yes.
Question? So, is… a GPS receiver.
Stratum 0 or stratum 1. I always get that confused.
Yes, so a GPS receiver. The receiver itself is stratum zero.
Okay.
It, uh, is plugged into, uh. A device that is basically… usually, you can query it, get a certain time, and then get pulses out of it.
For, uh, uh… to keep your actual clock regulated?
And then that network device that is plugged into the GPS receiver.
Is a stratum 1 clock. So you… you never actually query a stratum zero device.
Okay. I either…
Using NTP.
Right.
And then, of course, uh… you don't want to have your nice, high-precision Stratum 1 clock being just…
Pounded on by the unwashed masses, because. It's busy being a precise timepiece.
So, most things out on the internet that you're going to connect to, in theory, are Stratum 2s.
Uh, you will see occasionally they'll put a really beefy box out there with a GPS connector to it.
And you'll be querying directly a stratum 1. We'll see that here in a few minutes when I demo out.
Um, actually seeing what time sources we're talking to. But really, from there, in theory, you can go stratum 3, stratum N.
However far down the rabbit hole you want, but really. Time sources are cheap enough these days.
If you're going out to, like, Stratum 6 or Stratum 7, what are you doing?
Network time is cheap. You don't have to worry about those sort of things.
So, uh, the next, uh, thing to cover is. The three-watch problem.
I'm sure everyone's… or at least some people have heard of Seagal's Law.
A man with a watch knows what time it is. A man with two watches is never sure.
Uh, this saying actually wasn't originally coined by, uh. Lee Segal of, uh, the, uh, radio station KIXL.
But it actually came from a newspaper, the San Diego Union, all the way back in 193.
Uh, that basically was talking about that retail jewelers assert that every man should carry two watches.
But a man with one watch knows what time it is, and a man with two watches never could be sure.
Well, it was misquoted and named Seagal's Law, and it sort of stuck.
So, in theory, if I have one clock, you know, I think I know what time it is. I may be completely wrong.
But if I have two clocks and they don't… they disagree with each other.
Sort of like my, uh, microwave and my stove downstairs. I don't know what time it is, it might be 1, it might be the other, it might be somewhere in the middle.
But if I have 3 clocks and 2 of them agree, well, then… or at least are pretty close to each other, then I pretty much probably know what time it is.
Is the idea. And, uh, that will come to play here in just a little bit, uh, when we start talking about.
Setting up your, uh… NTP configurations.
So… where can we get time from? There are public sources.
The NIST, uh, has a great list of, uh. Government-paid-for sources of time.
Uh, if you go to tf.nist.gov. Uh, there's a great list of servers.
Or there's the NTP pool. Org, which is a bunch of servers all around the world of people who are just nice enough to.
Volunteer their time. But really, whenever you're grabbing time, if you're only grabbing one set of time, you probably better trust it.
Um, one, uh, here we can quick pop over to the, uh.
Ndp, uh, time… or if it will actually let me here.
Okay, nothing like doing things live. So, uh, if we go to TF… dot NIST dot gov…
Uh, here is the, uh, looking like it was written in, uh, the 90s, because it was.
List of different time sources, uh, from the government and where they exist.
And, uh. As you can see, there's a lot of stuff in Maryland, a little bit of stuff in Colorado.
And then, uh, more stuff in Colorado. Because that's where, uh… our wonderful, uh, NIST, uh.
Offices are based out of, so it makes sense. The best source to use is… time.nist.gov, it will be basically a round robin that will get you, uh.
A mix of everything, that way it's happily balanced across. All of the sources?
And, uh, we'll see it here in a few minutes in use, but it's a perfectly great and accurate source of time.
And if you really want to use the old-time protocol. It still does work, but don't… It's, uh…
It's not very useful, it's not very accurate, it doesn't give you a lot of information.
And it's better just to get with the, uh… future, because that and, uh, the daytime protocol.
On port 13, all of those are really, really old. Stuff that I'm surprised they're even, uh, supporting.
Anymore. Uh, the other source of really great time. Around the world is the NTP…
Ntp… pool… dot org…
And these are actually where most of your, uh… distros and everything else will get their time from.
And if you're feeling really, really nice, you can actually join the pool and start providing time out to the world.
Uh… you probably should, but… Especially if you just have an internet connection at home, it's probably not worth your time or trouble.
But, uh, as you can see, no matter where you are in the world.
You can, uh, connect up to, uh. A time server near you, and if you want to get multiple, uh, ones, they have a round robin 0123.
For, uh, time pools? Uh, and basically, you'll just get one of the random servers nearby you, if you're saying North America, you'll get…
North America, if you say U.S, you'll get U.S. And if you're one of those folks in one of these underserved countries, like Cuba.
Well, you probably will be more valuable to host versus, like, say, uh… The United States, where there's a thousand servers.
But as you can see, there's quite a few active, only a few inactive.
And, uh, they all stay… you have to have a certain amount of trustworthiness and… uptime and all those sort of things before you actually get put into the pool and live.
So, uh, yeah, that's, uh… a good spot to get, uh, time if you don't have a local friendly.
Time server sitting under your desk. Uh, me, personally, I have the finest grade of chinesium.
My, uh, stratum 1 time server, uh, there's a photo of it.
Um, as you can see there, uh, straight all the way from Alibaba Express.
Before the tariffs hit. And there's a nice little ceramic antenna that I have sitting in my window, and it…
Works great. And as we'll see in a few minutes, I have, uh… Less than a microsecond of, uh…
Less than a millisecond of time, uh, between me and my time, and it's, uh, great.
So, this is actually the UI for it. As you can see, it looks just like what you'd expect a, uh.
Somebody who, uh, coded up a time clock to have written.
And if we take a look here, just in the, uh… The interface for it, as you can see, it speaks Chinese first, English second.
And, uh, otherwise, it's really a no-frills sort of thing. You can see I'm pulling, uh, GPS, or…
Data from all of the major, uh, constellations. And I'm using… a good diversity of time.
If you really wanted to, uh, come and find me, yeah, those are my lat-long, down.
Painful, uh, locations, including my altitude, so… Uh, yeah.
Not great, but here we are. Uh, you can do stuff like update the password.
You can, uh, turn on and off broadcast time, which is really not used very much anymore.
But, uh, as we'll find out in a few minutes, that actually ended up helping me.
Remember what IP address I'd set this thing to, because it had been a while since I'd set up a computer with it.
And, uh, otherwise, your Ethernet, uh, configurations, I mean, it's really no frills, as you can tell.
So, uh, with that, if there aren't any questions about the first part here, we can sort of dive into.
How do we keep time in Linux, and… Well, what, uh.
What are we doing here? And why are we even, uh… getting lost in, uh.
27 slides of fun. Okay, hearing no, uh, comments or snide remarks, that's great.
Feel free to shout them out if you do have any.
Um, this is sort of the standard, uh… There were 14 competing standards. We need to make one universal standard that covers every use case.
Well, now there's 15 competing standards. In this case, I'm talking about, uh.
Time-based programs, not necessarily, uh. Different standards? Yes, Tony?
Um, yeah, I had a question. Um. So you said, um, NTP doesn't talk to Stratum 1 servers, or Stratum 0 servers.
So, like, there's a different protocol, I guess, or…?
Uh, so in the case of most of the, uh… Uh, atomic clocks and, uh, GPS devices.
What you can do is you can query what time it is.
Uh, basically through, like, a USB interface or something like that.
And then you can also get a pulse. A regulated pulse out of it of every.
Every, like, X milliseconds is a pulse, and that you can use that to regulate your… timekeeper, basically.
So it's not reporting in time, it's giving you a signal.
If that makes sense?
Yeah, basically, you can see, okay, I don't know if anyone's ever been a big, uh.
Shortwave radio nerd, but you can hear, uh… Uh, if you tune in to wwwv, uh.
At the tone, X hours, X minutes, universal coordinated time. Beep. Beep, beep, beep, beep.
That's basically what you're getting, and… That time synchronization signal is what is basically both governing your.
Is basically conditioning your clock to stay in sync. And all those sort of things.
If that makes sense.
Yeah, that's good. So, um… A lot of people… a lot of people just installed Debian and don't think about this stuff, so what… what server are they using? What is, like, the default?
For, like, Debian or Ubuntu or something.
Yep, um, thank you as a great lead-in here to, uh… Uh, really, it depend… the easy answer is…
It depends which, uh… which OS are you talking about, and which, uh…
Which, uh… era of release are you talking about?
Uh, in the case of Debian. Uh, if you're going with just the slimmest version.
Uh, you're talking about System D timeSign D? And, uh, that, that is, uh, it's a system that we're going to be talking about first here.
If you're talking about an older version or something that's more set up to be a server.
Install of it, it's probably going to be running NTPD. And Ubuntu and, uh, I believe Red Hat.
And several of the more newer ones here, as of… pretty much this year's release.
Have, uh, moved on to something called Crony. And, of course, we'll get to that here in a couple slides.
So the easy answer is it depends. Just like everything good.
This is Stan, I have a question. Uh, well, kind of a…
Yes, go ahead, Stan.
I've read that there's a difference between. The way Microsoft.
Windows does time in relation to the BIOS versus how Linux does it, is that…?
Through? I just couldn't quite understand what the… The narrative I read was talking about.
Yeah, so by my understanding. Microsoft, by default.
Uses whatever time is set for the system time, the, the, uh… the clock that's set in BIOS?
Right?
And so if you change your time zone to, say. Like, in Windows, if you pop open the control panel and you switch from Central Time to.
I don't know, UTC plus 14. Uh, it's going to change your BIOS time as well.
And what effect does that have versus the way Linux does whatever it does?
And so, Linux, by default. Uh, use your, uh…
Bios time is set to… UTC time.
So I'm doing it wrong?
And then, uh, you set a time zone. Well, and then, uh, if, as you install your Linux program.
You see the little, uh… you always come to that window that asks, where are you.
Uh, without fail, I always choose Chicago, yeah, for your time zone.
Time zone. Okay.
And so it adjusts your time. That you see in the… unlike your, uh…
Start bar… well… Yeah, whatever you call, uh.
Yeah. What are applications?
Yeah, your application bar up here. Or when you run the command line, uh, time…
Okay.
And it sends you out your thing. That's actually, uh… showing you, uh, the adjusted time, uh, to match your client.
Because in theory, I could be SSH'd into this from some other place, too, and… I could have my personal time set differently, or stuff like that.
So, hopefully that clears that up a little bit.
Yeah, I seem to remember years back that there was a big difference.
That it… that I never see anymore, because I've always… dealing with the graphical change.
And it used to be… I vaguely recall you had to do a different command.
To set the time in Linux. It's been a long time, so…
From the fall. The Microsoft way…
The post's hardware clock is set to. Whatever time is in your time zone.
Whereas Linux and Unix way is it is set to UTC.
Okay.
When you say UTC, do you mean, like, London time, or an offset from London time?
Um, there's also kind of a hybrid. Where…
No actual coordinal universal time. Um, sometimes called GMT.
Yeah, it's… Sorry, go ahead.
There's also… Go ahead and… There's also, uh, kind of a third hybrid variant that I used to see in Slackware installs.
Where it asked… Are you dual booting?
A Windows system. So, it would actually put.
Time in a third mode. Where it knew the hardware clock… was set to the local time zone, and it would not do the offset calculation. So you could do all the… and be compatible instead.
Having to mess with time.
Okay, that's kind of what I remembered that. Partially remembered.
It's really… what is the hardware clock set to?
That seemed to confuse me.
Is it set to Universal Time? Or is it set to your local time zone?
Microsoft, it's your local time zone.
Oh, the church… okay. I'm still reading that when you transition.
From Microsoft to Linux that may be something to deal with.
I think dealing with that has been abstracted away about 18 different rabbit holes.
Right.
Turtles all the way down type thing, and I have not needed to actually care about it in more than a decade, unless I'm dealing with.
Right. Okay.
Very ancient, I believe.
Thank you all, that's exactly what it kind of… explains to me what I'm misremembering.
Yeah. Richards…
Yeah, and now that you mention it, I have seen that before, but yeah, it's been years since I've had to worry about that.
Okay.
Richard Schmidt in the comments said that, uh. Not really true. He says, Linux time is the seconds from the.
Epoch of one January 1970.
That's… that… of the value of the number.
Yes, that's fair.
But it's starting as… If it's universal cordinal time.
Versus Central Daylight Time. Here in mid-Missouri, type thing. That's how…
Correct.
Stored internally.
Well, I'm really glad I threw a curveball into everything here. Can we continue?
Bye.
I'm okay with continuing. Any other questions here before we dive in, now that the train's off the tracks?
There's another comment, you know…
Yeah, I guess Richard was saying that Tobias does not involve time zones.
There's…
It's not close to UTC.
Let's agree to disagree and digress.
And I guess…
Yeah, anyway, so, uh, starting with the, uh… most simplest and dumbest one here first.
The System D, uh, TimeSyncD, uh. Which, actually, there is another, uh, older, uh.
Time, uh… command that basically was just dropped into cron.
And once a day, it would reset your clock for you.
Which probably is the dumbest, simplest way of doing it, but… So, yeah, um, SystemD, TimeSyncD…
Is a simple client. Um, but it's, uh, built on top of System D, and it's the default.
For, uh, Debian right now, if you're doing the, the, uh… lightest weight of the installs, uh…
Ubuntu, uh… for a certain time periods.
Uh, there was a… I think it was, uh. The, uh, 18-dot, uh… release that was, uh, when they switched from.
Ntpd, and then went to this, and now they're moving. Towards, uh, crony, as I'd mentioned.
Because of some of the downfall, uh, the faults in it.
And then, of course, Arch is also using it as well.
It is a client only, it's just very simple. It, um, takes… it, again, fairly lightweight. It will fall back if you have a list of, uh, multiple servers to hit.
It will just grab the first one, say, hey, what time is it?
And if it doesn't respond, it will move on to the next one.
But if that first one responds. Uh, it, uh, will just go with that, that time, and whether or not how accurate it is.
That doesn't matter. It gave me the time, so that's what time it is.
Uh, and it, uh, does, uh, it is network aware, which is one of the faults of SystemD.
Or one of the faults of NTPD. Um, so if your network is, uh, disconnected, it's fine with it.
Uh… It does also save, uh, how much of an offset there is, and how…
Jittery your clock is and stuff like that. It will save it to a file so that it at boot time.
If you don't have a very good clock. It can, uh, give you something to start out with.
Uh, like, for example, a Raspberry Pi or something like… one of those sort of systems.
And for the most part, it gives you good enough time.
For logging or stuff like that, it gets you pretty close.
And, uh, one of the ways that you can tell if it's running.
Is you can just run this command, uh, system control. Status, SystemD timeSyncD.
And then, of course, as you can see, just like any other happy server.
Service, it will tell you, uh, what's going on.
So, uh, next up is NTPD. Which, uh, will, uh, happily, uh, is the old granddaddy of them all.
Uh, it dates all the way back to 1985, when Dr. David Mills, uh, came up with the NTP protocol.
And, uh… that was at the University of Delaware. It's still… it's basically one of the absolute oldest internet protocols that still is in regular use.
It's gone through several different, uh, iterations, but really the heart of it is still there.
And of course, NTPD is the reference implementation. It was developed at the same time, written in C.
Designed to run continuously as a daemon. To keep your time, uh, in sync.
Uh, basically the evaluation… the evolution of it was from the 90s to 2000.
It was the standard of time synchronization for, uh. Unix-like people everywhere.
And why was it popular? Uh… It had this idea of stratum-based hierarchy, like we've already talked about.
It can, uh, talk to multiple different time sources at the same time.
And suss out what time it is based off of. What you get, and what are the accurate, trustworthy times in there?
It will actually look at your hardware clock and say, you know, I know you think that you're running on time.
But here, this is actually… you're, uh… a few ticks short of a full minute, so let's go ahead and adjust that.
It can actually do authentication. And, uh, validate that the time you're getting is actually from who you think it is?
And it has had, uh… sort of security enhancements through the time, there was an idea of having something called AutoKey for authentication.
And apparently, it's been deprecated because it was a great idea, but it also has security vulnerabilities in it.
And it also has support for things like IPv6. Multicast and network broadcast.
Which, uh, network broadcast, uh, as you see, saw in my, uh.
Stratum 1, uh, time source. Basically, all it does is every second or every minute, or whatever.
It shouts out to the world, hey, if anyone cares, it's this time.
And, uh. Be basically over and over again, whether or not anyone's actually listening.
Which, uh, it's great, but it also, uh. Is lacking some of the niceties that you get from direct polling.
Uh, one of the ideas that you can run in, uh.
Is that you can actually ask it. Uh…
What are my time sources, and what are good and bad?
Uh, the pluses are being used… And the star… or the star is the one that it's gone with, the pluses are responding.
But, uh, aren't being used. Uh…
I thought the pluses were candidates. If this don't stop responding.
Or, yes. Yes, that's… that's correct.
Because all of them, I believe, are responding, it's just… which ones are backup.
Their candidate, yeah. Yep, sorry, I was thinking crony… crony. Um…
Gotcha.
Yeah, as you can tell, there's a lot of slightly variations on the same idea, and some of them are kind of hard to keep track of unless you're looking at them every day, and if you're looking at how your time sources are doing every day.
You have a far more boring life than I do, and I don't have that interesting of a life.
So… Uh, but anyway, though, if you truly do care about time, there are those time nerds that.
Probably have their own, uh, cesium clock down in their basement. I've tried to get one.
My wife won't let me. You can be… you'd be surprised what you can buy off of eBay. But anyway… I digress.
The third option, because of course, you can't be… one of the big problems with NTPD that they've run into is.
Number one, it's very, uh… Old and crufty, it's been being developed for…
30-some-odd years now, and uh… has all of the cruft that C code from back then and etc. Has, uh, drawn on, so it's…
Slower in development time. As well as it doesn't handle, uh, network, uh, changes very well at all.
So the third option is crony. It's, uh, fast, it's the fastest at, uh, actually.
Going from straight-up startup to, uh, being online and running. It has support for offline support. Basically, if the network drops off.
It will, uh… basically based on what it's learned about your local time source, it will still keep that going.
It's a lot more power efficient and memory efficient. Ntpd tends to be pretty heavy.
It's also very flexible, it, uh… Uh, is, uh, very, uh, good at…
Being able to, uh… handle all sorts of different time sources, including if you have a pulse generator, like, from your, uh.
Gps source or whatever, it can talk and handle that. As well as, uh, different reference clocks, and also just manual input, too.
Uh, it has some nice monitoring tools that come with it, etc.
And, uh, part of the power efficiency is it's designed with mobile and low-power laptops.
In mind, and it's great for, like, laptops, embedded systems, and virtual machines and stuff like that, that… really aren't, uh.
Prepared for running really heavy stuff or System D. Which is part of the reason why.
Ubuntu and several other repo, or, uh, distros are looking at moving towards it.
And we'll actually, uh… play around with it a bit here in a minute, but…
Here's an example of the output you get from, uh… Uh, running, uh, the, the crony C command for tracking. As you can see, I'm getting.
My reference time, uh, in UTC. I'm pretty darn close to being actually accurate.
I have just a little bit of drift. But it's still, uh…
I'm not very far off of being… Uh, right in time.
And I'm checking in every 1.4 seconds, which… Again, seems insane, but here we are.
And, uh, this is the output if you, uh… are looking at what are all of my, uh, times, and this is where I was getting, uh, confused.
Cronies decided that my local time source is the best. As you can see, it's a stratum 1.
And, uh. I'm about 1 millisecond off of…
Uh, for latency. So, I mean, that's… I'm probably about as good as you can do over Wi-Fi.
And as you can see, some of the NTP servers. This one there, they're claiming as stratum 0, uh, probably because of how they're defining it.
Um, or… Or no, sorry, uh… that there was no response.
Time E, uh, B is actually a, uh, IPv6 server. And that's why it didn't respond, I think.
But, uh, as you can see, we've got a smattering of, uh, canonicals, uh, time servers are all stratum 2s, as we'd expect.
Um, and then, uh, Indiana… colorado.edu, uh, ends up being a stratum 1, just like I am.
They're just a little bit further away from me, surprise, I'm here in Des Moines. I wouldn't expect any less.
And then, as you can see… my, uh, IPv6 clocks…
Since I don't have IPv6, uh, on my local network. Yay, thank you, uh, fiber provider.
Those just don't respond right now. Now, if I were to connect to my, uh… Wi-Fi, or to my, uh, cellular, uh, modem, yeah, I'd have IPv6 then, but…
Oh, well, here we are, the future is now. So, one gotcha that I did find, and uh… our friendly LLM in Copilot, uh.
Steered me wrong. Uh, crony does not handle, uh, broadcast, uh, NTP time.
Uh, my, uh… Hardware, uh, unfortunately, is not smart enough to do multicasting, so…
We're at somewhat of an impasse. Uh, we're, uh, one doesn't support the other, and the other doesn't support the other thing, so…
Multicast is basically the more modern way that you can just do a broadcast.
Network, uh, basically one of the, uh… multicast IP addresses and just shout to the… anyone who's listening on that address that.
This is what time it is. Uh, and the one problem I did run into is.
I'd set this thing up months ago and forgotten about it, and uh… my laptop, I hadn't ever set up because I've… I recently re-imaged this thing, so…
I forgot what IP address I'd set it to. Uh, and, uh… Yeah, so, um…
My trick was, I just… went ahead and… did, uh, TCP dump.
Listening on my network, uh, for port 123, which is the time.
Ntp, uh, port. And as you can see, I've got some various traffic that I was able to capture.
Here's one coming from, uh, Colorado's, uh, time… NTP server…
And another one… And then here you can see is where my favorite local little source that's sitting under my desk.
I gave a broadcast. That was, uh, giving out what time it is.
And what time… uh… how much of a delay there is.
And then, what time it transmitted at, and etc. So, as you can see.
Uh, you don't get the, uh, quite the network delay, uh.
Information, uh, that makes it a little harder to get as down to the nitty-gritty as TimeSync.
But, uh, it at least gave me the hint that I needed to figure out what IP address I was working on.
Uh… Sorry, was there a question?
Okay, hearing none, uh… Basically, the big compare-contrast matrix here of… between crony and NTB.
They, uh, both are great. Crony's just a little more accurate.
With, uh, goofy network latency and connectivity and stuff like that.
Uh, NTPD is pretty rigid, and… doesn't really like if you drop on and offline.
Doesn't really matter if you never restart your computer, but, uh… crony syncs quickly at boot.
And, uh. Is a lot better at handling those sort of things.
It also, uh… does a better job at.
Keeping track of, uh, time, even when your network sources drop off.
And will help correct, uh, drift. Even if, uh, it's no longer connected.
It really is good for a lot more energy efficient around battery usage and stuff like that.
It actually understands when things, uh. Go to suspend mode, or resume, or stuff like that.
And it's a little bit less Byzantine in its configurations. And it still has, uh…
It has a little bit more updated security stance in being able to tell that, yes, this is a secured, signed.
Uh, timestamp… And it's actually being a lot more maintained and modern.
Uh, this is sort of the exorg versus, uh… Uh, your other graphical display system sort of argument as well.
And, uh, broadcast, uh, multicast support. Is limited, but, uh, if you really want that sort of thing, NTP is…
What's, uh, what you want.
I have another difference for ya. Um, NTPD.
Yes, go ahead.
Does not like x86 virtualization. The way time slot happens.
Yes.
Because it actually tries to run, which consumes more power and kills batteries.
A tight loop or counters. To calculate, and time slicing x86 messes with that.
So, that may be another reason that Crony. Which plays much nicer in virtualization.
Is being favored.
Yep, uh, yes, that is another factor. That's also part of the reason why.
Uh, their, uh, Ubuntu's moving away from the system D-based systems.
Because a lot of the virtualization. Containers don't have SystemD running by default?
And so, um… It just adds another very annoying thing to deal with if you're wanting to run it inside of.
Virtualization. Not the end of the world, but it… it is just yet another thing you have to be doing.
And especially if you're on a low-memory environment. It's nice not having to have SystemD being a hog.
So, now that we've talked a little bit about how we keep track of time, uh.
Before we do that, we can go ahead and jump in and do some just digging around here on the command line, just to prove that I'm… Not completely making things up.
Only some things. We can look at the, the, uh, cranny, uh.
Config files, which is just, uh… in your, uh, ETC cranny…
Uh, cranny.coff. And here you can see my configuration.
The big, uh… fun, notable spot here is… this is my home server.
And, uh, iBurst means that on startup, it will hit it a few more times faster.
So that, uh, it will, uh, get… synchronized and going a little bit faster.
Uh, you have two different, uh, ways that you can ask for servers.
If you say server, you say, well, this is just one machine.
Pool means that there are multiple ones. So it just hits and gets a list of them.
That's why, if you noticed, I had multiple NIST.gov servers listed, including the, uh.
Colorado, um, university ones. And you can actually say, well, I want at least 2 of those.
Or, uh, ntp.umbuntu is grabbing 4 sources, these are… those are, from here on down.
Their, uh, default, uh, defaults that I just didn't bother changing.
Uh, and as you can see with the NTP.org pools. A lot of the vendors will have their own name.
Um, so, like, say your wireless router. Probably will be, uh, like, Cisco or something like that.
And the reason why NDP.org does that is, like, say… Ubuntu has a configuration problem, and they just start pounding the heck out of.
The pool, and behaving very, very badly. Ntp.org can actually go out and just…
Block them and stop. The bad actor from breaking things until it gets fixed.
Uh, since we don't necessarily have that, if we're just setting it up ourselves.
That we say, okay, I want to get U.S. Time sources, because I'm here in the US.
That's where you can change what your DNS address is, or what your, uh.
Addresses there, and you can control that since you're not a vendor doing very big, bad things.
The rest of this, I just have kept default, because I haven't needed to change anything.
But, uh, you can actually set your sources in a sources.d folder if you really care about it.
Uh, you can also set your key files if you want to, uh… coordinate and, uh, authenticate your time.
If this was a system that truly had to care about those sort of things.
You can turn on logging to be able to track, uh, statistics and stuff like that.
And you can also… set it so that it will only update itself so much at a time.
Skewing itself slowly into being straight, uh, and correct. Because, uh, certain applications really hate it if you just start pulling the rug out from under it and, uh, changing what time.
Things are at. Uh, you can also…
That's… I mean, if… You are ahead of time and need to go backwards.
And you have database transaction logs. You don't want time to go backwards.
So it goes forward really slow. Until it is at the correct time.
Yes, correctly. Correct. It's one of those things that. Time matters, and simple things such as leap seconds.
And all those sort of things really, really break things. And our edge cases.
Which is part of the reason why they want to get rid of them.
And these are just, again, other, uh… really, uh… fun, uh, little adjustments that you can make.
And really, for the most part, I've never had to set them.
And you're probably doing deep, dark magic if, uh. You are actually needing to set them.
Which is fine, I encourage it, but… Yeah. So, as we can see here, we've got…
Uh, activity will give you, how are we doing? It's okay.
We've got 11 sources online. None of them are offline right now.
Everything looks good. Uh, tracking-wise, it will report, yeah, we're sitting at stratum 2 because we're talking to a stratum 1 server.
And, uh, we're… pretty darn close to being right on time.
Yeah, leap seconds are… are bad. Yeah, we're going to talk about some of those, uh.
Things that you think you know about time that are wrong, those are coming up here in just a minute.
Uh, so, uh, yes, we're going to get into that fun.
Trivia of, uh… Time is hard. It's wibbly-wobbly and all those things.
Uh, but as you can see from my sources list here.
Uh, I'm… uh, 744 microseconds in this specific time when I last ran this, let's… See how I'm doing. I'm a little bit off, but then again, I'm on Wi-Fi at the moment, so that's not a huge surprise.
And plus, I'm doing other things with my computer, and uh… working at harder than normal because Zoom is fun.
So, yeah, um… Let's talk a little bit about time zones.
And how much fun, uh, those can be. Making sure I'm not… yep, okay.
Wanted to make sure I wasn't skipping ahead, or missing anything, so if we just copy and paste… Uh, let's make this a little bit bigger so those in the back can see.
Maybe?
Uh, come on. There we go.
So here you can see, uh, we're on… American Chicago time.
We're synchronized… And, uh… our, uh, real-time clock is not in the local time zone.
So, if we want to list and see what are all of our time zones that we can possibly, uh, have.
Time, date, CTL list time zones. Is the magic that will show you all of the possible time zones that you can be dealing with.
There's a lot of them. And we'll deal… with that here in a few minutes, of just exactly what are these time zones, and why are there so many of them, and why are there.
Just so broken. Uh, if you wanted to set your time zone, it's really easy.
You just copy and paste whatever time zone you want. Into this, uh, magic incantation.
And you can see just exactly what's going on there. And if you want to see, uh.
What's in your… because, of course, in Linux, everything is a file.
Uh, let's go, uh, and LSR, uh, local time.
So right now, we're saying ls-l list. Everything that's in ETC local time.
And here you can see that really all we're doing under the hood is local time.
Is just a pointer to… America, Chicago.
And that's really all that this command ends up doing under the hood.
Uh, yes, there can be, uh…
Uh… yes, you can copy that as well. That's just the implementation that Ubuntu, uh.
Used, but yes, you can just do a copy. But if we want to do a time zone dump to, uh, sort of query our.
And see what's in our time zone file.
You can see, uh, the different times that… things changed on what is an offset, and, uh.
Different time zones that it… or times that the time has changed.
Historically, if we, uh, wanted to see a lot more than.
Uh… 10… You can see that they're… there's a… basically, it's predicting out when it's go… the time zones are gonna change for Central Standard Time.
Uh, all the way on out to, uh, way in the future.
Assuming that those times actually happen. And that sort of information is for every single time zone that we have listed there.
And there's a keeper of all of that data, and… It changes fairly frequently, at least in some time zone around the world.
That, uh, this data is always updating and always… being, uh, changed.
So, of course, this sort of brings up the great quote from the doctor.
That people assume that time is a straight progression of cause to effect, but… Actually, from a nonlinear, non-subjective viewpoint, it's more like a big ball of wibbly-wobbly, timey-wimey stuff.
Time zones are hard. They keep changing, and… Uh, just whenever you think that you know what time it is.
It's always changing somewhere, and it's… In the spirit of, there's been several different, uh, sort of lists for software developers of.
Uh, things that you know about. Addresses, or things that you know about, email addresses, or stuff like that, that…
You're totally wrong about. I have some, uh… time zones is hard. Time is hard.
And these are the easy ones first. Everyone knows that a day is always 24 hours, right?
Well, no, daylight savings time and leap seconds can make it either 23, 24, or 25 hours, depending on how.
Things work out. Uh… February is always 28 days? No…
Leap years have 29. And historically, it gets even weirder than that.
The good news is we don't live historically, so I don't have to worry about those, usually.
Um, the next thing is UTC offset is totally from 12… Negative 12 to plus 12, right? I mean, it just makes sense. It's split around the world.
No, no, uh… Uh, it ranges from negative 12 to plus 14.
Because of some really nice islands out in the middle of nowhere.
Uh, your body, uh… So, uh, time zones never change. Central time is always…
The Chicago time is always Central Time, and we know when Daylight Savings Time happens.
Well, no, even in my lifetime, uh… George W. Bush changed it, and we shifted what time it was.
Uh, in theory, if the great people in Washington had, uh, passed the.
Daylight Savings Time Act, uh, here, I think, a couple years ago now, we wouldn't have had Daylight Savings Time ever again, uh, so… Yeah, but the cooler heads, uh…
Stepped in and, uh, talked them out of that. Uh, but yeah. So anyway, though, uh… Time is hard.
Uh… the idea of, uh, there's one offset for one time zone.
No, uh, as I said, the rules change all the time.
Uh, and they have different histories and different, uh, rules and… different times. Uh, there's one time zone per country. Us in the U.S. Here know that's wrong.
China has one. Russia has 11. Uh, daylight savings time rules are consistent? No, they vary wildly. Uh, sometimes you get less than a week's notice that, uh, some countries decided to get rid of daylight savings time, or put it in place.
Lawmakers are, uh, weird and, uh, very unpredictable, especially in certain countries.
Uh… leap seconds are predictable. No, they're announced roughly 6 months ahead of time, like I said.
Uh, and it's basically all around as the Earth slows down, or speeds up.
Uh, in theory. They can be negative as well.
And of course, all time zones are whole hours. No. No, no, no, they're not.
Uh, some are plus 30 minutes offset, or even plus 45 minutes offset. I'm looking at you, India and Nepal.
Uh, the harder ones that start breaking your head a little bit more.
Uh, leap seconds make, uh, minutes 61 seconds. Uh, smearing, as we talked about, one of the ways that systems deal with that is rather than dealing with.
Having to count, uh, 61 seconds, they just sort of… smear across. Uh, yes, Indiana is another one that is very…
Messy, because depending on how close you are to Chicago. Changes what time zone you're in, uh, just because time… Chicago's, uh, gravitational pull, uh, skews time, I guess.
Also, whether or not Daylight Savings Time exists or not, I think, is another one that gets really messy.
With, uh, Indiana? Yes, I've never really liked Indiana and how they keep time.
But that's neither here nor there. We like every state equally. Uh… Some days never, ever happened.
Uh, for example, in our recent lifetime here. Arizona's another one that, uh… time zone, or…
Daylight Savings Time just doesn't happen. But in, uh, December 30, if you think you lived in Samoa in December 30th happened in 2011.
No, you're wrong, it didn't. Excel actually has a bug that they know about, and it's not a bug, it's a feature.
Even though 1900 was not a leap year. They say that it is.
Just to maintain compatibility with Lotus 123. Because they got it wrong.
Upcoming in the year 2038, hopefully everyone's, uh, adjusted their stuff from 32-bit to 64-bit, because.
Time t will overflow and blow up in your face. There can be negative leap seconds, because if the Earth slows down, or speeds up a little bit.
We have to adjust it. They can just pull a second off and call it good.
Again, that's gonna break everything, because there hasn't been one yet.
So I'm willing to bet that's an edge case that somebody's missed.
Um… 24 colon 00 is a perfectly valid timestamp.
Uh, but it's perfectly valid in ISO 8601. But most of the parsers get it wrong.
And Bombay, uh, was just completely weird, and they're bad, and they should feel bad for being so weird, because historically.
They were UTC plus. 4 hours and 51 minutes.
Why? I'm not exactly sure. That one minute… yeah.
It's obnoxious. And all of those things are codified in…
The… in your time zone files. So that if you want to figure out what time it was.
Uh, in, uh, or what… how many days it's been since, uh.
January 1st, you can do it and still get an… theoretically accurate answer.
Uh, based off of your time zone and when skips were made and stuff like that.
Um, the other really, really broken… messed up time zone. Uh, in most, uh, GUIs.
Out there, you'll find UTC-12. Is going to be, uh, missing from their dropdowns, just because there are no permanent residents there.
It is in the time zone file, it's ETC, GMT plus 12.
It's also, uh, called… ahead of everyone, or the latest on Earth.
And it's a great edge case if you're testing to see if your system works.
Or if, uh, you want to make sure that it's the last in before, uh.
A deadline or something like that. And there you can see a little screenshot of where it is in the world.
Uh, it is. Uh, basically…
Uh, right here. So as you can see, anybody who's anybody.
Skips and, uh, avoids it if at all possible. So then the even more weirder one…
Is, of course, UTC plus 14. And, uh, that's… This little tiny hunk of, uh…
Land. Spread out over 33 atolls.
And 3.5 million square kilometers out in the Pacific. And it's UTC plus 14.
So, it's, uh… theoretically, um, technically 26 hours ahead of…
Utc 12. And why did they go with that? Because, uh, they spread out in a way that, uh, basically they're…
Parts of their country were, uh. Straddling the date line.
And, uh, it made for things to be really, really messy if you tried to have.
Government office and tried to coordinate with them, because, of course, uh.
On Mondays, their Monday was, uh, or someone's Sunday was someone else's Monday.
And someone's Friday was someone else's Saturday, and… So it made that, uh, basically, uh.
There were, uh, only a few days that you could actually get ahold of people.
Uh, if you were working from one end of the country over to the other.
And so they just fixed it by smashing everybody together and calling it good.
And that leads us to the end of, uh… the time that I was planning for tonight, uh, we can totally dive into more.
Minutiae, or I'm willing to learn at the feet of masters because, uh.
I don't understand all of time. It's wibbly-wobbly, and uh… it's just really messy.
There, I thought a Kerbati… print plus 14 because they wanted.
To be on the same day as Australia instead of… Hawaii, because that's who they were… trading with most of the time.
That was also another factor why they jumped… why they decided to mess with it in the first place, was that they wanted to make it easier.
But then also… Switching over to that helped with dealing with Hawaii, or with, um…
Australia, too. Yes, that's… that's a good point.
And I do apologize, some of the polishing is a little rough here because, uh.
Uh, yet again, this was one of those where, uh. I got the phone call, uh, to, uh, speak, uh.
With only a few days' notice, so, uh, we're… We're running a little short on time for Polish. I admit I finished some of this over lunch this afternoon, so…
They had a great speaker lined up. And very great, well-renowned, loved by all.
Unfortunately, they weren't available, so you guys got me.
Would it be fair to say it could have used a little more time in the oven?
Oh, totally.
Hi, this is Rich Schmidt. Can you hear me?
Yes.
Yes?
Hi, this is Rich Smith from the Naval Observatory. Gary Meyer invited me to tune in, and I'm very pleased that I did. I thought your.
Presentation was outstanding, and. I particularly like the detail that you went into discussing.
The merits of crony. That, um… It was a decision that the Naval Observatory made not too long ago to.
Drop our 30-something years with NTPD and move on to crony.
Amazing!
And it was actually driven by the need. For more modern…
Um, stratum Zero sources. Uh, so when you were talking about.
Stratum Zero sources, um, in… the legacy version of NTP that Dave Mills produced, and.
Harlan Sten's organization. Maintains. Um…
You had to write, um, ref clock drivers. To talk to GPS.
Receivers or radio clocks, or… iRigBee, uh, time and frequency boards, and that sort of thing.
But in… Uh, crony, one of the great things, of course, is that it continued the ability to use shared memory.
Which is very useful for. Some of the more modern PCI-based, uh.
I'm… bus clocks that you can buy. And Cronie can talk to them.
Through shared memory interface, but Croni also has. Um, RefClock drivers for PTP hardware clocks.
So, if you want to do nanosecond-type synchronization within your local area network at home.
You can put, um, various Intel or Broadcom. Network interfaces in that can use Linux PTP software.
To synchronize your systems. In the low tens of nanoseconds.
So, the new NTP servers at the Naval Observatory. Uh, we'll use…
Three master… three atomic… talk ensembles, three master clocks.
As our input sources. And each one of them will have 3, um, PTP.
Nick's, uh, Ethernet cabled up to a PTP, um. Master Clock, Grand Master, 3 grandmasters.
And we also have 3 PCI cards. That are synchronized to a source of iRigBee, which is a.
An analog time code. But anyway. Crony has been extremely…
Um, helpful to us in developing the latest NTP servers, whereas.
Harlan's group, uh… after Dave Mills left, has been entirely unresponsive to the needs of the Naval Observatory for Stratum 1 servers.
And basically, he has just blown us off because we're not giving them money.
So, the other thing is, um… negative leap seconds. There was a great fear.
Last year that we're heading towards. A negative leap second.
And this happens when… there's an excess length of the day over…
24 hours, and… I don't know, am I able to…
To share…
No, no, don't do that. Um, but…
Uh, you should be able to here. I may have to drop, uh, my sharing… Oh, it's fine, I don't have anything more to show here anyway, so…
Yeah, you should be able to share now, I think.
Let me see if I…
While you're doing that, will you please expand. The acronym, it sounded like… Paul, Tom, Paul?
Yeah, so that's correct. Um, let me see if this is working.
And I know that's the wrong phonetics, but… Okay.
Precision time protocol.
Thank you.
All right, I'm gonna skip this. I'm gonna skip this share, sorry.
I'll let you go back. Um… Precise time protocol, if you go to…
Linuxptp.org. Or, um…
Yeah. Linuxccp.org, or… It's now, um, the network content.
Um… It's software that runs under Linux, and it talks to a device called a PTP hardware clock that's on a, like, an i350.
Or an I-210 network interface card? Um, and it…
Can… it has a two-way, uh, handshake with PTP Grand Master Clock, and there are many of those available, or you can use your own… one of your other servers.
As a Grand Master Clock. But it… whereas NTP does synchronization on a local network.
In, um, about a tenth of a millisecond, PTP does it in the low nanoseconds.
Um, 10 to the minus 9th seconds, so it's… it's really quite nice.
Uh, but what I was going to say about that was that.
Um, we kind of dodged a bullet with respect to negative leap seconds.
Because it now looks like. Um, the Earth is pulling away from a trend.
In the short term, uh, to speed up. And, um, require us to do the opposite of inserting a leap second.
Croni and NTPD both have always had support for negative leap seconds, but we don't know.
What effects it'll have worldwide.
So was part of that, uh, I heard, uh. That, uh, some of the tie… some of the speed of the Earth was, uh, actually adjusted by.
The Chinese Three Gorges Dam filling up.
Uh, no, that was a pretty small effect, but there are seasonal effects.
Um, I should say there are… when you look at a… a plot of, from the International Earth.
Orientation Service, IERS, you can… look at a plot of what they call LOD, or excess length of day, and that shows that.
From 1975 to the present. Uh, the Earth has slowed down.
A total of about 3 milliseconds. Um, per 24 hours.
But there are, on top of this. The effects of the moon.
Um, and tidal effects that are, um… the, um, height of the atmosphere, that's a seasonal-dependent thing.
So, change of season, change of atmosphere, change of temperature in the atmosphere.
Changes the… moment of inertia of the Earth, like the ice skater pulling.
Um, her arms in, she speeds up faster. And so, if snow is melting.
On the, um, tops of mountain peaks and so on, we're going to see the.
Mass, um, concentration of the Earth move closer to the mean surface level of the Earth. And the long-term trend.
Will be, um, a gradual speeding up. Of the Earth to counteract the long-term.
Slowing down in there is that we've been seeing in… since.
1975, when they started. Measuring this stuff. But I just wanted to say that.
It looks currently that we're not going to have a negative leap second.
For the next 4 years, but then… Um, this becomes a much greater possibility.
Anyway, I really enjoyed your talk. I think you covered many, many useful things for people. Thank you.
Thank you.
I believe the, uh… compute modules, or the Raspberry Pi compute modules, have that.
Yeah, this…
Ability to do that PTP. Uh, protocol.
I don't know why, but, you know… Jeff Geerling did a whole…
If I remember right, a whole thing on that.
Yeah, it's really just a function of. What network interface they put on the Raspberry Pis. The Raspberry Pis I have do not support PTP.
Right. Right. The regular one. The one that's the same factor as the original.
Without a motherboard.
But the compute module. Which is the weird one.
Well, that needs the… to snap into something.
You have a compute module.
Um, it. Has the ability to… interact via PTP.
I would have naively thought the… Ethernet was on the… carrier card that the compute module plugged into.
Nope, it's… pass… passed through these connectors, and it's on the actual.
Okay.
Compute module itself.
All right. Yeah, very interesting presentation. Thank you, Andrew.
But that…
Thank you.
Makes me want to dig out the, uh… USB-based receiver I have.
That I think I can get PPS out of. Hey, one thing you did not touch on that is kind of germane for my use case is.
I don't care if the time in my house. Is precise with the rest of the world.
I just want all of my devices. To be synchronized with each other, and all wrong in the same way.
Yes, so in that case, what you could do is, uh… You have one, uh, time source.
Excuse me.
In your house. Stratum X, whatever value you want. Have that clot, NDP server.
Set and running. And then just synchronize all your times to that.
Right?
So in my case, since I have a Stratum 1. I could just synchronize all my computers to that, and…
I'll have not only very precise time, but they should all agree with each other.
In theory, uh, the old use case of broadcast. If all of them were listening to the broadcast and setting themselves off of that.
They'd also satisfy that itch. You can also set, uh, in your, uh.
Yep.
Dhcp, uh, server. Announcing that… that… that time is… that NTP server is available.
And in theory, if your stuff is set up correctly, it will… take that and run with it?
We'll have to set that up on my Raspberry Pi. You have to… port 123 open, right?
Um.
Uh, yes. It was pork, uh…
123 sounds right. So, devices on your LAN would need to be able to reach UDP port 123.
On your Raspberry Pi if it, presuming it's your NTP server.
But I believe… it can do stateful.
Out to the internet, and it is perfectly happy going through even multiple layers of NAT.
Oh, yeah, I just… I have the cranny extension set up on my…
Home Assistant build. And… it was just 3rd as…
That's a clock on the home assistant was synced. But I've been running… or looking to start running some LDAP servers, and.
Heavenly knows that if those LDAP server… GSAs get out of sync by more than.
10 seconds… They get cranky.
I was just gonna ask that question, like, how… how… Um, that does it have to get before there's problems, and… and it's just… just… is this just network?
Networking problems, or is this single system problems, or…?
Well, with… With LDAP, what happens is… You can write to any of the…
Servers, or GSAs. And if you have multiple DSAs.
Oh, you've got multiple, sir.
They will fight it out in the back end, and it's always by who wrote last.
Okay, that makes more sense.
So if it's… if your clocks are out of sync. Then that screws up the synchronization between the two. I had this problem with my…
Tsas on my… product in… at work, and it was a royal pain in the rear end to get those things set up, because I had to go find.
The internal NTIC and get… make sure everything was synced, then run the install.
Just because, for some odd reason, VMware wasn't keeping the hardware clocked.
The…
And the VM clocks. Properly in sync.
Um, are those… what you're trying to synchronize are those, uh, virtual machines themselves?
Yes, they're virtual. They… the ones that work are… virtual machines, and the ones here will be virtual machines, but…
Great. Um… I kind of alluded to it earlier.
Time in virtual machines on x86. Is problematic.
Because of the way virtualization happens on x86 compared. To spark, or power, or other architectures.
Well, that's why ELA has to be within, like, 10 seconds.
Um. Right.
Doesn't have to be ultra-precise. It just needs to be close enough that.
If somebody writes to both of them. Which occurred likely good that the one that got written last is reported to be written last.
Yeah, I wonder if…
And I would multicast or broadcast. Might help in that. Um…
Andrew, how often were those. Broadcast time announcements from your stratum one?
Yeah, so I was… I had it set to, uh… I believe it was, uh, 1 a second. Uh, I'm looking right now.
Sign in… Uh… NTP status…
Uh, yes. Oh, I'm, yeah, doing it every 60 seconds, but you can turn it lower than that if you want.
Right. Um… I wonder if the VSAs have an option to receive.
A broadcast or a multicast? So, something could… do that broadcast or multicast of, hey, the time now is.
Ding-type thing. Uh, and the VSA's update.
Because with x86 virtualization. Time basically stops.
In between clock cycles. Whereas non-X86, it still continues, and…
It… minutiae. It's weird, deep in the weeds.
That's something where Kroni can also help. It, uh, handles that weirdness a lot better.
And you can set it for a frequent pull.
Yeah, and I think the advance… or al… actually call it the disadvantage that NTP has.
Is NTP. Tries to have a loop running where it's calculating real time, and.
If time stops in the middle of that loop for a short amount of time.
When it resumes, that loop is now way off, and it's just… busy going, oh, I'm wrong, gotta re-correct myself.
And Karami doesn't try to do that. It's a, okay, what time is it now?
Frequently, and it just adjusts itself.
Which makes a lot of sense. Virtualization is hard.
Mm-hmm.
Uh, to answer Tony's question of how out of sync do things have to be before things start to break.
It depends on how paranoid is whatever thing you have, uh, that's connecting.
I know, uh, I'm trying to remember for… Kerberos, I think it's within a couple hours.
But, uh, it really just depends on, uh. There are some things, like, as we were mentioning there, uh…
That tends of seconds can really mess with stuff.
I have heard from multiple reliable sources. That Kerberos under Active Directory.
Things start breaking. At about the 10-minute mark.
That would make sense.
Um. Conversely, Novell, or Netware Directory Services, 20-some years ago.
It wanted single-digit number of seconds, like, maybe within 2 seconds, and it was obscenely tight.
Compared to other things at the time.
Yeah, I just know that I've seen stuff break because the clock wasn't set right.
I never bothered to suss out what point that was. It was just.
Oh, NTP got turned off. Why? Okay, well, let's fix it.
For a walk, or…
Because especially if you have cheap, crappy hardware. System clocks don't… really stay synced all that well.
I remember for a while, while I was at IBM. The recommendation from the VMware team.
Was to not… run an NTP daemon.
And to rely on the host to synchronize the guest's time.
And it turns out, it only does it. On boot, or when VMware Tools restarted.
So if you had a long-running VM. Its clock would just start.
Getting it further and further and further out of sync. In a reboot or restart of the tools, or bike an NTP date command.
Would go, oh, hey, let's fix this.
Yeah, I'm seeing that, uh. Uh, anything more than 5 minutes causes authentication failures in Kerberos, apparently.
With my quick Dr. Google search.
Yeah.
But yeah, the prevailing wisdom always was that. If you're a guest, you should trust your host to keep their clocks synced.
But, of course, as we all know, that… That doesn't play nice.
Mm-hmm. Especially when it only does it. Under certain configurations on boot.
Oh, this is Gary. I'm gonna stick in just a couple of footnotes, uh… Uh, first off, David Mills was mentioned. He's the original author of NTP, I believe.
Uh, that is not the same David Mills, who is the president of the user group here in St. Louis.
Uh, the Unix users group. Uh, uh, in the 1990s, I don't have the exact dates.
But, uh, Dave was our longtime president, uh, moved out of St. Louis, uh, somewhere out into rural Missouri.
And, uh, haven't heard from him in a while, but that is not the Dave Mills of network fame.
Awesome. Another name we just mentioned, uh, uh, the name Harland Sten was mentioned.
Uh, Harlan… I believe this is the same Harlan that ran plus 5 computing, uh, which was at Page and I-170 here in St. Louis.
Uh, Harlan was actually, uh, I don't even know if they had the title of president, but he was certainly one of the lead organizers of what was the first, uh, Unix users group, as far as I know of, here in St. Louis.
It existed in the late 1970s, uh. And, uh, I guess in 1982, 84, somewhere in that range.
As AT&T was released from its non-compete clauses in its antitrust provisions.
At&t felt that, well, they should be running the, uh, uh.
Unix users group, because they own the name Unix. And so, uh, AT&T formed a Unix users group here in St. Louis.
And the original, uh, which Harlan and, uh, some of the guys at St. Louis U and.
Uh, later wash you were involved in. Uh, they, uh, figured, okay, well, AT&T has the money, they're gonna fund it, they're gonna run it well.
And, uh, so they just turned it all over to AT&T and, uh, of course, as we all know, AT&T.
Uh, later in the 80s, uh, decided to get out of the, uh.
Computer business. So, uh, uh, and that's when… The third or fourth integration of the Unit Trusage Group came about. That's what exists today.
So, uh, uh… So, anyway, those are the, uh, the brief histories. As I say, I think it's the same Harlan stem, uh, but I haven't checked.
The one other name that we heard tonight, Richard Schmidt, uh, uh, gave us some interesting information there. Richard works at the, uh.
Naval Observatory, as he said. Richard did a presentation for us. I haven't looked up.
The date I was thinking of doing it, I didn't… do it. The archive, but it was like… I don't know, Richard, was it 15 years ago, maybe?
I know it was when we were still physically meeting at, uh, Sunnen Products.
Uh, and uh, Richard was on it. On a trip here to St. Louis, and uh… was kind enough to give us a presentation back then.
So, uh… So those are my little historical footnotes. Let's check in there.
Yeah.
I don't remember the presentation, but I do remember the pizza.
That's the important thing.
Yeah, if I could jump in there, I don't know… I think he told me one time, uh.
Richard, uh, did you have an Uncle Fred? Or did you know of Fred Schmidt, uh… You're using ham radio?
Oh, yes. He was in Hap Radio, my dad, Eugene, was also in ham radio.
Okay.
And my late brother, Tom. Um, whose funeral I was in town for on…
Saturday. He was also, um, a ham user.
Okay. Yeah, I knew Fred. Uh, I always considered him a good friend. He was a good guy.
Yeah, so I had dinner with his, uh, many children, they were quite great.
And we went down one time. Him and one of his brothers, I think, who said he built a cabin or whatever, and I forget… Oh, yeah.
Okay, yeah, let me jump off of here. I just, uh, went.
Acknowledge it.
I was gonna ask if anybody got… I was gonna ask if anybody had experience using, uh, para-virtualized clocks.
On x86 VMs. Um, it's a way of supporting time synchronization, particularly.
With, uh, systems. Um… that have timestamp counters.
As a clock source. Many CPUs do now.
I had never considered a pair of virtualized clock. But I'm currently in the background waiting on.
A brand new Zen… virtualization host.
To, uh, build… And I will certainly be looking into.
Para-virtualized clocks, if it supports it. Um, I know para-virtualization is an interesting thing.
And not as well supported across all the. Common hypervisors.
Yeah, and to find out if your hardware supports, um.
The systems I'm using on are… older boxes that almost certainly don't have it.
You know, that fell off the truck on the way to e-waste type thing.
Uh, that's too bad. I was gonna say, if you do a CAT, a proc CPU info and grow up, uh.
Constant underscore TSC, you can see if your CPO has a constant.
Give me more. Charlie, Sam, Tom?
Timestamp calendar.
Correct. Constant underscore. Tom, Sam, Charlie.
Working. No, it does not.
But I'm not surprised on this thing.
I should say that… Gary Meyer's been very, very active in all this.
Is it by chance… Tom Sam Charlie?
Yes. Const… constant underscore TSC.
Okay, I'd search for Charlie Sam Tonff. Yes, this does have… constant underscore.
Tom, Sam, Charlie. Hmm… very, very, very interesting.
10… 10…
Well, Gary, I've got to say, this has been one… well, is one of the better and more interesting meetings I've attended.
Yeah. If I could, I just wanted to put in a thank you to Gary Maher for inviting me to this.
Um, presentation. Gary asked me… If I would be willing to come back and give a talk, because we're doing completely new networks, all of our equipment has been upgraded, and all of our NTP servers have been redesigned.
Um, I've been designing NTP servers for the Naval Observatory for.
31 years. But unfortunately, we've reached a level of…
Um, security classification that I'm no longer allowed to divulge any of the.
Interesting details about our hardware. That's… just too bad.
Is it American-made or Chinese-made?
No, I mean, I can say that much of the hardware is stuff that we lifted off of spacecraft from Alien 51.
Area 51. Um… But one of the things that it's nice to announce is that we will soon be supporting NTS.
Um, the network time security protocol, it's not. Encrypted, um… NTP, but it's an extension to NTP.
Um, that eliminates the need to store symmetric keys. On servers, because it's completely stateless for servers.
Has anyone… been familiar with NTP, NTS.
I am not. I'm draw… wondering if there's a paralysis. Our parallelism… with, like, DNSSEC.
For DNS, but signatures… for time, especially since you say it's not encrypted.
Signed for authentication.
Exactly. It's… it's used for… Authentication of servers to clients.
And it uses… TLS on port 4460.
And the way it works is that. Um, when you have set up, and you can set this up under Croni, if you have Croni on Linux.
A client initiates a key exchange with a server. And the server provides the client with a set of cookies that.
And close, um, a secret key that the server has. And when a client asks for time from the server.
It sends along one of these cookies, and the NTP server unpacks it, and uses the keys inside.
Um, to validate the signature. Of the person who sent the query, and then sign a response. So it's very much like…
Um, port 443 HTTPS. When the client gets back the NTP packet from the server, it has an extension on it.
Um, that includes the signature. And the client can validate the signature and accept the time.
So, unlike… symmetric key authentication, it's extremely scalable.
So we at USNO, we got. We do a billion, uh, NDP exchanges per day.
On the public servers. So, we have not been able to allow authenticated NTP on the public side.
To enable people to. Avoid Chinese man-in-the-middle attacks, but with NTS, it's completely scalable, and we will be offering.
Nts, both on the public side and on our. Are, uh, DISA DOD networks as well.
So that will give people. Um, a great deal of security, which is going to be important.
Because of… are changing relationship with our friends, the Chinese.
Very, very, very interesting.
Oh, and one more thing I should say is that. The client only asks for these cookies once.
And keeps them. And when the client turns himself off and turns himself back on, he's got this.
Cookie store, um, you saw Andrew showed one of the paths where the NTS.
Cookies are stored in his configuration. So, it only has to do the TLS on TCP port 4460 once.
It gets the cookies from our server. And then it switches back to UDP123 for ordinary NTP, and that continues on.
Oh.
You know, as long as you want.
Okay, I thought it was going to be a new protocol, but it's really an extension to the existing.
And plus a side helper protocol.
Sounds like.
And unfortunately, it's not in NTPD. But it is an NTP SAC, which is a different, um…
Version of NTP put out by another group. And it comes for free within Crony.
Which is the default, um… package that's on Red Hat and some of the other.
Linux is today.
Well, if you ever end up. Being given permission to give a, uh…
Watered down, or shall I say, selective availability version. Of some history about the infrastructure you've worked on, I would love.
To hear it.
And if you ever have the opportunity to come out to Washington, I'm sure we can arrange a tour under certain circumstances.
Group trip…
Okay, I'm gonna say goodbye and sign off before I get roped into anything. Thank you very much, everybody.
Smart man.
Thank you, sir. Great.
Hey, thank you. Thank you very, very much, Richard.
Thank you.
And also, thank you, Andrew. Also very, very, very interesting.
Presentation. Um… for better or worse, it is going to spur me to try some things.
And a big thank you to Andrew as he kind of mentioned. We had thought we had someone lined up to speak this month, and, uh.
Uh, yep, last week, it was like, okay, are you back from Labor Day yet? Or are you back for… are you still on vacation? So, yeah, we kind of had a fall-through there, apparently, and uh… Uh, Andrew was nice enough on, uh, just, uh, some quick notice at last.
Tweaked to… to step in and, uh. Do this as a quick fill-in for us.
Um, although, as he said, he'd been, uh. You've done a similar talk earlier, so I knew he had access to some good information, but uh…
Uh, he spent his weekend polishing it up, obviously. So, thank you very much again, Andrew.
And thank you all for coming this evening. Anybody have any other comments, questions, anything they need to bring up, point out?
Seems to be…
That silence I take as a no. I… I suggest we, uh, end the recording.
Just because we end the recording doesn't mean people have to leave, though.
I suppose so.
Bookend.
Yep. Um, so, um, but as we've said earlier, uh. Uh, when we're off our recording, other people could still be recording, so be aware of it.
Uh, the bookend for our recording, uh, this was Wednesday, September the 10th, uh, 2025.
It was the monthly general meeting of the SLUG St. Louis Unix.
St. Louis Linux Unix Users Group. Our main… Each of the night was Andrew Denner, who is, uh, talking to us about, uh, network time protocol and setting, uh, setting time on your Linux systems.
Uh, our, uh, uh, basic tutorial earlier this evening. Was, uh, uh, uh, Ken, um.
Help. Stan.
In Johnson. Ken… Ken Johnson.
Ken's last name. Johnson. Yeah, Ken Johnson, I'm sorry, I… my brain is obviously scrambled. So, uh…
Yeah, Ken Johnson, uh, and uh, Ken did a, uh, tutorial for us.
On, uh… I… my brain really is scrambled tonight. I've been on MTP so long, I've already forgotten, what was our tutorial on?
Ben Johnson's presentation was network diagnostics. From a mobile phone.
Sam. Tutorial.
Ah. Network… network diagnostic tools from an Android mobile phone. He gave us, uh.
Uh, at least four, uh, really nice tools he was using. And, uh, so it was… I remember it was a very good tutorial, I just didn't, uh… blanking on here. As I say, it's been a long day, long week.
If only it was recorded. Go back and put it on there.
So, thank you all for coming tonight. Uh, we're turning off the recording.